<div dir="ltr">Hi<div><br></div><div>From the log snippet you shared it seems that the the failure was never detected by the other Pgpool-II node, Can you please share the pgpoo.conf files and log files for all Pgpool nodes.</div><div><br></div><div>Thanks</div><div>Best Regards</div><div>Muhammad Usama<br><div class="gmail_extra"><br><div class="gmail_quote">On Thu, May 3, 2018 at 5:20 PM, Vlad G <span dir="ltr"><<a href="mailto:omenvlad@gmail.com" target="_blank">omenvlad@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hey Guys,<br>
I have a cluster with Pgpool-II-pg96-3.7.3 and postgresql-9.6.<br>
(3 x pgpool and 3 x postgresql<br>
The same scheme as:<br>
<a href="http://www.pgpool.net/docs/latest/en/html/example-cluster.html" rel="noreferrer" target="_blank">http://www.pgpool.net/docs/<wbr>latest/en/html/example-<wbr>cluster.html</a><br>
<br>
When master node of postgresql (pgpoolpsql-1) goes down the master node of pgpool ( pgpool-1) does not get second vote from one of the standby pgpool nodes (pgpool-2 and pgpool-3). <br>
<br>
If I set:<br>
failover_require_consensus = off<br>
Everything works fine.<br>
<br>
May 03 13:02:45 pgpool-1 pgpool[24216]: 2018-05-03 13:02:45: pid 24237: LOG: failed to connect to PostgreSQL server on "pgpoolpsql-1:5432", getsockopt() detected error "Connection refused"<br>
May 03 13:02:45 pgpool-1 pgpool[24216]: 2018-05-03 13:02:45: pid 24237: LOG: received degenerate backend request for node_id: 0 from pid [24237]<br>
May 03 13:02:45 pgpool-1 pgpool[24216]: 2018-05-03 13:02:45: pid 24217: LOG: new IPC connection received<br>
May 03 13:02:45 pgpool-1 pgpool[24216]: 2018-05-03 13:02:45: pid 24217: LOG: watchdog received the failover command from local pgpool-II on IPC interface<br>
May 03 13:02:45 pgpool-1 pgpool[24216]: 2018-05-03 13:02:45: pid 24217: LOG: watchdog is processing the failover command [DEGENERATE_BACKEND_REQUEST] received from local pgpool-II on IPC interface<br>
May 03 13:02:45 pgpool-1 pgpool[24216]: 2018-05-03 13:02:45: pid 24217: LOG: failover requires the majority vote, waiting for consensus<br>
May 03 13:02:45 pgpool-1 pgpool[24216]: 2018-05-03 13:02:45: pid 24217: DETAIL: failover request noted<br>
May 03 13:02:45 pgpool-1 pgpool[24216]: 2018-05-03 13:02:45: pid 24217: LOG: failover command [DEGENERATE_BACKEND_REQUEST] request from pgpool-II node "pgpool-1:9999 Linux pgpool-1" is queued, waiting for the confirmation from other nodes<br>
May 03 13:02:45 pgpool-1 pgpool[24216]: 2018-05-03 13:02:45: pid 24237: LOG: degenerate backend request for node_id: 0 from pid [24237], will be handled by watchdog, which is building consensus for request<br>
May 03 13:02:45 pgpool-1 pgpool[24216]: 2018-05-03 13:02:45: pid 24237: FATAL: failed to create a backend connection<br>
May 03 13:02:45 pgpool-1 pgpool[24216]: 2018-05-03 13:02:45: pid 24237: DETAIL: executing failover on backend<br>
May 03 13:02:45 pgpool-1 pgpool[24216]: 2018-05-03 13:02:45: pid 24216: LOG: child process with pid: 24237 exits with status 256<br>
May 03 13:02:45 pgpool-1 pgpool[24216]: 2018-05-03 13:02:45: pid 24216: LOG: fork a new child process with pid: 24268<br>
May 03 13:02:46 pgpool-1 pgpool[24216]: 2018-05-03 13:02:46: pid 24228: LOG: failed to connect to PostgreSQL server on "pgpoolpsql-1:5432", getsockopt() detected error "Connection refused"<br>
May 03 13:02:46 pgpool-1 pgpool[24216]: 2018-05-03 13:02:46: pid 24228: LOG: received degenerate backend request for node_id: 0 from pid [24228]<br>
May 03 13:02:46 pgpool-1 pgpool[24216]: 2018-05-03 13:02:46: pid 24217: LOG: new IPC connection received<br>
May 03 13:02:46 pgpool-1 pgpool[24216]: 2018-05-03 13:02:46: pid 24217: LOG: watchdog received the failover command from local pgpool-II on IPC interface<br>
May 03 13:02:46 pgpool-1 pgpool[24216]: 2018-05-03 13:02:46: pid 24217: LOG: watchdog is processing the failover command [DEGENERATE_BACKEND_REQUEST] received from local pgpool-II on IPC interface<br>
May 03 13:02:46 pgpool-1 pgpool[24216]: 2018-05-03 13:02:46: pid 24217: LOG: Duplicate failover request from "pgpool-1:9999 Linux pgpool-1" node<br>
May 03 13:02:46 pgpool-1 pgpool[24216]: 2018-05-03 13:02:46: pid 24217: DETAIL: request ignored<br>
May 03 13:02:46 pgpool-1 pgpool[24216]: 2018-05-03 13:02:46: pid 24217: LOG: failover requires the majority vote, waiting for consensus<br>
May 03 13:02:46 pgpool-1 pgpool[24216]: 2018-05-03 13:02:46: pid 24217: DETAIL: failover request noted<br>
May 03 13:02:46 pgpool-1 pgpool[24216]: 2018-05-03 13:02:46: pid 24228: LOG: degenerate backend request for 1 node(s) from pid [24228], is changed to quarantine node request by watchdog<br>
May 03 13:02:46 pgpool-1 pgpool[24216]: 2018-05-03 13:02:46: pid 24228: DETAIL: watchdog is taking time to build consensus<br>
May 03 13:02:46 pgpool-1 pgpool[24216]: 2018-05-03 13:02:46: pid 24228: FATAL: failed to create a backend connection<br>
May 03 13:02:46 pgpool-1 pgpool[24216]: 2018-05-03 13:02:46: pid 24228: DETAIL: executing failover on backend<br>
May 03 13:02:46 pgpool-1 pgpool[24216]: 2018-05-03 13:02:46: pid 24216: LOG: Pgpool-II parent process has received failover request<br>
May 03 13:02:46 pgpool-1 pgpool[24216]: 2018-05-03 13:02:46: pid 24217: LOG: new IPC connection received<br>
May 03 13:02:46 pgpool-1 pgpool[24216]: 2018-05-03 13:02:46: pid 24217: LOG: received the failover indication from Pgpool-II on IPC interface<br>
May 03 13:02:46 pgpool-1 pgpool[24216]: 2018-05-03 13:02:46: pid 24217: LOG: watchdog is informed of failover end by the main process<br>
May 03 13:02:46 pgpool-1 pgpool[24216]: 2018-05-03 13:02:46: pid 24216: LOG: starting quarantine. shutdown host pgpoolpsql-1(5432)<br>
May 03 13:02:46 pgpool-1 pgpool[24216]: 2018-05-03 13:02:46: pid 24216: LOG: Restart all children<br>
May 03 13:02:46 pgpool-1 pgpool[24216]: 2018-05-03 13:02:46: pid 24216: LOG: failover: set new primary node: -1<br>
May 03 13:02:46 pgpool-1 pgpool[24216]: 2018-05-03 13:02:46: pid 24216: LOG: failover: set new master node: 1<br>
May 03 13:02:46 pgpool-1 pgpool[24216]: 2018-05-03 13:02:46: pid 24252: LOG: worker process received restart request<br>
May 03 13:02:46 pgpool-1 pgpool[24216]: 2018-05-03 13:02:46: pid 24217: LOG: new IPC connection received<br>
May 03 13:02:46 pgpool-1 pgpool[24216]: 2018-05-03 13:02:46: pid 24217: LOG: received the failover indication from Pgpool-II on IPC interface<br>
May 03 13:02:46 pgpool-1 pgpool[24216]: 2018-05-03 13:02:46: pid 24217: LOG: watchdog is informed of failover start by the main process<br>
May 03 13:02:46 pgpool-1 pgpool[24216]: quarantine done. shutdown host pgpoolpsql-1(5432)2018-05-03 13:02:46: pid 24216: LOG: quarantine done. shutdown host pgpoolpsql-1(5432)<br>
May 03 13:02:47 pgpool-1 pgpool[24216]: 2018-05-03 13:02:47: pid 24251: LOG: restart request received in pcp child process<br>
May 03 13:02:47 pgpool-1 pgpool[24216]: 2018-05-03 13:02:47: pid 24216: LOG: PCP child 24251 exits with status 0 in failover()<br>
May 03 13:02:47 pgpool-1 pgpool[24216]: 2018-05-03 13:02:47: pid 24216: LOG: fork a new PCP child pid 24301 in failover()<br>
May 03 13:02:47 pgpool-1 pgpool[24216]: 2018-05-03 13:02:47: pid 24216: LOG: child process with pid: 24219 exits with status 0<br>
May 03 13:02:47 pgpool-1 pgpool[24216]: 2018-05-03 13:02:47: pid 24216: LOG: child process with pid: 24219 exited with success and will not be restarted<br>
May 03 13:02:47 pgpool-1 pgpool[24216]: 2018-05-03 13:02:47: pid 24216: LOG: child process with pid: 24220 exits with status 0<br>
May 03 13:02:47 pgpool-1 pgpool[24216]: 2018-05-03 13:02:47: pid 24216: LOG: child process with pid: 24220 exited with success and will not be restarted<br>
May 03 13:02:47 pgpool-1 pgpool[24216]: 2018-05-03 13:02:47: pid 24216: LOG: child process with pid: 24221 exits with status 0<br>
<br>
Around a month ago it woked fine (It seems I tested it on pgpool-3.7.2), but now it does not work. Could you tell me some parameters what it depends on or you have other thoughts. <br>
<br>
Best regards,<br>
Vladyslav<br>
<br>
______________________________<wbr>_________________<br>
pgpool-general mailing list<br>
<a href="mailto:pgpool-general@pgpool.net">pgpool-general@pgpool.net</a><br>
<a href="http://www.pgpool.net/mailman/listinfo/pgpool-general" rel="noreferrer" target="_blank">http://www.pgpool.net/mailman/<wbr>listinfo/pgpool-general</a><br>
</blockquote></div><br></div></div></div>