<div dir="ltr">Hi<div><br>Thanks for the logs and config files.</div><div>As per the logs and pgpool.conf files, This is what is happening.</div><div><br></div><div>You have health check disabled on all Pgpool-II nodes, So only way to detect the backend failure is through fail_over_on_backend error( which only works when client connection</div><div>detects the error) . But since the clients are only connecting to the master Pgpool-II node, so only master Pgpool-II node can notice the backend PostgreSQL node failure</div><div>and because of consensus requirement it will keep waiting for the detection of backend failure by other Pgpool-II nodes, Which never arrives because other two Pgpool-II nodes</div><div>are sitting idle and didn't detected the error.</div><div>So you either need to enable the health check on all pgpool-II nodes ( Which is the recommended setting for HA) or just disable the consensus requirements (as you did when failover</div><div>was working fine)</div><div><br></div><div>Thanks</div><div>Best Rgeards</div><div>Muhammad Usama<br><div class="gmail_extra"><br><div class="gmail_quote">On Tue, May 8, 2018 at 7:54 PM, Vlad G <span dir="ltr"><<a href="mailto:omenvlad@gmail.com" target="_blank">omenvlad@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word;line-break:after-white-space">Hey Guys. <div>Thank you for your answer.</div><div>I attached the configuration files of pgpool and logs.</div><div>I hope you сan help. </div></div></blockquote><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word;line-break:after-white-space"><div><br></div><div></div></div><br><div style="word-wrap:break-word;line-break:after-white-space"><div></div></div><br><div style="word-wrap:break-word;line-break:after-white-space"><div></div></div><br><div style="word-wrap:break-word;line-break:after-white-space"><div></div></div><br><div style="word-wrap:break-word;line-break:after-white-space"><div></div></div><br><div style="word-wrap:break-word;line-break:after-white-space"><div></div><div><br></div><div><div style="margin:0px;font-stretch:normal;line-height:normal"><span style="font-kerning:none">Best regards,</span></div><div style="margin:0px;font-stretch:normal;line-height:normal"><span style="font-kerning:none">Vladyslav</span></div></div><div><br></div><div><div><br><blockquote type="cite"><div>On May 7, 2018, at 16:05, Muhammad Usama <<a href="mailto:m.usama@gmail.com" target="_blank">m.usama@gmail.com</a>> wrote:</div><br class="m_-8382865875936540422Apple-interchange-newline"><div><div dir="ltr">Hi<div><br></div><div>From the log snippet you shared it seems that the the failure was never detected by the other Pgpool-II node, Can you please share the pgpoo.conf files and log files for all Pgpool nodes.</div><div><br></div><div>Thanks</div><div>Best Regards</div><div>Muhammad Usama<br><div class="gmail_extra"><br><div class="gmail_quote">On Thu, May 3, 2018 at 5:20 PM, Vlad G <span dir="ltr"><<a href="mailto:omenvlad@gmail.com" target="_blank">omenvlad@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hey Guys,<br>
I have a cluster with Pgpool-II-pg96-3.7.3 and postgresql-9.6.<br>
(3 x pgpool and 3 x postgresql<br>
The same scheme as:<br>
<a href="http://www.pgpool.net/docs/latest/en/html/example-cluster.html" rel="noreferrer" target="_blank">http://www.pgpool.net/docs/lat<wbr>est/en/html/example-cluster.<wbr>html</a><br>
<br>
When master node of postgresql (pgpoolpsql-1) goes down the master node of pgpool ( pgpool-1) does not get second vote from one of the standby pgpool nodes (pgpool-2 and pgpool-3). <br>
<br>
If I set:<br>
failover_require_consensus = off<br>
Everything works fine.<br>
<br>
May 03 13:02:45 pgpool-1 pgpool[24216]: 2018-05-03 13:02:45: pid 24237: LOG: failed to connect to PostgreSQL server on "pgpoolpsql-1:5432", getsockopt() detected error "Connection refused"<br>
May 03 13:02:45 pgpool-1 pgpool[24216]: 2018-05-03 13:02:45: pid 24237: LOG: received degenerate backend request for node_id: 0 from pid [24237]<br>
May 03 13:02:45 pgpool-1 pgpool[24216]: 2018-05-03 13:02:45: pid 24217: LOG: new IPC connection received<br>
May 03 13:02:45 pgpool-1 pgpool[24216]: 2018-05-03 13:02:45: pid 24217: LOG: watchdog received the failover command from local pgpool-II on IPC interface<br>
May 03 13:02:45 pgpool-1 pgpool[24216]: 2018-05-03 13:02:45: pid 24217: LOG: watchdog is processing the failover command [DEGENERATE_BACKEND_REQUEST] received from local pgpool-II on IPC interface<br>
May 03 13:02:45 pgpool-1 pgpool[24216]: 2018-05-03 13:02:45: pid 24217: LOG: failover requires the majority vote, waiting for consensus<br>
May 03 13:02:45 pgpool-1 pgpool[24216]: 2018-05-03 13:02:45: pid 24217: DETAIL: failover request noted<br>
May 03 13:02:45 pgpool-1 pgpool[24216]: 2018-05-03 13:02:45: pid 24217: LOG: failover command [DEGENERATE_BACKEND_REQUEST] request from pgpool-II node "pgpool-1:9999 Linux pgpool-1" is queued, waiting for the confirmation from other nodes<br>
May 03 13:02:45 pgpool-1 pgpool[24216]: 2018-05-03 13:02:45: pid 24237: LOG: degenerate backend request for node_id: 0 from pid [24237], will be handled by watchdog, which is building consensus for request<br>
May 03 13:02:45 pgpool-1 pgpool[24216]: 2018-05-03 13:02:45: pid 24237: FATAL: failed to create a backend connection<br>
May 03 13:02:45 pgpool-1 pgpool[24216]: 2018-05-03 13:02:45: pid 24237: DETAIL: executing failover on backend<br>
May 03 13:02:45 pgpool-1 pgpool[24216]: 2018-05-03 13:02:45: pid 24216: LOG: child process with pid: 24237 exits with status 256<br>
May 03 13:02:45 pgpool-1 pgpool[24216]: 2018-05-03 13:02:45: pid 24216: LOG: fork a new child process with pid: 24268<br>
May 03 13:02:46 pgpool-1 pgpool[24216]: 2018-05-03 13:02:46: pid 24228: LOG: failed to connect to PostgreSQL server on "pgpoolpsql-1:5432", getsockopt() detected error "Connection refused"<br>
May 03 13:02:46 pgpool-1 pgpool[24216]: 2018-05-03 13:02:46: pid 24228: LOG: received degenerate backend request for node_id: 0 from pid [24228]<br>
May 03 13:02:46 pgpool-1 pgpool[24216]: 2018-05-03 13:02:46: pid 24217: LOG: new IPC connection received<br>
May 03 13:02:46 pgpool-1 pgpool[24216]: 2018-05-03 13:02:46: pid 24217: LOG: watchdog received the failover command from local pgpool-II on IPC interface<br>
May 03 13:02:46 pgpool-1 pgpool[24216]: 2018-05-03 13:02:46: pid 24217: LOG: watchdog is processing the failover command [DEGENERATE_BACKEND_REQUEST] received from local pgpool-II on IPC interface<br>
May 03 13:02:46 pgpool-1 pgpool[24216]: 2018-05-03 13:02:46: pid 24217: LOG: Duplicate failover request from "pgpool-1:9999 Linux pgpool-1" node<br>
May 03 13:02:46 pgpool-1 pgpool[24216]: 2018-05-03 13:02:46: pid 24217: DETAIL: request ignored<br>
May 03 13:02:46 pgpool-1 pgpool[24216]: 2018-05-03 13:02:46: pid 24217: LOG: failover requires the majority vote, waiting for consensus<br>
May 03 13:02:46 pgpool-1 pgpool[24216]: 2018-05-03 13:02:46: pid 24217: DETAIL: failover request noted<br>
May 03 13:02:46 pgpool-1 pgpool[24216]: 2018-05-03 13:02:46: pid 24228: LOG: degenerate backend request for 1 node(s) from pid [24228], is changed to quarantine node request by watchdog<br>
May 03 13:02:46 pgpool-1 pgpool[24216]: 2018-05-03 13:02:46: pid 24228: DETAIL: watchdog is taking time to build consensus<br>
May 03 13:02:46 pgpool-1 pgpool[24216]: 2018-05-03 13:02:46: pid 24228: FATAL: failed to create a backend connection<br>
May 03 13:02:46 pgpool-1 pgpool[24216]: 2018-05-03 13:02:46: pid 24228: DETAIL: executing failover on backend<br>
May 03 13:02:46 pgpool-1 pgpool[24216]: 2018-05-03 13:02:46: pid 24216: LOG: Pgpool-II parent process has received failover request<br>
May 03 13:02:46 pgpool-1 pgpool[24216]: 2018-05-03 13:02:46: pid 24217: LOG: new IPC connection received<br>
May 03 13:02:46 pgpool-1 pgpool[24216]: 2018-05-03 13:02:46: pid 24217: LOG: received the failover indication from Pgpool-II on IPC interface<br>
May 03 13:02:46 pgpool-1 pgpool[24216]: 2018-05-03 13:02:46: pid 24217: LOG: watchdog is informed of failover end by the main process<br>
May 03 13:02:46 pgpool-1 pgpool[24216]: 2018-05-03 13:02:46: pid 24216: LOG: starting quarantine. shutdown host pgpoolpsql-1(5432)<br>
May 03 13:02:46 pgpool-1 pgpool[24216]: 2018-05-03 13:02:46: pid 24216: LOG: Restart all children<br>
May 03 13:02:46 pgpool-1 pgpool[24216]: 2018-05-03 13:02:46: pid 24216: LOG: failover: set new primary node: -1<br>
May 03 13:02:46 pgpool-1 pgpool[24216]: 2018-05-03 13:02:46: pid 24216: LOG: failover: set new master node: 1<br>
May 03 13:02:46 pgpool-1 pgpool[24216]: 2018-05-03 13:02:46: pid 24252: LOG: worker process received restart request<br>
May 03 13:02:46 pgpool-1 pgpool[24216]: 2018-05-03 13:02:46: pid 24217: LOG: new IPC connection received<br>
May 03 13:02:46 pgpool-1 pgpool[24216]: 2018-05-03 13:02:46: pid 24217: LOG: received the failover indication from Pgpool-II on IPC interface<br>
May 03 13:02:46 pgpool-1 pgpool[24216]: 2018-05-03 13:02:46: pid 24217: LOG: watchdog is informed of failover start by the main process<br>
May 03 13:02:46 pgpool-1 pgpool[24216]: quarantine done. shutdown host pgpoolpsql-1(5432)2018-05-03 13:02:46: pid 24216: LOG: quarantine done. shutdown host pgpoolpsql-1(5432)<br>
May 03 13:02:47 pgpool-1 pgpool[24216]: 2018-05-03 13:02:47: pid 24251: LOG: restart request received in pcp child process<br>
May 03 13:02:47 pgpool-1 pgpool[24216]: 2018-05-03 13:02:47: pid 24216: LOG: PCP child 24251 exits with status 0 in failover()<br>
May 03 13:02:47 pgpool-1 pgpool[24216]: 2018-05-03 13:02:47: pid 24216: LOG: fork a new PCP child pid 24301 in failover()<br>
May 03 13:02:47 pgpool-1 pgpool[24216]: 2018-05-03 13:02:47: pid 24216: LOG: child process with pid: 24219 exits with status 0<br>
May 03 13:02:47 pgpool-1 pgpool[24216]: 2018-05-03 13:02:47: pid 24216: LOG: child process with pid: 24219 exited with success and will not be restarted<br>
May 03 13:02:47 pgpool-1 pgpool[24216]: 2018-05-03 13:02:47: pid 24216: LOG: child process with pid: 24220 exits with status 0<br>
May 03 13:02:47 pgpool-1 pgpool[24216]: 2018-05-03 13:02:47: pid 24216: LOG: child process with pid: 24220 exited with success and will not be restarted<br>
May 03 13:02:47 pgpool-1 pgpool[24216]: 2018-05-03 13:02:47: pid 24216: LOG: child process with pid: 24221 exits with status 0<br>
<br>
Around a month ago it woked fine (It seems I tested it on pgpool-3.7.2), but now it does not work. Could you tell me some parameters what it depends on or you have other thoughts. <br>
<br>
Best regards,<br>
Vladyslav<br>
<br>
______________________________<wbr>_________________<br>
pgpool-general mailing list<br>
<a href="mailto:pgpool-general@pgpool.net" target="_blank">pgpool-general@pgpool.net</a><br>
<a href="http://www.pgpool.net/mailman/listinfo/pgpool-general" rel="noreferrer" target="_blank">http://www.pgpool.net/mailman/<wbr>listinfo/pgpool-general</a><br>
</blockquote></div><br></div></div></div>
</div></blockquote></div><br></div></div><br>______________________________<wbr>_________________<br>
pgpool-general mailing list<br>
<a href="mailto:pgpool-general@pgpool.net">pgpool-general@pgpool.net</a><br>
<a href="http://www.pgpool.net/mailman/listinfo/pgpool-general" rel="noreferrer" target="_blank">http://www.pgpool.net/mailman/<wbr>listinfo/pgpool-general</a><br>
<br></blockquote></div><br></div></div></div>