<html><head></head><body>Any update on this?<br>Cheers,<br> Anssi<br><br><div class="gmail_quote">On 2 September 2020 13:16:59 EEST, Anssi Kanninen <anssi@iki.fi> wrote:<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<pre class="k9mail">Thanks Bo Peng,<br><br>Here are the "power off" logs with debugging enabled.<br><br>Cheers,<br> - Anssi Kanninen<br><br>Sep 2 11:20:13<br><br><br><br><br>On Wed, 2 Sep 2020, Bo Peng wrote:<br><br><blockquote class="gmail_quote" style="margin: 0pt 0pt 1ex 0.8ex; border-left: 1px solid #729fcf; padding-left: 1ex;"> Hi,<br><br> Thank you for sharing log and connfig files.<br><br> I can't see log like:<br><br> "LOG: watchdog: lifecheck started"<br><br> Watchdog lifecheck may not being performed properly.<br> Sometimes watchdog lifecheck takes time,<br> could you wait a while and check the watchdog status.<br><br> To check watchdog lifecheck behaviour details you need to enable debug mode.<br> Is possible to enable debug mode and try again?<br><br> To enable debug mode (If you installed pgpool from rpm):<br><br> vi /etc/sysconfig/pgpool<br><br> OPTS=" -D -n" => OPTS="-d -D -n"<br><br><br> On Tue, 1 Sep 2020 12:04:35 +0300 (FLE Daylight Time)<br> Anssi Kanninen <anssi@iki.fi> wrote:<br><br><blockquote class="gmail_quote" style="margin: 0pt 0pt 1ex 0.8ex; border-left: 1px solid #ad7fa8; padding-left: 1ex;"> Here are my configs and logs for nodes 0, 1 and 2 regarding the "power<br> off" problem.<br><br> CLEAN SHUTDOWN LOGS: pgpool-shutdown.log.nodeX<br><br> ***** Node status from node 0 after shutdown of node 1:<br><br> [node0]$ pcp_watchdog_info -w -h localhost<br> 3 NO centos8i3-int:5432 Linux centos8i3.localdomain centos8i3-int<br><br> centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int 5432 9000 7 STANDBY<br> centos8i2-int:5432 Linux centos8i2.localdomain centos8i2-int 5432 9000 10 SHUTDOWN<br> centos8i3-int:5432 Linux centos8i3.localdomain centos8i3-int 5432 9000 4 MASTER<br><br> ***** Node status from node 2 after shutdown of node 1:<br><br> [node2]$ pcp_watchdog_info -w -h localhost<br> 3 YES centos8i3-int:5432 Linux centos8i3.localdomain centos8i3-int<br><br> centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int 5432 9000 7 STANDBY<br> centos8i2-int:5432 Linux centos8i2.localdomain centos8i2-int 5432 9000 10 SHUTDOWN<br> centos8i3-int:5432 Linux centos8i3.localdomain centos8i3-int 5432 9000 4 MASTER<br><br> =====> Everything is ok<br><br><br> NON-CLEAN ("power switch off") SHUTDOWN LOGS: pgpool-poweroff.log.nodeX<br><br> ***** Node status from node 0 after power off of node 1:<br><br> [node0]$ pcp_watchdog_info -w -h localhost<br> 3 YES centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int<br><br> centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int 5432 9000 4 MASTER<br> centos8i2-int:5432 Linux centos8i2.localdomain centos8i2-int 5432 9000 8 LOST<br> centos8i3-int:5432 Linux centos8i3.localdomain centos8i3-int 5432 9000 7 STANDBY<br><br><br> ***** Node status from node 2 after power off of node 1:<br><br> [node2]$ pcp_watchdog_info -w -h localhost<br> 3 NO centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int<br><br> centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int 5432 9000 4 MASTER<br> centos8i2-int:5432 Linux centos8i2.localdomain centos8i2-int 5432 9000 7 STANDBY<br> centos8i3-int:5432 Linux centos8i3.localdomain centos8i3-int 5432 9000 7 STANDBY<br><br> =====> Node 2 is thinks node 1 is still in standby mode<br><br> Cheers!<br> - Anssi Kanninen<br><br><br> On Mon, 31 Aug 2020, Bo Peng wrote:<br><br><blockquote class="gmail_quote" style="margin: 0pt 0pt 1ex 0.8ex; border-left: 1px solid #8ae234; padding-left: 1ex;"> Hello,<br><br> On Fri, 28 Aug 2020 12:27:48 +0300 (FLE Daylight Time)<br> Anssi Kanninen <anssi@iki.fi> wrote:<br><br><blockquote class="gmail_quote" style="margin: 0pt 0pt 1ex 0.8ex; border-left: 1px solid #fcaf3e; padding-left: 1ex;"> Hi everyone!<br><br> I'm having a problem with information exchange between PgPool instances. I<br> have 3 nodes, each containing one DB backend instance and one PgPool<br> instance.<br><br> If I shut down one standby node cleanly, everything seems to go ok. The<br> master PgPool notices that and informs the remaining standby PgPool about<br> it.<br><br> But the situation changes if a standby node just vahishes from the network<br> by powering it off without clean shutdown. The master PgPool marks the<br> node as "lost" but the remaining standby PgPool still thinks we are having<br> another standby PgPool. It doesn't get any information about a lost node.<br></blockquote><br> How did you shutdown pgpool node?<br> Could you share the pgpool.log of each node?<br><br><blockquote class="gmail_quote" style="margin: 0pt 0pt 1ex 0.8ex; border-left: 1px solid #fcaf3e; padding-left: 1ex;"> Here it goes. In the example I'm checking the statuses by connecting each<br> node with pcp_watchdog_info . I have sorted the results by node hostname.<br><br> Nodes are:<br> * ID 0 (centos8i1-int)<br> * ID 1 (centos8i2-int)<br> * ID 2 (centos8i3-int).<br><br> ***** INITIAL SETUP *****<br><br> $ pcp_watchdog_info -w -h centos8i1-int<br> 3 YES centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int<br><br> centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int 5432 9000 4 MASTER<br> centos8i2-int:5432 Linux centos8i2.localdomain centos8i2-int 5432 9000 7 STANDBY<br> centos8i3-int:5432 Linux centos8i3.localdomain centos8i3-int 5432 9000 7 STANDBY<br><br> $ pcp_watchdog_info -w -h centos8i2-int<br> 3 NO centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int<br><br> centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int 5432 9000 4 MASTER<br> centos8i2-int:5432 Linux centos8i2.localdomain centos8i2-int 5432 9000 7 STANDBY<br> centos8i3-int:5432 Linux centos8i3.localdomain centos8i3-int 5432 9000 7 STANDBY<br><br> $ pcp_watchdog_info -w -h centos8i3-int<br> 3 NO centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int<br><br> centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int 5432 9000 4 MASTER<br> centos8i2-int:5432 Linux centos8i2.localdomain centos8i2-int 5432 9000 7 STANDBY<br> centos8i3-int:5432 Linux centos8i3.localdomain centos8i3-int 5432 9000 7 STANDBY<br><br> ***** SHUTDOWN node ID 1 *****<br><br> $ pcp_watchdog_info -w -h centos8i1-int<br> 3 YES centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int<br><br> centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int 5432 9000 4 MASTER<br> centos8i2-int:5432 Linux centos8i2.localdomain centos8i2-int 5432 9000 10 SHUTDOWN<br> centos8i3-int:5432 Linux centos8i3.localdomain centos8i3-int 5432 9000 7 STANDBY<br><br> $ pcp_watchdog_info -w -h centos8i3-int<br> 3 NO centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int<br><br> centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int 5432 9000 4 MASTER<br> centos8i2-int:5432 Linux centos8i2.localdomain centos8i2-int 5432 9000 10 SHUTDOWN<br> centos8i3-int:5432 Linux centos8i3.localdomain centos8i3-int 5432 9000 7 STANDBY<br><br> ***** RESTART node ID 1 *****<br><br> $ pcp_watchdog_info -w -h centos8i1-int<br> 3 YES centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int<br><br> centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int 5432 9000 4 MASTER<br> centos8i2-int:5432 Linux centos8i2.localdomain centos8i2-int 5432 9000 7 STANDBY<br> centos8i3-int:5432 Linux centos8i3.localdomain centos8i3-int 5432 9000 7 STANDBY<br><br> $ pcp_watchdog_info -w -h centos8i2-int<br> 3 NO centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int<br><br> centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int 5432 9000 4 MASTER<br> centos8i2-int:5432 Linux centos8i2.localdomain centos8i2-int 5432 9000 7 STANDBY<br> centos8i3-int:5432 Linux centos8i3.localdomain centos8i3-int 5432 9000 7 STANDBY<br><br> $ pcp_watchdog_info -w -h centos8i3-int<br> 3 NO centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int<br><br> centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int 5432 9000 4 MASTER<br> centos8i2-int:5432 Linux centos8i2.localdomain centos8i2-int 5432 9000 7 STANDBY<br> centos8i3-int:5432 Linux centos8i3.localdomain centos8i3-int 5432 9000 7 STANDBY<br><br> ***** POWER OFF node ID 1 *****<br><br> $ pcp_watchdog_info -w -h centos8i1-int<br> 3 YES centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int<br><br> centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int 5432 9000 4 MASTER<br> centos8i2-int:5432 Linux centos8i2.localdomain centos8i2-int 5432 9000 8 LOST<br> centos8i3-int:5432 Linux centos8i3.localdomain centos8i3-int 5432 9000 7 STANDBY<br><br> $ pcp_watchdog_info -w -h centos8i3-int<br> 3 NO centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int<br><br> centos8i1-int:5432 Linux centos8i1.localdomain centos8i1-int 5432 9000 4 MASTER<br> centos8i2-int:5432 Linux centos8i2.localdomain centos8i2-int 5432 9000 7 STANDBY<br> centos8i3-int:5432 Linux centos8i3.localdomain centos8i3-int 5432 9000 7 STANDBY<br><br><br> Best regards,<br> Anssi Kanninen<br><br><br><br><br> --<br> anssi@iki.fi<hr> pgpool-general mailing list<br> pgpool-general@pgpool.net<br> <a href="http://www.pgpool.net/mailman/listinfo/pgpool-general">http://www.pgpool.net/mailman/listinfo/pgpool-general</a><br></blockquote><br><br> --<br> Bo Peng <pengbo@sraoss.co.jp><br> SRA OSS, Inc. Japan<hr> pgpool-general mailing list<br> pgpool-general@pgpool.net<br> <a href="http://www.pgpool.net/mailman/listinfo/pgpool-general">http://www.pgpool.net/mailman/listinfo/pgpool-general</a><br><br></blockquote><br> --<br> anssi@iki.fi<br></blockquote><br><br> -- <br> Bo Peng <pengbo@sraoss.co.jp><br> SRA OSS, Inc. Japan<hr> pgpool-general mailing list<br> pgpool-general@pgpool.net<br> <a href="http://www.pgpool.net/mailman/listinfo/pgpool-general">http://www.pgpool.net/mailman/listinfo/pgpool-general</a><br><br></blockquote></pre></blockquote></div></body></html>