<div dir="ltr"><div><div><div>Hi Tatsuo,<br></div>I shutdown one watchdog instead of both, I can't reproduce the problem.<br></div><br></div><div>Here is the details:<br></div><div>server0 pgpool watchdog is disabled<br>
</div><div>server1 pgpool watchdog is enabled and it is a primary database for streaming replication, failover & failback works just fine; except that the virtual ip will not be migrated to the other pgpool server because<br>
</div><div>watchdog on server0 is not running.<br><br></div><div>FYI: as i reported on the other email thread, running watchdog on both server will not allow me to failover & failback more than once which I am still looking for root cause.<br>
</div><div><br></div><div>1) both node shows pool_nodes as state 2<br></div><div>2) shutdown database on server1, then cause the DB to failover to server0, server0 is now primary<br></div><div>3) execute pcp_recovery on server0 to bring the server1 failed database back online and connects to server0 as a standby; however, pool_nodes on server1 shows the following:<br>
[root@server1 data]# psql -c "show pool_nodes" -p 9999<br> node_id | hostname | port | status | lb_weight | role<br>---------+----------+------+--------+-----------+---------<br> 0 | server0 | 5432 | 2 | 0.500000 | primary<br>
1 | server1 | 5432 | 3 | 0.500000 | standby<br>(2 rows)<br><br></div><div>As shows, server1 pgpool think itself as in state 3.<br></div><div>Replication however is working fine.<br><br></div><div>4) i have to execute pcp_attach_node on server1 to bring its pool_nodes state to 2, however, server0 pool_nodes info about server1 becomes 3. see below for both servers output:<br>
[root@server1 data]# psql -c "show pool_nodes" -p 9999<br> node_id | hostname | port | status | lb_weight | role<br>---------+----------+------+--------+-----------+---------<br> 0 | server0 | 5432 | 2 | 0.500000 | primary<br>
1 | server1 | 5432 | 2 | 0.500000 | standby<br><br>[root@server0 ~]# psql -c "show pool_nodes" -p 9999<br> node_id | hostname | port | status | lb_weight | role<br>---------+----------+------+--------+-----------+---------<br>
0 | server0 | 5432 | 2 | 0.500000 | primary<br> 1 | server1 | 5432 | 3 | 0.500000 | standby<br><br><br></div><div>5) execute the following command on server1 will bring the server1 status to 2 on both node:<br>
/usr/local/bin/pcp_attach_node 10 server0 9898 pgpool [passwd] 1<br><br>[root@server1 data]# psql -c "show pool_nodes" -p 9999<br> node_id | hostname | port | status | lb_weight | role<br>---------+----------+------+--------+-----------+---------<br>
0 | server0 | 5432 | 2 | 0.500000 | primary<br> 1 | server1 | 5432 | 2 | 0.500000 | standby<br><br>[root@server0 ~]# psql -c "show pool_nodes" -p 9999<br> node_id | hostname | port | status | lb_weight | role<br>
---------+----------+------+--------+-----------+---------<br> 0 | server0 | 5432 | 2 | 0.500000 | primary<br> 1 | server1 | 5432 | 2 | 0.500000 | standby<br><br></div><div>Please advise the next step.<br>
<br>Thanks~<br>Ning<br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Sun, Mar 3, 2013 at 6:03 PM, Tatsuo Ishii <span dir="ltr"><<a href="mailto:ishii@postgresql.org" target="_blank">ishii@postgresql.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im">> Mar 1 10:45:31 server1 pgpool[3339]: bind(:) failed. reason: Success<br>
<br>
</div>This error messge seems pretty strange. ":" should be something like<br>
"/tmp/.s.PGSQL.9898". Also it's weired because 2failed. reason:<br>
Success". To isolate the problem, can please disable watchdog and try<br>
again?<br>
--<br>
Tatsuo Ishii<br>
SRA OSS, Inc. Japan<br>
English: <a href="http://www.sraoss.co.jp/index_en.php" target="_blank">http://www.sraoss.co.jp/index_en.php</a><br>
Japanese: <a href="http://www.sraoss.co.jp" target="_blank">http://www.sraoss.co.jp</a><br>
<div class="HOEnZb"><div class="h5"><br>
<br>
> Hi All,<br>
> After upgrade to pgPool-II 3.2.3 and I tested my failover/ failback setup,<br>
> and start / stop pgpool mutlip times, I see one of the pgpool goes in to an<br>
> unrecoverable state.<br>
><br>
> Mar 1 10:45:25 server1 pgpool[3007]: received smart shutdown request<br>
> Mar 1 10:45:25 server1 pgpool[3007]: watchdog_pid: 3010<br>
> Mar 1 10:45:31 server1 pgpool[3338]: wd_chk_sticky: ifup[/sbin/ip] doesn't<br>
> have sticky bit<br>
> Mar 1 10:45:31 server1 pgpool[3339]: bind(:) failed. reason: Success<br>
> Mar 1 10:45:31 server1 pgpool[3339]: unlink(/tmp/.s.PGSQL.9898) failed: No<br>
> such file or directory<br>
><br>
><br>
> netstat shows the following:<br>
> [root@server1 ~]# netstat -na |egrep "9898|9999"<br>
> tcp 0 0 <a href="http://0.0.0.0:9898" target="_blank">0.0.0.0:9898</a> 0.0.0.0:*<br>
> LISTEN<br>
> tcp 0 0 <a href="http://0.0.0.0:9999" target="_blank">0.0.0.0:9999</a> 0.0.0.0:*<br>
> LISTEN<br>
> tcp 0 0 <a href="http://172.16.6.154:46650" target="_blank">172.16.6.154:46650</a> <a href="http://172.16.6.153:9999" target="_blank">172.16.6.153:9999</a><br>
> TIME_WAIT<br>
> tcp 9 0 <a href="http://172.16.6.154:9999" target="_blank">172.16.6.154:9999</a> <a href="http://172.16.6.153:51868" target="_blank">172.16.6.153:51868</a><br>
> CLOSE_WAIT<br>
> tcp 9 0 <a href="http://172.16.6.154:9999" target="_blank">172.16.6.154:9999</a> <a href="http://172.16.6.153:51906" target="_blank">172.16.6.153:51906</a><br>
> CLOSE_WAIT<br>
> tcp 0 0 <a href="http://172.16.6.154:9999" target="_blank">172.16.6.154:9999</a> <a href="http://172.16.6.154:50624" target="_blank">172.16.6.154:50624</a><br>
> TIME_WAIT<br>
> tcp 9 0 <a href="http://172.16.6.154:9999" target="_blank">172.16.6.154:9999</a> <a href="http://172.16.6.153:51946" target="_blank">172.16.6.153:51946</a><br>
> CLOSE_WAIT<br>
> unix 2 [ ACC ] STREAM LISTENING 18698 /tmp/.s.PGSQL.9898<br>
> unix 2 [ ACC ] STREAM LISTENING 18685 /tmp/.s.PGSQL.9999<br>
><br>
> Is this a known issue?<br>
><br>
> I will have to reboot the server in order to start pgpool back online.<br>
><br>
> My cluster has two servers (server0 & server1) which each of them are<br>
> running pgpool, and postgreSQL with streaming Replication setup.<br>
><br>
> Thanks~<br>
> Ning<br>
</div></div></blockquote></div><br></div>