<div dir="ltr">Hey,<div><br></div><div>You are right - reading back that is all a bit vague. I will perform more investigation, I suppose I just want to confirm this isn't as expected and that a new pgpool node should sync with a currently active node.</div>
<div><br></div><div>I have installed from the EnterpriseDB PPAS suite installer.</div><div><br></div><div>My steps are:</div><div><br></div><div><ol><li>Start pgpool1</li><li>Start pgpool2</li><li>Verify show_nodes is correct from both pgpool nodes</li>
<li>Run pcp_detach to detach the standby PostgreSQL node<span style="font-family:arial,sans-serif;font-size:13px"> (pcp_detach_node 1 pgpool1 9898 postgres postgres 0)</span></li><li>Verify show_nodes is correct from both pgpool nodes</li>
<li>Stop all pgpool services on pgpool2 (check with ps)</li><li>Start pgpool on pgpool2</li><li>show_nodes no longer shows the same output on pgpool1 and pgpool2</li></ol><div>I'll bump up logging and do a bit more digging.<br>
</div><div><br></div></div><div>Cheers,</div><div><br></div></div><div class="gmail_extra"><br clear="all"><div><div style="color:#b8276a;font:bold 14px Arial,Helvetica,sans-serif">
<br>
<span style="color:#323b62">James Sewell,</span><br>
<span style="color:#a9a9a9"><span style="font:12px Arial,Helvetica,sans-serif">PostgreSQL Team Lead / Solutions Architect </span></span><br>
<span style="color:#a9a9a9">______________________________________</span><br>
</div>
<img alt="" src="http://www.lisasoft.com/sites/lisasoft/files/u1/logo1.jpg" style="margin:0px;width:153px;height:50px"><br>
<div style="font:normal 12px/20px Arial,Helvetica,sans-serif;color:#404040;margin:0">
<span style="color:#a9a9a9">Level 2, 50 Queen St, Melbourne VIC 3000</span><br>
<br>
<strong>P </strong><span style="color:#a9a9a9"><span style="font-family:Arial,Verdana,sans-serif">(+61) 3 8370 8000</span></span><span style="color:rgb(169,169,169)"> </span><strong> </strong><span style="color:rgb(50,59,98)"><span style="width:15px;display:inline-block"><strong>W</strong></span></span> <a style="color:rgb(64,64,64);margin:2px 0px;text-decoration:none"><span style="color:#a9a9a9">www.lisasoft.com</span></a> <span style="color:rgb(50,59,98)"><span style="width:15px;display:inline-block"><strong>F </strong></span></span><span style="color:#a9a9a9"><span style="font-family:Arial,Verdana,sans-serif">(+61) 3 8370 8099</span></span></div>
<div style="padding-top:8px">
</div></div>
<br><br><div class="gmail_quote">On Fri, Jul 25, 2014 at 12:19 PM, Richard Michael <span dir="ltr"><<a href="mailto:rmichael@edgeofthenet.org" target="_blank">rmichael@edgeofthenet.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr"><br><div class="gmail_extra"><br><br><div class="gmail_quote"><div><div class="h5">On Thu, Jul 24, 2014 at 8:36 PM, James Sewell <span dir="ltr"><<a href="mailto:james.sewell@lisasoft.com" target="_blank">james.sewell@lisasoft.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">Hello all,<div><br></div><div>I have two pgpool nodes which I am using a TCP load balancer to spread between. I am using watchdog to synchronise PostgreSQL node information between the two and an external HA solution (with ALLOW_TO_FAILOVER).</div>
<div><br></div><div>If I start both my pgpool nodes up I get the following inital state:<br></div><div><br></div><div><div>postgres=# show pool_nodes;</div><div> node_id | hostname | port | status | lb_weight | role</div>
<div>---------+-------------+------+--------+-----------+---------</div>
<div> 0 | 10.10.10.1 | 5432 | 2 | 0.500000 | standby</div><div> 1 | 10.10.10.2 | 5432 | 2 | 0.500000 | primary</div><div>(2 rows)</div></div><div><br></div><div>And then I run the following command:</div>
<div><br></div><div> pcp_detach_node 1 load_balancer 9898 postgres postgres 0<br></div><div><br></div><div>Now both pgpool nodes show the following:</div><div><br></div><div><div>postgres=# show pool_nodes;</div><div> node_id | hostname | port | status | lb_weight | role</div>
<div>---------+-------------+------+--------+-----------+---------</div><div> 0 | 10.10.10.1 | 5432 | 3 | 0.500000 | standby</div><div> 1 | 10.10.10.2 | 5432 | 2 | 0.500000 | primary</div><div>(2 rows)</div>
</div><div><br></div><div>This proves watchdog is working, as the command is sent to one pgpool node through the load_balancer.</div><div><br></div><div>Now if I restart pgpool node two I would expect it to come back up with the config above, not the initial config - as it would have talked to the watchdog on the other node. This is not the case though.</div>
<div><br></div><div>The non restarted node still reports:</div><div><br></div><div><div>postgres=# show pool_nodes;</div><div> node_id | hostname | port | status | lb_weight | role</div><div>---------+-------------+------+--------+-----------+---------</div>
<div> 0 | 10.10.10.1 | 5432 | 3 | 0.500000 | standby</div><div> 1 | 10.10.10.2 | 5432 | 2 | 0.500000 | primary</div><div>(2 rows)</div></div><div><br></div><div>And the restarted node reports:</div>
<div><br></div><div><div>postgres=# show pool_nodes;</div><div> node_id | hostname | port | status | lb_weight | role</div><div>---------+-------------+------+--------+-----------+---------</div><div> 0 | 10.10.10.1 | 5432 | 2 | 0.500000 | standby</div>
<div> 1 | 10.10.10.2 | 5432 | 2 | 0.500000 | primary</div><div>(2 rows)</div></div><div><br></div><div>Any ideas on how to fix this?</div></div></blockquote><div><br></div></div></div><div>Relevant portions (or tails) of log files (during the restarts/detaches/attaches) from both nodes would be helpful. Permissions on the status file [on each node] should also be verified.</div>
<div><br></div><div>Which version of pgpool; and, from where -- RPMs from <a href="http://pgpool.net" target="_blank">pgpool.net</a>, or your distro? Basically -- what SHA1 of source? (Note, this is different from version "3.3.3" because the RPMs -1, -2, -3 from <a href="http://pgpool.net" target="_blank">pgpool.net</a> are built from source bumped down the 3_3_STABLE branch, not simply tag 3.3.3). Peruse the recent commits on V3_3_STABLE and the bug tracker for several issues preventing nodes from communicating during restart.</div>
<div><br></div><div>I've needed to run pgp attach_node to teach pgpool that the failed node is back; it did not detect it in my situation (although, I have perhaps overlooked configuration). I intend to investigate this.</div>
<div><br></div><div>What is "restart" exactly? Shutdown and use ps to confirm pgpool and all related processes (watchdog, etc.) are gone, then start it again? Or, HUP or .. ?</div><div><br></div><div><br></div>
<div>Just two remarks aside, I think it would be easier to follow your discussion/explanation if:</div><div><br></div><div>1/ you referred to the machines by node_id, as opposed to ".. I restart node two .. ". I assume you mean: you restarted node_id 0 -- the status 3 (down) node -- and it came back up knowing that it was back up (status 2, and as standby), but the "always up" node_id 1 remains unaware node_id 0 has returned. (Explicitly telling node_id 1 that node_id 0 is back with pgp attach will probably "solve" this, as I mention above, but that's not automated.)</div>
<div><br></div><div> 2/ you indicated the IP (hence node_id) through which you've run show_nodes, e.g., "psql -h 10.10.10.1 -c 'SELECT show_nodes();'"</div><div><br></div><div>Hope some of this helps.</div>
<div><br></div><div>Regards,</div><div>Richard</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div class="">
<div dir="ltr"><div><div><div style="color:rgb(184,39,106);font-weight:bold;font-style:normal;font-variant:normal;font-size:14px;line-height:normal;font-family:Arial,Helvetica,sans-serif">
<br>
<span style="color:rgb(50,59,98)">James Sewell,</span><br>
<span style="color:rgb(169,169,169)"><span style="font-style:normal;font-variant:normal;font-weight:normal;font-size:12px;line-height:normal;font-family:Arial,Helvetica,sans-serif">PostgreSQL Team Lead / Solutions Architect </span></span><br>
<span style="color:rgb(169,169,169)">______________________________________</span><br>
</div>
<img alt="" src="http://www.lisasoft.com/sites/lisasoft/files/u1/logo1.jpg" style="margin:0px;width:153px;min-height:50px"><br>
<div style="font-style:normal;font-variant:normal;font-weight:normal;font-size:12px;line-height:20px;font-family:Arial,Helvetica,sans-serif;color:rgb(64,64,64);margin:0px">
<span style="color:rgb(169,169,169)">Level 2, 50 Queen St, Melbourne VIC 3000</span><br>
<br>
<strong>P </strong><span style="color:rgb(169,169,169)"><span style="font-family:Arial,Verdana,sans-serif"><a href="tel:%28%2B61%29%203%208370%208000" value="+61383708000" target="_blank">(+61) 3 8370 8000</a></span></span><span style="color:rgb(169,169,169)"> </span><strong> </strong><span style="color:rgb(50,59,98)"><span style="width:15px;display:inline-block"><strong>W</strong></span></span> <a style="color:rgb(64,64,64);margin:2px 0px;text-decoration:none"><span style="color:rgb(169,169,169)">www.lisasoft.com</span></a> <span style="color:rgb(50,59,98)"><span style="width:15px;display:inline-block"><strong>F </strong></span></span><span style="color:rgb(169,169,169)"><span style="font-family:Arial,Verdana,sans-serif"><a href="tel:%28%2B61%29%203%208370%208099" value="+61383708099" target="_blank">(+61) 3 8370 8099</a></span></span></div>
<div style="padding-top:8px">
</div></div>
</div></div>
<br>
<p></p></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:14.5454540252686px;background-color:rgb(255,255,255)"><hr><font size="1" color="Gray" face="Arial">The contents of this email are confidential and may be subject to legal or professional privilege and copyright. No representation is made that this email is free of viruses or other defects. If you have received this communication in error, you may not copy or distribute any part of it or otherwise disclose its contents to anyone. Please advise the sender of your incorrect receipt of this correspondence.</font></div>
<p></p><br>_______________________________________________<br>
pgpool-general mailing list<br>
<a href="mailto:pgpool-general@pgpool.net" target="_blank">pgpool-general@pgpool.net</a><br>
<a href="http://www.pgpool.net/mailman/listinfo/pgpool-general" target="_blank">http://www.pgpool.net/mailman/listinfo/pgpool-general</a><br>
<br></blockquote></div><br></div></div>
</blockquote></div><br></div>
<br>
<p><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:14.545454025268555px;background-color:rgb(255,255,255)"><hr><font size="1" color="Gray" face="Arial">The contents of this email are confidential and may be subject to legal or professional privilege and copyright. No representation is made that this email is free of viruses or other defects. If you have received this communication in error, you may not copy or distribute any part of it or otherwise disclose its contents to anyone. Please advise the sender of your incorrect receipt of this correspondence.</font></div></p>