<div dir="ltr">I *think* I've sorted this by comparing %d with %P - backend ID of detached node to Old primary node ID - only promote %H:%r (New Master hostname:New master port number) when they're the same.<div><br></div><div>Does that sound reasonable?</div><div><br></div><div>I'm still not entirely sure why I'm using old primary node id rather than master node id, though, and why I'm getting failover_command called with different master and primary node ids - I do just ignore them, though, which seems to work :)</div><div><br></div><div>Many Thanks</div><div><br></div><div>Chris</div><div class="gmail_extra"><br><div class="gmail_quote">On 15 December 2014 at 15:37, Chris Beauchamp <span dir="ltr"><<a href="mailto:chris@chillibean.tv" target="_blank">chris@chillibean.tv</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_quote">Re-sending to include the list too :( Apologies</div><div class="gmail_quote">-----<div><div class="h5"><br><div dir="ltr">Hi, thanks for replying - I've attached my pgpool.conf and failover script - it's indirectly calling promote.sh on the postgres server which does:<div><br></div><div>pg_ctl promote</div><div><br></div><div>on the target server.</div><div><br></div><div>I think the failover script should do more checking as to which node is failing, and what the master or primary is and was, but I don't know which I'm meant to be checking</div><div><br></div><div>I think I'm getting to roughly under what circumstances I see this: If I try to pcp_node_attach a server which isn't actually up, it then calls the failover_command with old and new master as node 0, but old primary as 4 (which is correct) and new master port as the port of node 0. It's also logging about failback events - in reaction to this:</div><div><br></div><div><div>2014-12-15 15:17:23: pid 1: DETAIL: connection to host:"<a href="http://10.10.50.229:5702" target="_blank">10.10.50.229:5702</a>" failed</div><div>2014-12-15 15:17:23: pid 1: LOG: setting backend node 1 status to NODE DOWN</div><div>2014-12-15 15:17:23: pid 1: LOG: starting degeneration. shutdown host 10.10.50.229(5702)</div><div>2014-12-15 15:17:23: pid 1: LOG: Restart all children</div><div>2014-12-15 15:17:23: pid 2210: LOG: child process received shutdown request signal 3</div><div>2014-12-15 15:17:23: pid 2213: LOG: child process received shutdown request signal 3</div><div>... lots more of these then...</div><div>2014-12-15 15:17:23: pid 1: LOG: execute command: /dockerfail.rb 1 10.10.50.229 5702 /data1 0 0 10.10.50.229 4 5701 /data0</div><div><br></div><div><a href="http://10.10.50.229:5701" target="_blank">10.10.50.229:5701</a> - 1f16e7a97bc686b387612bb268ee41012296adcb1f985406d2613320a8144a0a</div><div>server promoting <- output from dockerfail.rb command (id of new master docker container and output from pg_ctl)</div><div>2014-12-15 15:17:23: pid 1: LOG: failover: set new primary node: 4</div><div>2014-12-15 15:17:23: pid 1: LOG: failover: set new master node: 0</div><div><br></div><div>(failover_command is defined as: failover_command = '/dockerfail.rb %d %h %p %D %M %m %H %P %r %R')</div><div><br></div><div>I'm afraid I'm a bit confused!</div><div><br></div><div>Thanks</div><div><br></div><div>Chris</div><div class="gmail_extra"><div><div><br><div class="gmail_quote">On 15 December 2014 at 14:12, Tatsuo Ishii <span dir="ltr"><<a href="mailto:ishii@postgresql.org" target="_blank">ishii@postgresql.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Probably there's something wrong with your failover script.<br>
Please show the pgpool.conf and the failover script.<br>
<br>
Best regards,<br>
--<br>
Tatsuo Ishii<br>
SRA OSS, Inc. Japan<br>
English: <a href="http://www.sraoss.co.jp/index_en.php" target="_blank">http://www.sraoss.co.jp/index_en.php</a><br>
Japanese:<a href="http://www.sraoss.co.jp" target="_blank">http://www.sraoss.co.jp</a><br>
<span><br>
> Hi PGPool list,<br>
><br>
> I'm new(ish) to both postgres and pgpool, but learning fast :)<br>
><br>
> I'm trying to setup streaming replication with pgpool, and doing automatic<br>
> failovers etc.<br>
><br>
> I've setup a mostly working configurations (lots of docker containers), and<br>
> a failing master will cause pgpool to promote another slave and then<br>
> re-direct the other slaves to the new master.<br>
><br>
> However, now, when I kill a slave, and the master *isn't* the first node in<br>
> the pool, I get failover_command called, but with the old and new master<br>
> ids set to the first node in the pool (which isn't the master in terms of<br>
> replication), and the old primary node id set the the second (which was the<br>
> master).<br>
><br>
> I then end up with two master nodes, and everything replicating from the<br>
> wrong node.<br>
><br>
> Is this a bug or a feature? I guess I'm just not clear on the difference<br>
> between a master node and a primary node - the documentation doesn't seem<br>
> to be very clear, especially where streaming replication is concerned.<br>
><br>
> Thanks!<br>
><br>
> Chris<br>
><br>
> --<br>
><br>
</span>> *Chris Beauchamp*<br>
<div><div>> Systems Administrator<br>
> ChilliBean Ltd<br>
><br>
> 22, D'Arblay Street, London, W1F 8EQ, United Kingdom<br>
> tel <a href="tel:%2B44%20%280%2920%207479%207030" value="+442074797030" target="_blank">+44 (0)20 7479 7030</a> | mobile <a href="tel:%2B44%20%280%297770%20643582" value="+447770643582" target="_blank">+44 (0)7770 643582</a> | email<br>
> <a href="mailto:chris@chillibean.tv" target="_blank">chris@chillibean.tv</a><br>
><br>
> <a href="http://www.chillibean.tv" target="_blank">www.chillibean.tv</a> | <a href="http://www.chillivault.tv" target="_blank">www.chillivault.tv</a> | <a href="http://www.sohosoho.tv" target="_blank">www.sohosoho.tv</a><br>
><br>
> This email and its attachments may be confidential and are intended solely<br>
> for the use of the individual to whom it is addressed. Any views or<br>
> opinions expressed are solely those of the author and do not necessarily<br>
> represent those of Chillibean Limited. If you are not the intended<br>
> recipient of this email and its attachments, you must take no action based<br>
> upon them, nor must you copy or show them to anyone. Please contact the<br>
> sender if you believe you have received this email in error.<br>
><br>
> Chillibean Limited of 10 Livonia Street, London W1F 8AF, United Kingdom is<br>
> a limited company registered in England under No. 4775994 VAT Registration<br>
> No. 835 760 804.<br>
</div></div></blockquote></div><br><br clear="all"><div><br></div></div></div><span><font color="#888888">-- <br></font></span><div><div dir="ltr"><span><font color="#888888"><div><b><br></b></div><div><b>Chris Beauchamp</b></div></font></span><span><div>Systems Administrator</div><div>ChilliBean Ltd</div><div><br></div><div>22, D'Arblay Street, London, W1F 8EQ, United Kingdom</div><div>tel <a href="tel:%2B44%20%280%2920%207479%207030" value="+442074797030" target="_blank">+44 (0)20 7479 7030</a> | mobile <a href="tel:%2B44%20%280%297770%20643582" value="+447770643582" target="_blank">+44 (0)7770 643582</a> | email <a href="mailto:chris@chillibean.tv" target="_blank">chris@chillibean.tv</a></div><div><br></div><div><a href="http://www.chillibean.tv" target="_blank">www.chillibean.tv</a> | <a href="http://www.chillivault.tv" target="_blank">www.chillivault.tv</a> | <a href="http://www.sohosoho.tv" target="_blank">www.sohosoho.tv</a></div><div><br></div><div>This email and its attachments may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Chillibean Limited. If you are not the intended recipient of this email and its attachments, you must take no action based upon them, nor must you copy or show them to anyone. Please contact the sender if you believe you have received this email in error.</div><div><br></div><div>Chillibean Limited of 10 Livonia Street, London W1F 8AF, United Kingdom is a limited company registered in England under No. 4775994 VAT Registration No. 835 760 804.</div></span></div></div>
</div></div></div>
</div></div></div><div><div class="h5"><br><br clear="all"><div><br></div>-- <br><div><div dir="ltr"><div><b><br></b></div><div><b>Chris Beauchamp</b></div><div>Systems Administrator</div><div>ChilliBean Ltd</div><div><br></div><div>22, D'Arblay Street, London, W1F 8EQ, United Kingdom</div><div>tel <a href="tel:%2B44%20%280%2920%207479%207030" value="+442074797030" target="_blank">+44 (0)20 7479 7030</a> | mobile <a href="tel:%2B44%20%280%297770%20643582" value="+447770643582" target="_blank">+44 (0)7770 643582</a> | email <a href="mailto:chris@chillibean.tv" target="_blank">chris@chillibean.tv</a></div><div><br></div><div><a href="http://www.chillibean.tv" target="_blank">www.chillibean.tv</a> | <a href="http://www.chillivault.tv" target="_blank">www.chillivault.tv</a> | <a href="http://www.sohosoho.tv" target="_blank">www.sohosoho.tv</a></div><div><br></div><div>This email and its attachments may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Chillibean Limited. If you are not the intended recipient of this email and its attachments, you must take no action based upon them, nor must you copy or show them to anyone. Please contact the sender if you believe you have received this email in error.</div><div><br></div><div>Chillibean Limited of 10 Livonia Street, London W1F 8AF, United Kingdom is a limited company registered in England under No. 4775994 VAT Registration No. 835 760 804.</div></div></div>
</div></div></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><div dir="ltr"><div><b><br></b></div><div><b>Chris Beauchamp</b></div><div>Systems Administrator</div><div>ChilliBean Ltd</div><div><br></div><div>22, D'Arblay Street, London, W1F 8EQ, United Kingdom</div><div>tel +44 (0)20 7479 7030 | mobile +44 (0)7770 643582 | email <a href="mailto:chris@chillibean.tv" target="_blank">chris@chillibean.tv</a></div><div><br></div><div><a href="http://www.chillibean.tv" target="_blank">www.chillibean.tv</a> | <a href="http://www.chillivault.tv" target="_blank">www.chillivault.tv</a> | <a href="http://www.sohosoho.tv" target="_blank">www.sohosoho.tv</a></div><div><br></div><div>This email and its attachments may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Chillibean Limited. If you are not the intended recipient of this email and its attachments, you must take no action based upon them, nor must you copy or show them to anyone. Please contact the sender if you believe you have received this email in error.</div><div><br></div><div>Chillibean Limited of 10 Livonia Street, London W1F 8AF, United Kingdom is a limited company registered in England under No. 4775994 VAT Registration No. 835 760 804.</div></div></div>
</div></div>