<div dir="ltr">Hi Usama / Tatsuo,<div><br></div><div> Received the email notification today, sorry for the delayed response. </div><div>Please find attached the pgpool-II log for the same.</div><div><br></div><div>So basically below is the short summary of the issue:</div><div><br></div><div><br></div><div>Node -1 : Pgpool Master + Postgres Master<br></div><div><br></div><div>Node -2 : Pgpool Standby + Postgres Standby</div><div><br></div><div>Node-3 : Pgpool Standby + Postgres Standby</div><div><br></div><div><br></div><div>When network failure happens and Node-1 goes out of network, below is the status :</div><div><br></div><div>Node-1 : Pgpool Lost status + Postgres Standby (down)</div><div><br></div><div>Node -2 : Pgpool Master + Postgres Master</div><div><br></div><div>Node-3 : Pgpool Standby + Postgres Standby</div><div><br></div><div><br></div><div>Now when Node-1 comes back to network , below is the status causing the pgpool cluster to get into imbalance :</div><div><br></div><div><br></div><div><br></div><div>lcm-34-189:~ # psql -h 10.198.34.191 -p 9999 -U pgpool postgres -c "show pool_nodes"<br>Password for user pgpool:<br> node_id | hostname | port | status | lb_weight | role | select_cnt | load_balance_node | replication_delay | last_status_change<br>---------+---------------+------+--------+-----------+---------+------------+-------------------+-------------------+---------------------<br> 0 | 10.198.34.188 | 5432 | up | 0.333333 | primary | 0 | true | 0 | 2019-08-31 16:40:26<br> 1 | 10.198.34.189 | 5432 | up | 0.333333 | standby | 0 | false | 1013552 | 2019-08-31 16:40:26<br> 2 | 10.198.34.190 | 5432 | up | 0.333333 | standby | 0 | false | 0 | 2019-08-31 16:40:26<br>(3 rows)<br></div><div><br></div><div>lcm-34-189:~ # /usr/local/bin/pcp_watchdog_info -p 9898 -h 10.198.34.191 -U pgpool<br>Password:<br>3 NO lcm-34-188.dev.lcm.local:9999 Linux lcm-34-188.dev.lcm.local 10.198.34.188<br><br>lcm-34-189.dev.lcm.local:9999 Linux lcm-34-189.dev.lcm.local lcm-34-189.dev.lcm.local 9999 9000 7 STANDBY<br>lcm-34-188.dev.lcm.local:9999 Linux lcm-34-188.dev.lcm.local 10.198.34.188 9999 9000 4 MASTER<br>lcm-34-190.dev.lcm.local:9999 Linux lcm-34-190.dev.lcm.local 10.198.34.190 9999 9000 4 MASTER<br>lcm-34-189:~ #<br></div><div><br></div><div><br></div><div><br></div><div>Thanks And Regards,</div><div><br></div><div> Lakshmi Y M</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Aug 20, 2019 at 8:55 AM Tatsuo Ishii <<a href="mailto:ishii@sraoss.co.jp">ishii@sraoss.co.jp</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">> On Sat, Aug 17, 2019 at 12:28 PM Tatsuo Ishii <<a href="mailto:ishii@sraoss.co.jp" target="_blank">ishii@sraoss.co.jp</a>> wrote:<br>
> <br>
>> > Hi Pgpool Team,<br>
>> ><br>
>> > *We are nearing a production release and running into the<br>
>> > below issues.*<br>
>> > Replies at the earliest would be highly helpful and greatly appreciated.<br>
>> > Please let us know on how to get rid of the below issues.<br>
>> ><br>
>> > We have a 3 node pgpool + postgres cluster - M1 , M2, M3. The pgpool.conf<br>
>> > is as attached.<br>
>> ><br>
>> > *Case I : *<br>
>> > M1 - Pgpool Master + Postgres Master<br>
>> > M2 , M3 - Pgpool slave + Postgres slave<br>
>> ><br>
>> > - M1 goes out of network. its marked as LOST in the pgpool cluster<br>
>> > - M2 becomes postgres master<br>
>> > - M3 becomes pgpool master.<br>
>> > - When M1 comes back to the network, pgpool is able to solve split brain.<br>
>> > However, its changing the postgres master back to M1 by logging a<br>
>> statement<br>
>> > - "LOG: primary node was chenged after the sync from new master", so<br>
>> since<br>
>> > M2 was already postgres master (and its trigger file is not touched) its<br>
>> > not able to sync to the new master.<br>
>> > *I somehow want to avoid this postgres master change..please let us know<br>
>> if<br>
>> > there is a way to avoid it*<br>
>><br>
>> Sorry but I don't know how to prevent this. Probably when former<br>
>> watchdog master recovers from an network outage and there's already<br>
>> PostgreSQL primary server, the watchdog master should not sync the<br>
>> state. What do you think Usama?<br>
>><br>
> <br>
> Yes, that's true, there is no functionality that exists in Pgpool-II to<br>
> disable the backend node status synch. In fact that<br>
> would be hazardous if we somehow disable the node status syncing.<br>
> <br>
> But having said that, In the mentioned scenario when the M1 comes back and<br>
> join the watchdog cluster Pgpool-II should have<br>
> kept the M2 as the true master while resolving the split-brain. The<br>
> algorithm used to resolve the true master considers quite a<br>
> few parameters and for the scenario, you explained, M2 should have kept the<br>
> master node status while M1 should have resigned<br>
> after joining back the cluster and effectively the M1 node should have been<br>
> syncing the status from M2 ( keeping the proper primary node)<br>
> not the other way around.<br>
> Can you please share the Pgpool-II log files so that I can have a look at<br>
> what went wrong in this case.<br>
<br>
Usama,<br>
<br>
Ok, the scenario (PostgreSQL primary x 2 in the end) should have not<br>
happend. That's a good news.<br>
<br>
Lakshmi,<br>
<br>
Can you please provide the Pgpool-II log files as Usama requested?<br>
<br>
Best regards,<br>
--<br>
Tatsuo Ishii<br>
SRA OSS, Inc. Japan<br>
English: <a href="http://www.sraoss.co.jp/index_en.php" rel="noreferrer" target="_blank">http://www.sraoss.co.jp/index_en.php</a><br>
Japanese:<a href="http://www.sraoss.co.jp" rel="noreferrer" target="_blank">http://www.sraoss.co.jp</a><br>
</blockquote></div>