<div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Aug 17, 2019 at 12:28 PM Tatsuo Ishii <<a href="mailto:ishii@sraoss.co.jp">ishii@sraoss.co.jp</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">> Hi Pgpool Team,<br>
> <br>
> *We are nearing a production release and running into the<br>
> below issues.*<br>
> Replies at the earliest would be highly helpful and greatly appreciated.<br>
> Please let us know on how to get rid of the below issues.<br>
> <br>
> We have a 3 node pgpool + postgres cluster - M1 , M2, M3. The pgpool.conf<br>
> is as attached.<br>
> <br>
> *Case I : *<br>
> M1 - Pgpool Master + Postgres Master<br>
> M2 , M3 - Pgpool slave + Postgres slave<br>
> <br>
> - M1 goes out of network. its marked as LOST in the pgpool cluster<br>
> - M2 becomes postgres master<br>
> - M3 becomes pgpool master.<br>
> - When M1 comes back to the network, pgpool is able to solve split brain.<br>
> However, its changing the postgres master back to M1 by logging a statement<br>
> - "LOG: primary node was chenged after the sync from new master", so since<br>
> M2 was already postgres master (and its trigger file is not touched) its<br>
> not able to sync to the new master.<br>
> *I somehow want to avoid this postgres master change..please let us know if<br>
> there is a way to avoid it*<br>
<br>
Sorry but I don't know how to prevent this. Probably when former<br>
watchdog master recovers from an network outage and there's already<br>
PostgreSQL primary server, the watchdog master should not sync the<br>
state. What do you think Usama?<br></blockquote><div><br></div><div>Yes, that's true, there is no functionality that exists in Pgpool-II to disable the backend node status synch. In fact that</div><div>would be hazardous if we somehow disable the node status syncing.</div><div><br></div><div>But having said that, In the mentioned scenario when the M1 comes back and join the watchdog cluster Pgpool-II should have</div><div>kept the M2 as the true master while resolving the split-brain. The algorithm used to resolve the true master considers quite a</div><div>few parameters and for the scenario, you explained, M2 should have kept the master node status while M1 should have resigned</div><div>after joining back the cluster and effectively the M1 node should have been syncing the status from M2 ( keeping the proper primary node)</div><div>not the other way around.</div><div>Can you please share the Pgpool-II log files so that I can have a look at what went wrong in this case.</div><div><br></div><div>Thanks</div><div>Best Regards</div><div>Muhammad Usama</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
In the mean time you could manually stop the primary PostgreSQL on M1<br>
then execute pcp_recovery_node on it to avoid the dual primary server<br>
situation.<br>
<br>
> *Case II:*<br>
> <br>
> M1 - Pgpool Master + Postgres Master<br>
> M2 , M3 - Pgpool slave + Postgres slave<br>
> <br>
> - Shut down M1, M2<br>
> - M3 is rightly elected as the pgpool master.<br>
> - However when failover request kicks in, the watchdog rejects with the<br>
> below log .* Is there a way to make M3 as the postgres master, inspite of<br>
> the quorum ?*<br>
> *Please let me know.*<br>
> <br>
> <br>
> 2019-08-16T11:15:04+00:00 lcm-34-182 pgpool[11002]: [92-1] 2019-08-16<br>
> 11:15:04: pid 11002: LOG: watchdog is processing the failover command<br>
> [DEGENERATE_BACKEND_REQUEST] received from local pgpool-II on IPC interface<br>
> 2019-08-16T11:15:04+00:00 lcm-34-182 pgpool[11002]: [93-1] 2019-08-16<br>
> 11:15:04: pid 11002: LOG: failover requires the quorum to hold, which is<br>
> not present at the moment<br>
> 2019-08-16T11:15:04+00:00 lcm-34-182 pgpool[11002]: [93-2] 2019-08-16<br>
> 11:15:04: pid 11002: DETAIL: Rejecting the failover request<br>
> 2019-08-16T11:15:04+00:00 lcm-34-182 pgpool[11002]: [94-1] 2019-08-16<br>
> 11:15:04: pid 11002: LOG: failover command [DEGENERATE_BACKEND_REQUEST]<br>
> request from pgpool-II node "lcm-34-182.dev.lcm.local:9999 Linux<br>
> lcm-34-182.dev.lcm.local" is rejected because the watchdog cluster does not<br>
> hold the quorum<br>
> 2019-08-16T11:15:04+00:00 lcm-34-182 pgpool[11049]: [23-1] 2019-08-16<br>
> 11:15:04: pid 11049: LOG: degenerate backend request for 1 node(s) from pid<br>
> [11049], is changed to quarantine node request by watchdog<br>
<br>
You could manualy promote PostgreSQL on M3 by using pg_ctl promote<br>
command, then execute pcp_promote_node to make it primary server from<br>
Pgpool-II's point of view.<br>
<br>
Best regards,<br>
--<br>
Tatsuo Ishii<br>
SRA OSS, Inc. Japan<br>
English: <a href="http://www.sraoss.co.jp/index_en.php" rel="noreferrer" target="_blank">http://www.sraoss.co.jp/index_en.php</a><br>
Japanese:<a href="http://www.sraoss.co.jp" rel="noreferrer" target="_blank">http://www.sraoss.co.jp</a><br>
</blockquote></div></div>