<div dir="ltr"><div>Hi Ishii-San</div><div><br></div><div>:-) Thanks for the confirmation.</div><div><br></div><div>Best Regards</div><div>Muhammad Usama</div><div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Aug 8, 2019 at 1:28 PM Tatsuo Ishii <<a href="mailto:ishii@sraoss.co.jp">ishii@sraoss.co.jp</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Oops. I should have connected to the master Pgpool-II:<br>
<br>
t-ishii$ psql -p 50004 -c "show pool_nodes" test<br>
node_id | hostname | port | status | lb_weight | role | select_cnt | load_balance_node | replication_delay | replication_state | replication_sync_state | last_status_change <br>
---------+----------+-------+--------+-----------+---------+------------+-------------------+-------------------+-------------------+------------------------+---------------------<br>
0 | /tmp | 51000 | up | 0.333333 | primary | 0 | true | 0 | | | 2019-08-08 17:26:27<br>
1 | /tmp | 51001 | up | 0.333333 | standby | 0 | false | 0 | streaming | async | 2019-08-08 17:26:27<br>
2 | /tmp | 51002 | up | 0.333333 | standby | 0 | false | 0 | streaming | async | 2019-08-08 17:26:27<br>
(3 rows)<br>
<br>
Now node 0 is primary, works great as I expected.<br>
Sorry for confusion.<br>
<br>
> Hi Usama,<br>
> <br>
> Thanks for the commit. Unfortunately the fix did not work for me.<br>
> <br>
> psql -p 50000 -c "show pool_nodes" test<br>
> node_id | hostname | port | status | lb_weight | role | select_cnt | load_balance_node | replication_delay | replication_state | replication_sync_state | last_status_change <br>
> ---------+----------+-------+------------+-----------+---------+------------+-------------------+-------------------+-------------------+------------------------+---------------------<br>
> 0 | /tmp | 51000 | quarantine | 0.333333 | standby | 0 | false | 0 | | | 2019-08-08 17:09:17<br>
> 1 | /tmp | 51001 | up | 0.333333 | standby | 0 | false | 0 | | | 2019-08-08 17:09:00<br>
> 2 | /tmp | 51002 | up | 0.333333 | standby | 0 | true | 0 | | | 2019-08-08 17:09:00<br>
> (3 rows)<br>
> <br>
> There's no primary PostgreSQL node after node was quarantined.<br>
> Pgpool log attached.<br>
> <br>
> From: Muhammad Usama <<a href="mailto:m.usama@gmail.com" target="_blank">m.usama@gmail.com</a>><br>
> Subject: Re: Behavior of resigned master watchdog node<br>
> Date: Thu, 8 Aug 2019 11:21:24 +0500<br>
> Message-ID: <CAEJvTzUpYZHc=T8FvOB12N=<a href="mailto:gC_Qj961dZEpRZTggsLHz_gJqSQ@mail.gmail.com" target="_blank">gC_Qj961dZEpRZTggsLHz_gJqSQ@mail.gmail.com</a>><br>
> <br>
>> Hi Ishii-San<br>
>> <br>
>> Thanks for providing the log files along with the steps to reproduce the<br>
>> issue. I have pushed the fix for it<br>
>> <br>
>> <a href="https://git.postgresql.org/gitweb/?p=pgpool2.git;a=commitdiff;h=2d7702c6961e8949e97992c23a42c8616af36d84" rel="noreferrer" target="_blank">https://git.postgresql.org/gitweb/?p=pgpool2.git;a=commitdiff;h=2d7702c6961e8949e97992c23a42c8616af36d84</a><br>
>> <br>
>> Thanks<br>
>> Best Regards<br>
>> Muhammad Usama<br>
>> <br>
>> <br>
>> On Fri, Aug 2, 2019 at 4:50 PM Tatsuo Ishii <<a href="mailto:ishii@sraoss.co.jp" target="_blank">ishii@sraoss.co.jp</a>> wrote:<br>
>> <br>
>>> > Hi Ishii-San<br>
>>> ><br>
>>> > On Fri, Aug 2, 2019 at 10:25 AM Tatsuo Ishii <<a href="mailto:ishii@sraoss.co.jp" target="_blank">ishii@sraoss.co.jp</a>> wrote:<br>
>>> ><br>
>>> >> Hi Usama,<br>
>>> >><br>
>>> >> In your commit:<br>
>>> >><br>
>>> >><br>
>>> <a href="https://git.postgresql.org/gitweb/?p=pgpool2.git;a=commit;h=33df0d33df1ce701f07fecaeef5b87a2707c08f2" rel="noreferrer" target="_blank">https://git.postgresql.org/gitweb/?p=pgpool2.git;a=commit;h=33df0d33df1ce701f07fecaeef5b87a2707c08f2</a><br>
>>> >><br>
>>> >> "the master watchdog node should resign from master responsibilities<br>
>>> >> if the primary backend node gets into quarantine state on that."<br>
>>> >><br>
>>> >> While testing this feature, I noticed that the master watchdog node<br>
>>> >> resign from master responsibilities as expected in this case. However<br>
>>> >> none of standby nodes gets promoted to new primary. As a result,<br>
>>> >> there's no primary node any more after the master watchdog resign.<br>
>>> >><br>
>>> >> This is not very good because users cannot<br>
>>> ><br>
>>> ><br>
>>> > This is not at all expected behavior, Can you please share the logs or<br>
>>> > steps to reproduce the issue.<br>
>>> > I will look into this on priority.<br>
>>><br>
>>> Sure. Pgpool.logs attached.<br>
>>><br>
>>> Here are steps to reproduce the issue.<br>
>>><br>
>>> - Pgpool-II master branch head with patch [pgpool-hackers: 3361].<br>
>>><br>
>>> 1) create 3 watchdog/3 PostgreSQL cluster: watchdog_setup -wn 3 -n 3<br>
>>><br>
>>> $ psql -p 50000 -c "show pool_nodes" test<br>
>>><br>
>>> node_id | hostname | port | status | lb_weight | role | select_cnt |<br>
>>> load_balance_node | replication_delay | replication_state |<br>
>>> replication_sync_state | last_status_change<br>
>>><br>
>>> ---------+----------+-------+--------+-----------+---------+------------+-------------------+-------------------+-------------------+------------------------+---------------------<br>
>>> 0 | /tmp | 51000 | up | 0.333333 | primary | 0 |<br>
>>> true | 0 | |<br>
>>> | 2019-08-02 20:38:25<br>
>>> 1 | /tmp | 51001 | up | 0.333333 | standby | 0 |<br>
>>> false | 0 | streaming | async<br>
>>> | 2019-08-02 20:38:25<br>
>>> 2 | /tmp | 51002 | up | 0.333333 | standby | 0 |<br>
>>> false | 0 | streaming | async<br>
>>> | 2019-08-02 20:38:25<br>
>>> (3 rows)<br>
>>><br>
>>> $ pcp_watchdog_info -p 50001 -v -w<br>
>>><br>
>>> Watchdog Cluster Information<br>
>>> Total Nodes : 3<br>
>>> Remote Nodes : 2<br>
>>> Quorum state : QUORUM EXIST<br>
>>> Alive Remote Nodes : 2<br>
>>> VIP up on local node : YES<br>
>>> Master Node Name : localhost:50000 Linux tishii-CFSV7-1<br>
>>> Master Host Name : localhost<br>
>>><br>
>>> Watchdog Node Information<br>
>>> Node Name : localhost:50000 Linux tishii-CFSV7-1<br>
>>> Host Name : localhost<br>
>>> Delegate IP : Not_Set<br>
>>> Pgpool port : 50000<br>
>>> Watchdog port : 50002<br>
>>> Node priority : 3<br>
>>> Status : 4<br>
>>> Status Name : MASTER<br>
>>><br>
>>> Node Name : localhost:50004 Linux tishii-CFSV7-1<br>
>>> Host Name : localhost<br>
>>> Delegate IP : Not_Set<br>
>>> Pgpool port : 50004<br>
>>> Watchdog port : 50006<br>
>>> Node priority : 2<br>
>>> Status : 7<br>
>>> Status Name : STANDBY<br>
>>><br>
>>> Node Name : localhost:50008 Linux tishii-CFSV7-1<br>
>>> Host Name : localhost<br>
>>> Delegate IP : Not_Set<br>
>>> Pgpool port : 50008<br>
>>> Watchdog port : 50010<br>
>>> Node priority : 1<br>
>>> Status : 7<br>
>>> Status Name : STANDBY<br>
>>><br>
>>> 2) echo "0 down" > pgpoo0/log<br>
>>><br>
>>> 3) check the result.<br>
>>><br>
>>> $psql -p 50000 -c "show pool_nodes" test<br>
>>><br>
>>> node_id | hostname | port | status | lb_weight | role | select_cnt |<br>
>>> load_balance_node | replication_delay | replication_state |<br>
>>> replication_sync_state | last_status_change<br>
>>><br>
>>> ---------+----------+-------+--------+-----------+---------+------------+-------------------+-------------------+-------------------+------------------------+---------------------<br>
>>> 0 | /tmp | 51000 | up | 0.333333 | standby | 0 |<br>
>>> true | 0 | |<br>
>>> | 2019-08-02 20:39:25<br>
>>> 1 | /tmp | 51001 | up | 0.333333 | standby | 0 |<br>
>>> false | 0 | |<br>
>>> | 2019-08-02 20:39:25<br>
>>> 2 | /tmp | 51002 | up | 0.333333 | standby | 0 |<br>
>>> false | 0 | |<br>
>>> | 2019-08-02 20:39:25<br>
>>> (3 rows)<br>
>>><br>
>>> $ pcp_watchdog_info -p 50001 -v -w<br>
>>> Watchdog Cluster Information<br>
>>> Total Nodes : 3<br>
>>> Remote Nodes : 2<br>
>>> Quorum state : QUORUM EXIST<br>
>>> Alive Remote Nodes : 2<br>
>>> VIP up on local node : NO<br>
>>> Master Node Name : localhost:50004 Linux tishii-CFSV7-1<br>
>>> Master Host Name : localhost<br>
>>><br>
>>> Watchdog Node Information<br>
>>> Node Name : localhost:50000 Linux tishii-CFSV7-1<br>
>>> Host Name : localhost<br>
>>> Delegate IP : Not_Set<br>
>>> Pgpool port : 50000<br>
>>> Watchdog port : 50002<br>
>>> Node priority : 3<br>
>>> Status : 7<br>
>>> Status Name : STANDBY<br>
>>><br>
>>> Node Name : localhost:50004 Linux tishii-CFSV7-1<br>
>>> Host Name : localhost<br>
>>> Delegate IP : Not_Set<br>
>>> Pgpool port : 50004<br>
>>> Watchdog port : 50006<br>
>>> Node priority : 2<br>
>>> Status : 4<br>
>>> Status Name : MASTER<br>
>>><br>
>>> Node Name : localhost:50008 Linux tishii-CFSV7-1<br>
>>> Host Name : localhost<br>
>>> Delegate IP : Not_Set<br>
>>> Pgpool port : 50008<br>
>>> Watchdog port : 50010<br>
>>> Node priority : 1<br>
>>> Status : 7<br>
>>> Status Name : STANDBY<br>
>>><br>
</blockquote></div></div>