<div dir="ltr">Hi Ishii-San<div><br></div><div>Thanks for providing the log files along with the steps to reproduce the issue. I have pushed the fix for it</div><div><br></div><div><a href="https://git.postgresql.org/gitweb/?p=pgpool2.git;a=commitdiff;h=2d7702c6961e8949e97992c23a42c8616af36d84">https://git.postgresql.org/gitweb/?p=pgpool2.git;a=commitdiff;h=2d7702c6961e8949e97992c23a42c8616af36d84</a><br></div><div><br></div><div>Thanks</div><div>Best Regards</div><div>Muhammad Usama</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Aug 2, 2019 at 4:50 PM Tatsuo Ishii <<a href="mailto:ishii@sraoss.co.jp">ishii@sraoss.co.jp</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">> Hi Ishii-San<br>
> <br>
> On Fri, Aug 2, 2019 at 10:25 AM Tatsuo Ishii <<a href="mailto:ishii@sraoss.co.jp" target="_blank">ishii@sraoss.co.jp</a>> wrote:<br>
> <br>
>> Hi Usama,<br>
>><br>
>> In your commit:<br>
>><br>
>> <a href="https://git.postgresql.org/gitweb/?p=pgpool2.git;a=commit;h=33df0d33df1ce701f07fecaeef5b87a2707c08f2" rel="noreferrer" target="_blank">https://git.postgresql.org/gitweb/?p=pgpool2.git;a=commit;h=33df0d33df1ce701f07fecaeef5b87a2707c08f2</a><br>
>><br>
>> "the master watchdog node should resign from master responsibilities<br>
>> if the primary backend node gets into quarantine state on that."<br>
>><br>
>> While testing this feature, I noticed that the master watchdog node<br>
>> resign from master responsibilities as expected in this case. However<br>
>> none of standby nodes gets promoted to new primary. As a result,<br>
>> there's no primary node any more after the master watchdog resign.<br>
>><br>
>> This is not very good because users cannot<br>
> <br>
> <br>
> This is not at all expected behavior, Can you please share the logs or<br>
> steps to reproduce the issue.<br>
> I will look into this on priority.<br>
<br>
Sure. Pgpool.logs attached.<br>
<br>
Here are steps to reproduce the issue.<br>
<br>
- Pgpool-II master branch head with patch [pgpool-hackers: 3361].<br>
<br>
1) create 3 watchdog/3 PostgreSQL cluster: watchdog_setup -wn 3 -n 3<br>
<br>
$ psql -p 50000 -c "show pool_nodes" test<br>
<br>
node_id | hostname | port | status | lb_weight | role | select_cnt | load_balance_node | replication_delay | replication_state | replication_sync_state | last_status_change <br>
---------+----------+-------+--------+-----------+---------+------------+-------------------+-------------------+-------------------+------------------------+---------------------<br>
0 | /tmp | 51000 | up | 0.333333 | primary | 0 | true | 0 | | | 2019-08-02 20:38:25<br>
1 | /tmp | 51001 | up | 0.333333 | standby | 0 | false | 0 | streaming | async | 2019-08-02 20:38:25<br>
2 | /tmp | 51002 | up | 0.333333 | standby | 0 | false | 0 | streaming | async | 2019-08-02 20:38:25<br>
(3 rows)<br>
<br>
$ pcp_watchdog_info -p 50001 -v -w<br>
<br>
Watchdog Cluster Information <br>
Total Nodes : 3<br>
Remote Nodes : 2<br>
Quorum state : QUORUM EXIST<br>
Alive Remote Nodes : 2<br>
VIP up on local node : YES<br>
Master Node Name : localhost:50000 Linux tishii-CFSV7-1<br>
Master Host Name : localhost<br>
<br>
Watchdog Node Information <br>
Node Name : localhost:50000 Linux tishii-CFSV7-1<br>
Host Name : localhost<br>
Delegate IP : Not_Set<br>
Pgpool port : 50000<br>
Watchdog port : 50002<br>
Node priority : 3<br>
Status : 4<br>
Status Name : MASTER<br>
<br>
Node Name : localhost:50004 Linux tishii-CFSV7-1<br>
Host Name : localhost<br>
Delegate IP : Not_Set<br>
Pgpool port : 50004<br>
Watchdog port : 50006<br>
Node priority : 2<br>
Status : 7<br>
Status Name : STANDBY<br>
<br>
Node Name : localhost:50008 Linux tishii-CFSV7-1<br>
Host Name : localhost<br>
Delegate IP : Not_Set<br>
Pgpool port : 50008<br>
Watchdog port : 50010<br>
Node priority : 1<br>
Status : 7<br>
Status Name : STANDBY<br>
<br>
2) echo "0 down" > pgpoo0/log<br>
<br>
3) check the result.<br>
<br>
$psql -p 50000 -c "show pool_nodes" test<br>
<br>
node_id | hostname | port | status | lb_weight | role | select_cnt | load_balance_node | replication_delay | replication_state | replication_sync_state | last_status_change <br>
---------+----------+-------+--------+-----------+---------+------------+-------------------+-------------------+-------------------+------------------------+---------------------<br>
0 | /tmp | 51000 | up | 0.333333 | standby | 0 | true | 0 | | | 2019-08-02 20:39:25<br>
1 | /tmp | 51001 | up | 0.333333 | standby | 0 | false | 0 | | | 2019-08-02 20:39:25<br>
2 | /tmp | 51002 | up | 0.333333 | standby | 0 | false | 0 | | | 2019-08-02 20:39:25<br>
(3 rows)<br>
<br>
$ pcp_watchdog_info -p 50001 -v -w<br>
Watchdog Cluster Information <br>
Total Nodes : 3<br>
Remote Nodes : 2<br>
Quorum state : QUORUM EXIST<br>
Alive Remote Nodes : 2<br>
VIP up on local node : NO<br>
Master Node Name : localhost:50004 Linux tishii-CFSV7-1<br>
Master Host Name : localhost<br>
<br>
Watchdog Node Information <br>
Node Name : localhost:50000 Linux tishii-CFSV7-1<br>
Host Name : localhost<br>
Delegate IP : Not_Set<br>
Pgpool port : 50000<br>
Watchdog port : 50002<br>
Node priority : 3<br>
Status : 7<br>
Status Name : STANDBY<br>
<br>
Node Name : localhost:50004 Linux tishii-CFSV7-1<br>
Host Name : localhost<br>
Delegate IP : Not_Set<br>
Pgpool port : 50004<br>
Watchdog port : 50006<br>
Node priority : 2<br>
Status : 4<br>
Status Name : MASTER<br>
<br>
Node Name : localhost:50008 Linux tishii-CFSV7-1<br>
Host Name : localhost<br>
Delegate IP : Not_Set<br>
Pgpool port : 50008<br>
Watchdog port : 50010<br>
Node priority : 1<br>
Status : 7<br>
Status Name : STANDBY<br>
</blockquote></div>