<div dir="ltr">Hi Ishii-San<div><br></div><div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Sep 8, 2017 at 5:40 AM, Tatsuo Ishii <span dir="ltr"><<a href="mailto:ishii@sraoss.co.jp" target="_blank">ishii@sraoss.co.jp</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">I have tested the patch a little bit using 004 watchdog regression<br>
test. After the test ends, I manually started master and standby<br>
Pgpool-II.<br>
<br>
1) Stop master PostgreSQL. Since there's only one PostgreSQL is<br>
configured, I expected:<br>
<br>
psql: ERROR: pgpool is not accepting any new connections<br>
DETAIL: all backend nodes are down, pgpool requires at least one valid node<br>
HINT: repair the backend nodes and restart pgpool<br>
<br>
but master Pgpool-II replies:<br>
<br>
psql: FATAL: failed to create a backend connection<br>
DETAIL: executing failover on backend<br>
<br>
Is this normal?<br></blockquote><div><br></div><div>Finally I am able to reproduce this behaviour and I don't think it is normal.</div><div>But this is not caused by this patch and I get the same response without the patch and even without watchdog.</div><div class="gmail_quote"><br></div>This is an existing problem in <font face="monospace, monospace">degenerate_backend_set_ex</font>() function with RAW mode where it does not</div><div class="gmail_quote">considers RAW mode while checking for valid backends.</div><div class="gmail_quote"><br></div><div class="gmail_quote">I will take care of this issue separately. </div><div class="gmail_quote"><br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br>
2) I shutdown the master node to see if the standby escalates.<br>
<br>
After shutting down the master, I see this using pcp_watchdog_info:<br>
<br>
pcp_watchdog_info -p 11105<br>
localhost:11100 Linux tishii-CF-SX3HE4BP localhost 11100 21104 4 MASTER<br>
localhost:11000 Linux tishii-CF-SX3HE4BP localhost 11000 21004 10 SHUTDOWN<br>
<br>
Seems ok but I want to confirm.<br></blockquote><div><br></div><div>Yes this seems perfect :-) </div><div><br></div><div>Thanks</div><div>Best regards</div><div>Muhammad Usama</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
master and standby pgpool logs attached.<br>
<div class="gmail-HOEnZb"><div class="gmail-h5"><br>
Best regards,<br>
--<br>
Tatsuo Ishii<br>
SRA OSS, Inc. Japan<br>
English: <a href="http://www.sraoss.co.jp/index_en.php" rel="noreferrer" target="_blank">http://www.sraoss.co.jp/index_<wbr>en.php</a><br>
Japanese:<a href="http://www.sraoss.co.jp" rel="noreferrer" target="_blank">http://www.sraoss.co.<wbr>jp</a><br>
<br>
> On Fri, Aug 25, 2017 at 5:05 PM, Tatsuo Ishii <<a href="mailto:ishii@sraoss.co.jp">ishii@sraoss.co.jp</a>> wrote:<br>
><br>
>> > On Fri, Aug 25, 2017 at 12:53 PM, Tatsuo Ishii <<a href="mailto:ishii@sraoss.co.jp">ishii@sraoss.co.jp</a>><br>
>> wrote:<br>
>> ><br>
>> >> Usama,<br>
>> >><br>
>> >> With the new patch, the regression tests all passed.<br>
>> >><br>
>> ><br>
>> > Glad to hear that :-)<br>
>> > Did you had a chance to look at the node quarantine state I added. What<br>
>> are<br>
>> > your thoughts on that ?<br>
>><br>
>> I'm going to look into the patch this weekend.<br>
>><br>
><br>
> Many thanks<br>
><br>
> Best Regards<br>
> Muhammad Usama<br>
><br>
>><br>
>> Best regards,<br>
>> --<br>
>> Tatsuo Ishii<br>
>> SRA OSS, Inc. Japan<br>
>> English: <a href="http://www.sraoss.co.jp/index_en.php" rel="noreferrer" target="_blank">http://www.sraoss.co.jp/index_<wbr>en.php</a><br>
>> Japanese:<a href="http://www.sraoss.co.jp" rel="noreferrer" target="_blank">http://www.sraoss.co.<wbr>jp</a><br>
>><br>
>> >> > Hi Ishii-San<br>
>> >> ><br>
>> >> > Please fine the updated patch, It fixes the regression issue you were<br>
>> >> > facing and also another bug which I encountered during my testing.<br>
>> >> ><br>
>> >> > -- Adding Yugo to the thread,<br>
>> >> > Hi Yugo,<br>
>> >> ><br>
>> >> > Since you are an expert of watchdog feature, So I thought you might<br>
>> have<br>
>> >> > something to say especially regarding the discussion points mentioned<br>
>> in<br>
>> >> > the initial mail.<br>
>> >> ><br>
>> >> ><br>
>> >> > Thanks<br>
>> >> > Best Regards<br>
>> >> > Muhammad Usama<br>
>> >> ><br>
>> >> ><br>
>> >> > On Thu, Aug 24, 2017 at 11:25 AM, Muhammad Usama <<a href="mailto:m.usama@gmail.com">m.usama@gmail.com</a>><br>
>> >> wrote:<br>
>> >> ><br>
>> >> >><br>
>> >> >><br>
>> >> >> On Thu, Aug 24, 2017 at 4:34 AM, Tatsuo Ishii <<a href="mailto:ishii@sraoss.co.jp">ishii@sraoss.co.jp</a>><br>
>> >> wrote:<br>
>> >> >><br>
>> >> >>> After applying the patch, many of regression tests fail. It seems<br>
>> >> >>> pgpool.conf.sample has bogus comment which causes the pgpool.conf<br>
>> >> >>> parser to complain parse error.<br>
>> >> >>><br>
>> >> >>> 2017-08-24 08:22:36: pid 6017: FATAL: syntex error in configuration<br>
>> >> file<br>
>> >> >>> "/home/t-ishii/work/pgpool-II/<wbr>current/pgpool2/src/test/regre<br>
>> >> >>> ssion/tests/004.watchdog/<wbr>standby/etc/pgpool.conf"<br>
>> >> >>> 2017-08-24 08:22:36: pid 6017: DETAIL: parse error at line 568 '*'<br>
>> >> token<br>
>> >> >>> = 8<br>
>> >> >>><br>
>> >> >><br>
>> >> >> Really sorry, Somehow I overlooked the sample config file changes I<br>
>> made<br>
>> >> >> at the last minute.<br>
>> >> >> Will send you the updated version.<br>
>> >> >><br>
>> >> >> Thanks<br>
>> >> >> Best Regards<br>
>> >> >> Muhammad Usama<br>
>> >> >><br>
>> >> >>><br>
>> >> >>> Best regards,<br>
>> >> >>> --<br>
>> >> >>> Tatsuo Ishii<br>
>> >> >>> SRA OSS, Inc. Japan<br>
>> >> >>> English: <a href="http://www.sraoss.co.jp/index_en.php" rel="noreferrer" target="_blank">http://www.sraoss.co.jp/index_<wbr>en.php</a><br>
>> >> >>> Japanese:<a href="http://www.sraoss.co.jp" rel="noreferrer" target="_blank">http://www.sraoss.co.<wbr>jp</a><br>
>> >> >>><br>
>> >> >>> > Usama,<br>
>> >> >>> ><br>
>> >> >>> > Thanks for the patch. I am going to review it.<br>
>> >> >>> ><br>
>> >> >>> > In the mean time when I apply your patch, I got some trailing<br>
>> >> >>> > whitespace errors. Can you please fix them?<br>
>> >> >>> ><br>
>> >> >>> > /home/t-ishii/quorum_aware_<wbr>failover.diff:470: trailing<br>
>> whitespace.<br>
>> >> >>> ><br>
>> >> >>> > /home/t-ishii/quorum_aware_<wbr>failover.diff:485: trailing<br>
>> whitespace.<br>
>> >> >>> ><br>
>> >> >>> > /home/t-ishii/quorum_aware_<wbr>failover.diff:564: trailing<br>
>> whitespace.<br>
>> >> >>> ><br>
>> >> >>> > /home/t-ishii/quorum_aware_<wbr>failover.diff:1428: trailing<br>
>> whitespace.<br>
>> >> >>> ><br>
>> >> >>> > /home/t-ishii/quorum_aware_<wbr>failover.diff:1450: trailing<br>
>> whitespace.<br>
>> >> >>> ><br>
>> >> >>> > warning: squelched 3 whitespace errors<br>
>> >> >>> > warning: 8 lines add whitespace errors.<br>
>> >> >>> ><br>
>> >> >>> > Best regards,<br>
>> >> >>> > --<br>
>> >> >>> > Tatsuo Ishii<br>
>> >> >>> > SRA OSS, Inc. Japan<br>
>> >> >>> > English: <a href="http://www.sraoss.co.jp/index_en.php" rel="noreferrer" target="_blank">http://www.sraoss.co.jp/index_<wbr>en.php</a><br>
>> >> >>> > Japanese:<a href="http://www.sraoss.co.jp" rel="noreferrer" target="_blank">http://www.sraoss.co.<wbr>jp</a><br>
>> >> >>> ><br>
>> >> >>> >> Hi<br>
>> >> >>> >><br>
>> >> >>> >> I was working on the new feature to make the backend node<br>
>> failover<br>
>> >> >>> quorum<br>
>> >> >>> >> aware and on the half way through the implementation I also added<br>
>> >> the<br>
>> >> >>> >> majority consensus feature for the same.<br>
>> >> >>> >><br>
>> >> >>> >> So please find the first version of the patch for review that<br>
>> makes<br>
>> >> the<br>
>> >> >>> >> backend node failover consider the watchdog cluster quorum status<br>
>> >> and<br>
>> >> >>> seek<br>
>> >> >>> >> the majority consensus before performing failover.<br>
>> >> >>> >><br>
>> >> >>> >> *Changes in the Failover mechanism with watchdog.*<br>
>> >> >>> >> For this new feature I have modified the Pgpool-II's existing<br>
>> >> failover<br>
>> >> >>> >> mechanism with watchdog.<br>
>> >> >>> >> Previously as you know when the Pgpool-II require to perform a<br>
>> node<br>
>> >> >>> >> operation (failover, failback, promote-node) with the watchdog.<br>
>> The<br>
>> >> >>> >> watchdog used to propagated the failover request to all the<br>
>> >> Pgpool-II<br>
>> >> >>> nodes<br>
>> >> >>> >> in the watchdog cluster and as soon as the request was received<br>
>> by<br>
>> >> the<br>
>> >> >>> >> node, it used to initiate the local failover and that failover<br>
>> was<br>
>> >> >>> >> synchronised on all nodes using the distributed locks.<br>
>> >> >>> >><br>
>> >> >>> >> *Now Only the Master node performs the failover.*<br>
>> >> >>> >> The attached patch changes the mechanism of synchronised<br>
>> failover,<br>
>> >> and<br>
>> >> >>> now<br>
>> >> >>> >> only the Pgpool-II of master watchdog node performs the failover,<br>
>> >> and<br>
>> >> >>> all<br>
>> >> >>> >> other standby nodes sync the backend statuses after the master<br>
>> >> >>> Pgpool-II is<br>
>> >> >>> >> finished with the failover.<br>
>> >> >>> >><br>
>> >> >>> >> *Overview of new failover mechanism.*<br>
>> >> >>> >> -- If the failover request is received to the standby watchdog<br>
>> >> >>> node(from<br>
>> >> >>> >> local Pgpool-II), That request is forwarded to the master<br>
>> watchdog<br>
>> >> and<br>
>> >> >>> the<br>
>> >> >>> >> Pgpool-II main process is returned with the<br>
>> >> FAILOVER_RES_WILL_BE_DONE<br>
>> >> >>> >> return code. And upon receiving the FAILOVER_RES_WILL_BE_DONE<br>
>> from<br>
>> >> the<br>
>> >> >>> >> watchdog for the failover request the requesting Pgpool-II moves<br>
>> >> >>> forward<br>
>> >> >>> >> without doing anything further for the particular failover<br>
>> command.<br>
>> >> >>> >><br>
>> >> >>> >> -- Now when the failover request from standby node is received by<br>
>> >> the<br>
>> >> >>> >> master watchdog, after performing the validation, applying the<br>
>> >> >>> consensus<br>
>> >> >>> >> rules the failover request is triggered on the local Pgpool-II .<br>
>> >> >>> >><br>
>> >> >>> >> -- When the failover request is received to the master watchdog<br>
>> node<br>
>> >> >>> from<br>
>> >> >>> >> the local Pgpool-II (On the IPC channel) the watchdog process<br>
>> inform<br>
>> >> >>> the<br>
>> >> >>> >> Pgpool-II requesting process to proceed with failover (provided<br>
>> all<br>
>> >> >>> >> failover rules are satisfied).<br>
>> >> >>> >><br>
>> >> >>> >> -- After the failover is finished on the master Pgpool-II, the<br>
>> >> failover<br>
>> >> >>> >> function calls the *wd_failover_end*() which sends the backend<br>
>> sync<br>
>> >> >>> >> required message to all standby watchdogs.<br>
>> >> >>> >><br>
>> >> >>> >> -- Upon receiving the sync required message from master watchdog<br>
>> >> node<br>
>> >> >>> all<br>
>> >> >>> >> Pgpool-II sync the new statuses of each backend node from the<br>
>> master<br>
>> >> >>> >> watchdog.<br>
>> >> >>> >><br>
>> >> >>> >> *No More Failover locks*<br>
>> >> >>> >> Since with this new failover mechanism we do not require any<br>
>> >> >>> >> synchronisation and guards against the execution of<br>
>> >> failover_commands<br>
>> >> >>> by<br>
>> >> >>> >> multiple Pgpool-II nodes, So the patch removes all the<br>
>> distributed<br>
>> >> >>> locks<br>
>> >> >>> >> from failover function, This makes the failover simpler and<br>
>> faster.<br>
>> >> >>> >><br>
>> >> >>> >> *New kind of Failover operation NODE_QUARANTINE_REQUEST*<br>
>> >> >>> >> The patch adds the new kind of backend node operation<br>
>> >> NODE_QUARANTINE<br>
>> >> >>> which<br>
>> >> >>> >> is effectively same as the NODE_DOWN, but with node_quarantine<br>
>> the<br>
>> >> >>> >> failover_command is not triggered.<br>
>> >> >>> >> The NODE_DOWN_REQUEST is automatically converted to the<br>
>> >> >>> >> NODE_QUARANTINE_REQUEST when the failover is requested on the<br>
>> >> backend<br>
>> >> >>> node<br>
>> >> >>> >> but watchdog cluster does not holds the quorum.<br>
>> >> >>> >> This means in the absence of quorum the failed backend nodes are<br>
>> >> >>> >> quarantined and when the quorum becomes available again the<br>
>> >> Pgpool-II<br>
>> >> >>> >> performs the failback operation on all quarantine nodes.<br>
>> >> >>> >> And again when the failback is performed on the quarantine<br>
>> backend<br>
>> >> >>> node the<br>
>> >> >>> >> failover function does not trigger the failback_command.<br>
>> >> >>> >><br>
>> >> >>> >> *Controlling the Failover behaviour.*<br>
>> >> >>> >> The patch adds three new configuration parameters to configure<br>
>> the<br>
>> >> >>> failover<br>
>> >> >>> >> behaviour from user side.<br>
>> >> >>> >><br>
>> >> >>> >> *failover_when_quorum_exists*<br>
>> >> >>> >> When enabled the failover command will only be executed when the<br>
>> >> >>> watchdog<br>
>> >> >>> >> cluster holds the quorum. And when the quorum is absent and<br>
>> >> >>> >> failover_when_quorum_exists is enabled the failed backend nodes<br>
>> will<br>
>> >> >>> get<br>
>> >> >>> >> quarantine until the quorum becomes available again.<br>
>> >> >>> >> disabling it will enable the old behaviour of failover commands.<br>
>> >> >>> >><br>
>> >> >>> >><br>
>> >> >>> >> *failover_require_consensus*<wbr>This new configuration parameter<br>
>> can be<br>
>> >> >>> used to<br>
>> >> >>> >> make sure we get the majority vote before performing the<br>
>> failover on<br>
>> >> >>> the<br>
>> >> >>> >> node. When *failover_require_consensus* is enabled then the<br>
>> >> failover is<br>
>> >> >>> >> only performed after receiving the failover request from the<br>
>> >> majority<br>
>> >> >>> or<br>
>> >> >>> >> Pgpool-II nodes.<br>
>> >> >>> >> For example in three nodes cluster the failover will not be<br>
>> >> performed<br>
>> >> >>> until<br>
>> >> >>> >> at least two nodes ask for performing the failover on the<br>
>> particular<br>
>> >> >>> >> backend node.<br>
>> >> >>> >><br>
>> >> >>> >> It is also worthwhile to mention here that<br>
>> >> *failover_require_consensus*<br>
>> >> >>> >> only works when failover_when_quorum_exists is enables.<br>
>> >> >>> >><br>
>> >> >>> >><br>
>> >> >>> >> *enable_multiple_failover_<wbr>requests_from_node*<br>
>> >> >>> >> This parameter works in connection with<br>
>> *failover_require_consensus*<br>
>> >> >>> >> config. When enabled a single Pgpool-II node can vote for<br>
>> failover<br>
>> >> >>> multiple<br>
>> >> >>> >> times.<br>
>> >> >>> >> For example in the three nodes cluster if one Pgpool-II node<br>
>> sends<br>
>> >> the<br>
>> >> >>> >> failover request of particular node twice that would be counted<br>
>> as<br>
>> >> two<br>
>> >> >>> >> votes in favour of failover and the failover will be performed<br>
>> even<br>
>> >> if<br>
>> >> >>> we<br>
>> >> >>> >> do not get a vote from other two nodes.<br>
>> >> >>> >><br>
>> >> >>> >> And when *enable_multiple_failover_<wbr>requests_from_node* is<br>
>> disabled,<br>
>> >> >>> Only<br>
>> >> >>> >> the first vote from each Pgpool-II will be accepted and all other<br>
>> >> >>> >> subsequent votes will be marked duplicate and rejected.<br>
>> >> >>> >> So in that case we will require a majority votes from distinct<br>
>> >> nodes to<br>
>> >> >>> >> execute the failover.<br>
>> >> >>> >> Again this *enable_multiple_failover_<wbr>requests_from_node* only<br>
>> >> becomes<br>
>> >> >>> >> effective when both *failover_when_quorum_exists* and<br>
>> >> >>> >> *failover_require_consensus* are enabled.<br>
>> >> >>> >><br>
>> >> >>> >><br>
>> >> >>> >> *Controlling the failover: The Coding perspective.*<br>
>> >> >>> >> Although the failover functions are made quorum and consensus<br>
>> aware<br>
>> >> but<br>
>> >> >>> >> there is still a way to bypass the quorum conditions, and<br>
>> >> requirement<br>
>> >> >>> of<br>
>> >> >>> >> consensus.<br>
>> >> >>> >><br>
>> >> >>> >> For this the patch uses the existing request_details flags in<br>
>> >> >>> >> POOL_REQUEST_NODE to control the behaviour of failover.<br>
>> >> >>> >><br>
>> >> >>> >> Here are the newly added flags values.<br>
>> >> >>> >><br>
>> >> >>> >> *REQ_DETAIL_WATCHDOG*:<br>
>> >> >>> >> Setting this flag while issuing the failover command will not<br>
>> send<br>
>> >> the<br>
>> >> >>> >> failover request to the watchdog. But this flag may not be<br>
>> useful in<br>
>> >> >>> any<br>
>> >> >>> >> other place than where it is already used.<br>
>> >> >>> >> Mostly this flag can be used to avoid the failover command from<br>
>> >> going<br>
>> >> >>> to<br>
>> >> >>> >> watchdog that is already originated from watchdog. Otherwise we<br>
>> can<br>
>> >> >>> end up<br>
>> >> >>> >> in infinite loop.<br>
>> >> >>> >><br>
>> >> >>> >> *REQ_DETAIL_CONFIRMED*:<br>
>> >> >>> >> Setting this flag will bypass the *failover_require_consensus*<br>
>> >> >>> >> configuration and immediately perform the failover if quorum is<br>
>> >> >>> present.<br>
>> >> >>> >> This flag can be used to issue the failover request originated<br>
>> from<br>
>> >> PCP<br>
>> >> >>> >> command.<br>
>> >> >>> >><br>
>> >> >>> >> *REQ_DETAIL_UPDATE*:<br>
>> >> >>> >> This flag is used for the command where we are failing back the<br>
>> >> >>> quarantine<br>
>> >> >>> >> nodes. Setting this flag will not trigger the failback_command.<br>
>> >> >>> >><br>
>> >> >>> >> *Some conditional flags used:*<br>
>> >> >>> >> I was not sure about the configuration of each type of failover<br>
>> >> >>> operation.<br>
>> >> >>> >> As we have three main failover operations NODE_UP_REQUEST,<br>
>> >> >>> >> NODE_DOWN_REQUEST, and PROMOTE_NODE_REQUEST<br>
>> >> >>> >> So I was thinking do we need to give the configuration option to<br>
>> the<br>
>> >> >>> users,<br>
>> >> >>> >> if they want to enable/disable quorum checking and consensus for<br>
>> >> >>> individual<br>
>> >> >>> >> failover operation type.<br>
>> >> >>> >> For example: is it a practical configuration where a user would<br>
>> >> want to<br>
>> >> >>> >> ensure quorum while preforming NODE_DOWN operation while does not<br>
>> >> want<br>
>> >> >>> it<br>
>> >> >>> >> for NODE_UP.<br>
>> >> >>> >> So in this patch I use three compile time defines to enable<br>
>> disable<br>
>> >> the<br>
>> >> >>> >> individual failover operation, while we can decide on the best<br>
>> >> >>> solution.<br>
>> >> >>> >><br>
>> >> >>> >> NODE_UP_REQUIRE_CONSENSUS: defining it will enable quorum<br>
>> checking<br>
>> >> >>> feature<br>
>> >> >>> >> for NODE_UP_REQUESTs<br>
>> >> >>> >><br>
>> >> >>> >> NODE_DOWN_REQUIRE_CONSENSUS: defining it will enable quorum<br>
>> checking<br>
>> >> >>> >> feature for NODE_DOWN_REQUESTs<br>
>> >> >>> >><br>
>> >> >>> >> NODE_PROMOTE_REQUIRE_<wbr>CONSENSUS: defining it will enable quorum<br>
>> >> >>> checking<br>
>> >> >>> >> feature for PROMOTE_NODE_REQUESTs<br>
>> >> >>> >><br>
>> >> >>> >> *Some Point for Discussion:*<br>
>> >> >>> >><br>
>> >> >>> >> *Do we really need to check ReqInfo->switching flag before<br>
>> enqueuing<br>
>> >> >>> >> failover request.*<br>
>> >> >>> >> While working on the patch I was wondering why do we disallow<br>
>> >> >>> enqueuing the<br>
>> >> >>> >> failover command when the failover is already in progress? For<br>
>> >> example<br>
>> >> >>> in<br>
>> >> >>> >> *pcp_process_command*() function if we see the<br>
>> *Req_info->switching*<br>
>> >> >>> flag<br>
>> >> >>> >> set we bailout with the error instead of enqueuing the command.<br>
>> Is<br>
>> >> is<br>
>> >> >>> >> really necessary?<br>
>> >> >>> >><br>
>> >> >>> >> *Do we need more granule control over each failover operation:*<br>
>> >> >>> >> As described in section "Some conditional flags used" I want the<br>
>> >> >>> opinion on<br>
>> >> >>> >> do we need configuration parameters in pgpool.conf to enable<br>
>> disable<br>
>> >> >>> quorum<br>
>> >> >>> >> and consensus checking on individual failover types.<br>
>> >> >>> >><br>
>> >> >>> >> *Which failover should be mark as Confirmed:*<br>
>> >> >>> >> As defined in the above section of REQ_DETAIL_CONFIRMED, We can<br>
>> mark<br>
>> >> >>> the<br>
>> >> >>> >> failover request to not need consensus, currently the requests<br>
>> from<br>
>> >> >>> the PCP<br>
>> >> >>> >> commands are fired with this flag. But I was wondering there may<br>
>> be<br>
>> >> >>> more<br>
>> >> >>> >> places where we many need to use the flag.<br>
>> >> >>> >> For example I currently use the same confirmed flag when<br>
>> failover is<br>
>> >> >>> >> triggered because of *replication_stop_on_mismatch*<wbr>.<br>
>> >> >>> >><br>
>> >> >>> >> I think we should think this flag for each place of failover,<br>
>> like<br>
>> >> >>> when the<br>
>> >> >>> >> failover is triggered<br>
>> >> >>> >> because of health_check failure.<br>
>> >> >>> >> because of replication mismatch<br>
>> >> >>> >> because of backend_error<br>
>> >> >>> >> e.t.c<br>
>> >> >>> >><br>
>> >> >>> >> *Node Quarantine behaviour.*<br>
>> >> >>> >> What do you think about the node quarantine used by this patch.<br>
>> Can<br>
>> >> you<br>
>> >> >>> >> think of some problem which can be caused by this?<br>
>> >> >>> >><br>
>> >> >>> >> *What should be the default values for each newly added config<br>
>> >> >>> parameters.*<br>
>> >> >>> >><br>
>> >> >>> >><br>
>> >> >>> >><br>
>> >> >>> >> *TODOs*<br>
>> >> >>> >><br>
>> >> >>> >> -- Updating the documentation is still todo. Will do that once<br>
>> every<br>
>> >> >>> aspect<br>
>> >> >>> >> of the feature will be finalised.<br>
>> >> >>> >> -- Some code warnings and cleanups are still not done.<br>
>> >> >>> >> -- I am still little short on testing<br>
>> >> >>> >> -- Regression test cases for the feature<br>
>> >> >>> >><br>
>> >> >>> >><br>
>> >> >>> >> Thoughts and suggestions are most welcome.<br>
>> >> >>> >><br>
>> >> >>> >> Thanks<br>
>> >> >>> >> Best regards<br>
>> >> >>> >> Muhammad Usama<br>
>> >> >>> > ______________________________<wbr>_________________<br>
>> >> >>> > pgpool-hackers mailing list<br>
>> >> >>> > <a href="mailto:pgpool-hackers@pgpool.net">pgpool-hackers@pgpool.net</a><br>
>> >> >>> > <a href="http://www.pgpool.net/mailman/listinfo/pgpool-hackers" rel="noreferrer" target="_blank">http://www.pgpool.net/mailman/<wbr>listinfo/pgpool-hackers</a><br>
>> >> >>><br>
>> >> >><br>
>> >> >><br>
>> >><br>
>><br>
</div></div><br>2017-09-08 09:29:29: pid 4434: LOG: Backend status file /home/t-ishii/work/pgpool-II/<wbr>current/pgpool2/src/test/<wbr>regression/tests/004.watchdog/<wbr>master/log/pgpool_status discarded<br>
2017-09-08 09:29:29: pid 4434: LOG: waiting for watchdog to initialize<br>
2017-09-08 09:29:29: pid 4436: LOG: setting the local watchdog node name to "localhost:11000 Linux tishii-CF-SX3HE4BP"<br>
2017-09-08 09:29:29: pid 4436: LOG: watchdog cluster is configured with 1 remote nodes<br>
2017-09-08 09:29:29: pid 4436: LOG: watchdog remote node:0 on localhost:21104<br>
2017-09-08 09:29:29: pid 4436: LOG: interface monitoring is disabled in watchdog<br>
2017-09-08 09:29:29: pid 4436: LOG: watchdog node state changed from [DEAD] to [LOADING]<br>
2017-09-08 09:29:34: pid 4436: LOG: watchdog node state changed from [LOADING] to [JOINING]<br>
2017-09-08 09:29:38: pid 4436: LOG: watchdog node state changed from [JOINING] to [INITIALIZING]<br>
2017-09-08 09:29:39: pid 4436: LOG: I am the only alive node in the watchdog cluster<br>
2017-09-08 09:29:39: pid 4436: HINT: skiping stand for coordinator state<br>
2017-09-08 09:29:39: pid 4436: LOG: watchdog node state changed from [INITIALIZING] to [MASTER]<br>
2017-09-08 09:29:39: pid 4436: LOG: I am announcing my self as master/coordinator watchdog node<br>
2017-09-08 09:29:41: pid 4436: LOG: new watchdog node connection is received from "<a href="http://127.0.0.1:33967" rel="noreferrer" target="_blank">127.0.0.1:33967</a>"<br>
2017-09-08 09:29:41: pid 4436: LOG: new node joined the cluster hostname:"localhost" port:21104 pgpool_port:11100<br>
2017-09-08 09:29:42: pid 4436: LOG: adding watchdog node "localhost:11100 Linux tishii-CF-SX3HE4BP" to the standby list<br>
2017-09-08 09:29:42: pid 4436: LOG: I am the cluster leader node<br>
2017-09-08 09:29:42: pid 4436: DETAIL: our declare coordinator message is accepted by all nodes<br>
2017-09-08 09:29:42: pid 4436: LOG: setting the local node "localhost:11000 Linux tishii-CF-SX3HE4BP" as watchdog cluster master<br>
2017-09-08 09:29:42: pid 4436: LOG: I am the cluster leader node. Starting escalation process<br>
2017-09-08 09:29:42: pid 4436: LOG: escalation process started with PID:4444<br>
2017-09-08 09:29:42: pid 4434: LOG: watchdog process is initialized<br>
2017-09-08 09:29:42: pid 4444: LOG: watchdog: escalation started<br>
2017-09-08 09:29:42: pid 4436: LOG: new IPC connection received<br>
2017-09-08 09:29:42: pid 4436: LOG: new IPC connection received<br>
2017-09-08 09:29:42: pid 4446: LOG: 2 watchdog nodes are configured for lifecheck<br>
2017-09-08 09:29:42: pid 4446: LOG: watchdog nodes ID:0 Name:"localhost:11000 Linux tishii-CF-SX3HE4BP"<br>
2017-09-08 09:29:42: pid 4446: DETAIL: Host:"localhost" WD Port:21004 pgpool-II port:11000<br>
2017-09-08 09:29:42: pid 4446: LOG: watchdog nodes ID:1 Name:"localhost:11100 Linux tishii-CF-SX3HE4BP"<br>
2017-09-08 09:29:42: pid 4446: DETAIL: Host:"localhost" WD Port:21104 pgpool-II port:11100<br>
2017-09-08 09:29:42: pid 4434: LOG: Setting up socket for <a href="http://127.0.0.1:11000" rel="noreferrer" target="_blank">127.0.0.1:11000</a><br>
2017-09-08 09:29:42: pid 4436: LOG: watchdog escalation process with pid: 4444 exit with SUCCESS.<br>
2017-09-08 09:29:42: pid 4434: LOG: pgpool-II successfully started. version 3.7devel (amefuriboshi)<br>
2017-09-08 09:29:43: pid 4448: LOG: set SO_REUSEPORT option to the socket<br>
2017-09-08 09:29:43: pid 4448: LOG: creating socket for sending heartbeat<br>
2017-09-08 09:29:43: pid 4448: DETAIL: set SO_REUSEPORT<br>
2017-09-08 09:29:43: pid 4447: LOG: set SO_REUSEPORT option to the socket<br>
2017-09-08 09:29:43: pid 4447: LOG: creating watchdog heartbeat receive socket.<br>
2017-09-08 09:29:43: pid 4447: DETAIL: set SO_REUSEPORT<br>
2017-09-08 09:29:45: pid 4436: LOG: new outbond connection to localhost:21104<br>
2017-09-08 09:29:52: pid 4446: LOG: watchdog: lifecheck started<br>
2017-09-08 09:31:42: pid 4498: LOG: failed to connect to PostgreSQL server by unix domain socket<br>
2017-09-08 09:31:42: pid 4498: DETAIL: connect to "/tmp/.s.PGSQL.11002" failed with error "No such file or directory"<br>
2017-09-08 09:31:42: pid 4498: ERROR: failed to make persistent db connection<br>
2017-09-08 09:31:42: pid 4498: DETAIL: connection to host:"/tmp:11002" failed<br>
2017-09-08 09:31:42: pid 4498: LOG: health check failed on node 0 (timeout:0)<br>
2017-09-08 09:31:42: pid 4498: LOG: received degenerate backend request for node_id: 0 from pid [4498]<br>
2017-09-08 09:31:42: pid 4436: LOG: new IPC connection received<br>
2017-09-08 09:31:42: pid 4436: LOG: watchdog received the failover command from local pgpool-II on IPC interface<br>
2017-09-08 09:31:42: pid 4436: LOG: watchdog is processing the failover command [DEGENERATE_BACKEND_REQUEST] received from local pgpool-II on IPC interface<br>
2017-09-08 09:31:42: pid 4436: LOG: we do not require majority votes to proceed with failover<br>
2017-09-08 09:31:42: pid 4436: DETAIL: proceeding with the failover<br>
2017-09-08 09:31:42: pid 4436: HINT: failover_require_consensus is set to false<br>
2017-09-08 09:31:42: pid 4434: LOG: Pgpool-II parent process has received failover request<br>
2017-09-08 09:31:42: pid 4436: LOG: new IPC connection received<br>
2017-09-08 09:31:42: pid 4436: LOG: received the failover indication from Pgpool-II on IPC interface<br>
2017-09-08 09:31:42: pid 4436: LOG: watchdog is informed of failover end<br>
2017-09-08 09:31:42: pid 4434: LOG: starting degeneration. shutdown host /tmp(11002)<br>
2017-09-08 09:31:42: pid 4434: WARNING: All the DB nodes are in down status and skip writing status file.<br>
2017-09-08 09:31:42: pid 4434: LOG: failover: no valid backends node found<br>
2017-09-08 09:31:42: pid 4434: LOG: Restart all children<br>
2017-09-08 09:31:42: pid 4434: LOG: failover: set new primary node: -1<br>
2017-09-08 09:31:42: pid 4436: LOG: watchdog received the failover command from remote pgpool-II node "localhost:11100 Linux tishii-CF-SX3HE4BP"<br>
2017-09-08 09:31:42: pid 4436: LOG: watchdog is processing the failover command [DEGENERATE_BACKEND_REQUEST] received from localhost:11100 Linux tishii-CF-SX3HE4BP<br>
2017-09-08 09:31:42: pid 4436: LOG: we do not require majority votes to proceed with failover<br>
2017-09-08 09:31:42: pid 4436: DETAIL: proceeding with the failover<br>
2017-09-08 09:31:42: pid 4436: HINT: failover_require_consensus is set to false<br>
2017-09-08 09:31:42: pid 4436: LOG: received degenerate backend request for node_id: 0 from pid [4436]<br>
2017-09-08 09:31:42: pid 4497: LOG: worker process received restart request<br>
2017-09-08 09:31:42: pid 4436: LOG: new IPC connection received<br>
2017-09-08 09:31:42: pid 4436: LOG: received the failover indication from Pgpool-II on IPC interface<br>
2017-09-08 09:31:42: pid 4436: LOG: watchdog is informed of failover start<br>
failover done. shutdown host /tmp(11002)2017-09-08 09:31:42: pid 4434: LOG: failover done. shutdown host /tmp(11002)<br>
2017-09-08 09:31:42: pid 4436: LOG: new IPC connection received<br>
2017-09-08 09:31:42: pid 4436: LOG: received the failover indication from Pgpool-II on IPC interface<br>
2017-09-08 09:31:42: pid 4436: LOG: watchdog is informed of failover end<br>
2017-09-08 09:31:42: pid 4434: LOG: starting degeneration. shutdown host /tmp(11002)<br>
2017-09-08 09:31:42: pid 4434: WARNING: All the DB nodes are in down status and skip writing status file.<br>
2017-09-08 09:31:42: pid 4434: LOG: failover: no valid backends node found<br>
2017-09-08 09:31:42: pid 4434: LOG: Restart all children<br>
2017-09-08 09:31:42: pid 4434: LOG: failover: set new primary node: -1<br>
2017-09-08 09:31:42: pid 4436: LOG: new IPC connection received<br>
2017-09-08 09:31:42: pid 4436: LOG: received the failover indication from Pgpool-II on IPC interface<br>
2017-09-08 09:31:42: pid 4436: LOG: watchdog is informed of failover start<br>
failover done. shutdown host /tmp(11002)2017-09-08 09:31:42: pid 4434: LOG: failover done. shutdown host /tmp(11002)<br>
2017-09-08 09:31:43: pid 4670: LOG: failed to connect to PostgreSQL server by unix domain socket<br>
2017-09-08 09:31:43: pid 4670: DETAIL: connect to "/tmp/.s.PGSQL.11002" failed with error "No such file or directory"<br>
2017-09-08 09:31:43: pid 4670: LOG: received degenerate backend request for node_id: 0 from pid [4670]<br>
2017-09-08 09:31:43: pid 4436: LOG: new IPC connection received<br>
2017-09-08 09:31:43: pid 4436: LOG: watchdog received the failover command from local pgpool-II on IPC interface<br>
2017-09-08 09:31:43: pid 4436: LOG: watchdog is processing the failover command [DEGENERATE_BACKEND_REQUEST] received from local pgpool-II on IPC interface<br>
2017-09-08 09:31:43: pid 4436: LOG: Duplicate failover request from "localhost:11000 Linux tishii-CF-SX3HE4BP" node<br>
2017-09-08 09:31:43: pid 4436: DETAIL: request ignored<br>
2017-09-08 09:31:43: pid 4436: LOG: we do not require majority votes to proceed with failover<br>
2017-09-08 09:31:43: pid 4436: DETAIL: proceeding with the failover<br>
2017-09-08 09:31:43: pid 4436: HINT: failover_require_consensus is set to false<br>
2017-09-08 09:31:43: pid 4670: FATAL: failed to create a backend connection<br>
2017-09-08 09:31:43: pid 4670: DETAIL: executing failover on backend<br>
2017-09-08 09:31:43: pid 4496: LOG: restart request received in pcp child process<br>
2017-09-08 09:31:43: pid 4434: LOG: PCP child 4496 exits with status 0 in failover()<br>
2017-09-08 09:31:43: pid 4434: LOG: fork a new PCP child pid 4676 in failover()<br>
2017-09-08 09:31:43: pid 4434: LOG: Pgpool-II parent process has received failover request<br>
2017-09-08 09:31:43: pid 4436: LOG: new IPC connection received<br>
2017-09-08 09:31:43: pid 4436: LOG: received the failover indication from Pgpool-II on IPC interface<br>
2017-09-08 09:31:43: pid 4436: LOG: watchdog is informed of failover end<br>
2017-09-08 09:31:43: pid 4434: LOG: starting degeneration. shutdown host /tmp(11002)<br>
2017-09-08 09:31:43: pid 4434: WARNING: All the DB nodes are in down status and skip writing status file.<br>
2017-09-08 09:31:43: pid 4434: LOG: failover: no valid backends node found<br>
2017-09-08 09:31:43: pid 4434: LOG: Restart all children<br>
2017-09-08 09:31:43: pid 4434: LOG: failover: set new primary node: -1<br>
2017-09-08 09:31:43: pid 4436: LOG: new IPC connection received<br>
2017-09-08 09:31:43: pid 4436: LOG: received the failover indication from Pgpool-II on IPC interface<br>
2017-09-08 09:31:43: pid 4436: LOG: watchdog is informed of failover start<br>
failover done. shutdown host /tmp(11002)2017-09-08 09:31:43: pid 4434: LOG: failover done. shutdown host /tmp(11002)<br>
2017-09-08 09:31:44: pid 4676: LOG: restart request received in pcp child process<br>
2017-09-08 09:31:44: pid 4434: LOG: PCP child 4676 exits with status 0 in failover()<br>
2017-09-08 09:31:44: pid 4434: LOG: fork a new PCP child pid 4709 in failover()<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4449 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4449 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4450 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4450 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4452 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4452 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4454 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4454 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4456 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4456 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4459 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4459 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4460 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4460 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4462 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4462 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4465 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4465 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4467 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4467 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4469 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4469 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4471 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4471 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4473 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4473 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4475 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4475 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4476 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4476 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4477 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4477 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4479 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4479 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4481 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4481 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4482 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4482 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4483 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4483 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4484 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4484 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4485 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4485 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4486 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4486 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4487 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4487 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4488 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4488 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4489 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4489 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4490 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4490 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4491 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4491 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4492 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4492 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4493 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4493 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4494 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4494 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4495 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4495 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: worker child process with pid: 4497 exits with status 256<br>
2017-09-08 09:31:44: pid 4434: LOG: fork a new worker child process with pid: 4710<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4578 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4578 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4579 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4579 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4580 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4580 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4581 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4581 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4582 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4582 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4583 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4583 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4584 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4584 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4585 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4585 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4586 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4586 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4587 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4587 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4588 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4588 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4589 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4589 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4590 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4590 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4591 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4591 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4592 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4592 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4593 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4593 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4594 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4594 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4595 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4595 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4596 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4596 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4597 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4597 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4598 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4598 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4599 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4599 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4600 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4600 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4601 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4601 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4614 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4614 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4616 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4616 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4618 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4618 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4619 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4619 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4620 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4620 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4621 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4621 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4623 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4623 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4624 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4624 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4642 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4642 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4643 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4643 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4644 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4644 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4645 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4645 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4646 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4646 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4647 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4647 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4648 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4648 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4649 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4649 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4650 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4650 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4651 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4651 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4652 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4652 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4653 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4653 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4654 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4654 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4655 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4655 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4656 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4656 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4657 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4657 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4658 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4658 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4659 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4659 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4660 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4660 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4661 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4661 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4662 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4662 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4663 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4663 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4664 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4664 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4665 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4665 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4666 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4666 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4668 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4668 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4669 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4669 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4670 exits with status 256<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4670 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4671 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4671 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4672 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4672 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4673 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4673 exited with success and will not be restarted<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4674 exits with status 0<br>
2017-09-08 09:31:44: pid 4434: LOG: child process with pid: 4674 exited with success and will not be restarted<br>
2017-09-08 09:31:46: pid 4706: LOG: failed to connect to PostgreSQL server by unix domain socket<br>
2017-09-08 09:31:46: pid 4706: DETAIL: connect to "/tmp/.s.PGSQL.11002" failed with error "No such file or directory"<br>
2017-09-08 09:31:46: pid 4706: LOG: received degenerate backend request for node_id: 0 from pid [4706]<br>
2017-09-08 09:31:46: pid 4436: LOG: new IPC connection received<br>
2017-09-08 09:31:46: pid 4436: LOG: watchdog received the failover command from local pgpool-II on IPC interface<br>
2017-09-08 09:31:46: pid 4436: LOG: watchdog is processing the failover command [DEGENERATE_BACKEND_REQUEST] received from local pgpool-II on IPC interface<br>
2017-09-08 09:31:46: pid 4436: LOG: Duplicate failover request from "localhost:11000 Linux tishii-CF-SX3HE4BP" node<br>
2017-09-08 09:31:46: pid 4436: DETAIL: request ignored<br>
2017-09-08 09:31:46: pid 4436: LOG: we do not require majority votes to proceed with failover<br>
2017-09-08 09:31:46: pid 4436: DETAIL: proceeding with the failover<br>
2017-09-08 09:31:46: pid 4436: HINT: failover_require_consensus is set to false<br>
2017-09-08 09:31:46: pid 4706: FATAL: failed to create a backend connection<br>
2017-09-08 09:31:46: pid 4706: DETAIL: executing failover on backend<br>
2017-09-08 09:31:46: pid 4434: LOG: Pgpool-II parent process has received failover request<br>
2017-09-08 09:31:46: pid 4436: LOG: new IPC connection received<br>
2017-09-08 09:31:46: pid 4436: LOG: received the failover indication from Pgpool-II on IPC interface<br>
2017-09-08 09:31:46: pid 4436: LOG: watchdog is informed of failover end<br>
2017-09-08 09:31:46: pid 4434: LOG: starting degeneration. shutdown host /tmp(11002)<br>
2017-09-08 09:31:46: pid 4434: WARNING: All the DB nodes are in down status and skip writing status file.<br>
2017-09-08 09:31:46: pid 4434: LOG: failover: no valid backends node found<br>
2017-09-08 09:31:46: pid 4434: LOG: Restart all children<br>
2017-09-08 09:31:46: pid 4434: LOG: failover: set new primary node: -1<br>
2017-09-08 09:31:46: pid 4710: LOG: worker process received restart request<br>
2017-09-08 09:31:46: pid 4436: LOG: new IPC connection received<br>
2017-09-08 09:31:46: pid 4436: LOG: received the failover indication from Pgpool-II on IPC interface<br>
2017-09-08 09:31:46: pid 4436: LOG: watchdog is informed of failover start<br>
failover done. shutdown host /tmp(11002)2017-09-08 09:31:46: pid 4434: LOG: failover done. shutdown host /tmp(11002)<br>
2017-09-08 09:31:47: pid 4709: LOG: restart request received in pcp child process<br>
2017-09-08 09:31:47: pid 4434: LOG: PCP child 4709 exits with status 0 in failover()<br>
2017-09-08 09:31:47: pid 4434: LOG: fork a new PCP child pid 4744 in failover()<br>
2017-09-08 09:31:47: pid 4434: LOG: child process with pid: 4677 exits with status 0<br>
2017-09-08 09:31:47: pid 4434: LOG: child process with pid: 4677 exited with success and will not be restarted<br>
2017-09-08 09:31:47: pid 4434: LOG: child process with pid: 4678 exits with status 0<br>
2017-09-08 09:31:47: pid 4434: LOG: child process with pid: 4678 exited with success and will not be restarted<br>
2017-09-08 09:31:47: pid 4434: LOG: child process with pid: 4679 exits with status 0<br>
2017-09-08 09:31:47: pid 4434: LOG: child process with pid: 4679 exited with success and will not be restarted<br>
2017-09-08 09:31:47: pid 4434: LOG: child process with pid: 4680 exits with status 0<br>
2017-09-08 09:31:47: pid 4434: LOG: child process with pid: 4680 exited with success and will not be restarted<br>
2017-09-08 09:31:47: pid 4434: LOG: child process with pid: 4681 exits with status 0<br>
2017-09-08 09:31:47: pid 4434: LOG: child process with pid: 4681 exited with success and will not be restarted<br>
2017-09-08 09:31:47: pid 4434: LOG: child process with pid: 4682 exits with status 0<br>
2017-09-08 09:31:47: pid 4434: LOG: child process with pid: 4682 exited with success and will not be restarted<br>
2017-09-08 09:31:47: pid 4434: LOG: child process with pid: 4683 exits with status 0<br>
2017-09-08 09:31:47: pid 4434: LOG: child process with pid: 4683 exited with success and will not be restarted<br>
2017-09-08 09:31:47: pid 4434: LOG: child process with pid: 4684 exits with status 0<br>
2017-09-08 09:31:47: pid 4434: LOG: child process with pid: 4684 exited with success and will not be restarted<br>
2017-09-08 09:31:47: pid 4434: LOG: child process with pid: 4685 exits with status 0<br>
2017-09-08 09:31:47: pid 4434: LOG: child process with pid: 4685 exited with success and will not be restarted<br>
2017-09-08 09:31:47: pid 4434: LOG: child process with pid: 4686 exits with status 0<br>
2017-09-08 09:31:47: pid 4434: LOG: child process with pid: 4686 exited with success and will not be restarted<br>
2017-09-08 09:31:47: pid 4434: LOG: child process with pid: 4687 exits with status 0<br>
2017-09-08 09:31:47: pid 4434: LOG: child process with pid: 4687 exited with success and will not be restarted<br>
2017-09-08 09:31:47: pid 4434: LOG: child process with pid: 4688 exits with status 0<br>
2017-09-08 09:31:47: pid 4434: LOG: child process with pid: 4688 exited with success and will not be restarted<br>
2017-09-08 09:31:47: pid 4434: LOG: child process with pid: 4689 exits with status 0<br>
2017-09-08 09:31:47: pid 4434: LOG: child process with pid: 4689 exited with success and will not be restarted<br>
2017-09-08 09:31:47: pid 4434: LOG: child process with pid: 4690 exits with status 0<br>
2017-09-08 09:31:47: pid 4434: LOG: child process with pid: 4690 exited with success and will not be restarted<br>
2017-09-08 09:31:47: pid 4434: LOG: child process with pid: 4691 exits with status 0<br>
2017-09-08 09:31:47: pid 4434: LOG: child process with pid: 4691 exited with success and will not be restarted<br>
2017-09-08 09:31:47: pid 4434: LOG: child process with pid: 4692 exits with status 0<br>
2017-09-08 09:31:47: pid 4434: LOG: child process with pid: 4692 exited with success and will not be restarted<br>
2017-09-08 09:31:47: pid 4434: LOG: child process with pid: 4693 exits with status 0<br>
2017-09-08 09:31:47: pid 4434: LOG: child process with pid: 4693 exited with success and will not be restarted<br>
2017-09-08 09:31:47: pid 4434: LOG: child process with pid: 4694 exits with status 0<br>
2017-09-08 09:31:47: pid 4434: LOG: child process with pid: 4694 exited with success and will not be restarted<br>
2017-09-08 09:31:47: pid 4434: LOG: child process with pid: 4695 exits with status 0<br>
2017-09-08 09:31:47: pid 4434: LOG: child process with pid: 4695 exited with success and will not be restarted<br>
2017-09-08 09:31:47: pid 4434: LOG: child process with pid: 4696 exits with status 0<br>
2017-09-08 09:31:47: pid 4434: LOG: child process with pid: 4696 exited with success and will not be restarted<br>
2017-09-08 09:31:47: pid 4434: LOG: child process with pid: 4697 exits with status 0<br>
2017-09-08 09:31:47: pid 4434: LOG: child process with pid: 4697 exited with success and will not be restarted<br>
2017-09-08 09:31:47: pid 4434: LOG: child process with pid: 4698 exits with status 0<br>
2017-09-08 09:31:47: pid 4434: LOG: child process with pid: 4698 exited with success and will not be restarted<br>
2017-09-08 09:31:47: pid 4434: LOG: child process with pid: 4699 exits with status 0<br>
2017-09-08 09:31:47: pid 4434: LOG: child process with pid: 4699 exited with success and will not be restarted<br>
2017-09-08 09:31:47: pid 4434: LOG: child process with pid: 4700 exits with status 0<br>
2017-09-08 09:31:47: pid 4434: LOG: child process with pid: 4700 exited with success and will not be restarted<br>
2017-09-08 09:31:47: pid 4434: LOG: child process with pid: 4701 exits with status 0<br>
2017-09-08 09:31:47: pid 4434: LOG: child process with pid: 4701 exited with success and will not be restarted<br>
2017-09-08 09:31:47: pid 4434: LOG: child process with pid: 4702 exits with status 0<br>
2017-09-08 09:31:47: pid 4434: LOG: child process with pid: 4702 exited with success and will not be restarted<br>
2017-09-08 09:31:47: pid 4434: LOG: child process with pid: 4703 exits with status 0<br>
2017-09-08 09:31:47: pid 4434: LOG: child process with pid: 4703 exited with success and will not be restarted<br>
2017-09-08 09:31:47: pid 4434: LOG: child process with pid: 4704 exits with status 0<br>
2017-09-08 09:31:47: pid 4434: LOG: child process with pid: 4704 exited with success and will not be restarted<br>
2017-09-08 09:31:47: pid 4434: LOG: child process with pid: 4705 exits with status 0<br>
2017-09-08 09:31:47: pid 4434: LOG: child process with pid: 4705 exited with success and will not be restarted<br>
2017-09-08 09:31:47: pid 4434: LOG: child process with pid: 4706 exits with status 256<br>
2017-09-08 09:31:47: pid 4434: LOG: child process with pid: 4706 exited with success and will not be restarted<br>
2017-09-08 09:31:47: pid 4434: LOG: child process with pid: 4707 exits with status 0<br>
2017-09-08 09:31:47: pid 4434: LOG: child process with pid: 4707 exited with success and will not be restarted<br>
2017-09-08 09:31:47: pid 4434: LOG: child process with pid: 4708 exits with status 0<br>
2017-09-08 09:31:47: pid 4434: LOG: child process with pid: 4708 exited with success and will not be restarted<br>
2017-09-08 09:31:47: pid 4434: LOG: worker child process with pid: 4710 exits with status 256<br>
2017-09-08 09:31:47: pid 4434: LOG: fork a new worker child process with pid: 4745<br>
2017-09-08 09:33:10: pid 4434: LOG: received fast shutdown request<br>
2017-09-08 09:33:10: pid 4434: LOG: shutdown request. closing listen socket<br>
2017-09-08 09:33:10: pid 4436: LOG: Watchdog is shutting down<br>
2017-09-08 09:33:10: pid 4766: LOG: watchdog: de-escalation started<br>
2017-09-08 09:33:10: pid 4434: WARNING: All the DB nodes are in down status and skip writing status file.<br>
<br>2017-09-08 09:29:41: pid 4441: LOG: Backend status file /home/t-ishii/work/pgpool-II/<wbr>current/pgpool2/src/test/<wbr>regression/tests/004.watchdog/<wbr>master/log/pgpool_status does not exist<br>
2017-09-08 09:29:41: pid 4441: LOG: waiting for watchdog to initialize<br>
2017-09-08 09:29:41: pid 4443: LOG: setting the local watchdog node name to "localhost:11100 Linux tishii-CF-SX3HE4BP"<br>
2017-09-08 09:29:41: pid 4443: LOG: watchdog cluster is configured with 1 remote nodes<br>
2017-09-08 09:29:41: pid 4443: LOG: watchdog remote node:0 on localhost:21004<br>
2017-09-08 09:29:41: pid 4443: LOG: interface monitoring is disabled in watchdog<br>
2017-09-08 09:29:41: pid 4443: LOG: watchdog node state changed from [DEAD] to [LOADING]<br>
2017-09-08 09:29:41: pid 4443: LOG: new outbond connection to localhost:21004<br>
2017-09-08 09:29:41: pid 4443: LOG: setting the remote node "localhost:11000 Linux tishii-CF-SX3HE4BP" as watchdog cluster master<br>
2017-09-08 09:29:41: pid 4443: LOG: watchdog node state changed from [LOADING] to [INITIALIZING]<br>
2017-09-08 09:29:42: pid 4443: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY]<br>
2017-09-08 09:29:42: pid 4443: LOG: successfully joined the watchdog cluster as standby node<br>
2017-09-08 09:29:42: pid 4443: DETAIL: our join coordinator request is accepted by cluster leader node "localhost:11000 Linux tishii-CF-SX3HE4BP"<br>
2017-09-08 09:29:42: pid 4441: LOG: watchdog process is initialized<br>
2017-09-08 09:29:42: pid 4443: LOG: new IPC connection received<br>
2017-09-08 09:29:42: pid 4441: LOG: we have joined the watchdog cluster as STANDBY node<br>
2017-09-08 09:29:42: pid 4441: DETAIL: syncing the backend states from the MASTER watchdog node<br>
2017-09-08 09:29:42: pid 4443: LOG: new IPC connection received<br>
2017-09-08 09:29:42: pid 4443: LOG: received the get data request from local pgpool-II on IPC interface<br>
2017-09-08 09:29:42: pid 4443: LOG: get data request from local pgpool-II node received on IPC interface is forwarded to master watchdog node "localhost:11000 Linux tishii-CF-SX3HE4BP"<br>
2017-09-08 09:29:42: pid 4443: DETAIL: waiting for the reply...<br>
2017-09-08 09:29:42: pid 4443: LOG: new IPC connection received<br>
2017-09-08 09:29:42: pid 4441: LOG: master watchdog node "localhost:11000 Linux tishii-CF-SX3HE4BP" returned status for 1 backend nodes<br>
2017-09-08 09:29:42: pid 4441: LOG: Setting up socket for <a href="http://127.0.0.1:11100" rel="noreferrer" target="_blank">127.0.0.1:11100</a><br>
2017-09-08 09:29:42: pid 4445: LOG: 2 watchdog nodes are configured for lifecheck<br>
2017-09-08 09:29:42: pid 4445: LOG: watchdog nodes ID:0 Name:"localhost:11100 Linux tishii-CF-SX3HE4BP"<br>
2017-09-08 09:29:42: pid 4445: DETAIL: Host:"localhost" WD Port:21104 pgpool-II port:11100<br>
2017-09-08 09:29:42: pid 4445: LOG: watchdog nodes ID:1 Name:"localhost:11000 Linux tishii-CF-SX3HE4BP"<br>
2017-09-08 09:29:42: pid 4445: DETAIL: Host:"localhost" WD Port:21004 pgpool-II port:11000<br>
2017-09-08 09:29:42: pid 4441: LOG: pgpool-II successfully started. version 3.7devel (amefuriboshi)<br>
2017-09-08 09:29:43: pid 4480: LOG: set SO_REUSEPORT option to the socket<br>
2017-09-08 09:29:43: pid 4478: LOG: set SO_REUSEPORT option to the socket<br>
2017-09-08 09:29:43: pid 4478: LOG: creating watchdog heartbeat receive socket.<br>
2017-09-08 09:29:43: pid 4478: DETAIL: set SO_REUSEPORT<br>
2017-09-08 09:29:43: pid 4480: LOG: creating socket for sending heartbeat<br>
2017-09-08 09:29:43: pid 4480: DETAIL: set SO_REUSEPORT<br>
2017-09-08 09:29:45: pid 4443: LOG: new watchdog node connection is received from "<a href="http://127.0.0.1:21121" rel="noreferrer" target="_blank">127.0.0.1:21121</a>"<br>
2017-09-08 09:29:45: pid 4443: LOG: new node joined the cluster hostname:"localhost" port:21004 pgpool_port:11000<br>
2017-09-08 09:29:52: pid 4445: LOG: watchdog: lifecheck started<br>
2017-09-08 09:31:42: pid 4521: LOG: failed to connect to PostgreSQL server by unix domain socket<br>
2017-09-08 09:31:42: pid 4521: DETAIL: connect to "/tmp/.s.PGSQL.11002" failed with error "No such file or directory"<br>
2017-09-08 09:31:42: pid 4521: ERROR: failed to make persistent db connection<br>
2017-09-08 09:31:42: pid 4521: DETAIL: connection to host:"/tmp:11002" failed<br>
2017-09-08 09:31:42: pid 4521: LOG: health check failed on node 0 (timeout:0)<br>
2017-09-08 09:31:42: pid 4521: LOG: received degenerate backend request for node_id: 0 from pid [4521]<br>
2017-09-08 09:31:42: pid 4441: LOG: Pgpool-II parent process received sync backend signal from watchdog<br>
2017-09-08 09:31:42: pid 4443: LOG: new IPC connection received<br>
2017-09-08 09:31:42: pid 4443: LOG: new IPC connection received<br>
2017-09-08 09:31:42: pid 4443: LOG: failover request from local pgpool-II node received on IPC interface is forwarded to master watchdog node "localhost:11000 Linux tishii-CF-SX3HE4BP"<br>
2017-09-08 09:31:42: pid 4443: DETAIL: waiting for the reply...<br>
2017-09-08 09:31:42: pid 4441: LOG: master watchdog has performed failover<br>
2017-09-08 09:31:42: pid 4441: DETAIL: syncing the backend states from the MASTER watchdog node<br>
2017-09-08 09:31:42: pid 4443: LOG: new IPC connection received<br>
2017-09-08 09:31:42: pid 4443: LOG: received the get data request from local pgpool-II on IPC interface<br>
2017-09-08 09:31:42: pid 4443: LOG: get data request from local pgpool-II node received on IPC interface is forwarded to master watchdog node "localhost:11000 Linux tishii-CF-SX3HE4BP"<br>
2017-09-08 09:31:42: pid 4443: DETAIL: waiting for the reply...<br>
2017-09-08 09:31:42: pid 4521: LOG: degenerate backend request for 1 node(s) from pid [4521], will be handled by watchdog<br>
2017-09-08 09:31:42: pid 4441: LOG: master watchdog node "localhost:11000 Linux tishii-CF-SX3HE4BP" returned status for 1 backend nodes<br>
2017-09-08 09:31:42: pid 4441: LOG: backend:0 is set to down status<br>
2017-09-08 09:31:42: pid 4441: DETAIL: backend:0 is DOWN on cluster master "localhost:11000 Linux tishii-CF-SX3HE4BP"<br>
2017-09-08 09:31:42: pid 4441: LOG: node status was chenged after the sync from "localhost:11000 Linux tishii-CF-SX3HE4BP"<br>
2017-09-08 09:31:42: pid 4441: DETAIL: all children needs to be restarted as we are not in streaming replication mode<br>
2017-09-08 09:31:42: pid 4441: LOG: Pgpool-II parent process received sync backend signal from watchdog<br>
2017-09-08 09:31:42: pid 4443: LOG: new IPC connection received<br>
2017-09-08 09:31:42: pid 4441: LOG: master watchdog has performed failover<br>
2017-09-08 09:31:42: pid 4441: DETAIL: syncing the backend states from the MASTER watchdog node<br>
2017-09-08 09:31:42: pid 4443: LOG: new IPC connection received<br>
2017-09-08 09:31:42: pid 4443: LOG: received the get data request from local pgpool-II on IPC interface<br>
2017-09-08 09:31:42: pid 4443: LOG: get data request from local pgpool-II node received on IPC interface is forwarded to master watchdog node "localhost:11000 Linux tishii-CF-SX3HE4BP"<br>
2017-09-08 09:31:42: pid 4443: DETAIL: waiting for the reply...<br>
2017-09-08 09:31:42: pid 4441: LOG: master watchdog node "localhost:11000 Linux tishii-CF-SX3HE4BP" returned status for 1 backend nodes<br>
2017-09-08 09:31:42: pid 4441: LOG: backend nodes status remains same after the sync from "localhost:11000 Linux tishii-CF-SX3HE4BP"<br>
2017-09-08 09:31:42: pid 4441: LOG: child process with pid: 4451 exits with status 0<br>
2017-09-08 09:31:42: pid 4441: LOG: child process with pid: 4451 exited with success and will not be restarted<br>
2017-09-08 09:31:42: pid 4441: LOG: child process with pid: 4453 exits with status 0<br>
2017-09-08 09:31:42: pid 4441: LOG: child process with pid: 4453 exited with success and will not be restarted<br>
2017-09-08 09:31:42: pid 4441: LOG: child process with pid: 4455 exits with status 0<br>
2017-09-08 09:31:42: pid 4441: LOG: child process with pid: 4455 exited with success and will not be restarted<br>
2017-09-08 09:31:42: pid 4441: LOG: child process with pid: 4457 exits with status 0<br>
2017-09-08 09:31:42: pid 4441: LOG: child process with pid: 4457 exited with success and will not be restarted<br>
2017-09-08 09:31:42: pid 4441: LOG: child process with pid: 4458 exits with status 0<br>
2017-09-08 09:31:42: pid 4441: LOG: child process with pid: 4458 exited with success and will not be restarted<br>
2017-09-08 09:31:42: pid 4441: LOG: child process with pid: 4461 exits with status 0<br>
2017-09-08 09:31:42: pid 4441: LOG: child process with pid: 4461 exited with success and will not be restarted<br>
2017-09-08 09:31:42: pid 4441: LOG: child process with pid: 4463 exits with status 0<br>
2017-09-08 09:31:42: pid 4441: LOG: child process with pid: 4463 exited with success and will not be restarted<br>
2017-09-08 09:31:42: pid 4441: LOG: child process with pid: 4464 exits with status 0<br>
2017-09-08 09:31:42: pid 4441: LOG: child process with pid: 4464 exited with success and will not be restarted<br>
2017-09-08 09:31:42: pid 4441: LOG: child process with pid: 4466 exits with status 0<br>
2017-09-08 09:31:42: pid 4441: LOG: child process with pid: 4466 exited with success and will not be restarted<br>
2017-09-08 09:31:42: pid 4441: LOG: child process with pid: 4468 exits with status 0<br>
2017-09-08 09:31:42: pid 4441: LOG: child process with pid: 4468 exited with success and will not be restarted<br>
2017-09-08 09:31:42: pid 4441: LOG: child process with pid: 4470 exits with status 0<br>
2017-09-08 09:31:42: pid 4441: LOG: child process with pid: 4470 exited with success and will not be restarted<br>
2017-09-08 09:31:42: pid 4441: LOG: child process with pid: 4472 exits with status 0<br>
2017-09-08 09:31:42: pid 4441: LOG: child process with pid: 4472 exited with success and will not be restarted<br>
2017-09-08 09:31:42: pid 4441: LOG: child process with pid: 4474 exits with status 0<br>
2017-09-08 09:31:42: pid 4441: LOG: child process with pid: 4474 exited with success and will not be restarted<br>
2017-09-08 09:31:42: pid 4441: LOG: child process with pid: 4499 exits with status 0<br>
2017-09-08 09:31:42: pid 4441: LOG: child process with pid: 4499 exited with success and will not be restarted<br>
2017-09-08 09:31:42: pid 4441: LOG: child process with pid: 4500 exits with status 0<br>
2017-09-08 09:31:42: pid 4441: LOG: child process with pid: 4500 exited with success and will not be restarted<br>
2017-09-08 09:31:42: pid 4441: LOG: child process with pid: 4501 exits with status 0<br>
2017-09-08 09:31:42: pid 4441: LOG: child process with pid: 4501 exited with success and will not be restarted<br>
2017-09-08 09:31:42: pid 4441: LOG: child process with pid: 4502 exits with status 0<br>
2017-09-08 09:31:42: pid 4441: LOG: child process with pid: 4502 exited with success and will not be restarted<br>
2017-09-08 09:31:42: pid 4441: LOG: child process with pid: 4503 exits with status 0<br>
2017-09-08 09:31:42: pid 4441: LOG: child process with pid: 4503 exited with success and will not be restarted<br>
2017-09-08 09:31:42: pid 4441: LOG: child process with pid: 4504 exits with status 0<br>
2017-09-08 09:31:42: pid 4441: LOG: child process with pid: 4504 exited with success and will not be restarted<br>
2017-09-08 09:31:42: pid 4441: LOG: child process with pid: 4505 exits with status 0<br>
2017-09-08 09:31:42: pid 4441: LOG: child process with pid: 4505 exited with success and will not be restarted<br>
2017-09-08 09:31:42: pid 4441: LOG: child process with pid: 4506 exits with status 0<br>
2017-09-08 09:31:42: pid 4441: LOG: child process with pid: 4506 exited with success and will not be restarted<br>
2017-09-08 09:31:42: pid 4441: LOG: child process with pid: 4507 exits with status 0<br>
2017-09-08 09:31:42: pid 4441: LOG: child process with pid: 4507 exited with success and will not be restarted<br>
2017-09-08 09:31:42: pid 4441: LOG: child process with pid: 4508 exits with status 0<br>
2017-09-08 09:31:42: pid 4441: LOG: child process with pid: 4508 exited with success and will not be restarted<br>
2017-09-08 09:31:42: pid 4441: LOG: child process with pid: 4509 exits with status 0<br>
2017-09-08 09:31:42: pid 4441: LOG: child process with pid: 4509 exited with success and will not be restarted<br>
2017-09-08 09:31:42: pid 4441: LOG: child process with pid: 4510 exits with status 0<br>
2017-09-08 09:31:42: pid 4441: LOG: child process with pid: 4510 exited with success and will not be restarted<br>
2017-09-08 09:31:42: pid 4441: LOG: child process with pid: 4512 exits with status 0<br>
2017-09-08 09:31:42: pid 4441: LOG: child process with pid: 4512 exited with success and will not be restarted<br>
2017-09-08 09:31:42: pid 4441: LOG: child process with pid: 4514 exits with status 0<br>
2017-09-08 09:31:42: pid 4441: LOG: child process with pid: 4514 exited with success and will not be restarted<br>
2017-09-08 09:31:42: pid 4441: LOG: child process with pid: 4516 exits with status 0<br>
2017-09-08 09:31:42: pid 4441: LOG: child process with pid: 4516 exited with success and will not be restarted<br>
2017-09-08 09:31:42: pid 4441: LOG: child process with pid: 4517 exits with status 0<br>
2017-09-08 09:31:42: pid 4441: LOG: child process with pid: 4517 exited with success and will not be restarted<br>
2017-09-08 09:31:42: pid 4441: LOG: child process with pid: 4518 exits with status 0<br>
2017-09-08 09:31:42: pid 4441: LOG: child process with pid: 4518 exited with success and will not be restarted<br>
2017-09-08 09:31:42: pid 4441: LOG: child process with pid: 4513 exits with status 0<br>
2017-09-08 09:31:42: pid 4441: LOG: child process with pid: 4513 exited with success and will not be restarted<br>
2017-09-08 09:31:42: pid 4441: LOG: child process with pid: 4511 exits with status 0<br>
2017-09-08 09:31:42: pid 4441: LOG: child process with pid: 4511 exited with success and will not be restarted<br>
2017-09-08 09:31:43: pid 4441: LOG: Pgpool-II parent process received sync backend signal from watchdog<br>
2017-09-08 09:31:43: pid 4443: LOG: new IPC connection received<br>
2017-09-08 09:31:43: pid 4441: LOG: master watchdog has performed failover<br>
2017-09-08 09:31:43: pid 4441: DETAIL: syncing the backend states from the MASTER watchdog node<br>
2017-09-08 09:31:43: pid 4443: LOG: new IPC connection received<br>
2017-09-08 09:31:43: pid 4443: LOG: received the get data request from local pgpool-II on IPC interface<br>
2017-09-08 09:31:43: pid 4443: LOG: get data request from local pgpool-II node received on IPC interface is forwarded to master watchdog node "localhost:11000 Linux tishii-CF-SX3HE4BP"<br>
2017-09-08 09:31:43: pid 4443: DETAIL: waiting for the reply...<br>
2017-09-08 09:31:43: pid 4441: LOG: master watchdog node "localhost:11000 Linux tishii-CF-SX3HE4BP" returned status for 1 backend nodes<br>
2017-09-08 09:31:43: pid 4441: LOG: backend nodes status remains same after the sync from "localhost:11000 Linux tishii-CF-SX3HE4BP"<br>
2017-09-08 09:31:46: pid 4441: LOG: Pgpool-II parent process received sync backend signal from watchdog<br>
2017-09-08 09:31:46: pid 4443: LOG: new IPC connection received<br>
2017-09-08 09:31:46: pid 4441: LOG: master watchdog has performed failover<br>
2017-09-08 09:31:46: pid 4441: DETAIL: syncing the backend states from the MASTER watchdog node<br>
2017-09-08 09:31:46: pid 4443: LOG: new IPC connection received<br>
2017-09-08 09:31:46: pid 4443: LOG: received the get data request from local pgpool-II on IPC interface<br>
2017-09-08 09:31:46: pid 4443: LOG: get data request from local pgpool-II node received on IPC interface is forwarded to master watchdog node "localhost:11000 Linux tishii-CF-SX3HE4BP"<br>
2017-09-08 09:31:46: pid 4443: DETAIL: waiting for the reply...<br>
2017-09-08 09:31:46: pid 4441: LOG: master watchdog node "localhost:11000 Linux tishii-CF-SX3HE4BP" returned status for 1 backend nodes<br>
2017-09-08 09:31:46: pid 4441: LOG: backend nodes status remains same after the sync from "localhost:11000 Linux tishii-CF-SX3HE4BP"<br>
2017-09-08 09:31:56: pid 4622: FATAL: pgpool is not accepting any new connections<br>
2017-09-08 09:31:56: pid 4622: DETAIL: all backend nodes are down, pgpool requires at least one valid node<br>
2017-09-08 09:31:56: pid 4622: HINT: repair the backend nodes and restart pgpool<br>
2017-09-08 09:31:56: pid 4441: LOG: child process with pid: 4622 exits with status 256<br>
2017-09-08 09:31:56: pid 4441: LOG: fork a new child process with pid: 4748<br>
2017-09-08 09:32:10: pid 4637: FATAL: pgpool is not accepting any new connections<br>
2017-09-08 09:32:10: pid 4637: DETAIL: all backend nodes are down, pgpool requires at least one valid node<br>
2017-09-08 09:32:10: pid 4637: HINT: repair the backend nodes and restart pgpool<br>
2017-09-08 09:32:10: pid 4441: LOG: child process with pid: 4637 exits with status 256<br>
2017-09-08 09:32:10: pid 4441: LOG: fork a new child process with pid: 4753<br>
2017-09-08 09:33:10: pid 4443: LOG: remote node "localhost:11000 Linux tishii-CF-SX3HE4BP" is shutting down<br>
2017-09-08 09:33:10: pid 4443: LOG: watchdog cluster has lost the coordinator node<br>
2017-09-08 09:33:10: pid 4443: LOG: unassigning the remote node "localhost:11000 Linux tishii-CF-SX3HE4BP" from watchdog cluster master<br>
2017-09-08 09:33:10: pid 4443: LOG: We have lost the cluster master node "localhost:11000 Linux tishii-CF-SX3HE4BP"<br>
2017-09-08 09:33:10: pid 4443: LOG: watchdog node state changed from [STANDBY] to [JOINING]<br>
2017-09-08 09:33:14: pid 4443: LOG: watchdog node state changed from [JOINING] to [INITIALIZING]<br>
2017-09-08 09:33:15: pid 4443: LOG: I am the only alive node in the watchdog cluster<br>
2017-09-08 09:33:15: pid 4443: HINT: skiping stand for coordinator state<br>
2017-09-08 09:33:15: pid 4443: LOG: watchdog node state changed from [INITIALIZING] to [MASTER]<br>
2017-09-08 09:33:15: pid 4443: LOG: I am announcing my self as master/coordinator watchdog node<br>
2017-09-08 09:33:19: pid 4443: LOG: I am the cluster leader node<br>
2017-09-08 09:33:19: pid 4443: DETAIL: our declare coordinator message is accepted by all nodes<br>
2017-09-08 09:33:19: pid 4443: LOG: setting the local node "localhost:11100 Linux tishii-CF-SX3HE4BP" as watchdog cluster master<br>
2017-09-08 09:33:19: pid 4443: LOG: I am the cluster leader node. Starting escalation process<br>
2017-09-08 09:33:19: pid 4443: LOG: escalation process started with PID:4769<br>
2017-09-08 09:33:19: pid 4443: LOG: new IPC connection received<br>
2017-09-08 09:33:19: pid 4769: LOG: watchdog: escalation started<br>
2017-09-08 09:33:19: pid 4443: LOG: watchdog escalation process with pid: 4769 exit with SUCCESS.<br>
2017-09-08 09:33:40: pid 4445: LOG: informing the node status change to watchdog<br>
2017-09-08 09:33:40: pid 4445: DETAIL: node id :1 status = "NODE DEAD" message:"No heartbeat signal from node"<br>
2017-09-08 09:33:40: pid 4443: LOG: new IPC connection received<br>
2017-09-08 09:33:40: pid 4443: LOG: received node status change ipc message<br>
2017-09-08 09:33:40: pid 4443: DETAIL: No heartbeat signal from node<br>
2017-09-08 09:33:40: pid 4443: LOG: remote node "localhost:11000 Linux tishii-CF-SX3HE4BP" is shutting down<br>
2017-09-08 09:36:57: pid 4519: LOG: forked new pcp worker, pid=4813 socket=6<br>
2017-09-08 09:36:57: pid 4443: LOG: new IPC connection received<br>
2017-09-08 09:36:57: pid 4519: LOG: PCP process with pid: 4813 exit with SUCCESS.<br>
2017-09-08 09:36:57: pid 4519: LOG: PCP process with pid: 4813 exits with status 0<br>
<br></blockquote></div><br></div></div></div>