<div dir="ltr">The watchdog information has nothing to do with the PostgreSQL backends. The master shown by the watchdog info against the Pgpool-II node is the Pgpool-II master/coordinator node in the Pgpool-II watchdog cluster. While the master/primary node in discussion above is the master PostgreSQL backend node in streaming replication.<div><br><div><br></div><div>Thanks</div><div>Best regards</div><div>Muhammad Usama</div><div><br><div><br></div><div><br><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Jan 12, 2017 at 2:49 PM, Bhattacharyya, Subhro <span dir="ltr"><<a href="mailto:s.bhattacharyya@sap.com" target="_blank">s.bhattacharyya@sap.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-US" link="blue" vlink="purple">
<div class="m_5026338835933139239WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">Hi,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">As pointed out, this is the log of node 1
<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">2017-01-11 11:17:52: pid 28787: LOG: watchdog node state changed from [STANDBY] to [JOINING]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">2017-01-11 11:17:52: pid 28787: LOG: watchdog node state changed from [JOINING] to [INITIALIZING]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">2017-01-11 11:17:52: pid 28787: LOG: read from socket failed with error :"Connection reset by peer"<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">2017-01-11 11:17:52: pid 28787: LOG: read from socket failed, remote end closed the connection<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">2017-01-11 11:17:53: pid 28787: LOG: watchdog node state changed from [INITIALIZING] to [STANDING FOR MASTER]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">2017-01-11 11:17:53: pid 28787: LOG: watchdog node state changed from [STANDING FOR MASTER] to [MASTER]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">2017-01-11 11:17:53: pid 28787: LOG: I am announcing my self as master/coordinator watchdog node<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">2017-01-11 11:17:53: pid 28787: LOG: I am the cluster leader node<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">2017-01-11 11:17:53: pid 28787: DETAIL: our declare coordinator message is accepted by all nodes<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">2017-01-11 11:17:53: pid 28787: LOG: I am the cluster leader node. Starting escalation process<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">2017-01-11 11:17:53: pid 28787: LOG: escalation process started with PID:30979<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">2017-01-11 11:17:53: pid 30979: LOG: watchdog: escalation started<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">2017-01-11 11:17:53: pid 28787: LOG: watchdog escalation process with pid: 30979 exit with SUCCESS.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">2017-01-11 11:17:56: pid 30978: ERROR: Failed to check replication time lag<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">2017-01-11 11:17:56: pid 30978: DETAIL: Query to node (1) returned no data<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">2017-01-11 11:17:56: pid 30978: CONTEXT: while checking replication time lag<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">2017-01-11 11:17:57: pid 30814: LOG: statement: show pool_version;<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">This has indeed become master.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">The other two nodes have cone up as standby.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">However the watchdog information shows that the last node is master, not node 1.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">How do we stop pgpool-II from thinking that node 1 is standby and stop the nodes from querying node 1<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">Thanks<u></u><u></u></span></p>
<p class="MsoNormal"><a name="m_5026338835933139239__MailEndCompose"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></a></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Muhammad Usama [mailto:<a href="mailto:m.usama@gmail.com" target="_blank">m.usama@gmail.com</a>]
<br>
<b>Sent:</b> Thursday, January 12, 2017 2:50 PM<br>
<b>To:</b> Bhattacharyya, Subhro <<a href="mailto:s.bhattacharyya@sap.com" target="_blank">s.bhattacharyya@sap.com</a>><br>
<b>Cc:</b> <a href="mailto:pgpool-general@pgpool.net" target="_blank">pgpool-general@pgpool.net</a><br>
<b>Subject:</b> Re: [pgpool-general: 5228] Failed to check replication time lag - infinitely occurring error<u></u><u></u></span></p><div><div class="h5">
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">Pgpool-II sends (<span style="font-family:"Courier New"">SELECT pg_last_xlog_replay_location()</span><wbr>) query to standby backend nodes for calculating the replication time lag.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">And the error "<span style="font-family:"Courier New"">Query to node (1) returned no data</span>" indicates that the backend node 1 is not returning any results for the query.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt">Usually this happens when Pgpool-II thinks that the node is a standby while actually it is not. So you should check the backend node 1 if it is in the standby mode and it is not <span class="m_5026338835933139239gmail-gingersoftwaremark">accidentally</span>
promoted <span class="m_5026338835933139239gmail-il">to</span> primary.</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt">Thanks</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt">Regards</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt">Muhammad Usama</span><u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">On Thu, Jan 12, 2017 at 12:20 PM, Bhattacharyya, Subhro <<a href="mailto:s.bhattacharyya@sap.com" target="_blank">s.bhattacharyya@sap.com</a>> wrote:<u></u><u></u></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">Hi,<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">We have a 2 node postgresql cluster along with 3 pgpool-II nodes.<u></u><u></u></p>
<p class="MsoNormal">The version of pgpool-II that we are using is 3.5.2<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">We keep getting the following logs infinitely in the logs of all the pgpool-II nodes<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal"><b>2017-01-12 07:09:55: pid 2534: ERROR: Failed to check replication time lag</b><u></u><u></u></p>
<p class="MsoNormal"><b>2017-01-12 07:09:55: pid 2534: DETAIL: Query to node (1) returned no data</b><u></u><u></u></p>
<p class="MsoNormal"><b>2017-01-12 07:09:55: pid 2534: CONTEXT: while checking replication time lag</b><u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal" style="text-autospace:none">
<span style="font-size:9.5pt;font-family:"Segoe UI",sans-serif">Following is the watchdog cluster/node information:</span><u></u><u></u></p>
<p class="MsoNormal" style="text-autospace:none">
<span style="font-size:9.5pt;font-family:"Segoe UI",sans-serif"> </span><u></u><u></u></p>
<p class="MsoNormal" style="text-autospace:none">
<span style="font-size:9.5pt;font-family:"Segoe UI",sans-serif">Watchdog Cluster Information</span><u></u><u></u></p>
<p class="MsoNormal" style="text-autospace:none">
<span style="font-size:9.5pt;font-family:"Segoe UI",sans-serif">Total Nodes : 3</span><u></u><u></u></p>
<p class="MsoNormal" style="text-autospace:none">
<span style="font-size:9.5pt;font-family:"Segoe UI",sans-serif">Remote Nodes : 2</span><u></u><u></u></p>
<p class="MsoNormal" style="text-autospace:none">
<span style="font-size:9.5pt;font-family:"Segoe UI",sans-serif">Quorum state : QUORUM EXIST</span><u></u><u></u></p>
<p class="MsoNormal" style="text-autospace:none">
<span style="font-size:9.5pt;font-family:"Segoe UI",sans-serif">Alive Remote Nodes : 2</span><u></u><u></u></p>
<p class="MsoNormal" style="text-autospace:none">
<span style="font-size:9.5pt;font-family:"Segoe UI",sans-serif">VIP up on local node : NO</span><u></u><u></u></p>
<p class="MsoNormal" style="text-autospace:none">
<span style="font-size:9.5pt;font-family:"Segoe UI",sans-serif">Master Node Name : Linux_270f905f-37ef-46a2-9f7b-<wbr>9b92b96a6ea9_9999</span><u></u><u></u></p>
<p class="MsoNormal" style="text-autospace:none">
<span style="font-size:9.5pt;font-family:"Segoe UI",sans-serif">Master Host Name : 10.11.61.50</span><u></u><u></u></p>
<p class="MsoNormal" style="text-autospace:none">
<span style="font-size:9.5pt;font-family:"Segoe UI",sans-serif"> </span><u></u><u></u></p>
<p class="MsoNormal" style="text-autospace:none">
<span style="font-size:9.5pt;font-family:"Segoe UI",sans-serif">Watchdog Node Information</span><u></u><u></u></p>
<p class="MsoNormal" style="text-autospace:none">
<span style="font-size:9.5pt;font-family:"Segoe UI",sans-serif">Node Name : Linux_26acb17c-dadc-49ca-bb14-<wbr>44820ab6a78a_9999</span><u></u><u></u></p>
<p class="MsoNormal" style="text-autospace:none">
<span style="font-size:9.5pt;font-family:"Segoe UI",sans-serif">Host Name : 10.11.61.52</span><u></u><u></u></p>
<p class="MsoNormal" style="text-autospace:none">
<span style="font-size:9.5pt;font-family:"Segoe UI",sans-serif">Delegate IP : Not_Set</span><u></u><u></u></p>
<p class="MsoNormal" style="text-autospace:none">
<span style="font-size:9.5pt;font-family:"Segoe UI",sans-serif">Pgpool port : 9999</span><u></u><u></u></p>
<p class="MsoNormal" style="text-autospace:none">
<span style="font-size:9.5pt;font-family:"Segoe UI",sans-serif">Watchdog port : 9000</span><u></u><u></u></p>
<p class="MsoNormal" style="text-autospace:none">
<span style="font-size:9.5pt;font-family:"Segoe UI",sans-serif">Node priority : 1</span><u></u><u></u></p>
<p class="MsoNormal" style="text-autospace:none">
<span style="font-size:9.5pt;font-family:"Segoe UI",sans-serif">Status : 7</span><u></u><u></u></p>
<p class="MsoNormal" style="text-autospace:none">
<span style="font-size:9.5pt;font-family:"Segoe UI",sans-serif">Status Name : STANDBY</span><u></u><u></u></p>
<p class="MsoNormal" style="text-autospace:none">
<span style="font-size:9.5pt;font-family:"Segoe UI",sans-serif"> </span><u></u><u></u></p>
<p class="MsoNormal" style="text-autospace:none">
<span style="font-size:9.5pt;font-family:"Segoe UI",sans-serif">Node Name : Linux_2ae9f3fd-31fc-4921-aeb7-<wbr>8e320cf347c4_9999</span><u></u><u></u></p>
<p class="MsoNormal" style="text-autospace:none">
<span style="font-size:9.5pt;font-family:"Segoe UI",sans-serif">Host Name : 10.11.61.51</span><u></u><u></u></p>
<p class="MsoNormal" style="text-autospace:none">
<span style="font-size:9.5pt;font-family:"Segoe UI",sans-serif">Delegate IP : Not_Set</span><u></u><u></u></p>
<p class="MsoNormal" style="text-autospace:none">
<span style="font-size:9.5pt;font-family:"Segoe UI",sans-serif">Pgpool port : 9999</span><u></u><u></u></p>
<p class="MsoNormal" style="text-autospace:none">
<span style="font-size:9.5pt;font-family:"Segoe UI",sans-serif">Watchdog port : 9000</span><u></u><u></u></p>
<p class="MsoNormal" style="text-autospace:none">
<span style="font-size:9.5pt;font-family:"Segoe UI",sans-serif">Node priority : 1</span><u></u><u></u></p>
<p class="MsoNormal" style="text-autospace:none">
<span style="font-size:9.5pt;font-family:"Segoe UI",sans-serif">Status : 7</span><u></u><u></u></p>
<p class="MsoNormal" style="text-autospace:none">
<span style="font-size:9.5pt;font-family:"Segoe UI",sans-serif">Status Name : STANDBY</span><u></u><u></u></p>
<p class="MsoNormal" style="text-autospace:none">
<span style="font-size:9.5pt;font-family:"Segoe UI",sans-serif"> </span><u></u><u></u></p>
<p class="MsoNormal" style="text-autospace:none">
<span style="font-size:9.5pt;font-family:"Segoe UI",sans-serif">Node Name : Linux_270f905f-37ef-46a2-9f7b-<wbr>9b92b96a6ea9_9999</span><u></u><u></u></p>
<p class="MsoNormal" style="text-autospace:none">
<span style="font-size:9.5pt;font-family:"Segoe UI",sans-serif">Host Name : 10.11.61.50</span><u></u><u></u></p>
<p class="MsoNormal" style="text-autospace:none">
<span style="font-size:9.5pt;font-family:"Segoe UI",sans-serif">Delegate IP : Not_Set</span><u></u><u></u></p>
<p class="MsoNormal" style="text-autospace:none">
<span style="font-size:9.5pt;font-family:"Segoe UI",sans-serif">Pgpool port : 9999</span><u></u><u></u></p>
<p class="MsoNormal" style="text-autospace:none">
<span style="font-size:9.5pt;font-family:"Segoe UI",sans-serif">Watchdog port : 9000</span><u></u><u></u></p>
<p class="MsoNormal" style="text-autospace:none">
<span style="font-size:9.5pt;font-family:"Segoe UI",sans-serif">Node priority : 1</span><u></u><u></u></p>
<p class="MsoNormal" style="text-autospace:none">
<span style="font-size:9.5pt;font-family:"Segoe UI",sans-serif">Status : 4</span><u></u><u></u></p>
<p class="MsoNormal" style="text-autospace:none">
<span style="font-size:9.5pt;font-family:"Segoe UI",sans-serif">Status Name : MASTER </span><u></u><u></u></p>
<p class="MsoNormal" style="text-autospace:none">
<span style="font-size:9.5pt;font-family:"Segoe UI",sans-serif"> </span><u></u><u></u></p>
<p class="MsoNormal" style="text-autospace:none">
<span style="font-size:9.5pt;font-family:"Segoe UI",sans-serif">Please let us know how to rectify this issue.</span><u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
______________________________<wbr>_________________<br>
pgpool-general mailing list<br>
<a href="mailto:pgpool-general@pgpool.net" target="_blank">pgpool-general@pgpool.net</a><br>
<a href="http://www.pgpool.net/mailman/listinfo/pgpool-general" target="_blank">http://www.pgpool.net/mailman/<wbr>listinfo/pgpool-general</a><u></u><u></u></p>
</blockquote>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div></div></div>
</div>
</blockquote></div><br></div></div></div></div></div>