<div dir="ltr">Hi <span style="font-size:12.8px">Guptesh</span><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">I am looking into the issue. meanwhile I had recently committed a fix related to watchdog heartbeat problem, So could you also try building from the latest source code to verify if you still face the issue. And if the issue still presists please share the complete pgpool-II log file with debug enabled. That would help to solve the problem more quickly.</span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">Kind regards</span></div><div><span style="font-size:12.8px">Muhammad Usama</span></div><div><span style="font-size:12.8px"><br></span></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Aug 2, 2016 at 9:25 PM, Guptesh Kumar <span dir="ltr"><<a href="mailto:guptesh.cg4@gmail.com" target="_blank">guptesh.cg4@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div><div><div><div>Hi Usama,<br><br></div>Please let me know if I can provide any more input which can help you debug/reproduce this issue.<br></div>We are considering pgpool(with watchdog) as a high availability solution for DB. And this issue is kind of show stopper because gracefully handling server crash/network problems is key in a Highly Available solution.<br><br></div>I think this issue is same as Bug <a href="http://www.pgpool.net/mantisbt/view.php?id=227" target="_blank">http://www.pgpool.net/mantisbt/view.php?id=227</a> .<br><br></div>Looking forward for a solution of this issue.<br><br></div>Thanks & Regards,<br></div>Guptesh<br></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Sat, Jul 30, 2016 at 12:17 AM, Guptesh Kumar <span dir="ltr"><<a href="mailto:guptesh.cg4@gmail.com" target="_blank">guptesh.cg4@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div><div><div><div><div><div>Hi Usama,<br><br><br></div>I am still facing issues here-<br><br></div>I shutdown primary(Node0) and standby(Node1) became primary successfully.<br></div>When Node0 came up, i recovered postgres0(postgres on Node0). Now Node0 is standby and Node1 is primary.<br></div>Again
if I shutdown Node1(now primary), failover is not happening on Node0.
Below are the logs after health_check retry count is expired<br><br><br><br>Jul 29 23:50:02 ]: 2016-07-29 23:50:02: pid 4211: DETAIL: timed out. retrying...<br>Jul 29 23:50:02 ]: 2016-07-29 23:50:02: pid 4185: LOG: trying connecting to PostgreSQL server on "<a href="http://192.168.160.5:5432" target="_blank">192.168.160.5:5432</a>" by INET socket<br>Jul 29 23:50:02 ]: 2016-07-29 23:50:02: pid 4185: DETAIL: timed out. retrying...<br>Jul 29 23:50:04 ]: 2016-07-29 23:50:04: pid 4312: LOG: trying connecting to PostgreSQL server on "<a href="http://192.168.160.5:5432" target="_blank">192.168.160.5:5432</a>" by INET socket<br>Jul 29 23:50:04 ]: 2016-07-29 23:50:04: pid 4312: DETAIL: timed out. retrying...<br>Jul 29 23:50:04 ]: 2016-07-29 23:50:04: pid 4279: LOG: trying connecting to PostgreSQL server on "<a href="http://192.168.160.5:5432" target="_blank">192.168.160.5:5432</a>" by INET socket<br>Jul 29 23:50:04 ]: 2016-07-29 23:50:04: pid 4279: DETAIL: timed out. retrying...<br>Jul 29 23:50:04 ]: 2016-07-29 23:50:04: pid 4257: LOG: trying connecting to PostgreSQL server on "<a href="http://192.168.160.5:5432" target="_blank">192.168.160.5:5432</a>" by INET socket<br>Jul 29 23:50:04 ]: 2016-07-29 23:50:04: pid 4257: DETAIL: timed out. retrying...<br>Jul 29 23:50:04 ]: 2016-07-29 23:50:04: pid 4230: LOG: trying connecting to PostgreSQL server on "<a href="http://192.168.160.5:5432" target="_blank">192.168.160.5:5432</a>" by INET socket<br>Jul 29 23:50:04 ]: 2016-07-29 23:50:04: pid 4230: DETAIL: timed out. retrying...<br>Jul 29 23:50:04 ]: 2016-07-29 23:50:04: pid 4210: LOG: trying connecting to PostgreSQL server on "<a href="http://192.168.160.5:5432" target="_blank">192.168.160.5:5432</a>" by INET socket<br>Jul 29 23:50:04 ]: 2016-07-29 23:50:04: pid 4210: DETAIL: timed out. retrying...<br>Jul 29 23:50:04 ]: 2016-07-29 23:50:04: pid 4278: LOG: trying connecting to PostgreSQL server on "<a href="http://192.168.160.5:5432" target="_blank">192.168.160.5:5432</a>" by INET socket<br>Jul 29 23:50:04 ]: 2016-07-29 23:50:04: pid 4278: DETAIL: timed out. retrying...<br>Jul 29 23:50:06 ]: 2016-07-29 23:50:06: pid 4304: LOG: trying connecting to PostgreSQL server on "<a href="http://192.168.160.5:5432" target="_blank">192.168.160.5:5432</a>" by INET socket<br>Jul 29 23:50:06 ]: 2016-07-29 23:50:06: pid 4304: DETAIL: timed out. retrying...<br>Jul 29 23:50:06 ]: 2016-07-29 23:50:06: pid 4227: LOG: trying connecting to PostgreSQL server on "<a href="http://192.168.160.5:5432" target="_blank">192.168.160.5:5432</a>" by INET socket<br>Jul 29 23:50:06 ]: 2016-07-29 23:50:06: pid 4227: DETAIL: timed out. retrying...<br>Jul 29 23:50:06 ]: 2016-07-29 23:50:06: pid 4276: LOG: trying connecting to PostgreSQL server on "<a href="http://192.168.160.5:5432" target="_blank">192.168.160.5:5432</a>" by INET socket<br>Jul 29 23:50:06 ]: 2016-07-29 23:50:06: pid 4276: DETAIL: timed out. retrying...<br>Jul 29 23:50:06 ]: 2016-07-29 23:50:06: pid 4261: LOG: trying connecting to PostgreSQL server on "<a href="http://192.168.160.5:5432" target="_blank">192.168.160.5:5432</a>" by INET socket<br>Jul 29 23:50:06 ]: 2016-07-29 23:50:06: pid 4261: DETAIL: timed out. retrying...<br>Jul 29 23:50:06 ]: 2016-07-29 23:50:06: pid 4297: LOG: trying connecting to PostgreSQL server on "<a href="http://192.168.160.5:5432" target="_blank">192.168.160.5:5432</a>" by INET socket<br>Jul 29 23:50:06 ]: 2016-07-29 23:50:06: pid 4297: DETAIL: timed out. retrying...<br>Jul 29 23:50:06 ]: 2016-07-29 23:50:06: pid 4204: LOG: trying connecting to PostgreSQL server on "<a href="http://192.168.160.5:5432" target="_blank">192.168.160.5:5432</a>" by INET socket<br>Jul 29 23:50:06 ]: 2016-07-29 23:50:06: pid 4204: DETAIL: timed out. retrying...<br>Jul 29 23:50:06 ]: 2016-07-29 23:50:06: pid 4295: LOG: trying connecting to PostgreSQL server on "<a href="http://192.168.160.5:5432" target="_blank">192.168.160.5:5432</a>" by INET socket<br>Jul 29 23:50:06 ]: 2016-07-29 23:50:06: pid 4295: DETAIL: timed out. retrying...<br>Jul 29 23:50:06 ]: 2016-07-29 23:50:06: pid 4291: LOG: trying connecting to PostgreSQL server on "<a href="http://192.168.160.5:5432" target="_blank">192.168.160.5:5432</a>" by INET socket<br>Jul 29 23:50:06 ]: 2016-07-29 23:50:06: pid 4291: DETAIL: timed out. retrying...<br>Jul 29 23:50:06 ]: 2016-07-29 23:50:06: pid 4228: LOG: trying connecting to PostgreSQL server on "<a href="http://192.168.160.5:5432" target="_blank">192.168.160.5:5432</a>" by INET socket<br>Jul 29 23:50:06 ]: 2016-07-29 23:50:06: pid 4228: DETAIL: timed out. retrying...<br>Jul 29 23:50:07 ]: 2016-07-29 23:50:07: pid 4211: LOG: trying connecting to PostgreSQL server on "<a href="http://192.168.160.5:5432" target="_blank">192.168.160.5:5432</a>" by INET socket<br>Jul 29 23:50:07 ]: 2016-07-29 23:50:07: pid 4211: DETAIL: timed out. retrying...<br>Jul 29 23:50:07 ]: 2016-07-29 23:50:07: pid 4185: LOG: trying connecting to PostgreSQL server on "<a href="http://192.168.160.5:5432" target="_blank">192.168.160.5:5432</a>" by INET socket<br>Jul 29 23:50:07 ]: 2016-07-29 23:50:07: pid 4185: DETAIL: timed out. retrying...<br>Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4228: LOG: failed to connect to PostgreSQL server on "<a href="http://192.168.160.5:5432" target="_blank">192.168.160.5:5432</a>", getsockopt() detected error "Connection timed out"<br>Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4291: LOG: failed to connect to PostgreSQL server on "<a href="http://192.168.160.5:5432" target="_blank">192.168.160.5:5432</a>", getsockopt() detected error "Connection timed out"<br>Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4261: LOG: failed to connect to PostgreSQL server on "<a href="http://192.168.160.5:5432" target="_blank">192.168.160.5:5432</a>", getsockopt() detected error "Connection timed out"<br>Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4297: LOG: failed to connect to PostgreSQL server on "<a href="http://192.168.160.5:5432" target="_blank">192.168.160.5:5432</a>", getsockopt() detected error "Connection timed out"<br>Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4295: LOG: failed to connect to PostgreSQL server on "<a href="http://192.168.160.5:5432" target="_blank">192.168.160.5:5432</a>", getsockopt() detected error "Connection timed out"<br>Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4227: LOG: failed to connect to PostgreSQL server on "<a href="http://192.168.160.5:5432" target="_blank">192.168.160.5:5432</a>", getsockopt() detected error "Connection timed out"<br>Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4276: LOG: failed to connect to PostgreSQL server on "<a href="http://192.168.160.5:5432" target="_blank">192.168.160.5:5432</a>", getsockopt() detected error "Connection timed out"<br>Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4228: LOG: received degenerate backend request for node_id: 1 from pid [4228]<br>Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4261: LOG: received degenerate backend request for node_id: 1 from pid [4261]<br>Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4297: LOG: received degenerate backend request for node_id: 1 from pid [4297]<br>Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4291: LOG: received degenerate backend request for node_id: 1 from pid [4291]<br>Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4295: LOG: received degenerate backend request for node_id: 1 from pid [4295]<br>Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4227: LOG: received degenerate backend request for node_id: 1 from pid [4227]<br>Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4276: LOG: received degenerate backend request for node_id: 1 from pid [4276]<br>Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4204: LOG: failed to connect to PostgreSQL server on "<a href="http://192.168.160.5:5432" target="_blank">192.168.160.5:5432</a>", getsockopt() detected error "Connection timed out"<br>Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4204: LOG: received degenerate backend request for node_id: 1 from pid [4204]<br>Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4304: LOG: failed to connect to PostgreSQL server on "<a href="http://192.168.160.5:5432" target="_blank">192.168.160.5:5432</a>", getsockopt() detected error "Connection timed out"<br>Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4304: LOG: received degenerate backend request for node_id: 1 from pid [4304]<br>Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4179: LOG: new IPC connection received<br>Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4179: LOG: new IPC connection received<br>Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4179: LOG: new IPC connection received<br>Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4179: LOG: new IPC connection received<br>Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4179: LOG: new IPC connection received<br>Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4179: LOG: new IPC connection received<br>Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4179: LOG: new IPC connection received<br>Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4179: LOG: new IPC connection received<br>Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4179: LOG: new IPC connection received<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4257: LOG: trying connecting to PostgreSQL server on "<a href="http://192.168.160.5:5432" target="_blank">192.168.160.5:5432</a>" by INET socket<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4257: DETAIL: timed out. retrying...<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4279: LOG: trying connecting to PostgreSQL server on "<a href="http://192.168.160.5:5432" target="_blank">192.168.160.5:5432</a>" by INET socket<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4279: DETAIL: timed out. retrying...<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4230: LOG: trying connecting to PostgreSQL server on "<a href="http://192.168.160.5:5432" target="_blank">192.168.160.5:5432</a>" by INET socket<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4230: DETAIL: timed out. retrying...<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4210: LOG: trying connecting to PostgreSQL server on "<a href="http://192.168.160.5:5432" target="_blank">192.168.160.5:5432</a>" by INET socket<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4210: DETAIL: timed out. retrying...<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4312: LOG: trying connecting to PostgreSQL server on "<a href="http://192.168.160.5:5432" target="_blank">192.168.160.5:5432</a>" by INET socket<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4312: DETAIL: timed out. retrying...<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4278: LOG: trying connecting to PostgreSQL server on "<a href="http://192.168.160.5:5432" target="_blank">192.168.160.5:5432</a>" by INET socket<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4278: DETAIL: timed out. retrying...<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4179: LOG: read from socket failed, remote end closed the connection<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4250: WARNING: degenerate backend set command failed<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4250: DETAIL: ipc command timeout<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4329: WARNING: degenerate backend set command failed<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4329: DETAIL: ipc command timeout<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4269: WARNING: degenerate backend set command failed<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4269: DETAIL: ipc command timeout<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4289: WARNING: degenerate backend set command failed<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4289: DETAIL: ipc command timeout<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4217: WARNING: degenerate backend set command failed<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4217: DETAIL: ipc command timeout<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4266: WARNING: degenerate backend set command failed<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4266: DETAIL: ipc command timeout<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4179: LOG: read from socket failed, remote end closed the connection<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4329: FATAL: failed to create a backend connection<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4329: DETAIL: executing failover on backend<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4217: FATAL: failed to create a backend connection<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4217: DETAIL: executing failover on backend<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4266: FATAL: failed to create a backend connection<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4266: DETAIL: executing failover on backend<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4269: FATAL: failed to create a backend connection<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4269: DETAIL: executing failover on backend<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4289: FATAL: failed to create a backend connection<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4289: DETAIL: executing failover on backend<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4179: LOG: read from socket failed, remote end closed the connection<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4179: LOG: read from socket failed, remote end closed the connection<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4250: FATAL: failed to create a backend connection<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4250: DETAIL: executing failover on backend<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4179: LOG: read from socket failed, remote end closed the connection<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4179: LOG: read from socket failed, remote end closed the connection<br>Jul 29 23:50:12 ]: 2016-07-29 23:50:12: pid 4185: LOG: trying connecting to PostgreSQL server on "<a href="http://192.168.160.5:5432" target="_blank">192.168.160.5:5432</a>" by INET socket<br>Jul 29 23:50:12 ]: 2016-07-29 23:50:12: pid 4185: DETAIL: timed out. retrying...<br>Jul 29 23:50:12 ]: 2016-07-29 23:50:12: pid 4211: LOG: trying connecting to PostgreSQL server on "<a href="http://192.168.160.5:5432" target="_blank">192.168.160.5:5432</a>" by INET socket<br>Jul 29 23:50:12 ]: 2016-07-29 23:50:12: pid 4211: DETAIL: timed out. retrying...<br>Jul 29 23:50:14 ]: 2016-07-29 23:50:14: pid 4230: LOG: trying connecting to PostgreSQL server on "<a href="http://192.168.160.5:5432" target="_blank">192.168.160.5:5432</a>" by INET socket<br>Jul 29 23:50:14 ]: 2016-07-29 23:50:14: pid 4230: DETAIL: timed out. retrying...<br>Jul 29 23:50:14 ]: 2016-07-29 23:50:14: pid 4312: LOG: trying connecting to PostgreSQL server on "<a href="http://192.168.160.5:5432" target="_blank">192.168.160.5:5432</a>" by INET socket<br>Jul 29 23:50:14 ]: 2016-07-29 23:50:14: pid 4312: DETAIL: timed out. retrying...<br>Jul 29 23:50:14 ]: 2016-07-29 23:50:14: pid 4279: LOG: trying connecting to PostgreSQL server on "<a href="http://192.168.160.5:5432" target="_blank">192.168.160.5:5432</a>" by INET socket<br>Jul 29 23:50:14 ]: 2016-07-29 23:50:14: pid 4279: DETAIL: timed out. retrying...<br>Jul 29 23:50:14 ]: 2016-07-29 23:50:14: pid 4257: LOG: trying connecting to PostgreSQL server on "<a href="http://192.168.160.5:5432" target="_blank">192.168.160.5:5432</a>" by INET socket<br>Jul 29 23:50:14 ]: 2016-07-29 23:50:14: pid 4257: DETAIL: timed out. retrying...<br>Jul 29 23:50:14 ]: 2016-07-29 23:50:14: pid 4210: LOG: trying connecting to PostgreSQL server on "<a href="http://192.168.160.5:5432" target="_blank">192.168.160.5:5432</a>" by INET socket<br>Jul 29 23:50:14 ]: 2016-07-29 23:50:14: pid 4210: DETAIL: timed out. retrying...<br>Jul 29 23:50:14 ]: 2016-07-29 23:50:14: pid 4278: LOG: trying connecting to PostgreSQL server on "<a href="http://192.168.160.5:5432" target="_blank">192.168.160.5:5432</a>" by INET socket<br>Jul 29 23:50:14 ]: 2016-07-29 23:50:14: pid 4278: DETAIL: timed out. retrying...<br>Jul 29 23:50:16 ]: 2016-07-29 23:50:16: pid 4210: LOG: failed to connect to PostgreSQL server on "<a href="http://192.168.160.5:5432" target="_blank">192.168.160.5:5432</a>", getsockopt() detected error "Connection timed out"<br>Jul 29 23:50:16 ]: 2016-07-29 23:50:16: pid 4230: LOG: failed to connect to PostgreSQL server on "<a href="http://192.168.160.5:5432" target="_blank">192.168.160.5:5432</a>", getsockopt() detected error "Connection timed out"<br>Jul 29 23:50:16 ]: 2016-07-29 23:50:16: pid 4312: LOG: failed to connect to PostgreSQL server on "<a href="http://192.168.160.5:5432" target="_blank">192.168.160.5:5432</a>", getsockopt() detected error "Connection timed out"<br>Jul 29 23:50:16 ]: 2016-07-29 23:50:16: pid 4279: LOG: failed to connect to PostgreSQL server on "<a href="http://192.168.160.5:5432" target="_blank">192.168.160.5:5432</a>", getsockopt() detected error "Connection timed out"<br>Jul 29 23:50:16 ]: 2016-07-29 23:50:16: pid 4278: LOG: failed to connect to PostgreSQL server on "<a href="http://192.168.160.5:5432" target="_blank">192.168.160.5:5432</a>", getsockopt() detected error "Connection timed out"<br>Jul 29 23:50:16 ]: 2016-07-29 23:50:16: pid 4312: LOG: received degenerate backend request for node_id: 1 from pid [4312]<br>Jul 29 23:50:16 ]: 2016-07-29 23:50:16: pid 4230: LOG: received degenerate backend request for node_id: 1 from pid [4230]<br>Jul 29 23:50:16 ]: 2016-07-29 23:50:16: pid 4210: LOG: received degenerate backend request for node_id: 1 from pid [4210]<br>Jul 29 23:50:16 ]: 2016-07-29 23:50:16: pid 4279: LOG: received degenerate backend request for node_id: 1 from pid [4279]<br>Jul 29 23:50:16 ]: 2016-07-29 23:50:16: pid 4278: LOG: received degenerate backend request for node_id: 1 from pid [4278]<br>Jul 29 23:50:16 ]: 2016-07-29 23:50:16: pid 4179: LOG: new IPC connection received<br>Jul 29 23:50:16 ]: 2016-07-29 23:50:16: pid 4179: LOG: new IPC connection received<br>Jul 29 23:50:16 ]: 2016-07-29 23:50:16: pid 4257: LOG: failed to connect to PostgreSQL server on "<a href="http://192.168.160.5:5432" target="_blank">192.168.160.5:5432</a>", getsockopt() detected error "Connection timed out"<br>Jul 29 23:50:16 ]: 2016-07-29 23:50:16: pid 4257: LOG: received degenerate backend request for node_id: 1 from pid [4257]<br>Jul 29 23:50:16 ]: 2016-07-29 23:50:16: pid 4179: LOG: new IPC connection received<br>Jul 29 23:50:16 ]: 2016-07-29 23:50:16: pid 4179: LOG: new IPC connection received<br>Jul 29 23:50:16 ]: 2016-07-29 23:50:16: pid 4179: LOG: new IPC connection received<br>Jul 29 23:50:16 ]: 2016-07-29 23:50:16: pid 4179: LOG: new IPC connection received<br>Jul 29 23:50:17 ]: 2016-07-29 23:50:17: pid 4185: LOG: trying connecting to PostgreSQL server on "<a href="http://192.168.160.5:5432" target="_blank">192.168.160.5:5432</a>" by INET socket<br>Jul 29 23:50:17 ]: 2016-07-29 23:50:17: pid 4185: DETAIL: timed out. retrying...<br>Jul 29 23:50:17 ]: 2016-07-29 23:50:17: pid 4211: LOG: trying connecting to PostgreSQL server on "<a href="http://192.168.160.5:5432" target="_blank">192.168.160.5:5432</a>" by INET socket<br>Jul 29 23:50:17 ]: 2016-07-29 23:50:17: pid 4211: DETAIL: timed out. retrying...<br>Jul 29 23:50:18 ]: 2016-07-29 23:50:18: pid 4295: WARNING: degenerate backend set command failed<br>Jul 29 23:50:18 ]: 2016-07-29 23:50:18: pid 4295: DETAIL: ipc command timeout<br>Jul 29 23:50:18 ]: 2016-07-29 23:50:18: pid 4179: LOG: read from socket failed, remote end closed the connection<br>Jul 29 23:50:18 ]: 2016-07-29 23:50:18: pid 4295: FATAL: failed to create a backend connection<br>Jul 29 23:50:18 ]: 2016-07-29 23:50:18: pid 4295: DETAIL: executing failover on backend<br>Jul 29 23:50:18 ]: 2016-07-29 23:50:18: pid 4227: WARNING: degenerate backend set command failed<br>Jul 29 23:50:18 ]: 2016-07-29 23:50:18: pid 4227: DETAIL: ipc command timeout<br>Jul 29 23:50:18 ]: 2016-07-29 23:50:18: pid 4179: LOG: read from socket failed, remote end closed the connection<br>Jul 29 23:50:18 ]: 2016-07-29 23:50:18: pid 4227: FATAL: failed to create a backend connection<br>Jul 29 23:50:18 ]: 2016-07-29 23:50:18: pid 4227: DETAIL: executing failover on backend<br>Jul 29 23:50:18 ]: 2016-07-29 23:50:18: pid 4179: LOG: read from socket failed, remote end closed the connection<br>Jul 29 23:50:18 ]: 2016-07-29 23:50:18: pid 4179: LOG: read from socket failed, remote end closed the connection<br>Jul 29 23:50:18 ]: 2016-07-29 23:50:18: pid 4304: WARNING: degenerate backend set command failed<br>Jul 29 23:50:18 ]: 2016-07-29 23:50:18: pid 4304: DETAIL: ipc command timeout<br>Jul 29 23:50:18 ]: 2016-07-29 23:50:18: pid 4304: FATAL: failed to create a backend connection<br>Jul 29 23:50:18 ]: 2016-07-29 23:50:18: pid 4304: DETAIL: executing failover on backend<br>Jul 29 23:50:18 ]: 2016-07-29 23:50:18: pid 4179: LOG: read from socket failed, remote end closed the connection<br>Jul 29 23:50:18 ]: 2016-07-29 23:50:18: pid 4228: WARNING: degenerate backend set command failed<br>Jul 29 23:50:18 ]: 2016-07-29 23:50:18: pid 4228: DETAIL: ipc command timeout<br>Jul 29 23:50:18 ]: 2016-07-29 23:50:18: pid 4179: LOG: read from socket failed, remote end closed the connection<br>Jul 29 23:50:18 ]: 2016-07-29 23:50:18: pid 4228: FATAL: failed to create a backend connection<br>Jul 29 23:50:18 ]: 2016-07-29 23:50:18: pid 4228: DETAIL: executing failover on backend<br>Jul 29 23:50:18 ]: 2016-07-29 23:50:18: pid 4179: LOG: read from socket failed, remote end closed the connection<br>Jul 29 23:50:19 ]: 2016-07-29 23:50:19: pid 4185: LOG: failed to connect to PostgreSQL server on "<a href="http://192.168.160.5:5432" target="_blank">192.168.160.5:5432</a>", getsockopt() detected error "Connection timed out"<br>Jul 29 23:50:19 ]: 2016-07-29 23:50:19: pid 4211: LOG: failed to connect to PostgreSQL server on "<a href="http://192.168.160.5:5432" target="_blank">192.168.160.5:5432</a>", getsockopt() detected error "Connection timed out"<br>Jul 29 23:50:19 ]: 2016-07-29 23:50:19: pid 4185: LOG: received degenerate backend request for node_id: 1 from pid [4185]<br>Jul 29 23:50:19 ]: 2016-07-29 23:50:19: pid 4211: LOG: received degenerate backend request for node_id: 1 from pid [4211]<br>Jul 29 23:50:19 ]: 2016-07-29 23:50:19: pid 4179: LOG: new IPC connection received<br>Jul 29 23:50:19 ]: 2016-07-29 23:50:19: pid 4179: LOG: new IPC connection received<br>Jul 29 23:50:25 ]: 2016-07-29 23:50:25: pid 4312: WARNING: degenerate backend set command failed<br>Jul 29 23:50:25 ]: 2016-07-29 23:50:25: pid 4312: DETAIL: ipc command timeout<br>Jul 29 23:50:25 ]: 2016-07-29 23:50:25: pid 4279: WARNING: degenerate backend set command failed<br>Jul 29 23:50:25 ]: 2016-07-29 23:50:25: pid 4279: DETAIL: ipc command timeout<br>Jul 29 23:50:25 ]: 2016-07-29 23:50:25: pid 4278: WARNING: degenerate backend set command failed<br>Jul 29 23:50:25 ]: 2016-07-29 23:50:25: pid 4278: DETAIL: ipc command timeout<br>Jul 29 23:50:25 ]: 2016-07-29 23:50:25: pid 4312: FATAL: failed to create a backend connection<br>Jul 29 23:50:25 ]: 2016-07-29 23:50:25: pid 4312: DETAIL: executing failover on backend<br>Jul 29 23:50:26 ]: 2016-07-29 23:50:25: pid 4257: WARNING: degenerate backend set command failed<br>Jul 29 23:50:26 ]: 2016-07-29 23:50:25: pid 4257: DETAIL: ipc command timeout<br>Jul 29 23:50:26 ]: 2016-07-29 23:50:25: pid 4210: WARNING: degenerate backend set command failed<br>Jul 29 23:50:26 ]: 2016-07-29 23:50:25: pid 4210: DETAIL: ipc command timeout<br>Jul 29 23:50:26 ]: 2016-07-29 23:50:25: pid 4278: FATAL: failed to create a backend connection<br>Jul 29 23:50:26 ]: 2016-07-29 23:50:25: pid 4278: DETAIL: executing failover on backend<br>Jul 29 23:50:26 ]: 2016-07-29 23:50:25: pid 4279: FATAL: failed to create a backend connection<br>Jul 29 23:50:26 ]: 2016-07-29 23:50:25: pid 4279: DETAIL: executing failover on backend<br>Jul 29 23:50:26 ]: 2016-07-29 23:50:25: pid 4179: LOG: read from socket failed, remote end closed the connection<br>Jul 29 23:50:26 ]: 2016-07-29 23:50:25: pid 4210: FATAL: failed to create a backend connection<br>Jul 29 23:50:26 ]: 2016-07-29 23:50:25: pid 4210: DETAIL: executing failover on backend<br>Jul 29 23:50:26 ]: 2016-07-29 23:50:25: pid 4257: FATAL: failed to create a backend connection<br>Jul 29 23:50:26 ]: 2016-07-29 23:50:25: pid 4257: DETAIL: executing failover on backend<br>Jul 29 23:50:26 ]: 2016-07-29 23:50:25: pid 4179: LOG: read from socket failed, remote end closed the connection<br>Jul 29 23:50:26 ]: 2016-07-29 23:50:25: pid 4179: LOG: read from socket failed, remote end closed the connection<br>Jul 29 23:50:26 ]: 2016-07-29 23:50:25: pid 4179: LOG: read from socket failed, remote end closed the connection<br>Jul 29 23:50:26 ]: 2016-07-29 23:50:25: pid 4230: WARNING: degenerate backend set command failed<br>Jul 29 23:50:26 ]: 2016-07-29 23:50:25: pid 4230: DETAIL: ipc command timeout<br>Jul 29 23:50:26 ]: 2016-07-29 23:50:25: pid 4179: LOG: read from socket failed, remote end closed the connection<br>Jul 29 23:50:26 ]: 2016-07-29 23:50:25: pid 4230: FATAL: failed to create a backend connection<br>Jul 29 23:50:26 ]: 2016-07-29 23:50:25: pid 4230: DETAIL: executing failover on backend<br>Jul 29 23:50:26 ]: 2016-07-29 23:50:25: pid 4179: LOG: read from socket failed, remote end closed the connection<br>Jul 29 23:50:29 ]: 2016-07-29 23:50:29: pid 4211: WARNING: degenerate backend set command failed<br>Jul 29 23:50:29 ]: 2016-07-29 23:50:29: pid 4211: DETAIL: ipc command timeout<br>Jul 29 23:50:29 ]: 2016-07-29 23:50:29: pid 4185: WARNING: degenerate backend set command failed<br>Jul 29 23:50:29 ]: 2016-07-29 23:50:29: pid 4185: DETAIL: ipc command timeout<br>Jul 29 23:50:29 ]: 2016-07-29 23:50:29: pid 4179: LOG: read from socket failed, remote end closed the connection<br>Jul 29 23:50:29 ]: 2016-07-29 23:50:29: pid 4185: FATAL: failed to create a backend connection<br>Jul 29 23:50:29 ]: 2016-07-29 23:50:29: pid 4185: DETAIL: executing failover on backend<br>Jul 29 23:50:29 ]: 2016-07-29 23:50:29: pid 4211: FATAL: failed to create a backend connection<br>Jul 29 23:50:29 ]: 2016-07-29 23:50:29: pid 4211: DETAIL: executing failover on backend<br>Jul 29 23:50:29 ]: 2016-07-29 23:50:29: pid 4179: LOG: read from socket failed, remote end closed the connection<br>Jul 29 23:51:13 ]: 2016-07-29 23:51:13: pid 4179: LOG: read from socket failed with error :"Connection reset by peer"<br>Jul 29 23:51:19 ]: 2016-07-29 23:51:19: pid 4179: LOG: read from socket failed with error :"Connection reset by peer"<br><br><br></div></div></div>Thanks & Regards,<br></div>Guptesh</div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Jun 16, 2016 at 6:57 PM, Guptesh Kumar <span dir="ltr"><<a href="mailto:guptesh.cg4@gmail.com" target="_blank">guptesh.cg4@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div>Hi Usama,<br><br></div>I built from V3_5_STABLE head and tested. The problem is fixed :)<br><br><br></div><div>Thanks & Regards,<br></div><div>Guptesh<br></div></div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Jun 16, 2016 at 1:14 AM, Muhammad Usama <span dir="ltr"><<a href="mailto:m.usama@gmail.com" target="_blank">m.usama@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi<div><br></div><div>I have fixed the similar problem in pgpool-II 3.5 a few days back and it will be part of next point release, which is expected in a few days time</div><div><br></div><div><a href="http://git.postgresql.org/gitweb?p=pgpool2.git;a=commitdiff;h=6ae42e74d48391b8cfaa97927e00bd65aec68fe8" target="_blank">http://git.postgresql.org/gitweb?p=pgpool2.git;a=commitdiff;h=6ae42e74d48391b8cfaa97927e00bd65aec68fe8</a><br></div><div><br></div><div>You can try if building from the V3_5_STABLE head fixes the problem.</div><div><br></div><div>Regards</div><span><font color="#888888"><div>Muhammad Usama</div><div><br></div>
</font></span></div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Jun 15, 2016 at 11:24 PM, Guptesh Kumar <span dir="ltr"><<a href="mailto:guptesh.cg4@gmail.com" target="_blank">guptesh.cg4@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div><div><div><div><div>Hi Ricardo,<br><br></div>I am facing a similar issue -<br><br></div>Node 1 : Master pgpool, Master Postgresql<br></div>Node 2 : Slave Pgpool, Slave Postgresql<br><br></div>When I pull out the power cable of master node(Node 1), the Slave watchdog promotes itself to Master correctly. But the pgpool in NOW master node(Node 2) neither disconnects the Node 1 Postgresql from the pool nor it promotes the Node 2 postgresql as Master(failover does not happen). All my DB connection hangs.<br><br></div><div>I tried various values for health_check parameters but none of them helped.<br><br></div>Softwares Used are -<br>pgpool-II-pg94-3.5.2-1pgdg.rhel7.x86_64<br>postgresql94-server-9.4.7-1PGDG.rhel7.x86_64<br><br></div>Thanks & Regards,<br></div>Guptesh<br><div><div><div><br></div></div></div></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div>On Tue, May 10, 2016 at 11:36 PM, Ricardo Larrañaga <span dir="ltr"><<a href="mailto:ricardo.larranaga@gmail.com" target="_blank">ricardo.larranaga@gmail.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div><div dir="ltr"><div><div><div><div>Hi guys:<br></div>I have been working on this a little bit, and i encounter this problem (running pgpool 3.5.2 on a 2 node cluster with streaming replication, if i shut down the interface of the slave, pgpool on the master hangs the connections). And i realize that the problem only happens when i shut down the interface. If i either shut down the processes or use the firewall to block backend communication, pgpool reacts properly.<br><br></div>Does anyone have experience with this or have a 2 node cluster with streaming replication to test it. I am trying to identify what makes the setup hang, since shutting down the interface is the best way i have to simulate a node crash. When i shut down the interface, both postgres and pgpool loose connection, but this is a valid scenario that could happen in real life (when the whole server crashes).<br></div>Thanks a lot.<br></div>Regards<br><br></div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, May 5, 2016 at 4:55 PM, Ricardo Larrañaga <span dir="ltr"><<a href="mailto:ricardo.larranaga@gmail.com" target="_blank">ricardo.larranaga@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div>Hello guys:<br></div>I am running pgpool 3.5.2 in a 2 node cluster with postgresql-9.5 in master-slave and streaming replication mode.<br><br></div>I have been testing failover and failback for a while. While doing failover by shutting down the processes, everything looks fine.<br><br><br></div><div>The one test that is failing is when i shutdown the interface of one node.<br></div><div>Right now i am shutting down the interface of the slave node (both pgool slave and postgresql slave).<br><br></div><div>The problem i am running into is that after doing this, all connections to my database (through pgpool) hang. i am testing it with psql, and psql just hangs and does not give me an output. When i bring the interface of the slave back up, and try to connect to psql again, it looks like pgpool never marked the postgresql node as disconnected. <br><br></div><div>I tried both with and without health check, and also tried with different values of health_check_timeout. my connect timeout value is default (10 seconds).<br><br></div><div>Any one has encountered this issue? I just dont see pgpool attempting to do any failover. Pgpool is still running though, i can see the logs still comming. I just never see an error.<br><br></div><div>I am NOT using interface monitoring, and i would prefer not to use it.<br><br></div><div>Any pointers on how could i troubleshoot this?<br></div><div>Thanks.<br></div><div>Regards<br><br></div></div>
</blockquote></div><br></div>
</div></div><br></div></div>_______________________________________________<br>
pgpool-general mailing list<br>
<a href="mailto:pgpool-general@pgpool.net" target="_blank">pgpool-general@pgpool.net</a><br>
<a href="http://www.pgpool.net/mailman/listinfo/pgpool-general" rel="noreferrer" target="_blank">http://www.pgpool.net/mailman/listinfo/pgpool-general</a><br>
<br></blockquote></div><br></div>
<br>_______________________________________________<br>
pgpool-general mailing list<br>
<a href="mailto:pgpool-general@pgpool.net" target="_blank">pgpool-general@pgpool.net</a><br>
<a href="http://www.pgpool.net/mailman/listinfo/pgpool-general" rel="noreferrer" target="_blank">http://www.pgpool.net/mailman/listinfo/pgpool-general</a><br>
<br></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>