<div dir="ltr"><div><div><div><div><div><div>Hi Usama,<br><br><br></div>I am still facing issues here-<br><br></div>I shutdown primary(Node0) and standby(Node1) became primary successfully.<br></div>When Node0 came up, i recovered postgres0(postgres on Node0). Now Node0 is standby and Node1 is primary.<br></div>Again
if I shutdown Node1(now primary), failover is not happening on Node0.
Below are the logs after health_check retry count is expired<br><br><br><br>Jul 29 23:50:02 ]: 2016-07-29 23:50:02: pid 4211: DETAIL: timed out. retrying...<br>Jul 29 23:50:02 ]: 2016-07-29 23:50:02: pid 4185: LOG: trying connecting to PostgreSQL server on "<a target="_blank" href="http://192.168.160.5:5432">192.168.160.5:5432</a>" by INET socket<br>Jul 29 23:50:02 ]: 2016-07-29 23:50:02: pid 4185: DETAIL: timed out. retrying...<br>Jul 29 23:50:04 ]: 2016-07-29 23:50:04: pid 4312: LOG: trying connecting to PostgreSQL server on "<a target="_blank" href="http://192.168.160.5:5432">192.168.160.5:5432</a>" by INET socket<br>Jul 29 23:50:04 ]: 2016-07-29 23:50:04: pid 4312: DETAIL: timed out. retrying...<br>Jul 29 23:50:04 ]: 2016-07-29 23:50:04: pid 4279: LOG: trying connecting to PostgreSQL server on "<a target="_blank" href="http://192.168.160.5:5432">192.168.160.5:5432</a>" by INET socket<br>Jul 29 23:50:04 ]: 2016-07-29 23:50:04: pid 4279: DETAIL: timed out. retrying...<br>Jul 29 23:50:04 ]: 2016-07-29 23:50:04: pid 4257: LOG: trying connecting to PostgreSQL server on "<a target="_blank" href="http://192.168.160.5:5432">192.168.160.5:5432</a>" by INET socket<br>Jul 29 23:50:04 ]: 2016-07-29 23:50:04: pid 4257: DETAIL: timed out. retrying...<br>Jul 29 23:50:04 ]: 2016-07-29 23:50:04: pid 4230: LOG: trying connecting to PostgreSQL server on "<a target="_blank" href="http://192.168.160.5:5432">192.168.160.5:5432</a>" by INET socket<br>Jul 29 23:50:04 ]: 2016-07-29 23:50:04: pid 4230: DETAIL: timed out. retrying...<br>Jul 29 23:50:04 ]: 2016-07-29 23:50:04: pid 4210: LOG: trying connecting to PostgreSQL server on "<a target="_blank" href="http://192.168.160.5:5432">192.168.160.5:5432</a>" by INET socket<br>Jul 29 23:50:04 ]: 2016-07-29 23:50:04: pid 4210: DETAIL: timed out. retrying...<br>Jul 29 23:50:04 ]: 2016-07-29 23:50:04: pid 4278: LOG: trying connecting to PostgreSQL server on "<a target="_blank" href="http://192.168.160.5:5432">192.168.160.5:5432</a>" by INET socket<br>Jul 29 23:50:04 ]: 2016-07-29 23:50:04: pid 4278: DETAIL: timed out. retrying...<br>Jul 29 23:50:06 ]: 2016-07-29 23:50:06: pid 4304: LOG: trying connecting to PostgreSQL server on "<a target="_blank" href="http://192.168.160.5:5432">192.168.160.5:5432</a>" by INET socket<br>Jul 29 23:50:06 ]: 2016-07-29 23:50:06: pid 4304: DETAIL: timed out. retrying...<br>Jul 29 23:50:06 ]: 2016-07-29 23:50:06: pid 4227: LOG: trying connecting to PostgreSQL server on "<a target="_blank" href="http://192.168.160.5:5432">192.168.160.5:5432</a>" by INET socket<br>Jul 29 23:50:06 ]: 2016-07-29 23:50:06: pid 4227: DETAIL: timed out. retrying...<br>Jul 29 23:50:06 ]: 2016-07-29 23:50:06: pid 4276: LOG: trying connecting to PostgreSQL server on "<a target="_blank" href="http://192.168.160.5:5432">192.168.160.5:5432</a>" by INET socket<br>Jul 29 23:50:06 ]: 2016-07-29 23:50:06: pid 4276: DETAIL: timed out. retrying...<br>Jul 29 23:50:06 ]: 2016-07-29 23:50:06: pid 4261: LOG: trying connecting to PostgreSQL server on "<a target="_blank" href="http://192.168.160.5:5432">192.168.160.5:5432</a>" by INET socket<br>Jul 29 23:50:06 ]: 2016-07-29 23:50:06: pid 4261: DETAIL: timed out. retrying...<br>Jul 29 23:50:06 ]: 2016-07-29 23:50:06: pid 4297: LOG: trying connecting to PostgreSQL server on "<a target="_blank" href="http://192.168.160.5:5432">192.168.160.5:5432</a>" by INET socket<br>Jul 29 23:50:06 ]: 2016-07-29 23:50:06: pid 4297: DETAIL: timed out. retrying...<br>Jul 29 23:50:06 ]: 2016-07-29 23:50:06: pid 4204: LOG: trying connecting to PostgreSQL server on "<a target="_blank" href="http://192.168.160.5:5432">192.168.160.5:5432</a>" by INET socket<br>Jul 29 23:50:06 ]: 2016-07-29 23:50:06: pid 4204: DETAIL: timed out. retrying...<br>Jul 29 23:50:06 ]: 2016-07-29 23:50:06: pid 4295: LOG: trying connecting to PostgreSQL server on "<a target="_blank" href="http://192.168.160.5:5432">192.168.160.5:5432</a>" by INET socket<br>Jul 29 23:50:06 ]: 2016-07-29 23:50:06: pid 4295: DETAIL: timed out. retrying...<br>Jul 29 23:50:06 ]: 2016-07-29 23:50:06: pid 4291: LOG: trying connecting to PostgreSQL server on "<a target="_blank" href="http://192.168.160.5:5432">192.168.160.5:5432</a>" by INET socket<br>Jul 29 23:50:06 ]: 2016-07-29 23:50:06: pid 4291: DETAIL: timed out. retrying...<br>Jul 29 23:50:06 ]: 2016-07-29 23:50:06: pid 4228: LOG: trying connecting to PostgreSQL server on "<a target="_blank" href="http://192.168.160.5:5432">192.168.160.5:5432</a>" by INET socket<br>Jul 29 23:50:06 ]: 2016-07-29 23:50:06: pid 4228: DETAIL: timed out. retrying...<br>Jul 29 23:50:07 ]: 2016-07-29 23:50:07: pid 4211: LOG: trying connecting to PostgreSQL server on "<a target="_blank" href="http://192.168.160.5:5432">192.168.160.5:5432</a>" by INET socket<br>Jul 29 23:50:07 ]: 2016-07-29 23:50:07: pid 4211: DETAIL: timed out. retrying...<br>Jul 29 23:50:07 ]: 2016-07-29 23:50:07: pid 4185: LOG: trying connecting to PostgreSQL server on "<a target="_blank" href="http://192.168.160.5:5432">192.168.160.5:5432</a>" by INET socket<br>Jul 29 23:50:07 ]: 2016-07-29 23:50:07: pid 4185: DETAIL: timed out. retrying...<br>Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4228: LOG: failed to connect to PostgreSQL server on "<a target="_blank" href="http://192.168.160.5:5432">192.168.160.5:5432</a>", getsockopt() detected error "Connection timed out"<br>Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4291: LOG: failed to connect to PostgreSQL server on "<a target="_blank" href="http://192.168.160.5:5432">192.168.160.5:5432</a>", getsockopt() detected error "Connection timed out"<br>Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4261: LOG: failed to connect to PostgreSQL server on "<a target="_blank" href="http://192.168.160.5:5432">192.168.160.5:5432</a>", getsockopt() detected error "Connection timed out"<br>Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4297: LOG: failed to connect to PostgreSQL server on "<a target="_blank" href="http://192.168.160.5:5432">192.168.160.5:5432</a>", getsockopt() detected error "Connection timed out"<br>Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4295: LOG: failed to connect to PostgreSQL server on "<a target="_blank" href="http://192.168.160.5:5432">192.168.160.5:5432</a>", getsockopt() detected error "Connection timed out"<br>Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4227: LOG: failed to connect to PostgreSQL server on "<a target="_blank" href="http://192.168.160.5:5432">192.168.160.5:5432</a>", getsockopt() detected error "Connection timed out"<br>Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4276: LOG: failed to connect to PostgreSQL server on "<a target="_blank" href="http://192.168.160.5:5432">192.168.160.5:5432</a>", getsockopt() detected error "Connection timed out"<br>Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4228: LOG: received degenerate backend request for node_id: 1 from pid [4228]<br>Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4261: LOG: received degenerate backend request for node_id: 1 from pid [4261]<br>Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4297: LOG: received degenerate backend request for node_id: 1 from pid [4297]<br>Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4291: LOG: received degenerate backend request for node_id: 1 from pid [4291]<br>Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4295: LOG: received degenerate backend request for node_id: 1 from pid [4295]<br>Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4227: LOG: received degenerate backend request for node_id: 1 from pid [4227]<br>Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4276: LOG: received degenerate backend request for node_id: 1 from pid [4276]<br>Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4204: LOG: failed to connect to PostgreSQL server on "<a target="_blank" href="http://192.168.160.5:5432">192.168.160.5:5432</a>", getsockopt() detected error "Connection timed out"<br>Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4204: LOG: received degenerate backend request for node_id: 1 from pid [4204]<br>Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4304: LOG: failed to connect to PostgreSQL server on "<a target="_blank" href="http://192.168.160.5:5432">192.168.160.5:5432</a>", getsockopt() detected error "Connection timed out"<br>Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4304: LOG: received degenerate backend request for node_id: 1 from pid [4304]<br>Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4179: LOG: new IPC connection received<br>Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4179: LOG: new IPC connection received<br>Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4179: LOG: new IPC connection received<br>Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4179: LOG: new IPC connection received<br>Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4179: LOG: new IPC connection received<br>Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4179: LOG: new IPC connection received<br>Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4179: LOG: new IPC connection received<br>Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4179: LOG: new IPC connection received<br>Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4179: LOG: new IPC connection received<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4257: LOG: trying connecting to PostgreSQL server on "<a target="_blank" href="http://192.168.160.5:5432">192.168.160.5:5432</a>" by INET socket<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4257: DETAIL: timed out. retrying...<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4279: LOG: trying connecting to PostgreSQL server on "<a target="_blank" href="http://192.168.160.5:5432">192.168.160.5:5432</a>" by INET socket<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4279: DETAIL: timed out. retrying...<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4230: LOG: trying connecting to PostgreSQL server on "<a target="_blank" href="http://192.168.160.5:5432">192.168.160.5:5432</a>" by INET socket<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4230: DETAIL: timed out. retrying...<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4210: LOG: trying connecting to PostgreSQL server on "<a target="_blank" href="http://192.168.160.5:5432">192.168.160.5:5432</a>" by INET socket<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4210: DETAIL: timed out. retrying...<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4312: LOG: trying connecting to PostgreSQL server on "<a target="_blank" href="http://192.168.160.5:5432">192.168.160.5:5432</a>" by INET socket<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4312: DETAIL: timed out. retrying...<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4278: LOG: trying connecting to PostgreSQL server on "<a target="_blank" href="http://192.168.160.5:5432">192.168.160.5:5432</a>" by INET socket<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4278: DETAIL: timed out. retrying...<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4179: LOG: read from socket failed, remote end closed the connection<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4250: WARNING: degenerate backend set command failed<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4250: DETAIL: ipc command timeout<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4329: WARNING: degenerate backend set command failed<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4329: DETAIL: ipc command timeout<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4269: WARNING: degenerate backend set command failed<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4269: DETAIL: ipc command timeout<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4289: WARNING: degenerate backend set command failed<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4289: DETAIL: ipc command timeout<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4217: WARNING: degenerate backend set command failed<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4217: DETAIL: ipc command timeout<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4266: WARNING: degenerate backend set command failed<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4266: DETAIL: ipc command timeout<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4179: LOG: read from socket failed, remote end closed the connection<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4329: FATAL: failed to create a backend connection<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4329: DETAIL: executing failover on backend<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4217: FATAL: failed to create a backend connection<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4217: DETAIL: executing failover on backend<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4266: FATAL: failed to create a backend connection<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4266: DETAIL: executing failover on backend<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4269: FATAL: failed to create a backend connection<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4269: DETAIL: executing failover on backend<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4289: FATAL: failed to create a backend connection<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4289: DETAIL: executing failover on backend<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4179: LOG: read from socket failed, remote end closed the connection<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4179: LOG: read from socket failed, remote end closed the connection<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4250: FATAL: failed to create a backend connection<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4250: DETAIL: executing failover on backend<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4179: LOG: read from socket failed, remote end closed the connection<br>Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4179: LOG: read from socket failed, remote end closed the connection<br>Jul 29 23:50:12 ]: 2016-07-29 23:50:12: pid 4185: LOG: trying connecting to PostgreSQL server on "<a target="_blank" href="http://192.168.160.5:5432">192.168.160.5:5432</a>" by INET socket<br>Jul 29 23:50:12 ]: 2016-07-29 23:50:12: pid 4185: DETAIL: timed out. retrying...<br>Jul 29 23:50:12 ]: 2016-07-29 23:50:12: pid 4211: LOG: trying connecting to PostgreSQL server on "<a target="_blank" href="http://192.168.160.5:5432">192.168.160.5:5432</a>" by INET socket<br>Jul 29 23:50:12 ]: 2016-07-29 23:50:12: pid 4211: DETAIL: timed out. retrying...<br>Jul 29 23:50:14 ]: 2016-07-29 23:50:14: pid 4230: LOG: trying connecting to PostgreSQL server on "<a target="_blank" href="http://192.168.160.5:5432">192.168.160.5:5432</a>" by INET socket<br>Jul 29 23:50:14 ]: 2016-07-29 23:50:14: pid 4230: DETAIL: timed out. retrying...<br>Jul 29 23:50:14 ]: 2016-07-29 23:50:14: pid 4312: LOG: trying connecting to PostgreSQL server on "<a target="_blank" href="http://192.168.160.5:5432">192.168.160.5:5432</a>" by INET socket<br>Jul 29 23:50:14 ]: 2016-07-29 23:50:14: pid 4312: DETAIL: timed out. retrying...<br>Jul 29 23:50:14 ]: 2016-07-29 23:50:14: pid 4279: LOG: trying connecting to PostgreSQL server on "<a target="_blank" href="http://192.168.160.5:5432">192.168.160.5:5432</a>" by INET socket<br>Jul 29 23:50:14 ]: 2016-07-29 23:50:14: pid 4279: DETAIL: timed out. retrying...<br>Jul 29 23:50:14 ]: 2016-07-29 23:50:14: pid 4257: LOG: trying connecting to PostgreSQL server on "<a target="_blank" href="http://192.168.160.5:5432">192.168.160.5:5432</a>" by INET socket<br>Jul 29 23:50:14 ]: 2016-07-29 23:50:14: pid 4257: DETAIL: timed out. retrying...<br>Jul 29 23:50:14 ]: 2016-07-29 23:50:14: pid 4210: LOG: trying connecting to PostgreSQL server on "<a target="_blank" href="http://192.168.160.5:5432">192.168.160.5:5432</a>" by INET socket<br>Jul 29 23:50:14 ]: 2016-07-29 23:50:14: pid 4210: DETAIL: timed out. retrying...<br>Jul 29 23:50:14 ]: 2016-07-29 23:50:14: pid 4278: LOG: trying connecting to PostgreSQL server on "<a target="_blank" href="http://192.168.160.5:5432">192.168.160.5:5432</a>" by INET socket<br>Jul 29 23:50:14 ]: 2016-07-29 23:50:14: pid 4278: DETAIL: timed out. retrying...<br>Jul 29 23:50:16 ]: 2016-07-29 23:50:16: pid 4210: LOG: failed to connect to PostgreSQL server on "<a target="_blank" href="http://192.168.160.5:5432">192.168.160.5:5432</a>", getsockopt() detected error "Connection timed out"<br>Jul 29 23:50:16 ]: 2016-07-29 23:50:16: pid 4230: LOG: failed to connect to PostgreSQL server on "<a target="_blank" href="http://192.168.160.5:5432">192.168.160.5:5432</a>", getsockopt() detected error "Connection timed out"<br>Jul 29 23:50:16 ]: 2016-07-29 23:50:16: pid 4312: LOG: failed to connect to PostgreSQL server on "<a target="_blank" href="http://192.168.160.5:5432">192.168.160.5:5432</a>", getsockopt() detected error "Connection timed out"<br>Jul 29 23:50:16 ]: 2016-07-29 23:50:16: pid 4279: LOG: failed to connect to PostgreSQL server on "<a target="_blank" href="http://192.168.160.5:5432">192.168.160.5:5432</a>", getsockopt() detected error "Connection timed out"<br>Jul 29 23:50:16 ]: 2016-07-29 23:50:16: pid 4278: LOG: failed to connect to PostgreSQL server on "<a target="_blank" href="http://192.168.160.5:5432">192.168.160.5:5432</a>", getsockopt() detected error "Connection timed out"<br>Jul 29 23:50:16 ]: 2016-07-29 23:50:16: pid 4312: LOG: received degenerate backend request for node_id: 1 from pid [4312]<br>Jul 29 23:50:16 ]: 2016-07-29 23:50:16: pid 4230: LOG: received degenerate backend request for node_id: 1 from pid [4230]<br>Jul 29 23:50:16 ]: 2016-07-29 23:50:16: pid 4210: LOG: received degenerate backend request for node_id: 1 from pid [4210]<br>Jul 29 23:50:16 ]: 2016-07-29 23:50:16: pid 4279: LOG: received degenerate backend request for node_id: 1 from pid [4279]<br>Jul 29 23:50:16 ]: 2016-07-29 23:50:16: pid 4278: LOG: received degenerate backend request for node_id: 1 from pid [4278]<br>Jul 29 23:50:16 ]: 2016-07-29 23:50:16: pid 4179: LOG: new IPC connection received<br>Jul 29 23:50:16 ]: 2016-07-29 23:50:16: pid 4179: LOG: new IPC connection received<br>Jul 29 23:50:16 ]: 2016-07-29 23:50:16: pid 4257: LOG: failed to connect to PostgreSQL server on "<a target="_blank" href="http://192.168.160.5:5432">192.168.160.5:5432</a>", getsockopt() detected error "Connection timed out"<br>Jul 29 23:50:16 ]: 2016-07-29 23:50:16: pid 4257: LOG: received degenerate backend request for node_id: 1 from pid [4257]<br>Jul 29 23:50:16 ]: 2016-07-29 23:50:16: pid 4179: LOG: new IPC connection received<br>Jul 29 23:50:16 ]: 2016-07-29 23:50:16: pid 4179: LOG: new IPC connection received<br>Jul 29 23:50:16 ]: 2016-07-29 23:50:16: pid 4179: LOG: new IPC connection received<br>Jul 29 23:50:16 ]: 2016-07-29 23:50:16: pid 4179: LOG: new IPC connection received<br>Jul 29 23:50:17 ]: 2016-07-29 23:50:17: pid 4185: LOG: trying connecting to PostgreSQL server on "<a target="_blank" href="http://192.168.160.5:5432">192.168.160.5:5432</a>" by INET socket<br>Jul 29 23:50:17 ]: 2016-07-29 23:50:17: pid 4185: DETAIL: timed out. retrying...<br>Jul 29 23:50:17 ]: 2016-07-29 23:50:17: pid 4211: LOG: trying connecting to PostgreSQL server on "<a target="_blank" href="http://192.168.160.5:5432">192.168.160.5:5432</a>" by INET socket<br>Jul 29 23:50:17 ]: 2016-07-29 23:50:17: pid 4211: DETAIL: timed out. retrying...<br>Jul 29 23:50:18 ]: 2016-07-29 23:50:18: pid 4295: WARNING: degenerate backend set command failed<br>Jul 29 23:50:18 ]: 2016-07-29 23:50:18: pid 4295: DETAIL: ipc command timeout<br>Jul 29 23:50:18 ]: 2016-07-29 23:50:18: pid 4179: LOG: read from socket failed, remote end closed the connection<br>Jul 29 23:50:18 ]: 2016-07-29 23:50:18: pid 4295: FATAL: failed to create a backend connection<br>Jul 29 23:50:18 ]: 2016-07-29 23:50:18: pid 4295: DETAIL: executing failover on backend<br>Jul 29 23:50:18 ]: 2016-07-29 23:50:18: pid 4227: WARNING: degenerate backend set command failed<br>Jul 29 23:50:18 ]: 2016-07-29 23:50:18: pid 4227: DETAIL: ipc command timeout<br>Jul 29 23:50:18 ]: 2016-07-29 23:50:18: pid 4179: LOG: read from socket failed, remote end closed the connection<br>Jul 29 23:50:18 ]: 2016-07-29 23:50:18: pid 4227: FATAL: failed to create a backend connection<br>Jul 29 23:50:18 ]: 2016-07-29 23:50:18: pid 4227: DETAIL: executing failover on backend<br>Jul 29 23:50:18 ]: 2016-07-29 23:50:18: pid 4179: LOG: read from socket failed, remote end closed the connection<br>Jul 29 23:50:18 ]: 2016-07-29 23:50:18: pid 4179: LOG: read from socket failed, remote end closed the connection<br>Jul 29 23:50:18 ]: 2016-07-29 23:50:18: pid 4304: WARNING: degenerate backend set command failed<br>Jul 29 23:50:18 ]: 2016-07-29 23:50:18: pid 4304: DETAIL: ipc command timeout<br>Jul 29 23:50:18 ]: 2016-07-29 23:50:18: pid 4304: FATAL: failed to create a backend connection<br>Jul 29 23:50:18 ]: 2016-07-29 23:50:18: pid 4304: DETAIL: executing failover on backend<br>Jul 29 23:50:18 ]: 2016-07-29 23:50:18: pid 4179: LOG: read from socket failed, remote end closed the connection<br>Jul 29 23:50:18 ]: 2016-07-29 23:50:18: pid 4228: WARNING: degenerate backend set command failed<br>Jul 29 23:50:18 ]: 2016-07-29 23:50:18: pid 4228: DETAIL: ipc command timeout<br>Jul 29 23:50:18 ]: 2016-07-29 23:50:18: pid 4179: LOG: read from socket failed, remote end closed the connection<br>Jul 29 23:50:18 ]: 2016-07-29 23:50:18: pid 4228: FATAL: failed to create a backend connection<br>Jul 29 23:50:18 ]: 2016-07-29 23:50:18: pid 4228: DETAIL: executing failover on backend<br>Jul 29 23:50:18 ]: 2016-07-29 23:50:18: pid 4179: LOG: read from socket failed, remote end closed the connection<br>Jul 29 23:50:19 ]: 2016-07-29 23:50:19: pid 4185: LOG: failed to connect to PostgreSQL server on "<a target="_blank" href="http://192.168.160.5:5432">192.168.160.5:5432</a>", getsockopt() detected error "Connection timed out"<br>Jul 29 23:50:19 ]: 2016-07-29 23:50:19: pid 4211: LOG: failed to connect to PostgreSQL server on "<a target="_blank" href="http://192.168.160.5:5432">192.168.160.5:5432</a>", getsockopt() detected error "Connection timed out"<br>Jul 29 23:50:19 ]: 2016-07-29 23:50:19: pid 4185: LOG: received degenerate backend request for node_id: 1 from pid [4185]<br>Jul 29 23:50:19 ]: 2016-07-29 23:50:19: pid 4211: LOG: received degenerate backend request for node_id: 1 from pid [4211]<br>Jul 29 23:50:19 ]: 2016-07-29 23:50:19: pid 4179: LOG: new IPC connection received<br>Jul 29 23:50:19 ]: 2016-07-29 23:50:19: pid 4179: LOG: new IPC connection received<br>Jul 29 23:50:25 ]: 2016-07-29 23:50:25: pid 4312: WARNING: degenerate backend set command failed<br>Jul 29 23:50:25 ]: 2016-07-29 23:50:25: pid 4312: DETAIL: ipc command timeout<br>Jul 29 23:50:25 ]: 2016-07-29 23:50:25: pid 4279: WARNING: degenerate backend set command failed<br>Jul 29 23:50:25 ]: 2016-07-29 23:50:25: pid 4279: DETAIL: ipc command timeout<br>Jul 29 23:50:25 ]: 2016-07-29 23:50:25: pid 4278: WARNING: degenerate backend set command failed<br>Jul 29 23:50:25 ]: 2016-07-29 23:50:25: pid 4278: DETAIL: ipc command timeout<br>Jul 29 23:50:25 ]: 2016-07-29 23:50:25: pid 4312: FATAL: failed to create a backend connection<br>Jul 29 23:50:25 ]: 2016-07-29 23:50:25: pid 4312: DETAIL: executing failover on backend<br>Jul 29 23:50:26 ]: 2016-07-29 23:50:25: pid 4257: WARNING: degenerate backend set command failed<br>Jul 29 23:50:26 ]: 2016-07-29 23:50:25: pid 4257: DETAIL: ipc command timeout<br>Jul 29 23:50:26 ]: 2016-07-29 23:50:25: pid 4210: WARNING: degenerate backend set command failed<br>Jul 29 23:50:26 ]: 2016-07-29 23:50:25: pid 4210: DETAIL: ipc command timeout<br>Jul 29 23:50:26 ]: 2016-07-29 23:50:25: pid 4278: FATAL: failed to create a backend connection<br>Jul 29 23:50:26 ]: 2016-07-29 23:50:25: pid 4278: DETAIL: executing failover on backend<br>Jul 29 23:50:26 ]: 2016-07-29 23:50:25: pid 4279: FATAL: failed to create a backend connection<br>Jul 29 23:50:26 ]: 2016-07-29 23:50:25: pid 4279: DETAIL: executing failover on backend<br>Jul 29 23:50:26 ]: 2016-07-29 23:50:25: pid 4179: LOG: read from socket failed, remote end closed the connection<br>Jul 29 23:50:26 ]: 2016-07-29 23:50:25: pid 4210: FATAL: failed to create a backend connection<br>Jul 29 23:50:26 ]: 2016-07-29 23:50:25: pid 4210: DETAIL: executing failover on backend<br>Jul 29 23:50:26 ]: 2016-07-29 23:50:25: pid 4257: FATAL: failed to create a backend connection<br>Jul 29 23:50:26 ]: 2016-07-29 23:50:25: pid 4257: DETAIL: executing failover on backend<br>Jul 29 23:50:26 ]: 2016-07-29 23:50:25: pid 4179: LOG: read from socket failed, remote end closed the connection<br>Jul 29 23:50:26 ]: 2016-07-29 23:50:25: pid 4179: LOG: read from socket failed, remote end closed the connection<br>Jul 29 23:50:26 ]: 2016-07-29 23:50:25: pid 4179: LOG: read from socket failed, remote end closed the connection<br>Jul 29 23:50:26 ]: 2016-07-29 23:50:25: pid 4230: WARNING: degenerate backend set command failed<br>Jul 29 23:50:26 ]: 2016-07-29 23:50:25: pid 4230: DETAIL: ipc command timeout<br>Jul 29 23:50:26 ]: 2016-07-29 23:50:25: pid 4179: LOG: read from socket failed, remote end closed the connection<br>Jul 29 23:50:26 ]: 2016-07-29 23:50:25: pid 4230: FATAL: failed to create a backend connection<br>Jul 29 23:50:26 ]: 2016-07-29 23:50:25: pid 4230: DETAIL: executing failover on backend<br>Jul 29 23:50:26 ]: 2016-07-29 23:50:25: pid 4179: LOG: read from socket failed, remote end closed the connection<br>Jul 29 23:50:29 ]: 2016-07-29 23:50:29: pid 4211: WARNING: degenerate backend set command failed<br>Jul 29 23:50:29 ]: 2016-07-29 23:50:29: pid 4211: DETAIL: ipc command timeout<br>Jul 29 23:50:29 ]: 2016-07-29 23:50:29: pid 4185: WARNING: degenerate backend set command failed<br>Jul 29 23:50:29 ]: 2016-07-29 23:50:29: pid 4185: DETAIL: ipc command timeout<br>Jul 29 23:50:29 ]: 2016-07-29 23:50:29: pid 4179: LOG: read from socket failed, remote end closed the connection<br>Jul 29 23:50:29 ]: 2016-07-29 23:50:29: pid 4185: FATAL: failed to create a backend connection<br>Jul 29 23:50:29 ]: 2016-07-29 23:50:29: pid 4185: DETAIL: executing failover on backend<br>Jul 29 23:50:29 ]: 2016-07-29 23:50:29: pid 4211: FATAL: failed to create a backend connection<br>Jul 29 23:50:29 ]: 2016-07-29 23:50:29: pid 4211: DETAIL: executing failover on backend<br>Jul 29 23:50:29 ]: 2016-07-29 23:50:29: pid 4179: LOG: read from socket failed, remote end closed the connection<br>Jul 29 23:51:13 ]: 2016-07-29 23:51:13: pid 4179: LOG: read from socket failed with error :"Connection reset by peer"<br>Jul 29 23:51:19 ]: 2016-07-29 23:51:19: pid 4179: LOG: read from socket failed with error :"Connection reset by peer"<br><br><br></div>Thanks & Regards,<br></div>Guptesh</div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Jun 16, 2016 at 6:57 PM, Guptesh Kumar <span dir="ltr"><<a href="mailto:guptesh.cg4@gmail.com" target="_blank">guptesh.cg4@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div>Hi Usama,<br><br></div>I built from V3_5_STABLE head and tested. The problem is fixed :)<br><br><br></div><div>Thanks & Regards,<br></div><div>Guptesh<br></div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Jun 16, 2016 at 1:14 AM, Muhammad Usama <span dir="ltr"><<a href="mailto:m.usama@gmail.com" target="_blank">m.usama@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi<div><br></div><div>I have fixed the similar problem in pgpool-II 3.5 a few days back and it will be part of next point release, which is expected in a few days time</div><div><br></div><div><a href="http://git.postgresql.org/gitweb?p=pgpool2.git;a=commitdiff;h=6ae42e74d48391b8cfaa97927e00bd65aec68fe8" target="_blank">http://git.postgresql.org/gitweb?p=pgpool2.git;a=commitdiff;h=6ae42e74d48391b8cfaa97927e00bd65aec68fe8</a><br></div><div><br></div><div>You can try if building from the V3_5_STABLE head fixes the problem.</div><div><br></div><div>Regards</div><span><font color="#888888"><div>Muhammad Usama</div><div><br></div>
</font></span></div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Jun 15, 2016 at 11:24 PM, Guptesh Kumar <span dir="ltr"><<a href="mailto:guptesh.cg4@gmail.com" target="_blank">guptesh.cg4@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div><div><div><div><div>Hi Ricardo,<br><br></div>I am facing a similar issue -<br><br></div>Node 1 : Master pgpool, Master Postgresql<br></div>Node 2 : Slave Pgpool, Slave Postgresql<br><br></div>When I pull out the power cable of master node(Node 1), the Slave watchdog promotes itself to Master correctly. But the pgpool in NOW master node(Node 2) neither disconnects the Node 1 Postgresql from the pool nor it promotes the Node 2 postgresql as Master(failover does not happen). All my DB connection hangs.<br><br></div><div>I tried various values for health_check parameters but none of them helped.<br><br></div>Softwares Used are -<br>pgpool-II-pg94-3.5.2-1pgdg.rhel7.x86_64<br>postgresql94-server-9.4.7-1PGDG.rhel7.x86_64<br><br></div>Thanks & Regards,<br></div>Guptesh<br><div><div><div><br></div></div></div></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div>On Tue, May 10, 2016 at 11:36 PM, Ricardo Larrañaga <span dir="ltr"><<a href="mailto:ricardo.larranaga@gmail.com" target="_blank">ricardo.larranaga@gmail.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div><div dir="ltr"><div><div><div><div>Hi guys:<br></div>I have been working on this a little bit, and i encounter this problem (running pgpool 3.5.2 on a 2 node cluster with streaming replication, if i shut down the interface of the slave, pgpool on the master hangs the connections). And i realize that the problem only happens when i shut down the interface. If i either shut down the processes or use the firewall to block backend communication, pgpool reacts properly.<br><br></div>Does anyone have experience with this or have a 2 node cluster with streaming replication to test it. I am trying to identify what makes the setup hang, since shutting down the interface is the best way i have to simulate a node crash. When i shut down the interface, both postgres and pgpool loose connection, but this is a valid scenario that could happen in real life (when the whole server crashes).<br></div>Thanks a lot.<br></div>Regards<br><br></div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, May 5, 2016 at 4:55 PM, Ricardo Larrañaga <span dir="ltr"><<a href="mailto:ricardo.larranaga@gmail.com" target="_blank">ricardo.larranaga@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div>Hello guys:<br></div>I am running pgpool 3.5.2 in a 2 node cluster with postgresql-9.5 in master-slave and streaming replication mode.<br><br></div>I have been testing failover and failback for a while. While doing failover by shutting down the processes, everything looks fine.<br><br><br></div><div>The one test that is failing is when i shutdown the interface of one node.<br></div><div>Right now i am shutting down the interface of the slave node (both pgool slave and postgresql slave).<br><br></div><div>The problem i am running into is that after doing this, all connections to my database (through pgpool) hang. i am testing it with psql, and psql just hangs and does not give me an output. When i bring the interface of the slave back up, and try to connect to psql again, it looks like pgpool never marked the postgresql node as disconnected. <br><br></div><div>I tried both with and without health check, and also tried with different values of health_check_timeout. my connect timeout value is default (10 seconds).<br><br></div><div>Any one has encountered this issue? I just dont see pgpool attempting to do any failover. Pgpool is still running though, i can see the logs still comming. I just never see an error.<br><br></div><div>I am NOT using interface monitoring, and i would prefer not to use it.<br><br></div><div>Any pointers on how could i troubleshoot this?<br></div><div>Thanks.<br></div><div>Regards<br><br></div></div>
</blockquote></div><br></div>
</div></div><br></div></div>_______________________________________________<br>
pgpool-general mailing list<br>
<a href="mailto:pgpool-general@pgpool.net" target="_blank">pgpool-general@pgpool.net</a><br>
<a href="http://www.pgpool.net/mailman/listinfo/pgpool-general" rel="noreferrer" target="_blank">http://www.pgpool.net/mailman/listinfo/pgpool-general</a><br>
<br></blockquote></div><br></div>
<br>_______________________________________________<br>
pgpool-general mailing list<br>
<a href="mailto:pgpool-general@pgpool.net" target="_blank">pgpool-general@pgpool.net</a><br>
<a href="http://www.pgpool.net/mailman/listinfo/pgpool-general" rel="noreferrer" target="_blank">http://www.pgpool.net/mailman/listinfo/pgpool-general</a><br>
<br></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>