<div dir="ltr"><div>Hi,</div><div><br></div><div>I've identified that worker child process keeps failing when</div><div>there's no primary backend in the cluster at the moment.</div><div><br></div><div><div>We're running a streaming replication cluster with 3 nodes. Because of</div>
<div>a sudden H/W problem of the master server, I detached the node from</div><div>pgpool using `pcp_detach_node` command, in order to temporarily make</div><div>the cluster read-only. pgpool then worked as expected, it rejected</div>
<div>write requests and still served read requests using the remaining two</div><div>standby nodes. The problem is, the worker child process which checks</div><div>for replication lag, keeps failing and being recreated without a</div>
<div>pause. So naturally it generates massive amount of log messages, and</div><div>keeps creating backend connections, which in turn leaves a large</div><div>number of TIME_WAIT sockets on the system.</div></div><div><br>
</div><div>2014-03-27 15:33:57 LOG: pid: 21996 fork a new worker child pid 28453</div><div>2014-03-27 15:33:57 ERROR: pid: 28453 do_query: error message from backend: recovery is in progress. Exit this session.</div><div>
2014-03-27 15:33:57 LOG: pid: 21996 worker child 28453 exits with status 256</div><div>2014-03-27 15:33:57 LOG: pid: 21996 fork a new worker child pid 28455</div><div>2014-03-27 15:33:57 ERROR: pid: 28455 do_query: error message from backend: recovery is in progress. Exit this session.</div>
<div>2014-03-27 15:33:57 LOG: pid: 21996 worker child 28455 exits with status 256</div><div>2014-03-27 15:33:57 LOG: pid: 21996 fork a new worker child pid 28459</div><div>2014-03-27 15:33:57 ERROR: pid: 28459 do_query: error message from backend: recovery is in progress. Exit this session.</div>
<div>2014-03-27 15:33:57 LOG: pid: 21996 worker child 28459 exits with status 256</div><div>...</div><div><br></div><div>Looking at the code, I found that the cause is that it invokes</div><div>`pg_current_xlog_location()` on the first remaining standby node, when</div>
<div>it shouldn't. In fact, there's no reason to check for replication lag</div><div>in such case.</div><div><br></div><div>I've attached a simple patch, which makes worker child to skip checking</div><div>when there's no primary node.</div>
<div><br></div><div>Please take a look.</div><div><br></div><div>Thanks.</div><div>Junegunn Choi.</div></div>