<html><head></head><body><div class="ydp46944ba6yahoo-style-wrap" style="font-family:Helvetica Neue, Helvetica, Arial, sans-serif;font-size:16px;"><div><div>Strange, i don't see any reason in the config you sent.</div><div><br></div><div>But it looks like it stops always at 03 and 33, so every 2 times per hour. Isn't there a cron that stop it every 30 min ?</div><div><br></div><div>Maybe you can increase the log verbosity ?</div><div><br></div><div class="ydp46944ba6signature">Pierre</div></div>
<div><br></div><div><br></div>
</div><div id="ydp83016aa2yahoo_quoted_6000283494" class="ydp83016aa2yahoo_quoted">
<div style="font-family:'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:13px;color:#26282a;">
<div>
On Friday, April 12, 2019, 9:10:50 PM GMT+2, Rob Reinhardt <rreinhardt@eitccorp.com> wrote:
</div>
<div><br></div>
<div><br></div>
<div><div id="ydp83016aa2yiv5207885332"><div><div dir="ltr"><div dir="ltr">Thank you Pierre. That fixed that problem.<div><br clear="none"></div><div>I have another though. It will not stay running. Just sitting there doing nothing or just monitoring the node status every 10 seconds, it claims to the a stop request and shuts itself down.</div><div><br clear="none"></div><div>I'm on Redhat 7.5 if that helps.</div><div><br clear="none"></div><div>----</div><div>deleted the stuff in between start and automatic? stops</div><div><div><br clear="none"></div><div>Apr 12 17:37:31 pgpool[6291]: [6-1] 2019-04-12 17:37:31: pid 6291: LOG: pgpool-II successfully started. version 4.0.4 (torokiboshi)</div><div>Apr 12 18:03:10 pgpool[9426]: [1-1] 2019-04-12 18:03:10: pid 9426: LOG: stop request sent to pgpool. waiting for termination...</div><div>~26 minutes it stopped itself?</div><div><br clear="none"></div><div>Apr 12 18:10:42 pgpool[10288]: [6-1] 2019-04-12 18:10:42: pid 10288: LOG: pgpool-II successfully started. version 4.0.4 (torokiboshi)</div><div>Apr 12 18:33:10 pgpool[13333]: [1-1] 2019-04-12 18:33:10: pid 13333: LOG: stop request sent to pgpool. waiting for termination...</div><div>~23 minutes it stopped itself?</div><div><br clear="none"></div><div>@trying this time without the 10second watch monitor to see if that changes anything...</div><div>Apr 12 18:42:34 pgpool[14349]: [6-1] 2019-04-12 18:42:34: pid 14349: LOG: pgpool-II successfully started. version 4.0.4 (torokiboshi)</div><div>Apr 12 19:03:10 pgpool[16505]: [1-1] 2019-04-12 19:03:10: pid 16505: LOG: stop request sent to pgpool. waiting for termination...</div><div>~21 mins?</div></div><div><br clear="none"></div><div>I feel like I'm in the movie Independence Day and there is a countdown to something even less nice.</div><div><br clear="none"></div></div></div><br clear="none"><div class="ydp83016aa2yiv5207885332yqt6758387926" id="ydp83016aa2yiv5207885332yqt90395"><div class="ydp83016aa2yiv5207885332gmail_quote"><div class="ydp83016aa2yiv5207885332gmail_attr" dir="ltr">On Fri, Apr 12, 2019 at 1:09 PM Pierre Timmermans <<a shape="rect" href="mailto:ptim007@yahoo.com" rel="nofollow" target="_blank">ptim007@yahoo.com</a>> wrote:<br clear="none"></div><blockquote class="ydp83016aa2yiv5207885332gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex;"><div><div class="ydp83016aa2yiv5207885332gmail-m_6352526754335693224ydp63bef53byahoo-style-wrap"><div><div>Hello,</div><div><br clear="none"></div><div>In your config you have</div><div><br clear="none"></div><div><span>backend_hostname0 = '192.x.y.a'<br clear="none"><br clear="none"></span><div>3 times, it should be once backend_hostname0, backend_hostname1 and backend_hostname2 (the same for backend_port0, etc)<br clear="none"></div><div><br clear="none"></div></div><div class="ydp83016aa2yiv5207885332gmail-m_6352526754335693224ydp63bef53bsignature">Pierre</div></div>
<div><br clear="none"></div><div><br clear="none"></div>
</div><div class="ydp83016aa2yiv5207885332gmail-m_6352526754335693224ydp4fb3b377yahoo_quoted" id="ydp83016aa2yiv5207885332gmail-m_6352526754335693224ydp4fb3b377yahoo_quoted_5116818638">
<div>
<div>
On Friday, April 12, 2019, 5:05:05 PM GMT+2, Rob Reinhardt <<a shape="rect" href="mailto:rreinhardt@eitccorp.com" rel="nofollow" target="_blank">rreinhardt@eitccorp.com</a>> wrote:
</div>
<div><br clear="none"></div>
<div><br clear="none"></div>
<div><div id="ydp83016aa2yiv5207885332gmail-m_6352526754335693224ydp4fb3b377yiv5251187594"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div>I "feel like" it should be working since so much of it is working, except the main function of the s/w seems to be failing me.</div><div><br clear="none"></div><div>my repmgr95 says this:</div><div><br clear="none"></div><div><div>ID | Name | Role | Status | Upstream | Location | Connection string</div><div>----+---------+---------+-----------+----------+----------+----------------------------------------------------------</div><div>1 | r01sv05 | standby | running | r01sv04 | default | host=r01sv05 user=repmgr dbname=repmgr connect_timeout=2</div><div>2 | r01sv04 | primary | * running | | default | host=r01sv04 user=repmgr dbname=repmgr connect_timeout=2</div><div>3 | r01sv03 | standby | running | r01sv04 | default | host=r01sv03 user=repmgr dbname=repmgr connect_timeout=2</div></div><div><br clear="none"></div><div>(actually 05 is now the primary, that is an old shot)</div><div><br clear="none"></div><div>r01sv02 is the pgpool server btw, and they are all on the same subnet.</div><div><br clear="none"></div><div>my pgpool says this:</div><div><br clear="none"></div><div><div>-bash-4.2$ psql -U pgpool --dbname=pgpool --host r01sv02 -c "show pool_nodes"</div><div> node_id | hostname | port | status | lb_weight | role | select_cnt | load_balance_node | replication_delay | last_status_change </div><div>---------+----------+------+--------+-----------+---------+------------+-------------------+-------------------+---------------------</div><div> 0 | r01sv03 | 5432 | up | 1.000000 | standby | 0 | true | 0 | 2019-04-11 19:48:43</div><div>(1 row)</div></div><div><br clear="none"></div><div>pgpool keeps logging this:</div><div><br clear="none"></div><div><div>Apr 12 14:03:03 <a shape="rect" href="http://r01sv02.change.me" rel="nofollow" target="_blank">r01sv02.change.me</a> pgpool[14630]: [259-1] 2019-04-12 14:03:03: pid 14630: LOG: find_primary_node: standby node is 0</div><div>Apr 12 14:03:03 <a shape="rect" href="http://r01sv02.change.me" rel="nofollow" target="_blank">r01sv02.change.me</a> pgpool[14630]: [259-2] 2019-04-12 14:03:03: pid 14630: LOCATION: pgpool_main.c:3438</div><div>Apr 12 14:03:04 <a shape="rect" href="http://r01sv02.change.me" rel="nofollow" target="_blank">r01sv02.change.me</a> pgpool[14630]: [260-1] 2019-04-12 14:03:04: pid 14630: LOG: find_primary_node: standby node is 0</div><div>Apr 12 14:03:04 <a shape="rect" href="http://r01sv02.change.me" rel="nofollow" target="_blank">r01sv02.change.me</a> pgpool[14630]: [260-2] 2019-04-12 14:03:04: pid 14630: LOCATION: pgpool_main.c:3438</div><div>Apr 12 14:03:05 <a shape="rect" href="http://r01sv02.change.me" rel="nofollow" target="_blank">r01sv02.change.me</a> pgpool[14630]: [261-1] 2019-04-12 14:03:05: pid 14630: LOG: find_primary_node: standby node is 0</div></div><div>and occasionally the find_primary_node_repeatedly line</div><div><br clear="none"></div><div>Quick summary of my setup:</div><div>3 postgresql-9.5 db nodes, one is primary, the other two are standby, in a streaming replication cluster built and managed with repmgr95. This is working fine.</div><div><br clear="none"></div><div>1 pgpool 4.0.4 server that has the same version of postgresql-9.5 and postgres user setup as the other 3.</div><div>- pgpool is running as postgres</div><div><br clear="none"></div><div>what does work:</div><div>-the postgres user has ssh access to/from any of the four servers. I can remotely run repmgr from the pgpool server as postgres user with no problem</div>-psql can access all the db's says with simple \list or \dt or whatever from any of the 4 nodes asking for 5432 access from any of the four nodes, even from the pgpool server</div><div>-i can use the postgres user or pgpool user with psql</div><div>- dns is working too, but I changed from using the hostname to the IP's in the config file in case it made a difference, but it did not.</div><div><br clear="none"></div><div>I've even run this commands by hand and it gets the right answers:</div><div><br clear="none"></div><div><div>-bash-4.2$ psql -U pgpool --dbname=pgpool --host r01sv02 -c "SELECT pg_is_in_recovery();"</div><div> pg_is_in_recovery </div><div>-------------------</div><div> t</div><div>(1 row)</div><div><br clear="none"></div><div>-bash-4.2$ psql -U pgpool --dbname=pgpool --host r01sv03 -c "SELECT pg_is_in_recovery();"</div><div> pg_is_in_recovery </div><div>-------------------</div><div> t</div><div>(1 row)</div><div><br clear="none"></div><div>-bash-4.2$ psql -U pgpool --dbname=pgpool --host r01sv04 -c "SELECT pg_is_in_recovery();"</div><div> pg_is_in_recovery </div><div>-------------------</div><div> t</div><div>(1 row)</div><div><br clear="none"></div><div>-bash-4.2$ psql -U pgpool --dbname=pgpool --host r01sv05 -c "SELECT pg_is_in_recovery();"</div><div> pg_is_in_recovery </div><div>-------------------</div><div> f</div><div>(1 row)</div></div><div><br clear="none"></div><div>pgpool for some reason finds one of the three nodes, a standby node, and it has the right.</div><div><br clear="none"></div><div>the pgpool database I created, I created on my primary. I had thought that when pgpool started up it might put some stuff in that database, but I haven't seen anything, in case that is the problem. i found notes on creating said database and user, but have seen nothing on actually putting anything in it by hand.--anyway, I was just looking at that in case it is something</div><div><br clear="none"></div><div>Main question -- where are the other two nodes?</div><div><br clear="none"></div><div>Also, I've noted that each time I start pgpool, it throws those errors (above) until the steps reaches 300, then it finally says "successfully started" and at that point the pcp_* commands will work, before then it has not yet created the pcp socket. Don't know if that is normal/expected or not. Seemed odd to me, for basic commands to take 5 minutes to even be available.</div><div><br clear="none"></div><div>The other thing is that while it will come up for a while, pgpool seems to be stopping itself after about 10 minutes or so. the log just says that pgpool was told to stop (but I didn't do it).</div><div><br clear="none"></div><div>I've attached a sanitized version of my pgpool.conf file</div><div dir="ltr"><br clear="none"></div><div>In case it helps, here also is the sanitized contents of the .pgpass and .pcppass files in the postgres home dir of all four of my servers and the pool_passwd, in case you see a problem with these (they are 600 owned by postgres).</div><div><br clear="none"></div><div><div>-bash-4.2$ cat .pgpass</div><div>r01sv02:5432:*:pgpool:sanitized</div><div>r01sv05:5432:*:postgres:pgpool:sanitized</div><div>r01sv04:5432:*:postgres:pgpool:sanitized</div><div>r01sv03:5432:*:postgres:pgpool:sanitized</div><div>r01sv05:5432:replication:repmgr:pgpool:sanitized</div><div>r01sv04:5432:replication:repmgr:pgpool:sanitized</div><div>r01sv03:5432:replication:repmgr:pgpool:sanitized</div></div><div><br clear="none"></div><div><div>-bash-4.2$ cat .pcppass</div><div>*:*:pgpool:pgpool:sanitized</div><div>*:*:postgres:pgpool:sanitized</div></div><div><br clear="none"></div><div>pcp.conf</div><div><div>pgpool:sanitized</div><div>nrpe:sanitized</div><div>postgres:sanitized</div></div><div><br clear="none"></div><div>pool_passwd</div><div><div>pgpool:sanitized</div><div>nrpe:sanitized</div><div>postgres:sanitized</div></div><div><br clear="none"></div><div><br clear="none"></div><div><div>-bash-4.2$ cat pool_hba.conf</div><div># pgpool Client Authentication Configuration File</div><div><br clear="none"></div><div># "local" is for Unix domain socket connections only</div><div>local all all trust</div><div># IPv4 local connections:</div><div>host all all <a shape="rect" href="http://127.0.0.1/32" rel="nofollow" target="_blank">127.0.0.1/32</a> trust</div><div>host all all ::1/128 trust</div><div>host all all 192.x.y.0/24 md5</div></div><div><br clear="none"></div><div>Thanks,</div><div>Rob</div><div dir="ltr"><br clear="none"></div><div dir="ltr"><br clear="none"></div><div dir="ltr"><br clear="none"></div><div dir="ltr"><br clear="none"></div><div dir="ltr"><br clear="all"><div><br clear="none"></div></div></div></div></div></div></div></div></div></div></div></div>_______________________________________________<br clear="none">pgpool-general mailing list<br clear="none"><a shape="rect" href="mailto:pgpool-general@pgpool.net" rel="nofollow" target="_blank">pgpool-general@pgpool.net</a><br clear="none"><a shape="rect" href="http://www.pgpool.net/mailman/listinfo/pgpool-general" rel="nofollow" target="_blank">http://www.pgpool.net/mailman/listinfo/pgpool-general</a><br clear="none"></div>
</div>
</div></div></blockquote></div></div><br clear="all"><div><br clear="none"></div>-- <br clear="none"><div class="ydp83016aa2yiv5207885332gmail_signature" dir="ltr"><div dir="ltr"><div><div dir="ltr"><div dir="ltr">Rob Reinhardt<div>DevOps Engineer</div><div><span style="font-family:arial, helvetica, sans-serif;">Enlighten IT Consulting (EITC), a MacAulay-Brown, Inc. company</span><br clear="none"></div></div></div></div></div></div></div></div></div>
</div>
</div></body></html>