<div dir="ltr">Sorry, here is the pgpool.conf from the master node<div><br></div><div>So I delete >/dev/null from the script and here is the result:</div><div><br></div><div> + FAILED_NODE_ID=0<br> + FAILED_NODE_HOST=master<br> + FAILED_NODE_PORT=5432<br> + FAILED_NODE_PGDATA=/var/lib/pgsql/11/data<br> + NEW_MASTER_NODE_ID=1<br> + OLD_MASTER_NODE_ID=0<br> + NEW_MASTER_NODE_HOST=slave<br> + OLD_PRIMARY_NODE_ID=0<br> + NEW_MASTER_NODE_PORT=5432<br> + NEW_MASTER_NODE_PGDATA=/var/lib/pgsql/11/data<br> + PGHOME=/usr/pgsql-11<br> + ARCHIVEDIR=/var/lib/pgsql/archivedir<br> + REPL_USER=repl<br> + PCP_USER=pgpool<br> + PGPOOL_PATH=/usr/bin<br> + PCP_PORT=9898<br> + logger -i -p <a href="http://local1.info">local1.info</a> follow_master.sh: start: pg_basebackup for 0<br> + ssh -T -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null postgres@master /usr/pgsql-11/bin/pg_ctl -w -D /var/lib/pgsql/11/data status<br> Warning: Permanently added 'master,192.168.56.110' (ECDSA) to the list of known hosts.<br> Permission denied, please try again.<br> Permission denied, please try again.<br> Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).<br> + [[ 255 -eq 0 ]]<br> + logger -i -p <a href="http://local1.info">local1.info</a> follow_master.sh: failed_nod_id=0 is not running. skipping follow master command.<br>follow_master.sh: failed_nod_id=0 is not running. skipping follow master command.<br> + exit 0<br> [192-1] 2019-07-29 13:55:02: pid 2504: LOG: execute command: /etc/pgpool-II/follow_master.sh 2 reserve 5432 /var/lib/pgsql/11/data 1 0 slave 0 5432 /var/lib/pgsql/11/data<br> follow_master.sh: start: pg_basebackup for 2<br> + FAILED_NODE_ID=2<br> + FAILED_NODE_HOST=reserve<br> + FAILED_NODE_PORT=5432<br> + FAILED_NODE_PGDATA=/var/lib/pgsql/11/data<br> + NEW_MASTER_NODE_ID=1<br> + OLD_MASTER_NODE_ID=0<br> + NEW_MASTER_NODE_HOST=slave<br> + OLD_PRIMARY_NODE_ID=0<br> + NEW_MASTER_NODE_PORT=5432<br> + NEW_MASTER_NODE_PGDATA=/var/lib/pgsql/11/data<br> + PGHOME=/usr/pgsql-11<br> + ARCHIVEDIR=/var/lib/pgsql/archivedir<br> + REPL_USER=repl<br> + PCP_USER=pgpool<br> + PGPOOL_PATH=/usr/bin<br> + PCP_PORT=9898<br> + logger -i -p <a href="http://local1.info">local1.info</a> follow_master.sh: start: pg_basebackup for 2<br> + ssh -T -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null postgres@reserve /usr/pgsql-11/bin/pg_ctl -w -D /var/lib/pgsql/11/data status<br> Warning: Permanently added 'reserve,192.168.56.112' (ECDSA) to the list of known hosts.<br> Permission denied, please try again.<br> Permission denied, please try again.<br> Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).<br> + [[ 255 -eq 0 ]]<br> + logger -i -p <a href="http://local1.info">local1.info</a> follow_master.sh: failed_nod_id=2 is not running. skipping follow master command.<br> slave root[2550]: follow_master.sh: failed_nod_id=2 is not running. skipping follow master command.<br> + exit 0<br></div><div><br></div><div>I'm starting to think that there some problem with ssh connection, but not sure</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">вс, 28 июл. 2019 г. в 03:58, Tatsuo Ishii <<a href="mailto:ishii@sraoss.co.jp">ishii@sraoss.co.jp</a>>:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">I noticed followings in the log files:<br>
<br>
/home/t-ishii/slave log.txt:Jul 25 22:30:53 reserve root[2011]: follow_master.sh: failed_nod_id=1 is not running. skipping follow master command.<br>
/home/t-ishii/slave log.txt:Jul 25 22:30:53 reserve root[2019]: follow_master.sh: failed_nod_id=2 is not running. skipping follow master command.<br>
<br>
I don't know which is node 1 and 2 (because you didn't share<br>
pgpool.conf) , but I don't think two nodes were skipped by follow<br>
master command was normal because you have only 3 nodes and just one<br>
of 3 is already down.<br>
<br>
I suspect following code in follow_master.sh did not succeed:<br>
<br>
ssh -T -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \<br>
postgres@${FAILED_NODE_HOST} ${PGHOME}/bin/pg_ctl -w -D ${FAILED_NODE_PGDATA} status >/dev/null 2>&1<br>
<br>
You would want to remove ">/dev/null" to see what was going on there.<br>
<br>
Best regards,<br>
--<br>
Tatsuo Ishii<br>
SRA OSS, Inc. Japan<br>
English: <a href="http://www.sraoss.co.jp/index_en.php" rel="noreferrer" target="_blank">http://www.sraoss.co.jp/index_en.php</a><br>
Japanese:<a href="http://www.sraoss.co.jp" rel="noreferrer" target="_blank">http://www.sraoss.co.jp</a><br>
<br>
> "slave" -primary<br>
> "master" and "reserve"- standby<br>
> After I shut down "slave", "master" became primary, but "reserve" got<br>
> status down. Configs are same from the documentation (changed just<br>
> hostnames and ip's). Failover config is the same also<br>
> <br>
> пт, 26 июл. 2019 г. в 12:54, Tatsuo Ishii <<a href="mailto:ishii@sraoss.co.jp" target="_blank">ishii@sraoss.co.jp</a>>:<br>
> <br>
>> Hi,<br>
>><br>
>> Yes, please provide log and config files.<br>
>><br>
>> My intuition is that there's something wrong with the follow master<br>
>> command script or related settings (especially ssh), because the<br>
>> script shutdowns standby server to resync with new primary database<br>
>> server.<br>
>><br>
>> Best regards,<br>
>> --<br>
>> Tatsuo Ishii<br>
>> SRA OSS, Inc. Japan<br>
>> English: <a href="http://www.sraoss.co.jp/index_en.php" rel="noreferrer" target="_blank">http://www.sraoss.co.jp/index_en.php</a><br>
>> Japanese:<a href="http://www.sraoss.co.jp" rel="noreferrer" target="_blank">http://www.sraoss.co.jp</a><br>
>><br>
>> > Гиа Хурцилава <<a href="mailto:khurtsilava.g@gmail.com" target="_blank">khurtsilava.g@gmail.com</a>><br>
>> > чт, 25 июл., 13:56 (21 час назад)<br>
>> > кому: pgpool-general<br>
>> ><br>
>> > Hi there.<br>
>> ><br>
>> > I’ve got 3 machines with pgpool-4.0.5 and postgresql-11. I have done<br>
>> > configuration for pgpool from the official documentations (<br>
>> > <a href="http://www.pgpool.net/docs/latest/en/html/example-cluster.html" rel="noreferrer" target="_blank">http://www.pgpool.net/docs/latest/en/html/example-cluster.html</a>) and<br>
>> > everything works fine, except 1 thing. When I’m shutting down master<br>
>> node,<br>
>> > one of the slaves is correctly promoted, and another one is going down<br>
>> with<br>
>> > master. Just like that:<br>
>> ><br>
>> > node_id | hostname | port | status | lb_weight | role | select_cnt |<br>
>> > load_balance_node | replication_delay | last_status_change<br>
>> ><br>
>> ><br>
>> ---------+----------+------+--------+-----------+---------+------------+-------------------+-------------------+---------------------<br>
>> ><br>
>> > 0 | master | 5432 | down | 0.333333 | standby | 0 |<br>
>> > false | 0 | 2019-07-25 13:49:22<br>
>> ><br>
>> > 1 | slave | 5432 | up | 0.333333 | primary | 0<br>
>> > | true | 0 | 2019-07-25 13:49:22<br>
>> ><br>
>> > 2 | reserve | 5432 | down | 0.333333 | standby | 0 |<br>
>> > false | 0 | 2019-07-25 13:49:22<br>
>> ><br>
>> ><br>
>> ><br>
>> > What reason can be of this behavior? How can I fix it?<br>
>> ><br>
>> > If you’ll need logs or config files-let me know. Thanks.<br>
>><br>
</blockquote></div>