Sep 1 11:09:38 centos8i3 pgpool[11424]: [1-1] pid 11424: LOG: Backend status file /var/log/pgpool-II/pgpool_status discarded Sep 1 11:09:38 centos8i3 pgpool[11424]: [2-1] pid 11424: LOG: memory cache initialized Sep 1 11:09:38 centos8i3 pgpool[11424]: [2-2] pid 11424: DETAIL: memcache blocks :64 Sep 1 11:09:38 centos8i3 pgpool[11424]: [3-1] pid 11424: LOG: pool_discard_oid_maps: discarded memqcache oid maps Sep 1 11:09:38 centos8i3 pgpool[11424]: [4-1] pid 11424: LOG: waiting for watchdog to initialize Sep 1 11:09:38 centos8i3 pgpool[11426]: [4-1] pid 11426: LOG: setting the local watchdog node name to "centos8i3-int:5432 Linux centos8i3.localdomain" Sep 1 11:09:38 centos8i3 pgpool[11426]: [5-1] pid 11426: LOG: watchdog cluster is configured with 2 remote nodes Sep 1 11:09:38 centos8i3 pgpool[11426]: [6-1] pid 11426: LOG: watchdog remote node:0 on centos8i1-int:9000 Sep 1 11:09:38 centos8i3 pgpool[11426]: [7-1] pid 11426: LOG: watchdog remote node:1 on centos8i2-int:9000 Sep 1 11:09:38 centos8i3 pgpool[11426]: [8-1] pid 11426: LOG: interface monitoring is disabled in watchdog Sep 1 11:09:38 centos8i3 pgpool[11426]: [9-1] pid 11426: LOG: watchdog node state changed from [DEAD] to [LOADING] Sep 1 11:09:38 centos8i3 pgpool[11426]: [10-1] pid 11426: LOG: new outbound connection to centos8i1-int:9000 Sep 1 11:09:38 centos8i3 pgpool[11426]: [11-1] pid 11426: LOG: new watchdog node connection is received from "10.0.0.1:48781" Sep 1 11:09:38 centos8i3 pgpool[11426]: [12-1] pid 11426: LOG: new node joined the cluster hostname:"centos8i1-int" port:9000 pgpool_port:5432 Sep 1 11:09:38 centos8i3 pgpool[11426]: [12-2] pid 11426: DETAIL: Pgpool-II version:"4.1.2" watchdog messaging version: 1.1 Sep 1 11:09:38 centos8i3 pgpool[11426]: [13-1] pid 11426: LOG: new watchdog node connection is received from "10.0.0.2:55443" Sep 1 11:09:38 centos8i3 pgpool[11426]: [14-1] pid 11426: LOG: new node joined the cluster hostname:"centos8i2-int" port:9000 pgpool_port:5432 Sep 1 11:09:38 centos8i3 pgpool[11426]: [14-2] pid 11426: DETAIL: Pgpool-II version:"4.1.2" watchdog messaging version: 1.1 Sep 1 11:09:43 centos8i3 pgpool[11426]: [15-1] pid 11426: LOG: watchdog node state changed from [LOADING] to [INITIALIZING] Sep 1 11:09:43 centos8i3 pgpool[11426]: [16-1] pid 11426: LOG: watchdog node state changed from [INITIALIZING] to [STANDING FOR MASTER] Sep 1 11:09:43 centos8i3 pgpool[11426]: [17-1] pid 11426: LOG: our stand for coordinator request is rejected by node "centos8i1-int:5432 Linux centos8i1.localdomain" Sep 1 11:09:43 centos8i3 pgpool[11426]: [18-1] pid 11426: LOG: watchdog node state changed from [STANDING FOR MASTER] to [PARTICIPATING IN ELECTION] Sep 1 11:09:48 centos8i3 pgpool[11426]: [19-1] pid 11426: LOG: watchdog node state changed from [PARTICIPATING IN ELECTION] to [JOINING] Sep 1 11:09:48 centos8i3 pgpool[11426]: [20-1] pid 11426: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] Sep 1 11:09:49 centos8i3 pgpool[11426]: [21-1] pid 11426: LOG: watchdog node state changed from [INITIALIZING] to [STANDING FOR MASTER] Sep 1 11:09:49 centos8i3 pgpool[11426]: [22-1] pid 11426: LOG: new outbound connection to centos8i2-int:9000 Sep 1 11:09:49 centos8i3 pgpool[11426]: [23-1] pid 11426: LOG: our stand for coordinator request is rejected by node "centos8i1-int:5432 Linux centos8i1.localdomain" Sep 1 11:09:49 centos8i3 pgpool[11426]: [24-1] pid 11426: LOG: watchdog node state changed from [STANDING FOR MASTER] to [PARTICIPATING IN ELECTION] Sep 1 11:09:54 centos8i3 pgpool[11426]: [25-1] pid 11426: LOG: watchdog node state changed from [PARTICIPATING IN ELECTION] to [JOINING] Sep 1 11:09:54 centos8i3 pgpool[11426]: [26-1] pid 11426: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] Sep 1 11:09:55 centos8i3 pgpool[11426]: [27-1] pid 11426: LOG: watchdog node state changed from [INITIALIZING] to [STANDING FOR MASTER] Sep 1 11:09:55 centos8i3 pgpool[11426]: [28-1] pid 11426: LOG: our stand for coordinator request is rejected by node "centos8i1-int:5432 Linux centos8i1.localdomain" Sep 1 11:09:55 centos8i3 pgpool[11426]: [29-1] pid 11426: LOG: watchdog node state changed from [STANDING FOR MASTER] to [PARTICIPATING IN ELECTION] Sep 1 11:10:00 centos8i3 pgpool[11426]: [30-1] pid 11426: LOG: watchdog node state changed from [PARTICIPATING IN ELECTION] to [JOINING] Sep 1 11:10:00 centos8i3 pgpool[11426]: [31-1] pid 11426: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] Sep 1 11:10:01 centos8i3 pgpool[11426]: [32-1] pid 11426: LOG: watchdog node state changed from [INITIALIZING] to [STANDING FOR MASTER] Sep 1 11:10:01 centos8i3 pgpool[11426]: [33-1] pid 11426: LOG: our stand for coordinator request is rejected by node "centos8i2-int:5432 Linux centos8i2.localdomain" Sep 1 11:10:01 centos8i3 pgpool[11426]: [34-1] pid 11426: LOG: watchdog node state changed from [STANDING FOR MASTER] to [PARTICIPATING IN ELECTION] Sep 1 11:10:05 centos8i3 pgpool[11426]: [35-1] pid 11426: LOG: watchdog node state changed from [PARTICIPATING IN ELECTION] to [INITIALIZING] Sep 1 11:10:05 centos8i3 pgpool[11426]: [36-1] pid 11426: LOG: setting the remote node "centos8i2-int:5432 Linux centos8i2.localdomain" as watchdog cluster master Sep 1 11:10:06 centos8i3 pgpool[11426]: [37-1] pid 11426: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] Sep 1 11:10:06 centos8i3 pgpool[11426]: [38-1] pid 11426: LOG: successfully joined the watchdog cluster as standby node Sep 1 11:10:06 centos8i3 pgpool[11426]: [38-2] pid 11426: DETAIL: our join coordinator request is accepted by cluster leader node "centos8i2-int:5432 Linux centos8i2.localdomain" Sep 1 11:10:06 centos8i3 pgpool[11424]: [5-1] pid 11424: LOG: watchdog process is initialized Sep 1 11:10:06 centos8i3 pgpool[11424]: [5-2] pid 11424: DETAIL: watchdog messaging data version: 1.1 Sep 1 11:10:06 centos8i3 pgpool[11426]: [39-1] pid 11426: LOG: new IPC connection received Sep 1 11:10:06 centos8i3 pgpool[11426]: [40-1] pid 11426: LOG: new IPC connection received Sep 1 11:10:06 centos8i3 pgpool[11424]: [6-1] pid 11424: LOG: we have joined the watchdog cluster as STANDBY node Sep 1 11:10:06 centos8i3 pgpool[11424]: [6-2] pid 11424: DETAIL: syncing the backend states from the MASTER watchdog node Sep 1 11:10:06 centos8i3 pgpool[11426]: [41-1] pid 11426: LOG: new IPC connection received Sep 1 11:10:06 centos8i3 pgpool[11426]: [42-1] pid 11426: LOG: received the get data request from local pgpool-II on IPC interface Sep 1 11:10:06 centos8i3 pgpool[11426]: [43-1] pid 11426: LOG: get data request from local pgpool-II node received on IPC interface is forwarded to master watchdog node "centos8i2-int:5432 Linux centos8i2.localdomain" Sep 1 11:10:06 centos8i3 pgpool[11426]: [43-2] pid 11426: DETAIL: waiting for the reply... Sep 1 11:10:06 centos8i3 pgpool[11451]: [6-1] pid 11451: LOG: 3 watchdog nodes are configured for lifecheck Sep 1 11:10:06 centos8i3 pgpool[11451]: [7-1] pid 11451: LOG: watchdog nodes ID:0 Name:"centos8i3-int:5432 Linux centos8i3.localdomain" Sep 1 11:10:06 centos8i3 pgpool[11451]: [7-2] pid 11451: DETAIL: Host:"centos8i3-int" WD Port:9000 pgpool-II port:5432 Sep 1 11:10:06 centos8i3 pgpool[11451]: [8-1] pid 11451: LOG: watchdog nodes ID:1 Name:"centos8i1-int:5432 Linux centos8i1.localdomain" Sep 1 11:10:06 centos8i3 pgpool[11451]: [8-2] pid 11451: DETAIL: Host:"centos8i1-int" WD Port:9000 pgpool-II port:5432 Sep 1 11:10:06 centos8i3 pgpool[11451]: [9-1] pid 11451: LOG: watchdog nodes ID:2 Name:"centos8i2-int:5432 Linux centos8i2.localdomain" Sep 1 11:10:06 centos8i3 pgpool[11451]: [9-2] pid 11451: DETAIL: Host:"centos8i2-int" WD Port:9000 pgpool-II port:5432 Sep 1 11:10:06 centos8i3 pgpool[11424]: [7-1] pid 11424: LOG: master watchdog node "centos8i2-int:5432 Linux centos8i2.localdomain" returned status for 3 backend nodes Sep 1 11:10:06 centos8i3 pgpool[11424]: [8-1] pid 11424: LOG: Setting up socket for 0.0.0.0:5432 Sep 1 11:10:06 centos8i3 pgpool[11424]: [9-1] pid 11424: LOG: Setting up socket for :::5432 Sep 1 11:10:06 centos8i3 pgpool[11488]: [10-1] pid 11488: LOG: PCP process: 11488 started Sep 1 11:10:06 centos8i3 pgpool[11424]: [10-1] pid 11424: LOG: pgpool-II successfully started. version 4.1.2 (karasukiboshi) Sep 1 11:10:06 centos8i3 pgpool[11424]: [11-1] pid 11424: LOG: node status[0]: 0 Sep 1 11:10:06 centos8i3 pgpool[11424]: [12-1] pid 11424: LOG: node status[1]: 0 Sep 1 11:10:06 centos8i3 pgpool[11424]: [13-1] pid 11424: LOG: node status[2]: 0 Sep 1 11:10:07 centos8i3 pgpool[11452]: [10-1] pid 11452: LOG: set SO_REUSEPORT option to the socket Sep 1 11:10:07 centos8i3 pgpool[11452]: [11-1] pid 11452: LOG: creating watchdog heartbeat receive socket. Sep 1 11:10:07 centos8i3 pgpool[11452]: [11-2] pid 11452: DETAIL: set SO_REUSEPORT Sep 1 11:10:07 centos8i3 pgpool[11454]: [10-1] pid 11454: LOG: set SO_REUSEPORT option to the socket Sep 1 11:10:07 centos8i3 pgpool[11454]: [11-1] pid 11454: LOG: creating watchdog heartbeat receive socket. Sep 1 11:10:07 centos8i3 pgpool[11454]: [11-2] pid 11454: DETAIL: set SO_REUSEPORT Sep 1 11:10:07 centos8i3 pgpool[11453]: [10-1] pid 11453: LOG: set SO_REUSEPORT option to the socket Sep 1 11:10:07 centos8i3 pgpool[11453]: [11-1] pid 11453: LOG: creating socket for sending heartbeat Sep 1 11:10:07 centos8i3 pgpool[11453]: [11-2] pid 11453: DETAIL: set SO_REUSEPORT Sep 1 11:10:07 centos8i3 pgpool[11455]: [10-1] pid 11455: LOG: set SO_REUSEPORT option to the socket Sep 1 11:10:07 centos8i3 pgpool[11455]: [11-1] pid 11455: LOG: creating socket for sending heartbeat Sep 1 11:10:07 centos8i3 pgpool[11455]: [11-2] pid 11455: DETAIL: set SO_REUSEPORT ***** SHUTDOWN NODE centos8i2-int ***** Sep 1 11:19:37 centos8i3 pgpool[11426]: [44-1] pid 11426: LOG: remote node "centos8i2-int:5432 Linux centos8i2.localdomain" is shutting down Sep 1 11:19:37 centos8i3 pgpool[11426]: [45-1] pid 11426: LOG: watchdog cluster has lost the coordinator node Sep 1 11:19:37 centos8i3 pgpool[11426]: [46-1] pid 11426: LOG: removing the remote node "centos8i2-int:5432 Linux centos8i2.localdomain" from watchdog cluster master Sep 1 11:19:37 centos8i3 pgpool[11426]: [47-1] pid 11426: LOG: We have lost the cluster master node "centos8i2-int:5432 Linux centos8i2.localdomain" Sep 1 11:19:37 centos8i3 pgpool[11426]: [48-1] pid 11426: LOG: watchdog node state changed from [STANDBY] to [JOINING] Sep 1 11:19:37 centos8i3 pgpool[11426]: [49-1] pid 11426: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] Sep 1 11:19:38 centos8i3 pgpool[11426]: [50-1] pid 11426: LOG: watchdog node state changed from [INITIALIZING] to [STANDING FOR MASTER] Sep 1 11:19:38 centos8i3 pgpool[11426]: [51-1] pid 11426: LOG: our stand for coordinator request is rejected by node "centos8i1-int:5432 Linux centos8i1.localdomain" Sep 1 11:19:38 centos8i3 pgpool[11426]: [52-1] pid 11426: LOG: watchdog node state changed from [STANDING FOR MASTER] to [PARTICIPATING IN ELECTION] Sep 1 11:19:39 centos8i3 pgpool[11489]: [10-1] pid 11489: LOG: failed to connect to PostgreSQL server on "centos8i2-int:5433", getsockopt() detected error "Connection refused" Sep 1 11:19:39 centos8i3 pgpool[11489]: [11-1] pid 11489: ERROR: failed to make persistent db connection Sep 1 11:19:39 centos8i3 pgpool[11489]: [11-2] pid 11489: DETAIL: connection to host:"centos8i2-int:5433" failed Sep 1 11:19:43 centos8i3 pgpool[11426]: [53-1] pid 11426: LOG: watchdog node state changed from [PARTICIPATING IN ELECTION] to [JOINING] Sep 1 11:19:43 centos8i3 pgpool[11426]: [54-1] pid 11426: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] Sep 1 11:19:43 centos8i3 pgpool[11491]: [10-1] pid 11491: LOG: failed to connect to PostgreSQL server on "centos8i2-int:5433", timed out Sep 1 11:19:43 centos8i3 pgpool[11491]: [11-1] pid 11491: ERROR: failed to make persistent db connection Sep 1 11:19:43 centos8i3 pgpool[11491]: [11-2] pid 11491: DETAIL: connection to host:"centos8i2-int:5433" failed Sep 1 11:19:43 centos8i3 pgpool[11491]: [12-1] pid 11491: LOG: health check retrying on DB node: 1 (round:1) Sep 1 11:19:44 centos8i3 pgpool[11426]: [55-1] pid 11426: LOG: watchdog node state changed from [INITIALIZING] to [STANDING FOR MASTER] Sep 1 11:19:44 centos8i3 pgpool[11426]: [56-1] pid 11426: LOG: our stand for coordinator request is rejected by node "centos8i1-int:5432 Linux centos8i1.localdomain" Sep 1 11:19:44 centos8i3 pgpool[11426]: [57-1] pid 11426: LOG: watchdog node state changed from [STANDING FOR MASTER] to [PARTICIPATING IN ELECTION] Sep 1 11:19:49 centos8i3 pgpool[11426]: [58-1] pid 11426: LOG: watchdog node state changed from [PARTICIPATING IN ELECTION] to [JOINING] Sep 1 11:19:49 centos8i3 pgpool[11426]: [59-1] pid 11426: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] Sep 1 11:19:49 centos8i3 pgpool[11489]: [12-1] pid 11489: ERROR: Failed to check replication time lag Sep 1 11:19:49 centos8i3 pgpool[11489]: [12-2] pid 11489: DETAIL: No persistent db connection for the node 1 Sep 1 11:19:49 centos8i3 pgpool[11489]: [12-3] pid 11489: HINT: check sr_check_user and sr_check_password Sep 1 11:19:49 centos8i3 pgpool[11489]: [12-4] pid 11489: CONTEXT: while checking replication time lag Sep 1 11:19:50 centos8i3 pgpool[11426]: [60-1] pid 11426: LOG: watchdog node state changed from [INITIALIZING] to [STANDING FOR MASTER] Sep 1 11:19:54 centos8i3 pgpool[11426]: [61-1] pid 11426: LOG: watchdog node state changed from [STANDING FOR MASTER] to [MASTER] Sep 1 11:19:54 centos8i3 pgpool[11426]: [62-1] pid 11426: LOG: I am announcing my self as master/coordinator watchdog node Sep 1 11:19:54 centos8i3 pgpool[11426]: [63-1] pid 11426: LOG: I am the cluster leader node Sep 1 11:19:54 centos8i3 pgpool[11426]: [63-2] pid 11426: DETAIL: our declare coordinator message is accepted by all nodes Sep 1 11:19:54 centos8i3 pgpool[11426]: [64-1] pid 11426: LOG: setting the local node "centos8i3-int:5432 Linux centos8i3.localdomain" as watchdog cluster master Sep 1 11:19:54 centos8i3 pgpool[11426]: [65-1] pid 11426: LOG: I am the cluster leader node but we do not have enough nodes in cluster Sep 1 11:19:54 centos8i3 pgpool[11426]: [65-2] pid 11426: DETAIL: waiting for the quorum to start escalation process Sep 1 11:19:54 centos8i3 pgpool[11424]: [14-1] pid 11424: LOG: Pgpool-II parent process received watchdog quorum change signal from watchdog Sep 1 11:19:54 centos8i3 pgpool[11426]: [66-1] pid 11426: LOG: new IPC connection received Sep 1 11:19:54 centos8i3 pgpool[11426]: [67-1] pid 11426: LOG: new IPC connection received Sep 1 11:19:54 centos8i3 pgpool[11491]: [13-1] pid 11491: LOG: failed to connect to PostgreSQL server on "centos8i2-int:5433", timed out Sep 1 11:19:54 centos8i3 pgpool[11491]: [14-1] pid 11491: ERROR: failed to make persistent db connection Sep 1 11:19:54 centos8i3 pgpool[11491]: [14-2] pid 11491: DETAIL: connection to host:"centos8i2-int:5433" failed Sep 1 11:19:54 centos8i3 pgpool[11491]: [15-1] pid 11491: LOG: health check retrying on DB node: 1 (round:2) Sep 1 11:19:55 centos8i3 pgpool[11426]: [68-1] pid 11426: LOG: adding watchdog node "centos8i1-int:5432 Linux centos8i1.localdomain" to the standby list Sep 1 11:19:55 centos8i3 pgpool[11426]: [69-1] pid 11426: LOG: quorum found Sep 1 11:19:55 centos8i3 pgpool[11426]: [69-2] pid 11426: DETAIL: starting escalation process Sep 1 11:19:55 centos8i3 pgpool[11426]: [70-1] pid 11426: LOG: escalation process started with PID:12032 Sep 1 11:19:55 centos8i3 pgpool[12032]: [70-1] pid 12032: LOG: watchdog: escalation started Sep 1 11:19:55 centos8i3 pgpool[11424]: [15-1] pid 11424: LOG: Pgpool-II parent process received watchdog quorum change signal from watchdog Sep 1 11:19:55 centos8i3 pgpool[11426]: [71-1] pid 11426: LOG: new IPC connection received Sep 1 11:19:55 centos8i3 pgpool[11424]: [16-1] pid 11424: LOG: watchdog cluster now holds the quorum Sep 1 11:19:55 centos8i3 pgpool[11424]: [16-2] pid 11424: DETAIL: updating the state of quarantine backend nodes Sep 1 11:19:55 centos8i3 pgpool[11426]: [72-1] pid 11426: LOG: new IPC connection received Sep 1 11:19:59 centos8i3 pgpool[11489]: [13-1] pid 11489: LOG: trying connecting to PostgreSQL server on "centos8i2-int:5433" by INET socket Sep 1 11:19:59 centos8i3 pgpool[11489]: [13-2] pid 11489: DETAIL: timed out. retrying... Sep 1 11:19:59 centos8i3 pgpool[12032]: [71-1] pid 12032: LOG: successfully acquired the delegate IP:"192.168.56.120" Sep 1 11:19:59 centos8i3 pgpool[12032]: [71-2] pid 12032: DETAIL: 'if_up_cmd' returned with success Sep 1 11:19:59 centos8i3 pgpool[11426]: [73-1] pid 11426: LOG: watchdog escalation process with pid: 12032 exit with SUCCESS. Sep 1 11:20:05 centos8i3 pgpool[11491]: [16-1] pid 11491: LOG: failed to connect to PostgreSQL server on "centos8i2-int:5433", timed out Sep 1 11:20:05 centos8i3 pgpool[11491]: [17-1] pid 11491: ERROR: failed to make persistent db connection Sep 1 11:20:05 centos8i3 pgpool[11491]: [17-2] pid 11491: DETAIL: connection to host:"centos8i2-int:5433" failed Sep 1 11:20:05 centos8i3 pgpool[11491]: [18-1] pid 11491: LOG: health check retrying on DB node: 1 (round:3) Sep 1 11:20:09 centos8i3 pgpool[11489]: [14-1] pid 11489: LOG: trying connecting to PostgreSQL server on "centos8i2-int:5433" by INET socket Sep 1 11:20:09 centos8i3 pgpool[11489]: [14-2] pid 11489: DETAIL: timed out. retrying... Sep 1 11:20:12 centos8i3 pgpool[11426]: [74-1] pid 11426: LOG: watchdog received the failover command from remote pgpool-II node "centos8i1-int:5432 Linux centos8i1.localdomain" Sep 1 11:20:12 centos8i3 pgpool[11426]: [75-1] pid 11426: LOG: watchdog is processing the failover command [DEGENERATE_BACKEND_REQUEST] received from centos8i1-int:5432 Linux centos8i1.localdomain Sep 1 11:20:12 centos8i3 pgpool[11426]: [76-1] pid 11426: LOG: failover requires the majority vote, waiting for consensus Sep 1 11:20:12 centos8i3 pgpool[11426]: [76-2] pid 11426: DETAIL: failover request noted Sep 1 11:20:12 centos8i3 pgpool[11426]: [77-1] pid 11426: LOG: failover command [DEGENERATE_BACKEND_REQUEST] request from pgpool-II node "centos8i1-int:5432 Linux centos8i1.localdomain" is queued, waiting for the confirmation from other nodes Sep 1 11:20:12 centos8i3 pgpool[11424]: [17-1] pid 11424: LOG: Pgpool-II parent process received inform quarantine nodes signal from watchdog Sep 1 11:20:16 centos8i3 pgpool[11491]: [19-1] pid 11491: LOG: failed to connect to PostgreSQL server on "centos8i2-int:5433", timed out Sep 1 11:20:16 centos8i3 pgpool[11491]: [20-1] pid 11491: ERROR: failed to make persistent db connection Sep 1 11:20:16 centos8i3 pgpool[11491]: [20-2] pid 11491: DETAIL: connection to host:"centos8i2-int:5433" failed Sep 1 11:20:16 centos8i3 pgpool[11491]: [21-1] pid 11491: LOG: health check failed on node 1 (timeout:0) Sep 1 11:20:16 centos8i3 pgpool[11491]: [22-1] pid 11491: LOG: received degenerate backend request for node_id: 1 from pid [11491] Sep 1 11:20:16 centos8i3 pgpool[11426]: [78-1] pid 11426: LOG: new IPC connection received Sep 1 11:20:16 centos8i3 pgpool[11426]: [79-1] pid 11426: LOG: watchdog received the failover command from local pgpool-II on IPC interface Sep 1 11:20:16 centos8i3 pgpool[11426]: [80-1] pid 11426: LOG: watchdog is processing the failover command [DEGENERATE_BACKEND_REQUEST] received from local pgpool-II on IPC interface Sep 1 11:20:16 centos8i3 pgpool[11426]: [81-1] pid 11426: LOG: we have got the consensus to perform the failover Sep 1 11:20:16 centos8i3 pgpool[11426]: [81-2] pid 11426: DETAIL: 2 node(s) voted in the favor Sep 1 11:20:16 centos8i3 pgpool[11424]: [18-1] pid 11424: LOG: Pgpool-II parent process has received failover request Sep 1 11:20:16 centos8i3 pgpool[11426]: [82-1] pid 11426: LOG: new IPC connection received Sep 1 11:20:16 centos8i3 pgpool[11426]: [83-1] pid 11426: LOG: received the failover indication from Pgpool-II on IPC interface Sep 1 11:20:16 centos8i3 pgpool[11426]: [84-1] pid 11426: LOG: watchdog is informed of failover start by the main process Sep 1 11:20:16 centos8i3 pgpool[11424]: [19-1] pid 11424: LOG: starting degeneration. shutdown host centos8i2-int(5433) Sep 1 11:20:16 centos8i3 pgpool[11424]: [20-1] pid 11424: LOG: Do not restart children because we are switching over node id 1 host: centos8i2-int port: 5433 and we are in streaming replication mode Sep 1 11:20:16 centos8i3 pgpool[11424]: [21-1] pid 11424: LOG: execute command: /etc/pgpool-II/failover.sh 1 centos8i2-int 5433 /db/data 0 centos8i1-int 0 0 5433 /db/data centos8i1-int 5433 Sep 1 11:20:16 centos8i3 pgpool[12098]: failover.sh: start: failed_node_id=1 old_primary_node_id=0 failed_host=centos8i2-int new_master_host=centos8i1-int Sep 1 11:20:16 centos8i3 pgpool[12097]: Warning: Permanently added 'centos8i1-int,10.0.0.1' (ECDSA) to the list of known hosts. Sep 1 11:20:17 centos8i3 pgpool[12103]: failover.sh: Standby node is down. Skipping failover. Sep 1 11:20:17 centos8i3 pgpool[11424]: [22-1] pid 11424: LOG: failover: set new primary node: 0 Sep 1 11:20:17 centos8i3 pgpool[11424]: [23-1] pid 11424: LOG: failover: set new master node: 0 Sep 1 11:20:17 centos8i3 pgpool[11489]: [15-1] pid 11489: LOG: trying to connect to PostgreSQL server on "centos8i2-int:5433" using INET socket Sep 1 11:20:17 centos8i3 pgpool[11489]: [15-2] pid 11489: DETAIL: select() interrupted. retrying... Sep 1 11:20:17 centos8i3 pgpool[11426]: [85-1] pid 11426: LOG: new IPC connection received Sep 1 11:20:17 centos8i3 pgpool[11426]: [86-1] pid 11426: LOG: received the failover indication from Pgpool-II on IPC interface Sep 1 11:20:17 centos8i3 pgpool[11426]: [87-1] pid 11426: LOG: watchdog is informed of failover end by the main process Sep 1 11:20:17 centos8i3 pgpool[11424]: [24-1] pid 11424: LOG: failover done. shutdown host centos8i2-int(5433) Sep 1 11:20:18 centos8i3 pgpool[11488]: [11-1] pid 11488: LOG: restart request received in pcp child process Sep 1 11:20:18 centos8i3 pgpool[11424]: [25-1] pid 11424: LOG: PCP child 11488 exits with status 0 in failover() Sep 1 11:20:18 centos8i3 pgpool[11424]: [26-1] pid 11424: LOG: fork a new PCP child pid 12107 in failover() Sep 1 11:20:18 centos8i3 pgpool[12107]: [26-1] pid 12107: LOG: PCP process: 12107 started Sep 1 11:20:21 centos8i3 pgpool[11489]: [16-1] pid 11489: LOG: failed to connect to PostgreSQL server on "centos8i2-int:5433", getsockopt() detected error "No route to host" Sep 1 11:20:21 centos8i3 pgpool[11489]: [17-1] pid 11489: ERROR: failed to make persistent db connection Sep 1 11:20:21 centos8i3 pgpool[11489]: [17-2] pid 11489: DETAIL: connection to host:"centos8i2-int:5433" failed Sep 1 11:20:31 centos8i3 pgpool[11489]: [18-1] pid 11489: ERROR: Failed to check replication time lag Sep 1 11:20:31 centos8i3 pgpool[11489]: [18-2] pid 11489: DETAIL: No persistent db connection for the node 1 Sep 1 11:20:31 centos8i3 pgpool[11489]: [18-3] pid 11489: HINT: check sr_check_user and sr_check_password Sep 1 11:20:31 centos8i3 pgpool[11489]: [18-4] pid 11489: CONTEXT: while checking replication time lag Sep 1 11:20:31 centos8i3 pgpool[11489]: [19-1] pid 11489: LOG: worker process received restart request Sep 1 11:20:31 centos8i3 pgpool[11424]: [27-1] pid 11424: LOG: worker child process with pid: 11489 exits with status 256 Sep 1 11:20:31 centos8i3 pgpool[11424]: [28-1] pid 11424: LOG: fork a new worker child process with pid: 12117 Sep 1 11:21:47 centos8i3 pgpool[12107]: [27-1] pid 12107: LOG: forked new pcp worker, pid=12165 socket=9 Sep 1 11:21:47 centos8i3 pgpool[11426]: [88-1] pid 11426: LOG: new IPC connection received Sep 1 11:21:47 centos8i3 pgpool[12107]: [28-1] pid 12107: LOG: PCP process with pid: 12165 exit with SUCCESS. Sep 1 11:21:47 centos8i3 pgpool[12107]: [29-1] pid 12107: LOG: PCP process with pid: 12165 exits with status 0 Sep 1 11:22:27 centos8i3 pgpool[12097]: could not connect: Connection timed out Sep 1 11:22:27 centos8i3 pgpool[12097]: . . . message not sent.