Sep 1 11:26:22 centos8i1 pgpool[14367]: [1-1] pid 14367: LOG: Backend status file /var/log/pgpool-II/pgpool_status discarded Sep 1 11:26:22 centos8i1 pgpool[14367]: [2-1] pid 14367: LOG: memory cache initialized Sep 1 11:26:22 centos8i1 pgpool[14367]: [2-2] pid 14367: DETAIL: memcache blocks :64 Sep 1 11:26:22 centos8i1 pgpool[14367]: [3-1] pid 14367: LOG: pool_discard_oid_maps: discarded memqcache oid maps Sep 1 11:26:22 centos8i1 pgpool[14367]: [4-1] pid 14367: LOG: waiting for watchdog to initialize Sep 1 11:26:22 centos8i1 pgpool[14369]: [4-1] pid 14369: LOG: setting the local watchdog node name to "centos8i1-int:5432 Linux centos8i1.localdomain" Sep 1 11:26:22 centos8i1 pgpool[14369]: [5-1] pid 14369: LOG: watchdog cluster is configured with 2 remote nodes Sep 1 11:26:22 centos8i1 pgpool[14369]: [6-1] pid 14369: LOG: watchdog remote node:0 on centos8i2-int:9000 Sep 1 11:26:22 centos8i1 pgpool[14369]: [7-1] pid 14369: LOG: watchdog remote node:1 on centos8i3-int:9000 Sep 1 11:26:22 centos8i1 pgpool[14369]: [8-1] pid 14369: LOG: interface monitoring is disabled in watchdog Sep 1 11:26:22 centos8i1 pgpool[14369]: [9-1] pid 14369: LOG: watchdog node state changed from [DEAD] to [LOADING] Sep 1 11:26:22 centos8i1 pgpool[14369]: [10-1] pid 14369: LOG: new watchdog node connection is received from "10.0.0.2:30922" Sep 1 11:26:22 centos8i1 pgpool[14369]: [11-1] pid 14369: LOG: new node joined the cluster hostname:"centos8i2-int" port:9000 pgpool_port:5432 Sep 1 11:26:22 centos8i1 pgpool[14369]: [11-2] pid 14369: DETAIL: Pgpool-II version:"4.1.2" watchdog messaging version: 1.1 Sep 1 11:26:22 centos8i1 pgpool[14369]: [12-1] pid 14369: LOG: new watchdog node connection is received from "10.0.0.3:64202" Sep 1 11:26:22 centos8i1 pgpool[14369]: [13-1] pid 14369: LOG: new node joined the cluster hostname:"centos8i3-int" port:9000 pgpool_port:5432 Sep 1 11:26:22 centos8i1 pgpool[14369]: [13-2] pid 14369: DETAIL: Pgpool-II version:"4.1.2" watchdog messaging version: 1.1 Sep 1 11:26:27 centos8i1 pgpool[14369]: [14-1] pid 14369: LOG: watchdog node state changed from [LOADING] to [JOINING] Sep 1 11:26:27 centos8i1 pgpool[14369]: [15-1] pid 14369: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] Sep 1 11:26:28 centos8i1 pgpool[14369]: [16-1] pid 14369: LOG: watchdog node state changed from [INITIALIZING] to [STANDING FOR MASTER] Sep 1 11:26:28 centos8i1 pgpool[14369]: [17-1] pid 14369: LOG: our stand for coordinator request is rejected by node "centos8i2-int:5432 Linux centos8i2.localdomain" Sep 1 11:26:28 centos8i1 pgpool[14369]: [18-1] pid 14369: LOG: watchdog node state changed from [STANDING FOR MASTER] to [PARTICIPATING IN ELECTION] Sep 1 11:26:33 centos8i1 pgpool[14369]: [19-1] pid 14369: LOG: watchdog node state changed from [PARTICIPATING IN ELECTION] to [JOINING] Sep 1 11:26:33 centos8i1 pgpool[14369]: [20-1] pid 14369: LOG: new outbound connection to centos8i2-int:9000 Sep 1 11:26:33 centos8i1 pgpool[14369]: [21-1] pid 14369: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] Sep 1 11:26:33 centos8i1 pgpool[14369]: [22-1] pid 14369: LOG: new outbound connection to centos8i3-int:9000 Sep 1 11:26:34 centos8i1 pgpool[14369]: [23-1] pid 14369: LOG: watchdog node state changed from [INITIALIZING] to [STANDING FOR MASTER] Sep 1 11:26:34 centos8i1 pgpool[14369]: [24-1] pid 14369: LOG: our stand for coordinator request is rejected by node "centos8i3-int:5432 Linux centos8i3.localdomain" Sep 1 11:26:34 centos8i1 pgpool[14369]: [25-1] pid 14369: LOG: watchdog node state changed from [STANDING FOR MASTER] to [PARTICIPATING IN ELECTION] Sep 1 11:26:39 centos8i1 pgpool[14369]: [26-1] pid 14369: LOG: watchdog node state changed from [PARTICIPATING IN ELECTION] to [JOINING] Sep 1 11:26:39 centos8i1 pgpool[14369]: [27-1] pid 14369: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] Sep 1 11:26:40 centos8i1 pgpool[14369]: [28-1] pid 14369: LOG: watchdog node state changed from [INITIALIZING] to [STANDING FOR MASTER] Sep 1 11:26:40 centos8i1 pgpool[14369]: [29-1] pid 14369: LOG: our stand for coordinator request is rejected by node "centos8i3-int:5432 Linux centos8i3.localdomain" Sep 1 11:26:40 centos8i1 pgpool[14369]: [30-1] pid 14369: LOG: watchdog node state changed from [STANDING FOR MASTER] to [PARTICIPATING IN ELECTION] Sep 1 11:26:45 centos8i1 pgpool[14369]: [31-1] pid 14369: LOG: watchdog node state changed from [PARTICIPATING IN ELECTION] to [JOINING] Sep 1 11:26:45 centos8i1 pgpool[14369]: [32-1] pid 14369: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] Sep 1 11:26:46 centos8i1 pgpool[14369]: [33-1] pid 14369: LOG: watchdog node state changed from [INITIALIZING] to [STANDING FOR MASTER] Sep 1 11:26:46 centos8i1 pgpool[14369]: [34-1] pid 14369: LOG: our stand for coordinator request is rejected by node "centos8i2-int:5432 Linux centos8i2.localdomain" Sep 1 11:26:46 centos8i1 pgpool[14369]: [35-1] pid 14369: LOG: watchdog node state changed from [STANDING FOR MASTER] to [PARTICIPATING IN ELECTION] Sep 1 11:26:51 centos8i1 pgpool[14369]: [36-1] pid 14369: LOG: watchdog node state changed from [PARTICIPATING IN ELECTION] to [JOINING] Sep 1 11:26:51 centos8i1 pgpool[14369]: [37-1] pid 14369: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] Sep 1 11:26:52 centos8i1 pgpool[14369]: [38-1] pid 14369: LOG: watchdog node state changed from [INITIALIZING] to [STANDING FOR MASTER] Sep 1 11:26:56 centos8i1 pgpool[14369]: [39-1] pid 14369: LOG: watchdog node state changed from [STANDING FOR MASTER] to [MASTER] Sep 1 11:26:56 centos8i1 pgpool[14369]: [40-1] pid 14369: LOG: I am announcing my self as master/coordinator watchdog node Sep 1 11:26:56 centos8i1 pgpool[14369]: [41-1] pid 14369: LOG: I am the cluster leader node Sep 1 11:26:56 centos8i1 pgpool[14369]: [41-2] pid 14369: DETAIL: our declare coordinator message is accepted by all nodes Sep 1 11:26:56 centos8i1 pgpool[14369]: [42-1] pid 14369: LOG: setting the local node "centos8i1-int:5432 Linux centos8i1.localdomain" as watchdog cluster master Sep 1 11:26:56 centos8i1 pgpool[14369]: [43-1] pid 14369: LOG: I am the cluster leader node but we do not have enough nodes in cluster Sep 1 11:26:56 centos8i1 pgpool[14369]: [43-2] pid 14369: DETAIL: waiting for the quorum to start escalation process Sep 1 11:26:56 centos8i1 pgpool[14367]: [5-1] pid 14367: LOG: watchdog process is initialized Sep 1 11:26:56 centos8i1 pgpool[14367]: [5-2] pid 14367: DETAIL: watchdog messaging data version: 1.1 Sep 1 11:26:56 centos8i1 pgpool[14369]: [44-1] pid 14369: LOG: new IPC connection received Sep 1 11:26:56 centos8i1 pgpool[14367]: [6-1] pid 14367: LOG: Setting up socket for 0.0.0.0:5432 Sep 1 11:26:56 centos8i1 pgpool[14367]: [7-1] pid 14367: LOG: Setting up socket for :::5432 Sep 1 11:26:56 centos8i1 pgpool[14367]: [8-1] pid 14367: LOG: find_primary_node_repeatedly: waiting for finding a primary node Sep 1 11:26:56 centos8i1 pgpool[14369]: [45-1] pid 14369: LOG: new IPC connection received Sep 1 11:26:56 centos8i1 pgpool[14394]: [6-1] pid 14394: LOG: 3 watchdog nodes are configured for lifecheck Sep 1 11:26:56 centos8i1 pgpool[14394]: [7-1] pid 14394: LOG: watchdog nodes ID:0 Name:"centos8i1-int:5432 Linux centos8i1.localdomain" Sep 1 11:26:56 centos8i1 pgpool[14394]: [7-2] pid 14394: DETAIL: Host:"centos8i1-int" WD Port:9000 pgpool-II port:5432 Sep 1 11:26:56 centos8i1 pgpool[14394]: [8-1] pid 14394: LOG: watchdog nodes ID:1 Name:"centos8i2-int:5432 Linux centos8i2.localdomain" Sep 1 11:26:56 centos8i1 pgpool[14394]: [8-2] pid 14394: DETAIL: Host:"centos8i2-int" WD Port:9000 pgpool-II port:5432 Sep 1 11:26:56 centos8i1 pgpool[14394]: [9-1] pid 14394: LOG: watchdog nodes ID:2 Name:"centos8i3-int:5432 Linux centos8i3.localdomain" Sep 1 11:26:56 centos8i1 pgpool[14394]: [9-2] pid 14394: DETAIL: Host:"centos8i3-int" WD Port:9000 pgpool-II port:5432 Sep 1 11:26:56 centos8i1 pgpool[14367]: [9-1] pid 14367: LOG: find_primary_node: primary node is 0 Sep 1 11:26:56 centos8i1 pgpool[14367]: [10-1] pid 14367: LOG: find_primary_node: standby node is 1 Sep 1 11:26:56 centos8i1 pgpool[14367]: [11-1] pid 14367: LOG: find_primary_node: standby node is 2 Sep 1 11:26:56 centos8i1 pgpool[14432]: [12-1] pid 14432: LOG: PCP process: 14432 started Sep 1 11:26:56 centos8i1 pgpool[14367]: [12-1] pid 14367: LOG: pgpool-II successfully started. version 4.1.2 (karasukiboshi) Sep 1 11:26:56 centos8i1 pgpool[14367]: [13-1] pid 14367: LOG: node status[0]: 1 Sep 1 11:26:56 centos8i1 pgpool[14367]: [14-1] pid 14367: LOG: node status[1]: 2 Sep 1 11:26:56 centos8i1 pgpool[14367]: [15-1] pid 14367: LOG: node status[2]: 2 Sep 1 11:26:57 centos8i1 pgpool[14369]: [46-1] pid 14369: LOG: adding watchdog node "centos8i2-int:5432 Linux centos8i2.localdomain" to the standby list Sep 1 11:26:57 centos8i1 pgpool[14369]: [47-1] pid 14369: LOG: adding watchdog node "centos8i3-int:5432 Linux centos8i3.localdomain" to the standby list Sep 1 11:26:57 centos8i1 pgpool[14369]: [48-1] pid 14369: LOG: quorum found Sep 1 11:26:57 centos8i1 pgpool[14369]: [48-2] pid 14369: DETAIL: starting escalation process Sep 1 11:26:57 centos8i1 pgpool[14369]: [49-1] pid 14369: LOG: escalation process started with PID:14439 Sep 1 11:26:57 centos8i1 pgpool[14439]: [49-1] pid 14439: LOG: watchdog: escalation started Sep 1 11:26:57 centos8i1 pgpool[14367]: [16-1] pid 14367: LOG: Pgpool-II parent process received watchdog quorum change signal from watchdog Sep 1 11:26:57 centos8i1 pgpool[14428]: [10-1] pid 14428: LOG: set SO_REUSEPORT option to the socket Sep 1 11:26:57 centos8i1 pgpool[14428]: [11-1] pid 14428: LOG: creating watchdog heartbeat receive socket. Sep 1 11:26:57 centos8i1 pgpool[14428]: [11-2] pid 14428: DETAIL: set SO_REUSEPORT Sep 1 11:26:57 centos8i1 pgpool[14431]: [10-1] pid 14431: LOG: set SO_REUSEPORT option to the socket Sep 1 11:26:57 centos8i1 pgpool[14431]: [11-1] pid 14431: LOG: creating socket for sending heartbeat Sep 1 11:26:57 centos8i1 pgpool[14431]: [11-2] pid 14431: DETAIL: set SO_REUSEPORT Sep 1 11:26:57 centos8i1 pgpool[14430]: [10-1] pid 14430: LOG: set SO_REUSEPORT option to the socket Sep 1 11:26:57 centos8i1 pgpool[14430]: [11-1] pid 14430: LOG: creating watchdog heartbeat receive socket. Sep 1 11:26:57 centos8i1 pgpool[14430]: [11-2] pid 14430: DETAIL: set SO_REUSEPORT Sep 1 11:26:57 centos8i1 pgpool[14429]: [10-1] pid 14429: LOG: set SO_REUSEPORT option to the socket Sep 1 11:26:57 centos8i1 pgpool[14429]: [11-1] pid 14429: LOG: creating socket for sending heartbeat Sep 1 11:26:57 centos8i1 pgpool[14429]: [11-2] pid 14429: DETAIL: set SO_REUSEPORT Sep 1 11:26:57 centos8i1 pgpool[14369]: [50-1] pid 14369: LOG: new IPC connection received Sep 1 11:26:57 centos8i1 pgpool[14367]: [17-1] pid 14367: LOG: watchdog cluster now holds the quorum Sep 1 11:26:57 centos8i1 pgpool[14367]: [17-2] pid 14367: DETAIL: updating the state of quarantine backend nodes Sep 1 11:26:57 centos8i1 pgpool[14369]: [51-1] pid 14369: LOG: new IPC connection received Sep 1 11:27:02 centos8i1 pgpool[14439]: [50-1] pid 14439: LOG: successfully acquired the delegate IP:"192.168.56.120" Sep 1 11:27:02 centos8i1 pgpool[14439]: [50-2] pid 14439: DETAIL: 'if_up_cmd' returned with success Sep 1 11:27:02 centos8i1 pgpool[14369]: [52-1] pid 14369: LOG: watchdog escalation process with pid: 14439 exit with SUCCESS. ***** POWER OFF NODE centos8i2-int ***** Sep 1 11:27:56 centos8i1 pgpool[14435]: [12-1] pid 14435: LOG: failed to connect to PostgreSQL server on "centos8i2-int:5433", timed out Sep 1 11:27:56 centos8i1 pgpool[14435]: [13-1] pid 14435: ERROR: failed to make persistent db connection Sep 1 11:27:56 centos8i1 pgpool[14435]: [13-2] pid 14435: DETAIL: connection to host:"centos8i2-int:5433" failed Sep 1 11:27:56 centos8i1 pgpool[14435]: [14-1] pid 14435: LOG: health check retrying on DB node: 1 (round:1) Sep 1 11:27:56 centos8i1 pgpool[14433]: [12-1] pid 14433: LOG: trying connecting to PostgreSQL server on "centos8i2-int:5433" by INET socket Sep 1 11:27:56 centos8i1 pgpool[14433]: [12-2] pid 14433: DETAIL: timed out. retrying... Sep 1 11:27:57 centos8i1 pgpool[14369]: [53-1] pid 14369: LOG: remote node "centos8i2-int:5432 Linux centos8i2.localdomain" is not replying to our beacons Sep 1 11:27:57 centos8i1 pgpool[14369]: [53-2] pid 14369: DETAIL: missed beacon reply count:2 Sep 1 11:28:06 centos8i1 pgpool[14433]: [13-1] pid 14433: LOG: trying connecting to PostgreSQL server on "centos8i2-int:5433" by INET socket Sep 1 11:28:06 centos8i1 pgpool[14433]: [13-2] pid 14433: DETAIL: timed out. retrying... Sep 1 11:28:07 centos8i1 pgpool[14369]: [54-1] pid 14369: LOG: remote node "centos8i2-int:5432 Linux centos8i2.localdomain" is not replying to our beacons Sep 1 11:28:07 centos8i1 pgpool[14369]: [54-2] pid 14369: DETAIL: missed beacon reply count:3 Sep 1 11:28:07 centos8i1 pgpool[14435]: [15-1] pid 14435: LOG: failed to connect to PostgreSQL server on "centos8i2-int:5433", timed out Sep 1 11:28:07 centos8i1 pgpool[14435]: [16-1] pid 14435: ERROR: failed to make persistent db connection Sep 1 11:28:07 centos8i1 pgpool[14435]: [16-2] pid 14435: DETAIL: connection to host:"centos8i2-int:5433" failed Sep 1 11:28:07 centos8i1 pgpool[14435]: [17-1] pid 14435: LOG: health check retrying on DB node: 1 (round:2) Sep 1 11:28:16 centos8i1 pgpool[14433]: [14-1] pid 14433: LOG: trying connecting to PostgreSQL server on "centos8i2-int:5433" by INET socket Sep 1 11:28:16 centos8i1 pgpool[14433]: [14-2] pid 14433: DETAIL: timed out. retrying... Sep 1 11:28:17 centos8i1 pgpool[14369]: [55-1] pid 14369: LOG: remote node "centos8i2-int:5432 Linux centos8i2.localdomain" is not replying to our beacons Sep 1 11:28:17 centos8i1 pgpool[14369]: [55-2] pid 14369: DETAIL: missed beacon reply count:4 Sep 1 11:28:17 centos8i1 pgpool[14369]: [56-1] pid 14369: LOG: remote node "centos8i2-int:5432 Linux centos8i2.localdomain" is not responding to our beacon messages Sep 1 11:28:17 centos8i1 pgpool[14369]: [56-2] pid 14369: DETAIL: marking the node as lost Sep 1 11:28:17 centos8i1 pgpool[14369]: [57-1] pid 14369: LOG: remote node "centos8i2-int:5432 Linux centos8i2.localdomain" is lost Sep 1 11:28:17 centos8i1 pgpool[14369]: [58-1] pid 14369: LOG: removing watchdog node "centos8i2-int:5432 Linux centos8i2.localdomain" from the standby list Sep 1 11:28:18 centos8i1 pgpool[14435]: [18-1] pid 14435: LOG: failed to connect to PostgreSQL server on "centos8i2-int:5433", timed out Sep 1 11:28:18 centos8i1 pgpool[14435]: [19-1] pid 14435: ERROR: failed to make persistent db connection Sep 1 11:28:18 centos8i1 pgpool[14435]: [19-2] pid 14435: DETAIL: connection to host:"centos8i2-int:5433" failed Sep 1 11:28:18 centos8i1 pgpool[14435]: [20-1] pid 14435: LOG: health check retrying on DB node: 1 (round:3) Sep 1 11:28:25 centos8i1 pgpool[14369]: [59-1] pid 14369: LOG: watchdog received the failover command from remote pgpool-II node "centos8i3-int:5432 Linux centos8i3.localdomain" Sep 1 11:28:25 centos8i1 pgpool[14369]: [60-1] pid 14369: LOG: watchdog is processing the failover command [DEGENERATE_BACKEND_REQUEST] received from centos8i3-int:5432 Linux centos8i3.localdomain Sep 1 11:28:25 centos8i1 pgpool[14369]: [61-1] pid 14369: LOG: failover requires the majority vote, waiting for consensus Sep 1 11:28:25 centos8i1 pgpool[14369]: [61-2] pid 14369: DETAIL: failover request noted Sep 1 11:28:25 centos8i1 pgpool[14369]: [62-1] pid 14369: LOG: failover command [DEGENERATE_BACKEND_REQUEST] request from pgpool-II node "centos8i3-int:5432 Linux centos8i3.localdomain" is queued, waiting for the confirmation from other nodes Sep 1 11:28:25 centos8i1 pgpool[14367]: [18-1] pid 14367: LOG: Pgpool-II parent process received inform quarantine nodes signal from watchdog Sep 1 11:28:26 centos8i1 pgpool[14433]: [15-1] pid 14433: LOG: trying connecting to PostgreSQL server on "centos8i2-int:5433" by INET socket Sep 1 11:28:26 centos8i1 pgpool[14433]: [15-2] pid 14433: DETAIL: timed out. retrying... Sep 1 11:28:29 centos8i1 pgpool[14435]: [21-1] pid 14435: LOG: failed to connect to PostgreSQL server on "centos8i2-int:5433", timed out Sep 1 11:28:29 centos8i1 pgpool[14435]: [22-1] pid 14435: ERROR: failed to make persistent db connection Sep 1 11:28:29 centos8i1 pgpool[14435]: [22-2] pid 14435: DETAIL: connection to host:"centos8i2-int:5433" failed Sep 1 11:28:29 centos8i1 pgpool[14435]: [23-1] pid 14435: LOG: health check failed on node 1 (timeout:0) Sep 1 11:28:29 centos8i1 pgpool[14435]: [24-1] pid 14435: LOG: received degenerate backend request for node_id: 1 from pid [14435] Sep 1 11:28:29 centos8i1 pgpool[14369]: [63-1] pid 14369: LOG: new IPC connection received Sep 1 11:28:29 centos8i1 pgpool[14369]: [64-1] pid 14369: LOG: watchdog received the failover command from local pgpool-II on IPC interface Sep 1 11:28:29 centos8i1 pgpool[14369]: [65-1] pid 14369: LOG: watchdog is processing the failover command [DEGENERATE_BACKEND_REQUEST] received from local pgpool-II on IPC interface Sep 1 11:28:29 centos8i1 pgpool[14369]: [66-1] pid 14369: LOG: we have got the consensus to perform the failover Sep 1 11:28:29 centos8i1 pgpool[14369]: [66-2] pid 14369: DETAIL: 2 node(s) voted in the favor Sep 1 11:28:29 centos8i1 pgpool[14367]: [19-1] pid 14367: LOG: Pgpool-II parent process has received failover request Sep 1 11:28:29 centos8i1 pgpool[14369]: [67-1] pid 14369: LOG: new IPC connection received Sep 1 11:28:29 centos8i1 pgpool[14369]: [68-1] pid 14369: LOG: received the failover indication from Pgpool-II on IPC interface Sep 1 11:28:29 centos8i1 pgpool[14369]: [69-1] pid 14369: LOG: watchdog is informed of failover start by the main process Sep 1 11:28:29 centos8i1 pgpool[14367]: [20-1] pid 14367: LOG: starting degeneration. shutdown host centos8i2-int(5433) Sep 1 11:28:29 centos8i1 pgpool[14367]: [21-1] pid 14367: LOG: Do not restart children because we are switching over node id 1 host: centos8i2-int port: 5433 and we are in streaming replication mode Sep 1 11:28:29 centos8i1 pgpool[14367]: [22-1] pid 14367: LOG: execute command: /etc/pgpool-II/failover.sh 1 centos8i2-int 5433 /db/data 0 centos8i1-int 0 0 5433 /db/data centos8i1-int 5433 Sep 1 11:28:29 centos8i1 pgpool[14736]: failover.sh: start: failed_node_id=1 old_primary_node_id=0 failed_host=centos8i2-int new_master_host=centos8i1-int Sep 1 11:28:29 centos8i1 pgpool[14735]: Warning: Permanently added 'centos8i1-int,10.0.0.1' (ECDSA) to the list of known hosts. Sep 1 11:28:30 centos8i1 pgpool[14757]: failover.sh: Standby node is down. Skipping failover. Sep 1 11:28:30 centos8i1 pgpool[14367]: [23-1] pid 14367: LOG: failover: set new primary node: 0 Sep 1 11:28:30 centos8i1 pgpool[14367]: [24-1] pid 14367: LOG: failover: set new master node: 0 Sep 1 11:28:30 centos8i1 pgpool[14433]: [16-1] pid 14433: LOG: trying to connect to PostgreSQL server on "centos8i2-int:5433" using INET socket Sep 1 11:28:30 centos8i1 pgpool[14433]: [16-2] pid 14433: DETAIL: select() interrupted. retrying... Sep 1 11:28:30 centos8i1 pgpool[14369]: [70-1] pid 14369: LOG: new IPC connection received Sep 1 11:28:30 centos8i1 pgpool[14369]: [71-1] pid 14369: LOG: received the failover indication from Pgpool-II on IPC interface Sep 1 11:28:30 centos8i1 pgpool[14369]: [72-1] pid 14369: LOG: watchdog is informed of failover end by the main process Sep 1 11:28:30 centos8i1 pgpool[14367]: [25-1] pid 14367: LOG: failover done. shutdown host centos8i2-int(5433) Sep 1 11:28:31 centos8i1 pgpool[14432]: [13-1] pid 14432: LOG: restart request received in pcp child process Sep 1 11:28:31 centos8i1 pgpool[14367]: [26-1] pid 14367: LOG: PCP child 14432 exits with status 0 in failover() Sep 1 11:28:31 centos8i1 pgpool[14367]: [27-1] pid 14367: LOG: fork a new PCP child pid 14770 in failover() Sep 1 11:28:31 centos8i1 pgpool[14770]: [27-1] pid 14770: LOG: PCP process: 14770 started Sep 1 11:28:40 centos8i1 pgpool[14433]: [17-1] pid 14433: LOG: trying connecting to PostgreSQL server on "centos8i2-int:5433" by INET socket Sep 1 11:28:40 centos8i1 pgpool[14433]: [17-2] pid 14433: DETAIL: timed out. retrying... Sep 1 11:28:44 centos8i1 pgpool[14435]: [25-1] pid 14435: LOG: failed to connect to PostgreSQL server on "centos8i2-int:5433", timed out Sep 1 11:28:44 centos8i1 pgpool[14435]: [26-1] pid 14435: ERROR: failed to make persistent db connection Sep 1 11:28:44 centos8i1 pgpool[14435]: [26-2] pid 14435: DETAIL: connection to host:"centos8i2-int:5433" failed Sep 1 11:28:44 centos8i1 pgpool[14435]: [27-1] pid 14435: LOG: health check retrying on DB node: 1 (round:1) Sep 1 11:28:50 centos8i1 pgpool[14433]: [18-1] pid 14433: LOG: trying connecting to PostgreSQL server on "centos8i2-int:5433" by INET socket Sep 1 11:28:50 centos8i1 pgpool[14433]: [18-2] pid 14433: DETAIL: timed out. retrying... Sep 1 11:28:55 centos8i1 pgpool[14435]: [28-1] pid 14435: LOG: failed to connect to PostgreSQL server on "centos8i2-int:5433", timed out Sep 1 11:28:55 centos8i1 pgpool[14435]: [29-1] pid 14435: ERROR: failed to make persistent db connection Sep 1 11:28:55 centos8i1 pgpool[14435]: [29-2] pid 14435: DETAIL: connection to host:"centos8i2-int:5433" failed Sep 1 11:28:55 centos8i1 pgpool[14435]: [30-1] pid 14435: LOG: health check retrying on DB node: 1 (round:2) Sep 1 11:29:00 centos8i1 pgpool[14433]: [19-1] pid 14433: LOG: trying connecting to PostgreSQL server on "centos8i2-int:5433" by INET socket Sep 1 11:29:00 centos8i1 pgpool[14433]: [19-2] pid 14433: DETAIL: timed out. retrying... Sep 1 11:29:02 centos8i1 pgpool[14435]: [31-1] pid 14435: LOG: failed to connect to PostgreSQL server on "centos8i2-int:5433", getsockopt() detected error "No route to host" Sep 1 11:29:02 centos8i1 pgpool[14435]: [32-1] pid 14435: ERROR: failed to make persistent db connection Sep 1 11:29:02 centos8i1 pgpool[14435]: [32-2] pid 14435: DETAIL: connection to host:"centos8i2-int:5433" failed Sep 1 11:29:02 centos8i1 pgpool[14435]: [33-1] pid 14435: LOG: health check retrying on DB node: 1 (round:3) Sep 1 11:29:06 centos8i1 pgpool[14435]: [34-1] pid 14435: LOG: failed to connect to PostgreSQL server on "centos8i2-int:5433", getsockopt() detected error "No route to host" Sep 1 11:29:06 centos8i1 pgpool[14435]: [35-1] pid 14435: ERROR: failed to make persistent db connection Sep 1 11:29:06 centos8i1 pgpool[14435]: [35-2] pid 14435: DETAIL: connection to host:"centos8i2-int:5433" failed Sep 1 11:29:10 centos8i1 pgpool[14433]: [20-1] pid 14433: LOG: trying connecting to PostgreSQL server on "centos8i2-int:5433" by INET socket Sep 1 11:29:10 centos8i1 pgpool[14433]: [20-2] pid 14433: DETAIL: timed out. retrying... Sep 1 11:29:13 centos8i1 pgpool[14435]: [36-1] pid 14435: LOG: failed to connect to PostgreSQL server on "centos8i2-int:5433", getsockopt() detected error "No route to host" Sep 1 11:29:13 centos8i1 pgpool[14435]: [37-1] pid 14435: ERROR: failed to make persistent db connection Sep 1 11:29:13 centos8i1 pgpool[14435]: [37-2] pid 14435: DETAIL: connection to host:"centos8i2-int:5433" failed Sep 1 11:29:13 centos8i1 pgpool[14435]: [38-1] pid 14435: LOG: health check retrying on DB node: 1 (round:1) Sep 1 11:29:16 centos8i1 pgpool[14435]: [39-1] pid 14435: LOG: failed to connect to PostgreSQL server on "centos8i2-int:5433", getsockopt() detected error "No route to host" Sep 1 11:29:16 centos8i1 pgpool[14435]: [40-1] pid 14435: ERROR: failed to make persistent db connection Sep 1 11:29:16 centos8i1 pgpool[14435]: [40-2] pid 14435: DETAIL: connection to host:"centos8i2-int:5433" failed Sep 1 11:29:16 centos8i1 pgpool[14435]: [41-1] pid 14435: LOG: health check retrying on DB node: 1 (round:2) Sep 1 11:29:19 centos8i1 pgpool[14435]: [42-1] pid 14435: LOG: failed to connect to PostgreSQL server on "centos8i2-int:5433", getsockopt() detected error "No route to host" Sep 1 11:29:19 centos8i1 pgpool[14435]: [43-1] pid 14435: ERROR: failed to make persistent db connection Sep 1 11:29:19 centos8i1 pgpool[14435]: [43-2] pid 14435: DETAIL: connection to host:"centos8i2-int:5433" failed Sep 1 11:29:19 centos8i1 pgpool[14435]: [44-1] pid 14435: LOG: health check retrying on DB node: 1 (round:3) Sep 1 11:29:20 centos8i1 pgpool[14433]: [21-1] pid 14433: LOG: trying connecting to PostgreSQL server on "centos8i2-int:5433" by INET socket Sep 1 11:29:20 centos8i1 pgpool[14433]: [21-2] pid 14433: DETAIL: timed out. retrying... Sep 1 11:29:22 centos8i1 pgpool[14435]: [45-1] pid 14435: LOG: failed to connect to PostgreSQL server on "centos8i2-int:5433", getsockopt() detected error "No route to host" Sep 1 11:29:22 centos8i1 pgpool[14435]: [46-1] pid 14435: ERROR: failed to make persistent db connection Sep 1 11:29:22 centos8i1 pgpool[14435]: [46-2] pid 14435: DETAIL: connection to host:"centos8i2-int:5433" failed Sep 1 11:29:29 centos8i1 pgpool[14435]: [47-1] pid 14435: LOG: failed to connect to PostgreSQL server on "centos8i2-int:5433", getsockopt() detected error "No route to host" Sep 1 11:29:29 centos8i1 pgpool[14435]: [48-1] pid 14435: ERROR: failed to make persistent db connection Sep 1 11:29:29 centos8i1 pgpool[14435]: [48-2] pid 14435: DETAIL: connection to host:"centos8i2-int:5433" failed Sep 1 11:29:29 centos8i1 pgpool[14435]: [49-1] pid 14435: LOG: health check retrying on DB node: 1 (round:1) Sep 1 11:29:30 centos8i1 pgpool[14433]: [22-1] pid 14433: LOG: trying connecting to PostgreSQL server on "centos8i2-int:5433" by INET socket Sep 1 11:29:30 centos8i1 pgpool[14433]: [22-2] pid 14433: DETAIL: timed out. retrying... Sep 1 11:29:32 centos8i1 pgpool[14435]: [50-1] pid 14435: LOG: failed to connect to PostgreSQL server on "centos8i2-int:5433", getsockopt() detected error "No route to host" Sep 1 11:29:32 centos8i1 pgpool[14435]: [51-1] pid 14435: ERROR: failed to make persistent db connection Sep 1 11:29:32 centos8i1 pgpool[14435]: [51-2] pid 14435: DETAIL: connection to host:"centos8i2-int:5433" failed Sep 1 11:29:32 centos8i1 pgpool[14435]: [52-1] pid 14435: LOG: health check retrying on DB node: 1 (round:2) Sep 1 11:29:35 centos8i1 pgpool[14435]: [53-1] pid 14435: LOG: failed to connect to PostgreSQL server on "centos8i2-int:5433", getsockopt() detected error "No route to host" Sep 1 11:29:35 centos8i1 pgpool[14435]: [54-1] pid 14435: ERROR: failed to make persistent db connection Sep 1 11:29:35 centos8i1 pgpool[14435]: [54-2] pid 14435: DETAIL: connection to host:"centos8i2-int:5433" failed Sep 1 11:29:35 centos8i1 pgpool[14435]: [55-1] pid 14435: LOG: health check retrying on DB node: 1 (round:3) Sep 1 11:29:38 centos8i1 pgpool[14435]: [56-1] pid 14435: LOG: failed to connect to PostgreSQL server on "centos8i2-int:5433", getsockopt() detected error "No route to host" Sep 1 11:29:38 centos8i1 pgpool[14435]: [57-1] pid 14435: ERROR: failed to make persistent db connection Sep 1 11:29:38 centos8i1 pgpool[14435]: [57-2] pid 14435: DETAIL: connection to host:"centos8i2-int:5433" failed Sep 1 11:29:40 centos8i1 pgpool[14433]: [23-1] pid 14433: LOG: trying connecting to PostgreSQL server on "centos8i2-int:5433" by INET socket Sep 1 11:29:40 centos8i1 pgpool[14433]: [23-2] pid 14433: DETAIL: timed out. retrying... Sep 1 11:29:45 centos8i1 pgpool[14435]: [58-1] pid 14435: LOG: failed to connect to PostgreSQL server on "centos8i2-int:5433", getsockopt() detected error "No route to host" Sep 1 11:29:45 centos8i1 pgpool[14435]: [59-1] pid 14435: ERROR: failed to make persistent db connection Sep 1 11:29:45 centos8i1 pgpool[14435]: [59-2] pid 14435: DETAIL: connection to host:"centos8i2-int:5433" failed Sep 1 11:29:45 centos8i1 pgpool[14435]: [60-1] pid 14435: LOG: health check retrying on DB node: 1 (round:1) Sep 1 11:29:48 centos8i1 pgpool[14435]: [61-1] pid 14435: LOG: failed to connect to PostgreSQL server on "centos8i2-int:5433", getsockopt() detected error "No route to host" Sep 1 11:29:48 centos8i1 pgpool[14435]: [62-1] pid 14435: ERROR: failed to make persistent db connection Sep 1 11:29:48 centos8i1 pgpool[14435]: [62-2] pid 14435: DETAIL: connection to host:"centos8i2-int:5433" failed Sep 1 11:29:48 centos8i1 pgpool[14435]: [63-1] pid 14435: LOG: health check retrying on DB node: 1 (round:2) Sep 1 11:29:50 centos8i1 pgpool[14433]: [24-1] pid 14433: LOG: trying connecting to PostgreSQL server on "centos8i2-int:5433" by INET socket Sep 1 11:29:50 centos8i1 pgpool[14433]: [24-2] pid 14433: DETAIL: timed out. retrying... Sep 1 11:29:51 centos8i1 pgpool[14435]: [64-1] pid 14435: LOG: failed to connect to PostgreSQL server on "centos8i2-int:5433", getsockopt() detected error "No route to host" Sep 1 11:29:51 centos8i1 pgpool[14435]: [65-1] pid 14435: ERROR: failed to make persistent db connection Sep 1 11:29:51 centos8i1 pgpool[14435]: [65-2] pid 14435: DETAIL: connection to host:"centos8i2-int:5433" failed Sep 1 11:29:51 centos8i1 pgpool[14435]: [66-1] pid 14435: LOG: health check retrying on DB node: 1 (round:3) Sep 1 11:29:54 centos8i1 pgpool[14435]: [67-1] pid 14435: LOG: failed to connect to PostgreSQL server on "centos8i2-int:5433", getsockopt() detected error "No route to host" Sep 1 11:29:54 centos8i1 pgpool[14435]: [68-1] pid 14435: ERROR: failed to make persistent db connection Sep 1 11:29:54 centos8i1 pgpool[14435]: [68-2] pid 14435: DETAIL: connection to host:"centos8i2-int:5433" failed Sep 1 11:29:58 centos8i1 pgpool[14433]: [25-1] pid 14433: LOG: failed to connect to PostgreSQL server on "centos8i2-int:5433", getsockopt() detected error "Connection timed out" Sep 1 11:29:58 centos8i1 pgpool[14433]: [26-1] pid 14433: ERROR: failed to make persistent db connection Sep 1 11:29:58 centos8i1 pgpool[14433]: [26-2] pid 14433: DETAIL: connection to host:"centos8i2-int:5433" failed Sep 1 11:30:08 centos8i1 pgpool[14433]: [27-1] pid 14433: ERROR: Failed to check replication time lag Sep 1 11:30:08 centos8i1 pgpool[14433]: [27-2] pid 14433: DETAIL: No persistent db connection for the node 1 Sep 1 11:30:08 centos8i1 pgpool[14433]: [27-3] pid 14433: HINT: check sr_check_user and sr_check_password Sep 1 11:30:08 centos8i1 pgpool[14433]: [27-4] pid 14433: CONTEXT: while checking replication time lag Sep 1 11:30:08 centos8i1 pgpool[14433]: [28-1] pid 14433: LOG: worker process received restart request Sep 1 11:30:08 centos8i1 pgpool[14367]: [28-1] pid 14367: LOG: worker child process with pid: 14433 exits with status 256 Sep 1 11:30:08 centos8i1 pgpool[14367]: [29-1] pid 14367: LOG: fork a new worker child process with pid: 14823 Sep 1 11:30:38 centos8i1 pgpool[14770]: [28-1] pid 14770: LOG: forked new pcp worker, pid=14897 socket=9 Sep 1 11:30:38 centos8i1 pgpool[14369]: [73-1] pid 14369: LOG: new IPC connection received Sep 1 11:30:38 centos8i1 pgpool[14770]: [29-1] pid 14770: LOG: PCP process with pid: 14897 exit with SUCCESS. Sep 1 11:30:38 centos8i1 pgpool[14770]: [30-1] pid 14770: LOG: PCP process with pid: 14897 exits with status 0 Sep 1 11:30:39 centos8i1 pgpool[14735]: could not connect: Connection timed out Sep 1 11:30:39 centos8i1 pgpool[14735]: . . . message not sent.