Sep 1 11:09:38 centos8i1 pgpool[12229]: [1-1] pid 12229: LOG: Backend status file /var/log/pgpool-II/pgpool_status discarded Sep 1 11:09:38 centos8i1 pgpool[12229]: [2-1] pid 12229: LOG: memory cache initialized Sep 1 11:09:38 centos8i1 pgpool[12229]: [2-2] pid 12229: DETAIL: memcache blocks :64 Sep 1 11:09:38 centos8i1 pgpool[12229]: [3-1] pid 12229: LOG: pool_discard_oid_maps: discarded memqcache oid maps Sep 1 11:09:38 centos8i1 pgpool[12229]: [4-1] pid 12229: LOG: waiting for watchdog to initialize Sep 1 11:09:38 centos8i1 pgpool[12231]: [4-1] pid 12231: LOG: setting the local watchdog node name to "centos8i1-int:5432 Linux centos8i1.localdomain" Sep 1 11:09:38 centos8i1 pgpool[12231]: [5-1] pid 12231: LOG: watchdog cluster is configured with 2 remote nodes Sep 1 11:09:38 centos8i1 pgpool[12231]: [6-1] pid 12231: LOG: watchdog remote node:0 on centos8i2-int:9000 Sep 1 11:09:38 centos8i1 pgpool[12231]: [7-1] pid 12231: LOG: watchdog remote node:1 on centos8i3-int:9000 Sep 1 11:09:38 centos8i1 pgpool[12231]: [8-1] pid 12231: LOG: interface monitoring is disabled in watchdog Sep 1 11:09:38 centos8i1 pgpool[12231]: [9-1] pid 12231: LOG: watchdog node state changed from [DEAD] to [LOADING] Sep 1 11:09:38 centos8i1 pgpool[12231]: [10-1] pid 12231: LOG: new watchdog node connection is received from "10.0.0.3:54980" Sep 1 11:09:38 centos8i1 pgpool[12231]: [11-1] pid 12231: LOG: new outbound connection to centos8i3-int:9000 Sep 1 11:09:38 centos8i1 pgpool[12231]: [12-1] pid 12231: LOG: new node joined the cluster hostname:"centos8i3-int" port:9000 pgpool_port:5432 Sep 1 11:09:38 centos8i1 pgpool[12231]: [12-2] pid 12231: DETAIL: Pgpool-II version:"4.1.2" watchdog messaging version: 1.1 Sep 1 11:09:38 centos8i1 pgpool[12231]: [13-1] pid 12231: LOG: new watchdog node connection is received from "10.0.0.2:29321" Sep 1 11:09:38 centos8i1 pgpool[12231]: [14-1] pid 12231: LOG: new node joined the cluster hostname:"centos8i2-int" port:9000 pgpool_port:5432 Sep 1 11:09:38 centos8i1 pgpool[12231]: [14-2] pid 12231: DETAIL: Pgpool-II version:"4.1.2" watchdog messaging version: 1.1 Sep 1 11:09:43 centos8i1 pgpool[12231]: [15-1] pid 12231: LOG: watchdog node state changed from [LOADING] to [INITIALIZING] Sep 1 11:09:43 centos8i1 pgpool[12231]: [16-1] pid 12231: LOG: watchdog node state changed from [INITIALIZING] to [STANDING FOR MASTER] Sep 1 11:09:43 centos8i1 pgpool[12231]: [17-1] pid 12231: LOG: our stand for coordinator request is rejected by node "centos8i3-int:5432 Linux centos8i3.localdomain" Sep 1 11:09:43 centos8i1 pgpool[12231]: [18-1] pid 12231: LOG: watchdog node state changed from [STANDING FOR MASTER] to [PARTICIPATING IN ELECTION] Sep 1 11:09:48 centos8i1 pgpool[12231]: [19-1] pid 12231: LOG: watchdog node state changed from [PARTICIPATING IN ELECTION] to [JOINING] Sep 1 11:09:48 centos8i1 pgpool[12231]: [20-1] pid 12231: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] Sep 1 11:09:49 centos8i1 pgpool[12231]: [21-1] pid 12231: LOG: watchdog node state changed from [INITIALIZING] to [STANDING FOR MASTER] Sep 1 11:09:49 centos8i1 pgpool[12231]: [22-1] pid 12231: LOG: our stand for coordinator request is rejected by node "centos8i2-int:5432 Linux centos8i2.localdomain" Sep 1 11:09:49 centos8i1 pgpool[12231]: [23-1] pid 12231: LOG: watchdog node state changed from [STANDING FOR MASTER] to [PARTICIPATING IN ELECTION] Sep 1 11:09:49 centos8i1 pgpool[12231]: [24-1] pid 12231: LOG: new outbound connection to centos8i2-int:9000 Sep 1 11:09:54 centos8i1 pgpool[12231]: [25-1] pid 12231: LOG: watchdog node state changed from [PARTICIPATING IN ELECTION] to [JOINING] Sep 1 11:09:54 centos8i1 pgpool[12231]: [26-1] pid 12231: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] Sep 1 11:09:55 centos8i1 pgpool[12231]: [27-1] pid 12231: LOG: watchdog node state changed from [INITIALIZING] to [STANDING FOR MASTER] Sep 1 11:09:55 centos8i1 pgpool[12231]: [28-1] pid 12231: LOG: our stand for coordinator request is rejected by node "centos8i2-int:5432 Linux centos8i2.localdomain" Sep 1 11:09:55 centos8i1 pgpool[12231]: [29-1] pid 12231: LOG: watchdog node state changed from [STANDING FOR MASTER] to [PARTICIPATING IN ELECTION] Sep 1 11:10:00 centos8i1 pgpool[12231]: [30-1] pid 12231: LOG: watchdog node state changed from [PARTICIPATING IN ELECTION] to [JOINING] Sep 1 11:10:00 centos8i1 pgpool[12231]: [31-1] pid 12231: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] Sep 1 11:10:01 centos8i1 pgpool[12231]: [32-1] pid 12231: LOG: watchdog node state changed from [INITIALIZING] to [STANDING FOR MASTER] Sep 1 11:10:01 centos8i1 pgpool[12231]: [33-1] pid 12231: LOG: our stand for coordinator request is rejected by node "centos8i2-int:5432 Linux centos8i2.localdomain" Sep 1 11:10:01 centos8i1 pgpool[12231]: [34-1] pid 12231: LOG: watchdog node state changed from [STANDING FOR MASTER] to [PARTICIPATING IN ELECTION] Sep 1 11:10:05 centos8i1 pgpool[12231]: [35-1] pid 12231: LOG: watchdog node state changed from [PARTICIPATING IN ELECTION] to [INITIALIZING] Sep 1 11:10:05 centos8i1 pgpool[12231]: [36-1] pid 12231: LOG: setting the remote node "centos8i2-int:5432 Linux centos8i2.localdomain" as watchdog cluster master Sep 1 11:10:06 centos8i1 pgpool[12231]: [37-1] pid 12231: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] Sep 1 11:10:06 centos8i1 pgpool[12231]: [38-1] pid 12231: LOG: successfully joined the watchdog cluster as standby node Sep 1 11:10:06 centos8i1 pgpool[12229]: [5-1] pid 12229: LOG: watchdog process is initialized Sep 1 11:10:06 centos8i1 pgpool[12231]: [38-2] pid 12231: DETAIL: our join coordinator request is accepted by cluster leader node "centos8i2-int:5432 Linux centos8i2.localdomain" Sep 1 11:10:06 centos8i1 pgpool[12229]: [5-2] pid 12229: DETAIL: watchdog messaging data version: 1.1 Sep 1 11:10:06 centos8i1 pgpool[12229]: [6-1] pid 12229: LOG: Pgpool-II parent process received watchdog quorum change signal from watchdog Sep 1 11:10:06 centos8i1 pgpool[12231]: [39-1] pid 12231: LOG: new IPC connection received Sep 1 11:10:06 centos8i1 pgpool[12231]: [40-1] pid 12231: LOG: new IPC connection received Sep 1 11:10:06 centos8i1 pgpool[12229]: [7-1] pid 12229: LOG: watchdog cluster now holds the quorum Sep 1 11:10:06 centos8i1 pgpool[12229]: [7-2] pid 12229: DETAIL: updating the state of quarantine backend nodes Sep 1 11:10:06 centos8i1 pgpool[12259]: [6-1] pid 12259: LOG: 3 watchdog nodes are configured for lifecheck Sep 1 11:10:06 centos8i1 pgpool[12259]: [7-1] pid 12259: LOG: watchdog nodes ID:0 Name:"centos8i1-int:5432 Linux centos8i1.localdomain" Sep 1 11:10:06 centos8i1 pgpool[12259]: [7-2] pid 12259: DETAIL: Host:"centos8i1-int" WD Port:9000 pgpool-II port:5432 Sep 1 11:10:06 centos8i1 pgpool[12259]: [8-1] pid 12259: LOG: watchdog nodes ID:1 Name:"centos8i2-int:5432 Linux centos8i2.localdomain" Sep 1 11:10:06 centos8i1 pgpool[12259]: [8-2] pid 12259: DETAIL: Host:"centos8i2-int" WD Port:9000 pgpool-II port:5432 Sep 1 11:10:06 centos8i1 pgpool[12259]: [9-1] pid 12259: LOG: watchdog nodes ID:2 Name:"centos8i3-int:5432 Linux centos8i3.localdomain" Sep 1 11:10:06 centos8i1 pgpool[12259]: [9-2] pid 12259: DETAIL: Host:"centos8i3-int" WD Port:9000 pgpool-II port:5432 Sep 1 11:10:06 centos8i1 pgpool[12231]: [41-1] pid 12231: LOG: new IPC connection received Sep 1 11:10:06 centos8i1 pgpool[12231]: [42-1] pid 12231: LOG: new IPC connection received Sep 1 11:10:06 centos8i1 pgpool[12229]: [8-1] pid 12229: LOG: we have joined the watchdog cluster as STANDBY node Sep 1 11:10:06 centos8i1 pgpool[12229]: [8-2] pid 12229: DETAIL: syncing the backend states from the MASTER watchdog node Sep 1 11:10:06 centos8i1 pgpool[12231]: [43-1] pid 12231: LOG: new IPC connection received Sep 1 11:10:06 centos8i1 pgpool[12231]: [44-1] pid 12231: LOG: received the get data request from local pgpool-II on IPC interface Sep 1 11:10:06 centos8i1 pgpool[12231]: [45-1] pid 12231: LOG: get data request from local pgpool-II node received on IPC interface is forwarded to master watchdog node "centos8i2-int:5432 Linux centos8i2.localdomain" Sep 1 11:10:06 centos8i1 pgpool[12231]: [45-2] pid 12231: DETAIL: waiting for the reply... Sep 1 11:10:06 centos8i1 pgpool[12229]: [9-1] pid 12229: LOG: master watchdog node "centos8i2-int:5432 Linux centos8i2.localdomain" returned status for 3 backend nodes Sep 1 11:10:06 centos8i1 pgpool[12229]: [10-1] pid 12229: LOG: Setting up socket for 0.0.0.0:5432 Sep 1 11:10:06 centos8i1 pgpool[12229]: [11-1] pid 12229: LOG: Setting up socket for :::5432 Sep 1 11:10:06 centos8i1 pgpool[12296]: [12-1] pid 12296: LOG: PCP process: 12296 started Sep 1 11:10:06 centos8i1 pgpool[12229]: [12-1] pid 12229: LOG: pgpool-II successfully started. version 4.1.2 (karasukiboshi) Sep 1 11:10:06 centos8i1 pgpool[12229]: [13-1] pid 12229: LOG: node status[0]: 0 Sep 1 11:10:06 centos8i1 pgpool[12229]: [14-1] pid 12229: LOG: node status[1]: 0 Sep 1 11:10:06 centos8i1 pgpool[12229]: [15-1] pid 12229: LOG: node status[2]: 0 Sep 1 11:10:07 centos8i1 pgpool[12261]: [10-1] pid 12261: LOG: set SO_REUSEPORT option to the socket Sep 1 11:10:07 centos8i1 pgpool[12261]: [11-1] pid 12261: LOG: creating socket for sending heartbeat Sep 1 11:10:07 centos8i1 pgpool[12261]: [11-2] pid 12261: DETAIL: set SO_REUSEPORT Sep 1 11:10:07 centos8i1 pgpool[12262]: [10-1] pid 12262: LOG: set SO_REUSEPORT option to the socket Sep 1 11:10:07 centos8i1 pgpool[12262]: [11-1] pid 12262: LOG: creating watchdog heartbeat receive socket. Sep 1 11:10:07 centos8i1 pgpool[12262]: [11-2] pid 12262: DETAIL: set SO_REUSEPORT Sep 1 11:10:07 centos8i1 pgpool[12263]: [10-1] pid 12263: LOG: set SO_REUSEPORT option to the socket Sep 1 11:10:07 centos8i1 pgpool[12263]: [11-1] pid 12263: LOG: creating socket for sending heartbeat Sep 1 11:10:07 centos8i1 pgpool[12263]: [11-2] pid 12263: DETAIL: set SO_REUSEPORT Sep 1 11:10:07 centos8i1 pgpool[12260]: [10-1] pid 12260: LOG: set SO_REUSEPORT option to the socket Sep 1 11:10:07 centos8i1 pgpool[12260]: [11-1] pid 12260: LOG: creating watchdog heartbeat receive socket. Sep 1 11:10:07 centos8i1 pgpool[12260]: [11-2] pid 12260: DETAIL: set SO_REUSEPORT ***** SHUTDOWN NODE centos8i2-int ***** Sep 1 11:19:37 centos8i1 pgpool[12299]: [12-1] pid 12299: ERROR: failed to authenticate Sep 1 11:19:37 centos8i1 pgpool[12299]: [12-2] pid 12299: DETAIL: the database system is shutting down Sep 1 11:19:37 centos8i1 pgpool[12299]: [13-1] pid 12299: LOG: health check retrying on DB node: 1 (round:1) Sep 1 11:19:37 centos8i1 pgpool[12231]: [46-1] pid 12231: LOG: remote node "centos8i2-int:5432 Linux centos8i2.localdomain" is shutting down Sep 1 11:19:37 centos8i1 pgpool[12231]: [47-1] pid 12231: LOG: watchdog cluster has lost the coordinator node Sep 1 11:19:37 centos8i1 pgpool[12231]: [48-1] pid 12231: LOG: removing the remote node "centos8i2-int:5432 Linux centos8i2.localdomain" from watchdog cluster master Sep 1 11:19:37 centos8i1 pgpool[12231]: [49-1] pid 12231: LOG: We have lost the cluster master node "centos8i2-int:5432 Linux centos8i2.localdomain" Sep 1 11:19:37 centos8i1 pgpool[12231]: [50-1] pid 12231: LOG: watchdog node state changed from [STANDBY] to [JOINING] Sep 1 11:19:37 centos8i1 pgpool[12231]: [51-1] pid 12231: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] Sep 1 11:19:38 centos8i1 pgpool[12299]: [14-1] pid 12299: LOG: failed to connect to PostgreSQL server on "centos8i2-int:5433", getsockopt() detected error "Connection refused" Sep 1 11:19:38 centos8i1 pgpool[12299]: [15-1] pid 12299: ERROR: failed to make persistent db connection Sep 1 11:19:38 centos8i1 pgpool[12299]: [15-2] pid 12299: DETAIL: connection to host:"centos8i2-int:5433" failed Sep 1 11:19:38 centos8i1 pgpool[12299]: [16-1] pid 12299: LOG: health check retrying on DB node: 1 (round:2) Sep 1 11:19:38 centos8i1 pgpool[12231]: [52-1] pid 12231: LOG: watchdog node state changed from [INITIALIZING] to [STANDING FOR MASTER] Sep 1 11:19:38 centos8i1 pgpool[12231]: [53-1] pid 12231: LOG: our stand for coordinator request is rejected by node "centos8i3-int:5432 Linux centos8i3.localdomain" Sep 1 11:19:38 centos8i1 pgpool[12231]: [54-1] pid 12231: LOG: watchdog node state changed from [STANDING FOR MASTER] to [PARTICIPATING IN ELECTION] Sep 1 11:19:39 centos8i1 pgpool[12299]: [17-1] pid 12299: LOG: failed to connect to PostgreSQL server on "centos8i2-int:5433", getsockopt() detected error "Connection refused" Sep 1 11:19:39 centos8i1 pgpool[12299]: [18-1] pid 12299: ERROR: failed to make persistent db connection Sep 1 11:19:39 centos8i1 pgpool[12299]: [18-2] pid 12299: DETAIL: connection to host:"centos8i2-int:5433" failed Sep 1 11:19:39 centos8i1 pgpool[12299]: [19-1] pid 12299: LOG: health check retrying on DB node: 1 (round:3) Sep 1 11:19:39 centos8i1 pgpool[12297]: [12-1] pid 12297: LOG: failed to connect to PostgreSQL server on "centos8i2-int:5433", getsockopt() detected error "Connection refused" Sep 1 11:19:39 centos8i1 pgpool[12297]: [13-1] pid 12297: ERROR: failed to make persistent db connection Sep 1 11:19:39 centos8i1 pgpool[12297]: [13-2] pid 12297: DETAIL: connection to host:"centos8i2-int:5433" failed Sep 1 11:19:40 centos8i1 pgpool[12299]: [20-1] pid 12299: LOG: failed to connect to PostgreSQL server on "centos8i2-int:5433", getsockopt() detected error "Connection refused" Sep 1 11:19:40 centos8i1 pgpool[12299]: [21-1] pid 12299: ERROR: failed to make persistent db connection Sep 1 11:19:40 centos8i1 pgpool[12299]: [21-2] pid 12299: DETAIL: connection to host:"centos8i2-int:5433" failed Sep 1 11:19:40 centos8i1 pgpool[12299]: [22-1] pid 12299: LOG: health check failed on node 1 (timeout:0) Sep 1 11:19:40 centos8i1 pgpool[12299]: [23-1] pid 12299: LOG: received degenerate backend request for node_id: 1 from pid [12299] Sep 1 11:19:40 centos8i1 pgpool[12231]: [55-1] pid 12231: LOG: new IPC connection received Sep 1 11:19:40 centos8i1 pgpool[12231]: [56-1] pid 12231: LOG: unable to process the failover request received on IPC interface Sep 1 11:19:40 centos8i1 pgpool[12231]: [56-2] pid 12231: DETAIL: this watchdog node has not joined the cluster yet Sep 1 11:19:40 centos8i1 pgpool[12231]: [56-3] pid 12231: HINT: try again in few seconds Sep 1 11:19:40 centos8i1 pgpool[12299]: [24-1] pid 12299: LOG: degenerate backend request for 1 node(s) from pid [12299] is canceled by other pgpool Sep 1 11:19:43 centos8i1 pgpool[12231]: [57-1] pid 12231: LOG: watchdog node state changed from [PARTICIPATING IN ELECTION] to [JOINING] Sep 1 11:19:43 centos8i1 pgpool[12231]: [58-1] pid 12231: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] Sep 1 11:19:44 centos8i1 pgpool[12231]: [59-1] pid 12231: LOG: watchdog node state changed from [INITIALIZING] to [STANDING FOR MASTER] Sep 1 11:19:44 centos8i1 pgpool[12231]: [60-1] pid 12231: LOG: our stand for coordinator request is rejected by node "centos8i3-int:5432 Linux centos8i3.localdomain" Sep 1 11:19:44 centos8i1 pgpool[12231]: [61-1] pid 12231: LOG: watchdog node state changed from [STANDING FOR MASTER] to [PARTICIPATING IN ELECTION] Sep 1 11:19:49 centos8i1 pgpool[12231]: [62-1] pid 12231: LOG: watchdog node state changed from [PARTICIPATING IN ELECTION] to [JOINING] Sep 1 11:19:49 centos8i1 pgpool[12231]: [63-1] pid 12231: LOG: watchdog node state changed from [JOINING] to [INITIALIZING] Sep 1 11:19:49 centos8i1 pgpool[12297]: [14-1] pid 12297: ERROR: Failed to check replication time lag Sep 1 11:19:49 centos8i1 pgpool[12297]: [14-2] pid 12297: DETAIL: No persistent db connection for the node 1 Sep 1 11:19:49 centos8i1 pgpool[12297]: [14-3] pid 12297: HINT: check sr_check_user and sr_check_password Sep 1 11:19:49 centos8i1 pgpool[12297]: [14-4] pid 12297: CONTEXT: while checking replication time lag Sep 1 11:19:50 centos8i1 pgpool[12231]: [64-1] pid 12231: LOG: watchdog node state changed from [INITIALIZING] to [STANDING FOR MASTER] Sep 1 11:19:50 centos8i1 pgpool[12231]: [65-1] pid 12231: LOG: our stand for coordinator request is rejected by node "centos8i3-int:5432 Linux centos8i3.localdomain" Sep 1 11:19:50 centos8i1 pgpool[12231]: [66-1] pid 12231: LOG: watchdog node state changed from [STANDING FOR MASTER] to [PARTICIPATING IN ELECTION] Sep 1 11:19:54 centos8i1 pgpool[12231]: [67-1] pid 12231: LOG: watchdog node state changed from [PARTICIPATING IN ELECTION] to [INITIALIZING] Sep 1 11:19:54 centos8i1 pgpool[12231]: [68-1] pid 12231: LOG: setting the remote node "centos8i3-int:5432 Linux centos8i3.localdomain" as watchdog cluster master Sep 1 11:19:55 centos8i1 pgpool[12299]: [25-1] pid 12299: LOG: failed to connect to PostgreSQL server on "centos8i2-int:5433", timed out Sep 1 11:19:55 centos8i1 pgpool[12299]: [26-1] pid 12299: ERROR: failed to make persistent db connection Sep 1 11:19:55 centos8i1 pgpool[12299]: [26-2] pid 12299: DETAIL: connection to host:"centos8i2-int:5433" failed Sep 1 11:19:55 centos8i1 pgpool[12299]: [27-1] pid 12299: LOG: health check retrying on DB node: 1 (round:1) Sep 1 11:19:55 centos8i1 pgpool[12231]: [69-1] pid 12231: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] Sep 1 11:19:55 centos8i1 pgpool[12231]: [70-1] pid 12231: LOG: successfully joined the watchdog cluster as standby node Sep 1 11:19:55 centos8i1 pgpool[12231]: [70-2] pid 12231: DETAIL: our join coordinator request is accepted by cluster leader node "centos8i3-int:5432 Linux centos8i3.localdomain" Sep 1 11:19:55 centos8i1 pgpool[12231]: [71-1] pid 12231: LOG: new IPC connection received Sep 1 11:19:55 centos8i1 pgpool[12229]: [16-1] pid 12229: LOG: we have joined the watchdog cluster as STANDBY node Sep 1 11:19:55 centos8i1 pgpool[12229]: [16-2] pid 12229: DETAIL: syncing the backend states from the MASTER watchdog node Sep 1 11:19:55 centos8i1 pgpool[12231]: [72-1] pid 12231: LOG: new IPC connection received Sep 1 11:19:55 centos8i1 pgpool[12231]: [73-1] pid 12231: LOG: received the get data request from local pgpool-II on IPC interface Sep 1 11:19:55 centos8i1 pgpool[12231]: [74-1] pid 12231: LOG: get data request from local pgpool-II node received on IPC interface is forwarded to master watchdog node "centos8i3-int:5432 Linux centos8i3.localdomain" Sep 1 11:19:55 centos8i1 pgpool[12231]: [74-2] pid 12231: DETAIL: waiting for the reply... Sep 1 11:19:55 centos8i1 pgpool[12229]: [17-1] pid 12229: LOG: master watchdog node "centos8i3-int:5432 Linux centos8i3.localdomain" returned status for 3 backend nodes Sep 1 11:19:55 centos8i1 pgpool[12229]: [18-1] pid 12229: LOG: backend nodes status remains same after the sync from "centos8i3-int:5432 Linux centos8i3.localdomain" Sep 1 11:19:55 centos8i1 pgpool[12229]: [19-1] pid 12229: LOG: Pgpool-II parent process received watchdog quorum change signal from watchdog Sep 1 11:19:55 centos8i1 pgpool[12231]: [75-1] pid 12231: LOG: new IPC connection received Sep 1 11:19:55 centos8i1 pgpool[12229]: [20-1] pid 12229: LOG: watchdog cluster now holds the quorum Sep 1 11:19:55 centos8i1 pgpool[12229]: [20-2] pid 12229: DETAIL: updating the state of quarantine backend nodes Sep 1 11:19:55 centos8i1 pgpool[12231]: [76-1] pid 12231: LOG: new IPC connection received Sep 1 11:19:59 centos8i1 pgpool[12297]: [15-1] pid 12297: LOG: trying connecting to PostgreSQL server on "centos8i2-int:5433" by INET socket Sep 1 11:19:59 centos8i1 pgpool[12297]: [15-2] pid 12297: DETAIL: timed out. retrying... Sep 1 11:20:06 centos8i1 pgpool[12299]: [28-1] pid 12299: LOG: failed to connect to PostgreSQL server on "centos8i2-int:5433", timed out Sep 1 11:20:06 centos8i1 pgpool[12299]: [29-1] pid 12299: ERROR: failed to make persistent db connection Sep 1 11:20:06 centos8i1 pgpool[12299]: [29-2] pid 12299: DETAIL: connection to host:"centos8i2-int:5433" failed Sep 1 11:20:06 centos8i1 pgpool[12299]: [30-1] pid 12299: LOG: health check retrying on DB node: 1 (round:2) Sep 1 11:20:06 centos8i1 pgpool[12297]: [16-1] pid 12297: LOG: failed to connect to PostgreSQL server on "centos8i2-int:5433", getsockopt() detected error "No route to host" Sep 1 11:20:06 centos8i1 pgpool[12297]: [17-1] pid 12297: ERROR: failed to make persistent db connection Sep 1 11:20:06 centos8i1 pgpool[12297]: [17-2] pid 12297: DETAIL: connection to host:"centos8i2-int:5433" failed Sep 1 11:20:09 centos8i1 pgpool[12299]: [31-1] pid 12299: LOG: failed to connect to PostgreSQL server on "centos8i2-int:5433", getsockopt() detected error "No route to host" Sep 1 11:20:09 centos8i1 pgpool[12299]: [32-1] pid 12299: ERROR: failed to make persistent db connection Sep 1 11:20:09 centos8i1 pgpool[12299]: [32-2] pid 12299: DETAIL: connection to host:"centos8i2-int:5433" failed Sep 1 11:20:09 centos8i1 pgpool[12299]: [33-1] pid 12299: LOG: health check retrying on DB node: 1 (round:3) Sep 1 11:20:12 centos8i1 pgpool[12299]: [34-1] pid 12299: LOG: failed to connect to PostgreSQL server on "centos8i2-int:5433", getsockopt() detected error "No route to host" Sep 1 11:20:12 centos8i1 pgpool[12299]: [35-1] pid 12299: ERROR: failed to make persistent db connection Sep 1 11:20:12 centos8i1 pgpool[12299]: [35-2] pid 12299: DETAIL: connection to host:"centos8i2-int:5433" failed Sep 1 11:20:12 centos8i1 pgpool[12299]: [36-1] pid 12299: LOG: health check failed on node 1 (timeout:0) Sep 1 11:20:12 centos8i1 pgpool[12299]: [37-1] pid 12299: LOG: received degenerate backend request for node_id: 1 from pid [12299] Sep 1 11:20:12 centos8i1 pgpool[12231]: [77-1] pid 12231: LOG: new IPC connection received Sep 1 11:20:12 centos8i1 pgpool[12231]: [78-1] pid 12231: LOG: failover request from local pgpool-II node received on IPC interface is forwarded to master watchdog node "centos8i3-int:5432 Linux centos8i3.localdomain" Sep 1 11:20:12 centos8i1 pgpool[12231]: [78-2] pid 12231: DETAIL: waiting for the reply... Sep 1 11:20:12 centos8i1 pgpool[12231]: [79-1] pid 12231: LOG: remote node "centos8i3-int:5432 Linux centos8i3.localdomain" is asking to inform about quarantined backend nodes Sep 1 11:20:12 centos8i1 pgpool[12229]: [21-1] pid 12229: LOG: Pgpool-II parent process received inform quarantine nodes signal from watchdog Sep 1 11:20:12 centos8i1 pgpool[12299]: [38-1] pid 12299: LOG: degenerate backend request for node_id: 1 from pid [12299], will be handled by watchdog, which is building consensus for request Sep 1 11:20:16 centos8i1 pgpool[12297]: [18-1] pid 12297: ERROR: Failed to check replication time lag Sep 1 11:20:16 centos8i1 pgpool[12297]: [18-2] pid 12297: DETAIL: No persistent db connection for the node 1 Sep 1 11:20:16 centos8i1 pgpool[12297]: [18-3] pid 12297: HINT: check sr_check_user and sr_check_password Sep 1 11:20:16 centos8i1 pgpool[12297]: [18-4] pid 12297: CONTEXT: while checking replication time lag Sep 1 11:20:17 centos8i1 pgpool[12229]: [22-1] pid 12229: LOG: Pgpool-II parent process received sync backend signal from watchdog Sep 1 11:20:17 centos8i1 pgpool[12231]: [80-1] pid 12231: LOG: new IPC connection received Sep 1 11:20:17 centos8i1 pgpool[12229]: [23-1] pid 12229: LOG: master watchdog has performed failover Sep 1 11:20:17 centos8i1 pgpool[12229]: [23-2] pid 12229: DETAIL: syncing the backend states from the MASTER watchdog node Sep 1 11:20:17 centos8i1 pgpool[12231]: [81-1] pid 12231: LOG: new IPC connection received Sep 1 11:20:17 centos8i1 pgpool[12231]: [82-1] pid 12231: LOG: received the get data request from local pgpool-II on IPC interface Sep 1 11:20:17 centos8i1 pgpool[12231]: [83-1] pid 12231: LOG: get data request from local pgpool-II node received on IPC interface is forwarded to master watchdog node "centos8i3-int:5432 Linux centos8i3.localdomain" Sep 1 11:20:17 centos8i1 pgpool[12231]: [83-2] pid 12231: DETAIL: waiting for the reply... Sep 1 11:20:17 centos8i1 pgpool[12229]: [24-1] pid 12229: LOG: master watchdog node "centos8i3-int:5432 Linux centos8i3.localdomain" returned status for 3 backend nodes Sep 1 11:20:17 centos8i1 pgpool[12229]: [25-1] pid 12229: LOG: backend:1 is set to down status Sep 1 11:20:17 centos8i1 pgpool[12229]: [25-2] pid 12229: DETAIL: backend:1 is DOWN on cluster master "centos8i3-int:5432 Linux centos8i3.localdomain" Sep 1 11:20:17 centos8i1 pgpool[12229]: [26-1] pid 12229: LOG: 1 backend node(s) were detached because of backend status sync from "centos8i3-int:5432 Linux centos8i3.localdomain" Sep 1 11:20:17 centos8i1 pgpool[12229]: [26-2] pid 12229: DETAIL: restarting the children processes Sep 1 11:20:17 centos8i1 pgpool[12229]: [27-1] pid 12229: LOG: Node 0 is not down (status: 1) Sep 1 11:20:17 centos8i1 pgpool[12297]: [19-1] pid 12297: LOG: trying to connect to PostgreSQL server on "centos8i2-int:5433" using INET socket Sep 1 11:20:17 centos8i1 pgpool[12297]: [19-2] pid 12297: DETAIL: select() interrupted. retrying... Sep 1 11:20:19 centos8i1 pgpool[12297]: [20-1] pid 12297: LOG: failed to connect to PostgreSQL server on "centos8i2-int:5433", getsockopt() detected error "No route to host" Sep 1 11:20:19 centos8i1 pgpool[12297]: [21-1] pid 12297: ERROR: failed to make persistent db connection Sep 1 11:20:19 centos8i1 pgpool[12297]: [21-2] pid 12297: DETAIL: connection to host:"centos8i2-int:5433" failed Sep 1 11:20:29 centos8i1 pgpool[12297]: [22-1] pid 12297: ERROR: Failed to check replication time lag Sep 1 11:20:29 centos8i1 pgpool[12297]: [22-2] pid 12297: DETAIL: No persistent db connection for the node 1 Sep 1 11:20:29 centos8i1 pgpool[12297]: [22-3] pid 12297: HINT: check sr_check_user and sr_check_password Sep 1 11:20:29 centos8i1 pgpool[12297]: [22-4] pid 12297: CONTEXT: while checking replication time lag Sep 1 11:20:29 centos8i1 pgpool[12297]: [23-1] pid 12297: LOG: worker process received restart request Sep 1 11:20:29 centos8i1 pgpool[12229]: [28-1] pid 12229: LOG: worker child process with pid: 12297 exits with status 256 Sep 1 11:20:29 centos8i1 pgpool[12229]: [29-1] pid 12229: LOG: fork a new worker child process with pid: 12913 Sep 1 11:21:43 centos8i1 pgpool[12296]: [13-1] pid 12296: LOG: forked new pcp worker, pid=12963 socket=9 Sep 1 11:21:43 centos8i1 pgpool[12231]: [84-1] pid 12231: LOG: new IPC connection received Sep 1 11:21:43 centos8i1 pgpool[12296]: [14-1] pid 12296: LOG: PCP process with pid: 12963 exit with SUCCESS. Sep 1 11:21:43 centos8i1 pgpool[12296]: [15-1] pid 12296: LOG: PCP process with pid: 12963 exits with status 0