May 1 09:31:46 host pgpool[9741]: [1862-1] 2019-05-01 09:31:46: pid 9741: FATAL: pgpool is not accepting any new connections May 1 09:31:46 host pgpool[9741]: [1862-2] 2019-05-01 09:31:46: pid 9741: DETAIL: all backend nodes are down, pgpool requires at least one valid node May 1 09:31:46 host pgpool[9741]: [1862-3] 2019-05-01 09:31:46: pid 9741: HINT: repair the backend nodes and restart pgpool May 1 09:31:46 host pgpool[1701]: [1877-1] 2019-05-01 09:31:46: pid 1701: LOG: child process with pid: 9741 exits with status 256 May 1 09:31:46 host pgpool[1701]: [1878-1] 2019-05-01 09:31:46: pid 1701: LOG: fork a new child process with pid: 10794 May 1 09:32:09 host pgpool[10911]: [1-1] 2019-05-01 09:32:09: pid 10911: LOG: stop request sent to pgpool. waiting for termination... May 1 09:32:09 host pgpool[1702]: [57-1] 2019-05-01 09:32:09: pid 1702: LOG: Watchdog is shutting down May 1 09:32:09 host pgpool[10912]: [58-1] 2019-05-01 09:32:09: pid 10912: LOG: watchdog: de-escalation started May 1 09:32:09 host pgpool[10912]: [59-1] 2019-05-01 09:32:09: pid 10912: LOG: successfully released the delegate IP:"10.1.24.15" May 1 09:32:09 host pgpool[10912]: [59-2] 2019-05-01 09:32:09: pid 10912: DETAIL: 'if_down_cmd' returned with success May 1 09:32:09 host pgpool[1701]: [1879-1] 2019-05-01 09:32:09: pid 1701: WARNING: All the DB nodes are in down status and skip writing status file. May 1 09:32:09 host systemd-logind: Removed session 25180. May 1 09:32:18 host pgpool[10924]: [1-1] 2019-05-01 09:32:18: pid 10924: LOG: Backend status file /var/log//pgpool_status does not exist May 1 09:32:18 host pgpool[10924]: [2-1] 2019-05-01 09:32:18: pid 10924: LOG: waiting for watchdog to initialize May 1 09:32:18 host pgpool[10925]: [2-1] 2019-05-01 09:32:18: pid 10925: LOG: setting the local watchdog node name to "host.adm.cacc.ch:5432 Linux host.adm.cacc.ch" May 1 09:32:18 host pgpool[10925]: [3-1] 2019-05-01 09:32:18: pid 10925: LOG: watchdog cluster is configured with 1 remote nodes May 1 09:32:18 host pgpool[10925]: [4-1] 2019-05-01 09:32:18: pid 10925: LOG: watchdog remote node:0 on asa-pgpool02.adm.cacc.ch:9000 May 1 09:32:18 host pgpool[10925]: [5-1] 2019-05-01 09:32:18: pid 10925: LOG: interface monitoring is disabled in watchdog May 1 09:32:18 host pgpool[10925]: [6-1] 2019-05-01 09:32:18: pid 10925: LOG: watchdog node state changed from [DEAD] to [LOADING] May 1 09:32:18 host pgpool[10925]: [7-1] 2019-05-01 09:32:18: pid 10925: LOG: new outbound connection to asa-pgpool02.adm.cacc.ch:9000 May 1 09:32:18 host pgpool[10925]: [8-1] 2019-05-01 09:32:18: pid 10925: LOG: setting the remote node "asa-pgpool02.adm.cacc.ch:5432 Linux asa-pgpool02.adm.cacc.ch" as watchdog cluster master May 1 09:32:18 host pgpool[10925]: [9-1] 2019-05-01 09:32:18: pid 10925: LOG: watchdog node state changed from [LOADING] to [INITIALIZING] May 1 09:32:18 host pgpool[10925]: [10-1] 2019-05-01 09:32:18: pid 10925: LOG: new watchdog node connection is received from "10.1.24.12:45738" May 1 09:32:18 host pgpool[10925]: [11-1] 2019-05-01 09:32:18: pid 10925: LOG: new node joined the cluster hostname:"asa-pgpool02.adm.cacc.ch" port:9000 pgpool_port:5432 May 1 09:32:19 host pgpool[10925]: [12-1] 2019-05-01 09:32:19: pid 10925: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] May 1 09:32:19 host pgpool[10925]: [13-1] 2019-05-01 09:32:19: pid 10925: LOG: successfully joined the watchdog cluster as standby node May 1 09:32:19 host pgpool[10925]: [13-2] 2019-05-01 09:32:19: pid 10925: DETAIL: our join coordinator request is accepted by cluster leader node "asa-pgpool02.adm.cacc.ch:5432 Linux asa-pgpool02.adm.cacc.ch" May 1 09:32:19 host pgpool[10924]: [3-1] 2019-05-01 09:32:19: pid 10924: LOG: watchdog process is initialized May 1 09:32:19 host pgpool[10924]: [4-1] 2019-05-01 09:32:19: pid 10924: LOG: Pgpool-II parent process received watchdog quorum change signal from watchdog May 1 09:32:19 host pgpool[10925]: [14-1] 2019-05-01 09:32:19: pid 10925: LOG: new IPC connection received May 1 09:32:19 host pgpool[10924]: [5-1] 2019-05-01 09:32:19: pid 10924: LOG: watchdog cluster now holds the quorum May 1 09:32:19 host pgpool[10924]: [5-2] 2019-05-01 09:32:19: pid 10924: DETAIL: updating the state of quarantine backend nodes May 1 09:32:19 host pgpool[10925]: [15-1] 2019-05-01 09:32:19: pid 10925: LOG: new IPC connection received May 1 09:32:19 host pgpool[10925]: [16-1] 2019-05-01 09:32:19: pid 10925: LOG: new IPC connection received May 1 09:32:19 host pgpool[10924]: [6-1] 2019-05-01 09:32:19: pid 10924: LOG: we have joined the watchdog cluster as STANDBY node May 1 09:32:19 host pgpool[10924]: [6-2] 2019-05-01 09:32:19: pid 10924: DETAIL: syncing the backend states from the MASTER watchdog node May 1 09:32:19 host pgpool[10925]: [17-1] 2019-05-01 09:32:19: pid 10925: LOG: new IPC connection received May 1 09:32:19 host pgpool[10925]: [18-1] 2019-05-01 09:32:19: pid 10925: LOG: received the get data request from local pgpool-II on IPC interface May 1 09:32:19 host pgpool[10925]: [19-1] 2019-05-01 09:32:19: pid 10925: LOG: get data request from local pgpool-II node received on IPC interface is forwarded to master watchdog node "asa-pgpool02.adm.cacc.ch:5432 Linux asa-pgpool02.adm.cacc.ch" May 1 09:32:19 host pgpool[10925]: [19-2] 2019-05-01 09:32:19: pid 10925: DETAIL: waiting for the reply... May 1 09:32:19 host pgpool[10924]: [7-1] 2019-05-01 09:32:19: pid 10924: LOG: master watchdog node "asa-pgpool02.adm.cacc.ch:5432 Linux asa-pgpool02.adm.cacc.ch" returned status for 2 backend nodes May 1 09:32:19 host pgpool[10924]: [8-1] 2019-05-01 09:32:19: pid 10924: LOG: backend:0 is set to down status May 1 09:32:19 host pgpool[10924]: [8-2] 2019-05-01 09:32:19: pid 10924: DETAIL: backend:0 is DOWN on cluster master "asa-pgpool02.adm.cacc.ch:5432 Linux asa-pgpool02.adm.cacc.ch" May 1 09:32:19 host pgpool[10924]: [9-1] 2019-05-01 09:32:19: pid 10924: LOG: backend:1 is set to down status May 1 09:32:19 host pgpool[10924]: [9-2] 2019-05-01 09:32:19: pid 10924: DETAIL: backend:1 is DOWN on cluster master "asa-pgpool02.adm.cacc.ch:5432 Linux asa-pgpool02.adm.cacc.ch" May 1 09:32:19 host pgpool[10924]: [10-1] 2019-05-01 09:32:19: pid 10924: LOG: Setting up socket for 0.0.0.0:5432 May 1 09:32:19 host pgpool[10924]: [11-1] 2019-05-01 09:32:19: pid 10924: LOG: Setting up socket for :::5432 May 1 09:32:19 host pgpool[10924]: [12-1] 2019-05-01 09:32:19: pid 10924: LOG: find_primary_node_repeatedly: waiting for finding a primary node May 1 09:32:19 host pgpool[10924]: [13-1] 2019-05-01 09:32:19: pid 10924: LOG: find_primary_node: checking backend no 0 May 1 09:32:19 host pgpool[10924]: [14-1] 2019-05-01 09:32:19: pid 10924: LOG: find_primary_node: checking backend no 1 May 1 09:32:19 host pgpool[10925]: [20-1] 2019-05-01 09:32:19: pid 10925: LOG: new IPC connection received May 1 09:32:19 host pgpool[10927]: [4-1] 2019-05-01 09:32:19: pid 10927: LOG: 2 watchdog nodes are configured for lifecheck May 1 09:32:19 host pgpool[10927]: [5-1] 2019-05-01 09:32:19: pid 10927: LOG: watchdog nodes ID:0 Name:"host.adm.cacc.ch:5432 Linux host.adm.cacc.ch" May 1 09:32:19 host pgpool[10927]: [5-2] 2019-05-01 09:32:19: pid 10927: DETAIL: Host:"host.adm.cacc.ch" WD Port:9000 pgpool-II port:5432 May 1 09:32:19 host pgpool[10927]: [6-1] 2019-05-01 09:32:19: pid 10927: LOG: watchdog nodes ID:1 Name:"asa-pgpool02.adm.cacc.ch:5432 Linux asa-pgpool02.adm.cacc.ch" May 1 09:32:19 host pgpool[10927]: [6-2] 2019-05-01 09:32:19: pid 10927: DETAIL: Host:"asa-pgpool02.adm.cacc.ch" WD Port:9000 pgpool-II port:5432 May 1 09:32:19 host pgpool[10946]: [12-1] 2019-05-01 09:32:19: pid 10946: FATAL: pgpool is not accepting any new connections May 1 09:32:19 host pgpool[10946]: [12-2] 2019-05-01 09:32:19: pid 10946: DETAIL: all backend nodes are down, pgpool requires at least one valid node May 1 09:32:19 host pgpool[10946]: [12-3] 2019-05-01 09:32:19: pid 10946: HINT: repair the backend nodes and restart pgpool May 1 09:32:19 host pgpool[10924]: [15-1] 2019-05-01 09:32:19: pid 10924: LOG: child process with pid: 10946 exits with status 256 May 1 09:32:19 host pgpool[10924]: [16-1] 2019-05-01 09:32:19: pid 10924: LOG: fork a new child process with pid: 10960 May 1 09:32:20 host pgpool[10924]: [17-1] 2019-05-01 09:32:20: pid 10924: LOG: find_primary_node: checking backend no 0 May 1 09:32:20 host pgpool[10924]: [18-1] 2019-05-01 09:32:20: pid 10924: LOG: find_primary_node: checking backend no 1 May 1 09:32:21 host pgpool[10924]: [19-1] 2019-05-01 09:32:21: pid 10924: LOG: find_primary_node: checking backend no 0 May 1 09:32:21 host pgpool[10924]: [20-1] 2019-05-01 09:32:21: pid 10924: LOG: find_primary_node: checking backend no 1 May 1 09:32:22 host pgpool[10924]: [21-1] 2019-05-01 09:32:22: pid 10924: LOG: find_primary_node: checking backend no 0 May 1 09:32:22 host pgpool[10924]: [22-1] 2019-05-01 09:32:22: pid 10924: LOG: find_primary_node: checking backend no 1 May 1 09:32:23 host pgpool[10924]: [23-1] 2019-05-01 09:32:23: pid 10924: LOG: find_primary_node: checking backend no 0 May 1 09:32:23 host pgpool[10924]: [24-1] 2019-05-01 09:32:23: pid 10924: LOG: find_primary_node: checking backend no 1 May 1 09:32:24 host pgpool[10924]: [25-1] 2019-05-01 09:32:24: pid 10924: LOG: find_primary_node: checking backend no 0 May 1 09:32:24 host pgpool[10924]: [26-1] 2019-05-01 09:32:24: pid 10924: LOG: find_primary_node: checking backend no 1 May 1 09:32:25 host pgpool[10924]: [27-1] 2019-05-01 09:32:25: pid 10924: LOG: find_primary_node: checking backend no 0 May 1 09:32:25 host pgpool[10924]: [28-1] 2019-05-01 09:32:25: pid 10924: LOG: find_primary_node: checking backend no 1 May 1 09:32:26 host pgpool[10924]: [29-1] 2019-05-01 09:32:26: pid 10924: LOG: find_primary_node: checking backend no 0 May 1 09:32:26 host pgpool[10924]: [30-1] 2019-05-01 09:32:26: pid 10924: LOG: find_primary_node: checking backend no 1 May 1 09:32:27 host pgpool[10924]: [31-1] 2019-05-01 09:32:27: pid 10924: LOG: find_primary_node: checking backend no 0 May 1 09:32:27 host pgpool[10924]: [32-1] 2019-05-01 09:32:27: pid 10924: LOG: find_primary_node: checking backend no 1 May 1 09:32:28 host pgpool[10924]: [33-1] 2019-05-01 09:32:28: pid 10924: LOG: find_primary_node: checking backend no 0 May 1 09:32:28 host pgpool[10924]: [34-1] 2019-05-01 09:32:28: pid 10924: LOG: find_primary_node: checking backend no 1 May 1 09:32:29 host pgpool[10924]: [35-1] 2019-05-01 09:32:29: pid 10924: LOG: find_primary_node: checking backend no 0 May 1 09:32:29 host pgpool[10924]: [36-1] 2019-05-01 09:32:29: pid 10924: LOG: find_primary_node: checking backend no 1 May 1 09:32:29 host pgpool[10960]: [16-1] 2019-05-01 09:32:29: pid 10960: FATAL: pgpool is not accepting any new connections May 1 09:32:29 host pgpool[10960]: [16-2] 2019-05-01 09:32:29: pid 10960: DETAIL: all backend nodes are down, pgpool requires at least one valid node May 1 09:32:29 host pgpool[10960]: [16-3] 2019-05-01 09:32:29: pid 10960: HINT: repair the backend nodes and restart pgpool May 1 09:32:29 host pgpool[10924]: [37-1] 2019-05-01 09:32:29: pid 10924: LOG: child process with pid: 10960 exits with status 256 May 1 09:32:29 host pgpool[10924]: [38-1] 2019-05-01 09:32:29: pid 10924: LOG: fork a new child process with pid: 10980 May 1 09:32:30 host pgpool[10924]: [39-1] 2019-05-01 09:32:30: pid 10924: LOG: find_primary_node: checking backend no 0 May 1 09:32:30 host pgpool[10924]: [40-1] 2019-05-01 09:32:30: pid 10924: LOG: find_primary_node: checking backend no 1 May 1 09:32:31 host pgpool[10924]: [41-1] 2019-05-01 09:32:31: pid 10924: LOG: find_primary_node: checking backend no 0 May 1 09:32:31 host pgpool[10924]: [42-1] 2019-05-01 09:32:31: pid 10924: LOG: find_primary_node: checking backend no 1 May 1 09:32:32 host pgpool[10924]: [43-1] 2019-05-01 09:32:32: pid 10924: LOG: find_primary_node: checking backend no 0 May 1 09:32:32 host pgpool[10924]: [44-1] 2019-05-01 09:32:32: pid 10924: LOG: find_primary_node: checking backend no 1 May 1 09:32:33 host pgpool[10924]: [45-1] 2019-05-01 09:32:33: pid 10924: LOG: find_primary_node: checking backend no 0 May 1 09:32:33 host pgpool[10924]: [46-1] 2019-05-01 09:32:33: pid 10924: LOG: find_primary_node: checking backend no 1 May 1 09:32:34 host pgpool[10924]: [47-1] 2019-05-01 09:32:34: pid 10924: LOG: find_primary_node: checking backend no 0 May 1 09:32:34 host pgpool[10924]: [48-1] 2019-05-01 09:32:34: pid 10924: LOG: find_primary_node: checking backend no 1 May 1 09:32:35 host pgpool[10924]: [49-1] 2019-05-01 09:32:35: pid 10924: LOG: find_primary_node: checking backend no 0 May 1 09:32:35 host pgpool[10924]: [50-1] 2019-05-01 09:32:35: pid 10924: LOG: find_primary_node: checking backend no 1 May 1 09:32:36 host pgpool[10924]: [51-1] 2019-05-01 09:32:36: pid 10924: LOG: find_primary_node: checking backend no 0 May 1 09:32:36 host pgpool[10924]: [52-1] 2019-05-01 09:32:36: pid 10924: LOG: find_primary_node: checking backend no 1 May 1 09:32:37 host pgpool[10924]: [53-1] 2019-05-01 09:32:37: pid 10924: LOG: find_primary_node: checking backend no 0 May 1 09:32:37 host pgpool[10924]: [54-1] 2019-05-01 09:32:37: pid 10924: LOG: find_primary_node: checking backend no 1 May 1 09:32:38 host pgpool[10924]: [55-1] 2019-05-01 09:32:38: pid 10924: LOG: find_primary_node: checking backend no 0 May 1 09:32:38 host pgpool[10924]: [56-1] 2019-05-01 09:32:38: pid 10924: LOG: find_primary_node: checking backend no 1 May 1 09:32:39 host pgpool[10924]: [57-1] 2019-05-01 09:32:39: pid 10924: LOG: find_primary_node: checking backend no 0 May 1 09:32:39 host pgpool[10924]: [58-1] 2019-05-01 09:32:39: pid 10924: LOG: find_primary_node: checking backend no 1 May 1 09:32:40 host pgpool[10924]: [59-1] 2019-05-01 09:32:40: pid 10924: LOG: find_primary_node: checking backend no 0 May 1 09:32:40 host pgpool[10924]: [60-1] 2019-05-01 09:32:40: pid 10924: LOG: find_primary_node: checking backend no 1 May 1 09:32:41 host pgpool[10924]: [61-1] 2019-05-01 09:32:41: pid 10924: LOG: find_primary_node: checking backend no 0 May 1 09:32:41 host pgpool[10924]: [62-1] 2019-05-01 09:32:41: pid 10924: LOG: find_primary_node: checking backend no 1 May 1 09:32:42 host pgpool[10924]: [63-1] 2019-05-01 09:32:42: pid 10924: LOG: find_primary_node: checking backend no 0 May 1 09:32:42 host pgpool[10924]: [64-1] 2019-05-01 09:32:42: pid 10924: LOG: find_primary_node: checking backend no 1 May 1 09:32:43 host pgpool[10924]: [65-1] 2019-05-01 09:32:43: pid 10924: LOG: find_primary_node: checking backend no 0 May 1 09:32:43 host pgpool[10924]: [66-1] 2019-05-01 09:32:43: pid 10924: LOG: find_primary_node: checking backend no 1 May 1 09:32:44 host pgpool[10924]: [67-1] 2019-05-01 09:32:44: pid 10924: LOG: find_primary_node: checking backend no 0 May 1 09:32:44 host pgpool[10924]: [68-1] 2019-05-01 09:32:44: pid 10924: LOG: find_primary_node: checking backend no 1 May 1 09:32:45 host pgpool[10924]: [69-1] 2019-05-01 09:32:45: pid 10924: LOG: find_primary_node: checking backend no 0 May 1 09:32:45 host pgpool[10924]: [70-1] 2019-05-01 09:32:45: pid 10924: LOG: find_primary_node: checking backend no 1 May 1 09:32:46 host pgpool[10924]: [71-1] 2019-05-01 09:32:46: pid 10924: LOG: find_primary_node: checking backend no 0 May 1 09:32:46 host pgpool[10924]: [72-1] 2019-05-01 09:32:46: pid 10924: LOG: find_primary_node: checking backend no 1 May 1 09:32:47 host pgpool[10924]: [73-1] 2019-05-01 09:32:47: pid 10924: LOG: find_primary_node: checking backend no 0 May 1 09:32:47 host pgpool[10924]: [74-1] 2019-05-01 09:32:47: pid 10924: LOG: find_primary_node: checking backend no 1 May 1 09:32:48 host pgpool[10924]: [75-1] 2019-05-01 09:32:48: pid 10924: LOG: find_primary_node: checking backend no 0 May 1 09:32:48 host pgpool[10924]: [76-1] 2019-05-01 09:32:48: pid 10924: LOG: find_primary_node: checking backend no 1 May 1 09:32:49 host pgpool[10924]: [77-1] 2019-05-01 09:32:49: pid 10924: LOG: find_primary_node: checking backend no 0 May 1 09:32:49 host pgpool[10924]: [78-1] 2019-05-01 09:32:49: pid 10924: LOG: find_primary_node: checking backend no 1 May 1 09:32:50 host pgpool[10924]: [79-1] 2019-05-01 09:32:50: pid 10924: LOG: find_primary_node: checking backend no 0 May 1 09:32:50 host pgpool[10924]: [80-1] 2019-05-01 09:32:50: pid 10924: LOG: find_primary_node: checking backend no 1 May 1 09:32:51 host pgpool[10924]: [81-1] 2019-05-01 09:32:51: pid 10924: LOG: find_primary_node: checking backend no 0 May 1 09:32:51 host pgpool[10924]: [82-1] 2019-05-01 09:32:51: pid 10924: LOG: find_primary_node: checking backend no 1 May 1 09:32:52 host pgpool[10924]: [83-1] 2019-05-01 09:32:52: pid 10924: LOG: find_primary_node: checking backend no 0 May 1 09:32:52 host pgpool[10924]: [84-1] 2019-05-01 09:32:52: pid 10924: LOG: find_primary_node: checking backend no 1 May 1 09:32:53 host pgpool[10924]: [85-1] 2019-05-01 09:32:53: pid 10924: LOG: find_primary_node: checking backend no 0 May 1 09:32:53 host pgpool[10924]: [86-1] 2019-05-01 09:32:53: pid 10924: LOG: find_primary_node: checking backend no 1 May 1 09:32:54 host pgpool[10924]: [87-1] 2019-05-01 09:32:54: pid 10924: LOG: find_primary_node: checking backend no 0 May 1 09:32:54 host pgpool[10924]: [88-1] 2019-05-01 09:32:54: pid 10924: LOG: find_primary_node: checking backend no 1 May 1 09:32:55 host pgpool[10924]: [89-1] 2019-05-01 09:32:55: pid 10924: LOG: find_primary_node: checking backend no 0 May 1 09:32:55 host pgpool[10924]: [90-1] 2019-05-01 09:32:55: pid 10924: LOG: find_primary_node: checking backend no 1 May 1 09:32:56 host nagios: SERVICE NOTIFICATION: nagiosadmin;asapgsql02;Current Load;CRITICAL;notify-service-by-email;CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. May 1 09:32:56 host nagios: wproc: NOTIFY job 165270 from worker Core Worker 17733 is a non-check helper but exited with return code 75