-bash-4.1$ /usr/pgpool-9.5/bin/pgpool -d -f /etc/pgpool-II-95/pgpool.conf -n 2016-03-23 10:46:05: pid 58460: DEBUG: loading hba configuration 2016-03-23 10:46:05: pid 58460: DETAIL: loading file :"/etc/pgpool-II-95/pool_hba.conf" for client authentication configuration file 2016-03-23 10:46:05: pid 58460: WARNING: checking setuid bit of if_up_cmd 2016-03-23 10:46:05: pid 58460: DETAIL: ifup[/data/postgresql/data/pgp_ip_addr.sh] doesn't have setuid bit 2016-03-23 10:46:05: pid 58460: WARNING: checking setuid bit of if_down_cmd 2016-03-23 10:46:05: pid 58460: DETAIL: ifdown[/data/postgresql/data/pgp_ip_addr.sh] doesn't have setuid bit 2016-03-23 10:46:05: pid 58460: WARNING: checking setuid bit of arping command 2016-03-23 10:46:05: pid 58460: DETAIL: arping[/data/postgresql/data/pgp_arping.sh] doesn't have setuid bit 2016-03-23 10:46:05: pid 58460: LOG: reading status file: 1 th backend is set to down status 2016-03-23 10:46:05: pid 58460: DEBUG: pool_coninfo_size: num_init_children (30) * max_pool (4) * MAX_NUM_BACKENDS (128) * sizeof(ConnectionInfo) (136) = 2088960 bytes requested for shared memory 2016-03-23 10:46:05: pid 58460: DEBUG: ProcessInfo: num_init_children (30) * sizeof(ProcessInfo) (32) = 960 bytes requested for shared memory 2016-03-23 10:46:05: pid 58460: DEBUG: Request info are: sizeof(POOL_REQUEST_INFO) 5224 bytes requested for shared memory 2016-03-23 10:46:05: pid 58460: DEBUG: Recovery management area: sizeof(int) 4 bytes requested for shared memory 2016-03-23 10:46:05: pid 58460: DEBUG: WD_Node_List: sizeof(unsigned char) (1) * MAX_NUM_BACKENDS (128) = 128 bytes requested for shared memory 2016-03-23 10:46:05: pid 58460: LOG: waiting for watchdog to initialize 2016-03-23 10:46:05: pid 58461: LOG: setting the local watchdog node name to "Linux_serv-ed02_9999" 2016-03-23 10:46:05: pid 58461: LOG: watchdog cluster configured with 1 remote nodes 2016-03-23 10:46:05: pid 58461: LOG: watchdog remote node:0 on serv-ed01:9000 2016-03-23 10:46:05: pid 58461: LOG: interface monitoring is disabled in watchdog 2016-03-23 10:46:05: pid 58461: LOG: IPC socket path: "/tmp/.s.PGPOOLWD_CMD.9000" 2016-03-23 10:46:05: pid 58461: DEBUG: STATE MACHINE INVOKED WITH EVENT = STATE CHANGED Current State = LOADING 2016-03-23 10:46:05: pid 58461: LOG: new outbond connection to serv-ed01:9000 2016-03-23 10:46:05: pid 58461: DEBUG: STATE MACHINE INVOKED WITH EVENT = NEW OUTBOUND_CONNECTION Current State = LOADING 2016-03-23 10:46:05: pid 58461: DEBUG: sending watchdog packet Socket:7, Type:[ADD NODE], Command_ID:3, data Length:170 2016-03-23 10:46:05: pid 58461: DEBUG: received packet tyep I while need packet type 2016-03-23 10:46:05: pid 58461: DEBUG: STATE MACHINE INVOKED WITH EVENT = PACKET RECEIVED Current State = LOADING 2016-03-23 10:46:05: pid 58461: LOG: watchdog node state changed from [LOADING] to [INITIALIZING] 2016-03-23 10:46:05: pid 58461: DEBUG: STATE MACHINE INVOKED WITH EVENT = STATE CHANGED Current State = INITIALIZING 2016-03-23 10:46:05: pid 58461: DEBUG: sending watchdog packet Socket:7, Type:[NODE INFO], Command_ID:4, data Length:170 2016-03-23 10:46:06: pid 58461: DEBUG: STATE MACHINE INVOKED WITH EVENT = TIMEOUT Current State = INITIALIZING 2016-03-23 10:46:06: pid 58461: LOG: watchdog node state changed from [INITIALIZING] to [STANDBY] 2016-03-23 10:46:06: pid 58461: DEBUG: STATE MACHINE INVOKED WITH EVENT = STATE CHANGED Current State = STANDBY 2016-03-23 10:46:06: pid 58461: DEBUG: sending watchdog packet Socket:7, Type:[JOIN COORDINATOR], Command_ID:5, data Length:0 2016-03-23 10:46:06: pid 58461: DEBUG: sending watchdog packet Socket:7, Type:[NODE INFO], Command_ID:6, data Length:170 2016-03-23 10:46:06: pid 58461: DEBUG: received packet tyep G while need packet type 2016-03-23 10:46:06: pid 58461: DEBUG: STATE MACHINE INVOKED WITH EVENT = PACKET RECEIVED Current State = STANDBY 2016-03-23 10:46:06: pid 58461: DEBUG: Watchdog node "Linux_serv-ed01_9999" has replied for command id 5 2016-03-23 10:46:06: pid 58461: DEBUG: STATE MACHINE INVOKED WITH EVENT = COMMAND FINISHED Current State = STANDBY 2016-03-23 10:46:06: pid 58461: DEBUG: sending watchdog packet Socket:7, Type:[ASK FOR POOL CONFIG], Command_ID:7, data Length:0 2016-03-23 10:46:06: pid 58461: LOG: successfully joined the watchdog cluster as standby node 2016-03-23 10:46:06: pid 58461: DETAIL: our join coordinator request is accepted by cluster leader node "Linux_serv-ed01_9999" 2016-03-23 10:46:06: pid 58460: LOG: watchdog process is initialized 2016-03-23 10:46:06: pid 58462: DEBUG: I am watchdog lifecheck child with pid:58462 2016-03-23 10:46:06: pid 58460: LOG: Setting up socket for 0.0.0.0:9999 2016-03-23 10:46:06: pid 58460: LOG: Setting up socket for :::9999 2016-03-23 10:46:06: pid 58461: LOG: new IPC connection received 2016-03-23 10:46:06: pid 58461: DEBUG: received packet tyep Z while need packet type 2016-03-23 10:46:06: pid 58461: DEBUG: STATE MACHINE INVOKED WITH EVENT = PACKET RECEIVED Current State = STANDBY 2016-03-23 10:46:06: pid 58462: LOG: 2 watchdog nodes are configured for lifecheck 2016-03-23 10:46:06: pid 58462: LOG: watchdog nodes ID:0 Name:"Linux_serv-ed02_9999" 2016-03-23 10:46:06: pid 58462: DETAIL: Host:"serv-ed02" WD Port:9000 pgpool-II port:9999 2016-03-23 10:46:06: pid 58462: LOG: watchdog nodes ID:1 Name:"Linux_serv-ed01_9999" 2016-03-23 10:46:06: pid 58462: DETAIL: Host:"serv-ed01" WD Port:9000 pgpool-II port:9999 2016-03-23 10:46:06: pid 58463: DEBUG: initializing backend status 2016-03-23 10:46:06: pid 58462: DEBUG: watchdog checking life check is ready 2016-03-23 10:46:06: pid 58462: DETAIL: pgpool:1 at "serv-ed01:9999" has not send the heartbeat signal yet 2016-03-23 10:46:06: pid 58465: DEBUG: initializing backend status 2016-03-23 10:46:06: pid 58467: DEBUG: initializing backend status 2016-03-23 10:46:06: pid 58468: DEBUG: initializing backend status 2016-03-23 10:46:06: pid 58469: DEBUG: initializing backend status 2016-03-23 10:46:06: pid 58470: DEBUG: initializing backend status 2016-03-23 10:46:06: pid 58471: DEBUG: initializing backend status 2016-03-23 10:46:06: pid 58472: DEBUG: initializing backend status 2016-03-23 10:46:06: pid 58473: DEBUG: initializing backend status 2016-03-23 10:46:06: pid 58474: DEBUG: initializing backend status 2016-03-23 10:46:06: pid 58475: DEBUG: initializing backend status 2016-03-23 10:46:06: pid 58476: DEBUG: initializing backend status 2016-03-23 10:46:06: pid 58477: DEBUG: initializing backend status 2016-03-23 10:46:06: pid 58478: DEBUG: initializing backend status 2016-03-23 10:46:06: pid 58479: DEBUG: initializing backend status 2016-03-23 10:46:06: pid 58480: DEBUG: initializing backend status 2016-03-23 10:46:06: pid 58481: DEBUG: initializing backend status 2016-03-23 10:46:06: pid 58482: DEBUG: initializing backend status 2016-03-23 10:46:06: pid 58483: DEBUG: initializing backend status 2016-03-23 10:46:06: pid 58484: DEBUG: initializing backend status 2016-03-23 10:46:06: pid 58485: DEBUG: initializing backend status 2016-03-23 10:46:06: pid 58486: DEBUG: initializing backend status 2016-03-23 10:46:06: pid 58487: DEBUG: initializing backend status 2016-03-23 10:46:06: pid 58488: DEBUG: initializing backend status 2016-03-23 10:46:06: pid 58489: DEBUG: initializing backend status 2016-03-23 10:46:06: pid 58490: DEBUG: initializing backend status 2016-03-23 10:46:06: pid 58491: DEBUG: initializing backend status 2016-03-23 10:46:06: pid 58492: DEBUG: initializing backend status 2016-03-23 10:46:06: pid 58493: DEBUG: initializing backend status 2016-03-23 10:46:06: pid 58494: DEBUG: initializing backend status 2016-03-23 10:46:06: pid 58495: DEBUG: I am PCP child with pid:58495 2016-03-23 10:46:06: pid 58460: LOG: pgpool-II successfully started. version 3.5.0 (ekieboshi) 2016-03-23 10:46:06: pid 58460: LOG: find_primary_node: checking backend no 0 2016-03-23 10:46:06: pid 58496: DEBUG: I am 58496 2016-03-23 10:46:06: pid 58496: DEBUG: initializing backend status 2016-03-23 10:46:06: pid 58460: DEBUG: pool_read: read 327 bytes from backend 0 2016-03-23 10:46:06: pid 58460: DEBUG: authenticate kind = 0 2016-03-23 10:46:06: pid 58460: DEBUG: authenticate backend: key data received 2016-03-23 10:46:06: pid 58460: DEBUG: authenticate backend: transaction state: I 2016-03-23 10:46:06: pid 58460: DEBUG: do_query: extended:0 query:"SELECT pg_is_in_recovery()" 2016-03-23 10:46:06: pid 58460: DEBUG: pool_write: to backend: kind:Q 2016-03-23 10:46:06: pid 58496: DEBUG: pool_read: read 327 bytes from backend 0 2016-03-23 10:46:06: pid 58496: DEBUG: authenticate kind = 0 2016-03-23 10:46:06: pid 58496: DEBUG: authenticate backend: key data received 2016-03-23 10:46:06: pid 58496: DEBUG: authenticate backend: transaction state: I 2016-03-23 10:46:06: pid 58496: DEBUG: pool_write: to backend: kind:X 2016-03-23 10:46:06: pid 58460: DEBUG: pool_read: read 75 bytes from backend 0 2016-03-23 10:46:06: pid 58460: DEBUG: do_query: kind: 'T' 2016-03-23 10:46:06: pid 58460: DEBUG: do_query: received ROW DESCRIPTION ('T') 2016-03-23 10:46:06: pid 58460: DEBUG: do_query: row description: num_fileds: 1 2016-03-23 10:46:06: pid 58460: DEBUG: do_query: kind: 'D' 2016-03-23 10:46:06: pid 58460: DEBUG: do_query: received DATA ROW ('D') 2016-03-23 10:46:06: pid 58460: DEBUG: do_query: kind: 'C' 2016-03-23 10:46:06: pid 58460: DEBUG: do_query: received COMMAND COMPLETE ('C') 2016-03-23 10:46:06: pid 58460: DEBUG: do_query: kind: 'Z' 2016-03-23 10:46:06: pid 58460: DEBUG: do_query: received READY FOR QUERY ('Z') 2016-03-23 10:46:06: pid 58460: DEBUG: pool_write: to backend: kind:X 2016-03-23 10:46:06: pid 58460: LOG: find_primary_node: primary node id is 0 2016-03-23 10:46:06: pid 58460: DEBUG: starting health check 2016-03-23 10:46:06: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:46:06: pid 58460: DEBUG: doing health check against database:postgres user:postgres 2016-03-23 10:46:06: pid 58460: DEBUG: Backend DB node 0 status is 2 2016-03-23 10:46:06: pid 58460: DEBUG: Trying to make persistent DB connection to backend node 0 having status 2 2016-03-23 10:46:06: pid 58460: DEBUG: pool_read: read 327 bytes from backend 0 2016-03-23 10:46:06: pid 58460: DEBUG: authenticate kind = 0 2016-03-23 10:46:06: pid 58460: DEBUG: authenticate backend: key data received 2016-03-23 10:46:06: pid 58460: DEBUG: authenticate backend: transaction state: I 2016-03-23 10:46:06: pid 58460: DEBUG: persistent DB connection to backend node 0 having status 2 is successful 2016-03-23 10:46:06: pid 58460: DEBUG: pool_write: to backend: kind:X 2016-03-23 10:46:06: pid 58460: DEBUG: Backend DB node 1 status is 3 2016-03-23 10:46:06: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:46:06: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:46:07: pid 58464: LOG: failed to create watchdog heartbeat receive socket. 2016-03-23 10:46:07: pid 58464: DETAIL: setsockopt(SO_BINDTODEVICE) requies root privilege 2016-03-23 10:46:07: pid 58466: LOG: creating socket for sending heartbeat 2016-03-23 10:46:07: pid 58466: DETAIL: setsockopt(SO_BINDTODEVICE) requires root privilege 2016-03-23 10:46:07: pid 58464: LOG: set SO_REUSEPORT option to the socket 2016-03-23 10:46:07: pid 58466: LOG: set SO_REUSEPORT option to the socket 2016-03-23 10:46:07: pid 58464: LOG: creating watchdog heartbeat receive socket. 2016-03-23 10:46:07: pid 58464: DETAIL: set SO_REUSEPORT 2016-03-23 10:46:07: pid 58466: LOG: creating socket for sending heartbeat 2016-03-23 10:46:07: pid 58466: DETAIL: set SO_REUSEPORT 2016-03-23 10:46:07: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:46:08: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:46:09: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:46:09: pid 58461: DEBUG: received packet tyep M while need packet type 2016-03-23 10:46:09: pid 58461: DEBUG: STATE MACHINE INVOKED WITH EVENT = PACKET RECEIVED Current State = STANDBY 2016-03-23 10:46:09: pid 58461: DEBUG: sending watchdog packet Socket:7, Type:[NODE INFO], Command_ID:8, data Length:170 2016-03-23 10:46:10: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:46:11: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:46:11: pid 58461: LOG: new watchdog node connection is received from "10.129.124.46:26275" 2016-03-23 10:46:11: pid 58461: DEBUG: received packet tyep A while need packet type A 2016-03-23 10:46:11: pid 58461: DEBUG: ADD NODE MESSAGE from Hostname:"serv-ed01" PORT:9000 pgpool_port:9999 2016-03-23 10:46:11: pid 58461: NOTICE: New node joined the cluster Hostname:"serv-ed01" PORT:9000 pgpool_port:9999 2016-03-23 10:46:11: pid 58461: DEBUG: STATE MACHINE INVOKED WITH EVENT = PACKET RECEIVED Current State = STANDBY 2016-03-23 10:46:11: pid 58460: DEBUG: starting health check 2016-03-23 10:46:11: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:46:11: pid 58460: DEBUG: doing health check against database:postgres user:postgres 2016-03-23 10:46:11: pid 58460: DEBUG: Backend DB node 0 status is 2 2016-03-23 10:46:11: pid 58460: DEBUG: Trying to make persistent DB connection to backend node 0 having status 2 2016-03-23 10:46:11: pid 58460: DEBUG: pool_read: read 327 bytes from backend 0 2016-03-23 10:46:11: pid 58460: DEBUG: authenticate kind = 0 2016-03-23 10:46:11: pid 58460: DEBUG: authenticate backend: key data received 2016-03-23 10:46:11: pid 58460: DEBUG: authenticate backend: transaction state: I 2016-03-23 10:46:11: pid 58460: DEBUG: persistent DB connection to backend node 0 having status 2 is successful 2016-03-23 10:46:11: pid 58460: DEBUG: pool_write: to backend: kind:X 2016-03-23 10:46:11: pid 58460: DEBUG: Backend DB node 1 status is 3 2016-03-23 10:46:11: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:46:11: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:46:12: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:46:13: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:46:14: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:46:15: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:46:16: pid 58496: DEBUG: pool_read: read 327 bytes from backend 0 2016-03-23 10:46:16: pid 58496: DEBUG: authenticate kind = 0 2016-03-23 10:46:16: pid 58496: DEBUG: authenticate backend: key data received 2016-03-23 10:46:16: pid 58496: DEBUG: authenticate backend: transaction state: I 2016-03-23 10:46:16: pid 58496: DEBUG: pool_write: to backend: kind:X 2016-03-23 10:46:16: pid 58460: DEBUG: starting health check 2016-03-23 10:46:16: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:46:16: pid 58460: DEBUG: doing health check against database:postgres user:postgres 2016-03-23 10:46:16: pid 58460: DEBUG: Backend DB node 0 status is 2 2016-03-23 10:46:16: pid 58460: DEBUG: Trying to make persistent DB connection to backend node 0 having status 2 2016-03-23 10:46:16: pid 58460: DEBUG: pool_read: read 327 bytes from backend 0 2016-03-23 10:46:16: pid 58460: DEBUG: authenticate kind = 0 2016-03-23 10:46:16: pid 58460: DEBUG: authenticate backend: key data received 2016-03-23 10:46:16: pid 58460: DEBUG: authenticate backend: transaction state: I 2016-03-23 10:46:16: pid 58460: DEBUG: persistent DB connection to backend node 0 having status 2 is successful 2016-03-23 10:46:16: pid 58460: DEBUG: pool_write: to backend: kind:X 2016-03-23 10:46:16: pid 58460: DEBUG: Backend DB node 1 status is 3 2016-03-23 10:46:16: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:46:16: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:46:16: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:46:17: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:46:18: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:46:19: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:46:19: pid 58461: DEBUG: received packet tyep M while need packet type 2016-03-23 10:46:19: pid 58461: DEBUG: STATE MACHINE INVOKED WITH EVENT = PACKET RECEIVED Current State = STANDBY 2016-03-23 10:46:19: pid 58461: DEBUG: sending watchdog packet Socket:7, Type:[NODE INFO], Command_ID:10, data Length:170 2016-03-23 10:46:20: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:46:21: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:46:21: pid 58460: DEBUG: starting health check 2016-03-23 10:46:21: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:46:21: pid 58460: DEBUG: doing health check against database:postgres user:postgres 2016-03-23 10:46:21: pid 58460: DEBUG: Backend DB node 0 status is 2 2016-03-23 10:46:21: pid 58460: DEBUG: Trying to make persistent DB connection to backend node 0 having status 2 2016-03-23 10:46:21: pid 58460: DEBUG: pool_read: read 327 bytes from backend 0 2016-03-23 10:46:21: pid 58460: DEBUG: authenticate kind = 0 2016-03-23 10:46:21: pid 58460: DEBUG: authenticate backend: key data received 2016-03-23 10:46:21: pid 58460: DEBUG: authenticate backend: transaction state: I 2016-03-23 10:46:21: pid 58460: DEBUG: persistent DB connection to backend node 0 having status 2 is successful 2016-03-23 10:46:21: pid 58460: DEBUG: pool_write: to backend: kind:X 2016-03-23 10:46:21: pid 58460: DEBUG: Backend DB node 1 status is 3 2016-03-23 10:46:21: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:46:21: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:46:22: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:46:23: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:46:24: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:46:25: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:46:26: pid 58496: DEBUG: pool_read: read 327 bytes from backend 0 2016-03-23 10:46:26: pid 58496: DEBUG: authenticate kind = 0 2016-03-23 10:46:26: pid 58496: DEBUG: authenticate backend: key data received 2016-03-23 10:46:26: pid 58496: DEBUG: authenticate backend: transaction state: I 2016-03-23 10:46:26: pid 58496: DEBUG: pool_write: to backend: kind:X 2016-03-23 10:46:26: pid 58460: DEBUG: starting health check 2016-03-23 10:46:26: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:46:26: pid 58460: DEBUG: doing health check against database:postgres user:postgres 2016-03-23 10:46:26: pid 58460: DEBUG: Backend DB node 0 status is 2 2016-03-23 10:46:26: pid 58460: DEBUG: Trying to make persistent DB connection to backend node 0 having status 2 2016-03-23 10:46:26: pid 58460: DEBUG: pool_read: read 327 bytes from backend 0 2016-03-23 10:46:26: pid 58460: DEBUG: authenticate kind = 0 2016-03-23 10:46:26: pid 58460: DEBUG: authenticate backend: key data received 2016-03-23 10:46:26: pid 58460: DEBUG: authenticate backend: transaction state: I 2016-03-23 10:46:26: pid 58460: DEBUG: persistent DB connection to backend node 0 having status 2 is successful 2016-03-23 10:46:26: pid 58460: DEBUG: pool_write: to backend: kind:X 2016-03-23 10:46:26: pid 58460: DEBUG: Backend DB node 1 status is 3 2016-03-23 10:46:26: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:46:26: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:46:26: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:46:27: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:46:28: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:46:29: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:46:29: pid 58461: DEBUG: received packet tyep M while need packet type 2016-03-23 10:46:29: pid 58461: DEBUG: STATE MACHINE INVOKED WITH EVENT = PACKET RECEIVED Current State = STANDBY 2016-03-23 10:46:29: pid 58461: DEBUG: sending watchdog packet Socket:7, Type:[NODE INFO], Command_ID:11, data Length:170 2016-03-23 10:46:30: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:46:31: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:46:31: pid 58460: DEBUG: starting health check 2016-03-23 10:46:31: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:46:31: pid 58460: DEBUG: doing health check against database:postgres user:postgres 2016-03-23 10:46:31: pid 58460: DEBUG: Backend DB node 0 status is 2 2016-03-23 10:46:31: pid 58460: DEBUG: Trying to make persistent DB connection to backend node 0 having status 2 2016-03-23 10:46:31: pid 58460: DEBUG: pool_read: read 327 bytes from backend 0 2016-03-23 10:46:31: pid 58460: DEBUG: authenticate kind = 0 2016-03-23 10:46:31: pid 58460: DEBUG: authenticate backend: key data received 2016-03-23 10:46:31: pid 58460: DEBUG: authenticate backend: transaction state: I 2016-03-23 10:46:31: pid 58460: DEBUG: persistent DB connection to backend node 0 having status 2 is successful 2016-03-23 10:46:31: pid 58460: DEBUG: pool_write: to backend: kind:X 2016-03-23 10:46:31: pid 58460: DEBUG: Backend DB node 1 status is 3 2016-03-23 10:46:31: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:46:31: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:46:32: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:46:33: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:46:34: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:46:35: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:46:36: pid 58462: LOG: watchdog: lifecheck started 2016-03-23 10:46:36: pid 58496: DEBUG: pool_read: read 327 bytes from backend 0 2016-03-23 10:46:36: pid 58496: DEBUG: authenticate kind = 0 2016-03-23 10:46:36: pid 58496: DEBUG: authenticate backend: key data received 2016-03-23 10:46:36: pid 58496: DEBUG: authenticate backend: transaction state: I 2016-03-23 10:46:36: pid 58496: DEBUG: pool_write: to backend: kind:X 2016-03-23 10:46:36: pid 58460: DEBUG: starting health check 2016-03-23 10:46:36: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:46:36: pid 58460: DEBUG: doing health check against database:postgres user:postgres 2016-03-23 10:46:36: pid 58460: DEBUG: Backend DB node 0 status is 2 2016-03-23 10:46:36: pid 58460: DEBUG: Trying to make persistent DB connection to backend node 0 having status 2 2016-03-23 10:46:36: pid 58460: DEBUG: pool_read: read 327 bytes from backend 0 2016-03-23 10:46:36: pid 58460: DEBUG: authenticate kind = 0 2016-03-23 10:46:36: pid 58460: DEBUG: authenticate backend: key data received 2016-03-23 10:46:36: pid 58460: DEBUG: authenticate backend: transaction state: I 2016-03-23 10:46:36: pid 58460: DEBUG: persistent DB connection to backend node 0 having status 2 is successful 2016-03-23 10:46:36: pid 58460: DEBUG: pool_write: to backend: kind:X 2016-03-23 10:46:36: pid 58460: DEBUG: Backend DB node 1 status is 3 2016-03-23 10:46:36: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:46:36: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:46:36: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:46:36: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:46:37: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:46:38: pid 58462: DEBUG: watchdog executing ping 2016-03-23 10:46:38: pid 58462: DETAIL: succeed to ping serv-ew01 2016-03-23 10:46:38: pid 58462: DEBUG: watchdog ping 2016-03-23 10:46:38: pid 58462: DETAIL: ping data: PING serv-ew01.adm.fr.clara.net (10.129.124.41) 56(84) bytes of data. --- serv-ew01.adm.fr.clara.net ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2001ms rtt min/avg/max/mdev = 0.170/0.186/0.201/0.012 ms 2016-03-23 10:46:38: pid 58462: DEBUG: watchdog executing ping 2016-03-23 10:46:38: pid 58462: DETAIL: succeed to ping serv-ea03 2016-03-23 10:46:38: pid 58462: DEBUG: watchdog executing ping 2016-03-23 10:46:38: pid 58462: DETAIL: succeed to ping serv-ea01 2016-03-23 10:46:38: pid 58462: DEBUG: missing error text 2016-03-23 10:46:38: pid 58462: DEBUG: watchdog ping 2016-03-23 10:46:38: pid 58462: DETAIL: ping data: PING serv-ea01.adm.fr.clara.net (10.129.124.43) 56(84) bytes of data. --- serv-ea01.adm.fr.clara.net ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2001ms rtt min/avg/max/mdev = 0.171/0.251/0.349/0.075 ms 2016-03-23 10:46:38: pid 58460: DEBUG: reaper handler 2016-03-23 10:46:38: pid 58460: WARNING: watchdog lifecheck process with pid: 58462 was terminated by segmentation fault 2016-03-23 10:46:38: pid 58460: LOG: fork a new watchdog lifecheck process with pid: 59034 2016-03-23 10:46:38: pid 58460: DEBUG: reaper handler: exiting normally 2016-03-23 10:46:38: pid 59034: DEBUG: I am watchdog lifecheck child with pid:59034 2016-03-23 10:46:38: pid 58461: LOG: new IPC connection received 2016-03-23 10:46:38: pid 59034: LOG: 2 watchdog nodes are configured for lifecheck 2016-03-23 10:46:38: pid 59034: LOG: watchdog nodes ID:0 Name:"Linux_serv-ed02_9999" 2016-03-23 10:46:38: pid 59034: DETAIL: Host:"serv-ed02" WD Port:9000 pgpool-II port:9999 2016-03-23 10:46:38: pid 59034: LOG: watchdog nodes ID:1 Name:"Linux_serv-ed01_9999" 2016-03-23 10:46:38: pid 59034: DETAIL: Host:"serv-ed01" WD Port:9000 pgpool-II port:9999 2016-03-23 10:46:38: pid 59034: DEBUG: watchdog checking life check is ready 2016-03-23 10:46:38: pid 59034: DETAIL: pgpool:1 at "serv-ed01:9999" has not send the heartbeat signal yet 2016-03-23 10:46:38: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:46:38: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:46:39: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:46:39: pid 59035: LOG: failed to create watchdog heartbeat receive socket. 2016-03-23 10:46:39: pid 59035: DETAIL: setsockopt(SO_BINDTODEVICE) requies root privilege 2016-03-23 10:46:39: pid 59035: LOG: set SO_REUSEPORT option to the socket 2016-03-23 10:46:39: pid 59035: LOG: creating watchdog heartbeat receive socket. 2016-03-23 10:46:39: pid 59035: DETAIL: set SO_REUSEPORT 2016-03-23 10:46:39: pid 59036: LOG: creating socket for sending heartbeat 2016-03-23 10:46:39: pid 59036: DETAIL: setsockopt(SO_BINDTODEVICE) requires root privilege 2016-03-23 10:46:39: pid 59036: LOG: set SO_REUSEPORT option to the socket 2016-03-23 10:46:39: pid 59036: LOG: creating socket for sending heartbeat 2016-03-23 10:46:39: pid 59036: DETAIL: set SO_REUSEPORT 2016-03-23 10:46:39: pid 59036: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:46:39: pid 58461: DEBUG: received packet tyep M while need packet type 2016-03-23 10:46:39: pid 58461: DEBUG: STATE MACHINE INVOKED WITH EVENT = PACKET RECEIVED Current State = STANDBY 2016-03-23 10:46:39: pid 58461: DEBUG: sending watchdog packet Socket:7, Type:[NODE INFO], Command_ID:12, data Length:170 2016-03-23 10:46:40: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:46:40: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:46:41: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:46:41: pid 59036: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:46:41: pid 58460: DEBUG: starting health check 2016-03-23 10:46:41: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:46:41: pid 58460: DEBUG: doing health check against database:postgres user:postgres 2016-03-23 10:46:41: pid 58460: DEBUG: Backend DB node 0 status is 2 2016-03-23 10:46:41: pid 58460: DEBUG: Trying to make persistent DB connection to backend node 0 having status 2 2016-03-23 10:46:41: pid 58460: DEBUG: pool_read: read 327 bytes from backend 0 2016-03-23 10:46:41: pid 58460: DEBUG: authenticate kind = 0 2016-03-23 10:46:41: pid 58460: DEBUG: authenticate backend: key data received 2016-03-23 10:46:41: pid 58460: DEBUG: authenticate backend: transaction state: I 2016-03-23 10:46:41: pid 58460: DEBUG: persistent DB connection to backend node 0 having status 2 is successful 2016-03-23 10:46:41: pid 58460: DEBUG: pool_write: to backend: kind:X 2016-03-23 10:46:41: pid 58460: DEBUG: Backend DB node 1 status is 3 2016-03-23 10:46:41: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:46:41: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:46:42: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:46:42: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:46:43: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:46:43: pid 59036: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:46:44: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:46:44: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:46:45: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:46:45: pid 59036: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:46:46: pid 58496: DEBUG: pool_read: read 327 bytes from backend 0 2016-03-23 10:46:46: pid 58496: DEBUG: authenticate kind = 0 2016-03-23 10:46:46: pid 58496: DEBUG: authenticate backend: key data received 2016-03-23 10:46:46: pid 58496: DEBUG: authenticate backend: transaction state: I 2016-03-23 10:46:46: pid 58496: DEBUG: pool_write: to backend: kind:X 2016-03-23 10:46:46: pid 58460: DEBUG: starting health check 2016-03-23 10:46:46: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:46:46: pid 58460: DEBUG: doing health check against database:postgres user:postgres 2016-03-23 10:46:46: pid 58460: DEBUG: Backend DB node 0 status is 2 2016-03-23 10:46:46: pid 58460: DEBUG: Trying to make persistent DB connection to backend node 0 having status 2 2016-03-23 10:46:46: pid 58460: DEBUG: pool_read: read 327 bytes from backend 0 2016-03-23 10:46:46: pid 58460: DEBUG: authenticate kind = 0 2016-03-23 10:46:46: pid 58460: DEBUG: authenticate backend: key data received 2016-03-23 10:46:46: pid 58460: DEBUG: authenticate backend: transaction state: I 2016-03-23 10:46:46: pid 58460: DEBUG: persistent DB connection to backend node 0 having status 2 is successful 2016-03-23 10:46:46: pid 58460: DEBUG: pool_write: to backend: kind:X 2016-03-23 10:46:46: pid 58460: DEBUG: Backend DB node 1 status is 3 2016-03-23 10:46:46: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:46:46: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:46:46: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:46:46: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:46:47: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:46:47: pid 59036: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:46:48: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:46:48: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:46:49: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:46:49: pid 59036: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:46:49: pid 58461: DEBUG: received packet tyep M while need packet type 2016-03-23 10:46:49: pid 58461: DEBUG: STATE MACHINE INVOKED WITH EVENT = PACKET RECEIVED Current State = STANDBY 2016-03-23 10:46:49: pid 58461: DEBUG: sending watchdog packet Socket:7, Type:[NODE INFO], Command_ID:13, data Length:170 2016-03-23 10:46:50: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:46:50: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:46:51: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:46:51: pid 59036: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:46:51: pid 58460: DEBUG: starting health check 2016-03-23 10:46:51: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:46:51: pid 58460: DEBUG: doing health check against database:postgres user:postgres 2016-03-23 10:46:51: pid 58460: DEBUG: Backend DB node 0 status is 2 2016-03-23 10:46:51: pid 58460: DEBUG: Trying to make persistent DB connection to backend node 0 having status 2 2016-03-23 10:46:51: pid 58460: DEBUG: pool_read: read 327 bytes from backend 0 2016-03-23 10:46:51: pid 58460: DEBUG: authenticate kind = 0 2016-03-23 10:46:51: pid 58460: DEBUG: authenticate backend: key data received 2016-03-23 10:46:51: pid 58460: DEBUG: authenticate backend: transaction state: I 2016-03-23 10:46:51: pid 58460: DEBUG: persistent DB connection to backend node 0 having status 2 is successful 2016-03-23 10:46:51: pid 58460: DEBUG: pool_write: to backend: kind:X 2016-03-23 10:46:51: pid 58460: DEBUG: Backend DB node 1 status is 3 2016-03-23 10:46:51: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:46:51: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:46:52: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:46:52: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:46:53: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:46:53: pid 59036: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:46:54: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:46:54: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:46:55: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:46:55: pid 59036: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:46:56: pid 58496: DEBUG: pool_read: read 327 bytes from backend 0 2016-03-23 10:46:56: pid 58496: DEBUG: authenticate kind = 0 2016-03-23 10:46:56: pid 58496: DEBUG: authenticate backend: key data received 2016-03-23 10:46:56: pid 58496: DEBUG: authenticate backend: transaction state: I 2016-03-23 10:46:56: pid 58496: DEBUG: pool_write: to backend: kind:X 2016-03-23 10:46:56: pid 58460: DEBUG: starting health check 2016-03-23 10:46:56: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:46:56: pid 58460: DEBUG: doing health check against database:postgres user:postgres 2016-03-23 10:46:56: pid 58460: DEBUG: Backend DB node 0 status is 2 2016-03-23 10:46:56: pid 58460: DEBUG: Trying to make persistent DB connection to backend node 0 having status 2 2016-03-23 10:46:56: pid 58460: DEBUG: pool_read: read 327 bytes from backend 0 2016-03-23 10:46:56: pid 58460: DEBUG: authenticate kind = 0 2016-03-23 10:46:56: pid 58460: DEBUG: authenticate backend: key data received 2016-03-23 10:46:56: pid 58460: DEBUG: authenticate backend: transaction state: I 2016-03-23 10:46:56: pid 58460: DEBUG: persistent DB connection to backend node 0 having status 2 is successful 2016-03-23 10:46:56: pid 58460: DEBUG: pool_write: to backend: kind:X 2016-03-23 10:46:56: pid 58460: DEBUG: Backend DB node 1 status is 3 2016-03-23 10:46:56: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:46:56: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:46:56: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:46:56: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:46:57: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:46:57: pid 59036: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:46:58: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:46:58: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:46:59: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:46:59: pid 59036: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:46:59: pid 58461: DEBUG: received packet tyep M while need packet type 2016-03-23 10:46:59: pid 58461: DEBUG: STATE MACHINE INVOKED WITH EVENT = PACKET RECEIVED Current State = STANDBY 2016-03-23 10:46:59: pid 58461: DEBUG: sending watchdog packet Socket:7, Type:[NODE INFO], Command_ID:14, data Length:170 2016-03-23 10:47:00: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:00: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:01: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:47:01: pid 59036: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:47:01: pid 58460: DEBUG: starting health check 2016-03-23 10:47:01: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:47:01: pid 58460: DEBUG: doing health check against database:postgres user:postgres 2016-03-23 10:47:01: pid 58460: DEBUG: Backend DB node 0 status is 2 2016-03-23 10:47:01: pid 58460: DEBUG: Trying to make persistent DB connection to backend node 0 having status 2 2016-03-23 10:47:01: pid 58460: DEBUG: pool_read: read 327 bytes from backend 0 2016-03-23 10:47:01: pid 58460: DEBUG: authenticate kind = 0 2016-03-23 10:47:01: pid 58460: DEBUG: authenticate backend: key data received 2016-03-23 10:47:01: pid 58460: DEBUG: authenticate backend: transaction state: I 2016-03-23 10:47:01: pid 58460: DEBUG: persistent DB connection to backend node 0 having status 2 is successful 2016-03-23 10:47:01: pid 58460: DEBUG: pool_write: to backend: kind:X 2016-03-23 10:47:01: pid 58460: DEBUG: Backend DB node 1 status is 3 2016-03-23 10:47:01: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:47:01: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:47:02: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:02: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:03: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:47:03: pid 59036: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:47:04: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:04: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:05: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:47:05: pid 59036: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:47:06: pid 58496: DEBUG: pool_read: read 327 bytes from backend 0 2016-03-23 10:47:06: pid 58496: DEBUG: authenticate kind = 0 2016-03-23 10:47:06: pid 58496: DEBUG: authenticate backend: key data received 2016-03-23 10:47:06: pid 58496: DEBUG: authenticate backend: transaction state: I 2016-03-23 10:47:06: pid 58496: DEBUG: pool_write: to backend: kind:X 2016-03-23 10:47:06: pid 58460: DEBUG: starting health check 2016-03-23 10:47:06: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:47:06: pid 58460: DEBUG: doing health check against database:postgres user:postgres 2016-03-23 10:47:06: pid 58460: DEBUG: Backend DB node 0 status is 2 2016-03-23 10:47:06: pid 58460: DEBUG: Trying to make persistent DB connection to backend node 0 having status 2 2016-03-23 10:47:06: pid 58460: DEBUG: pool_read: read 327 bytes from backend 0 2016-03-23 10:47:06: pid 58460: DEBUG: authenticate kind = 0 2016-03-23 10:47:06: pid 58460: DEBUG: authenticate backend: key data received 2016-03-23 10:47:06: pid 58460: DEBUG: authenticate backend: transaction state: I 2016-03-23 10:47:06: pid 58460: DEBUG: persistent DB connection to backend node 0 having status 2 is successful 2016-03-23 10:47:06: pid 58460: DEBUG: pool_write: to backend: kind:X 2016-03-23 10:47:06: pid 58460: DEBUG: Backend DB node 1 status is 3 2016-03-23 10:47:06: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:47:06: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:47:06: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:06: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:07: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:47:07: pid 59036: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:47:08: pid 59034: LOG: watchdog: lifecheck started 2016-03-23 10:47:08: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:08: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:09: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:47:09: pid 59036: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:47:09: pid 58461: DEBUG: received packet tyep M while need packet type 2016-03-23 10:47:09: pid 58461: DEBUG: STATE MACHINE INVOKED WITH EVENT = PACKET RECEIVED Current State = STANDBY 2016-03-23 10:47:09: pid 58461: DEBUG: sending watchdog packet Socket:7, Type:[NODE INFO], Command_ID:15, data Length:170 2016-03-23 10:47:10: pid 59034: DEBUG: watchdog executing ping 2016-03-23 10:47:10: pid 59034: DETAIL: succeed to ping serv-ea01 2016-03-23 10:47:10: pid 59034: DEBUG: watchdog ping 2016-03-23 10:47:10: pid 59034: DETAIL: ping data: PING serv-ea01.adm.fr.clara.net (10.129.124.43) 56(84) bytes of data. --- serv-ea01.adm.fr.clara.net ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2001ms rtt min/avg/max/mdev = 0.169/0.213/0.247/0.034 ms 2016-03-23 10:47:10: pid 59034: DEBUG: watchdog executing ping 2016-03-23 10:47:10: pid 59034: DETAIL: succeed to ping serv-ew01 2016-03-23 10:47:10: pid 59034: DEBUG: watchdog ping 2016-03-23 10:47:10: pid 59034: DETAIL: ping data: PING serv-ew01.adm.fr.clara.net (10.129.124.41) 56(84) bytes of data. --- serv-ew01.adm.fr.clara.net ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2002ms rtt min/avg/max/mdev = 0.181/0.201/0.218/0.022 ms 2016-03-23 10:47:10: pid 59034: DEBUG: watchdog executing ping 2016-03-23 10:47:10: pid 59034: DETAIL: succeed to ping serv-ew02 2016-03-23 10:47:10: pid 59034: DEBUG: watchdog ping 2016-03-23 10:47:10: pid 59034: DETAIL: ping data: PING serv-ew02.adm.fr.clara.net (10.129.124.42) 56(84) bytes of data. --- serv-ew02.adm.fr.clara.net ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2001ms rtt min/avg/max/mdev = 0.154/0.243/0.290/0.062 ms 2016-03-23 10:47:10: pid 59034: DEBUG: watchdog executing ping 2016-03-23 10:47:10: pid 59034: DETAIL: succeed to ping serv-ea03 2016-03-23 10:47:10: pid 59034: DEBUG: watchdog ping 2016-03-23 10:47:10: pid 59034: DETAIL: ping data: PING serv-ea03.adm.fr.clara.net (10.129.124.45) 56(84) bytes of data. --- serv-ea03.adm.fr.clara.net ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2001ms rtt min/avg/max/mdev = 0.156/0.235/0.284/0.059 ms 2016-03-23 10:47:10: pid 59034: DEBUG: watchdog executing ping 2016-03-23 10:47:10: pid 59034: DETAIL: succeed to ping serv-ea02 2016-03-23 10:47:10: pid 59034: DEBUG: watchdog ping 2016-03-23 10:47:10: pid 59034: DETAIL: ping data: PING serv-ea02.adm.fr.clara.net (10.129.124.44) 56(84) bytes of data. --- serv-ea02.adm.fr.clara.net ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2001ms rtt min/avg/max/mdev = 0.118/0.131/0.155/0.021 ms 2016-03-23 10:47:10: pid 59034: DEBUG: watchdog life checking by heartbeat 2016-03-23 10:47:10: pid 59034: DETAIL: checking pgpool 1 (serv-ed01:9999) 2016-03-23 10:47:10: pid 59034: DEBUG: watchdog checking if pgpool is alive using heartbeat 2016-03-23 10:47:10: pid 59034: DETAIL: the last heartbeat from "serv-ed01:9999" received 1 seconds ago 2016-03-23 10:47:10: pid 59034: DEBUG: checking pgpool status by heartbeat 2016-03-23 10:47:10: pid 59034: DETAIL: OK; status OK 2016-03-23 10:47:10: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:10: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:11: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:47:11: pid 59036: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:47:11: pid 58460: DEBUG: starting health check 2016-03-23 10:47:11: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:47:11: pid 58460: DEBUG: doing health check against database:postgres user:postgres 2016-03-23 10:47:11: pid 58460: DEBUG: Backend DB node 0 status is 2 2016-03-23 10:47:11: pid 58460: DEBUG: Trying to make persistent DB connection to backend node 0 having status 2 2016-03-23 10:47:11: pid 58460: DEBUG: pool_read: read 327 bytes from backend 0 2016-03-23 10:47:11: pid 58460: DEBUG: authenticate kind = 0 2016-03-23 10:47:11: pid 58460: DEBUG: authenticate backend: key data received 2016-03-23 10:47:11: pid 58460: DEBUG: authenticate backend: transaction state: I 2016-03-23 10:47:11: pid 58460: DEBUG: persistent DB connection to backend node 0 having status 2 is successful 2016-03-23 10:47:11: pid 58460: DEBUG: pool_write: to backend: kind:X 2016-03-23 10:47:11: pid 58460: DEBUG: Backend DB node 1 status is 3 2016-03-23 10:47:11: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:47:11: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:47:12: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:12: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:13: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:47:13: pid 59036: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:47:13: pid 59034: DEBUG: Lifecheck child reaper handler 2016-03-23 10:47:13: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:14: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:14: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:15: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:47:15: pid 59036: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:47:15: pid 59034: DEBUG: watchdog executing ping 2016-03-23 10:47:15: pid 59034: DETAIL: succeed to ping serv-ew02 2016-03-23 10:47:15: pid 59034: DEBUG: watchdog ping 2016-03-23 10:47:15: pid 59034: DETAIL: ping data: PING serv-ew02.adm.fr.clara.net (10.129.124.42) 56(84) bytes of data. --- serv-ew02.adm.fr.clara.net ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2000ms rtt min/avg/max/mdev = 0.205/0.280/0.345/0.060 ms 2016-03-23 10:47:15: pid 59034: DEBUG: watchdog executing ping 2016-03-23 10:47:15: pid 59034: DETAIL: succeed to ping serv-ea01 2016-03-23 10:47:15: pid 59034: DEBUG: watchdog ping 2016-03-23 10:47:15: pid 59034: DETAIL: ping data: PING serv-ea01.adm.fr.clara.net (10.129.124.43) 56(84) bytes of data. --- serv-ea01.adm.fr.clara.net ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2001ms rtt min/avg/max/mdev = 0.152/0.227/0.280/0.054 ms 2016-03-23 10:47:15: pid 59034: DEBUG: watchdog executing ping 2016-03-23 10:47:15: pid 59034: DETAIL: succeed to ping serv-ea02 2016-03-23 10:47:15: pid 59034: DEBUG: watchdog ping 2016-03-23 10:47:15: pid 59034: DETAIL: ping data: PING serv-ea02.adm.fr.clara.net (10.129.124.44) 56(84) bytes of data. --- serv-ea02.adm.fr.clara.net ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2000ms rtt min/avg/max/mdev = 0.094/0.115/0.155/0.030 ms 2016-03-23 10:47:15: pid 59034: DEBUG: watchdog executing ping 2016-03-23 10:47:15: pid 59034: DETAIL: succeed to ping serv-ew01 2016-03-23 10:47:15: pid 59034: DEBUG: watchdog ping 2016-03-23 10:47:15: pid 59034: DETAIL: ping data: PING serv-ew01.adm.fr.clara.net (10.129.124.41) 56(84) bytes of data. --- serv-ew01.adm.fr.clara.net ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2001ms rtt min/avg/max/mdev = 0.168/0.182/0.206/0.022 ms 2016-03-23 10:47:15: pid 59034: DEBUG: watchdog executing ping 2016-03-23 10:47:15: pid 59034: DETAIL: succeed to ping serv-ea03 2016-03-23 10:47:15: pid 59034: DEBUG: watchdog ping 2016-03-23 10:47:15: pid 59034: DETAIL: ping data: PING serv-ea03.adm.fr.clara.net (10.129.124.45) 56(84) bytes of data. --- serv-ea03.adm.fr.clara.net ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2001ms rtt min/avg/max/mdev = 0.133/0.167/0.219/0.038 ms 2016-03-23 10:47:15: pid 59034: DEBUG: watchdog life checking by heartbeat 2016-03-23 10:47:15: pid 59034: DETAIL: checking pgpool 1 (serv-ed01:9999) 2016-03-23 10:47:15: pid 59034: DEBUG: watchdog checking if pgpool is alive using heartbeat 2016-03-23 10:47:15: pid 59034: DETAIL: the last heartbeat from "serv-ed01:9999" received 0 seconds ago 2016-03-23 10:47:15: pid 59034: DEBUG: checking pgpool status by heartbeat 2016-03-23 10:47:15: pid 59034: DETAIL: OK; status OK 2016-03-23 10:47:15: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:16: pid 58496: DEBUG: pool_read: read 327 bytes from backend 0 2016-03-23 10:47:16: pid 58496: DEBUG: authenticate kind = 0 2016-03-23 10:47:16: pid 58496: DEBUG: authenticate backend: key data received 2016-03-23 10:47:16: pid 58496: DEBUG: authenticate backend: transaction state: I 2016-03-23 10:47:16: pid 58496: DEBUG: pool_write: to backend: kind:X 2016-03-23 10:47:16: pid 58460: DEBUG: starting health check 2016-03-23 10:47:16: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:47:16: pid 58460: DEBUG: doing health check against database:postgres user:postgres 2016-03-23 10:47:16: pid 58460: DEBUG: Backend DB node 0 status is 2 2016-03-23 10:47:16: pid 58460: DEBUG: Trying to make persistent DB connection to backend node 0 having status 2 2016-03-23 10:47:16: pid 58460: DEBUG: pool_read: read 327 bytes from backend 0 2016-03-23 10:47:16: pid 58460: DEBUG: authenticate kind = 0 2016-03-23 10:47:16: pid 58460: DEBUG: authenticate backend: key data received 2016-03-23 10:47:16: pid 58460: DEBUG: authenticate backend: transaction state: I 2016-03-23 10:47:16: pid 58460: DEBUG: persistent DB connection to backend node 0 having status 2 is successful 2016-03-23 10:47:16: pid 58460: DEBUG: pool_write: to backend: kind:X 2016-03-23 10:47:16: pid 58460: DEBUG: Backend DB node 1 status is 3 2016-03-23 10:47:16: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:47:16: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:47:16: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:16: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:17: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:47:17: pid 59036: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:47:17: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:18: pid 59034: DEBUG: Lifecheck child reaper handler 2016-03-23 10:47:18: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:18: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:19: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:47:19: pid 59036: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:47:19: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:20: pid 59034: DEBUG: watchdog executing ping 2016-03-23 10:47:20: pid 59034: DETAIL: succeed to ping serv-ea02 2016-03-23 10:47:20: pid 59034: DEBUG: watchdog ping 2016-03-23 10:47:20: pid 59034: DETAIL: ping data: PING serv-ea02.adm.fr.clara.net (10.129.124.44) 56(84) bytes of data. --- serv-ea02.adm.fr.clara.net ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2000ms rtt min/avg/max/mdev = 0.121/0.130/0.144/0.013 ms 2016-03-23 10:47:20: pid 59034: DEBUG: watchdog executing ping 2016-03-23 10:47:20: pid 59034: DETAIL: succeed to ping serv-ew01 2016-03-23 10:47:20: pid 59034: DEBUG: watchdog ping 2016-03-23 10:47:20: pid 59034: DETAIL: ping data: PING serv-ew01.adm.fr.clara.net (10.129.124.41) 56(84) bytes of data. --- serv-ew01.adm.fr.clara.net ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2001ms rtt min/avg/max/mdev = 0.137/0.182/0.240/0.044 ms 2016-03-23 10:47:20: pid 59034: DEBUG: watchdog executing ping 2016-03-23 10:47:20: pid 59034: DETAIL: succeed to ping serv-ea01 2016-03-23 10:47:20: pid 59034: DEBUG: watchdog ping 2016-03-23 10:47:20: pid 59034: DETAIL: ping data: PING serv-ea01.adm.fr.clara.net (10.129.124.43) 56(84) bytes of data. --- serv-ea01.adm.fr.clara.net ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2001ms rtt min/avg/max/mdev = 0.181/0.252/0.337/0.064 ms 2016-03-23 10:47:20: pid 59034: DEBUG: watchdog executing ping 2016-03-23 10:47:20: pid 59034: DETAIL: succeed to ping serv-ew02 2016-03-23 10:47:20: pid 59034: DEBUG: watchdog ping 2016-03-23 10:47:20: pid 59034: DETAIL: ping data: PING serv-ew02.adm.fr.clara.net (10.129.124.42) 56(84) bytes of data. --- serv-ew02.adm.fr.clara.net ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2002ms rtt min/avg/max/mdev = 0.201/0.233/0.294/0.044 ms 2016-03-23 10:47:20: pid 59034: DEBUG: watchdog executing ping 2016-03-23 10:47:20: pid 59034: DETAIL: succeed to ping serv-ea03 2016-03-23 10:47:20: pid 59034: DEBUG: watchdog ping 2016-03-23 10:47:20: pid 59034: DETAIL: ping data: PING serv-ea03.adm.fr.clara.net (10.129.124.45) 56(84) bytes of data. --- serv-ea03.adm.fr.clara.net ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2001ms rtt min/avg/max/mdev = 0.157/0.203/0.266/0.047 ms 2016-03-23 10:47:20: pid 59034: DEBUG: watchdog life checking by heartbeat 2016-03-23 10:47:20: pid 59034: DETAIL: checking pgpool 1 (serv-ed01:9999) 2016-03-23 10:47:20: pid 59034: DEBUG: watchdog checking if pgpool is alive using heartbeat 2016-03-23 10:47:20: pid 59034: DETAIL: the last heartbeat from "serv-ed01:9999" received 0 seconds ago 2016-03-23 10:47:20: pid 59034: DEBUG: checking pgpool status by heartbeat 2016-03-23 10:47:20: pid 59034: DETAIL: OK; status OK 2016-03-23 10:47:20: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:20: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:20: pid 58461: DEBUG: received packet tyep M while need packet type 2016-03-23 10:47:20: pid 58461: DEBUG: STATE MACHINE INVOKED WITH EVENT = PACKET RECEIVED Current State = STANDBY 2016-03-23 10:47:20: pid 58461: DEBUG: sending watchdog packet Socket:7, Type:[NODE INFO], Command_ID:16, data Length:170 2016-03-23 10:47:21: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:47:21: pid 59036: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:47:21: pid 58460: DEBUG: starting health check 2016-03-23 10:47:21: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:47:21: pid 58460: DEBUG: doing health check against database:postgres user:postgres 2016-03-23 10:47:21: pid 58460: DEBUG: Backend DB node 0 status is 2 2016-03-23 10:47:21: pid 58460: DEBUG: Trying to make persistent DB connection to backend node 0 having status 2 2016-03-23 10:47:21: pid 58460: DEBUG: pool_read: read 327 bytes from backend 0 2016-03-23 10:47:21: pid 58460: DEBUG: authenticate kind = 0 2016-03-23 10:47:21: pid 58460: DEBUG: authenticate backend: key data received 2016-03-23 10:47:21: pid 58460: DEBUG: authenticate backend: transaction state: I 2016-03-23 10:47:21: pid 58460: DEBUG: persistent DB connection to backend node 0 having status 2 is successful 2016-03-23 10:47:21: pid 58460: DEBUG: pool_write: to backend: kind:X 2016-03-23 10:47:21: pid 58460: DEBUG: Backend DB node 1 status is 3 2016-03-23 10:47:21: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:47:21: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:47:21: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:22: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:22: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:23: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:47:23: pid 59036: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:47:23: pid 59034: DEBUG: Lifecheck child reaper handler 2016-03-23 10:47:23: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:24: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:24: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:25: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:47:25: pid 59036: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:47:25: pid 59034: DEBUG: watchdog executing ping 2016-03-23 10:47:25: pid 59034: DETAIL: succeed to ping serv-ea01 2016-03-23 10:47:25: pid 59034: DEBUG: watchdog ping 2016-03-23 10:47:25: pid 59034: DETAIL: ping data: PING serv-ea01.adm.fr.clara.net (10.129.124.43) 56(84) bytes of data. --- serv-ea01.adm.fr.clara.net ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2001ms rtt min/avg/max/mdev = 0.176/0.213/0.240/0.027 ms 2016-03-23 10:47:25: pid 59034: DEBUG: watchdog executing ping 2016-03-23 10:47:25: pid 59034: DETAIL: succeed to ping serv-ew01 2016-03-23 10:47:25: pid 59034: DEBUG: watchdog ping 2016-03-23 10:47:25: pid 59034: DETAIL: ping data: PING serv-ew01.adm.fr.clara.net (10.129.124.41) 56(84) bytes of data. --- serv-ew01.adm.fr.clara.net ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2001ms rtt min/avg/max/mdev = 0.155/0.166/0.177/0.013 ms 2016-03-23 10:47:25: pid 59034: DEBUG: watchdog executing ping 2016-03-23 10:47:25: pid 59034: DETAIL: succeed to ping serv-ew02 2016-03-23 10:47:25: pid 59034: DEBUG: watchdog ping 2016-03-23 10:47:25: pid 59034: DETAIL: ping data: PING serv-ew02.adm.fr.clara.net (10.129.124.42) 56(84) bytes of data. --- serv-ew02.adm.fr.clara.net ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2001ms rtt min/avg/max/mdev = 0.162/0.192/0.241/0.038 ms 2016-03-23 10:47:25: pid 59034: DEBUG: watchdog executing ping 2016-03-23 10:47:25: pid 59034: DETAIL: succeed to ping serv-ea02 2016-03-23 10:47:25: pid 59034: DEBUG: watchdog ping 2016-03-23 10:47:25: pid 59034: DETAIL: ping data: PING serv-ea02.adm.fr.clara.net (10.129.124.44) 56(84) bytes of data. --- serv-ea02.adm.fr.clara.net ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2001ms rtt min/avg/max/mdev = 0.091/0.105/0.124/0.016 ms 2016-03-23 10:47:25: pid 59034: DEBUG: watchdog executing ping 2016-03-23 10:47:25: pid 59034: DETAIL: succeed to ping serv-ea03 2016-03-23 10:47:25: pid 59034: DEBUG: watchdog ping 2016-03-23 10:47:25: pid 59034: DETAIL: ping data: PING serv-ea03.adm.fr.clara.net (10.129.124.45) 56(84) bytes of data. --- serv-ea03.adm.fr.clara.net ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2001ms rtt min/avg/max/mdev = 0.128/0.199/0.316/0.083 ms 2016-03-23 10:47:25: pid 59034: DEBUG: watchdog life checking by heartbeat 2016-03-23 10:47:25: pid 59034: DETAIL: checking pgpool 1 (serv-ed01:9999) 2016-03-23 10:47:25: pid 59034: DEBUG: watchdog checking if pgpool is alive using heartbeat 2016-03-23 10:47:25: pid 59034: DETAIL: the last heartbeat from "serv-ed01:9999" received 0 seconds ago 2016-03-23 10:47:25: pid 59034: DEBUG: checking pgpool status by heartbeat 2016-03-23 10:47:25: pid 59034: DETAIL: OK; status OK 2016-03-23 10:47:25: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:26: pid 58496: DEBUG: pool_read: read 327 bytes from backend 0 2016-03-23 10:47:26: pid 58496: DEBUG: authenticate kind = 0 2016-03-23 10:47:26: pid 58496: DEBUG: authenticate backend: key data received 2016-03-23 10:47:26: pid 58496: DEBUG: authenticate backend: transaction state: I 2016-03-23 10:47:26: pid 58496: DEBUG: pool_write: to backend: kind:X 2016-03-23 10:47:26: pid 58460: DEBUG: starting health check 2016-03-23 10:47:26: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:47:26: pid 58460: DEBUG: doing health check against database:postgres user:postgres 2016-03-23 10:47:26: pid 58460: DEBUG: Backend DB node 0 status is 2 2016-03-23 10:47:26: pid 58460: DEBUG: Trying to make persistent DB connection to backend node 0 having status 2 2016-03-23 10:47:26: pid 58460: DEBUG: pool_read: read 327 bytes from backend 0 2016-03-23 10:47:26: pid 58460: DEBUG: authenticate kind = 0 2016-03-23 10:47:26: pid 58460: DEBUG: authenticate backend: key data received 2016-03-23 10:47:26: pid 58460: DEBUG: authenticate backend: transaction state: I 2016-03-23 10:47:26: pid 58460: DEBUG: persistent DB connection to backend node 0 having status 2 is successful 2016-03-23 10:47:26: pid 58460: DEBUG: pool_write: to backend: kind:X 2016-03-23 10:47:26: pid 58460: DEBUG: Backend DB node 1 status is 3 2016-03-23 10:47:26: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:47:26: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:47:26: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:26: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:27: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:47:27: pid 59036: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:47:27: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:28: pid 59034: DEBUG: Lifecheck child reaper handler 2016-03-23 10:47:28: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:28: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:29: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:47:29: pid 59036: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:47:29: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:30: pid 59034: DEBUG: watchdog executing ping 2016-03-23 10:47:30: pid 59034: DETAIL: succeed to ping serv-ew01 2016-03-23 10:47:30: pid 59034: DEBUG: watchdog ping 2016-03-23 10:47:30: pid 59034: DETAIL: ping data: PING serv-ew01.adm.fr.clara.net (10.129.124.41) 56(84) bytes of data. --- serv-ew01.adm.fr.clara.net ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2001ms rtt min/avg/max/mdev = 0.153/0.204/0.260/0.046 ms 2016-03-23 10:47:30: pid 59034: DEBUG: watchdog executing ping 2016-03-23 10:47:30: pid 59034: DETAIL: succeed to ping serv-ew02 2016-03-23 10:47:30: pid 59034: DEBUG: watchdog ping 2016-03-23 10:47:30: pid 59034: DETAIL: ping data: PING serv-ew02.adm.fr.clara.net (10.129.124.42) 56(84) bytes of data. --- serv-ew02.adm.fr.clara.net ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2002ms rtt min/avg/max/mdev = 0.145/0.202/0.286/0.060 ms 2016-03-23 10:47:30: pid 59034: DEBUG: watchdog executing ping 2016-03-23 10:47:30: pid 59034: DETAIL: succeed to ping serv-ea02 2016-03-23 10:47:30: pid 59034: DEBUG: watchdog ping 2016-03-23 10:47:30: pid 59034: DETAIL: ping data: PING serv-ea02.adm.fr.clara.net (10.129.124.44) 56(84) bytes of data. --- serv-ea02.adm.fr.clara.net ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2002ms rtt min/avg/max/mdev = 0.095/0.112/0.124/0.012 ms 2016-03-23 10:47:30: pid 59034: DEBUG: watchdog executing ping 2016-03-23 10:47:30: pid 59034: DETAIL: succeed to ping serv-ea01 2016-03-23 10:47:30: pid 59034: DEBUG: watchdog ping 2016-03-23 10:47:30: pid 59034: DETAIL: ping data: PING serv-ea01.adm.fr.clara.net (10.129.124.43) 56(84) bytes of data. --- serv-ea01.adm.fr.clara.net ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2002ms rtt min/avg/max/mdev = 0.173/0.218/0.249/0.034 ms 2016-03-23 10:47:30: pid 59034: DEBUG: watchdog executing ping 2016-03-23 10:47:30: pid 59034: DETAIL: succeed to ping serv-ea03 2016-03-23 10:47:30: pid 59034: DEBUG: watchdog ping 2016-03-23 10:47:30: pid 59034: DETAIL: ping data: PING serv-ea03.adm.fr.clara.net (10.129.124.45) 56(84) bytes of data. --- serv-ea03.adm.fr.clara.net ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2002ms rtt min/avg/max/mdev = 0.229/0.236/0.240/0.005 ms 2016-03-23 10:47:30: pid 59034: DEBUG: watchdog life checking by heartbeat 2016-03-23 10:47:30: pid 59034: DETAIL: checking pgpool 1 (serv-ed01:9999) 2016-03-23 10:47:30: pid 59034: DEBUG: watchdog checking if pgpool is alive using heartbeat 2016-03-23 10:47:30: pid 59034: DETAIL: the last heartbeat from "serv-ed01:9999" received 0 seconds ago 2016-03-23 10:47:30: pid 59034: DEBUG: checking pgpool status by heartbeat 2016-03-23 10:47:30: pid 59034: DETAIL: OK; status OK 2016-03-23 10:47:30: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:30: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:30: pid 58461: DEBUG: received packet tyep M while need packet type 2016-03-23 10:47:30: pid 58461: DEBUG: STATE MACHINE INVOKED WITH EVENT = PACKET RECEIVED Current State = STANDBY 2016-03-23 10:47:30: pid 58461: DEBUG: sending watchdog packet Socket:7, Type:[NODE INFO], Command_ID:17, data Length:170 2016-03-23 10:47:31: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:47:31: pid 59036: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:47:31: pid 58460: DEBUG: starting health check 2016-03-23 10:47:31: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:47:31: pid 58460: DEBUG: doing health check against database:postgres user:postgres 2016-03-23 10:47:31: pid 58460: DEBUG: Backend DB node 0 status is 2 2016-03-23 10:47:31: pid 58460: DEBUG: Trying to make persistent DB connection to backend node 0 having status 2 2016-03-23 10:47:31: pid 58460: DEBUG: pool_read: read 327 bytes from backend 0 2016-03-23 10:47:31: pid 58460: DEBUG: authenticate kind = 0 2016-03-23 10:47:31: pid 58460: DEBUG: authenticate backend: key data received 2016-03-23 10:47:31: pid 58460: DEBUG: authenticate backend: transaction state: I 2016-03-23 10:47:31: pid 58460: DEBUG: persistent DB connection to backend node 0 having status 2 is successful 2016-03-23 10:47:31: pid 58460: DEBUG: pool_write: to backend: kind:X 2016-03-23 10:47:31: pid 58460: DEBUG: Backend DB node 1 status is 3 2016-03-23 10:47:31: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:47:31: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:47:31: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:32: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:32: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:33: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:47:33: pid 59036: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:47:33: pid 59034: DEBUG: Lifecheck child reaper handler 2016-03-23 10:47:33: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:34: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:34: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:35: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:47:35: pid 59036: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:47:35: pid 59034: DEBUG: watchdog executing ping 2016-03-23 10:47:35: pid 59034: DETAIL: succeed to ping serv-ea03 2016-03-23 10:47:35: pid 59034: DEBUG: watchdog ping 2016-03-23 10:47:35: pid 59034: DETAIL: ping data: PING serv-ea03.adm.fr.clara.net (10.129.124.45) 56(84) bytes of data. --- serv-ea03.adm.fr.clara.net ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2001ms rtt min/avg/max/mdev = 0.175/0.189/0.202/0.015 ms 2016-03-23 10:47:35: pid 59034: DEBUG: watchdog executing ping 2016-03-23 10:47:35: pid 59034: DETAIL: succeed to ping serv-ew01 2016-03-23 10:47:35: pid 59034: DEBUG: watchdog ping 2016-03-23 10:47:35: pid 59034: DETAIL: ping data: PING serv-ew01.adm.fr.clara.net (10.129.124.41) 56(84) bytes of data. --- serv-ew01.adm.fr.clara.net ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2002ms rtt min/avg/max/mdev = 0.174/0.195/0.228/0.026 ms 2016-03-23 10:47:35: pid 59034: DEBUG: watchdog executing ping 2016-03-23 10:47:35: pid 59034: DETAIL: succeed to ping serv-ea01 2016-03-23 10:47:35: pid 59034: DEBUG: watchdog ping 2016-03-23 10:47:35: pid 59034: DETAIL: ping data: PING serv-ea01.adm.fr.clara.net (10.129.124.43) 56(84) bytes of data. --- serv-ea01.adm.fr.clara.net ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2001ms rtt min/avg/max/mdev = 0.216/0.234/0.249/0.022 ms 2016-03-23 10:47:35: pid 59034: DEBUG: watchdog executing ping 2016-03-23 10:47:35: pid 59034: DETAIL: succeed to ping serv-ea02 2016-03-23 10:47:35: pid 59034: DEBUG: watchdog ping 2016-03-23 10:47:35: pid 59034: DETAIL: ping data: PING serv-ea02.adm.fr.clara.net (10.129.124.44) 56(84) bytes of data. --- serv-ea02.adm.fr.clara.net ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2002ms rtt min/avg/max/mdev = 0.116/0.120/0.126/0.004 ms 2016-03-23 10:47:35: pid 59034: DEBUG: watchdog executing ping 2016-03-23 10:47:35: pid 59034: DETAIL: succeed to ping serv-ew02 2016-03-23 10:47:35: pid 59034: DEBUG: watchdog ping 2016-03-23 10:47:35: pid 59034: DETAIL: ping data: PING serv-ew02.adm.fr.clara.net (10.129.124.42) 56(84) bytes of data. --- serv-ew02.adm.fr.clara.net ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2001ms rtt min/avg/max/mdev = 0.139/0.183/0.242/0.043 ms 2016-03-23 10:47:35: pid 59034: DEBUG: watchdog life checking by heartbeat 2016-03-23 10:47:35: pid 59034: DETAIL: checking pgpool 1 (serv-ed01:9999) 2016-03-23 10:47:35: pid 59034: DEBUG: watchdog checking if pgpool is alive using heartbeat 2016-03-23 10:47:35: pid 59034: DETAIL: the last heartbeat from "serv-ed01:9999" received 0 seconds ago 2016-03-23 10:47:35: pid 59034: DEBUG: checking pgpool status by heartbeat 2016-03-23 10:47:35: pid 59034: DETAIL: OK; status OK 2016-03-23 10:47:35: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:36: pid 58496: DEBUG: pool_read: read 327 bytes from backend 0 2016-03-23 10:47:36: pid 58496: DEBUG: authenticate kind = 0 2016-03-23 10:47:36: pid 58496: DEBUG: authenticate backend: key data received 2016-03-23 10:47:36: pid 58496: DEBUG: authenticate backend: transaction state: I 2016-03-23 10:47:36: pid 58496: DEBUG: pool_write: to backend: kind:X 2016-03-23 10:47:36: pid 58460: DEBUG: starting health check 2016-03-23 10:47:36: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:47:36: pid 58460: DEBUG: doing health check against database:postgres user:postgres 2016-03-23 10:47:36: pid 58460: DEBUG: Backend DB node 0 status is 2 2016-03-23 10:47:36: pid 58460: DEBUG: Trying to make persistent DB connection to backend node 0 having status 2 2016-03-23 10:47:36: pid 58460: DEBUG: pool_read: read 327 bytes from backend 0 2016-03-23 10:47:36: pid 58460: DEBUG: authenticate kind = 0 2016-03-23 10:47:36: pid 58460: DEBUG: authenticate backend: key data received 2016-03-23 10:47:36: pid 58460: DEBUG: authenticate backend: transaction state: I 2016-03-23 10:47:36: pid 58460: DEBUG: persistent DB connection to backend node 0 having status 2 is successful 2016-03-23 10:47:36: pid 58460: DEBUG: pool_write: to backend: kind:X 2016-03-23 10:47:36: pid 58460: DEBUG: Backend DB node 1 status is 3 2016-03-23 10:47:36: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:47:36: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:47:36: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:36: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:37: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:47:37: pid 59036: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:47:37: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:38: pid 59034: DEBUG: Lifecheck child reaper handler 2016-03-23 10:47:38: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:38: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:39: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:47:39: pid 59036: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:47:39: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:40: pid 59034: DEBUG: watchdog executing ping 2016-03-23 10:47:40: pid 59034: DETAIL: succeed to ping serv-ea01 2016-03-23 10:47:40: pid 59034: DEBUG: watchdog ping 2016-03-23 10:47:40: pid 59034: DETAIL: ping data: PING serv-ea01.adm.fr.clara.net (10.129.124.43) 56(84) bytes of data. --- serv-ea01.adm.fr.clara.net ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2001ms rtt min/avg/max/mdev = 0.218/0.275/0.321/0.047 ms 2016-03-23 10:47:40: pid 59034: DEBUG: watchdog executing ping 2016-03-23 10:47:40: pid 59034: DETAIL: succeed to ping serv-ew01 2016-03-23 10:47:40: pid 59034: DEBUG: watchdog ping 2016-03-23 10:47:40: pid 59034: DETAIL: ping data: PING serv-ew01.adm.fr.clara.net (10.129.124.41) 56(84) bytes of data. --- serv-ew01.adm.fr.clara.net ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2000ms rtt min/avg/max/mdev = 0.164/0.192/0.214/0.020 ms 2016-03-23 10:47:40: pid 59034: DEBUG: watchdog executing ping 2016-03-23 10:47:40: pid 59034: DETAIL: succeed to ping serv-ea02 2016-03-23 10:47:40: pid 59034: DEBUG: watchdog ping 2016-03-23 10:47:40: pid 59034: DETAIL: ping data: PING serv-ea02.adm.fr.clara.net (10.129.124.44) 56(84) bytes of data. --- serv-ea02.adm.fr.clara.net ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2001ms rtt min/avg/max/mdev = 0.114/0.127/0.149/0.018 ms 2016-03-23 10:47:40: pid 59034: DEBUG: watchdog executing ping 2016-03-23 10:47:40: pid 59034: DETAIL: succeed to ping serv-ea03 2016-03-23 10:47:40: pid 59034: DEBUG: watchdog ping 2016-03-23 10:47:40: pid 59034: DETAIL: ping data: PING serv-ea03.adm.fr.clara.net (10.129.124.45) 56(84) bytes of data. --- serv-ea03.adm.fr.clara.net ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2003ms rtt min/avg/max/mdev = 0.112/0.177/0.237/0.051 ms 2016-03-23 10:47:40: pid 59034: DEBUG: watchdog executing ping 2016-03-23 10:47:40: pid 59034: DETAIL: succeed to ping serv-ew02 2016-03-23 10:47:40: pid 59034: DEBUG: watchdog ping 2016-03-23 10:47:40: pid 59034: DETAIL: ping data: PING serv-ew02.adm.fr.clara.net (10.129.124.42) 56(84) bytes of data. --- serv-ew02.adm.fr.clara.net ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2004ms rtt min/avg/max/mdev = 0.184/0.256/0.310/0.053 ms 2016-03-23 10:47:40: pid 59034: DEBUG: watchdog life checking by heartbeat 2016-03-23 10:47:40: pid 59034: DETAIL: checking pgpool 1 (serv-ed01:9999) 2016-03-23 10:47:40: pid 59034: DEBUG: watchdog checking if pgpool is alive using heartbeat 2016-03-23 10:47:40: pid 59034: DETAIL: the last heartbeat from "serv-ed01:9999" received 0 seconds ago 2016-03-23 10:47:40: pid 59034: DEBUG: checking pgpool status by heartbeat 2016-03-23 10:47:40: pid 59034: DETAIL: OK; status OK 2016-03-23 10:47:40: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:40: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:40: pid 58461: DEBUG: received packet tyep M while need packet type 2016-03-23 10:47:40: pid 58461: DEBUG: STATE MACHINE INVOKED WITH EVENT = PACKET RECEIVED Current State = STANDBY 2016-03-23 10:47:40: pid 58461: DEBUG: sending watchdog packet Socket:7, Type:[NODE INFO], Command_ID:18, data Length:170 2016-03-23 10:47:41: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:47:41: pid 59036: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:47:41: pid 58460: DEBUG: starting health check 2016-03-23 10:47:41: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:47:41: pid 58460: DEBUG: doing health check against database:postgres user:postgres 2016-03-23 10:47:41: pid 58460: DEBUG: Backend DB node 0 status is 2 2016-03-23 10:47:41: pid 58460: DEBUG: Trying to make persistent DB connection to backend node 0 having status 2 2016-03-23 10:47:41: pid 58460: DEBUG: pool_read: read 327 bytes from backend 0 2016-03-23 10:47:41: pid 58460: DEBUG: authenticate kind = 0 2016-03-23 10:47:41: pid 58460: DEBUG: authenticate backend: key data received 2016-03-23 10:47:41: pid 58460: DEBUG: authenticate backend: transaction state: I 2016-03-23 10:47:41: pid 58460: DEBUG: persistent DB connection to backend node 0 having status 2 is successful 2016-03-23 10:47:41: pid 58460: DEBUG: pool_write: to backend: kind:X 2016-03-23 10:47:41: pid 58460: DEBUG: Backend DB node 1 status is 3 2016-03-23 10:47:41: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:47:41: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:47:41: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:42: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:42: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:43: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:47:43: pid 59036: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:47:43: pid 59034: DEBUG: Lifecheck child reaper handler 2016-03-23 10:47:43: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:44: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:44: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:45: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:47:45: pid 59036: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:47:45: pid 59034: DEBUG: watchdog executing ping 2016-03-23 10:47:45: pid 59034: DETAIL: succeed to ping serv-ea02 2016-03-23 10:47:45: pid 59034: DEBUG: watchdog ping 2016-03-23 10:47:45: pid 59034: DETAIL: ping data: PING serv-ea02.adm.fr.clara.net (10.129.124.44) 56(84) bytes of data. --- serv-ea02.adm.fr.clara.net ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2001ms rtt min/avg/max/mdev = 0.132/0.139/0.145/0.011 ms 2016-03-23 10:47:45: pid 59034: DEBUG: watchdog executing ping 2016-03-23 10:47:45: pid 59034: DETAIL: succeed to ping serv-ew01 2016-03-23 10:47:45: pid 59034: DEBUG: watchdog ping 2016-03-23 10:47:45: pid 59034: DETAIL: ping data: PING serv-ew01.adm.fr.clara.net (10.129.124.41) 56(84) bytes of data. --- serv-ew01.adm.fr.clara.net ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2001ms rtt min/avg/max/mdev = 0.190/0.227/0.269/0.034 ms 2016-03-23 10:47:45: pid 59034: DEBUG: watchdog executing ping 2016-03-23 10:47:45: pid 59034: DETAIL: succeed to ping serv-ew02 2016-03-23 10:47:45: pid 59034: DEBUG: watchdog ping 2016-03-23 10:47:45: pid 59034: DETAIL: ping data: PING serv-ew02.adm.fr.clara.net (10.129.124.42) 56(84) bytes of data. --- serv-ew02.adm.fr.clara.net ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2001ms rtt min/avg/max/mdev = 0.176/0.200/0.228/0.021 ms 2016-03-23 10:47:45: pid 59034: DEBUG: watchdog executing ping 2016-03-23 10:47:45: pid 59034: DETAIL: succeed to ping serv-ea03 2016-03-23 10:47:45: pid 59034: DEBUG: watchdog ping 2016-03-23 10:47:45: pid 59034: DETAIL: ping data: PING serv-ea03.adm.fr.clara.net (10.129.124.45) 56(84) bytes of data. --- serv-ea03.adm.fr.clara.net ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2001ms rtt min/avg/max/mdev = 0.150/0.185/0.219/0.030 ms 2016-03-23 10:47:45: pid 59034: DEBUG: watchdog executing ping 2016-03-23 10:47:45: pid 59034: DETAIL: succeed to ping serv-ea01 2016-03-23 10:47:45: pid 59034: DEBUG: watchdog ping 2016-03-23 10:47:45: pid 59034: DETAIL: ping data: PING serv-ea01.adm.fr.clara.net (10.129.124.43) 56(84) bytes of data. --- serv-ea01.adm.fr.clara.net ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2002ms rtt min/avg/max/mdev = 0.136/0.177/0.234/0.042 ms 2016-03-23 10:47:45: pid 59034: DEBUG: watchdog life checking by heartbeat 2016-03-23 10:47:45: pid 59034: DETAIL: checking pgpool 1 (serv-ed01:9999) 2016-03-23 10:47:45: pid 59034: DEBUG: watchdog checking if pgpool is alive using heartbeat 2016-03-23 10:47:45: pid 59034: DETAIL: the last heartbeat from "serv-ed01:9999" received 0 seconds ago 2016-03-23 10:47:45: pid 59034: DEBUG: checking pgpool status by heartbeat 2016-03-23 10:47:45: pid 59034: DETAIL: OK; status OK 2016-03-23 10:47:45: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:45: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:46: pid 58496: DEBUG: pool_read: read 327 bytes from backend 0 2016-03-23 10:47:46: pid 58496: DEBUG: authenticate kind = 0 2016-03-23 10:47:46: pid 58496: DEBUG: authenticate backend: key data received 2016-03-23 10:47:46: pid 58496: DEBUG: authenticate backend: transaction state: I 2016-03-23 10:47:46: pid 58496: DEBUG: pool_write: to backend: kind:X 2016-03-23 10:47:46: pid 58460: DEBUG: starting health check 2016-03-23 10:47:46: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:47:46: pid 58460: DEBUG: doing health check against database:postgres user:postgres 2016-03-23 10:47:46: pid 58460: DEBUG: Backend DB node 0 status is 2 2016-03-23 10:47:46: pid 58460: DEBUG: Trying to make persistent DB connection to backend node 0 having status 2 2016-03-23 10:47:46: pid 58460: DEBUG: pool_read: read 327 bytes from backend 0 2016-03-23 10:47:46: pid 58460: DEBUG: authenticate kind = 0 2016-03-23 10:47:46: pid 58460: DEBUG: authenticate backend: key data received 2016-03-23 10:47:46: pid 58460: DEBUG: authenticate backend: transaction state: I 2016-03-23 10:47:46: pid 58460: DEBUG: persistent DB connection to backend node 0 having status 2 is successful 2016-03-23 10:47:46: pid 58460: DEBUG: pool_write: to backend: kind:X 2016-03-23 10:47:46: pid 58460: DEBUG: Backend DB node 1 status is 3 2016-03-23 10:47:46: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:47:46: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:47:46: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:46: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:47: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:47:47: pid 59036: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:47:47: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:47: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:48: pid 59034: DEBUG: Lifecheck child reaper handler 2016-03-23 10:47:48: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:48: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:49: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:47:49: pid 59036: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:47:49: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:49: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:50: pid 59034: DEBUG: watchdog executing ping 2016-03-23 10:47:50: pid 59034: DETAIL: succeed to ping serv-ea02 2016-03-23 10:47:50: pid 59034: DEBUG: watchdog ping 2016-03-23 10:47:50: pid 59034: DETAIL: ping data: PING serv-ea02.adm.fr.clara.net (10.129.124.44) 56(84) bytes of data. --- serv-ea02.adm.fr.clara.net ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2000ms rtt min/avg/max/mdev = 0.102/0.109/0.115/0.005 ms 2016-03-23 10:47:50: pid 59034: DEBUG: watchdog executing ping 2016-03-23 10:47:50: pid 59034: DETAIL: succeed to ping serv-ea01 2016-03-23 10:47:50: pid 59034: DEBUG: watchdog ping 2016-03-23 10:47:50: pid 59034: DETAIL: ping data: PING serv-ea01.adm.fr.clara.net (10.129.124.43) 56(84) bytes of data. --- serv-ea01.adm.fr.clara.net ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2001ms rtt min/avg/max/mdev = 0.153/0.173/0.208/0.026 ms 2016-03-23 10:47:50: pid 59034: DEBUG: watchdog executing ping 2016-03-23 10:47:50: pid 59034: DETAIL: succeed to ping serv-ew01 2016-03-23 10:47:50: pid 59034: DEBUG: watchdog ping 2016-03-23 10:47:50: pid 59034: DETAIL: ping data: PING serv-ew01.adm.fr.clara.net (10.129.124.41) 56(84) bytes of data. --- serv-ew01.adm.fr.clara.net ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2001ms rtt min/avg/max/mdev = 0.180/0.195/0.210/0.020 ms 2016-03-23 10:47:50: pid 59034: DEBUG: watchdog executing ping 2016-03-23 10:47:50: pid 59034: DETAIL: succeed to ping serv-ew02 2016-03-23 10:47:50: pid 59034: DEBUG: watchdog ping 2016-03-23 10:47:50: pid 59034: DETAIL: ping data: PING serv-ew02.adm.fr.clara.net (10.129.124.42) 56(84) bytes of data. --- serv-ew02.adm.fr.clara.net ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2001ms rtt min/avg/max/mdev = 0.168/0.216/0.244/0.034 ms 2016-03-23 10:47:50: pid 59034: DEBUG: watchdog executing ping 2016-03-23 10:47:50: pid 59034: DETAIL: succeed to ping serv-ea03 2016-03-23 10:47:50: pid 59034: DEBUG: watchdog ping 2016-03-23 10:47:50: pid 59034: DETAIL: ping data: PING serv-ea03.adm.fr.clara.net (10.129.124.45) 56(84) bytes of data. --- serv-ea03.adm.fr.clara.net ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2001ms rtt min/avg/max/mdev = 0.179/0.188/0.193/0.013 ms 2016-03-23 10:47:50: pid 59034: DEBUG: watchdog life checking by heartbeat 2016-03-23 10:47:50: pid 59034: DETAIL: checking pgpool 1 (serv-ed01:9999) 2016-03-23 10:47:50: pid 59034: DEBUG: watchdog checking if pgpool is alive using heartbeat 2016-03-23 10:47:50: pid 59034: DETAIL: the last heartbeat from "serv-ed01:9999" received 0 seconds ago 2016-03-23 10:47:50: pid 59034: DEBUG: checking pgpool status by heartbeat 2016-03-23 10:47:50: pid 59034: DETAIL: OK; status OK 2016-03-23 10:47:50: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:50: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:51: pid 59036: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:47:51: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:47:51: pid 58460: DEBUG: starting health check 2016-03-23 10:47:51: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:47:51: pid 58460: DEBUG: doing health check against database:postgres user:postgres 2016-03-23 10:47:51: pid 58460: DEBUG: Backend DB node 0 status is 2 2016-03-23 10:47:51: pid 58460: DEBUG: Trying to make persistent DB connection to backend node 0 having status 2 2016-03-23 10:47:51: pid 58460: DEBUG: pool_read: read 327 bytes from backend 0 2016-03-23 10:47:51: pid 58460: DEBUG: authenticate kind = 0 2016-03-23 10:47:51: pid 58460: DEBUG: authenticate backend: key data received 2016-03-23 10:47:51: pid 58460: DEBUG: authenticate backend: transaction state: I 2016-03-23 10:47:51: pid 58460: DEBUG: persistent DB connection to backend node 0 having status 2 is successful 2016-03-23 10:47:51: pid 58460: DEBUG: pool_write: to backend: kind:X 2016-03-23 10:47:51: pid 58460: DEBUG: Backend DB node 1 status is 3 2016-03-23 10:47:51: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:47:51: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:47:51: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:51: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:51: pid 58461: DEBUG: received packet tyep M while need packet type 2016-03-23 10:47:51: pid 58461: DEBUG: STATE MACHINE INVOKED WITH EVENT = PACKET RECEIVED Current State = STANDBY 2016-03-23 10:47:51: pid 58461: DEBUG: sending watchdog packet Socket:7, Type:[NODE INFO], Command_ID:19, data Length:170 2016-03-23 10:47:52: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:52: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:53: pid 59036: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:47:53: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:47:53: pid 59034: DEBUG: Lifecheck child reaper handler 2016-03-23 10:47:53: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:53: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:54: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:54: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:55: pid 59036: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:47:55: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:47:55: pid 59034: DEBUG: watchdog executing ping 2016-03-23 10:47:55: pid 59034: DETAIL: succeed to ping serv-ew01 2016-03-23 10:47:55: pid 59034: DEBUG: watchdog ping 2016-03-23 10:47:55: pid 59034: DETAIL: ping data: PING serv-ew01.adm.fr.clara.net (10.129.124.41) 56(84) bytes of data. --- serv-ew01.adm.fr.clara.net ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2002ms rtt min/avg/max/mdev = 0.142/0.204/0.237/0.043 ms 2016-03-23 10:47:55: pid 59034: DEBUG: watchdog executing ping 2016-03-23 10:47:55: pid 59034: DETAIL: succeed to ping serv-ew02 2016-03-23 10:47:55: pid 59034: DEBUG: watchdog ping 2016-03-23 10:47:55: pid 59034: DETAIL: ping data: PING serv-ew02.adm.fr.clara.net (10.129.124.42) 56(84) bytes of data. --- serv-ew02.adm.fr.clara.net ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2001ms rtt min/avg/max/mdev = 0.163/0.221/0.303/0.059 ms 2016-03-23 10:47:55: pid 59034: DEBUG: watchdog executing ping 2016-03-23 10:47:55: pid 59034: DETAIL: succeed to ping serv-ea01 2016-03-23 10:47:55: pid 59034: DEBUG: watchdog ping 2016-03-23 10:47:55: pid 59034: DETAIL: ping data: PING serv-ea01.adm.fr.clara.net (10.129.124.43) 56(84) bytes of data. --- serv-ea01.adm.fr.clara.net ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2001ms rtt min/avg/max/mdev = 0.162/0.227/0.319/0.066 ms 2016-03-23 10:47:55: pid 59034: DEBUG: watchdog executing ping 2016-03-23 10:47:55: pid 59034: DETAIL: succeed to ping serv-ea02 2016-03-23 10:47:55: pid 59034: DEBUG: watchdog ping 2016-03-23 10:47:55: pid 59034: DETAIL: ping data: PING serv-ea02.adm.fr.clara.net (10.129.124.44) 56(84) bytes of data. --- serv-ea02.adm.fr.clara.net ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2003ms rtt min/avg/max/mdev = 0.112/0.141/0.163/0.025 ms 2016-03-23 10:47:55: pid 59034: DEBUG: watchdog executing ping 2016-03-23 10:47:55: pid 59034: DETAIL: succeed to ping serv-ea03 2016-03-23 10:47:55: pid 59034: DEBUG: watchdog ping 2016-03-23 10:47:55: pid 59034: DETAIL: ping data: PING serv-ea03.adm.fr.clara.net (10.129.124.45) 56(84) bytes of data. --- serv-ea03.adm.fr.clara.net ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2002ms rtt min/avg/max/mdev = 0.183/0.212/0.261/0.036 ms 2016-03-23 10:47:55: pid 59034: DEBUG: watchdog life checking by heartbeat 2016-03-23 10:47:55: pid 59034: DETAIL: checking pgpool 1 (serv-ed01:9999) 2016-03-23 10:47:55: pid 59034: DEBUG: watchdog checking if pgpool is alive using heartbeat 2016-03-23 10:47:55: pid 59034: DETAIL: the last heartbeat from "serv-ed01:9999" received 0 seconds ago 2016-03-23 10:47:55: pid 59034: DEBUG: checking pgpool status by heartbeat 2016-03-23 10:47:55: pid 59034: DETAIL: OK; status OK 2016-03-23 10:47:55: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:55: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:56: pid 58496: DEBUG: pool_read: read 327 bytes from backend 0 2016-03-23 10:47:56: pid 58496: DEBUG: authenticate kind = 0 2016-03-23 10:47:56: pid 58496: DEBUG: authenticate backend: key data received 2016-03-23 10:47:56: pid 58496: DEBUG: authenticate backend: transaction state: I 2016-03-23 10:47:56: pid 58496: DEBUG: pool_write: to backend: kind:X 2016-03-23 10:47:56: pid 58460: DEBUG: starting health check 2016-03-23 10:47:56: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:47:56: pid 58460: DEBUG: doing health check against database:postgres user:postgres 2016-03-23 10:47:56: pid 58460: DEBUG: Backend DB node 0 status is 2 2016-03-23 10:47:56: pid 58460: DEBUG: Trying to make persistent DB connection to backend node 0 having status 2 2016-03-23 10:47:56: pid 58460: DEBUG: pool_read: read 327 bytes from backend 0 2016-03-23 10:47:56: pid 58460: DEBUG: authenticate kind = 0 2016-03-23 10:47:56: pid 58460: DEBUG: authenticate backend: key data received 2016-03-23 10:47:56: pid 58460: DEBUG: authenticate backend: transaction state: I 2016-03-23 10:47:56: pid 58460: DEBUG: persistent DB connection to backend node 0 having status 2 is successful 2016-03-23 10:47:56: pid 58460: DEBUG: pool_write: to backend: kind:X 2016-03-23 10:47:56: pid 58460: DEBUG: Backend DB node 1 status is 3 2016-03-23 10:47:56: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:47:56: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:47:56: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:56: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:57: pid 59036: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:47:57: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:47:57: pid 58461: DEBUG: received packet tyep V while need packet type 2016-03-23 10:47:57: pid 58461: DEBUG: STATE MACHINE INVOKED WITH EVENT = PACKET RECEIVED Current State = STANDBY 2016-03-23 10:47:57: pid 58461: DEBUG: sending watchdog packet Socket:7, Type:[ACCEPT], Command_ID:20, data Length:0 2016-03-23 10:47:57: pid 58461: LOG: received failback request for node_id: 1 from pid [58461] 2016-03-23 10:47:57: pid 58461: DEBUG: sending 10 to the parent process with PID:58460 2016-03-23 10:47:57: pid 58460: DEBUG: failover handler called 2016-03-23 10:47:57: pid 58460: DEBUG: failover handler 2016-03-23 10:47:57: pid 58460: DETAIL: starting to select new master node 2016-03-23 10:47:57: pid 58460: LOG: starting fail back. reconnect host serv-ed02(5432) 2016-03-23 10:47:57: pid 58461: LOG: new IPC connection received 2016-03-23 10:47:57: pid 58461: LOG: processing sync request from IPC socket 2016-03-23 10:47:57: pid 58461: DEBUG: sending watchdog packet Socket:7, Type:[NULL], Command_ID:8, data Length:81 2016-03-23 10:47:57: pid 58461: LOG: sync request from IPC socket is forwarded to master watchdog node "Linux_serv-ed01_9999" 2016-03-23 10:47:57: pid 58461: DETAIL: waiting for the reply from master node... 2016-03-23 10:47:57: pid 58461: DEBUG: received packet tyep D while need packet type 2016-03-23 10:47:57: pid 58461: DEBUG: STATE MACHINE INVOKED WITH EVENT = PACKET RECEIVED Current State = STANDBY 2016-03-23 10:47:57: pid 58461: LOG: new IPC connection received 2016-03-23 10:47:57: pid 58461: LOG: processing sync request from IPC socket 2016-03-23 10:47:57: pid 58461: DEBUG: sending watchdog packet Socket:7, Type:[NULL], Command_ID:9, data Length:80 2016-03-23 10:47:57: pid 58461: LOG: sync request from IPC socket is forwarded to master watchdog node "Linux_serv-ed01_9999" 2016-03-23 10:47:57: pid 58461: DETAIL: waiting for the reply from master node... 2016-03-23 10:47:57: pid 58461: DEBUG: received packet tyep D while need packet type 2016-03-23 10:47:57: pid 58461: DEBUG: STATE MACHINE INVOKED WITH EVENT = PACKET RECEIVED Current State = STANDBY 2016-03-23 10:47:57: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:57: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:58: pid 58461: LOG: new IPC connection received 2016-03-23 10:47:58: pid 58461: LOG: processing sync request from IPC socket 2016-03-23 10:47:58: pid 58461: DEBUG: sending watchdog packet Socket:7, Type:[NULL], Command_ID:10, data Length:80 2016-03-23 10:47:58: pid 58461: LOG: sync request from IPC socket is forwarded to master watchdog node "Linux_serv-ed01_9999" 2016-03-23 10:47:58: pid 58461: DETAIL: waiting for the reply from master node... 2016-03-23 10:47:58: pid 58461: DEBUG: received packet tyep D while need packet type 2016-03-23 10:47:58: pid 58461: DEBUG: STATE MACHINE INVOKED WITH EVENT = PACKET RECEIVED Current State = STANDBY 2016-03-23 10:47:58: pid 58460: LOG: Do not restart children because we are failbacking node id 1 hostserv-ed02 port:5432 and we are in streaming replication mode 2016-03-23 10:47:58: pid 58461: LOG: new IPC connection received 2016-03-23 10:47:58: pid 58461: LOG: processing sync request from IPC socket 2016-03-23 10:47:58: pid 58461: DEBUG: sending watchdog packet Socket:7, Type:[NULL], Command_ID:11, data Length:81 2016-03-23 10:47:58: pid 58461: LOG: sync request from IPC socket is forwarded to master watchdog node "Linux_serv-ed01_9999" 2016-03-23 10:47:58: pid 58461: DETAIL: waiting for the reply from master node... 2016-03-23 10:47:58: pid 58461: DEBUG: received packet tyep D while need packet type 2016-03-23 10:47:58: pid 58461: DEBUG: STATE MACHINE INVOKED WITH EVENT = PACKET RECEIVED Current State = STANDBY 2016-03-23 10:47:58: pid 58460: LOG: find_primary_node_repeatedly: waiting for finding a primary node 2016-03-23 10:47:58: pid 58460: LOG: find_primary_node: checking backend no 0 2016-03-23 10:47:58: pid 58460: DEBUG: pool_read: read 327 bytes from backend 0 2016-03-23 10:47:58: pid 58460: DEBUG: authenticate kind = 0 2016-03-23 10:47:58: pid 58460: DEBUG: authenticate backend: key data received 2016-03-23 10:47:58: pid 58460: DEBUG: authenticate backend: transaction state: I 2016-03-23 10:47:58: pid 58460: DEBUG: do_query: extended:0 query:"SELECT pg_is_in_recovery()" 2016-03-23 10:47:58: pid 58460: DEBUG: pool_write: to backend: kind:Q 2016-03-23 10:47:58: pid 58460: DEBUG: pool_read: read 75 bytes from backend 0 2016-03-23 10:47:58: pid 58460: DEBUG: do_query: kind: 'T' 2016-03-23 10:47:58: pid 58460: DEBUG: do_query: received ROW DESCRIPTION ('T') 2016-03-23 10:47:58: pid 58460: DEBUG: do_query: row description: num_fileds: 1 2016-03-23 10:47:58: pid 58460: DEBUG: do_query: kind: 'D' 2016-03-23 10:47:58: pid 58460: DEBUG: do_query: received DATA ROW ('D') 2016-03-23 10:47:58: pid 58460: DEBUG: do_query: kind: 'C' 2016-03-23 10:47:58: pid 58460: DEBUG: do_query: received COMMAND COMPLETE ('C') 2016-03-23 10:47:58: pid 58460: DEBUG: do_query: kind: 'Z' 2016-03-23 10:47:58: pid 58460: DEBUG: do_query: received READY FOR QUERY ('Z') 2016-03-23 10:47:58: pid 58460: DEBUG: pool_write: to backend: kind:X 2016-03-23 10:47:58: pid 58460: LOG: find_primary_node: primary node id is 0 2016-03-23 10:47:58: pid 58460: LOG: failover: set new primary node: 0 2016-03-23 10:47:58: pid 58460: LOG: failover: set new master node: 0 2016-03-23 10:47:58: pid 58496: LOG: worker process received restart request 2016-03-23 10:47:58: pid 58461: LOG: new IPC connection received 2016-03-23 10:47:58: pid 58461: LOG: processing sync request from IPC socket 2016-03-23 10:47:58: pid 58461: DEBUG: sending watchdog packet Socket:7, Type:[NULL], Command_ID:12, data Length:79 2016-03-23 10:47:58: pid 58461: LOG: sync request from IPC socket is forwarded to master watchdog node "Linux_serv-ed01_9999" 2016-03-23 10:47:58: pid 58461: DETAIL: waiting for the reply from master node... 2016-03-23 10:47:58: pid 58461: DEBUG: received packet tyep D while need packet type 2016-03-23 10:47:58: pid 58461: DEBUG: STATE MACHINE INVOKED WITH EVENT = PACKET RECEIVED Current State = STANDBY 2016-03-23 10:47:58: pid 58460: LOG: failback done. reconnect host serv-ed02(5432) 2016-03-23 10:47:58: pid 59034: DEBUG: Lifecheck child reaper handler 2016-03-23 10:47:58: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:58: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:59: pid 59036: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:47:59: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:47:59: pid 58495: LOG: restart request received in pcp child process 2016-03-23 10:47:59: pid 58495: DEBUG: PCP child receives shutdown request signal 15, Forwarding to all children 2016-03-23 10:47:59: pid 58495: DEBUG: PCP child receives smart shutdown request 2016-03-23 10:47:59: pid 58460: LOG: PCP child 58495 exits with status 0 in failover() 2016-03-23 10:47:59: pid 58460: LOG: fork a new PCP child pid 60202 in failover() 2016-03-23 10:47:59: pid 58460: DEBUG: reaper handler 2016-03-23 10:47:59: pid 58460: LOG: worker child process with pid: 58496 exits with status 256 2016-03-23 10:47:59: pid 60202: DEBUG: I am PCP child with pid:60202 2016-03-23 10:47:59: pid 58460: LOG: fork a new worker child process with pid: 60203 2016-03-23 10:47:59: pid 58460: DEBUG: reaper handler: exiting normally 2016-03-23 10:47:59: pid 60203: DEBUG: I am 60203 2016-03-23 10:47:59: pid 60203: DEBUG: initializing backend status 2016-03-23 10:47:59: pid 60203: DEBUG: pool_read: read 327 bytes from backend 0 2016-03-23 10:47:59: pid 60203: DEBUG: authenticate kind = 0 2016-03-23 10:47:59: pid 60203: DEBUG: authenticate backend: key data received 2016-03-23 10:47:59: pid 60203: DEBUG: authenticate backend: transaction state: I 2016-03-23 10:47:59: pid 60203: DEBUG: pool_read: read 327 bytes from backend 0 2016-03-23 10:47:59: pid 60203: DEBUG: authenticate kind = 0 2016-03-23 10:47:59: pid 60203: DEBUG: authenticate backend: key data received 2016-03-23 10:47:59: pid 60203: DEBUG: authenticate backend: transaction state: I 2016-03-23 10:47:59: pid 60203: DEBUG: do_query: extended:0 query:"SELECT pg_current_xlog_location()" 2016-03-23 10:47:59: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:47:59: pid 60203: DEBUG: pool_write: to backend: kind:Q 2016-03-23 10:47:59: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:47:59: pid 60203: DEBUG: pool_read: read 91 bytes from backend 0 2016-03-23 10:47:59: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:47:59: pid 60203: DEBUG: do_query: kind: 'T' 2016-03-23 10:47:59: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:47:59: pid 60203: DEBUG: do_query: received ROW DESCRIPTION ('T') 2016-03-23 10:47:59: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:47:59: pid 60203: DEBUG: do_query: row description: num_fileds: 1 2016-03-23 10:47:59: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:47:59: pid 60203: DEBUG: do_query: kind: 'D' 2016-03-23 10:47:59: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:47:59: pid 60203: DEBUG: do_query: received DATA ROW ('D') 2016-03-23 10:47:59: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:47:59: pid 60203: DEBUG: do_query: kind: 'C' 2016-03-23 10:47:59: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:47:59: pid 60203: DEBUG: do_query: received COMMAND COMPLETE ('C') 2016-03-23 10:47:59: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:47:59: pid 60203: DEBUG: do_query: kind: 'Z' 2016-03-23 10:47:59: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:47:59: pid 60203: DEBUG: do_query: received READY FOR QUERY ('Z') 2016-03-23 10:47:59: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:47:59: pid 60203: DEBUG: do_query: extended:0 query:"SELECT pg_last_xlog_replay_location()" 2016-03-23 10:47:59: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:47:59: pid 60203: DEBUG: pool_write: to backend: kind:Q 2016-03-23 10:47:59: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:47:59: pid 60203: DEBUG: pool_read: read 95 bytes from backend 0 2016-03-23 10:47:59: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:47:59: pid 60203: DEBUG: do_query: kind: 'T' 2016-03-23 10:47:59: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:47:59: pid 60203: DEBUG: do_query: received ROW DESCRIPTION ('T') 2016-03-23 10:47:59: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:47:59: pid 60203: DEBUG: do_query: row description: num_fileds: 1 2016-03-23 10:47:59: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:47:59: pid 60203: DEBUG: do_query: kind: 'D' 2016-03-23 10:47:59: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:47:59: pid 60203: DEBUG: do_query: received DATA ROW ('D') 2016-03-23 10:47:59: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:47:59: pid 60203: DEBUG: do_query: kind: 'C' 2016-03-23 10:47:59: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:47:59: pid 60203: DEBUG: do_query: received COMMAND COMPLETE ('C') 2016-03-23 10:47:59: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:47:59: pid 60203: DEBUG: do_query: kind: 'Z' 2016-03-23 10:47:59: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:47:59: pid 60203: DEBUG: do_query: received READY FOR QUERY ('Z') 2016-03-23 10:47:59: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:47:59: pid 60203: DEBUG: pool_write: to backend: kind:X 2016-03-23 10:47:59: pid 60203: DEBUG: pool_write: to backend: kind:X 2016-03-23 10:47:59: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:47:59: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:00: pid 59034: DEBUG: watchdog executing ping 2016-03-23 10:48:00: pid 59034: DETAIL: succeed to ping serv-ew01 2016-03-23 10:48:00: pid 59034: DEBUG: watchdog ping 2016-03-23 10:48:00: pid 59034: DETAIL: ping data: PING serv-ew01.adm.fr.clara.net (10.129.124.41) 56(84) bytes of data. --- serv-ew01.adm.fr.clara.net ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2000ms rtt min/avg/max/mdev = 0.145/0.186/0.218/0.034 ms 2016-03-23 10:48:00: pid 59034: DEBUG: watchdog executing ping 2016-03-23 10:48:00: pid 59034: DETAIL: succeed to ping serv-ew02 2016-03-23 10:48:00: pid 59034: DEBUG: watchdog ping 2016-03-23 10:48:00: pid 59034: DETAIL: ping data: PING serv-ew02.adm.fr.clara.net (10.129.124.42) 56(84) bytes of data. --- serv-ew02.adm.fr.clara.net ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2001ms rtt min/avg/max/mdev = 0.193/0.274/0.318/0.057 ms 2016-03-23 10:48:00: pid 59034: DEBUG: watchdog executing ping 2016-03-23 10:48:00: pid 59034: DETAIL: succeed to ping serv-ea01 2016-03-23 10:48:00: pid 59034: DEBUG: watchdog ping 2016-03-23 10:48:00: pid 59034: DETAIL: ping data: PING serv-ea01.adm.fr.clara.net (10.129.124.43) 56(84) bytes of data. --- serv-ea01.adm.fr.clara.net ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2001ms rtt min/avg/max/mdev = 0.196/0.251/0.358/0.077 ms 2016-03-23 10:48:00: pid 59034: DEBUG: watchdog executing ping 2016-03-23 10:48:00: pid 59034: DETAIL: succeed to ping serv-ea02 2016-03-23 10:48:00: pid 59034: DEBUG: watchdog ping 2016-03-23 10:48:00: pid 59034: DETAIL: ping data: PING serv-ea02.adm.fr.clara.net (10.129.124.44) 56(84) bytes of data. --- serv-ea02.adm.fr.clara.net ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2001ms rtt min/avg/max/mdev = 0.106/0.138/0.157/0.024 ms 2016-03-23 10:48:00: pid 59034: DEBUG: watchdog executing ping 2016-03-23 10:48:00: pid 59034: DETAIL: succeed to ping serv-ea03 2016-03-23 10:48:00: pid 59034: DEBUG: watchdog ping 2016-03-23 10:48:00: pid 59034: DETAIL: ping data: PING serv-ea03.adm.fr.clara.net (10.129.124.45) 56(84) bytes of data. --- serv-ea03.adm.fr.clara.net ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2001ms rtt min/avg/max/mdev = 0.166/0.250/0.324/0.064 ms 2016-03-23 10:48:00: pid 59034: DEBUG: watchdog life checking by heartbeat 2016-03-23 10:48:00: pid 59034: DETAIL: checking pgpool 1 (serv-ed01:9999) 2016-03-23 10:48:00: pid 59034: DEBUG: watchdog checking if pgpool is alive using heartbeat 2016-03-23 10:48:00: pid 59034: DETAIL: the last heartbeat from "serv-ed01:9999" received 0 seconds ago 2016-03-23 10:48:00: pid 59034: DEBUG: checking pgpool status by heartbeat 2016-03-23 10:48:00: pid 59034: DETAIL: OK; status OK 2016-03-23 10:48:00: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:00: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:01: pid 59036: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:48:01: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:48:01: pid 58460: DEBUG: starting health check 2016-03-23 10:48:01: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:48:01: pid 58460: DEBUG: doing health check against database:postgres user:postgres 2016-03-23 10:48:01: pid 58460: DEBUG: Backend DB node 0 status is 2 2016-03-23 10:48:01: pid 58460: DEBUG: Trying to make persistent DB connection to backend node 0 having status 2 2016-03-23 10:48:01: pid 58460: DEBUG: pool_read: read 327 bytes from backend 0 2016-03-23 10:48:01: pid 58460: DEBUG: authenticate kind = 0 2016-03-23 10:48:01: pid 58460: DEBUG: authenticate backend: key data received 2016-03-23 10:48:01: pid 58460: DEBUG: authenticate backend: transaction state: I 2016-03-23 10:48:01: pid 58460: DEBUG: persistent DB connection to backend node 0 having status 2 is successful 2016-03-23 10:48:01: pid 58460: DEBUG: pool_write: to backend: kind:X 2016-03-23 10:48:01: pid 58460: DEBUG: Backend DB node 1 status is 1 2016-03-23 10:48:01: pid 58460: DEBUG: Trying to make persistent DB connection to backend node 1 having status 1 2016-03-23 10:48:01: pid 58460: DEBUG: pool_read: read 327 bytes from backend 0 2016-03-23 10:48:01: pid 58460: DEBUG: authenticate kind = 0 2016-03-23 10:48:01: pid 58460: DEBUG: authenticate backend: key data received 2016-03-23 10:48:01: pid 58460: DEBUG: authenticate backend: transaction state: I 2016-03-23 10:48:01: pid 58460: DEBUG: persistent DB connection to backend node 1 having status 1 is successful 2016-03-23 10:48:01: pid 58460: DEBUG: pool_write: to backend: kind:X 2016-03-23 10:48:01: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:48:01: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:48:01: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:01: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:02: pid 58461: DEBUG: received packet tyep M while need packet type 2016-03-23 10:48:02: pid 58461: DEBUG: STATE MACHINE INVOKED WITH EVENT = PACKET RECEIVED Current State = STANDBY 2016-03-23 10:48:02: pid 58461: DEBUG: sending watchdog packet Socket:7, Type:[NODE INFO], Command_ID:21, data Length:170 2016-03-23 10:48:02: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:02: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:03: pid 59036: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:48:03: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:48:03: pid 59034: DEBUG: Lifecheck child reaper handler 2016-03-23 10:48:03: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:03: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:04: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:04: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:05: pid 59036: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:48:05: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:48:05: pid 59034: DEBUG: watchdog executing ping 2016-03-23 10:48:05: pid 59034: DETAIL: succeed to ping serv-ea02 2016-03-23 10:48:05: pid 59034: DEBUG: watchdog ping 2016-03-23 10:48:05: pid 59034: DETAIL: ping data: PING serv-ea02.adm.fr.clara.net (10.129.124.44) 56(84) bytes of data. --- serv-ea02.adm.fr.clara.net ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2001ms rtt min/avg/max/mdev = 0.102/0.121/0.139/0.015 ms 2016-03-23 10:48:05: pid 59034: DEBUG: watchdog executing ping 2016-03-23 10:48:05: pid 59034: DETAIL: succeed to ping serv-ew02 2016-03-23 10:48:05: pid 59034: DEBUG: watchdog ping 2016-03-23 10:48:05: pid 59034: DETAIL: ping data: PING serv-ew02.adm.fr.clara.net (10.129.124.42) 56(84) bytes of data. --- serv-ew02.adm.fr.clara.net ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2001ms rtt min/avg/max/mdev = 0.179/0.217/0.281/0.045 ms 2016-03-23 10:48:05: pid 59034: DEBUG: watchdog executing ping 2016-03-23 10:48:05: pid 59034: DETAIL: succeed to ping serv-ea03 2016-03-23 10:48:05: pid 59034: DEBUG: watchdog ping 2016-03-23 10:48:05: pid 59034: DETAIL: ping data: PING serv-ea03.adm.fr.clara.net (10.129.124.45) 56(84) bytes of data. --- serv-ea03.adm.fr.clara.net ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2001ms rtt min/avg/max/mdev = 0.167/0.195/0.246/0.037 ms 2016-03-23 10:48:05: pid 58460: DEBUG: reaper handler 2016-03-23 10:48:05: pid 58460: WARNING: watchdog lifecheck process with pid: 59034 was terminated by segmentation fault 2016-03-23 10:48:05: pid 58460: LOG: fork a new watchdog lifecheck process with pid: 60301 2016-03-23 10:48:05: pid 58460: DEBUG: reaper handler: exiting normally 2016-03-23 10:48:05: pid 60301: DEBUG: I am watchdog lifecheck child with pid:60301 2016-03-23 10:48:05: pid 58461: LOG: new IPC connection received 2016-03-23 10:48:05: pid 60301: LOG: 2 watchdog nodes are configured for lifecheck 2016-03-23 10:48:05: pid 60301: LOG: watchdog nodes ID:0 Name:"Linux_serv-ed02_9999" 2016-03-23 10:48:05: pid 60301: DETAIL: Host:"serv-ed02" WD Port:9000 pgpool-II port:9999 2016-03-23 10:48:05: pid 60301: LOG: watchdog nodes ID:1 Name:"Linux_serv-ed01_9999" 2016-03-23 10:48:05: pid 60301: DETAIL: Host:"serv-ed01" WD Port:9000 pgpool-II port:9999 2016-03-23 10:48:05: pid 60301: DEBUG: watchdog checking life check is ready 2016-03-23 10:48:05: pid 60301: DETAIL: pgpool:1 at "serv-ed01:9999" has not send the heartbeat signal yet 2016-03-23 10:48:05: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:05: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:06: pid 60302: LOG: failed to create watchdog heartbeat receive socket. 2016-03-23 10:48:06: pid 60302: DETAIL: setsockopt(SO_BINDTODEVICE) requies root privilege 2016-03-23 10:48:06: pid 60302: LOG: set SO_REUSEPORT option to the socket 2016-03-23 10:48:06: pid 60302: LOG: creating watchdog heartbeat receive socket. 2016-03-23 10:48:06: pid 60302: DETAIL: set SO_REUSEPORT 2016-03-23 10:48:06: pid 60303: LOG: creating socket for sending heartbeat 2016-03-23 10:48:06: pid 60303: DETAIL: setsockopt(SO_BINDTODEVICE) requires root privilege 2016-03-23 10:48:06: pid 60303: LOG: set SO_REUSEPORT option to the socket 2016-03-23 10:48:06: pid 60303: LOG: creating socket for sending heartbeat 2016-03-23 10:48:06: pid 60303: DETAIL: set SO_REUSEPORT 2016-03-23 10:48:06: pid 60303: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:48:06: pid 58460: DEBUG: starting health check 2016-03-23 10:48:06: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:48:06: pid 58460: DEBUG: doing health check against database:postgres user:postgres 2016-03-23 10:48:06: pid 58460: DEBUG: Backend DB node 0 status is 2 2016-03-23 10:48:06: pid 58460: DEBUG: Trying to make persistent DB connection to backend node 0 having status 2 2016-03-23 10:48:06: pid 58460: DEBUG: pool_read: read 327 bytes from backend 0 2016-03-23 10:48:06: pid 58460: DEBUG: authenticate kind = 0 2016-03-23 10:48:06: pid 58460: DEBUG: authenticate backend: key data received 2016-03-23 10:48:06: pid 58460: DEBUG: authenticate backend: transaction state: I 2016-03-23 10:48:06: pid 58460: DEBUG: persistent DB connection to backend node 0 having status 2 is successful 2016-03-23 10:48:06: pid 58460: DEBUG: pool_write: to backend: kind:X 2016-03-23 10:48:06: pid 58460: DEBUG: Backend DB node 1 status is 1 2016-03-23 10:48:06: pid 58460: DEBUG: Trying to make persistent DB connection to backend node 1 having status 1 2016-03-23 10:48:06: pid 58460: DEBUG: pool_read: read 327 bytes from backend 0 2016-03-23 10:48:06: pid 58460: DEBUG: authenticate kind = 0 2016-03-23 10:48:06: pid 58460: DEBUG: authenticate backend: key data received 2016-03-23 10:48:06: pid 58460: DEBUG: authenticate backend: transaction state: I 2016-03-23 10:48:06: pid 58460: DEBUG: persistent DB connection to backend node 1 having status 1 is successful 2016-03-23 10:48:06: pid 58460: DEBUG: pool_write: to backend: kind:X 2016-03-23 10:48:06: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:48:06: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:48:06: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:06: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:07: pid 59036: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:48:07: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:48:07: pid 60302: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:07: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:08: pid 60303: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:48:08: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:08: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:09: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:48:09: pid 59036: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:48:09: pid 60203: DEBUG: pool_read: read 327 bytes from backend 0 2016-03-23 10:48:09: pid 60203: DEBUG: authenticate kind = 0 2016-03-23 10:48:09: pid 60203: DEBUG: authenticate backend: key data received 2016-03-23 10:48:09: pid 60203: DEBUG: authenticate backend: transaction state: I 2016-03-23 10:48:09: pid 60203: DEBUG: pool_read: read 327 bytes from backend 0 2016-03-23 10:48:09: pid 60203: DEBUG: authenticate kind = 0 2016-03-23 10:48:09: pid 60203: DEBUG: authenticate backend: key data received 2016-03-23 10:48:09: pid 60203: DEBUG: authenticate backend: transaction state: I 2016-03-23 10:48:09: pid 60203: DEBUG: do_query: extended:0 query:"SELECT pg_current_xlog_location()" 2016-03-23 10:48:09: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:09: pid 60203: DEBUG: pool_write: to backend: kind:Q 2016-03-23 10:48:09: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:09: pid 60203: DEBUG: pool_read: read 91 bytes from backend 0 2016-03-23 10:48:09: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:09: pid 60203: DEBUG: do_query: kind: 'T' 2016-03-23 10:48:09: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:09: pid 60203: DEBUG: do_query: received ROW DESCRIPTION ('T') 2016-03-23 10:48:09: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:09: pid 60203: DEBUG: do_query: row description: num_fileds: 1 2016-03-23 10:48:09: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:09: pid 60203: DEBUG: do_query: kind: 'D' 2016-03-23 10:48:09: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:09: pid 60203: DEBUG: do_query: received DATA ROW ('D') 2016-03-23 10:48:09: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:09: pid 60203: DEBUG: do_query: kind: 'C' 2016-03-23 10:48:09: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:09: pid 60203: DEBUG: do_query: received COMMAND COMPLETE ('C') 2016-03-23 10:48:09: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:09: pid 60203: DEBUG: do_query: kind: 'Z' 2016-03-23 10:48:09: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:09: pid 60203: DEBUG: do_query: received READY FOR QUERY ('Z') 2016-03-23 10:48:09: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:09: pid 60203: DEBUG: do_query: extended:0 query:"SELECT pg_last_xlog_replay_location()" 2016-03-23 10:48:09: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:09: pid 60203: DEBUG: pool_write: to backend: kind:Q 2016-03-23 10:48:09: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:09: pid 60203: DEBUG: pool_read: read 95 bytes from backend 0 2016-03-23 10:48:09: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:09: pid 60203: DEBUG: do_query: kind: 'T' 2016-03-23 10:48:09: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:09: pid 60203: DEBUG: do_query: received ROW DESCRIPTION ('T') 2016-03-23 10:48:09: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:09: pid 60203: DEBUG: do_query: row description: num_fileds: 1 2016-03-23 10:48:09: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:09: pid 60203: DEBUG: do_query: kind: 'D' 2016-03-23 10:48:09: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:09: pid 60203: DEBUG: do_query: received DATA ROW ('D') 2016-03-23 10:48:09: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:09: pid 60203: DEBUG: do_query: kind: 'C' 2016-03-23 10:48:09: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:09: pid 60203: DEBUG: do_query: received COMMAND COMPLETE ('C') 2016-03-23 10:48:09: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:09: pid 60203: DEBUG: do_query: kind: 'Z' 2016-03-23 10:48:09: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:09: pid 60203: DEBUG: do_query: received READY FOR QUERY ('Z') 2016-03-23 10:48:09: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:09: pid 60203: DEBUG: pool_write: to backend: kind:X 2016-03-23 10:48:09: pid 60203: DEBUG: pool_write: to backend: kind:X 2016-03-23 10:48:09: pid 60302: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:09: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:10: pid 60303: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:48:10: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:10: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:11: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:48:11: pid 59036: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:48:11: pid 58460: DEBUG: starting health check 2016-03-23 10:48:11: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:48:11: pid 58460: DEBUG: doing health check against database:postgres user:postgres 2016-03-23 10:48:11: pid 58460: DEBUG: Backend DB node 0 status is 2 2016-03-23 10:48:11: pid 58460: DEBUG: Trying to make persistent DB connection to backend node 0 having status 2 2016-03-23 10:48:11: pid 58460: DEBUG: pool_read: read 327 bytes from backend 0 2016-03-23 10:48:11: pid 58460: DEBUG: authenticate kind = 0 2016-03-23 10:48:11: pid 58460: DEBUG: authenticate backend: key data received 2016-03-23 10:48:11: pid 58460: DEBUG: authenticate backend: transaction state: I 2016-03-23 10:48:11: pid 58460: DEBUG: persistent DB connection to backend node 0 having status 2 is successful 2016-03-23 10:48:11: pid 58460: DEBUG: pool_write: to backend: kind:X 2016-03-23 10:48:11: pid 58460: DEBUG: Backend DB node 1 status is 1 2016-03-23 10:48:11: pid 58460: DEBUG: Trying to make persistent DB connection to backend node 1 having status 1 2016-03-23 10:48:11: pid 58460: DEBUG: pool_read: read 327 bytes from backend 0 2016-03-23 10:48:11: pid 58460: DEBUG: authenticate kind = 0 2016-03-23 10:48:11: pid 58460: DEBUG: authenticate backend: key data received 2016-03-23 10:48:11: pid 58460: DEBUG: authenticate backend: transaction state: I 2016-03-23 10:48:11: pid 58460: DEBUG: persistent DB connection to backend node 1 having status 1 is successful 2016-03-23 10:48:11: pid 58460: DEBUG: pool_write: to backend: kind:X 2016-03-23 10:48:11: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:48:11: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:48:11: pid 60302: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:11: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:12: pid 58461: DEBUG: received packet tyep M while need packet type 2016-03-23 10:48:12: pid 58461: DEBUG: STATE MACHINE INVOKED WITH EVENT = PACKET RECEIVED Current State = STANDBY 2016-03-23 10:48:12: pid 58461: DEBUG: sending watchdog packet Socket:7, Type:[NODE INFO], Command_ID:22, data Length:170 2016-03-23 10:48:12: pid 60303: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:48:12: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:12: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:13: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:48:13: pid 59036: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:48:13: pid 60302: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:13: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:14: pid 60303: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:48:14: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:14: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:15: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:48:15: pid 59036: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:48:15: pid 60302: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:15: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:16: pid 60303: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:48:16: pid 58460: DEBUG: starting health check 2016-03-23 10:48:16: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:48:16: pid 58460: DEBUG: doing health check against database:postgres user:postgres 2016-03-23 10:48:16: pid 58460: DEBUG: Backend DB node 0 status is 2 2016-03-23 10:48:16: pid 58460: DEBUG: Trying to make persistent DB connection to backend node 0 having status 2 2016-03-23 10:48:16: pid 58460: DEBUG: pool_read: read 327 bytes from backend 0 2016-03-23 10:48:16: pid 58460: DEBUG: authenticate kind = 0 2016-03-23 10:48:16: pid 58460: DEBUG: authenticate backend: key data received 2016-03-23 10:48:16: pid 58460: DEBUG: authenticate backend: transaction state: I 2016-03-23 10:48:16: pid 58460: DEBUG: persistent DB connection to backend node 0 having status 2 is successful 2016-03-23 10:48:16: pid 58460: DEBUG: pool_write: to backend: kind:X 2016-03-23 10:48:16: pid 58460: DEBUG: Backend DB node 1 status is 1 2016-03-23 10:48:16: pid 58460: DEBUG: Trying to make persistent DB connection to backend node 1 having status 1 2016-03-23 10:48:16: pid 58460: DEBUG: pool_read: read 327 bytes from backend 0 2016-03-23 10:48:16: pid 58460: DEBUG: authenticate kind = 0 2016-03-23 10:48:16: pid 58460: DEBUG: authenticate backend: key data received 2016-03-23 10:48:16: pid 58460: DEBUG: authenticate backend: transaction state: I 2016-03-23 10:48:16: pid 58460: DEBUG: persistent DB connection to backend node 1 having status 1 is successful 2016-03-23 10:48:16: pid 58460: DEBUG: pool_write: to backend: kind:X 2016-03-23 10:48:16: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:48:16: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:48:16: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:16: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:17: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:48:17: pid 59036: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:48:17: pid 60302: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:17: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:17: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:18: pid 60303: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:48:18: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:18: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:19: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:48:19: pid 59036: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:48:19: pid 60203: DEBUG: pool_read: read 327 bytes from backend 0 2016-03-23 10:48:19: pid 60203: DEBUG: authenticate kind = 0 2016-03-23 10:48:19: pid 60203: DEBUG: authenticate backend: key data received 2016-03-23 10:48:19: pid 60203: DEBUG: authenticate backend: transaction state: I 2016-03-23 10:48:19: pid 60203: DEBUG: pool_read: read 327 bytes from backend 0 2016-03-23 10:48:19: pid 60203: DEBUG: authenticate kind = 0 2016-03-23 10:48:19: pid 60203: DEBUG: authenticate backend: key data received 2016-03-23 10:48:19: pid 60203: DEBUG: authenticate backend: transaction state: I 2016-03-23 10:48:19: pid 60203: DEBUG: do_query: extended:0 query:"SELECT pg_current_xlog_location()" 2016-03-23 10:48:19: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:19: pid 60203: DEBUG: pool_write: to backend: kind:Q 2016-03-23 10:48:19: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:19: pid 60203: DEBUG: pool_read: read 91 bytes from backend 0 2016-03-23 10:48:19: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:19: pid 60203: DEBUG: do_query: kind: 'T' 2016-03-23 10:48:19: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:19: pid 60203: DEBUG: do_query: received ROW DESCRIPTION ('T') 2016-03-23 10:48:19: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:19: pid 60203: DEBUG: do_query: row description: num_fileds: 1 2016-03-23 10:48:19: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:19: pid 60203: DEBUG: do_query: kind: 'D' 2016-03-23 10:48:19: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:19: pid 60203: DEBUG: do_query: received DATA ROW ('D') 2016-03-23 10:48:19: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:19: pid 60203: DEBUG: do_query: kind: 'C' 2016-03-23 10:48:19: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:19: pid 60203: DEBUG: do_query: received COMMAND COMPLETE ('C') 2016-03-23 10:48:19: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:19: pid 60203: DEBUG: do_query: kind: 'Z' 2016-03-23 10:48:19: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:19: pid 60203: DEBUG: do_query: received READY FOR QUERY ('Z') 2016-03-23 10:48:19: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:19: pid 60203: DEBUG: do_query: extended:0 query:"SELECT pg_last_xlog_replay_location()" 2016-03-23 10:48:19: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:19: pid 60203: DEBUG: pool_write: to backend: kind:Q 2016-03-23 10:48:19: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:19: pid 60203: DEBUG: pool_read: read 95 bytes from backend 0 2016-03-23 10:48:19: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:19: pid 60203: DEBUG: do_query: kind: 'T' 2016-03-23 10:48:19: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:19: pid 60203: DEBUG: do_query: received ROW DESCRIPTION ('T') 2016-03-23 10:48:19: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:19: pid 60203: DEBUG: do_query: row description: num_fileds: 1 2016-03-23 10:48:19: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:19: pid 60203: DEBUG: do_query: kind: 'D' 2016-03-23 10:48:19: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:19: pid 60203: DEBUG: do_query: received DATA ROW ('D') 2016-03-23 10:48:19: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:19: pid 60203: DEBUG: do_query: kind: 'C' 2016-03-23 10:48:19: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:19: pid 60203: DEBUG: do_query: received COMMAND COMPLETE ('C') 2016-03-23 10:48:19: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:19: pid 60203: DEBUG: do_query: kind: 'Z' 2016-03-23 10:48:19: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:19: pid 60203: DEBUG: do_query: received READY FOR QUERY ('Z') 2016-03-23 10:48:19: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:19: pid 60203: DEBUG: pool_write: to backend: kind:X 2016-03-23 10:48:19: pid 60203: DEBUG: pool_write: to backend: kind:X 2016-03-23 10:48:19: pid 60302: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:19: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:19: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:20: pid 60303: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:48:20: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:20: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:21: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:48:21: pid 59036: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:48:21: pid 58460: DEBUG: starting health check 2016-03-23 10:48:21: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:48:21: pid 58460: DEBUG: doing health check against database:postgres user:postgres 2016-03-23 10:48:21: pid 58460: DEBUG: Backend DB node 0 status is 2 2016-03-23 10:48:21: pid 58460: DEBUG: Trying to make persistent DB connection to backend node 0 having status 2 2016-03-23 10:48:21: pid 58460: DEBUG: pool_read: read 327 bytes from backend 0 2016-03-23 10:48:21: pid 58460: DEBUG: authenticate kind = 0 2016-03-23 10:48:21: pid 58460: DEBUG: authenticate backend: key data received 2016-03-23 10:48:21: pid 58460: DEBUG: authenticate backend: transaction state: I 2016-03-23 10:48:21: pid 58460: DEBUG: persistent DB connection to backend node 0 having status 2 is successful 2016-03-23 10:48:21: pid 58460: DEBUG: pool_write: to backend: kind:X 2016-03-23 10:48:21: pid 58460: DEBUG: Backend DB node 1 status is 1 2016-03-23 10:48:21: pid 58460: DEBUG: Trying to make persistent DB connection to backend node 1 having status 1 2016-03-23 10:48:21: pid 58460: DEBUG: pool_read: read 327 bytes from backend 0 2016-03-23 10:48:21: pid 58460: DEBUG: authenticate kind = 0 2016-03-23 10:48:21: pid 58460: DEBUG: authenticate backend: key data received 2016-03-23 10:48:21: pid 58460: DEBUG: authenticate backend: transaction state: I 2016-03-23 10:48:21: pid 58460: DEBUG: persistent DB connection to backend node 1 having status 1 is successful 2016-03-23 10:48:21: pid 58460: DEBUG: pool_write: to backend: kind:X 2016-03-23 10:48:21: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:48:21: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:48:21: pid 60302: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:21: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:21: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:22: pid 60303: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:48:22: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:22: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:22: pid 58461: DEBUG: received packet tyep M while need packet type 2016-03-23 10:48:22: pid 58461: DEBUG: STATE MACHINE INVOKED WITH EVENT = PACKET RECEIVED Current State = STANDBY 2016-03-23 10:48:22: pid 58461: DEBUG: sending watchdog packet Socket:7, Type:[NODE INFO], Command_ID:23, data Length:170 2016-03-23 10:48:23: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:48:23: pid 59036: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:48:23: pid 60302: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:23: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:23: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:24: pid 60303: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:48:24: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:24: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:25: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:48:25: pid 59036: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:48:25: pid 60302: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:25: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:25: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:26: pid 60303: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:48:26: pid 58460: DEBUG: starting health check 2016-03-23 10:48:26: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:48:26: pid 58460: DEBUG: doing health check against database:postgres user:postgres 2016-03-23 10:48:26: pid 58460: DEBUG: Backend DB node 0 status is 2 2016-03-23 10:48:26: pid 58460: DEBUG: Trying to make persistent DB connection to backend node 0 having status 2 2016-03-23 10:48:26: pid 58460: DEBUG: pool_read: read 327 bytes from backend 0 2016-03-23 10:48:26: pid 58460: DEBUG: authenticate kind = 0 2016-03-23 10:48:26: pid 58460: DEBUG: authenticate backend: key data received 2016-03-23 10:48:26: pid 58460: DEBUG: authenticate backend: transaction state: I 2016-03-23 10:48:26: pid 58460: DEBUG: persistent DB connection to backend node 0 having status 2 is successful 2016-03-23 10:48:26: pid 58460: DEBUG: pool_write: to backend: kind:X 2016-03-23 10:48:26: pid 58460: DEBUG: Backend DB node 1 status is 1 2016-03-23 10:48:26: pid 58460: DEBUG: Trying to make persistent DB connection to backend node 1 having status 1 2016-03-23 10:48:26: pid 58460: DEBUG: pool_read: read 327 bytes from backend 0 2016-03-23 10:48:26: pid 58460: DEBUG: authenticate kind = 0 2016-03-23 10:48:26: pid 58460: DEBUG: authenticate backend: key data received 2016-03-23 10:48:26: pid 58460: DEBUG: authenticate backend: transaction state: I 2016-03-23 10:48:26: pid 58460: DEBUG: persistent DB connection to backend node 1 having status 1 is successful 2016-03-23 10:48:26: pid 58460: DEBUG: pool_write: to backend: kind:X 2016-03-23 10:48:26: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:48:26: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:48:26: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:26: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:27: pid 59036: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:48:27: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:48:27: pid 60302: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:27: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:27: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:28: pid 60303: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:48:28: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:28: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:29: pid 59036: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:48:29: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:48:29: pid 60203: DEBUG: pool_read: read 327 bytes from backend 0 2016-03-23 10:48:29: pid 60203: DEBUG: authenticate kind = 0 2016-03-23 10:48:29: pid 60203: DEBUG: authenticate backend: key data received 2016-03-23 10:48:29: pid 60203: DEBUG: authenticate backend: transaction state: I 2016-03-23 10:48:29: pid 60203: DEBUG: pool_read: read 327 bytes from backend 0 2016-03-23 10:48:29: pid 60203: DEBUG: authenticate kind = 0 2016-03-23 10:48:29: pid 60203: DEBUG: authenticate backend: key data received 2016-03-23 10:48:29: pid 60203: DEBUG: authenticate backend: transaction state: I 2016-03-23 10:48:29: pid 60203: DEBUG: do_query: extended:0 query:"SELECT pg_current_xlog_location()" 2016-03-23 10:48:29: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:29: pid 60203: DEBUG: pool_write: to backend: kind:Q 2016-03-23 10:48:29: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:29: pid 60203: DEBUG: pool_read: read 91 bytes from backend 0 2016-03-23 10:48:29: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:29: pid 60203: DEBUG: do_query: kind: 'T' 2016-03-23 10:48:29: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:29: pid 60203: DEBUG: do_query: received ROW DESCRIPTION ('T') 2016-03-23 10:48:29: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:29: pid 60203: DEBUG: do_query: row description: num_fileds: 1 2016-03-23 10:48:29: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:29: pid 60203: DEBUG: do_query: kind: 'D' 2016-03-23 10:48:29: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:29: pid 60203: DEBUG: do_query: received DATA ROW ('D') 2016-03-23 10:48:29: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:29: pid 60203: DEBUG: do_query: kind: 'C' 2016-03-23 10:48:29: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:29: pid 60203: DEBUG: do_query: received COMMAND COMPLETE ('C') 2016-03-23 10:48:29: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:29: pid 60203: DEBUG: do_query: kind: 'Z' 2016-03-23 10:48:29: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:29: pid 60203: DEBUG: do_query: received READY FOR QUERY ('Z') 2016-03-23 10:48:29: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:29: pid 60203: DEBUG: do_query: extended:0 query:"SELECT pg_last_xlog_replay_location()" 2016-03-23 10:48:29: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:29: pid 60203: DEBUG: pool_write: to backend: kind:Q 2016-03-23 10:48:29: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:29: pid 60203: DEBUG: pool_read: read 95 bytes from backend 0 2016-03-23 10:48:29: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:29: pid 60203: DEBUG: do_query: kind: 'T' 2016-03-23 10:48:29: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:29: pid 60203: DEBUG: do_query: received ROW DESCRIPTION ('T') 2016-03-23 10:48:29: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:29: pid 60203: DEBUG: do_query: row description: num_fileds: 1 2016-03-23 10:48:29: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:29: pid 60203: DEBUG: do_query: kind: 'D' 2016-03-23 10:48:29: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:29: pid 60203: DEBUG: do_query: received DATA ROW ('D') 2016-03-23 10:48:29: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:29: pid 60203: DEBUG: do_query: kind: 'C' 2016-03-23 10:48:29: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:29: pid 60203: DEBUG: do_query: received COMMAND COMPLETE ('C') 2016-03-23 10:48:29: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:29: pid 60203: DEBUG: do_query: kind: 'Z' 2016-03-23 10:48:29: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:29: pid 60203: DEBUG: do_query: received READY FOR QUERY ('Z') 2016-03-23 10:48:29: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:29: pid 60203: DEBUG: pool_write: to backend: kind:X 2016-03-23 10:48:29: pid 60203: DEBUG: pool_write: to backend: kind:X 2016-03-23 10:48:29: pid 60302: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:29: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:29: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:30: pid 60303: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:48:30: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:30: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:31: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:48:31: pid 59036: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:48:31: pid 58460: DEBUG: starting health check 2016-03-23 10:48:31: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:48:31: pid 58460: DEBUG: doing health check against database:postgres user:postgres 2016-03-23 10:48:31: pid 58460: DEBUG: Backend DB node 0 status is 2 2016-03-23 10:48:31: pid 58460: DEBUG: Trying to make persistent DB connection to backend node 0 having status 2 2016-03-23 10:48:31: pid 58460: DEBUG: pool_read: read 327 bytes from backend 0 2016-03-23 10:48:31: pid 58460: DEBUG: authenticate kind = 0 2016-03-23 10:48:31: pid 58460: DEBUG: authenticate backend: key data received 2016-03-23 10:48:31: pid 58460: DEBUG: authenticate backend: transaction state: I 2016-03-23 10:48:31: pid 58460: DEBUG: persistent DB connection to backend node 0 having status 2 is successful 2016-03-23 10:48:31: pid 58460: DEBUG: pool_write: to backend: kind:X 2016-03-23 10:48:31: pid 58460: DEBUG: Backend DB node 1 status is 1 2016-03-23 10:48:31: pid 58460: DEBUG: Trying to make persistent DB connection to backend node 1 having status 1 2016-03-23 10:48:31: pid 58460: DEBUG: pool_read: read 327 bytes from backend 0 2016-03-23 10:48:31: pid 58460: DEBUG: authenticate kind = 0 2016-03-23 10:48:31: pid 58460: DEBUG: authenticate backend: key data received 2016-03-23 10:48:31: pid 58460: DEBUG: authenticate backend: transaction state: I 2016-03-23 10:48:31: pid 58460: DEBUG: persistent DB connection to backend node 1 having status 1 is successful 2016-03-23 10:48:31: pid 58460: DEBUG: pool_write: to backend: kind:X 2016-03-23 10:48:31: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:48:31: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:48:31: pid 60302: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:31: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:31: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:32: pid 60303: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:48:32: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:32: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:32: pid 58461: DEBUG: received packet tyep M while need packet type 2016-03-23 10:48:32: pid 58461: DEBUG: STATE MACHINE INVOKED WITH EVENT = PACKET RECEIVED Current State = STANDBY 2016-03-23 10:48:32: pid 58461: DEBUG: sending watchdog packet Socket:7, Type:[NODE INFO], Command_ID:24, data Length:170 2016-03-23 10:48:33: pid 59036: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:48:33: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:48:33: pid 60302: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:33: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:33: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:34: pid 60303: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:48:34: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:34: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:35: pid 59036: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:48:35: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:48:35: pid 60301: LOG: watchdog: lifecheck started 2016-03-23 10:48:35: pid 60302: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:35: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:35: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:36: pid 60303: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:48:36: pid 58460: DEBUG: starting health check 2016-03-23 10:48:36: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:48:36: pid 58460: DEBUG: doing health check against database:postgres user:postgres 2016-03-23 10:48:36: pid 58460: DEBUG: Backend DB node 0 status is 2 2016-03-23 10:48:36: pid 58460: DEBUG: Trying to make persistent DB connection to backend node 0 having status 2 2016-03-23 10:48:36: pid 58460: DEBUG: pool_read: read 327 bytes from backend 0 2016-03-23 10:48:36: pid 58460: DEBUG: authenticate kind = 0 2016-03-23 10:48:36: pid 58460: DEBUG: authenticate backend: key data received 2016-03-23 10:48:36: pid 58460: DEBUG: authenticate backend: transaction state: I 2016-03-23 10:48:36: pid 58460: DEBUG: persistent DB connection to backend node 0 having status 2 is successful 2016-03-23 10:48:36: pid 58460: DEBUG: pool_write: to backend: kind:X 2016-03-23 10:48:36: pid 58460: DEBUG: Backend DB node 1 status is 1 2016-03-23 10:48:36: pid 58460: DEBUG: Trying to make persistent DB connection to backend node 1 having status 1 2016-03-23 10:48:36: pid 58460: DEBUG: pool_read: read 327 bytes from backend 0 2016-03-23 10:48:36: pid 58460: DEBUG: authenticate kind = 0 2016-03-23 10:48:36: pid 58460: DEBUG: authenticate backend: key data received 2016-03-23 10:48:36: pid 58460: DEBUG: authenticate backend: transaction state: I 2016-03-23 10:48:36: pid 58460: DEBUG: persistent DB connection to backend node 1 having status 1 is successful 2016-03-23 10:48:36: pid 58460: DEBUG: pool_write: to backend: kind:X 2016-03-23 10:48:36: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:48:36: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:48:36: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:36: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:37: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:48:37: pid 59036: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:48:37: pid 60301: DEBUG: watchdog executing ping 2016-03-23 10:48:37: pid 60301: DETAIL: succeed to ping serv-ea01 2016-03-23 10:48:37: pid 60301: DEBUG: watchdog ping 2016-03-23 10:48:37: pid 60301: DETAIL: ping data: PING serv-ea01.adm.fr.clara.net (10.129.124.43) 56(84) bytes of data. --- serv-ea01.adm.fr.clara.net ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2001ms rtt min/avg/max/mdev = 0.214/0.273/0.337/0.053 ms 2016-03-23 10:48:37: pid 60301: DEBUG: watchdog executing ping 2016-03-23 10:48:37: pid 60301: DETAIL: succeed to ping serv-ew01 2016-03-23 10:48:37: pid 60301: DEBUG: watchdog ping 2016-03-23 10:48:37: pid 60301: DETAIL: ping data: PING serv-ew01.adm.fr.clara.net (10.129.124.41) 56(84) bytes of data. --- serv-ew01.adm.fr.clara.net ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2001ms rtt min/avg/max/mdev = 0.202/0.210/0.217/0.017 ms 2016-03-23 10:48:37: pid 60301: DEBUG: watchdog ping 2016-03-23 10:48:37: pid 60301: DETAIL: ping data: PING serv-ea02.adm.fr.clara.net (10.129.124.44) 56(84) bytes of data. --- serv-ea02.adm.fr.clara.net ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2001ms rtt min/avg/max/mdev = 0.116/0.128/0.139/0.013 ms 2016-03-23 10:48:37: pid 60301: DEBUG: watchdog ping 2016-03-23 10:48:37: pid 60301: DETAIL: ping data: PING serv-ea02.adm.fr.clara.net (10.129.124.44) 56(84) bytes of data. --- serv-ea02.adm.fr.clara.net ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2001ms rtt min/avg/max/mdev = 0.116/0.128/0.139/0.013 ms 2016-03-23 10:48:37: pid 58460: DEBUG: reaper handler 2016-03-23 10:48:37: pid 58460: WARNING: watchdog lifecheck process with pid: 60301 was terminated by segmentation fault 2016-03-23 10:48:37: pid 58460: LOG: fork a new watchdog lifecheck process with pid: 60339 2016-03-23 10:48:37: pid 58460: DEBUG: reaper handler: exiting normally 2016-03-23 10:48:37: pid 60339: DEBUG: I am watchdog lifecheck child with pid:60339 2016-03-23 10:48:37: pid 58461: LOG: new IPC connection received 2016-03-23 10:48:37: pid 60339: LOG: 2 watchdog nodes are configured for lifecheck 2016-03-23 10:48:37: pid 60339: LOG: watchdog nodes ID:0 Name:"Linux_serv-ed02_9999" 2016-03-23 10:48:37: pid 60339: DETAIL: Host:"serv-ed02" WD Port:9000 pgpool-II port:9999 2016-03-23 10:48:37: pid 60339: LOG: watchdog nodes ID:1 Name:"Linux_serv-ed01_9999" 2016-03-23 10:48:37: pid 60339: DETAIL: Host:"serv-ed01" WD Port:9000 pgpool-II port:9999 2016-03-23 10:48:37: pid 60339: DEBUG: watchdog checking life check is ready 2016-03-23 10:48:37: pid 60339: DETAIL: pgpool:1 at "serv-ed01:9999" has not send the heartbeat signal yet 2016-03-23 10:48:37: pid 60302: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:37: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:37: pid 58464: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:38: pid 60303: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:48:38: pid 60340: LOG: failed to create watchdog heartbeat receive socket. 2016-03-23 10:48:38: pid 60340: DETAIL: setsockopt(SO_BINDTODEVICE) requies root privilege 2016-03-23 10:48:38: pid 60340: LOG: set SO_REUSEPORT option to the socket 2016-03-23 10:48:38: pid 60340: LOG: creating watchdog heartbeat receive socket. 2016-03-23 10:48:38: pid 60340: DETAIL: set SO_REUSEPORT 2016-03-23 10:48:38: pid 60341: LOG: creating socket for sending heartbeat 2016-03-23 10:48:38: pid 60341: DETAIL: setsockopt(SO_BINDTODEVICE) requires root privilege 2016-03-23 10:48:38: pid 60341: LOG: set SO_REUSEPORT option to the socket 2016-03-23 10:48:38: pid 60341: LOG: creating socket for sending heartbeat 2016-03-23 10:48:38: pid 60341: DETAIL: set SO_REUSEPORT 2016-03-23 10:48:38: pid 60341: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:48:38: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:38: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:39: pid 59036: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:48:39: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:48:39: pid 60203: DEBUG: pool_read: read 327 bytes from backend 0 2016-03-23 10:48:39: pid 60203: DEBUG: authenticate kind = 0 2016-03-23 10:48:39: pid 60203: DEBUG: authenticate backend: key data received 2016-03-23 10:48:39: pid 60203: DEBUG: authenticate backend: transaction state: I 2016-03-23 10:48:39: pid 60203: DEBUG: pool_read: read 327 bytes from backend 0 2016-03-23 10:48:39: pid 60203: DEBUG: authenticate kind = 0 2016-03-23 10:48:39: pid 60203: DEBUG: authenticate backend: key data received 2016-03-23 10:48:39: pid 60203: DEBUG: authenticate backend: transaction state: I 2016-03-23 10:48:39: pid 60203: DEBUG: do_query: extended:0 query:"SELECT pg_current_xlog_location()" 2016-03-23 10:48:39: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:39: pid 60203: DEBUG: pool_write: to backend: kind:Q 2016-03-23 10:48:39: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:39: pid 60203: DEBUG: pool_read: read 91 bytes from backend 0 2016-03-23 10:48:39: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:39: pid 60203: DEBUG: do_query: kind: 'T' 2016-03-23 10:48:39: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:39: pid 60203: DEBUG: do_query: received ROW DESCRIPTION ('T') 2016-03-23 10:48:39: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:39: pid 60203: DEBUG: do_query: row description: num_fileds: 1 2016-03-23 10:48:39: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:39: pid 60203: DEBUG: do_query: kind: 'D' 2016-03-23 10:48:39: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:39: pid 60203: DEBUG: do_query: received DATA ROW ('D') 2016-03-23 10:48:39: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:39: pid 60203: DEBUG: do_query: kind: 'C' 2016-03-23 10:48:39: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:39: pid 60203: DEBUG: do_query: received COMMAND COMPLETE ('C') 2016-03-23 10:48:39: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:39: pid 60203: DEBUG: do_query: kind: 'Z' 2016-03-23 10:48:39: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:39: pid 60203: DEBUG: do_query: received READY FOR QUERY ('Z') 2016-03-23 10:48:39: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:39: pid 60203: DEBUG: do_query: extended:0 query:"SELECT pg_last_xlog_replay_location()" 2016-03-23 10:48:39: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:39: pid 60203: DEBUG: pool_write: to backend: kind:Q 2016-03-23 10:48:39: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:39: pid 60203: DEBUG: pool_read: read 95 bytes from backend 0 2016-03-23 10:48:39: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:39: pid 60203: DEBUG: do_query: kind: 'T' 2016-03-23 10:48:39: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:39: pid 60203: DEBUG: do_query: received ROW DESCRIPTION ('T') 2016-03-23 10:48:39: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:39: pid 60203: DEBUG: do_query: row description: num_fileds: 1 2016-03-23 10:48:39: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:39: pid 60203: DEBUG: do_query: kind: 'D' 2016-03-23 10:48:39: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:39: pid 60203: DEBUG: do_query: received DATA ROW ('D') 2016-03-23 10:48:39: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:39: pid 60203: DEBUG: do_query: kind: 'C' 2016-03-23 10:48:39: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:39: pid 60203: DEBUG: do_query: received COMMAND COMPLETE ('C') 2016-03-23 10:48:39: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:39: pid 60203: DEBUG: do_query: kind: 'Z' 2016-03-23 10:48:39: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:39: pid 60203: DEBUG: do_query: received READY FOR QUERY ('Z') 2016-03-23 10:48:39: pid 60203: CONTEXT: while checking replication time lag 2016-03-23 10:48:39: pid 60203: DEBUG: pool_write: to backend: kind:X 2016-03-23 10:48:39: pid 60203: DEBUG: pool_write: to backend: kind:X 2016-03-23 10:48:39: pid 60340: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:39: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:39: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:40: pid 60303: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:48:40: pid 60341: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:48:40: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:40: pid 59035: NOTICE: received heartbeat signal is older than the latest, ignored 2016-03-23 10:48:41: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:48:41: pid 59036: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:48:41: pid 58460: DEBUG: starting health check 2016-03-23 10:48:41: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:48:41: pid 58460: DEBUG: doing health check against database:postgres user:postgres 2016-03-23 10:48:41: pid 58460: DEBUG: Backend DB node 0 status is 2 2016-03-23 10:48:41: pid 58460: DEBUG: Trying to make persistent DB connection to backend node 0 having status 2 2016-03-23 10:48:41: pid 58460: DEBUG: pool_read: read 327 bytes from backend 0 2016-03-23 10:48:41: pid 58460: DEBUG: authenticate kind = 0 2016-03-23 10:48:41: pid 58460: DEBUG: authenticate backend: key data received 2016-03-23 10:48:41: pid 58460: DEBUG: authenticate backend: transaction state: I 2016-03-23 10:48:41: pid 58460: DEBUG: persistent DB connection to backend node 0 having status 2 is successful 2016-03-23 10:48:41: pid 58460: DEBUG: pool_write: to backend: kind:X 2016-03-23 10:48:41: pid 58460: DEBUG: Backend DB node 1 status is 1 2016-03-23 10:48:41: pid 58460: DEBUG: Trying to make persistent DB connection to backend node 1 having status 1 2016-03-23 10:48:41: pid 58460: DEBUG: pool_read: read 327 bytes from backend 0 2016-03-23 10:48:41: pid 58460: DEBUG: authenticate kind = 0 2016-03-23 10:48:41: pid 58460: DEBUG: authenticate backend: key data received 2016-03-23 10:48:41: pid 58460: DEBUG: authenticate backend: transaction state: I 2016-03-23 10:48:41: pid 58460: DEBUG: persistent DB connection to backend node 1 having status 1 is successful 2016-03-23 10:48:41: pid 58460: DEBUG: pool_write: to backend: kind:X 2016-03-23 10:48:41: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:48:41: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:48:41: pid 60340: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:41: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:41: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:42: pid 60303: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:48:42: pid 60341: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:48:42: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:42: pid 59035: NOTICE: received heartbeat signal is older than the latest, ignored 2016-03-23 10:48:42: pid 58461: DEBUG: received packet tyep M while need packet type 2016-03-23 10:48:42: pid 58461: DEBUG: STATE MACHINE INVOKED WITH EVENT = PACKET RECEIVED Current State = STANDBY 2016-03-23 10:48:42: pid 58461: DEBUG: sending watchdog packet Socket:7, Type:[NODE INFO], Command_ID:25, data Length:170 2016-03-23 10:48:43: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:48:43: pid 59036: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:48:43: pid 60340: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:43: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:43: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:44: pid 60303: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:48:44: pid 60341: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:48:44: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:44: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:45: pid 59036: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:48:45: pid 58466: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:48:45: pid 60340: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:45: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:45: pid 59035: DEBUG: received heartbeat signal from "serv-ed01:9999" 2016-03-23 10:48:46: pid 60303: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:48:46: pid 60341: DEBUG: watchdog heartbeat: send heartbeat signal to serv-ed01:9694 2016-03-23 10:48:46: pid 58460: DEBUG: starting health check 2016-03-23 10:48:46: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:48:46: pid 58460: DEBUG: doing health check against database:postgres user:postgres 2016-03-23 10:48:46: pid 58460: DEBUG: Backend DB node 0 status is 2 2016-03-23 10:48:46: pid 58460: DEBUG: Trying to make persistent DB connection to backend node 0 having status 2 2016-03-23 10:48:46: pid 58460: DEBUG: pool_read: read 327 bytes from backend 0 2016-03-23 10:48:46: pid 58460: DEBUG: authenticate kind = 0 2016-03-23 10:48:46: pid 58460: DEBUG: authenticate backend: key data received 2016-03-23 10:48:46: pid 58460: DEBUG: authenticate backend: transaction state: I 2016-03-23 10:48:46: pid 58460: DEBUG: persistent DB connection to backend node 0 having status 2 is successful 2016-03-23 10:48:46: pid 58460: DEBUG: pool_write: to backend: kind:X 2016-03-23 10:48:46: pid 58460: DEBUG: Backend DB node 1 status is 1 2016-03-23 10:48:46: pid 58460: DEBUG: Trying to make persistent DB connection to backend node 1 having status 1 2016-03-23 10:48:46: pid 58460: DEBUG: pool_read: read 327 bytes from backend 0 2016-03-23 10:48:46: pid 58460: DEBUG: authenticate kind = 0 2016-03-23 10:48:46: pid 58460: DEBUG: authenticate backend: key data received 2016-03-23 10:48:46: pid 58460: DEBUG: authenticate backend: transaction state: I 2016-03-23 10:48:46: pid 58460: DEBUG: persistent DB connection to backend node 1 having status 1 is successful 2016-03-23 10:48:46: pid 58460: DEBUG: pool_write: to backend: kind:X 2016-03-23 10:48:46: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:48:46: pid 58460: DEBUG: health check: clearing alarm 2016-03-23 10:48:46: pid 60303: DEBUG: watchdog heartbeat sender child receives shutdown request signal 2 2016-03-23 10:48:46: pid 60202: DEBUG: PCP child receives shutdown request signal 2, Forwarding to all children 2016-03-23 10:48:46: pid 58494: LOG: child process received shutdown request signal 2 2016-03-23 10:48:46: pid 60202: DEBUG: PCP child receives fast shutdown request 2016-03-23 10:48:46: pid 60302: DEBUG: watchdog heartbeat receiver child receives shutdown request signal 2 2016-03-23 10:48:46: pid 58492: LOG: child process received shutdown request signal 2 2016-03-23 10:48:46: pid 58465: LOG: child process received shutdown request signal 2 2016-03-23 10:48:46: pid 58493: LOG: child process received shutdown request signal 2 2016-03-23 10:48:46: pid 58490: LOG: child process received shutdown request signal 2 2016-03-23 10:48:46: pid 58479: LOG: child process received shutdown request signal 2 2016-03-23 10:48:46: pid 60339: DEBUG: lifecheck child receives shutdown request signal 2, Forwarding to all children 2016-03-23 10:48:46: pid 60339: DEBUG: lifecheck child receives fast shutdown request 2016-03-23 10:48:46: pid 58461: LOG: Watchdog is shutting down 2016-03-23 10:48:46: pid 58461: DEBUG: sending watchdog packet Socket:7, Type:[INFORM I AM GOING DOWN], Command_ID:13, data Length:0 2016-03-23 10:48:46: pid 58491: LOG: child process received shutdown request signal 2 2016-03-23 10:48:46: pid 58488: LOG: child process received shutdown request signal 2 2016-03-23 10:48:46: pid 58476: LOG: child process received shutdown request signal 2 2016-03-23 10:48:46: pid 58489: LOG: child process received shutdown request signal 2 2016-03-23 10:48:46: pid 58464: DEBUG: watchdog heartbeat receiver child receives shutdown request signal 2 2016-03-23 10:48:46: pid 58486: LOG: child process received shutdown request signal 2 2016-03-23 10:48:46: pid 58475: LOG: child process received shutdown request signal 2 2016-03-23 10:48:46: pid 58487: LOG: child process received shutdown request signal 2 2016-03-23 10:48:46: pid 58466: DEBUG: watchdog heartbeat sender child receives shutdown request signal 2 2016-03-23 10:48:46: pid 58484: LOG: child process received shutdown request signal 2 2016-03-23 10:48:46: pid 58472: LOG: child process received shutdown request signal 2 2016-03-23 10:48:46: pid 58485: LOG: child process received shutdown request signal 2 2016-03-23 10:48:46: pid 58469: LOG: child process received shutdown request signal 2 2016-03-23 10:48:46: pid 58482: LOG: child process received shutdown request signal 2 2016-03-23 10:48:46: pid 60340: DEBUG: watchdog heartbeat receiver child receives shutdown request signal 2 2016-03-23 10:48:46: pid 58483: LOG: child process received shutdown request signal 2 2016-03-23 10:48:46: pid 58463: LOG: child process received shutdown request signal 2 2016-03-23 10:48:46: pid 59035: DEBUG: watchdog heartbeat receiver child receives shutdown request signal 2 2016-03-23 10:48:46: pid 58480: LOG: child process received shutdown request signal 2 2016-03-23 10:48:46: pid 58481: LOG: child process received shutdown request signal 2 2016-03-23 10:48:46: pid 58468: LOG: child process received shutdown request signal 2 2016-03-23 10:48:46: pid 59036: DEBUG: watchdog heartbeat sender child receives shutdown request signal 2 2016-03-23 10:48:46: pid 58478: LOG: child process received shutdown request signal 2 2016-03-23 10:48:46: pid 58477: LOG: child process received shutdown request signal 2 2016-03-23 10:48:46: pid 58460: LOG: received fast shutdown request 2016-03-23 10:48:46: pid 58460: LOG: shutdown request. closing listen socket 2016-03-23 10:48:46: pid 58467: LOG: child process received shutdown request signal 2 2016-03-23 10:48:46: pid 58474: LOG: child process received shutdown request signal 2 2016-03-23 10:48:46: pid 60341: DEBUG: watchdog heartbeat sender child receives shutdown request signal 2 2016-03-23 10:48:46: pid 58473: LOG: child process received shutdown request signal 2 2016-03-23 10:48:46: pid 60339: DEBUG: lifecheck child receives shutdown request signal 2, Forwarding to all children 2016-03-23 10:48:46: pid 60339: DEBUG: lifecheck child receives fast shutdown request 2016-03-23 10:48:46: pid 58471: LOG: child process received shutdown request signal 2 2016-03-23 10:48:46: pid 58470: LOG: child process received shutdown request signal 2 -bash-4.1$