May 08 16:09:29 pgpool-2 pgpool[13811]: 2018-05-08 16:09:29: pid 13811: LOG: reload config files. May 08 16:09:29 pgpool-2 systemd[1]: Reloaded Pgpool-II. May 08 16:09:29 pgpool-2 pgpool[13811]: 2018-05-08 16:09:29: pid 13848: LOG: reloading config file May 08 16:12:49 pgpool-2 pgpool[13811]: 2018-05-08 16:12:49: pid 13845: LOG: failed to connect to PostgreSQL server on "pgpoolpsql-1:5432", getsockopt() detected error "Connection refused" May 08 16:12:49 pgpool-2 pgpool[13811]: 2018-05-08 16:12:49: pid 13845: LOG: received degenerate backend request for node_id: 0 from pid [13845] May 08 16:12:49 pgpool-2 pgpool[13811]: 2018-05-08 16:12:49: pid 13812: LOG: new IPC connection received May 08 16:12:49 pgpool-2 pgpool[13811]: 2018-05-08 16:12:49: pid 13812: LOG: watchdog received the failover command from local pgpool-II on IPC interface May 08 16:12:49 pgpool-2 pgpool[13811]: 2018-05-08 16:12:49: pid 13812: LOG: watchdog is processing the failover command [DEGENERATE_BACKEND_REQUEST] received from local pgpool-II on IPC interface May 08 16:12:49 pgpool-2 pgpool[13811]: 2018-05-08 16:12:49: pid 13812: LOG: failover requires the majority vote, waiting for consensus May 08 16:12:49 pgpool-2 pgpool[13811]: 2018-05-08 16:12:49: pid 13812: DETAIL: failover request noted May 08 16:12:49 pgpool-2 pgpool[13811]: 2018-05-08 16:12:49: pid 13812: LOG: failover command [DEGENERATE_BACKEND_REQUEST] request from pgpool-II node "pgpool-2:9999 Linux pgpool-2" is queued, waiting for the confirmation from other nodes May 08 16:12:49 pgpool-2 pgpool[13811]: 2018-05-08 16:12:49: pid 13845: LOG: degenerate backend request for node_id: 0 from pid [13845], will be handled by watchdog, which is building consensus for request May 08 16:12:49 pgpool-2 pgpool[13811]: 2018-05-08 16:12:49: pid 13845: FATAL: failed to create a backend connection May 08 16:12:49 pgpool-2 pgpool[13811]: 2018-05-08 16:12:49: pid 13845: DETAIL: executing failover on backend May 08 16:12:49 pgpool-2 pgpool[13811]: 2018-05-08 16:12:49: pid 13811: LOG: child process with pid: 13845 exits with status 256 May 08 16:12:49 pgpool-2 pgpool[13811]: 2018-05-08 16:12:49: pid 13811: LOG: fork a new child process with pid: 20054 May 08 16:12:50 pgpool-2 pgpool[13811]: 2018-05-08 16:12:50: pid 13844: LOG: failed to connect to PostgreSQL server on "pgpoolpsql-1:5432", getsockopt() detected error "Connection refused" May 08 16:12:50 pgpool-2 pgpool[13811]: 2018-05-08 16:12:50: pid 13844: LOG: received degenerate backend request for node_id: 0 from pid [13844] May 08 16:12:50 pgpool-2 pgpool[13811]: 2018-05-08 16:12:50: pid 13812: LOG: new IPC connection received May 08 16:12:50 pgpool-2 pgpool[13811]: 2018-05-08 16:12:50: pid 13812: LOG: watchdog received the failover command from local pgpool-II on IPC interface May 08 16:12:50 pgpool-2 pgpool[13811]: 2018-05-08 16:12:50: pid 13812: LOG: watchdog is processing the failover command [DEGENERATE_BACKEND_REQUEST] received from local pgpool-II on IPC interface May 08 16:12:50 pgpool-2 pgpool[13811]: 2018-05-08 16:12:50: pid 13812: LOG: Duplicate failover request from "pgpool-2:9999 Linux pgpool-2" node May 08 16:12:50 pgpool-2 pgpool[13811]: 2018-05-08 16:12:50: pid 13812: DETAIL: request ignored May 08 16:12:50 pgpool-2 pgpool[13811]: 2018-05-08 16:12:50: pid 13812: LOG: failover requires the majority vote, waiting for consensus May 08 16:12:50 pgpool-2 pgpool[13811]: 2018-05-08 16:12:50: pid 13812: DETAIL: failover request noted May 08 16:12:50 pgpool-2 pgpool[13811]: 2018-05-08 16:12:50: pid 13844: LOG: degenerate backend request for 1 node(s) from pid [13844], is changed to quarantine node request by watchdog May 08 16:12:50 pgpool-2 pgpool[13811]: 2018-05-08 16:12:50: pid 13844: DETAIL: watchdog is taking time to build consensus May 08 16:12:50 pgpool-2 pgpool[13811]: 2018-05-08 16:12:50: pid 13844: FATAL: failed to create a backend connection May 08 16:12:50 pgpool-2 pgpool[13811]: 2018-05-08 16:12:50: pid 13844: DETAIL: executing failover on backend May 08 16:12:50 pgpool-2 pgpool[13811]: 2018-05-08 16:12:50: pid 13811: LOG: Pgpool-II parent process has received failover request May 08 16:12:50 pgpool-2 pgpool[13811]: 2018-05-08 16:12:50: pid 13812: LOG: new IPC connection received May 08 16:12:50 pgpool-2 pgpool[13811]: 2018-05-08 16:12:50: pid 13812: LOG: received the failover indication from Pgpool-II on IPC interface May 08 16:12:50 pgpool-2 pgpool[13811]: 2018-05-08 16:12:50: pid 13812: LOG: watchdog is informed of failover end by the main process May 08 16:12:50 pgpool-2 pgpool[13811]: 2018-05-08 16:12:50: pid 13811: LOG: starting quarantine. shutdown host pgpoolpsql-1(5432) May 08 16:12:50 pgpool-2 pgpool[13811]: 2018-05-08 16:12:50: pid 13811: LOG: Restart all children May 08 16:12:50 pgpool-2 pgpool[13811]: 2018-05-08 16:12:50: pid 13811: LOG: failover: set new primary node: -1 May 08 16:12:50 pgpool-2 pgpool[13811]: 2018-05-08 16:12:50: pid 13811: LOG: failover: set new master node: 1 May 08 16:12:50 pgpool-2 pgpool[13811]: 2018-05-08 16:12:50: pid 13848: LOG: worker process received restart request May 08 16:12:50 pgpool-2 pgpool[13811]: 2018-05-08 16:12:50: pid 13812: LOG: new IPC connection received May 08 16:12:50 pgpool-2 pgpool[13811]: 2018-05-08 16:12:50: pid 13812: LOG: received the failover indication from Pgpool-II on IPC interface May 08 16:12:50 pgpool-2 pgpool[13811]: 2018-05-08 16:12:50: pid 13812: LOG: watchdog is informed of failover start by the main process May 08 16:12:50 pgpool-2 pgpool[13811]: quarantine done. shutdown host pgpoolpsql-1(5432)2018-05-08 16:12:50: pid 13811: LOG: quarantine done. shutdown host pgpoolpsql-1(5432) May 08 16:12:51 pgpool-2 pgpool[13811]: 2018-05-08 16:12:51: pid 13847: LOG: restart request received in pcp child process May 08 16:12:51 pgpool-2 pgpool[13811]: 2018-05-08 16:12:51: pid 13811: LOG: PCP child 13847 exits with status 0 in failover() May 08 16:12:51 pgpool-2 pgpool[13811]: 2018-05-08 16:12:51: pid 13811: LOG: fork a new PCP child pid 20087 in failover() May 08 16:12:51 pgpool-2 pgpool[13811]: 2018-05-08 16:12:51: pid 13811: LOG: child process with pid: 13815 exits with status 0 May 08 16:12:51 pgpool-2 pgpool[13811]: 2018-05-08 16:12:51: pid 13811: LOG: child process with pid: 13815 exited with success and will not be restarted May 08 16:12:51 pgpool-2 pgpool[13811]: 2018-05-08 16:12:51: pid 13811: LOG: child process with pid: 13816 exits with status 0 May 08 16:12:51 pgpool-2 pgpool[13811]: 2018-05-08 16:12:51: pid 13811: LOG: child process with pid: 13816 exited with success and will not be restarted May 08 16:12:51 pgpool-2 pgpool[13811]: 2018-05-08 16:12:51: pid 13811: LOG: child process with pid: 13817 exits with status 0 May 08 16:12:51 pgpool-2 pgpool[13811]: 2018-05-08 16:12:51: pid 13811: LOG: child process with pid: 13817 exited with success and will not be restarted May 08 16:12:51 pgpool-2 pgpool[13811]: 2018-05-08 16:12:51: pid 13811: LOG: child process with pid: 13818 exits with status 0 May 08 16:12:51 pgpool-2 pgpool[13811]: 2018-05-08 16:12:51: pid 13811: LOG: child process with pid: 13818 exited with success and will not be restarted May 08 16:12:51 pgpool-2 pgpool[13811]: 2018-05-08 16:12:51: pid 13811: LOG: child process with pid: 13819 exits with status 0 May 08 16:12:51 pgpool-2 pgpool[13811]: 2018-05-08 16:12:51: pid 13811: LOG: child process with pid: 13819 exited with success and will not be restarted May 08 16:12:51 pgpool-2 pgpool[13811]: 2018-05-08 16:12:51: pid 13811: LOG: child process with pid: 13820 exits with status 0 May 08 16:12:51 pgpool-2 pgpool[13811]: 2018-05-08 16:12:51: pid 13811: LOG: child process with pid: 13820 exited with success and will not be restarted May 08 16:12:51 pgpool-2 pgpool[13811]: 2018-05-08 16:12:51: pid 13811: LOG: child process with pid: 13821 exits with status 0 May 08 16:12:51 pgpool-2 pgpool[13811]: 2018-05-08 16:12:51: pid 13811: LOG: child process with pid: 13821 exited with success and will not be restarted May 08 16:12:51 pgpool-2 pgpool[13811]: 2018-05-08 16:12:51: pid 13811: LOG: child process with pid: 13822 exits with status 0 May 08 16:12:51 pgpool-2 pgpool[13811]: 2018-05-08 16:12:51: pid 13811: LOG: child process with pid: 13822 exited with success and will not be restarted May 08 16:12:51 pgpool-2 pgpool[13811]: 2018-05-08 16:12:51: pid 13811: LOG: child process with pid: 13823 exits with status 0 May 08 16:12:51 pgpool-2 pgpool[13811]: 2018-05-08 16:12:51: pid 13811: LOG: child process with pid: 13823 exited with success and will not be restarted May 08 16:12:51 pgpool-2 pgpool[13811]: 2018-05-08 16:12:51: pid 13811: LOG: child process with pid: 13824 exits with status 0 May 08 16:12:51 pgpool-2 pgpool[13811]: 2018-05-08 16:12:51: pid 13811: LOG: child process with pid: 13824 exited with success and will not be restarted May 08 16:12:51 pgpool-2 pgpool[13811]: 2018-05-08 16:12:51: pid 13811: LOG: child process with pid: 13825 exits with status 0 May 08 16:12:51 pgpool-2 pgpool[13811]: 2018-05-08 16:12:51: pid 13811: LOG: child process with pid: 13825 exited with success and will not be restarted May 08 16:12:51 pgpool-2 pgpool[13811]: 2018-05-08 16:12:51: pid 13811: LOG: child process with pid: 13826 exits with status 0 May 08 16:12:51 pgpool-2 pgpool[13811]: 2018-05-08 16:12:51: pid 13811: LOG: child process with pid: 13826 exited with success and will not be restarted May 08 16:12:51 pgpool-2 pgpool[13811]: 2018-05-08 16:12:51: pid 13811: LOG: child process with pid: 13828 exits with status 0 May 08 16:12:51 pgpool-2 pgpool[13811]: 2018-05-08 16:12:51: pid 13811: LOG: child process with pid: 13828 exited with success and will not be restarted May 08 16:12:51 pgpool-2 pgpool[13811]: 2018-05-08 16:12:51: pid 13811: LOG: child process with pid: 13829 exits with status 0 May 08 16:12:51 pgpool-2 pgpool[13811]: 2018-05-08 16:12:51: pid 13811: LOG: child process with pid: 13829 exited with success and will not be restarted May 08 16:12:51 pgpool-2 pgpool[13811]: 2018-05-08 16:12:51: pid 13811: LOG: child process with pid: 13830 exits with status 0 May 08 16:12:51 pgpool-2 pgpool[13811]: 2018-05-08 16:12:51: pid 13811: LOG: child process with pid: 13830 exited with success and will not be restarted May 08 16:12:51 pgpool-2 pgpool[13811]: 2018-05-08 16:12:51: pid 13811: LOG: child process with pid: 13831 exits with status 0 May 08 16:12:51 pgpool-2 pgpool[13811]: 2018-05-08 16:12:51: pid 13811: LOG: child process with pid: 13831 exited with success and will not be restarted May 08 16:12:51 pgpool-2 pgpool[13811]: 2018-05-08 16:12:51: pid 13811: LOG: child process with pid: 13832 exits with status 0 May 08 16:12:51 pgpool-2 pgpool[13811]: 2018-05-08 16:12:51: pid 13811: LOG: child process with pid: 13832 exited with success and will not be restarted May 08 16:12:51 pgpool-2 pgpool[13811]: 2018-05-08 16:12:51: pid 13811: LOG: child process with pid: 13833 exits with status 0 May 08 16:12:51 pgpool-2 pgpool[13811]: 2018-05-08 16:12:51: pid 13811: LOG: child process with pid: 13833 exited with success and will not be restarted May 08 16:12:51 pgpool-2 pgpool[13811]: 2018-05-08 16:12:51: pid 13811: LOG: child process with pid: 13834 exits with status 0 May 08 16:12:51 pgpool-2 pgpool[13811]: 2018-05-08 16:12:51: pid 13811: LOG: child process with pid: 13834 exited with success and will not be restarted May 08 16:12:51 pgpool-2 pgpool[13811]: 2018-05-08 16:12:51: pid 13811: LOG: child process with pid: 13835 exits with status 0 May 08 16:12:51 pgpool-2 pgpool[13811]: 2018-05-08 16:12:51: pid 13811: LOG: child process with pid: 13835 exited with success and will not be restarted May 08 16:12:51 pgpool-2 pgpool[13811]: 2018-05-08 16:12:51: pid 13811: LOG: child process with pid: 13836 exits with status 0 May 08 16:12:51 pgpool-2 pgpool[13811]: 2018-05-08 16:12:51: pid 13811: LOG: child process with pid: 13836 exited with success and will not be restarted May 08 16:12:51 pgpool-2 pgpool[13811]: 2018-05-08 16:12:51: pid 13811: LOG: child process with pid: 13837 exits with status 0 May 08 16:12:51 pgpool-2 pgpool[13811]: 2018-05-08 16:12:51: pid 13811: LOG: child process with pid: 13837 exited with success and will not be restarted May 08 16:12:51 pgpool-2 pgpool[13811]: 2018-05-08 16:12:51: pid 13811: LOG: child process with pid: 13838 exits with status 0 May 08 16:12:51 pgpool-2 pgpool[13811]: 2018-05-08 16:12:51: pid 13811: LOG: child process with pid: 13838 exited with success and will not be restarted May 08 16:12:51 pgpool-2 pgpool[13811]: 2018-05-08 16:12:51: pid 13811: LOG: child process with pid: 13839 exits with status 0 May 08 16:12:51 pgpool-2 pgpool[13811]: 2018-05-08 16:12:51: pid 13811: LOG: child process with pid: 13839 exited with success and will not be restarted May 08 16:12:51 pgpool-2 pgpool[13811]: 2018-05-08 16:12:51: pid 13811: LOG: child process with pid: 13840 exits with status 0 May 08 16:12:51 pgpool-2 pgpool[13811]: 2018-05-08 16:12:51: pid 13811: LOG: child process with pid: 13840 exited with success and will not be restarted May 08 16:12:51 pgpool-2 pgpool[13811]: 2018-05-08 16:12:51: pid 13811: LOG: child process with pid: 13841 exits with status 0 May 08 16:12:51 pgpool-2 pgpool[13811]: 2018-05-08 16:12:51: pid 13811: LOG: child process with pid: 13841 exited with success and will not be restarted May 08 16:12:51 pgpool-2 pgpool[13811]: 2018-05-08 16:12:51: pid 13811: LOG: child process with pid: 13842 exits with status 0 May 08 16:12:51 pgpool-2 pgpool[13811]: 2018-05-08 16:12:51: pid 13811: LOG: child process with pid: 13842 exited with success and will not be restarted May 08 16:12:51 pgpool-2 pgpool[13811]: 2018-05-08 16:12:51: pid 13811: LOG: child process with pid: 13843 exits with status 0 May 08 16:12:51 pgpool-2 pgpool[13811]: 2018-05-08 16:12:51: pid 13811: LOG: child process with pid: 13843 exited with success and will not be restarted May 08 16:12:51 pgpool-2 pgpool[13811]: 2018-05-08 16:12:51: pid 13811: LOG: child process with pid: 13844 exits with status 256 May 08 16:12:51 pgpool-2 pgpool[13811]: 2018-05-08 16:12:51: pid 13811: LOG: child process with pid: 13844 exited with success and will not be restarted May 08 16:12:51 pgpool-2 pgpool[13811]: 2018-05-08 16:12:51: pid 13811: LOG: child process with pid: 13846 exits with status 0 May 08 16:12:51 pgpool-2 pgpool[13811]: 2018-05-08 16:12:51: pid 13811: LOG: child process with pid: 13846 exited with success and will not be restarted May 08 16:12:51 pgpool-2 pgpool[13811]: 2018-05-08 16:12:51: pid 13811: LOG: worker child process with pid: 13848 exits with status 256 May 08 16:12:51 pgpool-2 pgpool[13811]: 2018-05-08 16:12:51: pid 13811: LOG: fork a new worker child process with pid: 20088 May 08 16:12:51 pgpool-2 pgpool[13811]: 2018-05-08 16:12:51: pid 13811: LOG: child process with pid: 13878 exits with status 0 May 08 16:12:51 pgpool-2 pgpool[13811]: 2018-05-08 16:12:51: pid 13811: LOG: child process with pid: 13878 exited with success and will not be restarted May 08 16:12:51 pgpool-2 pgpool[13811]: 2018-05-08 16:12:51: pid 13811: LOG: child process with pid: 20054 exits with status 0 May 08 16:12:51 pgpool-2 pgpool[13811]: 2018-05-08 16:12:51: pid 13811: LOG: child process with pid: 20054 exited with success and will not be restarted