<span style="font-family: Arial, Helvetica, Sans-Serif; font-size: 12px"><div>All,</div>
<div> </div>
<div>Very very sorry!! Im ashamed. My mail-client rendered my email horribly. Wil resend my message with other mailclient.</div>
<div> </div>
<hr align="center" size="2" width="100%" />
<div><span style="font-family: tahoma,arial,sans-serif; font-size: 10pt;"><b>From</b>: "Michael Akkermans" <info@michael-akkermans.nl><br />
<b>Sent</b>: Saturday, January 23, 2016 6:32 PM<br />
<b>To</b>: pgpool-general@pgpool.net<br />
<b>Subject</b>: [pgpool-general: 4356] Watchdog on Debian Jessie platform with PGpool version(3.3.4, 3.4, 3.5) wont run when defining virtual IP?</span>
<div> </div>
<span style="font-family: Arial, Helvetica, Sans-Serif; font-size: 12px">Hi, English is not my native language; please excuse typing errors. === Summary === Goal: Use Watchdog with a virtual-IP between 2 PGpool instances. Reason i ask my question in General, is cause im sure im missing something very obvious. Problem description: Watchdog and PGpool wont run after configuring (any) virtual IP. I checked the following sources: - Mailinglist General archives PGpool - Web search "setup and Watchdog topics" (Startpage.com) - Stack Overflow, Server Fault searches - Manual: http://www.pgpool.net/docs/latest/pgpool-en.html - FAQ: http://www.pgpool.net/mediawiki/index.php/FAQ - Read this before sending this email: http://www.catb.org/esr/faqs/smart-questions.html - Checked sourcecode from Debian-os-repo(3.3.4) and PGpool-git-repo(3.5) NOTE: Im not a code guru, but i can read it and somewhat understand what it should do. I created a labsetup: - Labsetup (2x PGpool VMs, 2x PG-9.4 VM's) === Lab Setup === pgpool01 -- VM(proxmox), OS(Debian Jessie 8.2 - 3.16.0-4-686-pae), Hostname(pgpool01.lab.local), IP(10.100.4.81) pgpool02 -- VM(proxmox), OS(Debian Jessie 8.2 - 3.16.0-4-686-pae), Hostname(pgpool02.lab.local), IP(10.100.4.82) pgsql01 -- VM(proxmox), OS(Debian Jessie 8.2 - 3.16.0-4-686-pae), Hostname(pgsql01.lab.local), IP(10.100.4.91) pgsql02 -- VM(proxmox), OS(Debian Jessie 8.2 - 3.16.0-4-686-pae), Hostname(pgsql01.lab.local), IP(10.100.4.92) ********************************************* *** pgpool01(/etc/pgpool.conf) highlights *** ********************************************* listen_addresses = '*' port = 5432 pcp_port = 9898 -- backend_hostname0 = '10.100.4.91' backend_port0 = 5432 backend_weight0 = 1 backend_data_directory0 = '/var/lib/postgresql/9.4/main' backend_flag0 = 'DISALLOW_TO_FAILOVER' -- backend_hostname1 = '10.100.4.92' backend_port1 = 5432 backend_weight1 = 1 backend_data_directory1 = '/var/lib/postgresql/9.4/main' backend_flag1 = 'DISALLOW_TO_FAILOVER' -- log_destination = 'stderr' debug_level = 1 -- connection_cache = on replication_mode = off load_balance_mode = on master_slave_mode = off parallel_mode = off -- use_watchdog = on trusted_servers = '' wd_hostname = '10.100.4.81' wd_port = 9000 delegate_IP = '10.100.4.85' if_up_cmd = 'ifconfig eth0:0 inet $_IP_$ netmask 255.255.255.0' if_down_cmd = 'ifconfig eth0:0 down' arping_cmd = 'arping -U $_IP_$ -w 1' clear_memqcache_on_escalation = on -- wd_lifecheck_method = 'heartbeat' wd_heartbeat_port = 9694 heartbeat_destination0 = '10.100.4.82' heartbeat_destination_port0 = 9694 heartbeat_device0 = 'eth0' ********************************************* *** pgpool02(/etc/pgpool.conf) highlights *** ********************************************* Has a similar setup as pgpool01 with subtle differences listed below: wd_hostname = '10.100.4.82' heartbeat_destination0 = '10.100.4.81' ************************ *** pgsql01, pgsql02 *** ************************ Both clusters are online and respond to queries from PGpool. """ psql -h 10.100.4.81 -U postgres -d postgres -c "SELECT * FROM t1" col1 | col2 ------+------------- 1 | a (1 row) """ NOTE: I did not configured hot_standby, just 2 standalone clusters with the same t1 table + tuples. First i want Watchdog operational so i can test some PGpool failover scenario's. === Behaviour in Lab Setup === With the settings as described in this email this is the output of pgpool, """ 2016-01-23 12:38:58 LOG: pid 15187: wd_chk_setuid: ifup[/sbin/ifconfig] doesn't have setuid bit 2016-01-23 12:39:01 LOG: pid 15187: wd_create_send_socket: connect() reports failure (No route to host). You can safely ignore this while starting up. 2016-01-23 12:39:01 LOG: pid 15187: send_packet_4_nodes: packet for 10.100.4.82:9000 is canceled 2016-01-23 12:39:04 DEBUG: pid 15187: exec_ping: failed to ping 10.100.4.85: exit code 1 2016-01-23 12:39:04 LOG: pid 15187: wd_escalation: escalating to master pgpool 2016-01-23 12:39:04 DEBUG: pid 15187: exec_ifconfig: 'ifconfig eth0:0 inet $_IP_$ netmask 255.255.255.0' succeeded """ This is the output of netstat, """ netstat -tulpen | grep '5432\|9000\|9694' tcp 0 0 0.0.0.0:5432 LISTEN 15187/pgpool """ For some (to me unknown) reason Watchdog and Heartbeat are not running. The subinterface with virtual-IP 10.100.4.85 is up. Also the pid files are there in /var/run/postgresql/. Output (snippets) with strace (with settings as described in this email) """ access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory) write(2, "2016-01-23 17:50:38 LOG: pid 1"..., 992016-01-23 17:50:38 LOG: pid 14996: send_packet_4_nodes: packet for 10.100.4.82:9000 is canceled write(2, "2016-01-23 17:50:41 LOG: pid 1"..., 812016-01-23 17:50:41 LOG: pid 14996: wd_escalation: escalating to master pgpool """ === Findings === - The message "send_packet_4_nodes is cancelled" means the other host is not responding. But even when i check the other host and make sure port 9000 is up and listening for incomming packets, the message wont change. I should not prevent me from brining the master up? - The output states "wd_escalation: escalating to master pgpool", but why does Watchdog stay down? - Tried with PGpool version 3.5, version 3.4 stable and version 3.3.4, same problem, what am i missing here? - Searched on keywords from the logfiles with no luck. Many happy-flow examples, not so much Watchdog troubleshoot examples. - pcp_* commands wont work in this setup. === Questions === - Would someone provide a pointer? - What is my example missing? - What site should I have checked? - How can i get Watchdog to work? With kind regards, Michael Akkermans</span></div></span>