<div dir="ltr"><div>OK thanks a lot again!</div><div>Next are both config files (pgpool.conf)</div><div><br></div><div>======================================================pgpool-01============================================</div>
<div>listen_addresses = '*'</div><div>port = 5432</div><div>socket_dir = '/tmp'</div><div><br></div><div>pcp_port = 9898</div><div>pcp_socket_dir = '/tmp'</div><div><br></div><div>backend_hostname0 = '10.0.0.250'</div>
<div>backend_port0 = 5432</div><div>backend_weight0 = 1</div><div>backend_data_directory0 = '/var/lib/postgresql/9.1/main/'</div><div>backend_flag0 = 'ALLOW_TO_FAILOVER'</div><div>backend_hostname1 = '10.0.1.54'</div>
<div>backend_port1 = 5432</div><div>backend_weight1 = 1000</div><div>backend_data_directory1 = '/var/lib/postgresql/9.1/main/'</div><div>backend_flag1 = 'ALLOW_TO_FAILOVER'</div><div><br></div><div>enable_pool_hba = on</div>
<div>pool_passwd = 'pool_passwd'</div><div>authentication_timeout = 60</div><div><br></div><div>ssl = off</div><div><br></div><div>num_init_children = 32</div><div>max_pool = 4</div><div><br></div><div>child_life_time = 300</div>
<div>child_max_connections = 0</div><div>connection_life_time = 0</div><div>client_idle_limit = 0</div><div><br></div><div>log_destination = 'stderr'</div><div><br></div><div>print_timestamp = on</div><div><br></div>
<div>log_connections = off</div><div>log_hostname = off</div><div>log_statement = off</div><div>log_per_node_statement = off</div><div>log_standby_delay = 'none'</div><div><br></div><div>syslog_facility = 'LOCAL0'</div>
<div>syslog_ident = 'pgpool'</div><div><br></div><div>debug_level = 0</div><div><br></div><div>pid_file_name = '/var/run/pgpool/pgpool.pid'</div><div>logdir = '/tmp'</div><div><br></div><div>connection_cache = on</div>
<div><br></div><div>reset_query_list = 'ABORT; DISCARD ALL'</div><div><br></div><div>replication_mode = off</div><div>replicate_select = off</div><div><br></div><div>replication_mode = off</div><div>replicate_select = off</div>
<div><br></div><div>insert_lock = on</div><div>lobj_lock_table = ''</div><div><br></div><div>replication_stop_on_mismatch = off</div><div><br></div><div>failover_if_affected_tuples_mismatch = off</div><div><br></div>
<div>load_balance_mode = off</div><div>ignore_leading_white_space = on</div><div>white_function_list = ''</div><div>black_function_list = 'nextval,setval'</div><div><br></div><div>master_slave_mode = on</div>
<div>master_slave_sub_mode = 'stream'</div><div><br></div><div>sr_check_period = 0</div><div>sr_check_user = 'postgres'</div><div>sr_check_password = 'protected'</div><div>delay_threshold = 0</div>
<div><br></div><div>follow_master_command = ''</div><div><br></div><div>parallel_mode = off</div><div>pgpool2_hostname = ''</div><div><br></div><div>system_db_hostname = 'localhost'</div><div>system_db_port = 5432 </div>
<div>system_db_dbname = 'pgpool'</div><div>system_db_schema = 'pgpool_catalog'</div><div>system_db_user = 'pgpool'</div><div>system_db_password = ''</div><div><br></div><div>health_check_period = 1</div>
<div>health_check_timeout = 20</div><div>health_check_user = 'postgres'</div><div>health_check_password = 'protected'</div><div>health_check_max_retries = 0</div><div>health_check_retry_delay = 1</div><div>
<br></div><div>failover_command = '/usr/local/bin/failover_stream.sh %d %H /tmp/trigger_file'</div><div>failback_command = ''</div><div><br></div><div>fail_over_on_backend_error = on</div><div><br></div><div>
recovery_user = 'postgres'</div><div>recovery_password = ''</div><div>recovery_1st_stage_command = ''</div><div>recovery_2nd_stage_command = ''</div><div>recovery_timeout = 90</div><div>client_idle_limit_in_recovery = 0</div>
<div><br></div><div>use_watchdog = on </div><div>trusted_servers = ''</div><div>delegate_IP = '10.11.12.13'</div><div>wd_hostname = '10.0.0.21'</div><div>wd_port = 9000</div><div>wd_interval = 10</div>
<div>ping_path = '/bin'</div><div>ifconfig_path = '/sbin'</div><div>if_up_cmd = 'ifconfig eth0:0 inet $_IP_$ netmask 255.255.255.0'</div><div>if_down_cmd = 'ifconfig eth0:0 down'</div><div>
<br></div><div>arping_cmd = 'arping -U $_IP_$ -w 1'</div><div>wd_life_point = 3</div><div>wd_lifecheck_query = 'SELECT 1'</div><div><br></div><div>other_pgpool_hostname0 = '10.0.1.21'</div><div>other_pgpool_port0 = 9999</div>
<div>other_wd_port0 = 9000</div><div><br></div><div>relcache_expire = 0</div><div><br></div><div>relcache_size = 256</div><div><br></div><div>check_temp_table = on</div><div><br></div><div>memory_cache_enabled = off</div>
<div>memqcache_method = 'shmem'</div><div>memqcache_memcached_host = 'localhost'</div><div>memqcache_memcached_port = 11211</div><div>memqcache_total_size = 67108864</div><div>memqcache_max_num_cache = 1000000</div>
<div>memqcache_expire = 0</div><div>memqcache_auto_cache_invalidation = on</div><div>memqcache_maxcache = 409600</div><div>memqcache_cache_block_size = 1048576</div><div>memqcache_oiddir = '/var/log/pgpool/oiddir'</div>
<div>white_memqcache_table_list = ''</div><div>black_memqcache_table_list = ''</div><div>=========================================================FIM===============================================</div><div>
<br></div><div>======================================================pgpool-02============================================</div><div>listen_addresses = '*'</div><div>port = 5432</div><div>socket_dir = '/tmp'</div>
<div><br></div><div>pcp_port = 9898</div><div>pcp_socket_dir = '/tmp'</div><div><br></div><div>backend_hostname0 = '10.0.0.250'</div><div>backend_port0 = 5432</div><div>backend_weight0 = 1</div><div>backend_data_directory0 = '/var/lib/postgresql/9.1/main/'</div>
<div>backend_flag0 = 'ALLOW_TO_FAILOVER'</div><div>backend_hostname1 = '10.0.1.54'</div><div>backend_port1 = 5432</div><div>backend_weight1 = 1000</div><div>backend_data_directory1 = '/var/lib/postgresql/9.1/main/'</div>
<div>backend_flag1 = 'ALLOW_TO_FAILOVER'</div><div><br></div><div>enable_pool_hba = on</div><div>pool_passwd = 'pool_passwd'</div><div>authentication_timeout = 60</div><div><br></div><div>ssl = off</div><div>
<br></div><div>num_init_children = 32</div><div>max_pool = 4</div><div><br></div><div>child_life_time = 300</div><div>child_max_connections = 0</div><div>connection_life_time = 0</div><div>client_idle_limit = 0</div><div>
<br></div><div>log_destination = 'stderr'</div><div><br></div><div>print_timestamp = on</div><div><br></div><div>log_connections = off</div><div>log_hostname = off</div><div>log_statement = off</div><div>log_per_node_statement = off</div>
<div>log_standby_delay = 'none'</div><div><br></div><div>syslog_facility = 'LOCAL0'</div><div>syslog_ident = 'pgpool'</div><div><br></div><div>debug_level = 0</div><div><br></div><div>pid_file_name = '/var/run/pgpool/pgpool.pid'</div>
<div>logdir = '/tmp'</div><div><br></div><div>connection_cache = on</div><div><br></div><div>reset_query_list = 'ABORT; DISCARD ALL'</div><div><br></div><div>replication_mode = off</div><div>replicate_select = off</div>
<div><br></div><div>insert_lock = on</div><div>lobj_lock_table = ''</div><div><br></div><div>replication_stop_on_mismatch = off</div><div><br></div><div>failover_if_affected_tuples_mismatch = off</div><div><br></div>
<div>load_balance_mode = off</div><div>ignore_leading_white_space = on</div><div>white_function_list = ''</div><div>black_function_list = 'nextval,setval'</div><div><br></div><div>master_slave_mode = on</div>
<div>master_slave_sub_mode = 'stream'</div><div><br></div><div>sr_check_period = 0</div><div>sr_check_user = 'postgres'</div><div>sr_check_password = 'protected'</div><div>delay_threshold = 0</div>
<div><br></div><div>follow_master_command = ''</div><div><br></div><div>parallel_mode = off</div><div>pgpool2_hostname = ''</div><div><br></div><div>system_db_hostname = 'localhost'</div><div>system_db_port = 5432 </div>
<div>system_db_dbname = 'pgpool'</div><div>system_db_schema = 'pgpool_catalog'</div><div>system_db_user = 'pgpool'</div><div>system_db_password = ''</div><div><br></div><div>health_check_period = 1</div>
<div>health_check_timeout = 20</div><div>health_check_user = 'postgres'</div><div>health_check_password = 'protected'</div><div>health_check_max_retries = 0</div><div>health_check_retry_delay = 1</div><div>
<br></div><div>failover_command = '/usr/local/bin/failover_stream.sh %d %H /tmp/trigger_file'</div><div>failback_command = ''</div><div><br></div><div>fail_over_on_backend_error = on</div><div><br></div><div>
recovery_user = 'postgres'</div><div>recovery_password = ''</div><div>recovery_1st_stage_command = ''</div><div>recovery_2nd_stage_command = ''</div><div>recovery_timeout = 90</div><div>client_idle_limit_in_recovery = 0</div>
<div><br></div><div>use_watchdog = on </div><div>trusted_servers = ''</div><div>delegate_IP = '10.11.12.13'</div><div>wd_hostname = '10.0.1.21'</div><div>wd_port = 9000</div><div>wd_interval = 10</div>
<div>ping_path = '/bin'</div><div>ifconfig_path = '/sbin'</div><div>if_up_cmd = 'ifconfig eth0:0 inet $_IP_$ netmask 255.255.255.0'</div><div>if_down_cmd = 'ifconfig eth0:0 down'</div><div>
<br></div><div>arping_cmd = 'arping -U $_IP_$ -w 1'</div><div>wd_life_point = 3</div><div>wd_lifecheck_query = 'SELECT 1'</div><div><br></div><div>other_pgpool_hostname0 = '10.0.0.21'</div><div>other_pgpool_port0 = 9999</div>
<div>other_wd_port0 = 9000</div><div><br></div><div>relcache_expire = 0</div><div><br></div><div>relcache_size = 256</div><div><br></div><div>check_temp_table = on</div><div><br></div><div>memory_cache_enabled = off</div>
<div>memqcache_method = 'shmem'</div><div>memqcache_memcached_host = 'localhost'</div><div>memqcache_memcached_port = 11211</div><div>memqcache_total_size = 67108864</div><div>memqcache_max_num_cache = 1000000</div>
<div>memqcache_expire = 0</div><div>memqcache_auto_cache_invalidation = on</div><div>memqcache_maxcache = 409600</div><div>memqcache_cache_block_size = 1048576</div><div>memqcache_oiddir = '/var/log/pgpool/oiddir'</div>
<div>white_memqcache_table_list = ''</div><div>black_memqcache_table_list = ''</div><div>=========================================================FIM===============================================</div><div>
<br></div><div><br></div><div>01 - I start the pgpool-01:</div><div>/usr/local/bin/pgpool -n</div><div><br></div><div>log:</div><div>2013-08-05 16:11:55 LOG: pid 17526: wd_chk_sticky: ifup[/sbin/ifconfig] doesn't have sticky bit</div>
<div>2013-08-05 16:11:55 LOG: pid 17526: wd_create_send_socket: connect() reports failure (Connection refused). You can safely ignore this while starting up.</div><div>2013-08-05 16:12:07 LOG: pid 17526: wd_escalation: escalated to master pgpool</div>
<div>2013-08-05 16:12:07 LOG: pid 17526: wd_create_send_socket: connect() reports failure (Connection refused). You can safely ignore this while starting up.</div><div>2013-08-05 16:12:07 LOG: pid 17526: wd_escalation: escalated to delegate_IP holder</div>
<div>2013-08-05 16:12:07 LOG: pid 17526: wd_init: start watchdog</div><div>2013-08-05 16:12:07 LOG: pid 17526: pgpool-II successfully started. version 3.2.5 (namameboshi)</div><div>2013-08-05 16:12:07 LOG: pid 17526: find_primary_node: primary node id is 1</div>
<div><br></div><div>ifconfig:</div><div>eth0:0 Link encap:Ethernet HWaddr 16:45:08:ce:61:8b </div><div> inet addr:10.11.12.13 Bcast:10.11.12.255 Mask:255.255.255.0</div><div> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1</div>
<div> Interrupt:25</div><div><br></div><div>I can use the delegate ip from the application and works fine:</div><div>fbuzon@fernando-Inspiron-7520:~$ psql -h 10.11.12.13 -U postgres -W</div><div>Password for user postgres: </div>
<div>psql (9.1.9)</div><div>Type "help" for help.</div><div><br></div><div>postgres=#</div><div><br></div><div><br></div><div>02 - I start the pgpool-02:</div><div>/usr/local/bin/pgpool -n</div><div><br></div><div>
log:</div><div>2013-08-05 16:12:28 LOG: pid 11382: wd_chk_sticky: ifup[/sbin/ifconfig] doesn't have sticky bit</div><div>2013-08-05 16:12:28 LOG: pid 11382: wd_init: start watchdog</div><div>2013-08-05 16:12:28 LOG: pid 11382: pgpool-II successfully started. version 3.2.5 (namameboshi)</div>
<div>2013-08-05 16:12:28 LOG: pid 11382: find_primary_node: primary node id is 1</div><div>2013-08-05 16:12:29 LOG: pid 11385: watchdog: lifecheck started</div><div><br></div><div>ifconfig don't show the delegate_IP. OK.</div>
<div><br></div><div><br></div><div>03 - I stop the pgpool-01:</div><div>pgpool stop</div><div>stop request sent to pgpool. waiting for termination................done.</div><div><br></div><div>log pgpool-01:</div><div>2013-08-05 16:13:09 LOG: pid 17526: received smart shutdown request</div>
<div>2013-08-05 16:13:09 LOG: pid 17526: watchdog_pid: 17533</div><div>2013-08-05 16:13:21 LOG: pid 17533: wd_IP_down: ifconfig down succeeded</div><div><br></div><div>ifconfig no more shows the delegate_ip. OK.</div>
<div><br></div><div>log pgpool-02:</div><div>2013-08-05 16:13:30 LOG: pid 11384: wd_escalation: escalated to master pgpool</div><div>2013-08-05 16:13:30 LOG: pid 11384: wd_escalation: escalated to delegate_IP holder</div>
<div>2013-08-05 16:13:30 LOG: pid 11384: wd_escalation: escalated to master pgpool</div><div>2013-08-05 16:13:30 LOG: pid 11384: wd_escalation: escalated to delegate_IP holder</div><div>2013-08-05 16:13:45 LOG: pid 11385: wd_lifecheck: lifecheck failed 3 times. pgpool 1 (<a href="http://10.0.0.21:5432">10.0.0.21:5432</a>) seems not to be working</div>
<div>2013-08-05 16:13:55 LOG: pid 11385: wd_lifecheck: lifecheck failed 3 times. pgpool 1 (<a href="http://10.0.0.21:5432">10.0.0.21:5432</a>) seems not to be working</div><div><br></div><div>ifconfig:</div><div>eth0:0 Link encap:Ethernet HWaddr 16:45:08:e5:3f:34 </div>
<div> inet addr:10.11.12.13 Bcast:10.11.12.255 Mask:255.255.255.0</div><div> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1</div><div> Interrupt:25</div><div><br></div><div>I still using the delegate ip from the application and works fine:</div>
<div>fbuzon@fernando-Inspiron-7520:~$ psql -h 10.11.12.13 -U postgres -W</div><div>Password for user postgres: </div><div>psql (9.1.9)</div><div>Type "help" for help.</div><div><br></div><div>postgres=#</div><div>
<br></div><div>04 - I start again the pgpool-01:</div><div>/usr/local/bin/pgpool -n</div><div><br></div><div>log:</div><div>2013-08-05 16:17:24 LOG: pid 17576: wd_chk_sticky: ifup[/sbin/ifconfig] doesn't have sticky bit</div>
<div>2013-08-05 16:17:24 LOG: pid 17576: wd_init: start watchdog</div><div>2013-08-05 16:17:24 LOG: pid 17576: pgpool-II successfully started. version 3.2.5 (namameboshi)</div><div>2013-08-05 16:17:24 LOG: pid 17576: find_primary_node: primary node id is 1</div>
<div>2013-08-05 16:17:25 LOG: pid 17579: watchdog: lifecheck started</div><div><br></div><div>ifconfig no more shows the delegate_ip. OK.</div><div><br></div><div><br></div><div>05 - I stop the pgpool-02:</div><div>pgpool stop</div>
<div>stop request sent to pgpool. waiting for termination................done.</div><div><br></div><div>log pgpool-02:</div><div>2013-08-05 16:18:53 LOG: pid 11382: received smart shutdown request</div><div>2013-08-05 16:18:53 LOG: pid 11382: watchdog_pid: 11385</div>
<div>2013-08-05 16:19:05 LOG: pid 11385: wd_IP_down: ifconfig down succeeded</div><div><br></div><div>log pgpool-01:</div><div>2013-08-05 16:18:56 LOG: pid 17578: wd_escalation: escalated to master pgpool</div><div>2013-08-05 16:18:56 LOG: pid 17578: wd_escalation: escalated to delegate_IP holder</div>
<div><br></div><div><br></div><div>06 - Start again the pgpool-02</div><div>log:</div><div><br></div><div>2013-08-05 16:21:30 LOG: pid 11521: wd_chk_sticky: ifup[/sbin/ifconfig] doesn't have sticky bit</div><div>2013-08-05 16:21:30 LOG: pid 11521: wd_init: start watchdog</div>
<div>2013-08-05 16:21:30 LOG: pid 11521: pgpool-II successfully started. version 3.2.5 (namameboshi)</div><div>2013-08-05 16:21:30 LOG: pid 11521: find_primary_node: primary node id is 1</div><div>2013-08-05 16:21:31 LOG: pid 11524: watchdog: lifecheck started</div>
<div><br></div><div><br></div><div>FINAL:</div><div>Like I said, all is working nice.</div><div>And now I am with the 2 pgpools up and working again.</div><div>The escaled pgpool is pgpool-01.</div><div>I stop it with "killall -9 pgpool" and now wd_lifecheck worked fine on pgpool-02!</div>
<div>I dont now what was the problem early, but now is working!</div><div><br></div><div>log on pgpool-02:</div><div>2013-08-05 17:52:42 LOG: pid 11524: wd_lifecheck: lifecheck failed 3 times. pgpool 1 (<a href="http://10.0.0.21:5432">10.0.0.21:5432</a>) seems not to be working</div>
<div>2013-08-05 17:52:42 LOG: pid 11524: wd_escalation: escalated to master pgpool</div><div>2013-08-05 17:52:42 LOG: pid 11524: wd_escalation: escalated to delegate_IP holder</div><div>2013-08-05 17:52:52 LOG: pid 11524: wd_lifecheck: lifecheck failed 3 times. pgpool 1 (<a href="http://10.0.0.21:5432">10.0.0.21:5432</a>) seems not to be working</div>
<div><br></div><div>So rest only one problem, that is how to down delegate_ip from the pgpool-01?</div><div>Because both servers is responding to delegate_ip.</div><div><br></div><div><br></div><div>Thanks!</div></div><div class="gmail_extra">
<br><br><div class="gmail_quote">2013/8/5 Jeff Frost <span dir="ltr"><<a href="mailto:jeff@pgexperts.com" target="_blank">jeff@pgexperts.com</a>></span><br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"><div class="im">
<div>On 08/05/13 07:51, Fernando Buzon
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr"><br>
<div class="gmail_extra">2013/8/2 Jeff Frost <span dir="ltr"><<a href="mailto:jeff@pgexperts.com" target="_blank">jeff@pgexperts.com</a>></span><br>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div style="word-wrap:break-word">
<div>So are you indicating that your test is:</div>
<div><br>
</div>
<div>killall -9 pgpool </div>
<div><br>
</div>
<div>on the primary pgpool server?</div>
</div>
</blockquote>
<div>YES</div>
<div>
<br>
</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div style="word-wrap:break-word">
<div>
<br>
</div>
<div>I think I may have lost track of the sequence of
events. Maybe you could lay it out as a simple test
case for us?</div>
</div>
</blockquote>
<div>What should I do? </div>
<div><br>
</div>
<div> </div>
</div>
</div>
</div>
</blockquote>
<br></div>
You should post a step-by-step method for reproducing your issue
with appropriate log output after each step is completed.<br>
<br>
That will make it more obvious what's going on.<br>
<br>
<br>
Ex:<br>
<br>
pgpool.conf files from both nodes<br>
successful psql connecting to the delegate IP<br>
log lines indicating pgpool is up and running fine (on both hosts)<br>
ip addr or ifconfig output indicating which host has the delegate ip<br>
<br>
kill -9<br>
log output (on both hosts)<br>
ip addr or ifconfig output<br>
......<br>
<br>
so it's obvious how to reproduce your issue (or tell you how to
avoid it).<br>
</div>
</blockquote></div><br></div>