<div dir="ltr">Hi Tatsuo<div><br></div><div>My conclusion is same as yours for this issue. And I think your patch will fix the problem. I think as part of this patch, adding the error string to the error log if select () call fails will be helpful. Please find the updated patch which just adds the strerror() to the log message reporting the select call failure. Rest of the patch is unchanged.</div><div><br></div><div>Thanks</div><div>Best regards</div><div>Muhammad Usama</div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, May 28, 2015 at 1:29 PM, Tatsuo Ishii <span dir="ltr"><<a href="mailto:ishii@postgresql.org" target="_blank">ishii@postgresql.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Sorry I forgot to attach the patch.<br>
<div class="HOEnZb"><div class="h5"><br>
Best regards,<br>
--<br>
Tatsuo Ishii<br>
SRA OSS, Inc. Japan<br>
English: <a href="http://www.sraoss.co.jp/index_en.php" target="_blank">http://www.sraoss.co.jp/index_en.php</a><br>
Japanese:<a href="http://www.sraoss.co.jp" target="_blank">http://www.sraoss.co.jp</a><br>
<br>
> I suspect while the health check routine tries to connect to backend,<br>
> select(2) was interrupted by certain signal. The signal could be<br>
> SIGCHLD, since you have pretty large number (2000) of child process,<br>
> plus you enable child_life_time (300). This means that after pgpool<br>
> becomes idle, child process starts to exit within 0.15 second (=<br>
> 300/2000) in average. If health check is executed (which is every 10<br>
> seconds) at the same time when child exits, select(2) will be<br>
> interrupted.<br>
><br>
> Fix is, check if the signal is other than SIGALRM (which is used for<br>
> health check by the health check routine itself). If not, just retries<br>
> the select(2).<br>
><br>
> Attached patch implements it.<br>
><br>
> Best regards,<br>
> --<br>
> Tatsuo Ishii<br>
> SRA OSS, Inc. Japan<br>
> English: <a href="http://www.sraoss.co.jp/index_en.php" target="_blank">http://www.sraoss.co.jp/index_en.php</a><br>
> Japanese:<a href="http://www.sraoss.co.jp" target="_blank">http://www.sraoss.co.jp</a><br>
><br>
>> Hi Tatsuo<br>
>><br>
>> Sorry for the late reply. Attached pgpool.cnf, but also in text below for mailing list.<br>
>><br>
>><br>
>> Postgres: 9.3.5<br>
>><br>
>> Pgpool version: pgpool-II-pg93-3.4.0-3pgdg.rhel6.x86_64 (from rpm -qa)<br>
>><br>
>><br>
>> Pgpool.conf (with out comments)<br>
>><br>
>> listen_addresses = '*'<br>
>> port = 9999<br>
>> socket_dir = '/tmp'<br>
>> listen_backlog_multiplier = 2<br>
>> pcp_listen_addresses = '*'<br>
>> pcp_port = 9898<br>
>> pcp_socket_dir = '/tmp'<br>
>> backend_hostname0 = 'X.X.X.50'<br>
>> backend_port0 = 5432<br>
>> backend_weight0 = 4<br>
>> backend_data_directory0 = '/var/lib/pgsql/9.3/data'<br>
>> backend_flag0 = 'ALLOW_TO_FAILOVER'<br>
>> backend_hostname1 = 'X.X.X.51'<br>
>> backend_port1 = 5432<br>
>> backend_weight1 = 3<br>
>> backend_data_directory1 = '/var/lib/pgsql/9.3/data'<br>
>> backend_flag1 = 'ALLOW_TO_FAILOVER'<br>
>> backend_hostname2 = 'X.X.X.52'<br>
>> backend_port2 = 5432<br>
>> backend_weight2 = 3<br>
>> backend_data_directory2 = '/var/lib/pgsql/9.3/data'<br>
>> backend_flag2 = 'ALLOW_TO_FAILOVER'<br>
>> enable_pool_hba = on<br>
>> pool_passwd = 'pool_passwd'<br>
>> authentication_timeout = 60<br>
>> ssl = off<br>
>> num_init_children = 2000<br>
>> max_pool = 4<br>
>> child_life_time = 300<br>
>> child_max_connections = 0<br>
>> connection_life_time = 0<br>
>> client_idle_limit = 0<br>
>> log_destination = 'stderr'<br>
>> log_line_prefix = '%p %t '<br>
>> log_standby_delay = 'none'<br>
>> syslog_facility = 'LOCAL0'<br>
>> syslog_ident = 'pgpool'<br>
>> debug_level = 0<br>
>> pid_file_name = '/tmp/pgpool/pgpool.pid'<br>
>> logdir = '/var/log/pgpool'<br>
>> connection_cache = on<br>
>> reset_query_list = 'ABORT; DISCARD ALL'<br>
>> replication_mode = off<br>
>> replicate_select = off<br>
>> insert_lock = on<br>
>> lobj_lock_table = ''<br>
>><br>
>><br>
>> replication_stop_on_mismatch = off<br>
>><br>
>> failover_if_affected_tuples_mismatch = off<br>
>><br>
>><br>
>><br>
>> load_balance_mode = on<br>
>> ignore_leading_white_space = on<br>
>> white_function_list = ''<br>
>> black_function_list = 'breadcrumbtrails,f_round,t_round,fn_CalculateUserStatement,nextval,setval,nextval,setval,fn_settierlili,fn_GetImageForCategory,tblSessions,articles,fn_ManagerPoints,process,fn.*,fn_.*,fn_RepReport'<br>
>><br>
>> database_redirect_preference_list = 'otrstest:primary'<br>
>><br>
>> app_name_redirect_preference_list = ''<br>
>> allow_sql_comments = off<br>
>><br>
>><br>
>> master_slave_mode = on<br>
>> master_slave_sub_mode = 'stream'<br>
>><br>
>><br>
>> sr_check_period = 10<br>
>> sr_check_user = 'postgres'<br>
>> sr_check_password = ''<br>
>> delay_threshold = 1024<br>
>><br>
>><br>
>> follow_master_command = '/etc/pgpool-II/follow_master.sh %d %M %h'<br>
>><br>
>><br>
>> health_check_period = 10<br>
>> health_check_timeout = 20<br>
>> health_check_user = 'postgres'<br>
>> health_check_password = ''<br>
>> health_check_max_retries = 2<br>
>> health_check_retry_delay = 1<br>
>> connect_timeout = 10000<br>
>><br>
>><br>
>> failover_command = '/etc/pgpool-II/failover.sh %d %P %H %R'<br>
>> failback_command = ''<br>
>><br>
>> fail_over_on_backend_error = on<br>
>><br>
>> search_primary_node_timeout = 10<br>
>><br>
>><br>
>> recovery_user = 'postgres'<br>
>> recovery_password = ''<br>
>> recovery_1st_stage_command = 'recovery_1st_stage'<br>
>> recovery_2nd_stage_command = ''<br>
>> recovery_timeout = 90<br>
>> client_idle_limit_in_recovery = 0<br>
>><br>
>><br>
>><br>
>><br>
>> use_watchdog = on<br>
>><br>
>><br>
>> trusted_servers = ''<br>
>> ping_path = '/home/apache/sbin'<br>
>><br>
>><br>
>> wd_hostname = 'UI-IS-POOL1'<br>
>> wd_port = 9000<br>
>> wd_authkey = ''<br>
>><br>
>><br>
>> delegate_IP = 'X.X.X.60'<br>
>> ifconfig_path = '/home/apache/sbin'<br>
>> if_up_cmd = 'ifconfig ens32:0 inet $_IP_$ netmask 255.255.255.0'<br>
>> if_down_cmd = 'ifconfig ens32:0 down'<br>
>><br>
>><br>
>> arping_cmd = 'arping -U $_IP_$ -w 1'<br>
>><br>
>><br>
>> clear_memqcache_on_escalation = on<br>
>> wd_escalation_command = ''<br>
>><br>
>><br>
>><br>
>> wd_lifecheck_method = 'heartbeat'<br>
>> wd_interval = 3<br>
>><br>
>><br>
>> wd_heartbeat_port = 9694<br>
>> wd_heartbeat_keepalive = 2<br>
>> wd_heartbeat_deadtime = 30<br>
>><br>
>> heartbeat_destination0 = 'X.X.X.61'<br>
>> heartbeat_destination_port0 = 9694<br>
>> heartbeat_device0 = ''<br>
>> heartbeat_destination1 = 'X.X.X.62'<br>
>> heartbeat_destination_port1 = 9694<br>
>> heartbeat_device1 = ''<br>
>><br>
>><br>
>> wd_life_point = 3<br>
>> wd_lifecheck_query = 'SELECT 1'<br>
>> wd_lifecheck_dbname = 'template1'<br>
>> wd_lifecheck_user = 'postgres'<br>
>> wd_lifecheck_password = ''<br>
>><br>
>><br>
>> other_pgpool_hostname0 = 'X.X.X.62'<br>
>> other_pgpool_port0 = 9999<br>
>> other_wd_port0 = 9000<br>
>><br>
>><br>
>> relcache_expire = 0<br>
>><br>
>> relcache_size = 256<br>
>><br>
>> check_temp_table = on<br>
>><br>
>> check_unlogged_table = off<br>
>><br>
>> memory_cache_enabled = off<br>
>> memqcache_method = 'shmem'<br>
>> memqcache_memcached_host = 'localhost'<br>
>> memqcache_memcached_port = 11211<br>
>> memqcache_total_size = 67108864<br>
>> memqcache_max_num_cache = 1000000<br>
>> memqcache_expire = 0<br>
>> memqcache_auto_cache_invalidation = on<br>
>> memqcache_maxcache = 409600<br>
>> memqcache_cache_block_size = 1048576<br>
>> memqcache_oiddir = '/var/log/pgpool/oiddir'<br>
>> white_memqcache_table_list = ''<br>
>> black_memqcache_table_list = ''<br>
>><br>
>><br>
>><br>
>> Robert Kennedy| Software & Systems Developer|<br>
>><br>
>> Tel: (011) 557-5700 | Ext: 011 557 5729 | Cell: 084 998 9850<br>
>><br>
>> Email : <a href="mailto:robert@uwiniwin.co.za">robert@uwiniwin.co.za</a> | Site: <a href="http://www.uwiniwin.co.za" target="_blank">www.uwiniwin.co.za</a><br>
>> Address : The Oval Office Park , Ground Floor Kingsmead, Meadowbrook Lane, Bryanston<br>
>> -----Original Message-----<br>
>> From: Tatsuo Ishii [mailto:<a href="mailto:ishii@postgresql.org">ishii@postgresql.org</a>]<br>
>> Sent: Wednesday, May 27, 2015 11:11 AM<br>
>> To: Robert Kennedy<br>
>> Cc: <a href="mailto:pgpool-general@pgpool.net">pgpool-general@pgpool.net</a><br>
>> Subject: Re: [pgpool-general: 3756] Connection Interrupted<br>
>><br>
>> Can you please provide details:<br>
>><br>
>> - pgpool-II version (and RPM version if any)<br>
>> - PostgreSQL version<br>
>> - pgpool.conf<br>
>><br>
>> Best regards,<br>
>> --<br>
>> Tatsuo Ishii<br>
>> SRA OSS, Inc. Japan<br>
>> English: <a href="http://www.sraoss.co.jp/index_en.php" target="_blank">http://www.sraoss.co.jp/index_en.php</a><br>
>> Japanese:<a href="http://www.sraoss.co.jp" target="_blank">http://www.sraoss.co.jp</a><br>
>><br>
>>> Hi Guys<br>
>>><br>
>>> I keep getting this error randomly appearing in my logs.<br>
>>><br>
>>> 23734 2015-05-27 10:45:52 LOG: failed to connect to PostgreSQL server on "x.x.x.x:5432" using INET socket<br>
>>> 23734 2015-05-27 10:45:52 DETAIL: select() system call interrupted<br>
>>> 23734 2015-05-27 10:45:52 ERROR: failed to make persistent db connection<br>
>>> 23734 2015-05-27 10:45:52 DETAIL: connection to host:"x.x.x.x:5432" failed<br>
>>><br>
>>> However I can connect to the server on that perfectly, and it seems pgpool can connect as well as no failover event gets called.<br>
>>><br>
>>> My concern is I'm loosing data, as this is my primary(master) server. I do see the same error for my slave as well tho.<br>
>>><br>
>>> Please advise what may be causing this, and let me know if you require any more information.<br>
>>><br>
>>> Robert Kennedy| Software & Systems Developer<br>
>>><br>
>>> Tel: 011 557 5729 | Ext:229 | Cell: 084 998 9850<br>
>>><br>
>>> Email: <a href="mailto:robert@uwiniwin.co.za">robert@uwiniwin.co.za</a> | Site: <a href="http://www.uwiniwin.co.za" target="_blank">www.uwiniwin.co.za</a><br>
>>> Address: The Oval Office Park , Ground Floor Kingsmead, Meadowbrook Lane, Bryanston<br>
>>><br>
>>> [site-award-logo]<br>
>>><br>
>>> [site-aita-logo]<br>
>>><br>
>>> [cid:image003.jpg@01D0986B.205AF960]<<a href="https://twitter.com/intent/follow?original_referer=http%3A%2F%2Fwww.linkedin.com%2Fprofile%2Fview%3Fid%3D7202140%26locale%3Den_US%26trk%3Dtyah&region=follow_link&screen_name=sandmandavid&source=followbutton&variant=2.0" target="_blank">https://twitter.com/intent/follow?original_referer=http%3A%2F%2Fwww.linkedin.com%2Fprofile%2Fview%3Fid%3D7202140%26locale%3Den_US%26trk%3Dtyah&region=follow_link&screen_name=sandmandavid&source=followbutton&variant=2.0</a>><br>
>>><br>
>>> [czxc]<<a href="http://www.facebook.com/UwinIwinIncentives" target="_blank">http://www.facebook.com/UwinIwinIncentives</a>><br>
>>><br>
>>> [cid:image005.jpg@01D0986B.205AF960]<<a href="http://www.linkedin.com/company/uwin-iwin-incentives?trk=fc_badge" target="_blank">http://www.linkedin.com/company/uwin-iwin-incentives?trk=fc_badge</a>><br>
>>><br>
>>> [cid:image006.jpg@01D0986B.205AF960]<<a href="http://www.linkedin.com/company/uwin-iwin-incentives?trk=fc_badge" target="_blank">http://www.linkedin.com/company/uwin-iwin-incentives?trk=fc_badge</a>><br>
>>><br>
>>><br>
>>> [cid:image007.jpg@01D0986B.205AF960]<<a href="http://www.uwiniwin.net/" target="_blank">http://www.uwiniwin.net/</a>><br>
>>><br>
>>><br>
>>> Please consider the environment before printing this email-<br>
>>><br>
>>> The views expressed in this e-mail are, unless otherwise stated, those of the author and not those of Uwin Iwin Incentives (PTY) LTD or its management. The information in this e-mail is confidential and is intended solely for the addressee. Access to this e-mail by anyone else is unauthorised. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted in reliance on this is prohibited and may be unlawful. Whilst all reasonable steps are taken to ensure the accuracy and integrity of information and data transmitted electronically and to preserve the confidentiality thereof, no liability or responsibility whatsoever is accepted if information or data is, for whatever reason, corrupted or does not reach its intended destination.<br>
>>><br>
>>><br>
>>><br>
>>><br>
>>><br>
> _______________________________________________<br>
> pgpool-general mailing list<br>
> <a href="mailto:pgpool-general@pgpool.net">pgpool-general@pgpool.net</a><br>
> <a href="http://www.pgpool.net/mailman/listinfo/pgpool-general" target="_blank">http://www.pgpool.net/mailman/listinfo/pgpool-general</a><br>
</div></div><br>_______________________________________________<br>
pgpool-general mailing list<br>
<a href="mailto:pgpool-general@pgpool.net">pgpool-general@pgpool.net</a><br>
<a href="http://www.pgpool.net/mailman/listinfo/pgpool-general" target="_blank">http://www.pgpool.net/mailman/listinfo/pgpool-general</a><br>
<br></blockquote></div><br></div>