<div dir="ltr">Thanks for your support :)<div><br></div><div>> <span style="font-size:12.8px">Still I don't understand. Pgpool-II and PostgreSQL master are on the</span><span style="font-size:12.8px">same machine, that means you could set like "backend_hostname0 = </span><span style="font-size:12.8px">"127.0.0.1".</span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">Because I need the public address for pgpool_recovery() method to permit online recovery from remote nodes. </span><span style="font-size:12.8px">And pgPool like health_check process use <span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">backend_hostname0 to do so.</span></span></div><div><span style="font-size:12.8px"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></span></div><div><span style="font-size:12.8px">The setting health_check_hostname0 doesn't exist but trough, this is not a workaround.</span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">So according to the log, is the timeout error triggered by this "health_check_timeout = 6" or this "connect_timeout = 10000" ?</span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">I downed timeout to 2 seconds each and monitoring net paquets to find some details... Keep you in touch</span></div></div><div class="gmail_extra"><br><div class="gmail_quote">2018-04-27 5:00 GMT+02:00 <span dir="ltr"><<a href="mailto:pgpool-general-request@pgpool.net" target="_blank">pgpool-general-request@pgpool.net</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Send pgpool-general mailing list submissions to<br>
<a href="mailto:pgpool-general@pgpool.net">pgpool-general@pgpool.net</a><br>
<br>
To subscribe or unsubscribe via the World Wide Web, visit<br>
<a href="http://www.sraoss.jp/mailman/listinfo/pgpool-general" rel="noreferrer" target="_blank">http://www.sraoss.jp/mailman/<wbr>listinfo/pgpool-general</a><br>
or, via email, send a message with subject or body 'help' to<br>
<a href="mailto:pgpool-general-request@pgpool.net">pgpool-general-request@pgpool.<wbr>net</a><br>
<br>
You can reach the person managing the list at<br>
<a href="mailto:pgpool-general-owner@pgpool.net">pgpool-general-owner@pgpool.<wbr>net</a><br>
<br>
When replying, please edit your Subject line so it is more specific<br>
than "Re: Contents of pgpool-general digest..."<br>
<br>
<br>
Today's Topics:<br>
<br>
1. [pgpool-general: 6056] Re: "health check timer expired" on<br>
local machine (Tatsuo Ishii)<br>
2. [pgpool-general: 6057] Re: "health check timer expired" on<br>
local machine (Tatsuo Ishii)<br>
<br>
<br>
------------------------------<wbr>------------------------------<wbr>----------<br>
<br>
Message: 1<br>
Date: Fri, 27 Apr 2018 09:40:09 +0900 (JST)<br>
From: Tatsuo Ishii <<a href="mailto:ishii@sraoss.co.jp">ishii@sraoss.co.jp</a>><br>
To: <a href="mailto:psyckow.prod@gmail.com">psyckow.prod@gmail.com</a><br>
Cc: <a href="mailto:pgpool-general@pgpool.net">pgpool-general@pgpool.net</a><br>
Subject: [pgpool-general: 6056] Re: "health check timer expired" on<br>
local machine<br>
Message-ID: <<a href="mailto:20180427.094009.1280111065989297836.t-ishii@sraoss.co.jp">20180427.094009.<wbr>1280111065989297836.t-ishii@<wbr>sraoss.co.jp</a>><br>
Content-Type: Text/Plain; charset=us-ascii<br>
<br>
> 2018-04-26 20:38:10.225 CEST [23537] [unknown]@[unknown] LOG: could not<br>
> accept SSL connection: EOF detected<br>
> 2018-04-26 20:59:34.856 CEST [27744] LOG: trigger file found:<br>
> /var/lib/postgresql/9.6/main/<wbr>trigger<br>
> 2018-04-26 20:59:34.856 CEST [27746] FATAL: terminating walreceiver<br>
> process due to administrator command<br>
> 2018-04-26 20:59:34.857 CEST [27744] LOG: invalid record length at<br>
> 3/2133FD18: wanted 24, got 0<br>
> 2018-04-26 20:59:34.857 CEST [27744] LOG: redo done at 3/2133FCF0<br>
> 2018-04-26 20:59:34.857 CEST [27744] LOG: last completed transaction was<br>
> at log time 2018-04-26 20:59:29.852716+02<br>
> 2018-04-26 20:59:34.873 CEST [27744] LOG: selected new timeline ID: 94<br>
> 2018-04-26 20:59:34.994 CEST [27744] LOG: archive recovery complete<br>
> 2018-04-26 20:59:35.025 CEST [27744] LOG: MultiXact member wraparound<br>
> protections are now enabled<br>
> 2018-04-26 20:59:35.034 CEST [25506] LOG: autovacuum launcher started<br>
> 2018-04-26 20:59:35.034 CEST [27743] LOG: database system is ready to<br>
> accept connections<br>
> <br>
>> 2018-04-26 20:59:34.856 CEST [27744] LOG: trigger file found:<br>
> /var/lib/postgresql/9.6/main/<wbr>trigger<br>
> -> On this line I assume this is the standby who is talking, because there<br>
> is no /var/lib/postgresql/9.6/main directory on the master, data are mount<br>
> somewhere else. The failover process start at 20:59:29 on pgpool, and the<br>
> standby get promoted.<br>
<br>
Yes, that's my understanding too. So there's no emmitted log on the<br>
master around 2018-04-26 20:59:34.856 CEST, I assume.<br>
<br>
>> 2018-04-26 20:38:10.225 CEST [23537] [unknown]@[unknown] LOG: could not<br>
> accept SSL connection: EOF detected<br>
> This could be the weird boy. But it happened 20 minutes before the bug and<br>
> this have not much to do with the healtcheck process.<br>
<br>
No idea for this part.<br>
<br>
> No more revelant things on Postgres logs<br>
<br>
Ok.<br>
<br>
>> there's no reason for the heath check process to not accept 127.0.0.1.<br>
> <br>
> Like I said, the health process fetch PostgreSQL trough public ip. So it<br>
> get trough a different interface.<br>
<br>
Still I don't understand. Pgpool-II and PostgreSQL master are on the<br>
same machine, that means you could set like "backend_hostname0 =<br>
"127.0.0.1". But actually you did not prefer the way. The heath check<br>
process just uses the same hostname/ip using backend_hostname0.<br>
<br>
> At this time PostgreSQL was receiving ~5 inserts / second and that's all.<br>
> No error detected on the apps.<br>
<br>
Yeah, no big load.<br>
<br>
> So the only reason I could find is a problem on the public interface of<br>
> this server, but this is really really unsual as it come from a dedicated<br>
> server provider.<br>
<br>
>From the error message of heath check process:<br>
> 2018-04-26 20:59:29: pid 2153:LOG: failed to connect to PostgreSQL server<br>
> on "x.x.x.x:xxx" using INET socket<br>
> 2018-04-26 20:59:29: pid 2153:DETAIL: health check timer expired<br>
> 2018-04-26 20:59:29: pid 2153:ERROR: failed to make persistent db<br>
<br>
Pgpool-II health check process uses non-blocking socket for connecting<br>
to PostgreSQL. After issuing connect system call it waits for its<br>
completion using select system call with timeout: connect_timeout in<br>
pgpool.conf (in your case 10 seconds). On the other hand health_check<br>
timeout is 6 seconds. So after 6 seconds, an alarm interrupted the<br>
select system call and it returned with errno == EINTR, then the log<br>
emitted. Not sure why the connect system call did not respond for 6<br>
seconds.<br>
<br>
That's all what I know from the log.<br>
<br>
Best regards,<br>
--<br>
Tatsuo Ishii<br>
SRA OSS, Inc. Japan<br>
English: <a href="http://www.sraoss.co.jp/index_en.php" rel="noreferrer" target="_blank">http://www.sraoss.co.jp/index_<wbr>en.php</a><br>
Japanese:<a href="http://www.sraoss.co.jp" rel="noreferrer" target="_blank">http://www.sraoss.co.<wbr>jp</a><br>
<br>
<br>
------------------------------<br>
<br>
Message: 2<br>
Date: Fri, 27 Apr 2018 10:15:54 +0900 (JST)<br>
From: Tatsuo Ishii <<a href="mailto:ishii@sraoss.co.jp">ishii@sraoss.co.jp</a>><br>
To: <a href="mailto:psyckow.prod@gmail.com">psyckow.prod@gmail.com</a><br>
Cc: <a href="mailto:pgpool-general@pgpool.net">pgpool-general@pgpool.net</a><br>
Subject: [pgpool-general: 6057] Re: "health check timer expired" on<br>
local machine<br>
Message-ID: <<a href="mailto:20180427.101554.1927607666615743161.t-ishii@sraoss.co.jp">20180427.101554.<wbr>1927607666615743161.t-ishii@<wbr>sraoss.co.jp</a>><br>
Content-Type: Text/Plain; charset=us-ascii<br>
<br>
> Pgpool-II health check process uses non-blocking socket for connecting<br>
> to PostgreSQL. After issuing connect system call it waits for its<br>
> completion using select system call with timeout: connect_timeout in<br>
> pgpool.conf (in your case 10 seconds). On the other hand health_check<br>
> timeout is 6 seconds. So after 6 seconds, an alarm interrupted the<br>
> select system call and it returned with errno == EINTR, then the log<br>
> emitted. Not sure why the connect system call did not respond for 6<br>
> seconds.<br>
> <br>
> That's all what I know from the log.<br>
<br>
If you want to make research on this, packet dump is required.<br>
<br>
Best regards,<br>
--<br>
Tatsuo Ishii<br>
SRA OSS, Inc. Japan<br>
English: <a href="http://www.sraoss.co.jp/index_en.php" rel="noreferrer" target="_blank">http://www.sraoss.co.jp/index_<wbr>en.php</a><br>
Japanese:<a href="http://www.sraoss.co.jp" rel="noreferrer" target="_blank">http://www.sraoss.co.<wbr>jp</a><br>
<br>
<br>
------------------------------<br>
<br>
______________________________<wbr>_________________<br>
pgpool-general mailing list<br>
<a href="mailto:pgpool-general@pgpool.net">pgpool-general@pgpool.net</a><br>
<a href="http://www.pgpool.net/mailman/listinfo/pgpool-general" rel="noreferrer" target="_blank">http://www.pgpool.net/mailman/<wbr>listinfo/pgpool-general</a><br>
<br>
<br>
End of pgpool-general Digest, Vol 78, Issue 19<br>
******************************<wbr>****************<br>
</blockquote></div><br></div>