Tatsuo,<br><br>Did you restart iptables after adding rule?<br><br>Regards,<br>Stevo.<br><br><div class="gmail_quote">On Wed, Jan 11, 2012 at 11:12 AM, Stevo Slavić <span dir="ltr"><<a href="mailto:sslavic@gmail.com">sslavic@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Looking into this to verify if these are all necessary changes to have port unreachable message silently rejected (suspecting some kernel parameter tuning is needed).<br>
<br>Just to clarify it's not a problem that host is being detected by pgpool to be down, but the timing when that happens. On environment where issue is reproduced pgpool as part of health check attempt tries to connect to backend and hangs for tcp timeout instead of being interrupted by timeout alarm. Can you verify/confirm please the health check retry timings are not delayed?<br>
<br>Regards,<br>Stevo.<div class="HOEnZb"><div class="h5"><br><br><div class="gmail_quote">On Wed, Jan 11, 2012 at 10:50 AM, Tatsuo Ishii <span dir="ltr"><<a href="mailto:ishii@postgresql.org" target="_blank">ishii@postgresql.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Ok, I did:<br>
<div><br>
# iptables -A FORWARD -j REJECT --reject-with icmp-port-unreachable<br>
<br>
</div>on the host where pgpoo is running. And pull network cable from<br>
<div>backend0 host network interface. Pgpool detected the host being down<br>
</div>as expected...<br>
<div><div>--<br>
Tatsuo Ishii<br>
SRA OSS, Inc. Japan<br>
English: <a href="http://www.sraoss.co.jp/index_en.php" target="_blank">http://www.sraoss.co.jp/index_en.php</a><br>
Japanese: <a href="http://www.sraoss.co.jp" target="_blank">http://www.sraoss.co.jp</a><br>
<br>
> Backend is not destination of this message, pgpool host is, and we don't<br>
> want it to ever get it. With command I've sent you rule will be created for<br>
> any source and destination.<br>
><br>
> Regards,<br>
> Stevo.<br>
><br>
> On Wed, Jan 11, 2012 at 10:38 AM, Tatsuo Ishii <<a href="mailto:ishii@postgresql.org" target="_blank">ishii@postgresql.org</a>> wrote:<br>
><br>
>> I did following:<br>
>><br>
>> Do following on the host where pgpool is running on:<br>
>><br>
>> # iptables -A FORWARD -j REJECT --reject-with icmp-port-unreachable -d<br>
>> 133.137.177.124<br>
>> (133.137.177.124 is the host where backend is running on)<br>
>><br>
>> Pull network cable from backend0 host network interface. Pgpool<br>
>> detected the host being down as expected. Am I missing something?<br>
>> --<br>
>> Tatsuo Ishii<br>
>> SRA OSS, Inc. Japan<br>
>> English: <a href="http://www.sraoss.co.jp/index_en.php" target="_blank">http://www.sraoss.co.jp/index_en.php</a><br>
>> Japanese: <a href="http://www.sraoss.co.jp" target="_blank">http://www.sraoss.co.jp</a><br>
>><br>
>> > Hello Tatsuo,<br>
>> ><br>
>> > With backend0 on one host just configure following rule on other host<br>
>> where<br>
>> > pgpool is:<br>
>> ><br>
>> > iptables -A FORWARD -j REJECT --reject-with icmp-port-unreachable<br>
>> ><br>
>> > and then have pgpool startup with health checking and retrying<br>
>> configured,<br>
>> > and then pull network cable from backend0 host network interface.<br>
>> ><br>
>> > Regards,<br>
>> > Stevo.<br>
>> ><br>
>> > On Wed, Jan 11, 2012 at 6:27 AM, Tatsuo Ishii <<a href="mailto:ishii@postgresql.org" target="_blank">ishii@postgresql.org</a>><br>
>> wrote:<br>
>> ><br>
>> >> I want to try to test the situation you descrived:<br>
>> >><br>
>> >> >> > When system is configured for security reasons not to return<br>
>> >> destination<br>
>> >> >> > host unreachable messages, even though health_check_timeout is<br>
>> >><br>
>> >> But I don't know how to do it. I pulled out the network cable and<br>
>> >> pgpool detected it as expected. Also I configured the server which<br>
>> >> PostgreSQL is running on to disable the 5432 port. In this case<br>
>> >> connect(2) returned EHOSTUNREACH (No route to host) so pgpool detected<br>
>> >> the error as expected.<br>
>> >><br>
>> >> Could you please instruct me?<br>
>> >> --<br>
>> >> Tatsuo Ishii<br>
>> >> SRA OSS, Inc. Japan<br>
>> >> English: <a href="http://www.sraoss.co.jp/index_en.php" target="_blank">http://www.sraoss.co.jp/index_en.php</a><br>
>> >> Japanese: <a href="http://www.sraoss.co.jp" target="_blank">http://www.sraoss.co.jp</a><br>
>> >><br>
>> >> > Hello Tatsuo,<br>
>> >> ><br>
>> >> > Thank you for replying!<br>
>> >> ><br>
>> >> > I'm not sure what exactly is blocking, just by pgpool code analysis I<br>
>> >> > suspect it is the part where a connection is made to the db and it<br>
>> >> doesn't<br>
>> >> > seem to get interrupted by alarm. Tested thoroughly health check<br>
>> >> behaviour,<br>
>> >> > it works really well when host/ip is there and just backend/postgres<br>
>> is<br>
>> >> > down, but not when backend host/ip is down. I could see in log that<br>
>> >> initial<br>
>> >> > health check and each retry got delayed when host/ip is not reachable,<br>
>> >> > while when just backend is not listening (is down) on the reachable<br>
>> >> host/ip<br>
>> >> > then initial health check and all retries are exact to the settings in<br>
>> >> > pgpool.conf.<br>
>> >> ><br>
>> >> > PGCONNECT_TIMEOUT is listed as one of the libpq environment variables<br>
>> in<br>
>> >> > the docs (see<br>
>> >> <a href="http://www.postgresql.org/docs/9.1/static/libpq-envars.html" target="_blank">http://www.postgresql.org/docs/9.1/static/libpq-envars.html</a> )<br>
>> >> > There is equivalent parameter in libpq PGconnectdbParams ( see<br>
>> >> ><br>
>> >><br>
>> <a href="http://www.postgresql.org/docs/9.1/static/libpq-connect.html#LIBPQ-CONNECT-CONNECT-TIMEOUT" target="_blank">http://www.postgresql.org/docs/9.1/static/libpq-connect.html#LIBPQ-CONNECT-CONNECT-TIMEOUT</a><br>
>> >> )<br>
>> >> > At the beginning of that same page there are some important infos on<br>
>> >> using<br>
>> >> > these functions.<br>
>> >> ><br>
>> >> > psql respects PGCONNECT_TIMEOUT.<br>
>> >> ><br>
>> >> > Regards,<br>
>> >> > Stevo.<br>
>> >> ><br>
>> >> > On Wed, Jan 11, 2012 at 12:13 AM, Tatsuo Ishii <<a href="mailto:ishii@postgresql.org" target="_blank">ishii@postgresql.org</a>><br>
>> >> wrote:<br>
>> >> ><br>
>> >> >> > Hello pgpool community,<br>
>> >> >> ><br>
>> >> >> > When system is configured for security reasons not to return<br>
>> >> destination<br>
>> >> >> > host unreachable messages, even though health_check_timeout is<br>
>> >> >> configured,<br>
>> >> >> > socket call will block and alarm will not get raised until TCP<br>
>> timeout<br>
>> >> >> > occurs.<br>
>> >> >><br>
>> >> >> Interesting. So are you saying that read(2) cannot be interrupted by<br>
>> >> >> alarm signal if the system is configured not to return destination<br>
>> >> >> host unreachable message? Could you please guide me where I can get<br>
>> >> >> such that info? (I'm not a network expert).<br>
>> >> >><br>
>> >> >> > Not a C programmer, found some info that select call could be<br>
>> replace<br>
>> >> >> with<br>
>> >> >> > select/pselect calls. Maybe it would be best if PGCONNECT_TIMEOUT<br>
>> >> value<br>
>> >> >> > could be used here for connection timeout. pgpool has libpq as<br>
>> >> >> dependency,<br>
>> >> >> > why isn't it using libpq for the healthcheck db connect calls, then<br>
>> >> >> > PGCONNECT_TIMEOUT would be applied?<br>
>> >> >><br>
>> >> >> I don't think libpq uses select/pselect for establishing connection,<br>
>> >> >> but using libpq instead of homebrew code seems to be an idea. Let me<br>
>> >> >> think about it.<br>
>> >> >><br>
>> >> >> One question. Are you sure that libpq can deal with the case (not to<br>
>> >> >> return destination host unreachable messages) by using<br>
>> >> >> PGCONNECT_TIMEOUT?<br>
>> >> >> --<br>
>> >> >> Tatsuo Ishii<br>
>> >> >> SRA OSS, Inc. Japan<br>
>> >> >> English: <a href="http://www.sraoss.co.jp/index_en.php" target="_blank">http://www.sraoss.co.jp/index_en.php</a><br>
>> >> >> Japanese: <a href="http://www.sraoss.co.jp" target="_blank">http://www.sraoss.co.jp</a><br>
>> >> >><br>
>> >><br>
>><br>
</div></div></blockquote></div><br>
</div></div></blockquote></div><br>