<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jan 10, 2017 at 8:08 AM, Tatsuo Ishii <span dir="ltr"><<a href="mailto:ishii@sraoss.co.jp" target="_blank">ishii@sraoss.co.jp</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Usama,<br>
<br>
Problem is, there are some places where standard library functions<br>
that calls malloc() internally.<br>
(For example, asprintf(), getaddrinfo()).<br>
<br></blockquote><div>You are right, this problem makes this solution not feasible.</div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Also blocking signals/unblocking signals every time malloc() called<br>
might hurt performance (I have not done an actual measurement. So this<br>
just a guess).<br></blockquote><div><br></div><div>Yes the benchmark can revel the actual overhead, but I think the impact of this change should be</div><div>very small, since Memory manager allocates the memory in chunks and malloc is not issued for every palloc call,</div><div>But again its hard to guess the actual impact before doing the measurements.</div><div><br></div><div>So I think we can proceed with the simple solution for now and remove the ereports from the signal handlers.</div><div><br></div><div>Kind regards</div><div>Muhammad Usama</div><div><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="HOEnZb"><div class="h5"><br>
Best regards,<br>
--<br>
Tatsuo Ishii<br>
SRA OSS, Inc. Japan<br>
English: <a href="http://www.sraoss.co.jp/index_en.php" rel="noreferrer" target="_blank">http://www.sraoss.co.jp/index_<wbr>en.php</a><br>
Japanese:<a href="http://www.sraoss.co.jp" rel="noreferrer" target="_blank">http://www.sraoss.co.<wbr>jp</a><br>
<br>
> Hi Ishii-San<br>
><br>
> I have been thinking about this and I think there is one solution we can<br>
> adopt.<br>
><br>
> We can block signals before doing malloc() in, AllocSetAlloc() and<br>
> AllocSetContextCreate() function, And since pgpool-II only uses palloc and<br>
> friends to allocate memory and there are only few places where the malloc()<br>
> are issued by MemoryManager API, So if we block signals before calling the<br>
> malloc(), we can safely use elog/ereport from signal handler routines.<br>
> What are your thoughts on this. I am attaching the sample patch for the<br>
> reference.<br>
><br>
> Kind regards<br>
> Muhammad Usama<br>
><br>
><br>
> On Fri, Jan 6, 2017 at 12:03 PM, Muhammad Usama <<a href="mailto:m.usama@gmail.com">m.usama@gmail.com</a>> wrote:<br>
><br>
>> Hi Ishii-San<br>
>><br>
>><br>
>> On Fri, Jan 6, 2017 at 11:22 AM, Tatsuo Ishii <<a href="mailto:ishii@sraoss.co.jp">ishii@sraoss.co.jp</a>> wrote:<br>
>><br>
>>> After some investigations, I found PostgreSQL does not block all<br>
>>> signals: SIGTERM etc. still can interrupt.<br>
>>><br>
>>> Probably we should take out elog/ereport call from child.c:die().<br>
>>><br>
>><br>
>> Yes, this seems to be a good fix for now, But I am looking for some more<br>
>> future proof fix. Give me a couple of more days to get to the solution,<br>
>> otherwise we will go with removing the ereport from child.c:die.<br>
>><br>
>> Thanks<br>
>> Best regards<br>
>> Muhammad Usama<br>
>><br>
>><br>
>>> Best regards,<br>
>>> --<br>
>>> Tatsuo Ishii<br>
>>> SRA OSS, Inc. Japan<br>
>>> English: <a href="http://www.sraoss.co.jp/index_en.php" rel="noreferrer" target="_blank">http://www.sraoss.co.jp/index_<wbr>en.php</a><br>
>>> Japanese:<a href="http://www.sraoss.co.jp" rel="noreferrer" target="_blank">http://www.sraoss.co.<wbr>jp</a><br>
>>><br>
>>> > I have asked this how PostgreSQL solves similar problems like this.<br>
>>> > It turned out that PostgreSQL mostly blocks all signals:<br>
>>> ><br>
>>> > <a href="https://www.postgresql.org/message-id/18405.1482794285%40sss.pgh.pa.us" rel="noreferrer" target="_blank">https://www.postgresql.org/<wbr>message-id/18405.1482794285%<wbr>40sss.pgh.pa.us</a><br>
>>> ><br>
>>> > Probably we should do the same way...<br>
>>> ><br>
>>> > Best regards,<br>
>>> > --<br>
>>> > Tatsuo Ishii<br>
>>> > SRA OSS, Inc. Japan<br>
>>> > English: <a href="http://www.sraoss.co.jp/index_en.php" rel="noreferrer" target="_blank">http://www.sraoss.co.jp/index_<wbr>en.php</a><br>
>>> > Japanese:<a href="http://www.sraoss.co.jp" rel="noreferrer" target="_blank">http://www.sraoss.co.<wbr>jp</a><br>
>>> ><br>
>>> >> Usama,<br>
>>> >><br>
>>> >> If I hit CTRL-C key while executing the regression test, sometimes a<br>
>>> >> orphan pgpool child remains. Here's a stack trace of the process:<br>
>>> >><br>
>>> >> t-ishii@localhost: gdb temp/installed/bin/pgpool 3794<br>
>>> >> GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.2) 7.7.1<br>
>>> >> [snip]<br>
>>> >> Loaded symbols for /lib/x86_64-linux-gnu/libnss_<wbr>files.so.2<br>
>>> >> __lll_lock_wait_private () at ../nptl/sysdeps/unix/sysv/linu<br>
>>> x/x86_64/lowlevellock.S:95<br>
>>> >> 95 ../nptl/sysdeps/unix/sysv/<wbr>linux/x86_64/lowlevellock.S:<br>
>>> そのようなファイルやディレクトリはありません.<br>
>>> >> (gdb) bt<br>
>>> >> #0 __lll_lock_wait_private () at ../nptl/sysdeps/unix/sysv/linu<br>
>>> x/x86_64/lowlevellock.S:95<br>
>>> >> #1 0x00007f67fe20ccba in _L_lock_12808 () from<br>
>>> /lib/x86_64-linux-gnu/libc.so.<wbr>6<br>
>>> >> #2 0x00007f67fe20a6b5 in __GI___libc_malloc (bytes=15) at<br>
>>> malloc.c:2887<br>
>>> >> #3 0x00007f67fe21072a in __GI___strdup (s=0x7f67fe305dd8<br>
>>> "/etc/localtime") at strdup.c:42<br>
>>> >> #4 0x00007f67fe239f51 in tzset_internal (always=<optimized out>,<br>
>>> explicit=explicit@entry=1)<br>
>>> >> at tzset.c:444<br>
>>> >> #5 0x00007f67fe23a913 in __tz_convert (timer=timer@entry=0x7ffce1c1b<br>
>>> 7f8,<br>
>>> >> use_localtime=use_localtime@<wbr>entry=1, tp=tp@entry=0x7f67fe54bde0<br>
>>> <_tmbuf>) at tzset.c:632<br>
>>> >> #6 0x00007f67fe2387d1 in __GI_localtime (t=t@entry=0x7ffce1c1b7f8)<br>
>>> at localtime.c:42<br>
>>> >> #7 0x000000000045627b in log_line_prefix (buf=buf@entry=0x7ffce1c1b8d0,<br>
>>> line_prefix=<optimized out>,<br>
>>> >> edata=<optimized out>) at ../../src/utils/error/elog.c:<wbr>2059<br>
>>> >> #8 0x000000000045894d in send_message_to_server_log (edata=0x753320<br>
>>> <errordata>)<br>
>>> >> at ../../src/utils/error/elog.c:<wbr>2084<br>
>>> >> #9 EmitErrorReport () at ../../src/utils/error/elog.c:<wbr>1129<br>
>>> >> #10 0x0000000000456d8e in errfinish (dummy=<optimized out>) at<br>
>>> ../../src/utils/error/elog.c:<wbr>434<br>
>>> >> #11 0x0000000000421f57 in die (sig=2) at protocol/child.c:925<br>
>>> >> #12 <signal handler called><br>
>>> >> #13 _int_malloc (av=0x7f67fe546760 <main_arena>, bytes=4176) at<br>
>>> malloc.c:3302<br>
>>> >> #14 0x00007f67fe20a6c0 in __GI___libc_malloc (bytes=4176) at<br>
>>> malloc.c:2891<br>
>>> >> #15 0x000000000045561d in AllocSetAlloc (context=0xf90000, size=4128)<br>
>>> at ../../src/utils/mmgr/aset.c:<wbr>674<br>
>>> >> #16 0x000000000042cd1e in pool_init_cp () at<br>
>>> protocol/pool_connection_pool.<wbr>c:70<br>
>>> >> #17 0x0000000000423024 in do_child (fds=fds@entry=0xfa35c0) at<br>
>>> protocol/child.c:194<br>
>>> >> #18 0x00000000004086f5 in fork_a_child (fds=0xfa35c0, id=11) at<br>
>>> main/pgpool_main.c:723<br>
>>> >> #19 0x00000000004090cc in reaper () at main/pgpool_main.c:2427<br>
>>> >> #20 0x000000000040d869 in PgpoolMain (discard_status=discard_<wbr>status@entry=1<br>
>>> '\001',<br>
>>> >> clear_memcache_oidmaps=clear_<wbr>memcache_oidmaps@entry=0 '\000') at<br>
>>> main/pgpool_main.c:457<br>
>>> >> #21 0x0000000000406eb3 in main (argc=<optimized out>, argv=<optimized<br>
>>> out>) at main/main.c:305<br>
>>> >><br>
>>> >> As you can see, the process was stuck while calling malloc().<br>
>>> >><br>
>>> >> pool_init_cp() was interrupted by SIGINT (because I hit the key), and<br>
>>> >> the signal handler die() was called. In die(), elog() was<br>
>>> >> called. elog() calls log_line_prefix(), which calls<br>
>>> >> localtime(). localtime() calls malloc(), which seems to try to obtain<br>
>>> >> a lock, then it causes a dead lock since pool_init_cp() was<br>
>>> >> interrupted while calling malloc().<br>
>>> >><br>
>>> >> log_line_prefix() was copied from PostgreSQL but I see subtle<br>
>>> >> differences from the PostgreSQL's original log_line_prefix().:<br>
>>> >><br>
>>> >> pg_strftime(strfbuf, sizeof(strfbuf),<br>
>>> >> "%Y-%m-%d %H:%M:%S %Z",<br>
>>> >> pg_localtime(&stamp_time, log_timezone));<br>
>>> >><br>
>>> >> Ours:<br>
>>> >> strftime(strbuf, 128, "%Y-%m-%d %H:%M:%S", localtime(&now));<br>
>>> >><br>
>>> >> Apparently the call stuck shows that the hang happened inside<br>
>>> >> localtime() because it calls malloc(). On the other hand,<br>
>>> >> pg_localtime() seems to avoid calling localtime() by using their home<br>
>>> >> made version of localtime(), which does *not* call malloc(). I'm not<br>
>>> >> sure if it's a intention but it seems it'd better to follow<br>
>>> >> PostgreSQL's way to avoid the hang problem.<br>
>>> >><br>
>>> >> What do you think, Usama?<br>
>>> >><br>
>>> >> Best regards,<br>
>>> >> --<br>
>>> >> Tatsuo Ishii<br>
>>> >> SRA OSS, Inc. Japan<br>
>>> >> English: <a href="http://www.sraoss.co.jp/index_en.php" rel="noreferrer" target="_blank">http://www.sraoss.co.jp/index_<wbr>en.php</a><br>
>>> >> Japanese:<a href="http://www.sraoss.co.jp" rel="noreferrer" target="_blank">http://www.sraoss.co.<wbr>jp</a><br>
>>> >> ______________________________<wbr>_________________<br>
>>> >> pgpool-hackers mailing list<br>
>>> >> <a href="mailto:pgpool-hackers@pgpool.net">pgpool-hackers@pgpool.net</a><br>
>>> >> <a href="http://www.pgpool.net/mailman/listinfo/pgpool-hackers" rel="noreferrer" target="_blank">http://www.pgpool.net/mailman/<wbr>listinfo/pgpool-hackers</a><br>
>>> > ______________________________<wbr>_________________<br>
>>> > pgpool-hackers mailing list<br>
>>> > <a href="mailto:pgpool-hackers@pgpool.net">pgpool-hackers@pgpool.net</a><br>
>>> > <a href="http://www.pgpool.net/mailman/listinfo/pgpool-hackers" rel="noreferrer" target="_blank">http://www.pgpool.net/mailman/<wbr>listinfo/pgpool-hackers</a><br>
>>> ______________________________<wbr>_________________<br>
>>> pgpool-hackers mailing list<br>
>>> <a href="mailto:pgpool-hackers@pgpool.net">pgpool-hackers@pgpool.net</a><br>
>>> <a href="http://www.pgpool.net/mailman/listinfo/pgpool-hackers" rel="noreferrer" target="_blank">http://www.pgpool.net/mailman/<wbr>listinfo/pgpool-hackers</a><br>
>>><br>
>><br>
>><br>
</div></div></blockquote></div><br></div></div>