[pgpool-hackers: 473] Re: penultimate patch for exception and memory manager ( EXCEPTION_MGR branch)
Muhammad Usama
m.usama at gmail.com
Fri Mar 28 00:32:01 JST 2014
Hi Tatsuo.
I am looking into the problem of double free reported by you. but I am
unable to reproduce the scenario at my end. I tried the regression on Linux
Ubuntu x86_64 and on OS/X and regression is clean on both systems. Can you
please tell me if you can always produce the problem or with some specific
configure switches.
And if it is possible for you to send me the stack trace with symbols for
the problem that would be really helpful for identifying the exact issue.
thanks
Muhammad Usama
On Wed, Mar 19, 2014 at 6:20 PM, Muhammad Usama <m.usama at gmail.com> wrote:
> Hi Tatsuo
>
> Sorry for the delay. I was stuck in some other work, and now I am resuming
> the work on this patch. For some reason regression was working fine on my
> machine when I sent out the patch but will send you the updated patch after
> figuring out the problem.
>
>
> Thanks
> Usama
>
>
>
> On Wed, Feb 19, 2014 at 9:23 AM, Tatsuo Ishii <ishii at postgresql.org>wrote:
>
>> Usama,
>>
>> Thanks for the explanation.
>>
>> BTW, I ran a reggression test against EXEC_MANAGER branch and got
>> serious problems.
>>
>> testing 003.failover...*** glibc detected *** pgpool: worker process:
>> free(): invalid pointer: 0x00000000007756c8 ***
>>
>> testing 055.backend_all_down...*** glibc detected *** /home/t-ishii/work/
>> git.postgresql.org/pgpool2/src/test/regression/temp/installed/bin/pgpool:
>> double free or corruption (!prev): 0x000000000075c050 ***
>>
>> [ and regression test hangs]
>>
>> Detailed log attached.
>>
>> Best regards,
>> --
>> Tatsuo Ishii
>> SRA OSS, Inc. Japan
>> English: http://www.sraoss.co.jp/index_en.php
>> Japanese: http://www.sraoss.co.jp
>>
>> > On Thu, Feb 13, 2014 at 3:49 AM, Tatsuo Ishii <ishii at postgresql.org>
>> wrote:
>> >
>> >> Usama,
>> >>
>> >> Thanks for your hard work.
>> >>
>> >> One thing I noticed is, you moved src/tools/pcp to src/utils/pcp. Is
>> >> there any reason for this?
>> >>
>> >
>> > Since the pgpool binary and pcp tool binaries all requires the functions
>> > defined in pcp_stream.c for reading and writing to pcp streams. But with
>> > the arrival of memory manager API in pgpool, pcp stream functions
>> requires
>> > to distinguish between calls made from pgpool and pcp tools. As we need
>> to
>> > create reading and writing buffers in specific MemoryContexts when the
>> > function is invoked from pgpool, and as pcp tools does not have a
>> concept
>> > of MemoryContext so the step of selecting a MemoryContext before
>> allocating
>> > a memory needed to be skipped when in pcp tools environment.
>> >
>> > To achieve the above, the patch makes the libpcp exclusive for the tool
>> > binaries only (compile with POOL_PRIVATE flag) and now pgpool uses the
>> > pcp_stream.c function directly and no longer has the dependancy on the
>> lib.
>> >
>> > To make this new arrangement visible in pgpool source directory
>> structure
>> > the patch rearranges the source code organisation and moves the
>> > pcp_stream.c and pcp_error.c files to src/utils/pcp from src/lib/pcp
>> > directory, while the pcp.c file still remains at its original directory.
>> > Also the new arrangement of code seems more naturally fit to the rest of
>> > pgppol-II source structure as other stream related functions also
>> resides
>> > in src/utils directory.
>> >
>> > --
>> > Thanks
>> > Muhammad Usama
>> >
>> >
>> > --
>> >> Tatsuo Ishii
>> >> SRA OSS, Inc. Japan
>> >> English: http://www.sraoss.co.jp/index_en.php
>> >> Japanese: http://www.sraoss.co.jp
>> >>
>> >> From: Muhammad Usama <m.usama at gmail.com>
>> >> Subject: Re: penultimate patch for exception and memory manager (
>> >> EXCEPTION_MGR branch)
>> >> Date: Tue, 11 Feb 2014 19:55:48 +0500
>> >> Message-ID: <
>> >> CAEJvTzWEjDWgtJ3HGCuaSwkGpXzWBETmrM4ShrA5tcNTQFvEuA at mail.gmail.com>
>> >>
>> >> > Hi Tatsuo
>> >> >
>> >> > I have rebased the EXCEPTION_MGR branch to current master, please
>> find
>> >> the
>> >> > updated patch generated after rebasing.
>> >> >
>> >> > Thanks
>> >> > Muhammad Usama
>> >> >
>> >> >
>> >> > On Tue, Feb 11, 2014 at 11:17 AM, Tatsuo Ishii <ishii at postgresql.org
>> >
>> >> wrote:
>> >> >
>> >> >> Usama,
>> >> >>
>> >> >> I have checked EXCEPTION_MGR branch and it seems it is far behind
>> main
>> >> >> branch. The last patch brought to the EXCEPTION_MGR was dated
>> >> >> 2013/12/19. Can you please rebase the branch?
>> >> >>
>> >> >> Best regards,
>> >> >> --
>> >> >> Tatsuo Ishii
>> >> >> SRA OSS, Inc. Japan
>> >> >> English: http://www.sraoss.co.jp/index_en.php
>> >> >> Japanese: http://www.sraoss.co.jp
>> >> >>
>> >> >> From: Muhammad Usama <m.usama at gmail.com>
>> >> >> Subject: penultimate patch for exception and memory manager (
>> >> >> EXCEPTION_MGR branch)
>> >> >> Date: Mon, 10 Feb 2014 19:53:41 +0500
>> >> >> Message-ID: <CAEJvTzUZHKKHjz-ZvYaAsviwVVBKSoX-ByNq9rhANO1zPJYR=
>> >> >> w at mail.gmail.com>
>> >> >>
>> >> >> > Please find the penultimate patch for integration of memory and
>> >> exception
>> >> >> > manager APIs into pgpool.
>> >> >> > The patch is generated on the EXCEPTION_MGR development branch and
>> >> >> > integrates the said APIs in query processing portion of pgpool-II
>> >> code.
>> >> >> >
>> >> >> > Patch summary for review
>> >> >> > ======================
>> >> >> > The patch is very large mainly because of replacing of
>> pool_error(),
>> >> >> > pool_debug() and pool_log() function calls to ereport(ERROR,..)
>> >> >> > ereport(DEBUG,..) and ereport(LOG,..) and it would be little time
>> >> >> consuming
>> >> >> > to review the patch. Although all the above also needs the
>> thorough
>> >> >> review,
>> >> >> > as replacing pool_error() with ereport(ERROR) or ereport(FATAL) is
>> >> just
>> >> >> not
>> >> >> > a cosmetic but changes the flow of code, but there are few things,
>> >> that
>> >> >> > requires the more attention than others.
>> >> >> >
>> >> >> > Lifecycle of objects.
>> >> >> > ================
>> >> >> > Since with integration of memory manager, it is very important to
>> >> manage
>> >> >> > the life cycle of each object and making sure the object is
>> created in
>> >> >> the
>> >> >> > proper MemoryContext. Here is the summary of some important
>> objects
>> >> and
>> >> >> > their residences.
>> >> >> >
>> >> >> > Some Important Memory Contexts
>> >> >> > ============================
>> >> >> >
>> >> >> > ProcessLoopContext:
>> >> >> > This context resets at every main loop iteration of a process
>> >> >> >
>> >> >> > TopMemoryContext:
>> >> >> > This context is same as PostgreSQL's TopMemoryContext and lives
>> for a
>> >> >> life
>> >> >> > time of process.
>> >> >> >
>> >> >> > ErrorContext:
>> >> >> > Same as PostgreSQL's ErrorContext and used for error processing.
>> >> >> >
>> >> >> > QueryContext:
>> >> >> > Used by pgpool child process and it lives for single query cycle.
>> >> >> >
>> >> >> >
>> >> >> > Lifecycle of Important Objects
>> >> >> > ========================
>> >> >> >
>> >> >> > SessionContext:
>> >> >> > The session context consists of a backend and frontend connection
>> and
>> >> it
>> >> >> > lives for a single iteration of main child process loop.
>> >> >> > So the session context object is created in ProcessLoopContext.
>> >> >> >
>> >> >> > ProcessContext:
>> >> >> > ConnectionPool:
>> >> >> > BackendConnection:
>> >> >> > The above objects are required for the life of process hence are
>> >> created
>> >> >> in
>> >> >> > TopMemoryContext which lives throughout the process life time.
>> >> >> >
>> >> >> > Child Frontend connection:
>> >> >> > Frontend connection object is placed in the ProcessLoopContext
>> since
>> >> new
>> >> >> > frontend connection is created at every main loop iteration of
>> pgpool
>> >> >> child
>> >> >> > process.
>> >> >> >
>> >> >> > PCP Frontend connection:
>> >> >> > This object can live till the pcp process so the object for pcp
>> >> frontend
>> >> >> is
>> >> >> > created in TopMemoryContext.
>> >> >> >
>> >> >> > new configuration parameters
>> >> >> > ========================
>> >> >> >
>> >> >> > The patch also adds three new configuration parameters to control
>> the
>> >> >> error
>> >> >> > reporting style and details.
>> >> >> > Parameters names and their behaviours are borrowed from
>> PostgreSQL.
>> >> >> >
>> >> >> > int log_error_verbosity; /* controls how much detail about
>> error
>> >> >> should
>> >> >> > be emitted */
>> >> >> >
>> >> >> > int client_min_messages; /*controls which message should be
>> sent to
>> >> >> > client */
>> >> >> >
>> >> >> > int log_min_messages; /*controls which message should be
>> >> emitted to
>> >> >> > server log */
>> >> >> >
>> >> >> >
>> >> >> > Some other notable changes introduced by the patch.
>> >> >> > ==========================================
>> >> >> >
>> >> >> > The patch changes the error reporting behaviour and now pgpool-II
>> >> tries
>> >> >> to
>> >> >> > send all the error and log messages to the connected frontend
>> clients
>> >> >> > (controllable by "client_min_messages" configuration parameter ).
>> >> >> >
>> >> >> > The patch added an extra call back "on_system_exit" to the elog
>> API,
>> >> >> which
>> >> >> > is not present in the PostgreSQL.
>> >> >> > The call back is called at system exit by elog API just before
>> calling
>> >> >> the
>> >> >> > "on_shmem_exit" call backs.
>> >> >> > "on_system_exit" is a single call back and can be used to clean up
>> >> things
>> >> >> > which are dependent on shared memory.
>> >> >> >
>> >> >> > "repalloc()" behaviour in PostgreSQL palloc API is little
>> different
>> >> from
>> >> >> > C-API counterpart "realloc()".
>> >> >> > ("realloc()" behaves as "malloc()" if ptr argument is NULL. While
>> >> >> > "repalloc()" does not works on NULL pointers)
>> >> >> > So I have changed the behaviour of "repalloc()" to make it more
>> >> >> consistent
>> >> >> > with C-API.
>> >> >> >
>> >> >> > The patch makes the libpcp exclusive for the tool binaries only
>> >> (compile
>> >> >> > with POOL_PRIVATE flag ). pgpool binary no longer has the
>> dependancy
>> >> on
>> >> >> the
>> >> >> > lib, and uses the pcp_stream.c function directly.
>> >> >> > For this new arrangement the patch also rearranges the source code
>> >> >> > organisation and moves the pcp_stream.c and pcp_error.c files to
>> >> >> > src/utils/pcp directory from src/lib/pcp directory.
>> >> >> >
>> >> >> > What is still remaining.
>> >> >> > ==================
>> >> >> > 1- Need to decide the MemoryContext for memcache objects.
>> >> >> > 2- Watchdog and few rewrite function are still without the
>> managers.
>> >> >> > 3- Replace all remaining occurrences of pool_error, pool_debug,
>> and
>> >> >> > pool_log with appropriate ereport() calls
>> >> >> > 4 -Code cleanup
>> >> >> >
>> >> >> > Reviews comments and suggestions are most welcome
>> >> >> >
>> >> >> > Thanks
>> >> >> > Usama
>> >> >>
>> >>
>>
>> using pgpool-II at /home/t-ishii/work/
>> git.postgresql.org/pgpool2/src/test/regression/temp/installed
>> *************************
>> REGRESSION MODE : install
>> PGPOOL-II : /home/t-ishii/work/
>> git.postgresql.org/pgpool2/src/test/regression/temp/installed
>> PostgreSQL bin : /usr/local/pgsql/bin
>> pgbench : /usr/local/pgsql/bin/pgbench
>> PostgreSQL jdbc : /usr/local/pgsql/share/postgresql-9.2-1003.jdbc4.jar
>> *************************
>> testing 001.load_balance...ok.
>> testing 002.native_replication...ok.
>>
>> testing 003.failover...*** glibc detected *** pgpool: worker process:
>> free(): invalid pointer: 0x00000000007756c8 ***
>> ======= Backtrace: =========
>> /lib64/libc.so.6(+0x763b6)[0x7fd2922cf3b6]
>> pgpool: worker process[0x41d1ff]
>> pgpool: worker process[0x436363]
>> pgpool: worker process[0x436615]
>> pgpool: worker process[0x405e94]
>> pgpool: worker process[0x4072bf]
>> pgpool: worker process[0x407425]
>> pgpool: worker process[0x408d01]
>> pgpool: worker process[0x404ef8]
>> /lib64/libc.so.6(__libc_start_main+0xfd)[0x7fd292277b1d]
>> pgpool: worker process[0x4040e9]
>> ======= Memory map: ========
>> 00400000-0050c000 r-xp 00000000 08:05 2635374
>> /home/t-ishii/work/
>> git.postgresql.org/pgpool2/src/test/regression/temp/installed/bin/pgpool
>> 0070c000-0070e000<http://git.postgresql.org/pgpool2/src/test/regression/temp/installed/bin/pgpool0070c000-0070e000>rw-p 0010c000 08:05 2635374 /home/t-ishii/work/
>> git.postgresql.org/pgpool2/src/test/regression/temp/installed/bin/pgpool
>> 0070e000-00795000<http://git.postgresql.org/pgpool2/src/test/regression/temp/installed/bin/pgpool0070e000-00795000>rw-p 00000000 00:00 0 [heap]
>> 7fd29044d000-7fd290463000 r-xp 00000000 08:05 1704093
>> /lib64/libgcc_s-4.4.5.so.1
>> 7fd290463000-7fd290662000 ---p 00016000 08:05 1704093
>> /lib64/libgcc_s-4.4.5.so.1
>> 7fd290662000-7fd290663000 rw-p 00015000 08:05 1704093
>> /lib64/libgcc_s-4.4.5.so.1
>> 7fd290663000-7fd290883000 rw-s 00000000 00:04 61898777
>> /SYSV00000000 (deleted)
>> 7fd290883000-7fd29088f000 r-xp 00000000 08:05 1704191
>> /lib64/libnss_files-2.11.1.so
>> 7fd29088f000-7fd290a8e000 ---p 0000c000 08:05 1704191
>> /lib64/libnss_files-2.11.1.so
>> 7fd290a8e000-7fd290a8f000 r--p 0000b000 08:05 1704191
>> /lib64/libnss_files-2.11.1.so
>> 7fd290a8f000-7fd290a90000 rw-p 0000c000 08:05 1704191
>> /lib64/libnss_files-2.11.1.so
>> 7fd290a90000-7fd290a92000 r-xp 00000000 08:05 1704176
>> /lib64/libkeyutils-1.2.so
>> 7fd290a92000-7fd290c91000 ---p 00002000 08:05 1704176
>> /lib64/libkeyutils-1.2.so
>> 7fd290c91000-7fd290c92000 rw-p 00001000 08:05 1704176
>> /lib64/libkeyutils-1.2.so
>> 7fd290c92000-7fd290c9a000 r-xp 00000000 08:05 1704181
>> /lib64/libkrb5support.so.0.1
>> 7fd290c9a000-7fd290e99000 ---p 00008000 08:05 1704181
>> /lib64/libkrb5support.so.0.1
>> 7fd290e99000-7fd290e9a000 rw-p 00007000 08:05 1704181
>> /lib64/libkrb5support.so.0.1
>> 7fd290e9a000-7fd290e9c000 r-xp 00000000 08:05 1704128
>> /lib64/libdl-2.11.1.so
>> 7fd290e9c000-7fd29109c000 ---p 00002000 08:05 1704128
>> /lib64/libdl-2.11.1.so
>> 7fd29109c000-7fd29109d000 r--p 00002000 08:05 1704128
>> /lib64/libdl-2.11.1.so
>> 7fd29109d000-7fd29109e000 rw-p 00003000 08:05 1704128
>> /lib64/libdl-2.11.1.so
>> 7fd29109e000-7fd2910c5000 r-xp 00000000 08:05 1704166
>> /lib64/libk5crypto.so.3.1
>> 7fd2910c5000-7fd2912c4000 ---p 00027000 08:05 1704166
>> /lib64/libk5crypto.so.3.1
>> 7fd2912c4000-7fd2912c6000 rw-p 00026000 08:05 1704166
>> /lib64/libk5crypto.so.3.1
>> 7fd2912c6000-7fd2912c9000 r-xp 00000000 08:05 1704103
>> /lib64/libcom_err.so.2.1
>> 7fd2912c9000-7fd2914c8000 ---p 00003000 08:05 1704103
>> /lib64/libcom_err.so.2.1
>> 7fd2914c8000-7fd2914c9000 rw-p 00002000 08:05 1704103
>> /lib64/libcom_err.so.2.1
>> 7fd2914c9000-7fd291597000 r-xp 00000000 08:05 1704179
>> /lib64/libkrb5.so.3.3
>> 7fd291597000-7fd291796000 ---p 000ce000 08:05 1704179
>> /lib64/libkrb5.so.3.3
>> 7fd291796000-7fd2917a1000 rw-p 000cd000 08:05 1704179
>> /lib64/libkrb5.so.3.3
>> 7fd2917a1000-7fd2917d9000 r-xp 00000000 08:05 1704152
>> /lib64/libgssapi_krb5.so.2.2
>> 7fd2917d9000-7fd2919d8000 ---p 00038000 08:05 1704152
>> /lib64/libgssapi_krb5.so.2.2
>> 7fd2919d8000-7fd2919da000 rw-p 00037000 08:05 1704152
>> /lib64/libgssapi_krb5.so.2.2
>> 7fd2919da000-7fd291a36000 r-xp 00000000 08:05 1970855
>> /usr/lib64/libfreebl3.so
>> 7fd291a36000-7fd291c35000 ---p 0005c000 08:05 1970855
>> /usr/lib64/libfreebl3.so
>> 7fd291c35000-7fd291c37000 rw-p 0005b000 08:05 1970855
>> /usr/lib64/libfreebl3.so
>> 7fd291c37000-7fd291c3b000 rw-p 00000000 00:00 0
>> 7fd291c3b000-7fd291dd1000 r-xp 00000000 08:05 1703939
>> /lib64/libcrypto.so.1.0.0l
>> 7fd291dd1000-7fd291fd0000 ---p 00196000 08:05 1703939
>> /lib64/libcrypto.so.1.0.0l
>> 7fd291fd0000-7fd291ff3000 rw-p 00195000 08:05 1703939
>> /lib64/libcrypto.so.1.0.0l
>> 7fd291ff3000-7fd291ff7000 rw-p 00000000 00:00 0
>> 7fd291ff7000-7fd292052000 r-xp 00000000 08:05 1703951
>> /lib64/libssl.so.1.0.0l
>> 7fd292052000-7fd292251000 ---p 0005b000 08:05 1703951
>> /lib64/libssl.so.1.0.0l
>> 7fd292251000-7fd292259000 rw-p 0005a000 08:05 1703951
>> /lib64/libssl.so.1.0.0l
>> 7fd292259000-7fd2923ca000 r-xp 00000000 08:05 1703946
>> /lib64/libc-2.11.1.so
>> 7fd2923ca000-7fd2925ca000 ---p 00171000 08:05 1703946
>> /lib64/libc-2.11.1.so
>> 7fd2925ca000-7fd2925ce000 r--p 00171000 08:05 1703946
>> /lib64/libc-2.11.1.so
>> 7fd2925ce000-7fd2925cf000 rw-p 00175000 08:05 1703946
>> /lib64/libc-2.11.1.so
>> 7fd2925cf000-7fd2925d4000 rw-p 00000000 00:00 0
>> 7fd2925d4000-7fd292656000 r-xp 00000000 08:05 1704182
>> /lib64/libm-2.11.1.so
>> 7fd292656000-7fd292855000 ---p 00082000 08:05 1704182
>> /lib64/libm-2.11.1.so
>> 7fd292855000-7fd292856000 r--p 00081000 08:05 1704182
>> /lib64/libm-2.11.1.so
>> 7fd292856000-7fd292857000 rw-p 00082000 08:05 1704182
>> /lib64/libm-2.11.1.so
>> 7fd292857000-7fd29286d000 r-xp 00000000 08:05 1704185
>> /lib64/libnsl-2.11.1.so
>> 7fd29286d000-7fd292a6c000 ---p 00016000 08:05 1704185
>> /lib64/libnsl-2.11.1.so
>> 7fd292a6c000-7fd292a6d000 r--p 00015000 08:05 1704185
>> /lib64/libnsl-2.11.1.so
>> 7fd292a6d000-7fd292a6e000 rw-p 00016000 08:05 1704185
>> /lib64/libnsl-2.11.1.so
>> 7fd292a6e000-7fd292a70000 rw-p 00000000 00:00 0
>> 7fd292a70000-7fd292a86000 r-xp 00000000 08:05 1704202
>> /lib64/libresolv-2.11.1.so
>> 7fd292a86000-7fd292c85000 ---p 00016000 08:05 1704202
>> /lib64/libresolv-2.11.1.so
>> 7fd292c85000-7fd292c86000 r--p 00015000 08:05 1704202
>> /lib64/libresolv-2.11.1.so
>> 7fd292c86000-7fd292c87000 rw-p 00016000 08:05 1704202
>> /lib64/libresolv-2.11.1.so
>> 7fd292c87000-7fd292c89000 rw-p 00000000 00:00 0
>> 7fd292c89000-7fd292c90000 r-xp 00000000 08:05 1703950
>> /lib64/libcrypt-2.11.1.so
>> 7fd292c90000-7fd292e90000 ---p 00007000 08:05 1703950
>> /lib64/libcrypt-2.11.1.so
>> 7fd292e90000-7fd292e91000 r--p 00007000 08:05 1703950
>> /lib64/libcrypt-2.11.1.so
>> 7fd292e91000-7fd292e92000 rw-p 00008000 08:05 1703950
>> /lib64/libcrypt-2.11.1.so
>> 7fd292e92000-7fd292ec0000 rw-p 00000000 00:00 0
>> 7fd292ec0000-7fd292ed7000 r-xp 00000000 08:05 1703970
>> /lib64/libpthread-2.11.1.so
>> 7fd292ed7000-7fd2930d6000 ---p 00017000 08:05 1703970
>> /lib64/libpthread-2.11.1.so
>> 7fd2930d6000-7fd2930d7000 r--p 00016000 08:05 1703970
>> /lib64/libpthread-2.11.1.so
>> 7fd2930d7000-7fd2930d8000 rw-p 00017000 08:05 1703970
>> /lib64/libpthread-2.11.1.so
>> 7fd2930d8000-7fd2930dc000 rw-p 00000000 00:00 0
>> 7fd2930fb000-7fd293102000 r-xp 00000000 08:05 2635363
>> /home/t-ishii/work/
>> git.postgresql.org/pgpool2/src/test/regression/temp/installed/lib/libpcp.so.0.0.0
>> 7fd293102000-7fd293302000<http://git.postgresql.org/pgpool2/src/test/regression/temp/installed/lib/libpcp.so.0.0.07fd293102000-7fd293302000>---p 00007000 08:05 2635363 /home/t-ishii/work/
>> git.postgresql.org/pgpool2/src/test/regression/temp/installed/lib/libpcp.so.0.0.0
>> 7fd293302000-7fd293303000<http://git.postgresql.org/pgpool2/src/test/regression/temp/installed/lib/libpcp.so.0.0.07fd293302000-7fd293303000>rw-p 00007000 08:05 2635363 /home/t-ishii/work/
>> git.postgresql.org/pgpool2/src/test/regression/temp/installed/lib/libpcp.so.0.0.0
>> 7fd293303000-7fd29332b000<http://git.postgresql.org/pgpool2/src/test/regression/temp/installed/lib/libpcp.so.0.0.07fd293303000-7fd29332b000>r-xp 00000000 08:05 2004750
>> /usr/local/pgsql/lib/libpq.so.5.6
>> 7fd29332b000-7fd29352a000 ---p 00028000 08:05 2004750
>> /usr/local/pgsql/lib/libpq.so.5.6
>> 7fd29352a000-7fd29352d000 rw-p 00027000 08:05 2004750
>> /usr/local/pgsql/lib/libpq.so.5.6
>> 7fd29352d000-7fd29354b000 r-xp 00000000 08:05 1703940
>> /lib64/ld-2.11.1.so
>> 7fd29372e000-7fd29372f000 rw-s 00000000 00:04 61997084
>> /SYSV00000000 (deleted)
>> 7fd29372f000-7fd293730000 rw-s 00000000 00:04 61964315
>> /SYSV00000000 (deleted)
>> 7fd293730000-7fd293731000 rw-s 00000000 00:04 61931546
>> /SYSV00000000 (deleted)
>> 7fd293731000-7fd29373f000 rw-s 00000000 00:04 61866007
>> /SYSV00000000 (deleted)
>> 7fd29373f000-7fd29374a000 rw-p 00000000 00:00 0
>> 7fd29374a000-7fd29374b000 r--p 0001d000 08:05 1703940
>> /lib64/ld-2.11.1.so
>> 7fd29374b000-7fd29374c000 rw-p 0001e000 08:05 1703940
>> /lib64/ld-2.11.1.so
>> 7fd29374c000-7fd29374d000 rw-p 00000000 00:00 0
>> 7fffb3324000-7fffb3345000 rw-p 00000000 00:00 0
>> [stack]
>> 7fffb33ff000-7fffb3400000 r-xp 00000000 00:00 0
>> [vdso]
>> ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0
>> [vsyscall]
>>
>> *** glibc detected *** pgpool: worker process: free(): invalid pointer:
>> 0x00000000007756c8 ***
>> ======= Backtrace: =========
>> /lib64/libc.so.6(+0x763b6)[0x7fd2922cf3b6]
>> pgpool: worker process[0x41d1ff]
>> pgpool: worker process[0x436363]
>> pgpool: worker process[0x436615]
>> pgpool: worker process[0x405e94]
>> pgpool: worker process[0x4072bf]
>> pgpool: worker process[0x407425]
>> pgpool: worker process[0x408d01]
>> pgpool: worker process[0x404ef8]
>> /lib64/libc.so.6(__libc_start_main+0xfd)[0x7fd292277b1d]
>> pgpool: worker process[0x4040e9]
>> ======= Memory map: ========
>> 00400000-0050c000 r-xp 00000000 08:05 2635374
>> /home/t-ishii/work/
>> git.postgresql.org/pgpool2/src/test/regression/temp/installed/bin/pgpool
>> 0070c000-0070e000<http://git.postgresql.org/pgpool2/src/test/regression/temp/installed/bin/pgpool0070c000-0070e000>rw-p 0010c000 08:05 2635374 /home/t-ishii/work/
>> git.postgresql.org/pgpool2/src/test/regression/temp/installed/bin/pgpool
>> 0070e000-00795000<http://git.postgresql.org/pgpool2/src/test/regression/temp/installed/bin/pgpool0070e000-00795000>rw-p 00000000 00:00 0 [heap]
>> 7fd29044d000-7fd290463000 r-xp 00000000 08:05 1704093
>> /lib64/libgcc_s-4.4.5.so.1
>> 7fd290463000-7fd290662000 ---p 00016000 08:05 1704093
>> /lib64/libgcc_s-4.4.5.so.1
>> 7fd290662000-7fd290663000 rw-p 00015000 08:05 1704093
>> /lib64/libgcc_s-4.4.5.so.1
>> 7fd290663000-7fd290883000 rw-s 00000000 00:04 61898777
>> /SYSV00000000 (deleted)
>> 7fd290883000-7fd29088f000 r-xp 00000000 08:05 1704191
>> /lib64/libnss_files-2.11.1.so
>> 7fd29088f000-7fd290a8e000 ---p 0000c000 08:05 1704191
>> /lib64/libnss_files-2.11.1.so
>> 7fd290a8e000-7fd290a8f000 r--p 0000b000 08:05 1704191
>> /lib64/libnss_files-2.11.1.so
>> 7fd290a8f000-7fd290a90000 rw-p 0000c000 08:05 1704191
>> /lib64/libnss_files-2.11.1.so
>> 7fd290a90000-7fd290a92000 r-xp 00000000 08:05 1704176
>> /lib64/libkeyutils-1.2.so
>> 7fd290a92000-7fd290c91000 ---p 00002000 08:05 1704176
>> /lib64/libkeyutils-1.2.so
>> 7fd290c91000-7fd290c92000 rw-p 00001000 08:05 1704176
>> /lib64/libkeyutils-1.2.so
>> 7fd290c92000-7fd290c9a000 r-xp 00000000 08:05 1704181
>> /lib64/libkrb5support.so.0.1
>> 7fd290c9a000-7fd290e99000 ---p 00008000 08:05 1704181
>> /lib64/libkrb5support.so.0.1
>> 7fd290e99000-7fd290e9a000 rw-p 00007000 08:05 1704181
>> ok.
>> testing 004.watchdog...ok.
>> testing 005.jdbc...ok.
>> testing 006.memqcache...ok.
>> testing 050.bug58...ok.
>> testing 051.bug60...ok.
>> testing 052.do_query...ok.
>> testing 053.insert_lock_hangs...ok.
>> testing 054.postgres_fdw...ok.
>>
>> testing 055.backend_all_down...*** glibc detected *** /home/t-ishii/work/
>> git.postgresql.org/pgpool2/src/test/regression/temp/installed/bin/pgpool:
>> double free or corruption (!prev): 0x000000000075c050 ***
>> *** glibc detected *** pgpool: worker process: double free or corruption
>> (!prev): 0x000000000075d490 ***
>> ======= Backtrace: =========
>> /lib64/libc.so.6(+0x763b6)[0x7f0dca1f93b6]
>> /home/t-ishii/work/
>> git.postgresql.org/pgpool2/src/test/regression/temp/installed/bin/pgpool[0x41d1ff]
>>
>> /home/t-ishii/work/git.postgresql.org/pgpool2/src/test/regression/temp/installed/bin/pgpool[0x4058a7]
>>
>> /home/t-ishii/work/git.postgresql.org/pgpool2/src/test/regression/temp/installed/bin/pgpool[0x40882f]
>>
>> /home/t-ishii/work/git.postgresql.org/pgpool2/src/test/regression/temp/installed/bin/pgpool[0x404ef8]<http://git.postgresql.org/pgpool2/src/test/regression/temp/installed/bin/pgpool%5B0x41d1ff%5D/home/t-ishii/work/git.postgresql.org/pgpool2/src/test/regression/temp/installed/bin/pgpool%5B0x4058a7%5D/home/t-ishii/work/git.postgresql.org/pgpool2/src/test/regression/temp/installed/bin/pgpool%5B0x40882f%5D/home/t-ishii/work/git.postgresql.org/pgpool2/src/test/regression/temp/installed/bin/pgpool%5B0x404ef8%5D>
>> ======= Backtrace: =========
>> /lib64/libc.so.6(__libc_start_main+0xfd)[0x7f0dca1a1b1d]
>> /home/t-ishii/work/
>> git.postgresql.org/pgpool2/src/test/regression/temp/installed/bin/pgpool[0x4040e9]<http://git.postgresql.org/pgpool2/src/test/regression/temp/installed/bin/pgpool%5B0x4040e9%5D>
>> pgpool: worker process======= Memory map: ========
>> /lib64/libc.so.6(__libc_start_main+0xfd)[0x7f0dca1a1b1d]
>> pgpool: worker process[0x4040e9]
>> ======= Memory map: ========
>> 00400000-0050c000 r-xp 00000000 08:05 2635374
>> /home/t-ishii/work/
>> git.postgresql.org/pgpool2/src/test/regression/temp/installed/bin/pgpool
>> 0070c000-0070e000<http://git.postgresql.org/pgpool2/src/test/regression/temp/installed/bin/pgpool0070c000-0070e000>rw-p 0010c000 08:05 2635374 /home/t-ishii/work/
>> git.postgresql.org/pgpool2/src/test/regression/temp/installed/bin/pgpool
>> 0070e000-00771000<http://git.postgresql.org/pgpool2/src/test/regression/temp/installed/bin/pgpool0070e000-00771000>rw-p 00000000 00:00 0 [heap]
>> 7f0dc8377000-7f0dc838d000 r-xp 00000000 08:05 1704093
>> /lib64/libgcc_s-4.4.5.so.1
>> 7f0dc838d000-7f0dc858c000 ---p 00016000 08:05 1704093
>> /lib64/libgcc_s-4.4.5.so.1
>> 7f0dc858c000-7f0dc858d000 rw-p 00015000 08:05 1704093
>> /lib64/libgcc_s-4.4.5.so.1
>> 7f0dc858d000-7f0dc87ad000 rw-s 00000000 00:04 84148247
>> /SYSV00000000 (deleted)
>> 7f0dc87ad000-7f0dc87b9000 r-xp 00000000 08:05 1704191
>> /lib64/libnss_files-2.11.1.so
>> 7f0dc87b9000-7f0dc89b8000 ---p 0000c000 08:05 1704191
>> /lib64/libnss_files-2.11.1.so
>> 7f0dc89b8000-7f0dc89b9000 r--p 0000b000 08:05 1704191
>> /lib64/libnss_files-2.11.1.so
>> 7f0dc89b9000-7f0dc89ba000 rw-p 0000c000 08:05 1704191
>> /lib64/libnss_files-2.11.1.so
>> 7f0dc89ba000-7f0dc89bc000 r-xp 00000000 08:05 1704176
>> /lib64/libkeyutils-1.2.so
>> 7f0dc89bc000-7f0dc8bbb000 ---p 00002000 08:05 1704176
>> /lib64/libkeyutils-1.2.so
>> 7f0dc8bbb000-7f0dc8bbc000 rw-p 00001000 08:05 1704176
>> /lib64/libkeyutils-1.2.so
>> 7f0dc8bbc000-7f0dc8bc4000 r-xp 00000000 08:05 1704181
>> /lib64/libkrb5support.so.0.1
>> 7f0dc8bc4000-7f0dc8dc3000 ---p 00008000 08:05 1704181
>> /lib64/libkrb5support.so.0.1
>> 7f0dc8dc3000-7f0dc8dc4000 rw-p 00007000 08:05 1704181
>> /lib64/libkrb5support.so.0.1
>> 7f0dc8dc4000-7f0dc8dc6000 r-xp 00000000 08:05 1704128
>> /lib64/libdl-2.11.1.so
>> 7f0dc8dc6000-7f0dc8fc6000 ---p 00002000 08:05 1704128
>> /lib64/libdl-2.11.1.so
>> 7f0dc8fc6000-7f0dc8fc7000 r--p 00002000 08:05 1704128
>> /lib64/libdl-2.11.1.so
>> 7f0dc8fc7000-7f0dc8fc8000 rw-p 00003000 08:05 1704128
>> /lib64/libdl-2.11.1.so
>> 7f0dc8fc8000-7f0dc8fef000 r-xp 00000000 08:05 1704166
>> /lib64/libk5crypto.so.3.1
>> 7f0dc8fef000-7f0dc91ee000 ---p 00027000 08:05 1704166
>> /lib64/libk5crypto.so.3.1
>> 7f0dc91ee000-7f0dc91f0000 rw-p 00026000 08:05 1704166
>> /lib64/libk5crypto.so.3.1
>> 7f0dc91f0000-7f0dc91f3000 r-xp 00000000 08:05 1704103
>> /lib64/libcom_err.so.2.1
>> 7f0dc91f3000-7f0dc93f2000 ---p 00003000 08:05 1704103
>> /lib64/libcom_err.so.2.1
>> 7f0dc93f2000-7f0dc93f3000 rw-p 00002000 08:05 1704103
>> /lib64/libcom_err.so.2.1
>> 7f0dc93f3000-7f0dc94c1000 r-xp 00000000 08:05 1704179
>> /lib64/libkrb5.so.3.3
>> 7f0dc94c1000-7f0dc96c0000 ---p 000ce000 08:05 1704179
>> /lib64/libkrb5.so.3.3
>> 7f0dc96c0000-7f0dc96cb000 rw-p 000cd000 08:05 1704179
>> /lib64/libkrb5.so.3.3
>> 7f0dc96cb000-7f0dc9703000 r-xp 00000000 08:05 1704152
>> /lib64/libgssapi_krb5.so.2.2
>> 7f0dc9703000-7f0dc9902000 ---p 00038000 08:05 1704152
>> /lib64/libgssapi_krb5.so.2.2
>> 7f0dc9902000-7f0dc9904000 rw-p 00037000 08:05 1704152
>> /lib64/libgssapi_krb5.so.2.2
>> 7f0dc9904000-7f0dc9960000 r-xp 00000000 08:05 1970855
>> /usr/lib64/libfreebl3.so
>> 7f0dc9960000-7f0dc9b5f000 ---p 0005c000 08:05 1970855
>> /usr/lib64/libfreebl3.so
>> 7f0dc9b5f000-7f0dc9b61000 rw-p 0005b000 08:05 1970855
>> /usr/lib64/libfreebl3.so
>> 7f0dc9b61000-7f0dc9b65000 rw-p 00000000 00:00 0
>> 7f0dc9b65000-7f0dc9cfb000 r-xp 00000000 08:05 1703939
>> /lib64/libcrypto.so.1.0.0l
>> 7f0dc9cfb000-7f0dc9efa000 ---p 00196000 08:05 1703939
>> /lib64/libcrypto.so.1.0.0l
>> 7f0dc9efa000-7f0dc9f1d000 rw-p 00195000 08:05 1703939
>> /lib64/libcrypto.so.1.0.0l
>> 7f0dc9f1d000-7f0dc9f21000 rw-p 00000000 00:00 0
>> 7f0dc9f21000-7f0dc9f7c000 r-xp 00000000 08:05 1703951
>> /lib64/libssl.so.1.0.0l
>> 7f0dc9f7c000-7f0dca17b000 ---p 0005b000 08:05 1703951
>> /lib64/libssl.so.1.0.0l
>> 7f0dca17b000-7f0dca183000 rw-p 0005a000 08:05 1703951
>> /lib64/libssl.so.1.0.0l
>> 7f0dca183000-7f0dca2f4000 r-xp 00000000 08:05 1703946
>> /lib64/libc-2.11.1.so
>> 7f0dca2f4000-7f0dca4f4000 ---p 00171000 08:05 1703946
>> /lib64/libc-2.11.1.so
>> 7f0dca4f4000-7f0dca4f8000 r--p 00171000 08:05 1703946
>> /lib64/libc-2.11.1.so
>> 7f0dca4f8000-7f0dca4f9000 rw-p 00175000 08:05 1703946
>> /lib64/libc-2.11.1.so
>> 7f0dca4f9000-7f0dca4fe000 rw-p 00000000 00:00 0
>> 7f0dca4fe000-7f0dca580000 r-xp 00000000 08:05 1704182
>> /lib64/libm-2.11.1.so
>> 7f0dca580000-7f0dca77f000 ---p 00082000 08:05 1704182
>> /lib64/libm-2.11.1.so
>> 7f0dca77f000-7f0dca780000 r--p 00081000 08:05 1704182
>> /lib64/libm-2.11.1.so
>> 7f0dca780000-7f0dca781000 rw-p 00082000 08:05 1704182
>> /lib64/libm-2.11.1.so
>> 7f0dca781000-7f0dca797000 r-xp 00000000 08:05 1704185
>> /lib64/libnsl-2.11.1.so
>> 7f0dca797000-7f0dca996000 ---p 00016000 08:05 1704185
>> /lib64/libnsl-2.11.1.so
>> 7f0dca996000-7f0dca997000 r--p 00015000 08:05 1704185
>> /lib64/libnsl-2.11.1.so
>> 7f0dca997000-7f0dca998000 rw-p 00016000 08:05 1704185
>> /lib64/libnsl-2.11.1.so
>> 7f0dca998000-7f0dca99a000 rw-p 00000000 00:00 0
>> 7f0dca99a000-7f0dca9b0000 r-xp 00000000 08:05 1704202
>> /lib64/libresolv-2.11.1.so
>> 7f0dca9b0000-7f0dcabaf000 ---p 00016000 08:05 1704202
>> /lib64/libresolv-2.11.1.so
>> 7f0dcabaf000-7f0dcabb0000 r--p 00015000 08:05 1704202
>> /lib64/libresolv-2.11.1.so
>> 7f0dcabb0000-7f0dcabb1000 rw-p 00016000 08:05 1704202
>> /lib64/libresolv-2.11.1.so
>> 7f0dcabb1000-7f0dcabb3000 rw-p 00000000 00:00 0
>> 7f0dcabb3000-7f0dcabba000 r-xp 00000000 08:05 1703950
>> /lib64/libcrypt-2.11.1.so
>> 7f0dcabba000-7f0dcadba000 ---p 00007000 08:05 1703950
>> /lib64/libcrypt-2.11.1.so
>> 7f0dcadba000-7f0dcadbb000 r--p 00007000 08:05 1703950
>> /lib64/libcrypt-2.11.1.so
>> 7f0dcadbb000-7f0dcadbc000 rw-p 00008000 08:05 1703950
>> /lib64/libcrypt-2.11.1.so
>> 7f0dcadbc000-7f0dcadea000 rw-p 00000000 00:00 0
>> 7f0dcadea000-7f0dcae01000 r-xp 00000000 08:05 1703970
>> /lib64/libpthread-2.11.1.so
>> 7f0dcae01000-7f0dcb000000 ---p 00017000 08:05 1703970
>> /lib64/libpthread-2.11.1.so
>> 7f0dcb000000-7f0dcb001000 r--p 00016000 08:05 1703970
>> /lib64/libpthread-2.11.1.so
>> 7f0dcb001000-7f0dcb002000 rw-p 00017000 08:05 1703970
>> /lib64/libpthread-2.11.1.so
>> 7f0dcb002000-7f0dcb006000 rw-p 00000000 00:00 0
>> 7f0dcb025000-7f0dcb02c000 r-xp 00000000 08:05 2635363
>> /home/t-ishii/work/
>> git.postgresql.org/pgpool2/src/test/regression/temp/installed/lib/libpcp.so.0.0.0
>> 7f0dcb02c000-7f0dcb22c000<http://git.postgresql.org/pgpool2/src/test/regression/temp/installed/lib/libpcp.so.0.0.07f0dcb02c000-7f0dcb22c000>---p 00007000 08:05 2635363 /home/t-ishii/work/
>> git.postgresql.org/pgpool2/src/test/regression/temp/installed/lib/libpcp.so.0.0.0
>> 7f0dcb22c000-7f0dcb22d000<http://git.postgresql.org/pgpool2/src/test/regression/temp/installed/lib/libpcp.so.0.0.07f0dcb22c000-7f0dcb22d000>rw-p 00007000 08:05 2635363 /home/t-ishii/work/
>> git.postgresql.org/pgpool2/src/test/regression/temp/installed/lib/libpcp.so.0.0.0
>> 7f0dcb22d000-7f0dcb255000<http://git.postgresql.org/pgpool2/src/test/regression/temp/installed/lib/libpcp.so.0.0.07f0dcb22d000-7f0dcb255000>r-xp 00000000 08:05 2004750
>> /usr/local/pgsql/lib/libpq.so.5.6
>> 7f0dcb255000-7f0dcb454000 ---p 00028000 08:05 2004750
>> /usr/local/pgsql/lib/libpq.so.5.6
>> 7f0dcb454000-7f0dcb457000 rw-p 00027000 08:05 2004750
>> /usr/local/pgsql/lib/libpq.so.5.6
>> 7f0dcb457000-7f0dcb475000 r-xp 00000000 08:05 1703940
>> /lib64/ld-2.11.1.so
>> 7f0dcb658000-7f0dcb659000 rw-s 00000000 00:04 84246555
>> /SYSV00000000 (deleted)
>> 7f0dcb659000-7f0dcb65a000 rw-s 00000000 00:04 84213786
>> /SYSV00000000 (deleted)
>> 7f0dcb65a000-7f0dcb65b000 rw-s 00000000 00:04 84181017
>> /SYSV00000000 (deleted)
>> 7f0dcb65b000-7f0dcb669000 rw-s 00000000 00:04 84115478
>> /SYSV00000000 (deleted)
>> 7f0dcb669000-7f0dcb674000 rw-p 00000000 00:00 0
>> 7f0dcb674000-7f0dcb675000 r--p 0001d000 08:05 1703940
>> /lib64/ld-2.11.1.so
>> 7f0dcb675000-7f0dcb676000 rw-p 0001e000 08:05 1703940
>> /lib64/ld-2.11.1.so
>> 7f0dcb676000-7f0dcb677000 rw-p 00000000 00:00 0
>> 7fff746e1000-7fff74702000 rw-p 00000000 00:00 0
>> [stack]
>> 7fff747ff000-7fff74800000 r-xp 00000000 00:00 0
>> [vdso]
>> ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0
>> [vsyscall]
>> 7fd290c92000-7fd290c9a000 r-xp 00000000 08:05 1704181
>> /lib64/libkrb5support.so.0.1
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.sraoss.jp/pipermail/pgpool-hackers/attachments/20140327/4b8f496f/attachment-0001.html>
More information about the pgpool-hackers
mailing list