[pgpool-hackers: 462] Re: penultimate patch for exception and memory manager ( EXCEPTION_MGR branch)

Thu Feb 13 17:41:34 JST 2014

On Thu, Feb 13, 2014 at 3:49 AM, Tatsuo Ishii <ishii at postgresql.org> wrote:

> Usama,
>
> Thanks for your hard work.
>
> One thing I noticed is, you moved src/tools/pcp to src/utils/pcp. Is
> there any reason for this?
>

Since the pgpool binary and pcp tool binaries all requires the functions
defined in pcp_stream.c for reading and writing to pcp streams. But with
the arrival of memory manager API in pgpool, pcp stream functions requires
to distinguish between calls made from pgpool and pcp tools. As we need to
create reading and writing buffers in specific MemoryContexts when the
function is invoked from pgpool, and as pcp tools does not have a concept
of MemoryContext so the step of selecting a MemoryContext before allocating
a memory needed to be skipped when in pcp tools environment.

To achieve the above, the patch makes the libpcp exclusive for the tool
binaries only (compile with POOL_PRIVATE flag) and now pgpool uses the
pcp_stream.c function directly and no longer has the dependancy on the lib.

To make this new arrangement visible in pgpool source directory structure
the patch rearranges the source code organisation and moves the
pcp_stream.c and pcp_error.c files to src/utils/pcp from src/lib/pcp
directory, while the pcp.c file still remains at its original directory.
Also the new arrangement of code seems more naturally fit to the rest of
pgppol-II source structure as other stream related functions also resides
in src/utils directory.

--
Thanks
Muhammad Usama

--
> Tatsuo Ishii
> SRA OSS, Inc. Japan
> English: http://www.sraoss.co.jp/index_en.php
> Japanese: http://www.sraoss.co.jp
>
> From: Muhammad Usama <m.usama at gmail.com>
> Subject: Re: penultimate patch for exception and memory manager (
> EXCEPTION_MGR branch)
> Date: Tue, 11 Feb 2014 19:55:48 +0500
> Message-ID: <
> CAEJvTzWEjDWgtJ3HGCuaSwkGpXzWBETmrM4ShrA5tcNTQFvEuA at mail.gmail.com>
>
> > Hi Tatsuo
> >
> > I have rebased the EXCEPTION_MGR branch to current master, please find
> the
> > updated patch generated after rebasing.
> >
> > Thanks
> > Muhammad Usama
> >
> >
> > On Tue, Feb 11, 2014 at 11:17 AM, Tatsuo Ishii <ishii at postgresql.org>
> wrote:
> >
> >> Usama,
> >>
> >> I have checked EXCEPTION_MGR branch and it seems it is far behind main
> >> branch. The last patch brought to the EXCEPTION_MGR was dated
> >> 2013/12/19. Can you please rebase the branch?
> >>
> >> Best regards,
> >> --
> >> Tatsuo Ishii
> >> SRA OSS, Inc. Japan
> >> English: http://www.sraoss.co.jp/index_en.php
> >> Japanese: http://www.sraoss.co.jp
> >>
> >> From: Muhammad Usama <m.usama at gmail.com>
> >> Subject: penultimate patch for exception and memory manager (
> >> EXCEPTION_MGR branch)
> >> Date: Mon, 10 Feb 2014 19:53:41 +0500
> >> Message-ID: <CAEJvTzUZHKKHjz-ZvYaAsviwVVBKSoX-ByNq9rhANO1zPJYR=
> >> w at mail.gmail.com>
> >>
> >> > Please find the penultimate patch for integration of memory and
> exception
> >> > manager APIs into pgpool.
> >> > The patch is generated on the EXCEPTION_MGR development branch and
> >> > integrates the said APIs in query processing portion of pgpool-II
> code.
> >> >
> >> > Patch summary for review
> >> > ======================
> >> > The patch is very large mainly because of replacing of pool_error(),
> >> > pool_debug() and pool_log() function calls to ereport(ERROR,..)
> >> > ereport(DEBUG,..) and ereport(LOG,..) and it would be little time
> >> consuming
> >> > to review the patch. Although all the above also needs the thorough
> >> review,
> >> > as replacing pool_error() with ereport(ERROR) or ereport(FATAL) is
> just
> >> not
> >> > a cosmetic but changes the flow of code, but there are few things,
> that
> >> > requires the more attention than others.
> >> >
> >> > Lifecycle of objects.
> >> > ================
> >> > Since with integration of memory manager, it is very important to
> manage
> >> > the life cycle of each object and making sure the object is created in
> >> the
> >> > proper MemoryContext. Here is the summary of some important objects
> and
> >> > their residences.
> >> >
> >> > Some Important Memory Contexts
> >> > ============================
> >> >
> >> > ProcessLoopContext:
> >> > This context resets at every main loop iteration of a process
> >> >
> >> > TopMemoryContext:
> >> > This context is same as PostgreSQL's TopMemoryContext and lives for a
> >> life
> >> > time of process.
> >> >
> >> > ErrorContext:
> >> > Same as PostgreSQL's ErrorContext and used for error processing.
> >> >
> >> > QueryContext:
> >> > Used by pgpool child process and it lives for single query cycle.
> >> >
> >> >
> >> > Lifecycle of Important Objects
> >> > ========================
> >> >
> >> > SessionContext:
> >> > The session context consists of a backend and frontend connection and
> it
> >> > lives for a single iteration of main child process loop.
> >> > So the session context object is created in ProcessLoopContext.
> >> >
> >> > ProcessContext:
> >> > ConnectionPool:
> >> > BackendConnection:
> >> > The above objects are required for the life of process hence are
> created
> >> in
> >> > TopMemoryContext which lives throughout the process life time.
> >> >
> >> > Child Frontend connection:
> >> > Frontend connection object is placed in the ProcessLoopContext since
> new
> >> > frontend connection is created at every main loop iteration of pgpool
> >> child
> >> > process.
> >> >
> >> > PCP Frontend connection:
> >> > This object can live till the pcp process so the object for pcp
> frontend
> >> is
> >> > created in TopMemoryContext.
> >> >
> >> > new configuration parameters
> >> > ========================
> >> >
> >> > The patch also adds three new configuration parameters to control the
> >> error
> >> > reporting style and details.
> >> > Parameters names and their behaviours are borrowed from PostgreSQL.
> >> >
> >> > int log_error_verbosity;    /* controls how much detail about error
> >> should
> >> > be emitted */
> >> >
> >> > int client_min_messages;    /*controls which message should be sent to
> >> > client */
> >> >
> >> > int log_min_messages;       /*controls which message should be
> emitted to
> >> > server log */
> >> >
> >> >
> >> > Some other notable changes introduced by the patch.
> >> > ==========================================
> >> >
> >> > The patch changes the error reporting behaviour and now pgpool-II
> tries
> >> to
> >> > send all the error and log messages to the connected frontend clients
> >> > (controllable by "client_min_messages" configuration parameter ).
> >> >
> >> > The patch added an extra call back "on_system_exit" to the elog API,
> >> which
> >> > is not present in the PostgreSQL.
> >> > The call back is called at system exit by elog API just before calling
> >> the
> >> > "on_shmem_exit" call backs.
> >> > "on_system_exit" is a single call back and can be used to clean up
> things
> >> > which are dependent on shared memory.
> >> >
> >> > "repalloc()" behaviour in PostgreSQL palloc API is little different
> from
> >> > C-API counterpart "realloc()".
> >> > ("realloc()" behaves as "malloc()" if ptr argument is NULL. While
> >> > "repalloc()" does not works on NULL pointers)
> >> > So I have changed the behaviour of "repalloc()" to make it more
> >> consistent
> >> > with C-API.
> >> >
> >> > The patch makes the libpcp exclusive for the tool binaries only
> (compile
> >> > with POOL_PRIVATE flag ). pgpool binary no longer has the dependancy
> on
> >> the
> >> > lib, and uses the pcp_stream.c function directly.
> >> > For this new arrangement the patch also rearranges the source code
> >> > organisation and moves the pcp_stream.c and pcp_error.c files to
> >> > src/utils/pcp directory from src/lib/pcp directory.
> >> >
> >> > What is still remaining.
> >> > ==================
> >> > 1- Need to decide the MemoryContext for memcache objects.
> >> > 2- Watchdog and few rewrite function are still without the managers.
> >> > 3- Replace all remaining occurrences of  pool_error, pool_debug, and
> >> > pool_log with appropriate ereport() calls
> >> > 4 -Code cleanup
> >> >
> >> > Reviews comments and suggestions are most welcome
> >> >
> >> > Thanks
> >> > Usama
> >>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.sraoss.jp/pipermail/pgpool-hackers/attachments/20140213/7f8c0d47/attachment.html>