[pgpool-hackers: 354] Re: Import Postgres memory management API in pgpool

[Ahsan Hadi]

On Mon, Aug 26, 2013 at 8:45 PM, Tatsuo Ishii <ishii at postgresql.org> wrote:

> > Hi Tatsuo
> >
> >
> > On Sat, Aug 24, 2013 at 1:47 AM, Tatsuo Ishii <ishii at postgresql.org>
> wrote:
> >
> >> Hi Usama,
> >>
> >> Actually pgpool already has the memory manger, palloc and friends.
> >> See src/parser/pool_memory.c. It's basically a subset of PostgreSQL's
> >> memory manager.
> >>
> >
> > Thanks for clearing it out. Actually there are lot of instances of C API
> > memory functions
> > in the code like malloc which lead me to the assumption of missing memory
> > manager in most
> > parts of pgpool code, and my assumption was only parser API and session
> > context
> > is using the pool_memory API.
> >
> > usama$ grep -R malloc pgpool2/ | wc -l
> >      272
> >
> >
> >>
> >> Besides that, I think complete implementation of memory manager
> >> requires the exception manager (the one implemented in elog of
> >> PostgreSQL using setjmp/longjmp) as well. The current memory manager
> >> of pgpool does not have it. The exception manager will not only
> >> greatly reduce the amount of the source code but improves the
> >> reliability of pgpool. It's already proved in PostgreSQL. Introducing
> >> the exception manager has already been in Postgres-XC. So I think it's
> >> not impossible.
> >
> >
> >> So my thought is, if you are not going to introduce the exception
> >> manager of PostgreSQL as well, I think its not worth to touch the
> >> current implementation of memory manager.
> >>
> >
> > Totally agreed and importing the Postgres's exception manager is also in
> my
> > TODO list
> > and now I will prioritise it over importing the complete set of memory
> > manager.
>
> I think You cannot arbitrary prioritize the memory manager and the
> exception manager since the memory manager relies (calls) the
> exception manager inside. So you technically need to implement the
> exception manager first, then the memory manager if your plan is
> importing PostgreSQL's memory manager.
>

Hi Tatsuo,

I believe that's exactly what we plan to do and Usama stated that in his
last week. We will implement the PG exception manager first before doing an
implementation of PG memory manager.


> BTW, if you work on importing the memory manager, are you going to
> change every place in pgpool to use the memory manager by replacing
> the call to malloc/free at the same time? I'm asking because without
> the work, just importing the memory manager is meaningless.
>

Yes i believe that is the plan otherwise as you say it won't be useful.
 But we will get the exception manager done first before we import the
memory manager.

-- Ahsan

--
> Tatsuo Ishii
> SRA OSS, Inc. Japan
> English: http://www.sraoss.co.jp/index_en.php
> Japanese: http://www.sraoss.co.jp
>
> >> > Currently pgpool does not contains any local memory management API, it
> >> > relies on standard C Library memory allocation functions for acquiring
> >> and
> >> > releasing the memory. Absence of a memory manager especially
> >> >
> >> > with the growth of source code and feature makes it vulnerable to
> serious
> >> > memory issues/leakages. A proper implementation of memory manager for
> >> > pgpool can also aid in debugging and could improve the overall
> >> >
> >> > performance of the system. Postgres already contains the memory
> manager
> >> API
> >> > and this document proposes to adopt the memory management used by PG
> in
> >> > pgpool.
> >> >
> >> >
> >> > *Some notes about the PG's memory management API.
> >> > *
> >> >
> >> > The core of memory management in the Postgres is a MemoryContext and
> >> > Postgres do almost all memory allocation in the "memory contexts",
> which
> >> has
> >> > a defined lifecycle.
> >> >
> >> > *The basic operations on a memory context are:
> >> > ** create a context
> >> > * allocate a chunk of memory within a context (equivalent of standard
> C
> >> > library's malloc())
> >> > * delete a context (including freeing all the memory allocated
> therein)
> >> > * reset a context (free all memory allocated in the context, but not
> >> > the context object itself)
> >> >
> >> > Given a chunk of memory previously allocated from a context, one can
> free
> >> > it or reallocate it larger or smaller (corresponding to standard
> >> library's
> >> > free() and realloc() routines).
> >> > These operations return memory to or get more memory from the same
> >> context
> >> > the chunk was originally allocated in.
> >> > At all times there is a "current" context denoted by
> >> > the CurrentMemoryContext global variable.  The backend macro palloc()
> >> >  implicitly allocates space in that context.  The
> >> > MemoryContextSwitchTo() operation selects a new current context (and
> >> > returns the previous context, so that the caller can restore the
> previous
> >> > context before exiting).
> >> >
> >> > The main advantage of memory contexts over plain use of malloc/free is
> >> that
> >> > the entire contents of a memory context can be freed easily, this
> saves
> >> the
> >> > trouble of freeing individual chunks of memory.
> >> > This is both faster and more reliable than per-chunk bookkeeping.
> >> > PG already use this fact to clean up at transaction end, The memory is
> >> > reclaimed by resetting all the active contexts.
> >> >
> >> > *Some Notes About the PG memory API(palloc) Versus Standard C Library*
> >> >
> >> > The behaviour of palloc and friends is similar to the standard C
> >> > library's malloc and friends, but there are some deliberate
> differences
> >> > too.  Here are some notes to clarify the behaviour.
> >> >
> >> > * If out of memory, palloc and repalloc exit via elog(ERROR).  They
> >> > never return NULL, and it is not necessary or useful to test for such
> a
> >> > result.
> >> >
> >> > * palloc(0) is explicitly a valid operation.  It does not return a
> >> > NULL pointer, but a valid chunk of which no bytes may be used.
>  (However,
> >> > the chunk might later be repalloc'd larger; it can also be pfree'd
> >> > without error.)  disallowed palloc(0).  It seems more consistent to
> allow
> >> > it, however.) Similarly, repalloc allows realloc'ing to zero size.
> >> >
> >> > * pfree and repalloc do not accept a NULL pointer.
> >> >
> >> > *Using the palloc API in pgpool*
> >> >
> >> > As mentioned above, I propose to following the memory management
> approach
> >> > used in PG in PGPOOL. I am planning to do the following to hook the
> >> > Postgres memory management API in pgpool.
> >> >
> >> >
> >> > 1- Copy the the memory management source files from
> >> > PGSRC/src/backend/utils/mmgr/ (mcxt.c and aset.c) int the pgpool
> source.
> >> >
> >> > 2- Change the behaviour in case of allocation errors, as currently
> pgpool
> >> > does not contains elog API.
> >> >
> >> > 3- Create a single TopMemoryContext global memory context.
> >> >
> >> > 4- Replace all malloc, free and related memory management functions
> calls
> >> > with palloc and friends.
> >> >
> >> > 4- Stabilise the code.
> >> >
> >> > 5- Make more memory contexts, inline with the lifecycle of components
> of
> >> > pgpool
> >> >
> >> > 6- Stabilise the code.
> >> >
> >> > *References*
> >> > PGSRC/src/backend/utils/mmgr/README
> >> >
> >> > Thanks
> >> > Muhammad Usama
> >>
>



-- 
Ahsan Hadi
Snr Director Product Development
EnterpriseDB Corporation
The Enterprise Postgres Company

Phone: +92-51-8358874
Mobile: +92-333-5162114

Website: www.enterprisedb.com
EnterpriseDB Blog: http://blogs.enterprisedb.com/
Follow us on Twitter: http://www.twitter.com/enterprisedb

This e-mail message (and any attachment) is intended for the use of the
individual or entity to whom it is addressed. This message contains
information from EnterpriseDB Corporation that may be privileged,
confidential, or exempt from disclosure under applicable law. If you are
not the intended recipient or authorized to receive this for the intended
recipient, any use, dissemination, distribution, retention, archiving, or
copying of this communication is strictly prohibited. If you have received
this e-mail in error, please notify the sender immediately by reply e-mail
and delete this message.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.sraoss.jp/pipermail/pgpool-hackers/attachments/20130827/5489de9d/attachment.html>