[pgpool-hackers: 355] Re: Import Postgres memory management API in pgpool

[Tatsuo Ishii]

> On Mon, Aug 26, 2013 at 8:45 PM, Tatsuo Ishii <ishii at postgresql.org> wrote:
> 
>> > Hi Tatsuo
>> >
>> >
>> > On Sat, Aug 24, 2013 at 1:47 AM, Tatsuo Ishii <ishii at postgresql.org>
>> wrote:
>> >
>> >> Hi Usama,
>> >>
>> >> Actually pgpool already has the memory manger, palloc and friends.
>> >> See src/parser/pool_memory.c. It's basically a subset of PostgreSQL's
>> >> memory manager.
>> >>
>> >
>> > Thanks for clearing it out. Actually there are lot of instances of C API
>> > memory functions
>> > in the code like malloc which lead me to the assumption of missing memory
>> > manager in most
>> > parts of pgpool code, and my assumption was only parser API and session
>> > context
>> > is using the pool_memory API.
>> >
>> > usama$ grep -R malloc pgpool2/ | wc -l
>> >      272
>> >
>> >
>> >>
>> >> Besides that, I think complete implementation of memory manager
>> >> requires the exception manager (the one implemented in elog of
>> >> PostgreSQL using setjmp/longjmp) as well. The current memory manager
>> >> of pgpool does not have it. The exception manager will not only
>> >> greatly reduce the amount of the source code but improves the
>> >> reliability of pgpool. It's already proved in PostgreSQL. Introducing
>> >> the exception manager has already been in Postgres-XC. So I think it's
>> >> not impossible.
>> >
>> >
>> >> So my thought is, if you are not going to introduce the exception
>> >> manager of PostgreSQL as well, I think its not worth to touch the
>> >> current implementation of memory manager.
>> >>
>> >
>> > Totally agreed and importing the Postgres's exception manager is also in
>> my
>> > TODO list
>> > and now I will prioritise it over importing the complete set of memory
>> > manager.
>>
>> I think You cannot arbitrary prioritize the memory manager and the
>> exception manager since the memory manager relies (calls) the
>> exception manager inside. So you technically need to implement the
>> exception manager first, then the memory manager if your plan is
>> importing PostgreSQL's memory manager.
>>
> 
> Hi Tatsuo,
> 
> I believe that's exactly what we plan to do and Usama stated that in his
> last week. We will implement the PG exception manager first before doing an
> implementation of PG memory manager.

Ahsan, you are right.

Usama, sorry for misunderstanding your last statement.

>> BTW, if you work on importing the memory manager, are you going to
>> change every place in pgpool to use the memory manager by replacing
>> the call to malloc/free at the same time? I'm asking because without
>> the work, just importing the memory manager is meaningless.
>>
> 
> Yes i believe that is the plan otherwise as you say it won't be useful.
>  But we will get the exception manager done first before we import the
> memory manager.
> 
> -- Ahsan
> 
> --
>> Tatsuo Ishii
>> SRA OSS, Inc. Japan
>> English: http://www.sraoss.co.jp/index_en.php
>> Japanese: http://www.sraoss.co.jp
>>
>> >> > Currently pgpool does not contains any local memory management API, it
>> >> > relies on standard C Library memory allocation functions for acquiring
>> >> and
>> >> > releasing the memory. Absence of a memory manager especially
>> >> >
>> >> > with the growth of source code and feature makes it vulnerable to
>> serious
>> >> > memory issues/leakages. A proper implementation of memory manager for
>> >> > pgpool can also aid in debugging and could improve the overall
>> >> >
>> >> > performance of the system. Postgres already contains the memory
>> manager
>> >> API
>> >> > and this document proposes to adopt the memory management used by PG
>> in
>> >> > pgpool.
>> >> >
>> >> >
>> >> > *Some notes about the PG's memory management API.
>> >> > *
>> >> >
>> >> > The core of memory management in the Postgres is a MemoryContext and
>> >> > Postgres do almost all memory allocation in the "memory contexts",
>> which
>> >> has
>> >> > a defined lifecycle.
>> >> >
>> >> > *The basic operations on a memory context are:
>> >> > ** create a context
>> >> > * allocate a chunk of memory within a context (equivalent of standard
>> C
>> >> > library's malloc())
>> >> > * delete a context (including freeing all the memory allocated
>> therein)
>> >> > * reset a context (free all memory allocated in the context, but not
>> >> > the context object itself)
>> >> >
>> >> > Given a chunk of memory previously allocated from a context, one can
>> free
>> >> > it or reallocate it larger or smaller (corresponding to standard
>> >> library's
>> >> > free() and realloc() routines).
>> >> > These operations return memory to or get more memory from the same
>> >> context
>> >> > the chunk was originally allocated in.
>> >> > At all times there is a "current" context denoted by
>> >> > the CurrentMemoryContext global variable.  The backend macro palloc()
>> >> >  implicitly allocates space in that context.  The
>> >> > MemoryContextSwitchTo() operation selects a new current context (and
>> >> > returns the previous context, so that the caller can restore the
>> previous
>> >> > context before exiting).
>> >> >
>> >> > The main advantage of memory contexts over plain use of malloc/free is
>> >> that
>> >> > the entire contents of a memory context can be freed easily, this
>> saves
>> >> the
>> >> > trouble of freeing individual chunks of memory.
>> >> > This is both faster and more reliable than per-chunk bookkeeping.
>> >> > PG already use this fact to clean up at transaction end, The memory is
>> >> > reclaimed by resetting all the active contexts.
>> >> >
>> >> > *Some Notes About the PG memory API(palloc) Versus Standard C Library*
>> >> >
>> >> > The behaviour of palloc and friends is similar to the standard C
>> >> > library's malloc and friends, but there are some deliberate
>> differences
>> >> > too.  Here are some notes to clarify the behaviour.
>> >> >
>> >> > * If out of memory, palloc and repalloc exit via elog(ERROR).  They
>> >> > never return NULL, and it is not necessary or useful to test for such
>> a
>> >> > result.
>> >> >
>> >> > * palloc(0) is explicitly a valid operation.  It does not return a
>> >> > NULL pointer, but a valid chunk of which no bytes may be used.
>>  (However,
>> >> > the chunk might later be repalloc'd larger; it can also be pfree'd
>> >> > without error.)  disallowed palloc(0).  It seems more consistent to
>> allow
>> >> > it, however.) Similarly, repalloc allows realloc'ing to zero size.
>> >> >
>> >> > * pfree and repalloc do not accept a NULL pointer.
>> >> >
>> >> > *Using the palloc API in pgpool*
>> >> >
>> >> > As mentioned above, I propose to following the memory management
>> approach
>> >> > used in PG in PGPOOL. I am planning to do the following to hook the
>> >> > Postgres memory management API in pgpool.
>> >> >
>> >> >
>> >> > 1- Copy the the memory management source files from
>> >> > PGSRC/src/backend/utils/mmgr/ (mcxt.c and aset.c) int the pgpool
>> source.
>> >> >
>> >> > 2- Change the behaviour in case of allocation errors, as currently
>> pgpool
>> >> > does not contains elog API.
>> >> >
>> >> > 3- Create a single TopMemoryContext global memory context.
>> >> >
>> >> > 4- Replace all malloc, free and related memory management functions
>> calls
>> >> > with palloc and friends.
>> >> >
>> >> > 4- Stabilise the code.
>> >> >
>> >> > 5- Make more memory contexts, inline with the lifecycle of components
>> of
>> >> > pgpool
>> >> >
>> >> > 6- Stabilise the code.
>> >> >
>> >> > *References*
>> >> > PGSRC/src/backend/utils/mmgr/README
>> >> >
>> >> > Thanks
>> >> > Muhammad Usama
>> >>
>>
> 
> 
> 
> -- 
> Ahsan Hadi
> Snr Director Product Development
> EnterpriseDB Corporation
> The Enterprise Postgres Company
> 
> Phone: +92-51-8358874
> Mobile: +92-333-5162114
> 
> Website: www.enterprisedb.com
> EnterpriseDB Blog: http://blogs.enterprisedb.com/
> Follow us on Twitter: http://www.twitter.com/enterprisedb
> 
> This e-mail message (and any attachment) is intended for the use of the
> individual or entity to whom it is addressed. This message contains
> information from EnterpriseDB Corporation that may be privileged,
> confidential, or exempt from disclosure under applicable law. If you are
> not the intended recipient or authorized to receive this for the intended
> recipient, any use, dissemination, distribution, retention, archiving, or
> copying of this communication is strictly prohibited. If you have received
> this e-mail in error, please notify the sender immediately by reply e-mail
> and delete this message.