[pgpool-hackers: 350] Re: Import Postgres memory management API in pgpool

[Tatsuo Ishii]

Hi Usama,

Actually pgpool already has the memory manger, palloc and friends.
See src/parser/pool_memory.c. It's basically a subset of PostgreSQL's
memory manager.

Besides that, I think complete implementation of memory manager
requires the exception manager (the one implemented in elog of
PostgreSQL using setjmp/longjmp) as well. The current memory manager
of pgpool does not have it. The exception manager will not only
greatly reduce the amount of the source code but improves the
reliability of pgpool. It's already proved in PostgreSQL. Introducing
the exception manager has already been in Postgres-XC. So I think it's
not impossible.

So my thought is, if you are not going to introduce the exception
manager of PostgreSQL as well, I think its not worth to touch the
current implementation of memory manager.
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese: http://www.sraoss.co.jp

> Currently pgpool does not contains any local memory management API, it
> relies on standard C Library memory allocation functions for acquiring and
> releasing the memory. Absence of a memory manager especially
> 
> with the growth of source code and feature makes it vulnerable to serious
> memory issues/leakages. A proper implementation of memory manager for
> pgpool can also aid in debugging and could improve the overall
> 
> performance of the system. Postgres already contains the memory manager API
> and this document proposes to adopt the memory management used by PG in
> pgpool.
> 
> 
> *Some notes about the PG's memory management API.
> *
> 
> The core of memory management in the Postgres is a MemoryContext and
> Postgres do almost all memory allocation in the "memory contexts", which has
> a defined lifecycle.
> 
> *The basic operations on a memory context are:
> ** create a context
> * allocate a chunk of memory within a context (equivalent of standard C
> library's malloc())
> * delete a context (including freeing all the memory allocated therein)
> * reset a context (free all memory allocated in the context, but not
> the context object itself)
> 
> Given a chunk of memory previously allocated from a context, one can free
> it or reallocate it larger or smaller (corresponding to standard library's
> free() and realloc() routines).
> These operations return memory to or get more memory from the same context
> the chunk was originally allocated in.
> At all times there is a "current" context denoted by
> the CurrentMemoryContext global variable.  The backend macro palloc()
>  implicitly allocates space in that context.  The
> MemoryContextSwitchTo() operation selects a new current context (and
> returns the previous context, so that the caller can restore the previous
> context before exiting).
> 
> The main advantage of memory contexts over plain use of malloc/free is that
> the entire contents of a memory context can be freed easily, this saves the
> trouble of freeing individual chunks of memory.
> This is both faster and more reliable than per-chunk bookkeeping.
> PG already use this fact to clean up at transaction end, The memory is
> reclaimed by resetting all the active contexts.
> 
> *Some Notes About the PG memory API(palloc) Versus Standard C Library*
> 
> The behaviour of palloc and friends is similar to the standard C
> library's malloc and friends, but there are some deliberate differences
> too.  Here are some notes to clarify the behaviour.
> 
> * If out of memory, palloc and repalloc exit via elog(ERROR).  They
> never return NULL, and it is not necessary or useful to test for such a
> result.
> 
> * palloc(0) is explicitly a valid operation.  It does not return a
> NULL pointer, but a valid chunk of which no bytes may be used.  (However,
> the chunk might later be repalloc'd larger; it can also be pfree'd
> without error.)  disallowed palloc(0).  It seems more consistent to allow
> it, however.) Similarly, repalloc allows realloc'ing to zero size.
> 
> * pfree and repalloc do not accept a NULL pointer.
> 
> *Using the palloc API in pgpool*
> 
> As mentioned above, I propose to following the memory management approach
> used in PG in PGPOOL. I am planning to do the following to hook the
> Postgres memory management API in pgpool.
> 
> 
> 1- Copy the the memory management source files from
> PGSRC/src/backend/utils/mmgr/ (mcxt.c and aset.c) int the pgpool source.
> 
> 2- Change the behaviour in case of allocation errors, as currently pgpool
> does not contains elog API.
> 
> 3- Create a single TopMemoryContext global memory context.
> 
> 4- Replace all malloc, free and related memory management functions calls
> with palloc and friends.
> 
> 4- Stabilise the code.
> 
> 5- Make more memory contexts, inline with the lifecycle of components of
> pgpool
> 
> 6- Stabilise the code.
> 
> *References*
> PGSRC/src/backend/utils/mmgr/README
> 
> Thanks
> Muhammad Usama