<div dir="ltr">Hi Tatsuo<br><div class="gmail_extra"><br><br><div class="gmail_quote">On Sat, Aug 24, 2013 at 1:47 AM, Tatsuo Ishii <span dir="ltr"><<a href="mailto:ishii@postgresql.org" target="_blank">ishii@postgresql.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Hi Usama,<br>
<br>
Actually pgpool already has the memory manger, palloc and friends.<br>
See src/parser/pool_memory.c. It's basically a subset of PostgreSQL's<br>
memory manager.<br></blockquote><div><br></div><div>Thanks for clearing it out. Actually there are lot of instances of C API memory functions</div><div>in the code like malloc which lead me to the assumption of missing memory manager in most</div>
<div>parts of pgpool code, and my assumption was only parser API and session context</div><div>is using the pool_memory API. </div><div><div><br></div><div>usama$ grep -R malloc pgpool2/ | wc -l</div><div> 272</div></div>
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
Besides that, I think complete implementation of memory manager<br>
requires the exception manager (the one implemented in elog of<br>
PostgreSQL using setjmp/longjmp) as well. The current memory manager<br>
of pgpool does not have it. The exception manager will not only<br>
greatly reduce the amount of the source code but improves the<br>
reliability of pgpool. It's already proved in PostgreSQL. Introducing<br>
the exception manager has already been in Postgres-XC. So I think it's<br>
not impossible. </blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
So my thought is, if you are not going to introduce the exception<br>
manager of PostgreSQL as well, I think its not worth to touch the<br>
current implementation of memory manager.<br></blockquote><div><br></div><div>Totally agreed and importing the Postgres's exception manager is also in my TODO list</div><div>and now I will prioritise it over importing the complete set of memory manager.</div>
<div><br></div><div>Thanks</div><div>Muhammad Usama</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
--<br>
Tatsuo Ishii<br>
SRA OSS, Inc. Japan<br>
English: <a href="http://www.sraoss.co.jp/index_en.php" target="_blank">http://www.sraoss.co.jp/index_en.php</a><br>
Japanese: <a href="http://www.sraoss.co.jp" target="_blank">http://www.sraoss.co.jp</a><br>
<div class="im"><br>
> Currently pgpool does not contains any local memory management API, it<br>
> relies on standard C Library memory allocation functions for acquiring and<br>
> releasing the memory. Absence of a memory manager especially<br>
><br>
> with the growth of source code and feature makes it vulnerable to serious<br>
> memory issues/leakages. A proper implementation of memory manager for<br>
> pgpool can also aid in debugging and could improve the overall<br>
><br>
> performance of the system. Postgres already contains the memory manager API<br>
> and this document proposes to adopt the memory management used by PG in<br>
> pgpool.<br>
><br>
><br>
</div>> *Some notes about the PG's memory management API.<br>
<div class="im">> *<br>
><br>
> The core of memory management in the Postgres is a MemoryContext and<br>
> Postgres do almost all memory allocation in the "memory contexts", which has<br>
> a defined lifecycle.<br>
><br>
</div>> *The basic operations on a memory context are:<br>
> ** create a context<br>
<div class="im">> * allocate a chunk of memory within a context (equivalent of standard C<br>
> library's malloc())<br>
> * delete a context (including freeing all the memory allocated therein)<br>
> * reset a context (free all memory allocated in the context, but not<br>
> the context object itself)<br>
><br>
> Given a chunk of memory previously allocated from a context, one can free<br>
> it or reallocate it larger or smaller (corresponding to standard library's<br>
> free() and realloc() routines).<br>
> These operations return memory to or get more memory from the same context<br>
> the chunk was originally allocated in.<br>
> At all times there is a "current" context denoted by<br>
> the CurrentMemoryContext global variable. The backend macro palloc()<br>
> implicitly allocates space in that context. The<br>
> MemoryContextSwitchTo() operation selects a new current context (and<br>
> returns the previous context, so that the caller can restore the previous<br>
> context before exiting).<br>
><br>
> The main advantage of memory contexts over plain use of malloc/free is that<br>
> the entire contents of a memory context can be freed easily, this saves the<br>
> trouble of freeing individual chunks of memory.<br>
> This is both faster and more reliable than per-chunk bookkeeping.<br>
> PG already use this fact to clean up at transaction end, The memory is<br>
> reclaimed by resetting all the active contexts.<br>
><br>
</div>> *Some Notes About the PG memory API(palloc) Versus Standard C Library*<br>
<div class="im">><br>
> The behaviour of palloc and friends is similar to the standard C<br>
> library's malloc and friends, but there are some deliberate differences<br>
> too. Here are some notes to clarify the behaviour.<br>
><br>
> * If out of memory, palloc and repalloc exit via elog(ERROR). They<br>
> never return NULL, and it is not necessary or useful to test for such a<br>
> result.<br>
><br>
> * palloc(0) is explicitly a valid operation. It does not return a<br>
> NULL pointer, but a valid chunk of which no bytes may be used. (However,<br>
> the chunk might later be repalloc'd larger; it can also be pfree'd<br>
> without error.) disallowed palloc(0). It seems more consistent to allow<br>
> it, however.) Similarly, repalloc allows realloc'ing to zero size.<br>
><br>
> * pfree and repalloc do not accept a NULL pointer.<br>
><br>
</div>> *Using the palloc API in pgpool*<br>
<div class="im">><br>
> As mentioned above, I propose to following the memory management approach<br>
> used in PG in PGPOOL. I am planning to do the following to hook the<br>
> Postgres memory management API in pgpool.<br>
><br>
><br>
> 1- Copy the the memory management source files from<br>
> PGSRC/src/backend/utils/mmgr/ (mcxt.c and aset.c) int the pgpool source.<br>
><br>
> 2- Change the behaviour in case of allocation errors, as currently pgpool<br>
> does not contains elog API.<br>
><br>
> 3- Create a single TopMemoryContext global memory context.<br>
><br>
> 4- Replace all malloc, free and related memory management functions calls<br>
> with palloc and friends.<br>
><br>
> 4- Stabilise the code.<br>
><br>
> 5- Make more memory contexts, inline with the lifecycle of components of<br>
> pgpool<br>
><br>
> 6- Stabilise the code.<br>
><br>
</div>> *References*<br>
> PGSRC/src/backend/utils/mmgr/README<br>
><br>
> Thanks<br>
> Muhammad Usama<br>
</blockquote></div><br></div></div>