[pgpool-general: 8515] Re: Segfaults on PgPool-II 4.3.2

[Jordan Hurwich]

Thanks Tatsuo, we've got the cluster running with `enable_shared_relcache =
off` now and will report back if we have any more issues (we'll also take a
look at updating our conf for 4.3, thanks for pointing that out)

On Thu, Dec 8, 2022 at 5:23 PM Tatsuo Ishii <ishii at sraoss.co.jp> wrote:

> > Hello PgPool Team,
> >
> > We've been experiencing frequent segfaults from PgPool-II 4.3.2,
> sometimes
> > as frequently as every 30min, when the cluster is under significant
> (though
> > normal for us) load. We've been using pgpool without issue for a while,
> and
> > these issues began occurring after we launched a cluster of all new
> > machines with no changes on the pgpool machines, and updated
> shared_buffers
> > (from "8000MB" to "14000MB" on machines with 32GB RAM) and decreased
> > swappiness on the PSQL nodes (vm.swappiness=1 to vm.swappiness=0 in
> > sysctl.conf). Following the segfaults the cluster appears to continue
> > behaving normally, though it's still a bit early for us to know for sure.
> >
> > Details follow including core dump backtraces from gdb for two segfaults
> > we've observed recently (one in memcpy, another in
> > pool_find_item_on_shmem_cache), including `bt` and `bt full`. Please let
> us
> > know if there is any more information we can provide.
> >
> > Thanks,
> > Jordan
> >
> > Environment details (ubuntu-focal-20.04-amd64):
> >
> >    - Cluster contains 3 PgPool nodes and 3 PSQL nodes, all on AWS EC2
> >    instances, pgpool.conf attached. Segfaults occurred on the leader at
> the
> >    time.
> >    - $ uname -a
> >    Linux ip-172-30-64-83 5.4.0-1038-aws #40-Ubuntu SMP Fri Feb 5 23:50:40
> >    UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
> >    - # SHOW POOL_VERSION;
> >         pool_version
> >    -----------------------
> >     4.3.2 (tamahomeboshi)
> >    (1 row)
> >    - In both cases the log line for segfault is like:
> >    pgpool[45529]: segfault at 7ff9cfa1094e ip 000056215c2d53fe sp
> >    00007ffe3e7db160 error 6 in pgpool[56215c28c000+d9000]
> >    - Backtraces attached:
> >    - segfault_memcpy_202212208_1500
> >       - segfault_pool_find_item_on_shmem_cache_20221208_1500
>
> The pgpool.conf is not for 4.3.x, though it's not directly related to
> the issue, I think.
>
> It seems all the segfaults are related to shared relcache, which was
> introduced in 4.3. Can you disable the feature and see if the segfaults
> still occur?
> You can do it by adding:
>
> enable_shared_relcache = off
>
> Best reagards,
> --
> Tatsuo Ishii
> SRA OSS LLC
> English: http://www.sraoss.co.jp/index_en/
> Japanese:http://www.sraoss.co.jp
> _______________________________________________
> pgpool-general mailing list
> pgpool-general at pgpool.net
> http://www.pgpool.net/mailman/listinfo/pgpool-general
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.pgpool.net/pipermail/pgpool-general/attachments/20221209/7b2650d9/attachment.htm>