[pgpool-hackers: 365] Re: [pgpool-II 0000015]: pgpool-II 3.1.3 on FreeBSD regex patterns for blacklist/whitelist report errors
Tatsuo Ishii
ishii at postgresql.org
Fri Sep 13 23:38:03 JST 2013
Done.
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese: http://www.sraoss.co.jp
> Hi pgpool hackers,
>
> We got a complain from a FreeBSD user regarding regexp module in
> pgpool. The root of the problem is regexp module in FreeBSD's
> "compatibility" library (-lcompat). It behaves differently from what
> pgpool expects. Also I think it's not worth the trouble to make
> pgpool's code to adopt the FreeBSD's old regexp. As stated in the
> message bottom from the user, I think now we can safely remove
> "-lcompat" from our configure.in because it's really an ancient
> requirement and now the platforms where pgpool is running on do not
> need any more. Please say no, if you think this is not a good idea.
> --
> Tatsuo Ishii
> SRA OSS, Inc. Japan
> English: http://www.sraoss.co.jp/index_en.php
> Japanese: http://www.sraoss.co.jp
>
>> A NOTE has been added to this issue.
>> ======================================================================
>> http://www.pgpool.net/mantisbt/view.php?id=15
>> ======================================================================
>> Reported By: spork
>> Assigned To: t-ishii
>> ======================================================================
>> Project: pgpool-II
>> Issue ID: 15
>> Category: Bug
>> Reproducibility: always
>> Severity: major
>> Priority: normal
>> Status: assigned
>> ======================================================================
>> Date Submitted: 2012-06-07 15:41 JST
>> Last Modified: 2013-09-12 10:32 JST
>> ======================================================================
>> Summary: pgpool-II 3.1.3 on FreeBSD regex patterns for
>> blacklist/whitelist report errors
>> Description:
>> Using the default config which is just a simple list of comma separated keywords
>> causes pgpool to throw errors and likely ignore the configuration settings.
>>
>> config statement in pgpool.conf:
>>
>> black_function_list = 'currval,lastval,nextval,setval'
>> # Comma separated list of function names
>> # that write to database
>> # Regexp are accepted
>>
>> log:
>>
>> Jun 7 01:20:22 devrep pgpool[52608]: add_to_patterns: invalid regex pattern:
>> ^currval$
>> Jun 7 01:20:22 devrep pgpool[52608]: add_to_patterns: invalid regex pattern:
>> ^lastval$
>> Jun 7 01:20:22 devrep pgpool[52608]: add_to_patterns: invalid regex pattern:
>> ^nextval$
>> Jun 7 01:20:22 devrep pgpool[52608]: add_to_patterns: invalid regex pattern:
>> ^setval$
>>
>> This is on a FreeBSD 8.2 box.
>>
>> I see a similar report from about a year ago:
>>
>> http://answerpot.com/showthread.php?3105812-Pgpool+3.1+errors
>>
>> Steps to Reproduce:
>> Setup a config with sample config file, including the "black_function_list"
>> statement.
>>
>> Start pgpool, watch log output.
>> ======================================================================
>>
>> ----------------------------------------------------------------------
>> (0000037) t-ishii (developer) - 2012-06-07 16:58
>> http://www.pgpool.net/mantisbt/view.php?id=15#c37
>> ----------------------------------------------------------------------
>> Works for me (Linux). Maybe FreeBSD specific issue? I don't have FreeBSD at this
>> point, and I cannot confirm now. Sorry.
>>
>> BTW, the link you are referring is not related to the problem you are facing,
>> I think. Probably the problem stated by the link has been fixed by:
>>
>> 3.1.1 (hatsuiboshi) 2011/12/06
>>
>> * Version 3.1.1
>>
>> This is a bugfix release against pgpool-II 3.1.
>>
>> * Bug fixes
>>
>> - Fix add_regex_pattern(). It does not allocate enough memory
>> for each black/white_function_list items. The function adds
>> "^" and "$" to each function items which do not contain those
>> characters. Unfortunately the function forgot to add extra 2
>> bytes for those characters. This may lead to memory corruption
>> errors when pgpool starting up.
>>
>> ----------------------------------------------------------------------
>> (0000091) spork (reporter) - 2012-09-25 13:48
>> http://www.pgpool.net/mantisbt/view.php?id=15#c91
>> ----------------------------------------------------------------------
>> Still present in 3.1.2.
>>
>> ----------------------------------------------------------------------
>> (0000104) t-ishii (developer) - 2012-10-14 12:22
>> http://www.pgpool.net/mantisbt/view.php?id=15#c104
>> ----------------------------------------------------------------------
>> I have tested pgpool-II 3.2.1 and 3.1.5 on FreeBSD 9.0 box. I don't see no
>> problem with black function list so far.
>>
>> ----------------------------------------------------------------------
>> (0000109) spork (reporter) - 2012-10-18 14:55
>> http://www.pgpool.net/mantisbt/view.php?id=15#c109
>> ----------------------------------------------------------------------
>> Still see it on 3.2.1 (although there are some build issues on FreeBSD in the
>> watchdog code). Maybe this debug info helps. This is FreeBSD 8.3.
>>
>> Oct 18 01:49:18 k16 pgpool: DEBUG: pid 6545: key: white_function_list
>> Oct 18 01:49:18 k16 pgpool: DEBUG: pid 6545: value: '' kind: 4
>> Oct 18 01:49:18 k16 pgpool: DEBUG: pid 6545: key: black_function_list
>> Oct 18 01:49:18 k16 pgpool: DEBUG: pid 6545: value:
>> 'currval,lastval,nextval,setval' kind: 4
>> Oct 18 01:49:18 k16 pgpool: DEBUG: pid 6545: extract_string_tokens: token:
>> currval
>> Oct 18 01:49:18 k16 pgpool: DEBUG: pid 6545: extract_string_tokens: token:
>> lastval
>> Oct 18 01:49:18 k16 pgpool: DEBUG: pid 6545: extract_string_tokens: token:
>> nextval
>> Oct 18 01:49:18 k16 pgpool: DEBUG: pid 6545: extract_string_tokens: token:
>> setval
>> Oct 18 01:49:18 k16 pgpool: DEBUG: pid 6545: add_to_patterns: regex pattern:
>> ^currval$
>> Oct 18 01:49:18 k16 pgpool: ERROR: pid 6545: add_to_patterns: invalid regex
>> pattern: ^currval$
>> Oct 18 01:49:18 k16 pgpool: DEBUG: pid 6545: add_to_patterns: regex pattern:
>> ^lastval$
>> Oct 18 01:49:18 k16 pgpool: ERROR: pid 6545: add_to_patterns: invalid regex
>> pattern: ^lastval$
>> Oct 18 01:49:18 k16 pgpool: DEBUG: pid 6545: add_to_patterns: regex pattern:
>> ^nextval$
>> Oct 18 01:49:18 k16 pgpool: ERROR: pid 6545: add_to_patterns: invalid regex
>> pattern: ^nextval$
>> Oct 18 01:49:18 k16 pgpool: DEBUG: pid 6545: add_to_patterns: regex pattern:
>> ^setval$
>> Oct 18 01:49:18 k16 pgpool: ERROR: pid 6545: add_to_patterns: invalid regex
>> pattern: ^setval$
>>
>> At least it shows the config parser is getting the values right. :)
>>
>> Just thought I'd add this. 3.2.1 doesn't quite work on FreeBSD 8.3 (can't
>> connect to any db hosts for the health checks), but I wanted to at least
>> document the regex issue.
>>
>> ----------------------------------------------------------------------
>> (0000110) t-ishii (developer) - 2012-10-18 15:15
>> http://www.pgpool.net/mantisbt/view.php?id=15#c110
>> ----------------------------------------------------------------------
>> I have fixed some FreeBSD build issues. Next time please grab 3.2-STABLE head.
>>
>> ----------------------------------------------------------------------
>> (0000114) spork (reporter) - 2012-10-20 10:48
>> http://www.pgpool.net/mantisbt/view.php?id=15#c114
>> ----------------------------------------------------------------------
>> Just downloaded, built and installed 3.2 head.
>>
>> Build issues are totally resolved, but the regex errors are still there:
>>
>> DEBUG: pid 18118: add_to_patterns: regex pattern: ^currval$
>> ERROR: pid 18118: add_to_patterns: invalid regex pattern: ^currval$
>> DEBUG: pid 18118: add_to_patterns: regex pattern: ^lastval$
>> ERROR: pid 18118: add_to_patterns: invalid regex pattern: ^lastval$
>> DEBUG: pid 18118: add_to_patterns: regex pattern: ^nextval$
>> ERROR: pid 18118: add_to_patterns: invalid regex pattern: ^nextval$
>> DEBUG: pid 18118: add_to_patterns: regex pattern: ^setval$
>> ERROR: pid 18118: add_to_patterns: invalid regex pattern: ^setval$
>>
>> ----------------------------------------------------------------------
>> (0000118) t-ishii (developer) - 2012-10-20 17:39
>> http://www.pgpool.net/mantisbt/view.php?id=15#c118
>> ----------------------------------------------------------------------
>> I confirmed that with FreeBSD 8.3 the problem reproduced. Strange thing is, nm
>> command against pgpool shows following result:
>> [t-ishii at freebsd /usr/local/src/pgpool/pgpool-II-3.2.1]$ nm ~/bin/pgpool |grep
>> regcomp
>> 0000000000476df0 T regcomp
>>
>> "T" means regcomp is a global symbol in pgpool. This should be like this:
>> [t-ishii at localhost tmp]$ nm a.out|grep regcomp
>> U regcomp@@GLIBC_2.2.5
>>
>> a.out is produced from test program:
>> #include <regex.h>
>> #include <stdio.h>
>> main()
>> {
>> regex_t regexv;
>> int rtn;
>> int regex_flags = REG_NOSUB;
>> regex_flags |= REG_ICASE;
>> regex_flags |= REG_EXTENDED;
>> rtn = regcomp(®exv, "^nextval$", regex_flags);
>> if (rtn != 0)
>> {
>> printf("invalid regex pattern %d\n", rtn);
>> }
>> }
>>
>> This program does exactly same thing in pgpool and works fine of courese. My
>> wild guess is, pgpool uses wrong regcomp module in FreeBSD 8.3. I don't know why
>> though.
>>
>> ----------------------------------------------------------------------
>> (0000313) spork (reporter) - 2013-08-28 15:54
>> http://www.pgpool.net/mantisbt/view.php?id=15#c313
>> ----------------------------------------------------------------------
>> Just noting this still exists. We've had to stop using pgpool for now, as we
>> have some queries that use "nextval()" and they get sent to the slave, which
>> causes problems for us:
>>
>> Aug 19 10:18:24 k29.i postgres[83999]: [17-1] db=xxx,user=xxx ERROR: cannot
>> execute nextval() in a read-only transaction
>> Aug 19 10:18:24 k29.i postgres[83999]: [17-2] db=xxx,user=xxx STATEMENT: select
>> nextval('signup_id_seq');
>>
>> I've tested again, now on FreeBSD 8.4 and pgpool 3.3.0. Same error message when
>> pgpool exits:
>>
>> DEBUG: pid 96388: key: black_function_list
>> DEBUG: pid 96388: value: 'currval,lastval,nextval,setval' kind: 4
>> DEBUG: pid 96388: extract_string_tokens: token: currval
>> DEBUG: pid 96388: extract_string_tokens: token: lastval
>> DEBUG: pid 96388: extract_string_tokens: token: nextval
>> DEBUG: pid 96388: extract_string_tokens: token: setval
>> DEBUG: pid 96388: add_to_patterns: regex pattern: ^currval$
>> ERROR: pid 96388: add_to_patterns: invalid regex pattern: ^currval$
>> DEBUG: pid 96388: add_to_patterns: regex pattern: ^lastval$
>> ERROR: pid 96388: add_to_patterns: invalid regex pattern: ^lastval$
>> DEBUG: pid 96388: add_to_patterns: regex pattern: ^nextval$
>> ERROR: pid 96388: add_to_patterns: invalid regex pattern: ^nextval$
>> DEBUG: pid 96388: add_to_patterns: regex pattern: ^setval$
>> ERROR: pid 96388: add_to_patterns: invalid regex pattern: ^setval$
>>
>> I also added a call to the error routine to see what the regex flags are. I
>> don't understand bitwise stuff at all (and barely understand C), so I don't know
>> if "7" represents the correct flags or not:
>>
>> DEBUG: pid 3484: add_to_patterns: regex pattern: ^currval$
>> ERROR: pid 3484: add_to_patterns: invalid regex pattern: ^currval$
>> ERROR: pid 3484: add_to_patterns: regex flags: 7
>> DEBUG: pid 3484: add_to_patterns: regex pattern: ^lastval$
>> ERROR: pid 3484: add_to_patterns: invalid regex pattern: ^lastval$
>> ERROR: pid 3484: add_to_patterns: regex flags: 7
>> DEBUG: pid 3484: add_to_patterns: regex pattern: ^nextval$
>> ERROR: pid 3484: add_to_patterns: invalid regex pattern: ^nextval$
>> ERROR: pid 3484: add_to_patterns: regex flags: 7
>> DEBUG: pid 3484: add_to_patterns: regex pattern: ^setval$
>> ERROR: pid 3484: add_to_patterns: invalid regex pattern: ^setval$
>> ERROR: pid 3484: add_to_patterns: regex flags: 7
>>
>> What else can I give you to check this out? Is "7" the correct flag to bitwise
>> combine 0001 0002 and 0004?
>>
>> Putting another debug in there shows me "errno 2" which maps to "invalid regex".
>>
>> I'm stumped.
>>
>> Your test program above works as expected.
>>
>> ----------------------------------------------------------------------
>> (0000314) t-ishii (developer) - 2013-08-28 18:54
>> http://www.pgpool.net/mantisbt/view.php?id=15#c314
>> ----------------------------------------------------------------------
>> I have no idea how to debug further. As I said before, I think the problem must
>> be in the compiler tool chain, including gcc and linker on FreeBSD, not the
>> pgpool source itself. To verify this, you can do:
>>
>> - cd to pgpool source code
>> - compile
>> - see "nm pool_config.o|grep regcomp". If it's not something like:
>> U regcomp
>> You are in trouble.
>>
>> ----------------------------------------------------------------------
>> (0000315) spork (reporter) - 2013-08-29 00:38
>> http://www.pgpool.net/mantisbt/view.php?id=15#c315
>> ----------------------------------------------------------------------
>> [root at client1 /usr/local/src/pgpool-II-3.3.0]# nm pool_config.o |grep regcomp
>> U regcomp
>> [root at client1 /usr/local/src/pgpool-II-3.3.0]#
>>
>> ----------------------------------------------------------------------
>> (0000316) t-ishii (developer) - 2013-08-29 10:47
>> http://www.pgpool.net/mantisbt/view.php?id=15#c316
>> ----------------------------------------------------------------------
>> It turned out if we remove the linking of compat library, everything goes fine.
>> I did it by hand:
>> gcc -g -O2 -Wall -Wmissing-prototypes -Wmissing-declarations -o .libs/pgpool
>> main.o child.o pool_auth.o pool_config.o pool_error.o pool_process_query.o
>> pool_stream.o pool_connection_pool.o pool_params.o pool_signal.o pcp_child.o
>> md5.o pool_shmem.o pool_sema.o pool_system.o pool_rewrite_query.o
>> pool_rewrite_outfuncs.o pool_query_cache.o pool_hba.o pool_path.o pool_ip.o
>> ps_status.o strlcpy.o recovery.o pool_relcache.o pool_process_reporting.o
>> pool_ssl.o pool_timestamp.o pool_proto2.o pool_proto_modules.o pool_lobj.o
>> pool_process_context.o pool_memqcache.o pool_session_context.o
>> pool_query_context.o pool_worker_child.o pool_passwd.o pool_globals.o
>> pool_select_walker.o getopt_long.o parser/nodes.o -L/usr/local/pgsql/lib -lpq
>> parser/libsql-parser.a pcp/.libs/libpcp.so watchdog/lib-watchdog.a -lpthread
>> -lcrypt -lm -Wl,-rpath -Wl,/usr/local/lib -Wl,-rpath -Wl,/usr/local/pgsql/lib
>>
>> However, I don't know how to tell autoconf do that.
>>
>> ----------------------------------------------------------------------
>> (0000317) spork (reporter) - 2013-08-29 12:12
>> http://www.pgpool.net/mantisbt/view.php?id=15#c317
>> ----------------------------------------------------------------------
>> Just confirmed here that if I pull "-lcompat" from all Makefiles I get a clean
>> build and the error goes away.
>>
>> I have no idea how the whole autotools suite works though. I'm also not really
>> sure what the compat lib is for, trying to read up on it a bit. Thanks!
>>
>> ----------------------------------------------------------------------
>> (0000318) t-ishii (developer) - 2013-08-29 14:16
>> http://www.pgpool.net/mantisbt/view.php?id=15#c318
>> ----------------------------------------------------------------------
>> It seems "-lcompat" is in configure.in. Attached patch is just removing it.
>> Unfortunately I couldn't get neccessary tool chains such as autoconf to test it
>> put on my FreeBSD box. Can you please try it out?
>>
>> ----------------------------------------------------------------------
>> (0000327) spork (reporter) - 2013-09-12 10:32
>> http://www.pgpool.net/mantisbt/view.php?id=15#c327
>> ----------------------------------------------------------------------
>> I missed your last reply... I can confirm that removing the compat check from
>> configure.in results in a working build.
>>
>> I also think I sort of understand why the compat libs break things. Looking at
>> the source, I'm guessing this is for really, really old backward compatibility
>> and it appears to include it's own regex functions, which I imagine is how we
>> end up with the errors - the compat regex functions probably take arguments in a
>> different order or something. Does that sound about right?
>>
>> [root at client1 /usr/local/src/pgpool-II-3.3.0]# ls -R /usr/src/lib/libcompat/
>> 4.1 4.3 4.4 Makefile regexp
>>
>> /usr/src/lib/libcompat/4.1:
>> ascftime.c cftime.c ftime.c getpw.c
>> cftime.3 ftime.3 getpw.3
>>
>> /usr/src/lib/libcompat/4.3:
>> cfree.3 re_comp.3 rexec.3
>> cfree.c regex.c rexec.c
>>
>> /usr/src/lib/libcompat/4.4:
>> cuserid.3 cuserid.c
>>
>> /usr/src/lib/libcompat/regexp:
>> COPYRIGHT regerror.c regexp.c regsub.c
>> README regexp.3 regmagic.h
>>
>> Issue History
>> Date Modified Username Field Change
>> ======================================================================
>> 2012-06-07 15:41 spork New Issue
>> 2012-06-07 16:58 t-ishii Note Added: 0000037
>> 2012-09-25 13:48 spork Note Added: 0000091
>> 2012-10-14 12:21 t-ishii Assigned To => t-ishii
>> 2012-10-14 12:21 t-ishii Status new => assigned
>> 2012-10-14 12:22 t-ishii Note Added: 0000104
>> 2012-10-18 14:55 spork Note Added: 0000109
>> 2012-10-18 15:15 t-ishii Note Added: 0000110
>> 2012-10-20 10:48 spork Note Added: 0000114
>> 2012-10-20 17:39 t-ishii Note Added: 0000118
>> 2013-08-28 15:54 spork Note Added: 0000313
>> 2013-08-28 18:54 t-ishii Note Added: 0000314
>> 2013-08-29 00:38 spork Note Added: 0000315
>> 2013-08-29 10:47 t-ishii Note Added: 0000316
>> 2013-08-29 12:12 spork Note Added: 0000317
>> 2013-08-29 14:14 t-ishii File Added: configure.in.patch
>>
>> 2013-08-29 14:16 t-ishii Note Added: 0000318
>> 2013-09-12 10:32 spork Note Added: 0000327
>> ======================================================================
>>
> _______________________________________________
> pgpool-hackers mailing list
> pgpool-hackers at pgpool.net
> http://www.pgpool.net/mailman/listinfo/pgpool-hackers
More information about the pgpool-hackers
mailing list