[pgpool-hackers: 364] Re: [pgpool-II 0000015]: pgpool-II 3.1.3 on FreeBSD regex patterns for blacklist/whitelist report errors

Tatsuo Ishii ishii at postgresql.org
Thu Sep 12 13:28:44 JST 2013


Hi pgpool hackers,

We got a complain from a FreeBSD user regarding regexp module in
pgpool. The root of the problem is regexp module in FreeBSD's
"compatibility" library (-lcompat). It behaves differently from what
pgpool expects. Also I think it's not worth the trouble to make
pgpool's code to adopt the FreeBSD's old regexp. As stated in the
message bottom from the user, I think now we can safely remove
"-lcompat" from our configure.in because it's really an ancient
requirement and now the platforms where pgpool is running on do not
need any more. Please say no, if you think this is not a good idea.
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese: http://www.sraoss.co.jp

> A NOTE has been added to this issue. 
> ====================================================================== 
> http://www.pgpool.net/mantisbt/view.php?id=15 
> ====================================================================== 
> Reported By:                spork
> Assigned To:                t-ishii
> ====================================================================== 
> Project:                    pgpool-II
> Issue ID:                   15
> Category:                   Bug
> Reproducibility:            always
> Severity:                   major
> Priority:                   normal
> Status:                     assigned
> ====================================================================== 
> Date Submitted:             2012-06-07 15:41 JST
> Last Modified:              2013-09-12 10:32 JST
> ====================================================================== 
> Summary:                    pgpool-II 3.1.3 on FreeBSD regex patterns for
> blacklist/whitelist report errors
> Description: 
> Using the default config which is just a simple list of comma separated keywords
> causes pgpool to throw errors and likely ignore the configuration settings.
> 
> config statement in pgpool.conf:
> 
> black_function_list = 'currval,lastval,nextval,setval'
>                                    # Comma separated list of function names
>                                    # that write to database
>                                    # Regexp are accepted
> 
> log:
> 
> Jun  7 01:20:22 devrep pgpool[52608]: add_to_patterns: invalid regex pattern:
> ^currval$
> Jun  7 01:20:22 devrep pgpool[52608]: add_to_patterns: invalid regex pattern:
> ^lastval$
> Jun  7 01:20:22 devrep pgpool[52608]: add_to_patterns: invalid regex pattern:
> ^nextval$
> Jun  7 01:20:22 devrep pgpool[52608]: add_to_patterns: invalid regex pattern:
> ^setval$
> 
> This is on a FreeBSD 8.2 box.
> 
> I see a similar report from about a year ago:
> 
> http://answerpot.com/showthread.php?3105812-Pgpool+3.1+errors
> 
> Steps to Reproduce: 
> Setup a config with sample config file, including the "black_function_list"
> statement.
> 
> Start pgpool, watch log output.
> ====================================================================== 
> 
> ---------------------------------------------------------------------- 
>  (0000037) t-ishii (developer) - 2012-06-07 16:58
>  http://www.pgpool.net/mantisbt/view.php?id=15#c37 
> ---------------------------------------------------------------------- 
> Works for me (Linux). Maybe FreeBSD specific issue? I don't have FreeBSD at this
> point, and I cannot confirm now. Sorry.
> 
> BTW, the link you are referring is not related to the problem you are facing,
> I think. Probably the problem stated by the link has been fixed by:
> 
> 3.1.1 (hatsuiboshi) 2011/12/06
> 
> 	  * Version 3.1.1
> 
> 	  This is a bugfix release against pgpool-II 3.1.
> 
> 	  * Bug fixes
> 
> 	  - Fix add_regex_pattern(). It does not allocate enough memory
> 	   	for each black/white_function_list items. The function adds
> 	   	"^" and "$" to each function items which do not contain those
> 	   	characters. Unfortunately the function forgot to add extra 2
> 	   	bytes for those characters. This may lead to memory corruption
> 	   	errors when pgpool starting up. 
> 
> ---------------------------------------------------------------------- 
>  (0000091) spork (reporter) - 2012-09-25 13:48
>  http://www.pgpool.net/mantisbt/view.php?id=15#c91 
> ---------------------------------------------------------------------- 
> Still present in 3.1.2. 
> 
> ---------------------------------------------------------------------- 
>  (0000104) t-ishii (developer) - 2012-10-14 12:22
>  http://www.pgpool.net/mantisbt/view.php?id=15#c104 
> ---------------------------------------------------------------------- 
> I have tested pgpool-II 3.2.1 and 3.1.5 on FreeBSD 9.0 box. I don't see no
> problem with black function list so far. 
> 
> ---------------------------------------------------------------------- 
>  (0000109) spork (reporter) - 2012-10-18 14:55
>  http://www.pgpool.net/mantisbt/view.php?id=15#c109 
> ---------------------------------------------------------------------- 
> Still see it on 3.2.1 (although there are some build issues on FreeBSD in the
> watchdog code).  Maybe this debug info helps.  This is FreeBSD 8.3.
> 
> Oct 18 01:49:18 k16 pgpool: DEBUG: pid 6545: key: white_function_list
> Oct 18 01:49:18 k16 pgpool: DEBUG: pid 6545: value: '' kind: 4
> Oct 18 01:49:18 k16 pgpool: DEBUG: pid 6545: key: black_function_list
> Oct 18 01:49:18 k16 pgpool: DEBUG: pid 6545: value:
> 'currval,lastval,nextval,setval' kind: 4
> Oct 18 01:49:18 k16 pgpool: DEBUG: pid 6545: extract_string_tokens: token:
> currval
> Oct 18 01:49:18 k16 pgpool: DEBUG: pid 6545: extract_string_tokens: token:
> lastval
> Oct 18 01:49:18 k16 pgpool: DEBUG: pid 6545: extract_string_tokens: token:
> nextval
> Oct 18 01:49:18 k16 pgpool: DEBUG: pid 6545: extract_string_tokens: token:
> setval
> Oct 18 01:49:18 k16 pgpool: DEBUG: pid 6545: add_to_patterns: regex pattern:
> ^currval$
> Oct 18 01:49:18 k16 pgpool: ERROR: pid 6545: add_to_patterns: invalid regex
> pattern: ^currval$
> Oct 18 01:49:18 k16 pgpool: DEBUG: pid 6545: add_to_patterns: regex pattern:
> ^lastval$
> Oct 18 01:49:18 k16 pgpool: ERROR: pid 6545: add_to_patterns: invalid regex
> pattern: ^lastval$
> Oct 18 01:49:18 k16 pgpool: DEBUG: pid 6545: add_to_patterns: regex pattern:
> ^nextval$
> Oct 18 01:49:18 k16 pgpool: ERROR: pid 6545: add_to_patterns: invalid regex
> pattern: ^nextval$
> Oct 18 01:49:18 k16 pgpool: DEBUG: pid 6545: add_to_patterns: regex pattern:
> ^setval$
> Oct 18 01:49:18 k16 pgpool: ERROR: pid 6545: add_to_patterns: invalid regex
> pattern: ^setval$
> 
> At least it shows the config parser is getting the values right. :)
> 
> Just thought I'd add this.  3.2.1 doesn't quite work on FreeBSD 8.3 (can't
> connect to any db hosts for the health checks), but I wanted to at least
> document the regex issue. 
> 
> ---------------------------------------------------------------------- 
>  (0000110) t-ishii (developer) - 2012-10-18 15:15
>  http://www.pgpool.net/mantisbt/view.php?id=15#c110 
> ---------------------------------------------------------------------- 
> I have fixed some FreeBSD build issues. Next time please grab 3.2-STABLE head. 
> 
> ---------------------------------------------------------------------- 
>  (0000114) spork (reporter) - 2012-10-20 10:48
>  http://www.pgpool.net/mantisbt/view.php?id=15#c114 
> ---------------------------------------------------------------------- 
> Just downloaded, built and installed 3.2 head.
> 
> Build issues are totally resolved, but the regex errors are still there:
> 
> DEBUG: pid 18118: add_to_patterns: regex pattern: ^currval$
> ERROR: pid 18118: add_to_patterns: invalid regex pattern: ^currval$
> DEBUG: pid 18118: add_to_patterns: regex pattern: ^lastval$
> ERROR: pid 18118: add_to_patterns: invalid regex pattern: ^lastval$
> DEBUG: pid 18118: add_to_patterns: regex pattern: ^nextval$
> ERROR: pid 18118: add_to_patterns: invalid regex pattern: ^nextval$
> DEBUG: pid 18118: add_to_patterns: regex pattern: ^setval$
> ERROR: pid 18118: add_to_patterns: invalid regex pattern: ^setval$ 
> 
> ---------------------------------------------------------------------- 
>  (0000118) t-ishii (developer) - 2012-10-20 17:39
>  http://www.pgpool.net/mantisbt/view.php?id=15#c118 
> ---------------------------------------------------------------------- 
> I confirmed that with FreeBSD 8.3 the problem reproduced. Strange thing is, nm
> command against pgpool shows following result:
> [t-ishii at freebsd /usr/local/src/pgpool/pgpool-II-3.2.1]$ nm ~/bin/pgpool |grep
> regcomp
> 0000000000476df0 T regcomp
> 
> "T" means regcomp is a global symbol in pgpool. This should be like this:
> [t-ishii at localhost tmp]$ nm a.out|grep regcomp
>                  U regcomp@@GLIBC_2.2.5
> 
> a.out is produced from test program:
> #include <regex.h>
> #include <stdio.h>
> main()
> {
>         regex_t regexv;
>         int rtn;
>         int regex_flags = REG_NOSUB;
>         regex_flags |= REG_ICASE;
>         regex_flags |= REG_EXTENDED;
>         rtn = regcomp(&regexv, "^nextval$", regex_flags);
>         if (rtn != 0)
>         {
>                 printf("invalid regex pattern %d\n", rtn);
>         }
> }
> 
> This program does exactly same thing in pgpool and works fine of courese. My
> wild guess is, pgpool uses wrong regcomp module in FreeBSD 8.3. I don't know why
> though. 
> 
> ---------------------------------------------------------------------- 
>  (0000313) spork (reporter) - 2013-08-28 15:54
>  http://www.pgpool.net/mantisbt/view.php?id=15#c313 
> ---------------------------------------------------------------------- 
> Just noting this still exists.  We've had to stop using pgpool for now, as we
> have some queries that use "nextval()" and they get sent to the slave, which
> causes problems for us:
> 
> Aug 19 10:18:24 k29.i postgres[83999]: [17-1] db=xxx,user=xxx ERROR:  cannot
> execute nextval() in a read-only transaction
> Aug 19 10:18:24 k29.i postgres[83999]: [17-2] db=xxx,user=xxx STATEMENT:  select
> nextval('signup_id_seq');
> 
> I've tested again, now on FreeBSD 8.4 and pgpool 3.3.0.  Same error message when
> pgpool exits:
> 
> DEBUG: pid 96388: key: black_function_list
> DEBUG: pid 96388: value: 'currval,lastval,nextval,setval' kind: 4
> DEBUG: pid 96388: extract_string_tokens: token: currval
> DEBUG: pid 96388: extract_string_tokens: token: lastval
> DEBUG: pid 96388: extract_string_tokens: token: nextval
> DEBUG: pid 96388: extract_string_tokens: token: setval
> DEBUG: pid 96388: add_to_patterns: regex pattern: ^currval$
> ERROR: pid 96388: add_to_patterns: invalid regex pattern: ^currval$
> DEBUG: pid 96388: add_to_patterns: regex pattern: ^lastval$
> ERROR: pid 96388: add_to_patterns: invalid regex pattern: ^lastval$
> DEBUG: pid 96388: add_to_patterns: regex pattern: ^nextval$
> ERROR: pid 96388: add_to_patterns: invalid regex pattern: ^nextval$
> DEBUG: pid 96388: add_to_patterns: regex pattern: ^setval$
> ERROR: pid 96388: add_to_patterns: invalid regex pattern: ^setval$
> 
> I also added a call to the error routine to see what the regex flags are.  I
> don't understand bitwise stuff at all (and barely understand C), so I don't know
> if "7" represents the correct flags or not:
> 
> DEBUG: pid 3484: add_to_patterns: regex pattern: ^currval$
> ERROR: pid 3484: add_to_patterns: invalid regex pattern: ^currval$
> ERROR: pid 3484: add_to_patterns: regex flags: 7
> DEBUG: pid 3484: add_to_patterns: regex pattern: ^lastval$
> ERROR: pid 3484: add_to_patterns: invalid regex pattern: ^lastval$
> ERROR: pid 3484: add_to_patterns: regex flags: 7
> DEBUG: pid 3484: add_to_patterns: regex pattern: ^nextval$
> ERROR: pid 3484: add_to_patterns: invalid regex pattern: ^nextval$
> ERROR: pid 3484: add_to_patterns: regex flags: 7
> DEBUG: pid 3484: add_to_patterns: regex pattern: ^setval$
> ERROR: pid 3484: add_to_patterns: invalid regex pattern: ^setval$
> ERROR: pid 3484: add_to_patterns: regex flags: 7
> 
> What else can I give you to check this out?  Is "7" the correct flag to bitwise
> combine 0001 0002 and 0004?
> 
> Putting another debug in there shows me "errno 2" which maps to "invalid regex".
> 
> I'm stumped.  
> 
> Your test program above works as expected. 
> 
> ---------------------------------------------------------------------- 
>  (0000314) t-ishii (developer) - 2013-08-28 18:54
>  http://www.pgpool.net/mantisbt/view.php?id=15#c314 
> ---------------------------------------------------------------------- 
> I have no idea how to debug further. As I said before, I think the problem must
> be in the compiler tool chain, including gcc and linker on FreeBSD, not the
> pgpool source itself. To verify this, you can do:
> 
> - cd to pgpool source code
> - compile
> - see "nm pool_config.o|grep regcomp". If it's not something like:
>                  U regcomp
> You are in trouble. 
> 
> ---------------------------------------------------------------------- 
>  (0000315) spork (reporter) - 2013-08-29 00:38
>  http://www.pgpool.net/mantisbt/view.php?id=15#c315 
> ---------------------------------------------------------------------- 
> [root at client1 /usr/local/src/pgpool-II-3.3.0]# nm pool_config.o |grep regcomp
>          U regcomp
> [root at client1 /usr/local/src/pgpool-II-3.3.0]# 
> 
> ---------------------------------------------------------------------- 
>  (0000316) t-ishii (developer) - 2013-08-29 10:47
>  http://www.pgpool.net/mantisbt/view.php?id=15#c316 
> ---------------------------------------------------------------------- 
> It turned out if we remove the linking of compat library, everything goes fine.
> I did it by hand:
> gcc -g -O2 -Wall -Wmissing-prototypes -Wmissing-declarations -o .libs/pgpool
> main.o child.o pool_auth.o pool_config.o pool_error.o pool_process_query.o
> pool_stream.o pool_connection_pool.o pool_params.o pool_signal.o pcp_child.o
> md5.o pool_shmem.o pool_sema.o pool_system.o pool_rewrite_query.o
> pool_rewrite_outfuncs.o pool_query_cache.o pool_hba.o pool_path.o pool_ip.o
> ps_status.o strlcpy.o recovery.o pool_relcache.o pool_process_reporting.o
> pool_ssl.o pool_timestamp.o pool_proto2.o pool_proto_modules.o pool_lobj.o
> pool_process_context.o pool_memqcache.o pool_session_context.o
> pool_query_context.o pool_worker_child.o pool_passwd.o pool_globals.o
> pool_select_walker.o getopt_long.o parser/nodes.o  -L/usr/local/pgsql/lib -lpq
> parser/libsql-parser.a pcp/.libs/libpcp.so watchdog/lib-watchdog.a -lpthread
> -lcrypt -lm -Wl,-rpath -Wl,/usr/local/lib -Wl,-rpath -Wl,/usr/local/pgsql/lib
> 
> However, I don't know how to tell autoconf do that. 
> 
> ---------------------------------------------------------------------- 
>  (0000317) spork (reporter) - 2013-08-29 12:12
>  http://www.pgpool.net/mantisbt/view.php?id=15#c317 
> ---------------------------------------------------------------------- 
> Just confirmed here that if I pull "-lcompat" from all Makefiles I get a clean
> build and the error goes away.
> 
> I have no idea how the whole autotools suite works though.  I'm also not really
> sure what the compat lib is for, trying to read up on it a bit.  Thanks! 
> 
> ---------------------------------------------------------------------- 
>  (0000318) t-ishii (developer) - 2013-08-29 14:16
>  http://www.pgpool.net/mantisbt/view.php?id=15#c318 
> ---------------------------------------------------------------------- 
> It seems "-lcompat" is in configure.in. Attached patch is just removing it.
> Unfortunately I couldn't get neccessary tool chains such as autoconf to test it
> put on my FreeBSD box. Can you please try it out? 
> 
> ---------------------------------------------------------------------- 
>  (0000327) spork (reporter) - 2013-09-12 10:32
>  http://www.pgpool.net/mantisbt/view.php?id=15#c327 
> ---------------------------------------------------------------------- 
> I missed your last reply...  I can confirm that removing the compat check from
> configure.in results in a working build.
> 
> I also think I sort of understand why the compat libs break things.  Looking at
> the source, I'm guessing this is for really, really old backward compatibility
> and it appears to include it's own regex functions, which I imagine is how we
> end up with the errors - the compat regex functions probably take arguments in a
> different order or something.  Does that sound about right?
> 
> [root at client1 /usr/local/src/pgpool-II-3.3.0]# ls -R /usr/src/lib/libcompat/
> 4.1		4.3		4.4		Makefile	regexp
> 
> /usr/src/lib/libcompat/4.1:
> ascftime.c	cftime.c	ftime.c		getpw.c
> cftime.3	ftime.3		getpw.3
> 
> /usr/src/lib/libcompat/4.3:
> cfree.3		re_comp.3	rexec.3
> cfree.c		regex.c		rexec.c
> 
> /usr/src/lib/libcompat/4.4:
> cuserid.3	cuserid.c
> 
> /usr/src/lib/libcompat/regexp:
> COPYRIGHT	regerror.c	regexp.c	regsub.c
> README		regexp.3	regmagic.h 
> 
> Issue History 
> Date Modified    Username       Field                    Change               
> ====================================================================== 
> 2012-06-07 15:41 spork          New Issue                                    
> 2012-06-07 16:58 t-ishii        Note Added: 0000037                          
> 2012-09-25 13:48 spork          Note Added: 0000091                          
> 2012-10-14 12:21 t-ishii        Assigned To               => t-ishii         
> 2012-10-14 12:21 t-ishii        Status                   new => assigned     
> 2012-10-14 12:22 t-ishii        Note Added: 0000104                          
> 2012-10-18 14:55 spork          Note Added: 0000109                          
> 2012-10-18 15:15 t-ishii        Note Added: 0000110                          
> 2012-10-20 10:48 spork          Note Added: 0000114                          
> 2012-10-20 17:39 t-ishii        Note Added: 0000118                          
> 2013-08-28 15:54 spork          Note Added: 0000313                          
> 2013-08-28 18:54 t-ishii        Note Added: 0000314                          
> 2013-08-29 00:38 spork          Note Added: 0000315                          
> 2013-08-29 10:47 t-ishii        Note Added: 0000316                          
> 2013-08-29 12:12 spork          Note Added: 0000317                          
> 2013-08-29 14:14 t-ishii        File Added: configure.in.patch                  
>  
> 2013-08-29 14:16 t-ishii        Note Added: 0000318                          
> 2013-09-12 10:32 spork          Note Added: 0000327                          
> ======================================================================
> 


More information about the pgpool-hackers mailing list