[pgpool-hackers: 296] Re: pgpool-II testing framework

[Tatsuo Ishii]

I have added the regression test suit to git master.

$ sh regress.sh
testing 001.load_balance...ok.
testing 002.native_replication...ok.
testing 003.failover...ok.
testing 050.bug58...ok.
testing 051.bug60...ok.
testing 052.do_query...ok.
testing 053.insert_lock_hangs...ok.
out of 7 ok:7 failed:0

Apparently we need more tests but I hope this is a resonable starting
point. Here are excerptions from test/regress/README.

--------------------------------------------------------------------------
This directory contains files for simple regression tests.
To execute regress.sh you need to install:

- pgpool_setup
- PostgreSQL 8.3 or later (requires EXTENSION)
- pgbench
- perl PostgreSQL interface
- other tools or libraries required in each test cases

regress.sh: the regression test driver. To execute the test, type "sh regression.sh".

clean.sh: remove files including PostgreSQL DB clusters created by regress.sh.

tests: test scripts live here. Each test case has its own
	   subdirectory. In the directory the test script resides and the
	   script name must be "test.sh". The script should return exit
	   code 0 upon successfull completion of the test. Otherwise it
	   should return non 0 value. To add a new test case, simply
	   create a shell script under this.

	   The test name "001.*" to "050.*" is for regular regression
       tests, while after "051.*" are for bugs.

log: any output of the test files above is created under the
	  directory. The log file name is same as the test name.

libs.sh: contains some useful bash functions. The file name is
		 exported as "$TESTLIBS",
--------------------------------------------------------------------------

> Hi,
> 
> It's been a long time TODO item to create a general testing framework
> for pgpool-II. We already have certain test tools and they live in the
> "test" directory. However, they are for limited test cases and far
> from a general one. Here are requirements for the general testing
> framework from my point of view:
> 
> 1) must be able to test not only ordinary queries, but full
>    functionality including load balance, replication, failover and
>    query cache.
> 
> 2) the should be able to be performed against pgpool server and
>    multiple PostgreSQL database clusters.
> 
> Here is my proposal aiming for the general testing framework. The
> script is a set of simple bash scripts. At this point I created one
> function test (load balancing in both streaming replication mode and
> native replication mode) and some bugs fixed recently. In each test,
> fresh pgpool and PostgreSQL installation are created by using
> pgpool_setup.
> 
> $ sh regress.sh 
> testing 01.load_balance...ok.
> testing bug58...ok.
> testing bug60...ok.
> testing do_query...ok.
> testing insert_lock_hangs...ok.
> out of 5 ok:5 failed:0
> 
> Obviously there are not complete test cases but it is not terribly
> hard to add necessary test cases by using the framework.
> 
> Also I think it would be nice to execute the test every night like
> PostgreSQL's buildfarm.
> 
> Comments, suggestions are welcome.
> --
> Tatsuo Ishii
> SRA OSS, Inc. Japan
> English: http://www.sraoss.co.jp/index_en.php
> Japanese: http://www.sraoss.co.jp