<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Hello.<div><br></div><div>I am testing pgpool for high availability. I succeeded in setting up pgpool using the packages that come with current Debian (v 3.1), and streaming replication, and it worked well in the testing environment. </div><div><br></div><div>However, now I want to set up two pgpools that use watchdog to connect to each other. To do this I upgraded the Debian packages to v. 3.2, and everything seemed OK - until I decided to test server failover again. That is, the watchdog setup works, but streaming replication with pgpool fails.<br></div><div><br></div><div>I then disabled the watchdog and tried to do what succeeded in version 3.1 - just control two servers with one pgpool and no watchdog. The test I'm trying to do is bring pgpool up when one of the servers is down, and the other is active. My goal is for pgpool to see that it has one active server, and then to allow me to do a base backup from the live server to the "dead" server and start replicating again.</div><div><br></div><div>However, the main pgpool process dies immediately as I bring it up. All its subprocesses become attached to process #1.</div><div><br></div><div>This is what I see in the log (I set debug_level to 5):</div><div><br></div><div><div><font class="Apple-style-span" color="#602185">Apr 17 13:00:27 pool01 pgpool: 2013-04-17 13:00:27 DEBUG: pid 14023: pool_initialize_private_backend_status: initialize backend status</font></div><div><font class="Apple-style-span" color="#602185">Apr 17 13:00:27 pool01 pgpool: 2013-04-17 13:00:27 LOG: pid 13983: pgpool-II successfully started. version 3.2.2 (namameboshi)</font></div><div><font class="Apple-style-span" color="#602185">Apr 17 13:00:27 pool01 pgpool: 2013-04-17 13:00:27 DEBUG: pid 14025: I am 14025</font></div><div><font class="Apple-style-span" color="#602185">Apr 17 13:00:27 pool01 pgpool: 2013-04-17 13:00:27 DEBUG: pid 14025: pool_initialize_private_backend_status: initialize backend status</font></div><div><font class="Apple-style-span" color="#602185">Apr 17 13:00:27 pool01 pgpool: 2013-04-17 13:00:27 DEBUG: pid 13983: pool_ssl: SSL requested but SSL support is not available</font></div><div><font class="Apple-style-span" color="#602185">Apr 17 13:00:27 pool01 pgpool: 2013-04-17 13:00:27 ERROR: pid 13983: pool_flush_it: write failed to backend (0). reason: Connection refused offset: 0 wlen: 41</font></div><div><font class="Apple-style-span" color="#602185">Apr 17 13:00:27 pool01 pgpool: 2013-04-17 13:00:27 LOG: pid 13983: degenerate_backend_set: 0 fail over request from pid 13983</font></div><div><font class="Apple-style-span" color="#602185">Apr 17 13:00:27 pool01 pgpool: 2013-04-17 13:00:27 DEBUG: pid 14025: pool_ssl: SSL requested but SSL support is not available</font></div><div><font class="Apple-style-span" color="#602185">Apr 17 13:00:27 pool01 pgpool: 2013-04-17 13:00:27 ERROR: pid 14025: pool_flush_it: write failed to backend (0). reason: Connection refused offset: 0 wlen: 41</font></div><div><font class="Apple-style-span" color="#602185">Apr 17 13:00:27 pool01 pgpool: 2013-04-17 13:00:27 LOG: pid 14025: degenerate_backend_set: 0 fail over request from pid 14025</font></div></div><div><br></div><div>At this point, listing the processes, I see:</div><div><br></div><div><div><font class="Apple-style-span" color="#602185">UID PID PPID C STIME TTY TIME CMD</font></div></div><div><div><font class="Apple-style-span" color="#602185">postgres 13984 1 0 13:00 pts/1 00:00:00 logger -t pgpool -p <a href="http://local0.info">local0.info</a></font></div><div><font class="Apple-style-span" color="#602185">postgres 13992 1 0 13:00 pts/1 00:00:00 pgpool: wait for connection request</font></div><div><font class="Apple-style-span" color="#602185">postgres 13993 1 0 13:00 pts/1 00:00:00 pgpool: wait for connection request</font></div><div><font class="Apple-style-span" color="#602185">postgres 13994 1 0 13:00 pts/1 00:00:00 pgpool: wait for connection request</font></div><div><font class="Apple-style-span" color="#602185">postgres 13995 1 0 13:00 pts/1 00:00:00 pgpool: wait for connection request</font></div><div><font class="Apple-style-span" color="#602185">postgres 13996 1 0 13:00 pts/1 00:00:00 pgpool: wait for connection request</font></div><div><font class="Apple-style-span" color="#602185">postgres 13997 1 0 13:00 pts/1 00:00:00 pgpool: wait for connection request</font></div><div><font class="Apple-style-span" color="#602185">postgres 13998 1 0 13:00 pts/1 00:00:00 pgpool: wait for connection request</font></div><div><font class="Apple-style-span" color="#602185">postgres 13999 1 0 13:00 pts/1 00:00:00 pgpool: wait for connection request</font></div><div><font class="Apple-style-span" color="#602185">postgres 14000 1 0 13:00 pts/1 00:00:00 pgpool: wait for connection request</font></div><div><font class="Apple-style-span" color="#602185">postgres 14001 1 0 13:00 pts/1 00:00:00 pgpool: wait for connection request</font></div><div><font class="Apple-style-span" color="#602185">postgres 14002 1 0 13:00 pts/1 00:00:00 pgpool: wait for connection request</font></div><div><font class="Apple-style-span" color="#602185">postgres 14003 1 0 13:00 pts/1 00:00:00 pgpool: wait for connection request</font></div><div><font class="Apple-style-span" color="#602185">postgres 14004 1 0 13:00 pts/1 00:00:00 pgpool: wait for connection request</font></div><div><font class="Apple-style-span" color="#602185">postgres 14005 1 0 13:00 pts/1 00:00:00 pgpool: wait for connection request</font></div><div><font class="Apple-style-span" color="#602185">postgres 14006 1 0 13:00 pts/1 00:00:00 pgpool: wait for connection request</font></div><div><font class="Apple-style-span" color="#602185">postgres 14007 1 0 13:00 pts/1 00:00:00 pgpool: wait for connection request</font></div><div><font class="Apple-style-span" color="#602185">postgres 14008 1 0 13:00 pts/1 00:00:00 pgpool: wait for connection request</font></div><div><font class="Apple-style-span" color="#602185">postgres 14009 1 0 13:00 pts/1 00:00:00 pgpool: wait for connection request</font></div><div><font class="Apple-style-span" color="#602185">postgres 14010 1 0 13:00 pts/1 00:00:00 pgpool: wait for connection request</font></div><div><font class="Apple-style-span" color="#602185">postgres 14011 1 0 13:00 pts/1 00:00:00 pgpool: wait for connection request</font></div><div><font class="Apple-style-span" color="#602185">postgres 14012 1 0 13:00 pts/1 00:00:00 pgpool: wait for connection request</font></div><div><font class="Apple-style-span" color="#602185">postgres 14013 1 0 13:00 pts/1 00:00:00 pgpool: wait for connection request</font></div><div><font class="Apple-style-span" color="#602185">postgres 14014 1 0 13:00 pts/1 00:00:00 pgpool: wait for connection request</font></div><div><font class="Apple-style-span" color="#602185">postgres 14015 1 0 13:00 pts/1 00:00:00 pgpool: wait for connection request</font></div><div><font class="Apple-style-span" color="#602185">postgres 14016 1 0 13:00 pts/1 00:00:00 pgpool: wait for connection request</font></div><div><font class="Apple-style-span" color="#602185">postgres 14017 1 0 13:00 pts/1 00:00:00 pgpool: wait for connection request</font></div><div><font class="Apple-style-span" color="#602185">postgres 14018 1 0 13:00 pts/1 00:00:00 pgpool: wait for connection request</font></div><div><font class="Apple-style-span" color="#602185">postgres 14019 1 0 13:00 pts/1 00:00:00 pgpool: wait for connection request</font></div><div><font class="Apple-style-span" color="#602185">postgres 14020 1 0 13:00 pts/1 00:00:00 pgpool: wait for connection request</font></div><div><font class="Apple-style-span" color="#602185">postgres 14021 1 0 13:00 pts/1 00:00:00 pgpool: wait for connection request</font></div><div><font class="Apple-style-span" color="#602185">postgres 14022 1 0 13:00 pts/1 00:00:00 pgpool: wait for connection request</font></div><div><font class="Apple-style-span" color="#602185">postgres 14023 1 0 13:00 pts/1 00:00:00 pgpool: wait for connection request</font></div><div><font class="Apple-style-span" color="#602185">postgres 14024 1 0 13:00 pts/1 00:00:00 pgpool: PCP: wait for connection request</font></div></div><div><br></div><div>As you can see, process 14025 has died, and all its children are now children of process 1.</div><div><br></div><div>Attempting to psql (pgpool is at port 5432 so I don't need to add parameters):</div><div><br></div><div><div><font class="Apple-style-span" color="#2c7b2a">postgres@pool01:/etc/pgpool2$</font> psql</div><div><font class="Apple-style-span" color="#602185">psql: </font><font class="Apple-style-span" color="#2c7b2a">postgres@pool01:/etc/pgpool2$ </font></div></div><div><br></div><div>So it just prints "psql:" without newline and then dies. The exit status is 2.</div><div><br></div><div>This gives me the following in the log:</div><div><br></div><div><div><font class="Apple-style-span" color="#602185">Apr 17 13:14:20 pool01 pgpool: 2013-04-17 13:14:20 DEBUG: pid 14022: I am 14022 accept fd 6</font></div><div><font class="Apple-style-span" color="#602185">Apr 17 13:14:20 pool01 pgpool: 2013-04-17 13:14:20 LOG: pid 14022: connection received: host=[local]</font></div><div><font class="Apple-style-span" color="#602185">Apr 17 13:14:20 pool01 pgpool: 2013-04-17 13:14:20 DEBUG: pid 14022: read_startup_packet: application_name: psql</font></div><div><font class="Apple-style-span" color="#602185">Apr 17 13:14:20 pool01 pgpool: 2013-04-17 13:14:20 DEBUG: pid 14022: Protocol Major: 3 Minor: 0 database: postgres user: postgres</font></div><div><font class="Apple-style-span" color="#602185">Apr 17 13:14:20 pool01 pgpool: 2013-04-17 13:14:20 DEBUG: pid 14022: new_connection: connecting 0 backend</font></div><div><font class="Apple-style-span" color="#602185">Apr 17 13:14:20 pool01 pgpool: 2013-04-17 13:14:20 DEBUG: pid 14022: new_connection: connecting 1 backend</font></div><div><font class="Apple-style-span" color="#602185">Apr 17 13:14:20 pool01 pgpool: 2013-04-17 13:14:20 DEBUG: pid 14022: pool_ssl: SSL requested but SSL support is not available</font></div><div><font class="Apple-style-span" color="#602185">Apr 17 13:14:20 pool01 pgpool: 2013-04-17 13:14:20 ERROR: pid 14022: pool_flush_it: write failed to backend (0). reason: Connection refused offset: 0 wlen: 84</font></div><div><font class="Apple-style-span" color="#602185">Apr 17 13:14:20 pool01 pgpool: 2013-04-17 13:14:20 LOG: pid 14022: degenerate_backend_set: 0 fail over request from pid 14022</font></div><div><font class="Apple-style-span" color="#602185">Apr 17 13:14:20 pool01 pgpool: 2013-04-17 13:14:20 ERROR: pid 14022: pool_flush_it: write failed to backend (0). reason: Broken pipe offset: 0 wlen: 5</font></div></div><div><br></div><div>And after this, process 14022 is also dead.</div><div><br></div><div>What could be the problem here? Why doesn't it see that it has one backend and just connect to it? This is how it should work if in the past it had failed over to the second backend. By the way, here is what I get when I try to show the status of the backends:</div><div><br></div><div><div><font class="Apple-style-span" color="#2c7b2a">postgres@pool01:/etc/pgpool2$</font> /usr/sbin/pcp_node_info 10 localhost 9898 postgres pass123 0</div><div><font class="Apple-style-span" color="#602185">pstgr01 5432 2 0.500000</font></div><div><font class="Apple-style-span" color="#2c7b2a">postgres@pool01:/etc/pgpool2$</font> /usr/sbin/pcp_node_info 10 localhost 9898 postgres pass123 1</div><div><font class="Apple-style-span" color="#602185">pstgr02 5432 2 0.500000</font></div></div><div><br></div><div>But this seems to say that he thinks both backends are OK!</div><div><br></div><div>Can anybody explain what is happening or what I should look for?</div><div><br></div><div>Thank you,</div><div>Herouth</div></body></html>