<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">A quick rebuild and here we go:<br>
<br>
jeff@squeeze:/usr/local/pgpool2$ ps -ef|grep pgpool<br>
postgres 13073 1 0 12:35 pts/1 00:00:00 logger -t pgpool
-p local0.info<br>
postgres 13098 1 0 12:35 pts/1 00:00:00 pgpool:
watchdog <br>
postgres 13099 1 0 12:35 pts/1 00:00:00 pgpool:
lifecheck <br>
<br>
sudo gdb -p 13098<br>
<br>
(gdb) bt<br>
#0 0x00007f492036b3e3 in select () from /lib/libc.so.6<br>
#1 0x0000000000478d98 in wd_accept (sock=<value optimized
out>) at wd_packet.c:302<br>
#2 0x0000000000477137 in wd_child (fork_wait_time=1) at
wd_child.c:91<br>
#3 0x0000000000476db5 in wd_main (fork_wait_time=1) at
watchdog.c:127<br>
#4 0x00000000004086ec in main (argc=<value optimized out>,
argv=<value optimized out>) at main.c:632<br>
<br>
sudo gdb -p 13098<br>
<br>
(gdb) bt<br>
#0 0x00007f4920ef9b33 in wait () from /lib/libpthread.so.0<br>
#1 0x00000000004776bc in exec_ifconfig (path=0x7fffde882fe0
"/usr/bin/sudo", command=<value optimized out>) at
wd_if.c:191<br>
#2 0x0000000000477843 in wd_IP_down () at wd_if.c:79<br>
#3 0x0000000000479699 in wd_notice_server_down () at
wd_packet.c:119<br>
#4 0x0000000000476f30 in wd_exit (exit_signo=2) at watchdog.c:75<br>
#5 <signal handler called><br>
#6 0x00007f4920342c5d in nanosleep () from /lib/libc.so.6<br>
#7 0x00007f4920342ad0 in sleep () from /lib/libc.so.6<br>
#8 0x0000000000476e87 in wd_main (fork_wait_time=1) at
watchdog.c:160<br>
#9 0x00000000004086ec in main (argc=<value optimized out>,
argv=<value optimized out>) at main.c:632<br>
<br>
jeff@squeeze:/usr/local/pgpool2$ ps -ef|grep ifconfig<br>
root 2220 2121 0 12:45 pts/1 00:00:00 sudo ifconfig
eth0:1 10.10.10.28 netmask 255.255.255.0 down<br>
root 2221 2220 0 12:45 pts/1 00:00:00 [ifconfig]
<defunct><br>
<br>
jeff@squeeze:/usr/local/pgpool2$ sudo gdb -p 2220<br>
<br>
(gdb) bt<br>
#0 0x00007f1b412563c3 in select () from /lib/libc.so.6<br>
#1 0x0000000000409a23 in sudo_execve ()<br>
#2 0x000000000040e463 in run_command ()<br>
#3 0x000000000040fce0 in main ()<br>
<br>
jeff@squeeze:/usr/local/pgpool2$ sudo gdb -p 2221<br>
<br>
Attaching to process 2221<br>
ptrace: Operation not permitted.<br>
(gdb) bt<br>
No stack.<br>
<br>
An important note is that the ifconfig down is actually successful
even though it doesn't return - that is, the eth0:1 interface goes
away.<br>
<br>
So it looks like this:<br>
<br>
Jul 26 12:45:05 squeeze pgpool: 2013-07-26 12:45:05 ERROR: pid
2094: find_primary_node: make_persistent_connection failed<br>
Jul 26 12:45:05 squeeze pgpool: 2013-07-26 12:45:05 LOG: pid
2094: received fast shutdown request<br>
Jul 26 12:45:05 squeeze pgpool: 2013-07-26 12:45:05 LOG: pid
2094: watchdog_pid: 2121<br>
Jul 26 12:45:25 squeeze pgpool: 2013-07-26 12:45:25 ERROR: pid
2094: wait() failed. reason:Interrupted system call<br>
<br>
Till I kill -9 the sudo process:<br>
<br>
jeff@squeeze:/usr/local/pgpool2$ sudo kill -9 2220<br>
<br>
Then these two log lines are emitted:<br>
<br>
Jul 26 12:48:59 squeeze pgpool: 2013-07-26 12:48:59 ERROR: pid
2121: wd_IP_down: ifconfig down failed<br>
Jul 26 12:49:02 squeeze pgpool: 2013-07-26 12:49:02 LOG: pid
2121: wd_create_send_socket: connect() reports failure (No route
to host). You can safely ignore this while starting up.<br>
<br>
<br>
<br>
On 07/26/13 12:30, Jeff Frost wrote:<br>
</div>
<blockquote
cite="mid:WM!f396d2ab0b78ac7e7c3a2dcd78e285c9f8c3cad6b330b0371363027437fd1974e10b5314878e06c2303b30964ee0775c!@asav-3.01.com"
type="cite">
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
<div class="moz-cite-prefix">More info:<br>
<br>
<div dir="ltr">
<div class="markdown-here-wrapper"
id="markdown-here-wrapper-398933" style="">
<p style="margin:1.2em 0px!important">Here is a syslog
snippet:</p>
<pre style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;font-size:1em;line-height:1.2em;overflow:auto;margin:1.2em 0px"><code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:nowrap;border:1px solid rgb(234,234,234);background-color:rgb(248,248,248);border-top-left-radius:3px;border-top-right-radius:3px;border-bottom-right-radius:3px;border-bottom-left-radius:3px;display:inline;white-space:pre;border-top-left-radius:3px;border-top-right-radius:3px;border-bottom-right-radius:3px;border-bottom-left-radius:3px;border:1px solid rgb(204,204,204);padding:0.5em 0.7em;display:block;padding:0.5em;color:rgb(51,51,51);background-color:rgb(248,248,255);background-repeat:initial initial">Jul 26 12:06:33 pgpool01 pgpool: 2013-07-26 12:06:33 LOG: pid 12847: wd_create_send_socket: connect() reports failure (Connection refused). You can safely ignore this while starting up.
Jul 26 12:06:49 pgpool01 pgpool: 2013-07-26 12:06:49 LOG: pid 13243: wd_chk_sticy: all commands have sticky bit
Jul 26 12:06:49 pgpool01 pgpool: 2013-07-26 12:06:49 LOG: pid 13243: watchdog might call network commands which using sticky bit.
Jul 26 12:06:49 pgpool01 pgpool: 2013-07-26 12:06:49 LOG: pid 13243: wd_create_send_socket: connect() reports failure (Connection refused). You can safely ignore this while starting up.
Jul 26 12:06:52 pgpool01 pgpool: 2013-07-26 12:06:52 LOG: pid 13243: wd_escalation: escalated to master pgpool
Jul 26 12:06:54 pgpool01 pgpool: 2013-07-26 12:06:54 LOG: pid 13243: wd_create_send_socket: connect() reports failure (Connection refused). You can safely ignore this while starting up.
Jul 26 12:06:54 pgpool01 pgpool: 2013-07-26 12:06:54 LOG: pid 13243: wd_escalation: escaleted to delegate_IP holder
Jul 26 12:06:54 pgpool01 pgpool: 2013-07-26 12:06:54 LOG: pid 13243: wd_init: start watchdog
Jul 26 12:06:54 pgpool01 pgpool: 2013-07-26 12:06:54 LOG: pid 13243: pgpool-II successfully started. version 3.2.4 (namameboshi)
Jul 26 12:06:54 pgpool01 pgpool: 2013-07-26 12:06:54 LOG: pid 13243: find_primary_node: primary node id is 0
Jul 26 12:10:13 pgpool01 pgpool: 2013-07-26 12:10:13 LOG: pid 13243: received fast shutdown request
Jul 26 12:10:13 pgpool01 pgpool: 2013-07-26 12:10:13 LOG: pid 13243: watchdog_pid: 13257
<b>Jul 26 12:10:53 pgpool01 pgpool: 2013-07-26 12:10:53 ERROR: pid 13257: wd_IP_down: ifconfig down failed</b>
Jul 26 12:10:53 pgpool01 pgpool: 2013-07-26 12:10:53 LOG: pid 13257: wd_create_send_socket: connect() reports failure (Connection refused). You can safely ignore this while starting up.
Jul 26 12:11:19 pgpool01 pgpool: 2013-07-26 12:11:19 LOG: pid 13613: wd_chk_sticy: all commands have sticky bit</code></pre>
<p style="margin:1.2em 0px!important">I also attached 2
straces - one was the pgpool process as I issued a stop,
and the other was the watchdog process.</p>
</div>
</div>
Unfortunately, it doesn't look like apt.postgresql.org provides
dbg files, so I'll have to rebuild pgpool2 so I can get the
symbols for the backtraces.<br>
<br>
<br>
On 07/26/13 07:48, Jeff Frost wrote:<br>
</div>
<blockquote
cite="mid:WM!ebda45320d0432d0b032840fa7aff46b385fb236f4b33cb93217c9402814088e68d74e06b2c619e63906825209f9a6e6!@asav-3.01.com"
type="cite">
<pre wrap="">Yes, you can see the pgpool processes stuck in my ps output below.
They happily exit once I kill -9 the sudo process.
I'll see if I can get some stack traces but if you can't reproduce on Ubuntu or CentOS, I suspect it's something with Debian Squeeze's sudo or ifconfig commands.
On Jul 26, 2013, at 3:24 AM, Yugo Nagata <a moz-do-not-send="true" class="moz-txt-link-rfc2396E" href="mailto:nagata@sraoss.co.jp"><nagata@sraoss.co.jp></a> wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Hi,
Does pgpool hang as well as ifconfig when it is stopped?
I cannot reproduce this on CentOS and Ubuntu. Both pgpool and
ifconfig stops normally.
Could you please provide me the stack trace of hanging pgpool and
log msessages?
On Thu, 25 Jul 2013 09:56:36 -0700
Jeff Frost <a moz-do-not-send="true" class="moz-txt-link-rfc2396E" href="mailto:jeff@pgexperts.com"><jeff@pgexperts.com></a> wrote:
</pre>
<blockquote type="cite">
<pre wrap="">This seems to be the same on 3.2.3, 3.2.4 and 3.2.5.
The watchdog section of pgpool.conf looks like this:
use_watchdog = on
delegate_IP = '10.100.2.72'
wd_hostname = '10.100.2.70'
wd_port = 9000
ifconfig_path = '/usr/bin'
arping_path = '/usr/bin'
if_up_cmd = 'sudo ifconfig eth0:1 $_IP_$ netmask 255.255.255.0 up'
if_down_cmd = 'sudo ifconfig eth0:1 $_IP_$ netmask 255.255.255.0 down'
arping_cmd = 'sudo arping -U $_IP_$ -w 1'
wd_interval = 3
other_pgpool_hostname0 = '10.100.2.71'
other_pgpool_port0 = 9999
other_wd_port0 = 9000
virtual IP starts up great and properly moves to the secondary pgpool server
if you stop pgpool. However, the ifconfig becomes defunct and never exits
requiring a kill -9:
jeff@pgpool01:/tmp/pgpool$ ps -ef|grep pgpool
postgres 19974 1 0 09:51 pts/0 00:00:00 /tmp/pgpool/bin/pgpool -n
postgres 19975 1 0 09:51 pts/0 00:00:00 logger -t pgpool -p local0.info
postgres 19978 19974 0 09:51 pts/0 00:00:00 pgpool: watchdog
postgres 19979 19974 0 09:51 pts/0 00:00:00 pgpool: lifecheck
jeff 20735 1615 0 09:54 pts/0 00:00:00 grep pgpool
jeff@pgpool01:/tmp/pgpool$ ps -ef|grep ifconfig
root 20439 19979 0 09:52 pts/0 00:00:00 sudo ifconfig eth0:1
10.100.2.72 netmask 255.255.255.0 down
root 20440 20439 0 09:52 pts/0 00:00:00 [ifconfig] <defunct>
jeff 20737 1615 0 09:54 pts/0 00:00:00 grep ifconfig
System is Debian Squeeze. Any idea how to fix this? kill -9 of the sudo
allows pgpool to exit.
--
Jeff Frost <a moz-do-not-send="true" class="moz-txt-link-rfc2396E" href="mailto:jeff@pgexperts.com"><jeff@pgexperts.com></a>
CTO, PostgreSQL Experts, Inc.
Phone: 1-888-PG-EXPRT x506
FAX: 415-762-5122
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://www.pgexperts.com/">http://www.pgexperts.com/</a>
_______________________________________________
pgpool-general mailing list
<a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:pgpool-general@pgpool.net">pgpool-general@pgpool.net</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://www.pgpool.net/mailman/listinfo/pgpool-general">http://www.pgpool.net/mailman/listinfo/pgpool-general</a>
</pre>
</blockquote>
<pre wrap="">
--
Yugo Nagata <a moz-do-not-send="true" class="moz-txt-link-rfc2396E" href="mailto:nagata@sraoss.co.jp"><nagata@sraoss.co.jp></a>
</pre>
</blockquote>
<pre wrap="">---
Jeff Frost <a moz-do-not-send="true" class="moz-txt-link-rfc2396E" href="mailto:jeff@pgexperts.com"><jeff@pgexperts.com></a>
CTO, PostgreSQL Experts, Inc.
Phone: 1-888-PG-EXPRT x506
FAX: 415-762-5122
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://www.pgexperts.com/">http://www.pgexperts.com/</a>
_______________________________________________
pgpool-general mailing list
<a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:pgpool-general@pgpool.net">pgpool-general@pgpool.net</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://www.pgpool.net/mailman/listinfo/pgpool-general">http://www.pgpool.net/mailman/listinfo/pgpool-general</a>
</pre>
</blockquote>
<br>
<br>
<pre class="moz-signature" cols="78">--
Jeff Frost <a moz-do-not-send="true" class="moz-txt-link-rfc2396E" href="mailto:jeff@pgexperts.com"><jeff@pgexperts.com></a>
CTO, PostgreSQL Experts, Inc.
Phone: 1-888-PG-EXPRT x506
FAX: 415-762-5122
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://www.pgexperts.com/">http://www.pgexperts.com/</a>
</pre>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
pgpool-general mailing list
<a class="moz-txt-link-abbreviated" href="mailto:pgpool-general@pgpool.net">pgpool-general@pgpool.net</a>
<a class="moz-txt-link-freetext" href="http://www.pgpool.net/mailman/listinfo/pgpool-general">http://www.pgpool.net/mailman/listinfo/pgpool-general</a>
</pre>
</blockquote>
<br>
<br>
<pre class="moz-signature" cols="78">--
Jeff Frost <a class="moz-txt-link-rfc2396E" href="mailto:jeff@pgexperts.com"><jeff@pgexperts.com></a>
CTO, PostgreSQL Experts, Inc.
Phone: 1-888-PG-EXPRT x506
FAX: 415-762-5122
<a class="moz-txt-link-freetext" href="http://www.pgexperts.com/">http://www.pgexperts.com/</a>
</pre>
</body>
</html>