<html>
<head>
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
This error is occuring on a production server so I cannot upgrade it
to 3.3.5 for a while. We have test servers, but it appears that
this happens under high load and I have not been able to replicate
it on our test server.<br>
<br>
I have attached the pgpool.conf file from the system.<br>
<br>
The postgres version is "PostgreSQL 9.3.5 on
x86_64-unknown-linux-gnu, compiled by gcc (Debian 4.7.2-5) 4.7.2,
64-bit"<br>
<br>
Here is the snippet of the pool.log file then the problem occurs.<br>
<br>
Mar 5 12:49:56 ppasv57 pgpool[5500]: ProcessFrontendResponse:
failed to read kind from frontend. frontend abnormally exited<br>
Mar 5 12:50:03 ppasv57 pgpool[1421]: pgpool detected difference of
the number of inserted, updated or deleted tuples. Possible last
query was: "update ppadata.notebase set ntbmsgcount=$1,ntbupdated=$2
where ntbid=$3"<br>
Mar 5 12:50:03 ppasv57 pgpool[1421]: CommandComplete: Number of
affected tuples are: 1 0<br>
Mar 5 12:50:03 ppasv57 pgpool[1421]: ReadyForQuery: Degenerate
backends: 1<br>
Mar 5 12:50:03 ppasv57 pgpool[1421]: ReadyForQuery: Number of
affected tuples are: 1 0<br>
Mar 5 12:50:03 ppasv57 pgpool[1421]: degenerate_backend_set: 1 fail
over request from pid 1421<br>
Mar 5 12:50:03 ppasv57 pgpool[2896]: starting degeneration.
shutdown host ppasv59g(9999)<br>
Mar 5 12:50:03 ppasv57 pgpool[2896]: Restart all children<br>
Mar 5 12:50:03 ppasv57 pgpool[2896]: execute command: echo
host:ppasv59g, new master id:0, old master id:0 | mail -s pgpool\
failover <a class="moz-txt-link-abbreviated" href="mailto:notifier@rxhealthcare.org">notifier@rxhealthcare.org</a><br>
Mar 5 12:50:04 ppasv57 pgpool[2896]: failover: set new primary
node: -1<br>
Mar 5 12:50:04 ppasv57 pgpool[2896]: failover: set new master node:
0<br>
Mar 5 12:50:04 ppasv57 pgpool[2896]: failover done. shutdown host
ppasv59g(9999)<br>
Mar 5 12:50:04 ppasv57 pgpool[14731]: worker process received
restart request<br>
Mar 5 12:50:05 ppasv57 pgpool[14730]: pcp child process received
restart request<br>
Mar 5 12:50:05 ppasv57 pgpool[2896]: PCP child 14730 exits with
status 256 in failover()<br>
Mar 5 12:50:05 ppasv57 pgpool[2896]: fork a new PCP child pid 6222
in failover()<br>
Mar 5 12:50:05 ppasv57 pgpool[2896]: worker child 14731 exits with
status 256<br>
Mar 5 12:50:05 ppasv57 pgpool[2896]: fork a new worker child pid
6223<br>
<br>
Let me know if you need more information.<br>
<br>
Steve<br>
<br>
<div class="moz-cite-prefix">On 03/09/2015 10:29 AM, Muhammad Usama
wrote:<br>
</div>
<blockquote
cite="mid:CAEJvTzXyinx=ksZuHb7Cdr+JW0BnN4QeAcfumpa=JnecLD1DmA@mail.gmail.com"
type="cite">
<div dir="ltr">Hi
<div><br>
</div>
<div>I am trying to investigate the issue, meanwhile, can you
please share the pgpool.conf file, pgpool-II log file when the
problem occurs and PostgreSQL version information.</div>
<div><br>
</div>
<div>Also <span style="font-size:12.8000001907349px">pgpool-II
3.3.2 is little older. Can you confirm if the
problem persists after updating to the latest version </span><span
style="font-size:12.8000001907349px">3.3.5</span></div>
<div><span style="font-size:12.8000001907349px"><br>
</span></div>
<div><span style="font-size:12.8000001907349px">Thanks</span></div>
<div><span style="font-size:12.8000001907349px">Kind regards!</span></div>
<div><span style="font-size:12.8000001907349px">Muhammad Usama</span></div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Fri, Mar 6, 2015 at 8:01 PM, Steve
Kuekes <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:steve.kuekes@physicianspharmacy.com"
target="_blank">steve.kuekes@physicianspharmacy.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"> pgpool is handling
the replication<br>
<br>
I have<br>
<br>
replication_mode = on<br>
<br>
in my configuration. The question is how can the 2 nodes
get out of sync. All database access to the Postgres
nodes is going through pgpool.<span class="HOEnZb"><font
color="#888888"><br>
<br>
Steve</font></span>
<div>
<div class="h5"><br>
<br>
<div>On 03/06/2015 09:41 AM, Muhammad Usama wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Hi Steve<br>
<br>
What kind of replication are you using between
backend nodes?<br>
<br>
<i><b>replication_stop_on_mismatch</b></i> and <i><b>failover_if_affected_tuples_mismatch</b></i>
configuration parameters dictate the behavior of
pgpool-II, when the synchronization problem
between PostgreSQL backends is detected. These
parameters have nothing to do for keeping the data
consistent between nodes. For that you have to
check why one of the node lacks behind the other.
<br>
<br>
Thanks
<div>Kind regards!</div>
<div>Muhammad Usama<br>
<br>
On Fri, Mar 6, 2015 at 1:42 AM, Steve Kuekes
<<a moz-do-not-send="true"
href="mailto:steve.kuekes@physicianspharmacy.com"
target="_blank">steve.kuekes@physicianspharmacy.com</a>>
wrote:<br>
> I am running pgpool version pgpool-II
version 3.3.2 (tokakiboshi) in<br>
> replication mode with 2 back ends with load
balancing. We have a very busy<br>
> database and we cannot seem to keep both
back ends up for longer that 12-16<br>
> hours of production.<br>
><br>
> These are the messages I get in the pgpool
log when the failover occurs<br>
><br>
> Mar 5 12:50:03 ppasv57 pgpool[1421]:
pgpool detected difference of the<br>
> number of inserted, updated or deleted
tuples. Possible last query was:<br>
> "update ppadata.notebase set
ntbmsgcount=$1,ntbupdated=$2 where ntbid=$3"<br>
> Mar 5 12:50:03 ppasv57 pgpool[1421]:
CommandComplete: Number of affected<br>
> tuples are: 1 0<br>
> Mar 5 12:50:03 ppasv57 pgpool[1421]:
ReadyForQuery: Degenerate backends: 1<br>
> Mar 5 12:50:03 ppasv57 pgpool[1421]:
ReadyForQuery: Number of affected<br>
> tuples are: 1 0<br>
> Mar 5 12:50:03 ppasv57 pgpool[1421]:
degenerate_backend_set: 1 fail over<br>
> request from pid 1421<br>
> Mar 5 12:50:03 ppasv57 pgpool[2896]:
starting degeneration. shutdown host<br>
> ppasv59g(9999)<br>
> Mar 5 12:50:03 ppasv57 pgpool[2896]:
Restart all children<br>
><br>
> This update statement is being done within
a postgres transaction and it<br>
> appears after the failover the transaction
was successfully rolled back on<br>
> both back ends.<br>
><br>
> We have the following options set in the
pgpool.conf<br>
><br>
> replication_stop_on_mismatch = on<br>
> failover_if_affected_tuples_mismatch = on<br>
><br>
> Don't we need to have these options set to
make sure that both back ends<br>
> have exactly the same data?<br>
><br>
> What do I need to do to keep both back ends
up?<br>
><br>
> Thanks for any help<br>
><br>
> Steve<br>
><br>
> --<br>
> Steve Kuekes<br>
><br>
> Physicians Pharmacy Alliance<br>
> 118 MacKenan Drive, Suite 200<br>
> Cary, NC 27511<br>
> 919-465-5801 direct<br>
> 919-463-5555 main<br>
> 919-463-5566 fax<br>
><br>
> <a moz-do-not-send="true"
href="mailto:steve.kuekes@physicianspharmacy.com"
target="_blank">steve.kuekes@physicianspharmacy.com</a><br>
><br>
>
_______________________________________________<br>
> pgpool-general mailing list<br>
> <a moz-do-not-send="true"
href="mailto:pgpool-general@pgpool.net"
target="_blank">pgpool-general@pgpool.net</a><br>
> <a moz-do-not-send="true"
href="http://www.pgpool.net/mailman/listinfo/pgpool-general"
target="_blank">http://www.pgpool.net/mailman/listinfo/pgpool-general</a></div>
</div>
</blockquote>
<br>
<pre cols="72">--
Steve Kuekes
Physicians Pharmacy Alliance
118 MacKenan Drive, Suite 200
Cary, NC 27511
919-465-5801 direct
919-463-5555 main
919-463-5566 fax
<a moz-do-not-send="true" href="mailto:steve.kuekes@physicianspharmacy.com" target="_blank">steve.kuekes@physicianspharmacy.com</a>
</pre>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Steve Kuekes
Physicians Pharmacy Alliance
118 MacKenan Drive, Suite 200
Cary, NC 27511
919-465-5801 direct
919-463-5555 main
919-463-5566 fax
<a class="moz-txt-link-abbreviated" href="mailto:steve.kuekes@physicianspharmacy.com">steve.kuekes@physicianspharmacy.com</a>
</pre>
</body>
</html>