<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">Hi Muhammad<o:p></o:p></p>
<p class="MsoNormal">I’m sending you the pgpool.log from both node0 and node1. The pgpool.log files are when I used pgpool.conf from 4.1.0. Also sending the pgpool.conf from 4.0.5 and 4.1.0 When I start pgpool using 4.0.5 pgpool.conf I’m not getting any errors
but when I use pgpool.conf from 4.1.0 I’m getting “We are in split brain”<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I tar all files into pgpool.tar.gz <o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:10.0pt">Wolfgang Schwurack<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:10.0pt">Database/System Administrator<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:10.0pt">Utah Education Network<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:10.0pt">801-587-9444<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt"><a href="mailto:wolf@uen.org"><span style="color:#0B4CB4">wolf@uen.org</span></a><o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:12.0pt;color:black">From: </span></b><span style="font-size:12.0pt;color:black">Muhammad Usama <m.usama@gmail.com><br>
<b>Date: </b>Wednesday, March 4, 2020 at 2:20 AM<br>
<b>To: </b>Tatsuo Ishii <ishii@sraoss.co.jp><br>
<b>Cc: </b>Wolfgang Schwurack <wolf@uen.org>, PgPool General <pgpool-general@pgpool.net><br>
<b>Subject: </b>Re: [pgpool-general: 6865] Re: watchdog fails to start pgpool-4.1.0<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Hi <span style="font-size:10.0pt">Wolfgang,</span> <o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt">Sorry for the late reply. I just realized the email was sitting in my drafts folder and was never sent.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Is it possible if you can share the Pppool log files for both nodes preferably with the debug enabled?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Meanwhile, I am also trying to reproduce the scenario locally.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Thanks<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Best regards<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Muhammad Usama<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Tue, Feb 18, 2020 at 12:13 PM Tatsuo Ishii <<a href="mailto:ishii@sraoss.co.jp">ishii@sraoss.co.jp</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<p class="MsoNormal">Hi Usama,<br>
<br>
Any opinion on this?<br>
<br>
Best regards,<br>
--<br>
Tatsuo Ishii<br>
SRA OSS, Inc. Japan<br>
English: <a href="http://www.sraoss.co.jp/index_en.php" target="_blank">http://www.sraoss.co.jp/index_en.php</a><br>
Japanese:<a href="http://www.sraoss.co.jp" target="_blank">http://www.sraoss.co.jp</a><br>
<br>
> I turned on enable_consensus_with_half_votes which I’m getting the<br>
> acquired delegate IP on node 0. But now when I start pgpool on node 1<br>
> getting this in the log file which is repeating - see below. When I check<br>
> which node has the virtual IP is show that node 0 does which is the master<br>
> node. <br>
> <br>
> 2020-02-12 08:11:52: pid 29493: LOG: watchdog node state changed from<br>
> [INITIALIZING] to [MASTER]<br>
> 2020-02-12 08:11:52: pid 29493: LOG: I am announcing my self as<br>
> master/coordinator watchdog node<br>
> 2020-02-12 08:11:52: pid 29493: LOG: remote node "" decided it is the<br>
> true master<br>
> 2020-02-12 08:11:52: pid 29493: DETAIL: re-initializing the local<br>
> watchdog cluster state because of split-brain<br>
> 2020-02-12 08:11:52: pid 29493: LOG: watchdog node state changed from<br>
> [MASTER] to [JOINING]<br>
> 2020-02-12 08:11:53: pid 29493: LOG: new watchdog node connection is<br>
> received from "<a href="http://10.11.0.202:12399" target="_blank">10.11.0.202:12399</a>"<br>
> 2020-02-12 08:11:56: pid 29493: LOG: watchdog node state changed from<br>
> [JOINING] to [INITIALIZING]<br>
> 2020-02-12 08:11:57: pid 29493: LOG: I am the only alive node in the<br>
> watchdog cluster<br>
> 2020-02-12 08:11:57: pid 29493: HINT: skipping stand for coordinator state<br>
> <br>
> My environment<br>
> 2 pgpool hosts on Ubuntu 18<br>
> 2 postgresql hosts on Ubuntu 18 postgreSQL 11<br>
> <br>
> <br>
> Wolfgang Schwurack<br>
> Database/System Administrator<br>
> Utah Education Network<br>
> 801-587-9444<br>
> <a href="mailto:Wolf@uen.org" target="_blank">Wolf@uen.org</a><br>
> <br>
> <br>
> <br>
> <br>
> <br>
> On 2/11/20, 3:50 PM, "Tatsuo Ishii" <<a href="mailto:ishii@sraoss.co.jp" target="_blank">ishii@sraoss.co.jp</a>> wrote:<br>
> <br>
>>Have you turned on enable_consensus_with_half_votes?<br>
>>From 4.1 you need to turn on this if you use even number of Pgpool-II<br>
>>nodes.<br>
>>It's documented in the migration section in the doc:<br>
>><a href="https://www.pgpool.net/docs/latest/en/html/release-4-1-0.html" target="_blank">https://www.pgpool.net/docs/latest/en/html/release-4-1-0.html</a><br>
>><br>
>>Best regards,<br>
>>--<br>
>>Tatsuo Ishii<br>
>>SRA OSS, Inc. Japan<br>
>>English: <a href="http://www.sraoss.co.jp/index_en.php" target="_blank">http://www.sraoss.co.jp/index_en.php</a><br>
>>Japanese:<a href="http://www.sraoss.co.jp" target="_blank">http://www.sraoss.co.jp</a><br>
>><br>
>>From: Wolf Schwurack <<a href="mailto:wolf@uen.org" target="_blank">wolf@uen.org</a>><br>
>>Subject: [pgpool-general: 6865] Re: watchdog fails to start pgpool-4.1.0<br>
>>Date: Tue, 11 Feb 2020 18:10:25 +0000<br>
>>Message-ID: <<a href="mailto:56216C05-00F8-4C10-A32A-C793411C7891@umail.utah.edu" target="_blank">56216C05-00F8-4C10-A32A-C793411C7891@umail.utah.edu</a>><br>
>><br>
>>> After doing some more testing on version 4.1.0 I have notice that if<br>
>>>node 0 fails, node 1 never acquires the delegate IP. I compared this to<br>
>>>version 4.0.5 which when node 0 fails, node 1 acquires the delegate IP<br>
>>> <br>
>>> Wolfgang Schwurack<br>
>>> Database/System Administrator<br>
>>> Utah Education Network<br>
>>> 801-587-9444<br>
>>> <a href="mailto:wolf@uen.org" target="_blank">wolf@uen.org</a><mailto:<a href="mailto:wolf@uen.org" target="_blank">wolf@uen.org</a>><br>
>>> <br>
>>> From: "<a href="mailto:pgpool-general-bounces@pgpool.net" target="_blank">pgpool-general-bounces@pgpool.net</a>"<br>
>>><<a href="mailto:pgpool-general-bounces@pgpool.net" target="_blank">pgpool-general-bounces@pgpool.net</a>> on behalf of Wolfgang Schwurack<br>
>>><<a href="mailto:wolf@uen.org" target="_blank">wolf@uen.org</a>><br>
>>> Date: Tuesday, February 11, 2020 at 10:54 AM<br>
>>> To: "<a href="mailto:pgpool-general@pgpool.net" target="_blank">pgpool-general@pgpool.net</a>" <<a href="mailto:pgpool-general@pgpool.net" target="_blank">pgpool-general@pgpool.net</a>><br>
>>> Subject: [pgpool-general: 6864] Re: watchdog fails to start pgpool-4.1.0<br>
>>> <br>
>>> It seem that version 4.1.0 requires the second node to be started<br>
>>>before acquired the delegate IP<br>
>>> After starting pgpool on the node 1 I?m seeing that watchdog<br>
>>>successfully acquired the delegate IP on node 0<br>
>>> <br>
>>> 2020-02-11 10:45:26: pid 9928: LOG: watchdog: escalation started<br>
>>> 2020-02-11 10:45:33: pid 9928: LOG: successfully acquired the delegate<br>
>>>IP:"10.11.0.204"<br>
>>> 2020-02-11 10:45:33: pid 9928: DETAIL: 'if_up_cmd' returned with<br>
>>>success<br>
>>> 2020-02-11 10:45:33: pid 9577: LOG: watchdog escalation process with<br>
>>>pid: 9928 exit with SUCCESS.<br>
>>> <br>
>>> On previous versions watchdog would always acquire the delegate IP<br>
>>>without the second node being started.<br>
>>> <br>
>>> <br>
>>> From: "<a href="mailto:pgpool-general-bounces@pgpool.net" target="_blank">pgpool-general-bounces@pgpool.net</a>"<br>
>>><<a href="mailto:pgpool-general-bounces@pgpool.net" target="_blank">pgpool-general-bounces@pgpool.net</a>> on behalf of Wolfgang Schwurack<br>
>>><<a href="mailto:wolf@uen.org" target="_blank">wolf@uen.org</a>><br>
>>> Date: Tuesday, February 11, 2020 at 10:22 AM<br>
>>> To: "<a href="mailto:pgpool-general@pgpool.net" target="_blank">pgpool-general@pgpool.net</a>" <<a href="mailto:pgpool-general@pgpool.net" target="_blank">pgpool-general@pgpool.net</a>><br>
>>> Subject: [pgpool-general: 6863] watchdog fails to start pgpool-4.1.0<br>
>>> <br>
>>> I?m trying to get watchdog to start using pgpool-4.1.0 but fails to<br>
>>>start. I have been using pgpool-4.0.5 with watchdog no issues.<br>
>>> Has something changed in version 4.1.0 for watchdog?<br>
>>> Hosts - Ubuntu 18.0.4<br>
>>> PostgreSQL 11<br>
>>> <br>
>>> I?ve been using pgpool for a long time on each new release I have<br>
>>>always just done ./configure, make, make install<br>
>>> <br>
>>> This is my start command<br>
>>> <br>
>>> /usr/local/bin/pgpool -n -D -f /usr/local/etc/pgpool.conf ><br>
>>>/var/log/pgpool/pgpool.log 2>&1 &<br>
>>> In pgpool.log it would always show if acquired the delegate ip<br>
>>> Version 4.0.5 start up watchdog<br>
>>> <br>
>>> 2020-02-11 10:13:05: pid 2195: LOG: pgpool-II successfully started.<br>
>>>version 4.0.5 (torokiboshi)<br>
>>> <br>
>>> 2020-02-11 10:13:05: pid 2195: LOG: node status[0]: 1<br>
>>> <br>
>>> 2020-02-11 10:13:05: pid 2195: LOG: node status[1]: 2<br>
>>> <br>
>>> 2020-02-11 10:13:06: pid 2228: LOG: creating socket for sending<br>
>>>heartbeat<br>
>>> <br>
>>> 2020-02-11 10:13:06: pid 2228: DETAIL: bind send socket to device: eth0<br>
>>> <br>
>>> 2020-02-11 10:13:06: pid 2228: LOG: set SO_REUSEPORT option to the<br>
>>>socket<br>
>>> <br>
>>> 2020-02-11 10:13:06: pid 2228: LOG: creating socket for sending<br>
>>>heartbeat<br>
>>> <br>
>>> 2020-02-11 10:13:06: pid 2228: DETAIL: set SO_REUSEPORT<br>
>>> <br>
>>> 2020-02-11 10:13:06: pid 2227: LOG: createing watchdog heartbeat<br>
>>>receive socket.<br>
>>> <br>
>>> 2020-02-11 10:13:06: pid 2227: DETAIL: bind receive socket to device:<br>
>>>"eth0"<br>
>>> <br>
>>> 2020-02-11 10:13:06: pid 2227: LOG: set SO_REUSEPORT option to the<br>
>>>socket<br>
>>> <br>
>>> 2020-02-11 10:13:06: pid 2227: LOG: creating watchdog heartbeat<br>
>>>receive socket.<br>
>>> <br>
>>> 2020-02-11 10:13:06: pid 2227: DETAIL: set SO_REUSEPORT<br>
>>> <br>
>>> 2020-02-11 10:13:12: pid 2200: LOG: successfully acquired the delegate<br>
>>>IP:"10.11.0.204"<br>
>>> <br>
>>> 2020-02-11 10:13:12: pid 2200: DETAIL: 'if_up_cmd' returned with<br>
>>>success<br>
>>> <br>
>>> 2020-02-11 10:13:12: pid 2197: LOG: watchdog escalation process with<br>
>>>pid: 2200 exit with SUCCESS.<br>
>>> <br>
>>> Version 4.1.0 fails to start watchdog<br>
>>> <br>
>>> 2020-02-11 10:15:54: pid 8392: LOG: pgpool-II successfully started.<br>
>>>version 4.1.0 (karasukiboshi)<br>
>>> <br>
>>> 2020-02-11 10:15:54: pid 8392: LOG: node status[0]: 1<br>
>>> <br>
>>> 2020-02-11 10:15:54: pid 8392: LOG: node status[1]: 2<br>
>>> <br>
>>> 2020-02-11 10:15:55: pid 8425: LOG: creating socket for sending<br>
>>>heartbeat<br>
>>> <br>
>>> 2020-02-11 10:15:55: pid 8425: DETAIL: bind send socket to device: eth0<br>
>>> <br>
>>> 2020-02-11 10:15:55: pid 8425: LOG: set SO_REUSEPORT option to the<br>
>>>socket<br>
>>> <br>
>>> 2020-02-11 10:15:55: pid 8425: LOG: creating socket for sending<br>
>>>heartbeat<br>
>>> <br>
>>> 2020-02-11 10:15:55: pid 8425: DETAIL: set SO_REUSEPORT<br>
>>> <br>
>>> 2020-02-11 10:15:55: pid 8424: LOG: createing watchdog heartbeat<br>
>>>receive socket.<br>
>>> <br>
>>> 2020-02-11 10:15:55: pid 8424: DETAIL: bind receive socket to device:<br>
>>>"eth0"<br>
>>> <br>
>>> 2020-02-11 10:15:55: pid 8424: LOG: set SO_REUSEPORT option to the<br>
>>>socket<br>
>>> <br>
>>> 2020-02-11 10:15:55: pid 8424: LOG: creating watchdog heartbeat<br>
>>>receive socket.<br>
>>> <br>
>>> 2020-02-11 10:15:55: pid 8424: DETAIL: set SO_REUSEPORT<br>
>>> <br>
>>> <br>
>>> Wolfgang Schwurack<br>
>>> Database/System Administrator<br>
>>> Utah Education Network<br>
>>> 801-587-9444<br>
>>> <a href="mailto:wolf@uen.org" target="_blank">wolf@uen.org</a><mailto:<a href="mailto:wolf@uen.org" target="_blank">wolf@uen.org</a>><br>
>>> <br>
> <o:p></o:p></p>
</blockquote>
</div>
</div>
</body>
</html>