[pgpool-hackers: 1520] Re: Proposal: minimize process restart when fail over occurs

Tue Apr 26 17:35:41 JST 2016

From: Ahsan Hadi <ahsan.hadi at enterprisedb.com>
Subject: Re: [pgpool-hackers: 1504] Re: Proposal: minimize process restart when fail over occurs
Date: Tue, 26 Apr 2016 13:10:48 +0500
Message-ID: <CAGX_dr1RLL2MwcKdfN+Vi-8xAzm+Hsoy36Nqrwqc+nr88C+2Fg at mail.gmail.com>

> On Tue, Apr 19, 2016 at 4:20 PM, Ahsan Hadi <ahsan.hadi at enterprisedb.com>
> wrote:
> 
>> Hi Ishii.
>>
>> As per my understanding, the goal of this feature is to avoid doing the
>> following in-case a stand-by node failure :
>>
>> a) disconnect all the client connections
>> b) not accept new client connection until timeout
>>
>> The objective is continue serving new client connection and continue
>> serving existing connection (unless they were connected to the failed node)
>> in-case of a failover.
>>
>> Is my understanding correct?

My recent committ tried to address a), not b).

Best regards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp

> Hi Ishii-san,
> 
> Will this feature serve the above objectives?
> 
> -- Ahsan
> 
> 
>>
>> -- Ahsan
>>
>>
>>
>>
>>
>> On Sun, Apr 17, 2016 at 2:15 PM, Tatsuo Ishii <ishii at postgresql.org>
>> wrote:
>>
>>> Ok, I have succeeded in not restart child process when certain
>>> conditions are met.
>>>
>>>  - Streaming replication mode
>>>  - pcp_detach_node is used
>>>  - does not use the load balance node (that means the process does not
>>>    issue queries to the load balance node)
>>>  - the node is not primary node
>>>
>>> At this point this just enhance allow following use cases (we assume
>>> that pcp_detach_node detaches DB node N):
>>>
>>> 1) Lucky users connecting to the database server N are not affected by
>>>    the pcp_detach_node.
>>>
>>> 2) Planned DB shutdown. For demonstration purpose, I use pgbench -C.
>>>
>>>    - start pgbench -C
>>>    - change pgpool.conf to change the weight to 0 for backend N.
>>>    - pgpool reload
>>>    - pcp_detach_node N
>>>    - pgbench happily continues the benchmark
>>>
>>> Probably #2 is practically useful.
>>>
>>> I think we could expand this to certain cases such as PostgreSQL
>>> is shutdown by admin. Will continue to work on this.
>>> --
>>> Tatsuo Ishii
>>> SRA OSS, Inc. Japan
>>> English: http://www.sraoss.co.jp/index_en.php
>>> Japanese:http://www.sraoss.co.jp
>>>
>>> > I have moved forward a little bit with this. At this point I have just
>>> > a created necessary infrastructure to deal with the goal. See
>>> > [pgpool-committers: 3127] for more details.
>>> >
>>> > Best regards,
>>> > --
>>> > Tatsuo Ishii
>>> > SRA OSS, Inc. Japan
>>> > English: http://www.sraoss.co.jp/index_en.php
>>> > Japanese:http://www.sraoss.co.jp
>>> >
>>> >> So this is a proposal for pgpool-II 3.6.
>>> >>
>>> >> I already did some discussion on this:
>>> >>
>>> >> From: Tatsuo Ishii <ishii at postgresql.org>
>>> >> Subject: [pgpool-hackers: 1413] Item #11, torward pgpool-II 3.6
>>> >> Date: Fri, 19 Feb 2016 12:03:12 +0900 (JST)
>>> >> Message-ID: <20160219.120312.816223524770393776.t-ishii at sraoss.co.jp>
>>> >>
>>> >> Here is a more or less formal proposal which is replacing it.
>>> >>
>>> >> Goal:
>>> >>
>>> >> Currently pgpool-II kills all child process when fail over (or switch
>>> >> over by pcp_detach_node) occurs. Of course this leads to disconnecting
>>> >> of all existing client connections because the peer process which
>>> >> client is connecting is gone. This proposal is seeking a way to
>>> >> minimize such session disconnections.
>>> >>
>>> >> o Precondition:
>>> >>
>>> >> I assume this proposal is for streaming replication mode only. Maybe
>>> >> we could expand this for other modes in the future. I also assume the
>>> >> broken server is not primary.
>>> >>
>>> >> o Consideration:
>>> >>
>>> >> What is the reason why we need to kill child process? Basically the
>>> >> problem is the retry in the TCP/IP stack layer when the connection
>>> >> goes wrong, for example, the network cable is pulled out. In this case
>>> >> the only way to stop the retry is restarting the process.
>>> >>
>>> >> There are several chances where we could avoid the restarting:
>>> >>
>>> >> 1) Knowing that we are not dealing with a fail over caused by the
>>> >> cabling problem. There are at least two cases we know the problem is
>>> >> not a cabling:
>>> >>
>>> >>  a) the fail over is triggered by pcp_detach_node.
>>> >>
>>> >>  b) the fail over is triggered by posmaster shutdown.
>>> >>
>>> >> For other cases we need to find a way to know that the problem is a
>>> >> cabling or not. Currently we use timeout to detect such that
>>> >> situation. So if we could know if the timeout is occurred or not, then
>>> >> we could know the problem is a cabling or not.
>>> >>
>>> >> 2) Once we succeed in #1, next thing we need to do is, whether a
>>> >> session in question is using the broken server. This is fairly easy
>>> >> because we already have the info on shared memory. If the session uses
>>> >> the broken server, then we need to restart the process. No way. Other
>>> >> case we just close a connection to the broken backend (if any).
>>> >>
>>> >> o Things we need to do:
>>> >>
>>> >> - Invent a way to know if the fail over request is created by
>>> >>   pcp_detach_node. Probably we add a new flag to the fail over request
>>> >>   packet to indicate whether the origin of the request is
>>> >>   pcp_detach_node or not.
>>> >>
>>> >> - The same technique above can be used for the admin PostgreSQL
>>> >>   shutdown case.
>>> >>
>>> >> - Create a API to deal with connections using the broken server.
>>> >>
>>> >> o What are the benefit once above proposal is implemented?
>>> >>
>>> >> - If conditions below are met, the user session can be survives after
>>> fail over.
>>> >>
>>> >>  - Operated in streaming replication mode
>>> >>
>>> >>  - The failed server is not primary
>>> >>
>>> >>  - The session does not connect to the broken broken standby server
>>> >>
>>> >> Comments, opinions?
>>> >> --
>>> >> Tatsuo Ishii
>>> >> SRA OSS, Inc. Japan
>>> >> English: http://www.sraoss.co.jp/index_en.php
>>> >> Japanese:http://www.sraoss.co.jp
>>> >> _______________________________________________
>>> >> pgpool-hackers mailing list
>>> >> pgpool-hackers at pgpool.net
>>> >> http://www.pgpool.net/mailman/listinfo/pgpool-hackers
>>> > _______________________________________________
>>> > pgpool-hackers mailing list
>>> > pgpool-hackers at pgpool.net
>>> > http://www.pgpool.net/mailman/listinfo/pgpool-hackers
>>> _______________________________________________
>>> pgpool-hackers mailing list
>>> pgpool-hackers at pgpool.net
>>> http://www.pgpool.net/mailman/listinfo/pgpool-hackers
>>>
>>
>>
>>
>> --
>> Ahsan Hadi
>> Snr Director Product Development
>> EnterpriseDB Corporation
>> The Enterprise Postgres Company
>>
>> Phone: +92-51-8358874
>> Mobile: +92-333-5162114
>>
>> Website: www.enterprisedb.com
>> EnterpriseDB Blog: http://blogs.enterprisedb.com/
>> Follow us on Twitter: http://www.twitter.com/enterprisedb
>>
>> This e-mail message (and any attachment) is intended for the use of the
>> individual or entity to whom it is addressed. This message contains
>> information from EnterpriseDB Corporation that may be privileged,
>> confidential, or exempt from disclosure under applicable law. If you are
>> not the intended recipient or authorized to receive this for the intended
>> recipient, any use, dissemination, distribution, retention, archiving, or
>> copying of this communication is strictly prohibited. If you have received
>> this e-mail in error, please notify the sender immediately by reply e-mail
>> and delete this message.
>>
> 
> 
> 
> -- 
> Ahsan Hadi
> Snr Director Product Development
> EnterpriseDB Corporation
> The Enterprise Postgres Company
> 
> Phone: +92-51-8358874
> Mobile: +92-333-5162114
> 
> Website: www.enterprisedb.com
> EnterpriseDB Blog: http://blogs.enterprisedb.com/
> Follow us on Twitter: http://www.twitter.com/enterprisedb
> 
> This e-mail message (and any attachment) is intended for the use of the
> individual or entity to whom it is addressed. This message contains
> information from EnterpriseDB Corporation that may be privileged,
> confidential, or exempt from disclosure under applicable law. If you are
> not the intended recipient or authorized to receive this for the intended
> recipient, any use, dissemination, distribution, retention, archiving, or
> copying of this communication is strictly prohibited. If you have received
> this e-mail in error, please notify the sender immediately by reply e-mail
> and delete this message.