[pgpool-general: 8637] The cluster fails over even when failover_on_backend_error = off

[Ron]

PgPool II 4.3.5 and Postgresql 13.10.

Three nodes: FISPCCPGS302a, b & c.

"a" has the primary database, "b" has the replica, and "c" has the VIP (but 
no database).

One or more times a day, on each node, health_check times out when trying to 
reach one or two of the others.  (I'm working on figuring out why that happens.)

What else happens is that nodes "b" and "c" will vote to failover from "a" 
to "b", even though failover_on_backend_error = off.

Why does that happen? 
https://www.pgpool.net/docs/pgpool-II-4.3.5/en/html/runtime-config-failover.html 
says it should just "report an error and disconnect the session in case of 
such errors."

Is it because "if no client is connected, Pgpool-II will not detect the 
event even if the parameter is on"?  (This is a pre-production system, so 
the application servers aren't talking to it.)

Attached is a log file extract and PGPOOL SHOW command.

-- 
Born in Arizona, moved to Babylonia.
-------------- next part --------------
https://www.pgpool.net/docs/pgpool-II-4.3.5/en/html/runtime-config-failover.html

[postgres at FISPCCPGS302c ~]$ psql -p9999
psql (13.10)
Type "help" for help.

postgres=#
postgres=# pgpool show failover_on_backend_error;
 failover_on_backend_error
---------------------------
 off
(1 row)

2023-03-13 16:57:12.656: health_check0 pid 431235: LOG:  failed to connect to PostgreSQL server on "FISPCCPGS302a:5432", timed out
2023-03-13 16:57:12.656: health_check0 pid 431235: LOG:  health check failed on node 0 (timeout:0)
2023-03-13 16:57:12.656: health_check0 pid 431235: LOG:  received degenerate backend request for node_id: 0 from pid [431235]
2023-03-13 16:57:12.656: watchdog pid 431195: LOG:  watchdog is processing the failover command [DEGENERATE_BACKEND_REQUEST] received from local pgpool-II on IPC interface
2023-03-13 16:57:12.656: watchdog pid 431195: LOG:  failover requires the majority vote, waiting for consensus
2023-03-13 16:57:12.656: watchdog pid 431195: DETAIL:  failover request noted
2023-03-13 16:57:12.656: watchdog pid 431195: LOG:  failover command [DEGENERATE_BACKEND_REQUEST] request from pgpool-II node "FISPCCPGS302c:9999 Linux FISPCCPGS302c" is queued, waiting for the confirmation from other nodes
2023-03-13 16:57:12.656: health_check0 pid 431235: LOG:  degenerate backend request for node_id: 0 from pid [431235], will be handled by watchdog, which is building consensus for request
2023-03-13 16:57:15.934: watchdog pid 431195: LOG:  watchdog received the failover command from remote pgpool-II node "FISPCCPGS302b:9999 Linux FISPCCPGS302b"
2023-03-13 16:57:15.934: watchdog pid 431195: LOG:  watchdog is processing the failover command [DEGENERATE_BACKEND_REQUEST] received from FISPCCPGS302b:9999 Linux FISPCCPGS302b
2023-03-13 16:57:15.934: watchdog pid 431195: LOG:  we have got the consensus to perform the failover
2023-03-13 16:57:15.934: watchdog pid 431195: DETAIL:  2 node(s) voted in the favor
2023-03-13 16:57:15.934: watchdog pid 431195: LOG:  received degenerate backend request for node_id: 0 from pid [431195]