
<head></head><body>Hi all,<br><br>I've been working on a database project that involves using Postgres&nbsp;9.1, PgPool&nbsp;3.5.3, and SaltStack&nbsp;in Amazon EC2, and I apologize ahead of time with the long winded&nbsp;email.<br><br>As it currently stands, I have salt states that automate the setup of new new servers depending on their roles they are given at provision (postgres-master, postgres-slave). When the states run across all the running machines it sets up a working asynchronous&nbsp;streaming replication cluster. &nbsp;Wal-e&nbsp;is also being used for base backups and wal&nbsp;archving to&nbsp;S3.<br><br>For this example I'll be using a master with 5 slaves, the salt states also setup all the tools necessary as well as&nbsp;pgpool, and a&nbsp;config&nbsp;with a unique watchdog priority based on it's role. It's to the point that I check to make sure streaming replication is working correctly before I start pgpool&nbsp;up.<br><br><span style="color: rgb(51, 51, 51); font-family: Consolas, Menlo, Monaco, 'Lucida Console', 'Liberation Mono', 'DejaVu Sans Mono', 'Bitstream Vera Sans Mono', monospace, serif; font-size: 12px; line-height: 24px; orphans: 2; widows: 2;">&nbsp;</span><br><ol class="bash" style="color: rgb(172, 172, 172); margin-right: 0px; margin-bottom: 0px; margin-left: 0px; padding: 0px 0px 0px 55px; font-family: Consolas, Menlo, Monaco, 'Lucida Console', 'Liberation Mono', 'DejaVu Sans Mono', 'Bitstream Vera Sans Mono', monospace, serif; font-size: 12px; font-variant-ligatures: normal; line-height: 24px; orphans: 2; widows: 2; background: rgb(247, 247, 247);"><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">ubuntu@ip-172-31-2-26:~$ sudo salt '*' cmd.run "cat /etc/pgpool2/3.5.3/pgpool.conf| grep wd_priority"</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">ip-172-31-2-7.us-west-1.compute.internal:</div></li><li class="li2" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de2" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">&nbsp; &nbsp;wd_priority = 1</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">&nbsp; &nbsp;wd_priority = 5</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">&nbsp; &nbsp;wd_priority = 3</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">ip-172-31-2-6.us-west-1.compute.internal:</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">&nbsp; &nbsp;wd_priority = 1</div></li><li class="li2" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de2" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">&nbsp; &nbsp;wd_priority = 5</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">&nbsp; &nbsp;wd_priority = 2</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">ip-172-31-2-8.us-west-1.compute.internal:</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">&nbsp; &nbsp;wd_priority = 1</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">&nbsp; &nbsp;wd_priority = 1</div></li><li class="li2" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de2" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">&nbsp; &nbsp;wd_priority = 4</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">ip-172-31-2-9.us-west-1.compute.internal:</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">&nbsp; &nbsp;wd_priority = 1</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">&nbsp; &nbsp;wd_priority = 3</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">&nbsp; &nbsp;wd_priority = 5</div></li><li class="li2" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de2" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">ip-172-31-2-4.us-west-1.compute.internal:</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">&nbsp; &nbsp;wd_priority = 1</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">&nbsp; &nbsp;wd_priority = 6</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">&nbsp; &nbsp;wd_priority = 6</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">ip-172-31-2-5.us-west-1.compute.internal:</div></li><li class="li2" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de2" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">&nbsp; &nbsp;wd_priority = 1</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">&nbsp; &nbsp;wd_priority = 4</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background: rgb(248, 248, 248);">&nbsp; &nbsp;wd_priority = 1</div></li></ol><br>it took me awhile to get wd_priority to spit out a unique number based on it's role, that's why there's multiple entries. But let's note here that the postgres master in this case is 172.31.2.4, since my logic is set to give the master the highest number for watch dog priority, with 172.31.2.5 being next in line to take over the VIP in the event that the machine or pgpool fails.<br><br>Now I check to make sure that that 172.31.2.4 has actually been designated as master&nbsp;<br><br><ol class="bash" style="color: rgb(172, 172, 172); margin-right: 0px; margin-bottom: 0px; margin-left: 0px; padding: 0px 0px 0px 55px; font-family: Consolas, Menlo, Monaco, 'Lucida Console', 'Liberation Mono', 'DejaVu Sans Mono', 'Bitstream Vera Sans Mono', monospace, serif; font-size: 12px; font-variant-ligatures: normal; line-height: 24px; orphans: 2; widows: 2; background: rgb(247, 247, 247);"><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">ubuntu@ip-172-31-2-26:~$&nbsp;sudo&nbsp;salt&nbsp;-G&nbsp;'roles:postgres-master'&nbsp;test.ping</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background: rgb(248, 248, 248);">ip-172-31-2-4.us-west-1.compute.internal:</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">&nbsp; &nbsp; True</div></li></ol><div><br></div><div>Looks like salt thinks so, now let's make sure replication is setup correctly<br><br><ol class="bash" style="color: rgb(172, 172, 172); margin-right: 0px; margin-bottom: 0px; margin-left: 0px; padding: 0px 0px 0px 55px; font-family: Consolas, Menlo, Monaco, 'Lucida Console', 'Liberation Mono', 'DejaVu Sans Mono', 'Bitstream Vera Sans Mono', monospace, serif; font-size: 12px; font-variant-ligatures: normal; line-height: 24px; orphans: 2; widows: 2; background: rgb(247, 247, 247);"><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">ubuntu@ip-172-31-2-4:~$ sudo -u postgres psql</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">psql (9.1.22)</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">Type "help" for help.</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">&nbsp;</div></li><li class="li2" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de2" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">postgres=# select * from pg_stat_replication;</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">procpid | usesysid | &nbsp; usename &nbsp; | application_name | client_addr | client_hostname | client_port | &nbsp; &nbsp; &nbsp; &nbsp; backend_start &nbsp; &nbsp; &nbsp; &nbsp; | &nbsp; state &nbsp; | sent_location | write_location | flush_location | replay_location | sync_priority | sync_state</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">---------+----------+-------------+------------------+-------------+-----------------+-------------+-------------------------------+-----------+---------------+----------------+----------------+-----------------+---------------+------------</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">&nbsp; 28655 | &nbsp; &nbsp;16384 | replication | walreceiver &nbsp; &nbsp; &nbsp;| 172.31.2.5 &nbsp;| &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; | &nbsp; &nbsp; &nbsp; 44840 | 2016-08-02 22:21:34.352537+00 | streaming | 0/6000000 &nbsp; &nbsp; | 0/6000000 &nbsp; &nbsp; &nbsp;| 0/6000000 &nbsp; &nbsp; &nbsp;| 0/6000000 &nbsp; &nbsp; &nbsp; | &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 0 | async</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">&nbsp; 28901 | &nbsp; &nbsp;16384 | replication | walreceiver &nbsp; &nbsp; &nbsp;| 172.31.2.6 &nbsp;| &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; | &nbsp; &nbsp; &nbsp; 56999 | 2016-08-02 22:26:35.648484+00 | streaming | 0/6000000 &nbsp; &nbsp; | 0/6000000 &nbsp; &nbsp; &nbsp;| 0/6000000 &nbsp; &nbsp; &nbsp;| 0/6000000 &nbsp; &nbsp; &nbsp; | &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 0 | async</div></li><li class="li2" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de2" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">&nbsp; 28902 | &nbsp; &nbsp;16384 | replication | walreceiver &nbsp; &nbsp; &nbsp;| 172.31.2.8 &nbsp;| &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; | &nbsp; &nbsp; &nbsp; 55492 | 2016-08-02 22:26:35.751934+00 | streaming | 0/6000000 &nbsp; &nbsp; | 0/6000000 &nbsp; &nbsp; &nbsp;| 0/6000000 &nbsp; &nbsp; &nbsp;| 0/6000000 &nbsp; &nbsp; &nbsp; | &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 0 | async</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">&nbsp; 28903 | &nbsp; &nbsp;16384 | replication | walreceiver &nbsp; &nbsp; &nbsp;| 172.31.2.7 &nbsp;| &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; | &nbsp; &nbsp; &nbsp; 48535 | 2016-08-02 22:26:35.786888+00 | streaming | 0/6000000 &nbsp; &nbsp; | 0/6000000 &nbsp; &nbsp; &nbsp;| 0/6000000 &nbsp; &nbsp; &nbsp;| 0/6000000 &nbsp; &nbsp; &nbsp; | &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 0 | async</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background: rgb(248, 248, 248);">&nbsp; 28904 | &nbsp; &nbsp;16384 | replication | walreceiver &nbsp; &nbsp; &nbsp;| 172.31.2.9 &nbsp;| &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; | &nbsp; &nbsp; &nbsp; 51240 | 2016-08-02 22:26:35.818891+00 | streaming | 0/6000000 &nbsp; &nbsp; | 0/6000000 &nbsp; &nbsp; &nbsp;| 0/6000000 &nbsp; &nbsp; &nbsp;| 0/6000000 &nbsp; &nbsp; &nbsp; | &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 0 | async</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">(5 rows)</div></li></ol></div><div><br></div><div>(Sorry for not using extended format here) we can see that the master has the slaves replicating correclty, now I'm going to start up the pgpool2 service across my machines.<br><br><ol class="bash" style="color: rgb(172, 172, 172); margin-right: 0px; margin-bottom: 0px; margin-left: 0px; padding: 0px 0px 0px 55px; font-family: Consolas, Menlo, Monaco, 'Lucida Console', 'Liberation Mono', 'DejaVu Sans Mono', 'Bitstream Vera Sans Mono', monospace, serif; font-size: 12px; font-variant-ligatures: normal; line-height: 24px; orphans: 2; widows: 2; background: rgb(247, 247, 247);"><li class="li2" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de2" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">ubuntu@ip-172-31-2-26:~$&nbsp;sudo&nbsp;salt&nbsp;'*'&nbsp;cmd.run&nbsp;"service pgpool2 start"</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">ip-172-31-2-7.us-west-1.compute.internal:</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">&nbsp; &nbsp; &nbsp;*&nbsp;Starting pgpool-II pgpool</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">&nbsp; &nbsp; FD - Starting pgpool-II by executing:</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">&nbsp; &nbsp;&nbsp;/usr/sbin/pgpool&nbsp;-n&nbsp;-f&nbsp;/etc/pgpool2/3.5.3/pgpool.conf&nbsp;-F&nbsp;/etc/pgpool2/3.5.3/pcp.conf&nbsp;-a&nbsp;/etc/pgpool2/3.5.3/pool_hba.conf&nbsp;&gt;&gt;/var/log/pgpool/pgpool.log&nbsp;2&gt;&amp;1&nbsp;&amp;</div></li><li class="li2" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de2" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">&nbsp; &nbsp; &nbsp; &nbsp;...done.</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">ip-172-31-2-8.us-west-1.compute.internal:</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">&nbsp; &nbsp; &nbsp;*&nbsp;Starting pgpool-II pgpool</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">&nbsp; &nbsp; FD - Starting pgpool-II by executing:</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">&nbsp; &nbsp;&nbsp;/usr/sbin/pgpool&nbsp;-n&nbsp;-f&nbsp;/etc/pgpool2/3.5.3/pgpool.conf&nbsp;-F&nbsp;/etc/pgpool2/3.5.3/pcp.conf&nbsp;-a&nbsp;/etc/pgpool2/3.5.3/pool_hba.conf&nbsp;&gt;&gt;/var/log/pgpool/pgpool.log&nbsp;2&gt;&amp;1&nbsp;&amp;</div></li><li class="li2" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de2" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">&nbsp; &nbsp; &nbsp; &nbsp;...done.</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">ip-172-31-2-9.us-west-1.compute.internal:</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">&nbsp; &nbsp; &nbsp;*&nbsp;Starting pgpool-II pgpool</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">&nbsp; &nbsp; FD - Starting pgpool-II by executing:</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">&nbsp; &nbsp;&nbsp;/usr/sbin/pgpool&nbsp;-n&nbsp;-f&nbsp;/etc/pgpool2/3.5.3/pgpool.conf&nbsp;-F&nbsp;/etc/pgpool2/3.5.3/pcp.conf&nbsp;-a&nbsp;/etc/pgpool2/3.5.3/pool_hba.conf&nbsp;&gt;&gt;/var/log/pgpool/pgpool.log&nbsp;2&gt;&amp;1&nbsp;&amp;</div></li><li class="li2" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de2" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">&nbsp; &nbsp; &nbsp; &nbsp;...done.</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">ip-172-31-2-5.us-west-1.compute.internal:</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">&nbsp; &nbsp; &nbsp;*&nbsp;Starting pgpool-II pgpool</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">&nbsp; &nbsp; FD - Starting pgpool-II by executing:</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">&nbsp; &nbsp;&nbsp;/usr/sbin/pgpool&nbsp;-n&nbsp;-f&nbsp;/etc/pgpool2/3.5.3/pgpool.conf&nbsp;-F&nbsp;/etc/pgpool2/3.5.3/pcp.conf&nbsp;-a&nbsp;/etc/pgpool2/3.5.3/pool_hba.conf&nbsp;&gt;&gt;/var/log/pgpool/pgpool.log&nbsp;2&gt;&amp;1&nbsp;&amp;</div></li><li class="li2" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de2" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">&nbsp; &nbsp; &nbsp; &nbsp;...done.</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">ip-172-31-2-6.us-west-1.compute.internal:</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">&nbsp; &nbsp; &nbsp;*&nbsp;Starting pgpool-II pgpool</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">&nbsp; &nbsp; FD - Starting pgpool-II by executing:</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">&nbsp; &nbsp;&nbsp;/usr/sbin/pgpool&nbsp;-n&nbsp;-f&nbsp;/etc/pgpool2/3.5.3/pgpool.conf&nbsp;-F&nbsp;/etc/pgpool2/3.5.3/pcp.conf&nbsp;-a&nbsp;/etc/pgpool2/3.5.3/pool_hba.conf&nbsp;&gt;&gt;/var/log/pgpool/pgpool.log&nbsp;2&gt;&amp;1&nbsp;&amp;</div></li><li class="li2" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de2" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">&nbsp; &nbsp; &nbsp; &nbsp;...done.</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">ip-172-31-2-4.us-west-1.compute.internal:</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">&nbsp; &nbsp; &nbsp;*&nbsp;Starting pgpool-II pgpool</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">&nbsp; &nbsp; FD - Starting pgpool-II by executing:</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">&nbsp; &nbsp;&nbsp;/usr/sbin/pgpool&nbsp;-n&nbsp;-f&nbsp;/etc/pgpool2/3.5.3/pgpool.conf&nbsp;-F&nbsp;/etc/pgpool2/3.5.3/pcp.conf&nbsp;-a&nbsp;/etc/pgpool2/3.5.3/pool_hba.conf&nbsp;&gt;&gt;/var/log/pgpool/pgpool.log&nbsp;2&gt;&amp;1&nbsp;&amp;</div></li><li class="li2" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de2" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background: rgb(248, 248, 248);">&nbsp; &nbsp; &nbsp; &nbsp;...done.</div></li></ol></div><div><br></div><div>In salt i've set my designated Virtual IP for watchdog to be 172.31.2.30, all the appropriate IAM roles and security groups are applied to the machines, so let's see if I can connect to it from my salt server and see if it can see the other pool nodes. (note here i have postgresql running on port 5433 on each machine and pgpool is running as 5432)<br><br><ol class="bash" style="color: rgb(172, 172, 172); margin-right: 0px; margin-bottom: 0px; margin-left: 0px; padding: 0px 0px 0px 55px; font-family: Consolas, Menlo, Monaco, 'Lucida Console', 'Liberation Mono', 'DejaVu Sans Mono', 'Bitstream Vera Sans Mono', monospace, serif; font-size: 12px; font-variant-ligatures: normal; line-height: 24px; orphans: 2; widows: 2; background: rgb(247, 247, 247);"><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">ubuntu@ip-172-31-2-26:~$ psql -U postgres -h 172.31.2.30</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">psql (9.1.21, server 9.1.22)</div></li><li class="li2" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de2" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">Type "help" for help.</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">&nbsp;</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">postgres=# show pool_nodes;</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">node_id | &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; hostname &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; | port | status | lb_weight | &nbsp;role &nbsp; | select_cnt</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">---------+------------------------------------------+------+--------+-----------+---------+------------</div></li><li class="li2" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de2" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">0 &nbsp; &nbsp; &nbsp; | ip-172-31-2-6.us-west-1.compute.internal | 5433 | 2 &nbsp; &nbsp; &nbsp;| 0.166667 &nbsp;| standby | 0</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">1 &nbsp; &nbsp; &nbsp; | ip-172-31-2-5.us-west-1.compute.internal | 5433 | 2 &nbsp; &nbsp; &nbsp;| 0.166667 &nbsp;| standby | 0</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">2 &nbsp; &nbsp; &nbsp; | ip-172-31-2-9.us-west-1.compute.internal | 5433 | 2 &nbsp; &nbsp; &nbsp;| 0.166667 &nbsp;| standby | 0</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">3 &nbsp; &nbsp; &nbsp; | ip-172-31-2-4.us-west-1.compute.internal | 5433 | 2 &nbsp; &nbsp; &nbsp;| 0.166667 &nbsp;| primary | 0</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">4 &nbsp; &nbsp; &nbsp; | ip-172-31-2-8.us-west-1.compute.internal | 5433 | 2 &nbsp; &nbsp; &nbsp;| 0.166667 &nbsp;| standby | 0</div></li><li class="li2" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de2" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">5 &nbsp; &nbsp; &nbsp; | ip-172-31-2-7.us-west-1.compute.internal | 5433 | 2 &nbsp; &nbsp; &nbsp;| 0.166667 &nbsp;| standby | 0</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">(6 rows)</div></li></ol></div><div><br></div><div>So we can connect to our pgpool cluster, it can see the other nodes in the cluster, and we know streaming replication is working correctly. So far so good, now let's try to cause a failover event. In this case, I will go into the ec2 control panel and power down the master. If everything works out okay, The VIP should be moved to the next in line with the lowest watch dog priority, and pgpool should elect a new master and promote it as such. &nbsp;Now that the server is powered down let's check pool_nodes again.<br><br><ol class="bash" style="color: rgb(172, 172, 172); margin-right: 0px; margin-bottom: 0px; margin-left: 0px; padding: 0px 0px 0px 55px; font-family: Consolas, Menlo, Monaco, 'Lucida Console', 'Liberation Mono', 'DejaVu Sans Mono', 'Bitstream Vera Sans Mono', monospace, serif; font-size: 12px; font-variant-ligatures: normal; line-height: 24px; orphans: 2; widows: 2; background: rgb(247, 247, 247);"><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">postgres=# show pool_nodes;</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">node_id | &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; hostname &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; | port | status | lb_weight | &nbsp;role &nbsp; | select_cnt</div></li><li class="li2" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de2" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">---------+------------------------------------------+------+--------+-----------+---------+------------</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">0 &nbsp; &nbsp; &nbsp; | ip-172-31-2-6.us-west-1.compute.internal | 5433 | 2 &nbsp; &nbsp; &nbsp;| 0.166667 &nbsp;| primary | 0</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">1 &nbsp; &nbsp; &nbsp; | ip-172-31-2-5.us-west-1.compute.internal | 5433 | 2 &nbsp; &nbsp; &nbsp;| 0.166667 &nbsp;| standby | 0</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">2 &nbsp; &nbsp; &nbsp; | ip-172-31-2-9.us-west-1.compute.internal | 5433 | 2 &nbsp; &nbsp; &nbsp;| 0.166667 &nbsp;| standby | 0</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">3 &nbsp; &nbsp; &nbsp; | ip-172-31-2-4.us-west-1.compute.internal | 5433 | 3 &nbsp; &nbsp; &nbsp;| 0.166667 &nbsp;| standby | 0</div></li><li class="li2" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de2" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">4 &nbsp; &nbsp; &nbsp; | ip-172-31-2-8.us-west-1.compute.internal | 5433 | 2 &nbsp; &nbsp; &nbsp;| 0.166667 &nbsp;| standby | 0</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">5 &nbsp; &nbsp; &nbsp; | ip-172-31-2-7.us-west-1.compute.internal | 5433 | 2 &nbsp; &nbsp; &nbsp;| 0.166667 &nbsp;| standby | 0</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background: rgb(248, 248, 248);">(6 rows)</div></li></ol><br>Now we can see that our old master, 172.31.2.4 is now unavailable, and a new master, 172.31.2.6 has been elected as a new primary. &nbsp;Here's where I'm first really confused as to what to do next. It shows the status of all the other slaves as a status of 2, or available, however the other slaves are currently not replicating from the new master as shown below, why is this?&nbsp;<br><br><ol class="bash" style="color: rgb(172, 172, 172); margin-right: 0px; margin-bottom: 0px; margin-left: 0px; padding: 0px 0px 0px 55px; font-family: Consolas, Menlo, Monaco, 'Lucida Console', 'Liberation Mono', 'DejaVu Sans Mono', 'Bitstream Vera Sans Mono', monospace, serif; font-size: 12px; font-variant-ligatures: normal; line-height: 24px; orphans: 2; widows: 2; background: rgb(247, 247, 247);"><li class="li2" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de2" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">ubuntu@ip-172-31-2-6:~$ sudo -u postgres psql</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">psql (9.1.22)</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">Type "help" for help.</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">&nbsp;</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">postgres=# select * from pg_stat_replication;</div></li><li class="li2" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de2" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">procpid | usesysid | usename | application_name | client_addr | client_hostname | client_port | backend_start | state | sent_location | write_location | flush_location | replay_location | sync_priority | sync_state</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">---------+----------+---------+------------------+-------------+-----------------+-------------+---------------+-------+---------------+----------------+----------------+-----------------+---------------+------------</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">(0 rows)</div></li></ol></div><br>Now let's say I want to have the slaves replicating from the new master, in this case let's start replication on 172.31.2.5, I have a script that setups recovery.conf based on a few flags that are passed to it<br><br><ol class="bash" style="color: rgb(172, 172, 172); margin-right: 0px; margin-bottom: 0px; margin-left: 0px; padding: 0px 0px 0px 55px; font-family: Consolas, Menlo, Monaco, 'Lucida Console', 'Liberation Mono', 'DejaVu Sans Mono', 'Bitstream Vera Sans Mono', monospace, serif; font-size: 12px; font-variant-ligatures: normal; line-height: 24px; orphans: 2; widows: 2; background: rgb(247, 247, 247);"><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">ubuntu@ip-172-31-2-5:~$&nbsp;sudo&nbsp;-u&nbsp;postgres&nbsp;/etc/postgresql/9.1/main/replscripts/initiate_replication.sh&nbsp;-f&nbsp;-H&nbsp;ip-172-31-2-6.us-west-1.compute.internal&nbsp;-P&nbsp;5433&nbsp;-p&nbsp;replication</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">initiate_replication - Start</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">DEBUG: The script will be executed with the following arguments:</div></li><li class="li2" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de2" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">DEBUG:&nbsp;--trigger-file=/etc/postgresql/9.1/main/im_the_master</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">DEBUG: --standby_file=/etc/postgresql/9.1/main/im_slave</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">DEBUG:&nbsp;--primary-host=ip-172-31-2-6.us-west-1.compute.internal</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">DEBUG:&nbsp;--primary-port=5433</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">DEBUG:&nbsp;--slot-name=ip_172_31_2_5</div></li><li class="li2" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de2" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">DEBUG:&nbsp;--user=replication</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">DEBUG:&nbsp;--password=replication</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">DEBUG:&nbsp;--force</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">INFO: Checking&nbsp;if&nbsp;trigger&nbsp;file&nbsp;exists...</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">INFO: Checking&nbsp;if&nbsp;standby&nbsp;file&nbsp;exists...</div></li><li class="li2" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de2" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">INFO: Standby&nbsp;file&nbsp;not found. Creating new one...</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">INFO: Ensuring replication user and password&nbsp;in&nbsp;password&nbsp;file&nbsp;(.pgpass)...</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">&nbsp;*&nbsp;Stopping PostgreSQL&nbsp;9.1&nbsp;database server &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;[&nbsp;OK&nbsp;]</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">INFO: Deleting old data...</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">INFO: Getting the initial backup...</div></li><li class="li2" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de2" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">wal_e.main &nbsp; INFO &nbsp; &nbsp; MSG: starting WAL-E</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">&nbsp; &nbsp; &nbsp; &nbsp; DETAIL: The subcommand is&nbsp;"backup-fetch".</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">&nbsp; &nbsp; &nbsp; &nbsp; STRUCTURED:&nbsp;time=2016-08-02T23:00:15.870621-00&nbsp;pid=4027</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">wal_e.worker.s3.s3_worker INFO &nbsp; &nbsp; MSG: beginning partition download</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">&nbsp; &nbsp; &nbsp; &nbsp; DETAIL: The partition being downloaded is part_00000000.tar.lzo.</div></li><li class="li2" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de2" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">&nbsp; &nbsp; &nbsp; &nbsp; HINT: The absolute S3 key is wal-e-stage/basebackups_005/base_000000010000000000000002_00000032/tar_partitions/part_00000000.tar.lzo.</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">&nbsp; &nbsp; &nbsp; &nbsp; STRUCTURED:&nbsp;time=2016-08-02T23:00:22.453123-00&nbsp;pid=4027</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">INFO: Creating recovery.conf file...</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">INFO: Removing old postgresql.conf file...</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">INFO: Copying new postgresql.conf file...</div></li><li class="li2" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de2" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">INFO: Starting postgresql service...</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">&nbsp;*&nbsp;Starting PostgreSQL&nbsp;9.1&nbsp;database server &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;[&nbsp;OK&nbsp;]</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">initiate_replication - Done!</div></li></ol><br>Now let's verify on the new master that it has a new slave started.<br><br><ol class="bash" style="color: rgb(172, 172, 172); margin-right: 0px; margin-bottom: 0px; margin-left: 0px; padding: 0px 0px 0px 55px; font-family: Consolas, Menlo, Monaco, 'Lucida Console', 'Liberation Mono', 'DejaVu Sans Mono', 'Bitstream Vera Sans Mono', monospace, serif; font-size: 12px; font-variant-ligatures: normal; line-height: 24px; orphans: 2; widows: 2; background: rgb(247, 247, 247);"><li class="li2" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de2" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">Now&nbsp;let's check the master</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">ubuntu@ip-172-31-2-6:~$ sudo -u postgres psql</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">psql (9.1.22)</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">Type "help" for help.</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">&nbsp;</div></li><li class="li2" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de2" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">postgres=# select * from pg_stat_replication;</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">procpid | usesysid | &nbsp; usename &nbsp; | application_name | client_addr | client_hostname | client_port | &nbsp; &nbsp; &nbsp; &nbsp;backend_start &nbsp; &nbsp; &nbsp; &nbsp; | &nbsp; state &nbsp; | sent_location | write_location | flush_location | replay_location | sync_priority | sync_state</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">---------+----------+-------------+------------------+-------------+-----------------+-------------+------------------------------+-----------+---------------+----------------+----------------+-----------------+---------------+------------</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">&nbsp; &nbsp;3368 | &nbsp; &nbsp;16384 | replication | walreceiver &nbsp; &nbsp; &nbsp;| 172.31.2.5 &nbsp;| &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; | &nbsp; &nbsp; &nbsp; 49348 | 2016-08-02 23:00:30.00704+00 | streaming | 0/C000000 &nbsp; &nbsp; | 0/C000000 &nbsp; &nbsp; &nbsp;| 0/C000000 &nbsp; &nbsp; &nbsp;| 0/C000000 &nbsp; &nbsp; &nbsp; | &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 0 | async</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background: rgb(248, 248, 248);">(1 row)</div></li></ol><br>(Again sorry for not using extended display here) We can see that our master has a slave readded back to it, so what is the output from pgpool now?<br><br><ol class="bash" style="color: rgb(172, 172, 172); margin-right: 0px; margin-bottom: 0px; margin-left: 0px; padding: 0px 0px 0px 55px; font-family: Consolas, Menlo, Monaco, 'Lucida Console', 'Liberation Mono', 'DejaVu Sans Mono', 'Bitstream Vera Sans Mono', monospace, serif; font-size: 12px; font-variant-ligatures: normal; line-height: 24px; orphans: 2; widows: 2; background: rgb(247, 247, 247);"><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">ubuntu@ip-172-31-2-26:~$&nbsp;psql&nbsp;-U&nbsp;postgres&nbsp;-h&nbsp;172.31.2.30</div></li><li class="li2" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de2" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">psql&nbsp;(9.1.21, server 9.1.22)</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">Type&nbsp;"help"&nbsp;for&nbsp;help.</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">&nbsp;</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">postgres=# show pool_nodes;</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">&nbsp;node_id&nbsp;|&nbsp;&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;hostname&nbsp;&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;|&nbsp;port&nbsp;|&nbsp;status&nbsp;|&nbsp;lb_weight&nbsp;|&nbsp;&nbsp;role &nbsp;&nbsp;|&nbsp;select_cnt</div></li><li class="li2" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de2" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">---------+------------------------------------------+------+--------+-----------+---------+------------</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">&nbsp;0&nbsp;&nbsp; &nbsp; &nbsp;&nbsp;|&nbsp;ip-172-31-2-6.us-west-1.compute.internal&nbsp;|&nbsp;5433&nbsp;|&nbsp;2&nbsp;&nbsp; &nbsp; &nbsp;|&nbsp;0.166667&nbsp;&nbsp;|&nbsp;primary&nbsp;|&nbsp;0</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">&nbsp;1&nbsp;&nbsp; &nbsp; &nbsp;&nbsp;|&nbsp;ip-172-31-2-5.us-west-1.compute.internal&nbsp;|&nbsp;5433&nbsp;|&nbsp;3&nbsp;&nbsp; &nbsp; &nbsp;|&nbsp;0.166667&nbsp;&nbsp;|&nbsp;standby&nbsp;|&nbsp;0</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">&nbsp;2&nbsp;&nbsp; &nbsp; &nbsp;&nbsp;|&nbsp;ip-172-31-2-9.us-west-1.compute.internal&nbsp;|&nbsp;5433&nbsp;|&nbsp;2&nbsp;&nbsp; &nbsp; &nbsp;|&nbsp;0.166667&nbsp;&nbsp;|&nbsp;standby&nbsp;|&nbsp;0</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">&nbsp;3&nbsp;&nbsp; &nbsp; &nbsp;&nbsp;|&nbsp;ip-172-31-2-4.us-west-1.compute.internal&nbsp;|&nbsp;5433&nbsp;|&nbsp;3&nbsp;&nbsp; &nbsp; &nbsp;|&nbsp;0.166667&nbsp;&nbsp;|&nbsp;standby&nbsp;|&nbsp;0</div></li><li class="li2" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de2" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">&nbsp;4&nbsp;&nbsp; &nbsp; &nbsp;&nbsp;|&nbsp;ip-172-31-2-8.us-west-1.compute.internal&nbsp;|&nbsp;5433&nbsp;|&nbsp;2&nbsp;&nbsp; &nbsp; &nbsp;|&nbsp;0.166667&nbsp;&nbsp;|&nbsp;standby&nbsp;|&nbsp;0</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background-image: initial; background-attachment: initial; background-size: initial; background-origin: initial; background-clip: initial; background-position: initial; background-repeat: initial;">&nbsp;5&nbsp;&nbsp; &nbsp; &nbsp;&nbsp;|&nbsp;ip-172-31-2-7.us-west-1.compute.internal&nbsp;|&nbsp;5433&nbsp;|&nbsp;2&nbsp;&nbsp; &nbsp; &nbsp;|&nbsp;0.166667&nbsp;&nbsp;|&nbsp;standby&nbsp;|&nbsp;0</div></li><li class="li1" style="-webkit-user-select: none; margin: 0px 0px 0px -6px; background: rgb(255, 255, 255);"><div class="de1" style="-webkit-user-select: text; padding: 0px 8px; vertical-align: top; color: rgb(51, 51, 51); border-left-width: 1px; border-left-style: solid; border-left-color: rgb(221, 221, 221); margin: 0px 0px 0px -7px; position: relative; background: rgb(248, 248, 248);">(6&nbsp;rows)</div></li></ol><br>Looks like the master is still available, however the slave we have pointed to the new master now has a status of 3, while the other defunct slaves are still showing a status of 2. Is there any particular reason for this? Is there some sort of task I'm not performing that needs to be done? So really it's only two questions Why do defunct slaves show an available status, and how to I repair slaves to point to the promoted master and make them available in pgpool?&nbsp;<br><br>Thank you all,<br><br><!-- <signature> --><span style="font-weight: 600; font-family: Nylas-Pro, Helvetica, 'Lucidia Grande', sans-serif; font-size: 14.5px; line-height: 21.75px; color: rgb(102, 102, 102);">Ryan Fraser</span><div dir="ltr" style="max-width: 100%; font-family: Nylas-Pro, Helvetica, 'Lucidia Grande', sans-serif; font-size: 14.5px; line-height: 21.75px; color: rgb(136, 136, 136);"><div>DevOps Engineer</div><p style="margin-bottom: 0.0001pt; background-image: initial; background-repeat: initial;"><a href="http://www.redshelf.com/" target="_blank" title="http://www.redshelf.com/" style="color: rgb(17, 85, 204);"><img src="https://lh6.googleusercontent.com/-ZHUYf7dQ1C8/UhZLIsRHoNI/AAAAAAAAACg/dtOYhXEcVeI/w800-h800/Red+Shelf+Horizontal+Logo+Transparent+(w-+tagline)+(1).png" width="96" height="34" alt=" RedShelf" style="max-width: 100vw; max-height: 35.4167vw;"></a></p><a value="+13128788586" style="color: rgb(17, 85, 204);">312.878.8586</a><br><a href="mailto:ryan@redshelf.com" target="_blank" title="mailto:ryan@redshelf.com" style="color: rgb(16, 129, 247);">ryan@redshelf.com</a></div><!-- </signature> --></body>