<html><body><div style="color:#000; background-color:#fff; font-family:times new roman, new york, times, serif;font-size:14pt"><div>Dear Tatsuo,</div><div><br></div><div>&nbsp; &nbsp; &nbsp; Thanks for your reply, I have followed your recommendations and this time the errorcode e1012 pops up not on the third try but on the second try and&nbsp;<span style="font-size: 14pt;">within 2 seconds of clicking the recovery button.&nbsp;</span></div><div><br></div><div>First Try of Recovery button when primary was down</div><div>Recovery success on 172.16.80.49(when it was down manually) the backup log of 172.16.80.47 is as follows</div><div><br></div><div>******************************************************************</div><div>&nbsp;pg_start_backup&nbsp;</div><div>-----------------</div><div>&nbsp;1/3C000020</div><div>(1 row)</div><div><br></div><div>mkdir: cannot create directory `/usr/local/pgsql/data/pg_xlog': File exists</div><div>NOTICE: &nbsp;WAL
 archiving is not enabled; you must ensure that all required WAL segments are copied through other means to complete the backup</div><div>&nbsp;pg_stop_backup&nbsp;</div><div>----------------</div><div>&nbsp;1/3C0000D8</div><div>(1 row)</div><div>******************************************************************</div><div><br></div><div>Second Try of Recovery button when the new primary was down manually</div><div>Recovery fails on 172.16.80.47(when it was down manually) the backup log of 172.16.80.49 is as follows</div><div><br></div><div>******************************************************************</div><div>/usr/local/pgsql/data/basebackup.sh: line 12: psql: command not found</div><div>mkdir: cannot create directory `/usr/local/pgsql/data/pg_xlog': File exists</div><div>/usr/local/pgsql/data/basebackup.sh: line 33: psql: command not found</div><div>******************************************************************</div><div><br></div><div>The
 basebackup.sh on both servers is as follows with the added script for basebackup.log and uncommented recovery_target_timeline = 'latest'</div><div><br></div><div>*******************************************************************</div><div>#/bin/sh -x</div><div>exec &gt; /tmp/basebackup.log 2&gt;&amp;1</div><div># XXX We assume master and recovery host uses the same port number</div><div>PORT=5432</div><div>master_node_host_name=`hostname`</div><div>master_db_cluster=$1</div><div>recovery_node_host_name=$2</div><div>recovery_db_cluster=$3</div><div>tmp=/tmp/mytemp$$</div><div>trap "rm -f $tmp" 0 1 2 3 15</div><div><br></div><div>psql -p $PORT -c "SELECT pg_start_backup('Streaming Replication', true)" postgres</div><div><br></div><div>rsync -C -a -c --delete --exclude postgresql.conf --exclude postmaster.pid \</div><div>--exclude postmaster.opts --exclude pg_log \</div><div>--exclude recovery.conf --exclude recovery.done \</div><div>--exclude pg_xlog
 \</div><div>$master_db_cluster/ $recovery_node_host_name:$recovery_db_cluster</div><div><br></div><div>ssh -T $recovery_node_host_name mkdir $recovery_db_cluster/pg_xlog</div><div>ssh -T $recovery_node_host_name chmod 700 $recovery_db_cluster/pg_xlog</div><div>ssh -T $recovery_node_host_name rm -f $recovery_db_cluster/recovery.done</div><div><br></div><div>cat &gt; $tmp &lt;&lt;EOF</div><div>recovery_target_timeline = 'latest'</div><div>standby_mode &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;= 'on'</div><div>primary_conninfo &nbsp; &nbsp; &nbsp;= 'host=$master_node_host_name port=$PORT user=postgres'</div><div>trigger_file = '/var/log/pgpool/trigger/trigger_file1'</div><div>EOF</div><div><br></div><div>scp $tmp $recovery_node_host_name:$recovery_db_cluster/recovery.conf</div><div><br></div><div>psql -p $PORT -c "SELECT pg_stop_backup()" postgres</div><div>*******************************************************************</div><div><br></div><div>&nbsp; &nbsp;
 &nbsp; &nbsp;</div><div>&nbsp; &nbsp; &nbsp;Also, the reason for commenting (recovery_target_timeline = 'latest') was it was not mentioned in your "Simple Streaming replication setting with&nbsp;<span style="font-size: 14pt;">pgpool-II(multiple servers version)" http://www.pgpool.net/pgpool-web/contrib_docs/simple_sr_setting2_3.0/ page.&nbsp;</span></div><div>&nbsp; &nbsp; &nbsp;But after a long search on the net i found someone adding the line (recovery_target_timeline = 'latest') so for test purpose i have added it and once it did&nbsp;<span style="font-size: 14pt;">not solve the purpose I had commented it.</span></div><div><br></div><div>&nbsp; &nbsp; &nbsp;Request you to help on the issue ASAP.</div><div><br></div><div>Best Regards,</div><div>Syed Irfan</div><div>Sr Developer</div><div><span></span></div><div><br></div><div></div><div>&nbsp;</div><div>Thanks and Regards,</div><div><span style="font-size: 14pt;">Syed Irfan.</span><br></div><div><span
 style="font-size: 14pt;">Sr. Developer</span><br></div><div><br></div><div class="yahoo_quoted" style="display: block;"> <br> <br> <div style="font-family: 'times new roman', 'new york', times, serif; font-size: 14pt;"> <div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt;"> <div dir="ltr"> <font size="2" face="Arial"> On Tuesday, 28 January 2014 4:52 AM, Tatsuo Ishii &lt;ishii@postgresql.org&gt; wrote:<br> </font> </div>  <div class="y_msg_container">&gt; This is what the recovery.conf looks like.<br clear="none">&gt; *******************************************<br clear="none">&gt; #recovery_target_timeline = 'latest'<br clear="none">&gt; standby_mode &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;= 'on'<br clear="none">&gt; primary_conninfo &nbsp; &nbsp; &nbsp;= 'host=postgres-p.rolta.com port=5432 user=postgres'<br clear="none">&gt; trigger_file = '/var/log/pgpool/trigger/trigger_file1'<br clear="none">&gt;
 ********************************************************<br clear="none"><br clear="none">Why did you remove "recovery_target_timeline = 'latest'"?<br clear="none"><br clear="none">I suggesto to take an execution log of script. You change the very<br clear="none">begging of the script:<br clear="none"><br clear="none">#/bin/sh -x<br clear="none"><br clear="none">to:<br clear="none"><br clear="none">#/bin/sh -x<br clear="none">exec &gt; /tmp/basebackup.log 2&gt;&amp;1<br clear="none"><br clear="none">and please show us the content of /tmp/basebackup.log after execution of pcp_recovery_node.<br clear="none"><br clear="none">Best regards,<br clear="none">--<br clear="none">Tatsuo Ishii<br clear="none">SRA OSS, Inc. Japan<br clear="none">English: <a shape="rect" href="http://www.sraoss.co.jp/index_en.php" target="_blank">http://www.sraoss.co.jp/index_en.php</a><br clear="none">Japanese: <a shape="rect" href="http://www.sraoss.co.jp/"
 target="_blank">http://www.sraoss.co.jp</a><div class="yqt7916395328" id="yqtfd64652"><br clear="none"><br clear="none"><br clear="none">&gt; The basebackup.sh on both postgres databases is as follows<br clear="none">&gt; **************************************************<br clear="none">&gt; <br clear="none">&gt; #/bin/sh -x<br clear="none">&gt; #<br clear="none">&gt; # XXX We assume master and recovery host uses the same port number<br clear="none">&gt; PORT=5432<br clear="none">&gt; master_node_host_name=`hostname`<br clear="none">&gt; master_db_cluster=$1<br clear="none">&gt; recovery_node_host_name=$2<br clear="none">&gt; recovery_db_cluster=$3<br clear="none">&gt; tmp=/tmp/mytemp$$<br clear="none">&gt; trap "rm -f $tmp" 0 1 2 3 15<br clear="none">&gt; <br clear="none">&gt; psql -p $PORT -c "SELECT pg_start_backup('Streaming Replication', true)" postgres<br clear="none">&gt; <br clear="none">&gt; rsync -C -a -c --delete --exclude postgresql.conf
 --exclude postmaster.pid \<br clear="none">&gt; --exclude postmaster.opts --exclude pg_log \<br clear="none">&gt; --exclude recovery.conf --exclude recovery.done \<br clear="none">&gt; --exclude pg_xlog \<br clear="none">&gt; $master_db_cluster/ $recovery_node_host_name:$recovery_db_cluster<br clear="none">&gt; <br clear="none">&gt; ssh -T $recovery_node_host_name mkdir $recovery_db_cluster/pg_xlog<br clear="none">&gt; ssh -T $recovery_node_host_name chmod 700 $recovery_db_cluster/pg_xlog<br clear="none">&gt; ssh -T $recovery_node_host_name rm -f $recovery_db_cluster/recovery.done<br clear="none">&gt; <br clear="none">&gt; cat &gt; $tmp &lt;&lt;EOF<br clear="none">&gt; #recovery_target_timeline = 'latest'<br clear="none">&gt; standby_mode &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;= 'on'<br clear="none">&gt; primary_conninfo &nbsp; &nbsp; &nbsp;= 'host=$master_node_host_name port=$PORT user=postgres'<br clear="none">&gt; trigger_file =
 '/var/log/pgpool/trigger/trigger_file1'<br clear="none">&gt; EOF<br clear="none">&gt; <br clear="none">&gt; scp $tmp $recovery_node_host_name:$recovery_db_cluster/recovery.conf<br clear="none">&gt; <br clear="none">&gt; psql -p $PORT -c "SELECT pg_stop_backup()" postgres<br clear="none">&gt; ***********************************************<br clear="none">&gt; &nbsp;<br clear="none">&gt; Thanks and Regards,<br clear="none">&gt; Syed Irfan.<br clear="none">&gt; <br clear="none">&gt; Sr. Developer<br clear="none">&gt; <br clear="none">&gt; <br clear="none">&gt; <br clear="none">&gt; <br clear="none">&gt; <br clear="none">&gt; On Thursday, 23 January 2014 11:07 PM, Jeff Frost &lt;<a shape="rect" ymailto="mailto:jeff@pgexperts.com" href="mailto:jeff@pgexperts.com">jeff@pgexperts.com</a>&gt; wrote:<br clear="none">&gt;&nbsp; <br clear="none">&gt; <br clear="none">&gt; <br clear="none">&gt; On Jan 23, 2014, at 9:32 AM, Syed Irfan &lt;<a shape="rect"
 ymailto="mailto:syedirfan_77@yahoo.com" href="mailto:syedirfan_77@yahoo.com">syedirfan_77@yahoo.com</a>&gt; wrote:<br clear="none">&gt; <br clear="none">&gt; Dear Tatsuo Ishii,<br clear="none">&gt;&gt;<br clear="none">&gt;&gt;<br clear="none">&gt;&gt;&nbsp; &nbsp; &nbsp; &nbsp;I am still awaiting for your reply on this issue, I have tried your suggestions but still I am unable to successfully run the Recovery process the third time it's surprises me how does it work the first time but same thing fails in the third attempt.?<br clear="none">&gt;&gt;<br clear="none">&gt;&gt;<br clear="none">&gt;&gt;The Postgres log shows as below<br clear="none">&gt;&gt;<br clear="none">&gt;&gt;<br clear="none">&gt;&gt;<br clear="none">&gt;&gt;<br clear="none">&gt;&gt;&gt;&gt; 28038 2014-01-09 21:28:33 BDT FATAL:&nbsp; timeline 35 of the primary does not match recovery target timeline 36<br clear="none">&gt;&gt;&gt;&gt;&gt; 28039 2014-01-09 21:28:38 BDT FATAL:&nbsp;
 timeline 35 of the primary does not match recovery target timeline 36<br clear="none">&gt;&gt;<br clear="none">&gt;&gt;<br clear="none">&gt;&gt;<br clear="none">&gt;&gt;I urgently request you to help me in this impending issue.<br clear="none">&gt;&gt;<br clear="none">&gt; <br clear="none">&gt; This is usually caused by postgres trying to replay WAL files from the wrong source. &nbsp;Did you clean out the pg_xlog directory on the replica before taking the base backup?<br clear="none">&gt; <br clear="none">&gt; What does your recovery.conf look like?</div><br><br></div>  </div> </div>  </div> </div></body></html>