<html><body><div style="color:#000; background-color:#fff; font-family:times new roman, new york, times, serif;font-size:14pt"><div style="RIGHT: auto"><SPAN style="RIGHT: auto">Dear Tatsuo,</SPAN></div>
<div style="RIGHT: auto"><SPAN style="RIGHT: auto"></SPAN> </div>
<div style="BACKGROUND-COLOR: transparent; FONT-STYLE: normal; FONT-FAMILY: times new roman, new york, times, serif; COLOR: rgb(0,0,0); FONT-SIZE: 19px; RIGHT: auto"><SPAN style="RIGHT: auto"> The error e1012 popped up on the third try once again on my two physical systems and the basebackup.log says the following</SPAN></div>
<div style="BACKGROUND-COLOR: transparent; FONT-STYLE: normal; FONT-FAMILY: times new roman, new york, times, serif; COLOR: rgb(0,0,0); FONT-SIZE: 19px; RIGHT: auto"><SPAN style="RIGHT: auto">*************************</SPAN></div><SPAN style="RIGHT: auto"><SPAN style="RIGHT: auto" lang=EN>
<div>/usr/local/pgsql/data/basebackup.sh: line 12: psql: command not found</div>
<div>rsync: failed to set times on "/usr/local/pgsql/data/basebackup.sh": Operation not permitted (1)</div>
<div style="RIGHT: auto">rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1039) [sender=3.0.6]</div>
<div>/usr/local/pgsql/data/basebackup.sh: line 34: psql: command not found</div></SPAN></SPAN>
<DIV style="RIGHT: auto">***********************</DIV>
<DIV style="RIGHT: auto"> </DIV>
<DIV style="RIGHT: auto"> The same worked fine for my earlier virtual machines with same configuration and no error e1012 popped up the third time after applying the "rm -rf pg_xlog" line.</DIV>
<DIV style="RIGHT: auto"> </DIV>
<DIV style="RIGHT: auto"> Kindly help me on this unique issue i have checked the server time of both postgres database they are almost same only 1 to 2 sec apart.</DIV>
<DIV style="RIGHT: auto"> </DIV>
<DIV style="RIGHT: auto">Thanks and Regards,</DIV>
<div style="RIGHT: auto">Syed Irfan.</div>
<div style="RIGHT: auto">Sr. Developer<BR><BR style="RIGHT: auto"></div>
<DIV style="FONT-FAMILY: times new roman, new york, times, serif; FONT-SIZE: 14pt">
<DIV style="FONT-FAMILY: times new roman, new york, times, serif; FONT-SIZE: 12pt">
<DIV dir=ltr>
<DIV style="BORDER-BOTTOM: #ccc 1px solid; BORDER-LEFT: #ccc 1px solid; PADDING-BOTTOM: 0px; LINE-HEIGHT: 0; MARGIN: 5px 0px; PADDING-LEFT: 0px; PADDING-RIGHT: 0px; HEIGHT: 0px; FONT-SIZE: 0px; BORDER-TOP: #ccc 1px solid; BORDER-RIGHT: #ccc 1px solid; PADDING-TOP: 0px" class=hr contentEditable=false readonly="true"></DIV><FONT size=2 face=Arial><B><SPAN style="FONT-WEIGHT: bold">From:</SPAN></B> Tatsuo Ishii <ishii@postgresql.org><BR><B><SPAN style="FONT-WEIGHT: bold">To:</SPAN></B> syedirfan_77@yahoo.com <BR><B><SPAN style="FONT-WEIGHT: bold">Cc:</SPAN></B> pgpool-general@pgpool.net <BR><B><SPAN style="FONT-WEIGHT: bold">Sent:</SPAN></B> Thursday, 30 January 2014 5:20 AM<BR><B><SPAN style="FONT-WEIGHT: bold">Subject:</SPAN></B> Re: [pgpool-general: 2475] Help error code e1012 on pgpool II 3.3.0 while clicking Recovery button<BR></FONT></DIV>
<DIV class=y_msg_container><BR>It seems the cause of your problem is apparently this:<BR clear=none><BR clear=none>> /usr/local/pgsql/data/basebackup.sh: line 33: psql: command not found<BR clear=none><BR clear=none>Please fix it.<BR clear=none><BR clear=none>Best regards,<BR clear=none>--<BR clear=none>Tatsuo Ishii<BR clear=none>SRA OSS, Inc. Japan<BR clear=none>English: <A href="http://www.sraoss.co.jp/index_en.php" shape=rect target=_blank>http://www.sraoss.co.jp/index_en.php</A><BR clear=none>Japanese: <A href="http://www.sraoss.co.jp/" shape=rect target=_blank>http://www.sraoss.co.jp/</A>
<DIV id=yqtfd05782 class=yqt8830855540><BR clear=none><BR clear=none>> Dear Tatsuo,<BR clear=none>> <BR clear=none>> Thanks for your reply, I have followed your recommendations and this time the errorcode e1012 pops up not on the third try but on the second try and within 2 seconds of clicking the recovery button. <BR clear=none>> <BR clear=none>> First Try of Recovery button when primary was down<BR clear=none>> Recovery success on 172.16.80.49(when it was down manually) the backup log of 172.16.80.47 is as follows<BR clear=none>> <BR clear=none>> ******************************************************************<BR clear=none>> pg_start_backup <BR clear=none>> -----------------<BR clear=none>> 1/3C000020<BR clear=none>> (1 row)<BR clear=none>> <BR clear=none>> mkdir: cannot create directory `/usr/local/pgsql/data/pg_xlog': File exists<BR clear=none>> NOTICE:
WAL archiving is not enabled; you must ensure that all required WAL segments are copied through other means to complete the backup<BR clear=none>> pg_stop_backup <BR clear=none>> ----------------<BR clear=none>> 1/3C0000D8<BR clear=none>> (1 row)<BR clear=none>> ******************************************************************<BR clear=none>> <BR clear=none>> Second Try of Recovery button when the new primary was down manually<BR clear=none>> Recovery fails on 172.16.80.47(when it was down manually) the backup log of 172.16.80.49 is as follows<BR clear=none>> <BR clear=none>> ******************************************************************<BR clear=none>> /usr/local/pgsql/data/basebackup.sh: line 12: psql: command not found<BR clear=none>> mkdir: cannot create directory `/usr/local/pgsql/data/pg_xlog': File exists<BR clear=none>> /usr/local/pgsql/data/basebackup.sh: line 33: psql: command
not found<BR clear=none>> ******************************************************************<BR clear=none>> <BR clear=none>> The basebackup.sh on both servers is as follows with the added script for basebackup.log and uncommented recovery_target_timeline = 'latest'<BR clear=none>> <BR clear=none>> *******************************************************************<BR clear=none>> #/bin/sh -x<BR clear=none>> exec > /tmp/basebackup.log 2>&1<BR clear=none>> # XXX We assume master and recovery host uses the same port number<BR clear=none>> PORT=5432<BR clear=none>> master_node_host_name=`hostname`<BR clear=none>> master_db_cluster=$1<BR clear=none>> recovery_node_host_name=$2<BR clear=none>> recovery_db_cluster=$3<BR clear=none>> tmp=/tmp/mytemp$$<BR clear=none>> trap "rm -f $tmp" 0 1 2 3 15<BR clear=none>> <BR clear=none>> psql -p $PORT -c "SELECT pg_start_backup('Streaming Replication', true)"
postgres<BR clear=none>> <BR clear=none>> rsync -C -a -c --delete --exclude postgresql.conf --exclude postmaster.pid \<BR clear=none>> --exclude postmaster.opts --exclude pg_log \<BR clear=none>> --exclude recovery.conf --exclude recovery.done \<BR clear=none>> --exclude pg_xlog \<BR clear=none>> $master_db_cluster/ $recovery_node_host_name:$recovery_db_cluster<BR clear=none>> <BR clear=none>> ssh -T $recovery_node_host_name mkdir $recovery_db_cluster/pg_xlog<BR clear=none>> ssh -T $recovery_node_host_name chmod 700 $recovery_db_cluster/pg_xlog<BR clear=none>> ssh -T $recovery_node_host_name rm -f $recovery_db_cluster/recovery.done<BR clear=none>> <BR clear=none>> cat > $tmp <<EOF<BR clear=none>> recovery_target_timeline = 'latest'<BR clear=none>> standby_mode = 'on'<BR clear=none>> primary_conninfo = 'host=$master_node_host_name port=$PORT
user=postgres'<BR clear=none>> trigger_file = '/var/log/pgpool/trigger/trigger_file1'<BR clear=none>> EOF<BR clear=none>> <BR clear=none>> scp $tmp $recovery_node_host_name:$recovery_db_cluster/recovery.conf<BR clear=none>> <BR clear=none>> psql -p $PORT -c "SELECT pg_stop_backup()" postgres<BR clear=none>> *******************************************************************<BR clear=none>> <BR clear=none>> <BR clear=none>> Also, the reason for commenting (recovery_target_timeline = 'latest') was it was not mentioned in your "Simple Streaming replication setting with pgpool-II(multiple servers version)" <A href="http://www.pgpool.net/pgpool-web/contrib_docs/simple_sr_setting2_3.0/" shape=rect target=_blank>http://www.pgpool.net/pgpool-web/contrib_docs/simple_sr_setting2_3.0/</A>page. <BR clear=none>> But after a long search on the net i found
someone adding the line (recovery_target_timeline = 'latest') so for test purpose i have added it and once it did not solve the purpose I had commented it.<BR clear=none>> <BR clear=none>> Request you to help on the issue ASAP.<BR clear=none>> <BR clear=none>> Best Regards,<BR clear=none>> Syed Irfan<BR clear=none>> Sr Developer<BR clear=none>> <BR clear=none>> <BR clear=none>> Thanks and Regards,<BR clear=none>> Syed Irfan.<BR clear=none>> <BR clear=none>> Sr. Developer<BR clear=none>> <BR clear=none>> <BR clear=none>> <BR clear=none>> <BR clear=none>> <BR clear=none>> On Tuesday, 28 January 2014 4:52 AM, Tatsuo Ishii <<A href="mailto:ishii@postgresql.org" shape=rect ymailto="mailto:ishii@postgresql.org">ishii@postgresql.org</A>> wrote:<BR clear=none>> <BR clear=none>>> This is what the recovery.conf looks like.<BR clear=none>>>
*******************************************<BR clear=none>>> #recovery_target_timeline = 'latest'<BR clear=none>>> standby_mode = 'on'<BR clear=none>>> primary_conninfo = 'host=postgres-p.rolta.com port=5432 user=postgres'<BR clear=none>>> trigger_file = '/var/log/pgpool/trigger/trigger_file1'<BR clear=none>>> ********************************************************<BR clear=none>> <BR clear=none>> Why did you remove "recovery_target_timeline = 'latest'"?<BR clear=none>> <BR clear=none>> I suggesto to take an execution log of script. You change the very<BR clear=none>> begging of the script:<BR clear=none>> <BR clear=none>> #/bin/sh -x<BR clear=none>> <BR clear=none>> to:<BR clear=none>> <BR clear=none>> #/bin/sh -x<BR clear=none>> exec > /tmp/basebackup.log 2>&1<BR clear=none>> <BR clear=none>> and please show us the
content of /tmp/basebackup.log after execution of pcp_recovery_node.<BR clear=none>> <BR clear=none>> Best regards,<BR clear=none>> --<BR clear=none>> Tatsuo Ishii<BR clear=none>> SRA OSS, Inc. Japan<BR clear=none>> English: <A href="http://www.sraoss.co.jp/index_en.php" shape=rect target=_blank>http://www.sraoss.co.jp/index_en.php</A><BR clear=none>> Japanese: <A href="http://www.sraoss.co.jp/" shape=rect target=_blank>http://www.sraoss.co.jp/</A><BR clear=none>> <BR clear=none>> <BR clear=none>> <BR clear=none>>> The basebackup.sh on both postgres databases is as follows<BR clear=none>>> **************************************************<BR clear=none>>> <BR clear=none>>> #/bin/sh -x<BR clear=none>>> #<BR clear=none>>> # XXX We assume master and recovery host uses the same port number<BR clear=none>>> PORT=5432<BR clear=none>>> master_node_host_name=`hostname`<BR
clear=none>>> master_db_cluster=$1<BR clear=none>>> recovery_node_host_name=$2<BR clear=none>>> recovery_db_cluster=$3<BR clear=none>>> tmp=/tmp/mytemp$$<BR clear=none>>> trap "rm -f $tmp" 0 1 2 3 15<BR clear=none>>> <BR clear=none>>> psql -p $PORT -c "SELECT pg_start_backup('Streaming Replication', true)" postgres<BR clear=none>>> <BR clear=none>>> rsync -C -a -c --delete --exclude postgresql.conf --exclude postmaster.pid \<BR clear=none>>> --exclude postmaster.opts --exclude pg_log \<BR clear=none>>> --exclude recovery.conf --exclude recovery.done \<BR clear=none>>> --exclude pg_xlog \<BR clear=none>>> $master_db_cluster/ $recovery_node_host_name:$recovery_db_cluster<BR clear=none>>> <BR clear=none>>> ssh -T $recovery_node_host_name mkdir $recovery_db_cluster/pg_xlog<BR clear=none>>> ssh -T $recovery_node_host_name chmod 700 $recovery_db_cluster/pg_xlog<BR
clear=none>>> ssh -T $recovery_node_host_name rm -f $recovery_db_cluster/recovery.done<BR clear=none>>> <BR clear=none>>> cat > $tmp <<EOF<BR clear=none>>> #recovery_target_timeline = 'latest'<BR clear=none>>> standby_mode = 'on'<BR clear=none>>> primary_conninfo = 'host=$master_node_host_name port=$PORT user=postgres'<BR clear=none>>> trigger_file = '/var/log/pgpool/trigger/trigger_file1'<BR clear=none>>> EOF<BR clear=none>>> <BR clear=none>>> scp $tmp $recovery_node_host_name:$recovery_db_cluster/recovery.conf<BR clear=none>>> <BR clear=none>>> psql -p $PORT -c "SELECT pg_stop_backup()" postgres<BR clear=none>>> ***********************************************<BR clear=none>>> <BR clear=none>>> Thanks and Regards,<BR clear=none>>> Syed Irfan.<BR clear=none>>> <BR clear=none>>>
Sr. Developer<BR clear=none>>> <BR clear=none>>> <BR clear=none>>> <BR clear=none>>> <BR clear=none>>> <BR clear=none>>> On Thursday, 23 January 2014 11:07 PM, Jeff Frost <<A href="mailto:jeff@pgexperts.com" shape=rect ymailto="mailto:jeff@pgexperts.com">jeff@pgexperts.com</A>> wrote:<BR clear=none>>> <BR clear=none>>> <BR clear=none>>> <BR clear=none>>> On Jan 23, 2014, at 9:32 AM, Syed Irfan <<A href="mailto:syedirfan_77@yahoo.com" shape=rect ymailto="mailto:syedirfan_77@yahoo.com">syedirfan_77@yahoo.com</A>> wrote:<BR clear=none>>> <BR clear=none>>> Dear Tatsuo Ishii,<BR clear=none>>>><BR clear=none>>>><BR clear=none>>>> I am still awaiting for your reply on this issue, I have tried your suggestions but still I am unable to successfully run the Recovery process the third time it's surprises me how does it work
the first time but same thing fails in the third attempt.?<BR clear=none>>>><BR clear=none>>>><BR clear=none>>>>The Postgres log shows as below<BR clear=none>>>><BR clear=none>>>><BR clear=none>>>><BR clear=none>>>><BR clear=none>>>>>> 28038 2014-01-09 21:28:33 BDT FATAL: timeline 35 of the primary does not match recovery target timeline 36<BR clear=none>>>>>>> 28039 2014-01-09 21:28:38 BDT FATAL: timeline 35 of the primary does not match recovery target timeline 36<BR clear=none>>>><BR clear=none>>>><BR clear=none>>>><BR clear=none>>>>I urgently request you to help me in this impending issue.<BR clear=none>>>><BR clear=none>>> <BR clear=none>>> This is usually caused by postgres trying to replay WAL files from the wrong source. Did you clean out the pg_xlog directory on the replica
before taking the base backup?<BR clear=none>>> <BR clear=none>>> What does your recovery.conf look like?</DIV><BR><BR></DIV></DIV></DIV></div></body></html>