<div dir="ltr"><div><div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, May 12, 2020 at 2:51 PM 周建身 <<a href="mailto:zhoujianshen@highgo.com">zhoujianshen@highgo.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div style="font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255);font-family:Calibri,Arial,Helvetica,sans-serif">
<div style="font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255);font-family:calibri,arial,helvetica,sans-serif">
<div style="font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255);font-family:calibri,arial,helvetica,sans-serif">
<div style="font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255);font-family:calibri,arial,helvetica,sans-serif">
<p>Hi <span style="font-family:calibri,sans-serif;font-size:14.6667px;background-color:rgb(255,255,255)">Muhammad Usama</span>:<br>
</p>
<p><span> </span>Thanks for the suggestions you giving and the help you provided during my development. I have rebased my patch with the current head of the master branch.</p>
<p>And,in this patch,I used <span style="color:rgb(40,40,40);font-family:calibri,arial,helvetica,sans-serif;font-size:16px;background-color:rgb(255,255,255)"> (</span><span style="color:rgb(40,40,40);font-family:calibri,arial,helvetica,sans-serif;font-size:16px;background-color:rgb(255,255,255)">--scope=cluster)</span>
to implement the cluster mode.<br>
</p>
<p><br>
</p>
<div style="font-family:calibri,arial,helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px">
How to use the patch to test pcp_reload_config command<br>
</div>
<div style="font-family:calibri,arial,helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px">
1、do patch under the directory of pgpool<br>
</div>
<div style="font-family:calibri,arial,helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px">
patch -p1 < pcp_reload_config.patch<br>
</div>
<div style="font-family:calibri,arial,helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px">
2、you should need automake and autoconf<br>
</div>
<div style="font-family:calibri,arial,helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px">
<br>
</div>
<pre style="font-size:16px;margin-top:14pt;margin-bottom:14pt">for example:
$ autoheader
$ aclocal
$ automake -a -c
$ autoconf
</pre>
<div style="font-family:calibri,arial,helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px">
3、then to do make and make install<br>
</div>
<div style="font-family:calibri,arial,helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px">
<br>
</div>
<div style="font-family:calibri,arial,helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px">
4、under the directory of bin，you can find the pcp_reload_config <br>
</div>
<div style="font-family:calibri,arial,helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px">
<br>
</div>
<div style="font-family:calibri,arial,helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px">
./pcp_reload_config -s c or ./pcp_reload_config --scope=cluster<br>
</div>
<div style="font-family:calibri,arial,helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px">
this can reload the config file of all pgpool nodes<br>
</div>
<div style="font-family:calibri,arial,helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px">
<br>
</div>
<div style="font-family:calibri,arial,helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px">
./pcp_reload_config or ./pcp_reload_config -s l or ./pcp_reload_config --scope=local<br>
</div>
<div style="font-family:calibri,arial,helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px">
this can only reload the config file of local pgpool node<br>
</div>
<p><br>
</p>
<p style="font-family:calibri,arial,helvetica,sans-serif;font-size:16px"><span style="color:rgb(40,40,40);font-family:"microsoft yahei ui","microsoft yahei",微软雅黑,simsun,宋体,sans-serif;font-size:13.3333px;background-color:rgb(255,255,255)">Thanks</span><br style="color:rgb(40,40,40);font-family:"microsoft yahei ui","microsoft yahei",微软雅黑,simsun,宋体,sans-serif;font-size:13.3333px;background-color:rgb(255,255,255)">
<span style="color:rgb(40,40,40);font-family:"microsoft yahei ui","microsoft yahei",微软雅黑,simsun,宋体,sans-serif;font-size:13.3333px;background-color:rgb(255,255,255)">Best regards</span><br style="color:rgb(40,40,40);font-family:"microsoft yahei ui","microsoft yahei",微软雅黑,simsun,宋体,sans-serif;font-size:13.3333px;background-color:rgb(255,255,255)">
Zhou Jianshen</p>
<p style="font-family:calibri,arial,helvetica,sans-serif;font-size:16px"><span style="font-family:verdana;font-size:13.3333px;background-color:rgb(255,255,255)"><a href="mailto:zhoujianshen@highgo.com" target="_blank">zhoujianshen@highgo.com</a></span></p>
<div style="color:rgb(40,40,40)">
<hr style="display:inline-block;width:98%">
<div id="gmail-m_4478220402618920387divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>发件人:</b> Muhammad Usama <<a href="mailto:m.usama@gmail.com" target="_blank">m.usama@gmail.com</a>><br>
<b>发送时间:</b> 2020年5月4日 21:43<br>
<b>收件人:</b> 周建身<br>
<b>抄送:</b> pgpool-hackers; Muhammad<br>
<b>主题:</b> Re: [pgpool-hackers: 3606] add the command of pcp_reload_config</font>
<div> </div>
</div>
<div>
<div dir="ltr">
<div dir="ltr">Hi Zhou Jianshen,
<div><br>
</div>
<div>Welcome to the Pgpool-II community and thanks for the patch.</div>
<div><br>
</div>
<div>I just committed a reorganization patch and now this patch does not apply anymore</div>
<div>so can you please rebase it with the current head of the master branch.</div>
<div><br>
</div>
<div>Secondly, you need to implement the cluster mode (--scope=cluster) for the</div>
<div>PCP reload config utility as well.</div>
<div><br>
</div>
<div>For the reference, you can see the implementation same ( scope=cluster)</div>
<div>in the pcp_stop_pgpool utility.</div>
<div><br>
</div>
<div><a href="https://git.postgresql.org/gitweb/?p=pgpool2.git;a=commitdiff;h=87e6e11c1931555e90a1cc65cc3ee79759ebaaab" target="_blank">https://git.postgresql.org/gitweb/?p=pgpool2.git;a=commitdiff;h=87e6e11c1931555e90a1cc65cc3ee79759ebaaab</a><br>
</div>
<div><br>
</div>
<div>You can find more info on steps for executing the command over watchdog network</div>
<div>in the following email.</div>
<div><br>
</div>
<div><a href="https://www.pgpool.net/pipermail/pgpool-hackers/2020-May/003608.html" target="_blank">https://www.pgpool.net/pipermail/pgpool-hackers/2020-May/003608.html</a><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div>Do let me know if you need further information or clarification</div>
<div><br>
</div>
<div>Best Regards</div>
<div>
<div><span style="color:rgb(0,0,0);font-family:calibri,arial,helvetica,sans-serif;font-size:16px"><br>
</span></div>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Wed, Apr 29, 2020 at 12:10 PM 周建身 <<a href="mailto:zhoujianshen@highgo.com" target="_blank">zhoujianshen@highgo.com</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div style="font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255);font-family:calibri,arial,helvetica,sans-serif">
<p><span style="color:rgb(40,40,40);font-family:"microsoft yahei ui","microsoft yahei",微软雅黑,simsun,宋体,sans-serif;font-size:15px;background-color:rgb(255,255,255)">Hi Hackers,</span><br>
</p>
<p><span> </span>As Usama Muhammad said in his email before,we should enhance/add the PCP utilities more cluster-mode friendly.</p>
<p>There are four commands which need to be added or enhanced.The first command which should be added is pcp_reload_config.<br>
</p>
<p><span> </span><br>
</p>
<p><span> <span style="color:rgb(40,40,40);font-family:"microsoft yahei ui","microsoft yahei",微软雅黑,simsun,宋体,sans-serif;font-size:13.3333px;background-color:rgb(255,255,255)">1- pcp_reload_config</span><br style="color:rgb(40,40,40);font-family:"microsoft yahei ui","microsoft yahei",微软雅黑,simsun,宋体,sans-serif;font-size:13.3333px;background-color:rgb(255,255,255)">
<span style="color:rgb(40,40,40);font-family:"microsoft yahei ui","microsoft yahei",微软雅黑,simsun,宋体,sans-serif;font-size:13.3333px;background-color:rgb(255,255,255)">==================</span><br style="color:rgb(40,40,40);font-family:"microsoft yahei ui","microsoft yahei",微软雅黑,simsun,宋体,sans-serif;font-size:13.3333px;background-color:rgb(255,255,255)">
<span style="color:rgb(40,40,40);font-family:"microsoft yahei ui","microsoft yahei",微软雅黑,simsun,宋体,sans-serif;font-size:13.3333px;background-color:rgb(255,255,255)">The utility to make the whole Pgpool-II cluster to reload the</span><br style="color:rgb(40,40,40);font-family:"microsoft yahei ui","microsoft yahei",微软雅黑,simsun,宋体,sans-serif;font-size:13.3333px;background-color:rgb(255,255,255)">
<span style="color:rgb(40,40,40);font-family:"microsoft yahei ui","microsoft yahei",微软雅黑,simsun,宋体,sans-serif;font-size:13.3333px;background-color:rgb(255,255,255)">configuration files at once.</span><br style="color:rgb(40,40,40);font-family:"microsoft yahei ui","microsoft yahei",微软雅黑,simsun,宋体,sans-serif;font-size:13.3333px;background-color:rgb(255,255,255)">
<span style="color:rgb(40,40,40);font-family:"microsoft yahei ui","microsoft yahei",微软雅黑,simsun,宋体,sans-serif;font-size:13.3333px;background-color:rgb(255,255,255)">I think this is required because currently if some installation has</span><br style="color:rgb(40,40,40);font-family:"microsoft yahei ui","microsoft yahei",微软雅黑,simsun,宋体,sans-serif;font-size:13.3333px;background-color:rgb(255,255,255)">
<span style="color:rgb(40,40,40);font-family:"microsoft yahei ui","microsoft yahei",微软雅黑,simsun,宋体,sans-serif;font-size:13.3333px;background-color:rgb(255,255,255)">let say 3 node Pgpool-II cluster</span><br style="color:rgb(40,40,40);font-family:"microsoft yahei ui","microsoft yahei",微软雅黑,simsun,宋体,sans-serif;font-size:13.3333px;background-color:rgb(255,255,255)">
<span style="color:rgb(40,40,40);font-family:"microsoft yahei ui","microsoft yahei",微软雅黑,simsun,宋体,sans-serif;font-size:13.3333px;background-color:rgb(255,255,255)">and we want to change one specific configuration parameter ( for</span><br style="color:rgb(40,40,40);font-family:"microsoft yahei ui","microsoft yahei",微软雅黑,simsun,宋体,sans-serif;font-size:13.3333px;background-color:rgb(255,255,255)">
<span style="color:rgb(40,40,40);font-family:"microsoft yahei ui","microsoft yahei",微软雅黑,simsun,宋体,sans-serif;font-size:13.3333px;background-color:rgb(255,255,255)">example: adding a new backend node)</span><br style="color:rgb(40,40,40);font-family:"microsoft yahei ui","microsoft yahei",微软雅黑,simsun,宋体,sans-serif;font-size:13.3333px;background-color:rgb(255,255,255)">
<span style="color:rgb(40,40,40);font-family:"microsoft yahei ui","microsoft yahei",微软雅黑,simsun,宋体,sans-serif;font-size:13.3333px;background-color:rgb(255,255,255)">we would require to go on and edit Pgpool configuration files on each</span><br style="color:rgb(40,40,40);font-family:"microsoft yahei ui","microsoft yahei",微软雅黑,simsun,宋体,sans-serif;font-size:13.3333px;background-color:rgb(255,255,255)">
<span style="color:rgb(40,40,40);font-family:"microsoft yahei ui","microsoft yahei",微软雅黑,simsun,宋体,sans-serif;font-size:13.3333px;background-color:rgb(255,255,255)">node separately and then</span><br style="color:rgb(40,40,40);font-family:"microsoft yahei ui","microsoft yahei",微软雅黑,simsun,宋体,sans-serif;font-size:13.3333px;background-color:rgb(255,255,255)">
<span style="color:rgb(40,40,40);font-family:"microsoft yahei ui","microsoft yahei",微软雅黑,simsun,宋体,sans-serif;font-size:13.3333px;background-color:rgb(255,255,255)">issue pgpool reload on each node one by one.</span><br style="color:rgb(40,40,40);font-family:"microsoft yahei ui","microsoft yahei",微软雅黑,simsun,宋体,sans-serif;font-size:13.3333px;background-color:rgb(255,255,255)">
<br style="color:rgb(40,40,40);font-family:"microsoft yahei ui","microsoft yahei",微软雅黑,simsun,宋体,sans-serif;font-size:13.3333px;background-color:rgb(255,255,255)">
<span style="color:rgb(40,40,40);font-family:"microsoft yahei ui","microsoft yahei",微软雅黑,simsun,宋体,sans-serif;font-size:13.3333px;background-color:rgb(255,255,255)">In my opinion, we need multiple enhancement in this area.</span><br style="color:rgb(40,40,40);font-family:"microsoft yahei ui","microsoft yahei",微软雅黑,simsun,宋体,sans-serif;font-size:13.3333px;background-color:rgb(255,255,255)">
<span style="color:rgb(40,40,40);font-family:"microsoft yahei ui","microsoft yahei",微软雅黑,simsun,宋体,sans-serif;font-size:13.3333px;background-color:rgb(255,255,255)">We need some mechanisms to let pgpool-II cluster use the centralized</span><br style="color:rgb(40,40,40);font-family:"microsoft yahei ui","microsoft yahei",微软雅黑,simsun,宋体,sans-serif;font-size:13.3333px;background-color:rgb(255,255,255)">
<span style="color:rgb(40,40,40);font-family:"microsoft yahei ui","microsoft yahei",微软雅黑,simsun,宋体,sans-serif;font-size:13.3333px;background-color:rgb(255,255,255)">configuration</span><br style="color:rgb(40,40,40);font-family:"microsoft yahei ui","microsoft yahei",微软雅黑,simsun,宋体,sans-serif;font-size:13.3333px;background-color:rgb(255,255,255)">
<span style="color:rgb(40,40,40);font-family:"microsoft yahei ui","microsoft yahei",微软雅黑,simsun,宋体,sans-serif;font-size:13.3333px;background-color:rgb(255,255,255)">( except watchdog config) or at least a mechanism to push the</span><br style="color:rgb(40,40,40);font-family:"microsoft yahei ui","microsoft yahei",微软雅黑,simsun,宋体,sans-serif;font-size:13.3333px;background-color:rgb(255,255,255)">
<span style="color:rgb(40,40,40);font-family:"microsoft yahei ui","microsoft yahei",微软雅黑,simsun,宋体,sans-serif;font-size:13.3333px;background-color:rgb(255,255,255)">configuration settings using some</span><br style="color:rgb(40,40,40);font-family:"microsoft yahei ui","microsoft yahei",微软雅黑,simsun,宋体,sans-serif;font-size:13.3333px;background-color:rgb(255,255,255)">
<span style="color:rgb(40,40,40);font-family:"microsoft yahei ui","microsoft yahei",微软雅黑,simsun,宋体,sans-serif;font-size:13.3333px;background-color:rgb(255,255,255)">new utility like "pcp_load_config" that could take a pgpool.conf file</span><br style="color:rgb(40,40,40);font-family:"microsoft yahei ui","microsoft yahei",微软雅黑,simsun,宋体,sans-serif;font-size:13.3333px;background-color:rgb(255,255,255)">
<span style="color:rgb(40,40,40);font-family:"microsoft yahei ui","microsoft yahei",微软雅黑,simsun,宋体,sans-serif;font-size:13.3333px;background-color:rgb(255,255,255)">and propagate it to all nodes</span><br style="color:rgb(40,40,40);font-family:"microsoft yahei ui","microsoft yahei",微软雅黑,simsun,宋体,sans-serif;font-size:13.3333px;background-color:rgb(255,255,255)">
<span style="color:rgb(40,40,40);font-family:"microsoft yahei ui","microsoft yahei",微软雅黑,simsun,宋体,sans-serif;font-size:13.3333px;background-color:rgb(255,255,255)">using watchdog.</span><br style="color:rgb(40,40,40);font-family:"microsoft yahei ui","microsoft yahei",微软雅黑,simsun,宋体,sans-serif;font-size:13.3333px;background-color:rgb(255,255,255)">
<span style="color:rgb(40,40,40);font-family:"microsoft yahei ui","microsoft yahei",微软雅黑,simsun,宋体,sans-serif;font-size:13.3333px;background-color:rgb(255,255,255)">The centralized configuration part of the problem is a big task and I</span><br style="color:rgb(40,40,40);font-family:"microsoft yahei ui","microsoft yahei",微软雅黑,simsun,宋体,sans-serif;font-size:13.3333px;background-color:rgb(255,255,255)">
<span style="color:rgb(40,40,40);font-family:"microsoft yahei ui","microsoft yahei",微软雅黑,simsun,宋体,sans-serif;font-size:13.3333px;background-color:rgb(255,255,255)">believe we could take it on after the</span><br style="color:rgb(40,40,40);font-family:"microsoft yahei ui","microsoft yahei",微软雅黑,simsun,宋体,sans-serif;font-size:13.3333px;background-color:rgb(255,255,255)">
<span style="color:rgb(40,40,40);font-family:"microsoft yahei ui","microsoft yahei",微软雅黑,simsun,宋体,sans-serif;font-size:13.3333px;background-color:rgb(255,255,255)">"simplifying watchdog config" feature that Peng is working on.</span><br style="color:rgb(40,40,40);font-family:"microsoft yahei ui","microsoft yahei",微软雅黑,simsun,宋体,sans-serif;font-size:13.3333px;background-color:rgb(255,255,255)">
<span style="color:rgb(40,40,40);font-family:"microsoft yahei ui","microsoft yahei",微软雅黑,simsun,宋体,sans-serif;font-size:13.3333px;background-color:rgb(255,255,255)">Meanwhile, I think we can work on pcp_reload_config utility in</span><br style="color:rgb(40,40,40);font-family:"microsoft yahei ui","microsoft yahei",微软雅黑,simsun,宋体,sans-serif;font-size:13.3333px;background-color:rgb(255,255,255)">
<span style="color:rgb(40,40,40);font-family:"microsoft yahei ui","microsoft yahei",微软雅黑,simsun,宋体,sans-serif;font-size:13.3333px;background-color:rgb(255,255,255)">parallel to make reloading part easier.</span></span><br>
</p>
<p><span><br>
</span></p>
<p><span><br>
</span></p>
<p><span>I tried to solve this command.And I realized reloading the configuration of an individual node.After Peng's working on </span></p>
<p><span>watchdog config,we can eventually realize the reloading the configurations of all nodes.</span></p>
<p><span><br>
</span></p>
<p><span>How to use the patch,</span></p>
<p>1、do patch under the directory of pgpool-II-4.1.1<br>
</p>
<p>patch -p1 < pcp_reload_config.patch<br>
</p>
<p>2、you should need automake and autoconf<br>
</p>
<p><br>
</p>
<pre style="white-space:pre-wrap">for example:
$ autoheader
$ aclocal
$ automake -a -c
$ autoconf<br></pre>
<p>3、then to do make and make install</p>
<p><br>
</p>
<p>4、under the directory of bin，you can find the pcp_reload_config ,try is as ./pcp_reload_config<br>
</p>
<p><span> </span></p>
<p><span><br>
</span></p>
<p><span>Comments and suggestions are welcome.</span></p>
<p><span><br>
</span></p>
<p><span><span style="color:rgb(40,40,40);font-family:"microsoft yahei ui","microsoft yahei",微软雅黑,simsun,宋体,sans-serif;font-size:13.3333px;background-color:rgb(255,255,255)">Thanks</span><br style="color:rgb(40,40,40);font-family:"microsoft yahei ui","microsoft yahei",微软雅黑,simsun,宋体,sans-serif;font-size:13.3333px;background-color:rgb(255,255,255)">
<span style="color:rgb(40,40,40);font-family:"microsoft yahei ui","microsoft yahei",微软雅黑,simsun,宋体,sans-serif;font-size:13.3333px;background-color:rgb(255,255,255)">Best regards</span><br style="color:rgb(40,40,40);font-family:"microsoft yahei ui","microsoft yahei",微软雅黑,simsun,宋体,sans-serif;font-size:13.3333px;background-color:rgb(255,255,255)">
Zhou Jianshen</span></p>
<p><span><span style="font-family:verdana;font-size:13.3333px;background-color:rgb(255,255,255)"><a href="mailto:zhoujianshen@highgo.com" target="_blank">zhoujianshen@highgo.com</a></span><br>
</span></p>
<p><span><br>
</span></p>
</div>
</div>
_______________________________________________<br>
pgpool-hackers mailing list<br>
<a href="mailto:pgpool-hackers@pgpool.net" target="_blank">pgpool-hackers@pgpool.net</a><br>
<a href="http://www.pgpool.net/mailman/listinfo/pgpool-hackers" rel="noreferrer" target="_blank">http://www.pgpool.net/mailman/listinfo/pgpool-hackers</a><br>
</blockquote>
</div>
<br clear="all">
<div><br></div></div></div></div></div></div></div></div></div></blockquote>Hi Jianshen,<div><br></div><div>Thank you for the patch. Overall the patch looks and had few minor issues. </div><div><br></div><div>1- In process_reload_config() function the log message was mentioning</div><div>pcp_stop_pgpool instead of pgpool_reload_config<br><br>2- When you want to throw an error to client from PCP processor you can just</div><div>use 'ereport' instead of creating the error message by hand.</div><div><br>3- No need to use different PCP packet type (Z and z ) for cluster and local modes.</div><div>instead, we can send the command scope in the packet load.</div><div>The reason we used the different packet types in the pcp_stop_pgpool was to keep</div><div>the backward compatibility. And since this is a new utility so we can send that</div><div>information in the packet data without worrying about compatibility.<br><br>4- '-n' argument is not required in the pcp_reload_config<br>5- The patch contains some whitespace warnings.</div><div><br></div><div>I have committed the patch after fixing the above-mentioned issues along with</div><div>few code re-arrangements in the pcp_frontend_client.c</div><div><br></div><div><a href="https://git.postgresql.org/gitweb/?p=pgpool2.git;a=commitdiff;h=987b69945dc914b88099d1d0420c4b49f1097f2f">https://git.postgresql.org/gitweb/?p=pgpool2.git;a=commitdiff;h=987b69945dc914b88099d1d0420c4b49f1097f2f</a><br></div><div><br></div><div><br></div>Also, the documentation part for the pcp_reload_utility was not included in the<div>patch. So can you please also submit the documentation updates for the utility,</div><div>I have marked this as a remaining item in the commit message.</div><div><br></div><div><div>P.S updated patch that I committed is also attached</div><div></div></div><div><br></div><div>Thanks</div><div></div><div>Best Regards</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div style="font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255);font-family:Calibri,Arial,Helvetica,sans-serif"><div style="font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255);font-family:calibri,arial,helvetica,sans-serif"><div style="font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255);font-family:calibri,arial,helvetica,sans-serif"><div style="font-size:12pt;color:rgb(0,0,0);background-color:rgb(255,255,255);font-family:calibri,arial,helvetica,sans-serif"><div style="color:rgb(40,40,40)"><div><div dir="ltr"><div>
</div><br>
<div dir="ltr">
<div dir="ltr">
<div style="color:rgb(0,0,0)">
<div>
<div><span style="color:rgb(68,68,68)">...<br>
</span></div>
<div><span style="color:rgb(68,68,68)">Muhammad Usama<br>
</span></div>
<div><span><span style="font-family:lato"><span style="font-size:14px"><span style="color:rgb(68,68,68)">Highgo Software (Canada/China/Pakistan)</span><span><span style="color:rgb(68,68,68)"> </span></span></span></span></span><span style="color:rgb(68,68,68)"><br>
</span></div>
<div><span><span style="font-family:lato"><span style="font-size:14px"><span style="color:rgb(68,68,68)">URL :</span><span><span style="color:rgb(68,68,68)"> </span></span></span></span></span><a href="http://www.highgo.ca/" style="color:rgb(89,143,222);font-family:lato;font-size:14px" target="_blank"><span style="color:rgb(68,68,68)">http://www.highgo.ca</span></a><span><span style="font-family:lato"><span style="font-size:14px"><span><span style="color:rgb(68,68,68)"> </span></span></span></span></span><span style="color:rgb(68,68,68)"><br>
</span></div>
<div><span><span style="font-family:lato"><span style="font-size:14px"><span style="color:rgb(68,68,68)">ADDR: 10318 WHALLEY BLVD, Surrey, BC</span><span><span style="color:rgb(68,68,68)"> </span></span></span></span></span><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote></div></div></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div style="color:rgb(0,0,0)"><div><div><span style="color:rgb(68,68,68)">...<br></span></div><div><span style="color:rgb(68,68,68)">Muhammad Usama<br></span></div><div><span><span style="font-family:Lato"><span style="font-size:14px"><span style="color:rgb(68,68,68)">Highgo Software (Canada/China/Pakistan)</span><span><span style="color:rgb(68,68,68)"> </span></span></span></span></span><span style="color:rgb(68,68,68)"><br></span></div><div><span><span style="font-family:Lato"><span style="font-size:14px"><span style="color:rgb(68,68,68)">URL :</span><span><span style="color:rgb(68,68,68)"> </span></span></span></span></span><a href="http://www.highgo.ca/" style="color:rgb(89,143,222);font-family:Lato;font-size:14px" target="_blank"><span style="color:rgb(68,68,68)">http://www.highgo.ca</span></a><span><span style="font-family:Lato"><span style="font-size:14px"><span><span style="color:rgb(68,68,68)"> </span></span></span></span></span><span style="color:rgb(68,68,68)"><br></span></div><div><span><span style="font-family:Lato"><span style="font-size:14px"><span style="color:rgb(68,68,68)">ADDR: 10318 WHALLEY BLVD, Surrey, BC</span><span><span style="color:rgb(68,68,68)"> </span></span></span></span></span><br></div></div></div></div></div></div>