My master nameserver is on vps3. My outgoing SMTP server is now on vps3. My inbound smtp server is now split between vps2 and vps3. Varnish cache is now installed on vps3 and vps2.
Installing and configuring nginx is going to be almost as complex as installing and configuring the vps itself because of the number of things that need to go right for everything to work.
For example, if I make a typo with an IP address then a Web site might not work and finding that typo might take a lot of time.
This is, perhaps, where my ULA network can come to the rescue. Moving from vps2 to vps3 should be as simple as installing nginx; copying all the relevant configuration files, TLS keys, and certificates across; adding the public IPs to the interfaces; adding the public IPs and ports to the firewall; modifying the public (HTTP/HTTPS termination) IP settings for nginx, so they are vps3 IPs; modifying the private (ULA network) IP settings in nginx, so they point to the vps3 varnish instance; restarting nginx; modifying the DNS A and AAAA records in NSD and Cloudflare.
Not that it will be that simple, however.
Some of the Web sites I host use MySQL, so I will need to export the databases from vps2 to vps3, ensuring I make the databases on vps2 read only until the move is complete so there are no new records created that get lost after switchover.
Future Changes After Transition
At the moment DNS requests (excluding domains using Cloudflare) are made to 3 destinations: vps2 (2 DNS servers), vps3 (1 DNS server), and Hurricane Electric (4 DNS servers).
Something I am interested in doing is moving from Hurricane Electric to Esgob Limited for secondary DNS.
While Hurricane Electric helped me (and many others) understand IPv6, their free DNS service does have limitations. Some of those limitations are imposed (e.g. number of zones, number of records per zone) while others are due to lack of support (e.g. still no DNSSEC support).
If I make such a transition for secondary DNS provider, I will be moving from 7 unicast DNS servers to 1 unicast and 1 anycast.
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 00A6F0A3C300EE8C sudo nano /etc/apt/sources.list
# nginx deb http://ppa.launchpad.net/nginx/development/ubuntu trusty main deb-src http://ppa.launchpad.net/nginx/development/ubuntu trust main
sudo apt-get update sudo apt-get dist-upgrade sudo apt-get install nginx sudo apt-get autoremove sudo apt-get autoclean
As with Varnish, configuring nginx should be as simple as copying across the nginx configuration files, the SSL certificates and keys, and changing the IP addresses.
sudo su rsync -avrlogpP --progress -e 'ssh -p 8043 -i /home/thejc/.ssh/thejc.vps3' email@example.com:/etc/nginx/ /etc/nginx/ rsync -avrlogpP --progress -e 'ssh -p 8043 -i /home/thejc/.ssh/thejc.vps3' firstname.lastname@example.org:/etc/lighttpd/ /etc/lighttpd/ rsync -avrlogpP --progress -e 'ssh -p 8043 -i /home/thejc/.ssh/thejc.vps3' email@example.com:/etc/ssl/ /etc/ssl/ exit scp -P 8043 firstname.lastname@example.org:~/Scripts/ocsp-stapling.sh ~/Scripts/ scp -P 8043 email@example.com:~/Scripts/ocsp-stapling-cache-now.sh ~/Scripts/ sudo crontab -e
30 2 * * Mon,Wed,Fri,Sat /home/thejc/Scripts/ocsp-stapling-cache-now.sh
sudo ~/Scripts/ocsp-stapling-cache-now.sh sudo mkdir -p /var/cache/nginx sudo chown www-data:www-data /var/cache/nginx sudo service nginx restart
If everything is working as it should at this point (i.e. the OCSP stapling script can verify all the current certificates as good) then restarting nginx should fail.
The only errors timestamped at the most recent attempt at restarting nginx should be about problems binding to a requested IP address. It is now time to modify the IP addresses in the nginx configuration files on vps3.
This is going to be rather simple. Search for the string
2001:470:1f09:38d::80: and replace it with
cd /etc/nginx/ sudo su find . -type f -print0 | xargs -0 sed -i 's/2001:470:1f09:38d::80:/2a03:ca80:8001:769d::80:/g'
A recursive search of strings 2001 and 2a03, however, show that some issues remain. On my previous VPS I had two IPv6 IP addresses, so I need to do a recursive delete of lines containing
find . -type f -print0 | xargs -0 sed -i '/2a03:ca80:8000:/d'
Likewise, I need to make a similar change for IPv4 IP addresses. On my previous VPS I had 3 different IP(v4) addresses and one my new one I only have one (technically two, but it should be one at the start of next month as I've dropped the additional IP address).
Thus I need to search for
18.104.22.168 and replace them all with
find . -type f -print0 | xargs -0 sed -i 's/22.214.171.124/126.96.36.199/g' find . -type f -print0 | xargs -0 sed -i 's/188.8.131.52/184.108.40.206/g' find . -type f -print0 | xargs -0 sed -i 's/220.127.116.11/18.104.22.168/g' exit sudo service nginx restart
A final configuration issue: I haven't searched and replaced ULA IP addresses.
find . -type f -print0 | xargs -0 sed -i 's/fdd7:5938:e2e6:9660::80/fdd7:5938:e2e6:6c8d::80/g'
The final thing to do is disable mail3.thejc.me.uk:443 and [::]:80 on vps3, and to add the IP addresses for the Web sites that are currently enabled.
cd /etc/nginx/sites-enabled/ sudo rm default sudo rm mail3.thejc.me.uk grep -R "\[\(.*\)\]" .
./watfordjc.co.uk: listen [2a03:ca80:8001:769d::80:1]:443 ssl spdy; ./watfordjc.co.uk: listen [fdd7:5938:e2e6:6c8d::80:1]:443 ssl spdy; ./watfordjc.co.uk: listen [2a03:ca80:8001:769d::80:1]:80; ./watfordjc.co.uk: resolver [fcf4:90db:f24c:72ca:df4d:b9ee:be0b:c37d] [2001:470:1f09:38d::fcf4:53] [2001:470:20::2] 22.214.171.124 126.96.36.199; ./watfordjc.co.uk: listen [2a03:ca80:8001:769d::80:1]:443 ssl spdy; ./watfordjc.co.uk: listen [fdd7:5938:e2e6:6c8d::80:1]:443 ssl spdy; ./watfordjc.co.uk: listen [2a03:ca80:8001:769d::80:1]:80; ./watfordjc.co.uk: resolver [fcf4:90db:f24c:72ca:df4d:b9ee:be0b:c37d] [2001:470:1f09:38d::fcf4:53] [2001:470:20::2] 188.8.131.52 184.108.40.206; ./watfordjc.co.uk: listen [2a03:ca80:8001:769d::80:1]:443 ssl spdy; ./watfordjc.co.uk: listen [2a03:ca80:8001:769d::80:1]:80; ./watfordjc.co.uk: resolver [fcf4:90db:f24c:72ca:df4d:b9ee:be0b:c37d] [2001:470:1f09:38d::fcf4:53] [2001:470:20::2] 220.127.116.11 18.104.22.168; ./health.thejc.me.uk: listen [2a03:ca80:8001:769d::80:4]:80; ./gpg.thejc.me.uk: listen [2a03:ca80:8001:769d::80:a]:80; ./david-j-lane.co.uk: listen [2a03:ca80:8001:769d::80:3]:80; ./burnthefat2014.thejc.me.uk: listen [2a03:ca80:8001:769d::80:1]:80; ./blogs.thejc.me.uk: listen [2a03:ca80:8001:769d::80:4]:80;
sudo nano /etc/init/ipv6-eth0.conf
… pre-start script … /sbin/ip -6 addr add 2a03:ca80:8001:769d::80:1 dev eth0 /sbin/ip -6 addr add 2a03:ca80:8001:769d::80:3 dev eth0 /sbin/ip -6 addr add 2a03:ca80:8001:769d::80:4 dev eth0 /sbin/ip -6 addr add 2a03:ca80:8001:769d::80:a dev eth0 /sbin/ip -6 addr add 2a03:ca80:8001:769d::80:c dev eth0 initctl emit ipv6-eth0-ips end script pre-stop script … /sbin/ip -6 addr del 2a03:ca80:8001:769d::80:1 dev eth0 /sbin/ip -6 addr del 2a03:ca80:8001:769d::80:3 dev eth0 /sbin/ip -6 addr del 2a03:ca80:8001:769d::80:4 dev eth0 /sbin/ip -6 addr del 2a03:ca80:8001:769d::80:a dev eth0 /sbin/ip -6 addr del 2a03:ca80:8001:769d::80:c dev eth0 end script
sudo nano /etc/init/ipv6-lo.conf
… pre-start script … /sbin/ip -6 addr add fdd7:5938:e2e6:6c8d::80:1 dev lo /sbin/ip -6 addr add fdd7:5938:e2e6:6c8d::80:c dev lo initctl emit ipv6-lo-ips end script pre-stop script … /sbin/ip -6 addr del fdd7:5938:e2e6:6c8d::80:1 dev lo /sbin/ip -6 addr del fdd7:5938:e2e6:6c8d::80:c dev lo end script
After a reboot, everything should be functioning. Two of the sites that are functioning on vps2 use MySQL, so I still need to do some work on getting those working, and two of the resolvers listed are located on vps2 so I still need to add a recursive DNS server on vps3.
Another thing I have yet to do is to allow traffic to nginx through the firewall. That is a simple case of adding the IPv4 and IPv6 IP addresses to my ip6tables.save and iptables.save files and accepting incoming connections to those IPs on ports 80 and 443.
Back on vps2, I can now test my active sites on vps3 are working properly. First, I need to install gnutls:
sudo apt-get install gnutls-bin
With gnutls installed, I can now test that watfordjc.co.uk and johncook.co.uk are working:
gnutls-cli -p 443 2a03:ca80:8001:769d::80:1 --insecure GET /news.php HTTP/1.1 Host: web.watfordjc.co.uk
HTTP/1.1 502 Bad Gateway …
gnutls-cli -p 443 2a03:ca80:8001:769d::80:c --insecure GET / HTTP/1.1 Host: web.johncook.co.uk
HTTP/1.1 200 OK …
Web.JohnCook.Co.UK is working, but Web.WatfordJC.Co.UK is having gateway trouble. That is to be expected because the gateway used by nginx for web.watfordjc.co.uk is 127.0.0.1:9000 and I haven't (yet) installed PHP. Also, even if the gateway was working, I haven't copied over /home/www/var/www/ so there aren't any PHP files to process if the processor was installed.
According to netstat, I am using php5-fpm on vps2. I will use it on vps3 as well then.
sudo apt-get install php5-fpm sudo so rsync -avrpPlog --progress -e 'ssh -p 8043 -i /home/thejc/.ssh/thejc.vps3' firstname.lastname@example.org:/etc/php5/fpm/ /etc/php5/fpm/ mkdir -p /home/www/var/www/ mkdir /home/www/var/log/ chown -R www-data:www-data /home/www/ rsync -avrpPlog --progress -e 'ssh -p 8043 -i /home/thejc/.ssh/thejc.vps3' email@example.com:/home/www/var/www/ /home/www/var/www/ exit sudo service php5-fpm restart sudo service nginx restart
When testing Web.WatfordJC.Co.UK now, I get a 500 Internal Server Error. A look in /etc/php5/fpm/conf.d/ shows a lot of symbolic links are missing targets: I need to install some php modules.
sudo apt-get install php5-gd php5-mysql php5-readline
I now get a 200 OK response, and a MySQL error 1045 (access denied).
Exporting and Importing MySQL Databases and Users
There are two things that need to be done in order to (hopefully) get these two Web sites functioning. The first is to dump and import the relevant databases. The second is to re-create the users.
The easiest way to re-create specific users is to follow some of the instructions in the article MySQL - Migrate Users from Server to Server. It should be noted I said specific users—I don't want to re-create all the users as it might overwrite existing ones (like root) which is not what I want to do.
mysql -u root -N -p -s > ~/mysql-users
select Distinct CONCAT('show grants for `', user, '`@`', host, '`;') as query from mysql.user; quit
Now, edit ~/mysql-users and remove all the lines of users we aren't interested in (i.e. all the lines except those for the two database users we want).
With that done and mysql-users saved, we can now get all the grants we need for vps3.
mysql -u root -N -p -s < ~/mysql-users > ~/mysql-grants scp -P 8043 -i ~/.ssh/thejc.vps2 mysql-grants firstname.lastname@example.org:
Then, over on vps3, use mysql-grants to create the users:
sed -i 's/$/;/' ~/mysql-grants mysql -u root -p < ~/mysql-grants
Another test using gnutls-cli from vps2 and the MySQL error has changed to 1049 (unknown database). We are on the right track.
Now we need to export the two databases:
mysql -u root -p
flush tables with read lock; set global read_only = ON; exit
mysqldump -u root -p thejc_watfordjc > mysql-thejc_watfordjc-2015-05-12.sql mysqldump -u root -p djlane_djlane > mysql-djlane_djlane-2015-05-12.sql scp -P 8043 -i ~/.ssh/thejc.vps2 mysql-thejc_watfordjc-2015-05-12.sql email@example.com: scp -P 8043 -i ~/.ssh/thejc.vps2 mysql-djlane_djlane-2015-05-12.sql firstname.lastname@example.org:
Then, over on vps3:
mysql -u root -p
create database thejc_watfordjc; create database djlane_djlane; flush privileges; exit
mysql -u root -p thejc_watfordjc < ~/mysql-thejc_watfordjc-2015-05-12.sql mysql -u root -p djlane_djlane < ~/mysql-djlane_djlane-2015-05-12.sql
One more test from vps2 using gnutls-cli, and /news.php is returned without an error.
Changing IP Addresses
First up, we need to change the backend in varnish on vps3. This is a simple case of editing /etc/varnish/user.vcl, changing fdd7:5938:e2e6:9660::80:c to fdd7:5938:e2e6:6c8d::80:c and changing johncook_vps2_nginx to johncook_vps3_nginx (not forgetting to rename the backend in the fallback block).
A restart of varnish later, and we can now look at the nginx upstream servers.
Edit /etc/nginx/conf.d/upstream-servers.conf and replace :9660: with :6c8d:.
After restarting nginx, we can make the first test change. At this point it is advantageous that Web.JohnCook.UK is on Cloudflare because Cloudflare acts as a reverse proxy. Thus I can make a change to the IPs in Cloudflare and quickly revert without having to worry about DNS TTL waits (propogation).
Changing IP addresses in Cloudflare was a simple matter of copying a new IP to the clipboard, clicking on the value of the A/AAAA record, selecting all the text (triple clicking), pasting, and then clicking outside the table area.
With Cloudflare'd site still functioning, I edited the nsd zone files for the sites that have migrated pointing the A/AAAA records to the new locations.
While the migrated sites are working, there are a much larger number of sites that aren't working as I didn't restore them after restoring vps2 after an attempted upgrade. Those sites aren't a priority at the moment, and in fact most of them are to be merged with my new domains at a later date.
Anyway, at this point I have successfully completed the goal of this article: migrate Web sites from my old VPS to my new one and confirm they are still working. For now I will leave the MySQL databases on vps2 in read-only mode because the only database that may need writing to at a later date is the mail database.
I only add new e-mail addresses once a month on average anyway, but it should be noted this may be an issue at a later time.