Fatih Mehmet Güler

1. Initial ESXi Setup

Hetzner did it for us, for 25€. The thing is since ESXi cannot route, it needs a separate public IP to reach the management console, via vSphere client. And for our current plan we have software RAID, and surprise, ESXi does not support it, so we have two 3TB datastores. Add the second one from configuration -> storage. Other installation options are followed from: ​http://wiki.hetzner.de/index.php/VMware_ESXi/en

2. Virtual Switch and Gateway VM Setup

Since we cannot afford a new subnet we use a single machine as a reverse proxy, and set other VMs local IPs. Even if we happen to afford subnets, Hetzner guide says that it is required to route subnet to a router VM on ESXi (they don't want to do the routing). To add virtual switch, go to Configuration -> Network (in vSphere) and click Add Networking. Select Virtual Machine, create standard switch, give the label "Company Local Network". Create a new VM, named "Company Gateway", 64 bit Ubuntu Server (512MB), 2 NICs, one on Company Local Network, one on VM Network. Before finishing set the VM Network NIC MAC to the MAC given by Hetzner (00:50:56:00:XX:YY). Install Ubuntu Server. Install SSH server. Follow Initial User Management section from development infrastructure  page (restrict ssh access, remove root). Give 10.0.0.1 IP to the local NIC. Uncomment net.ipv4.ip_forward=1 on /etc/sysctl.conf and do echo 1 > /proc/sys/net/ipv4/ip_forward. See: ​https://nowhere.dk/articles/tip_nat_with_ubuntus_ufw_firewall, ​http://codeghar.wordpress.com/2012/05/02/ubuntu-12-04-ipv4-nat-gateway-and-dhcp-server/, and ​https://help.ubuntu.com/12.04/serverguide/firewall.html#ip-masquerading. Then, setup other VMs on the Network, with static IP's on 10.0.0.x network.

3. ESXi Backup with GhettoVCB

Since the dedicated machine with ESXi does not have software raid, we need to backup regularly in case of hard drive failure. Note that this backup is separate from data (db & files) backup, which is machine specific and taken to Dropbox (off-site backup). (​http://communities.vmware.com/docs/DOC-8760) To setup backups ssh to management interface, create a folder named ghettoVCB, upload ghettoVCB files here, update conf file and run with ./ghettoVCB.sh -a -g ghettoVCB.conf which will backup all machines to the location specified in conf file. It is this straightforward. To recover just copy back to first datastore or add to inventory directly. Additionally add crontab to backup daily. Edit /var/spool/cron/crontabs/root add: 0 0 * * * /vmfs/volumes/datastore2/config/ghettoVCB-master/backup.sh > /vmfs/volumes/datastore2/config/ghettoVCB-master/ghettoVCB-backup-$(date +\%s).log Note that this is not persisted accross reboots. Do not reboot unless very necessary. A lot of configuration is reset on reboots, need to check extensively. (NOTE: we can setup software raid or iSCSI for important data files, for instantaneous hard disk failure)

4. ESXi Security

I followed this link to secure ESXi ​http://serverfault.com/questions/309942/how-to-secure-an-vmware-esxi-host-on-a-root-server

Changing ssh port was very PITA. I could not access via SSH, since default firewall thinks that ssh port is 22 but I changed it to [some another port]. I had to access to console with LARA, change port back to 22 (/etc/services) and then add firewall rule as in ​http://communities.vmware.com/message/1853590. As netstat in ESXi use: "esxcli network ip connection list". From vSphere Client firewall, disabled all incoming ports except: 902, 68, 22 (no ssh), 161, 546, 53, [ssh port], 902, 443. Scanned with nmap and only 443,902 and [ssh port] ports are open. Add ssh keys and disable password login as in: ​http://www.howtoforge.com/ssh_key_based_logins_putty. Then changed access to 902/403 from 127.0.0.1/32 from vShpere client firewall. Follow: ​http://www.virtuallifestyle.nl/2010/03/tunneling-a-vsphere-client-connection-over-ssh/. The only open port is [ssh port].

5. Gateway VM Other Services Setup

NAT configuration with Gateway VM is explained at step 2. Here other services for Gateway VM such as VPN, Mail, DNS are configured. These services used to be at Development machine since it was the only machine (at linode), but here we install all infrastructure services to the gateway vm so that all other machines are just dump VMs at local network.

First enable key only login as in ​http://www.howtoforge.com/ssh_key_based_logins_putty.

Then, install OpenVPN according to ​https://help.ubuntu.com/12.04/serverguide/openvpn.html. 

Install nginx (future: consider cherokee). Note that nginx configuration files are at /etc/nginx/sites-available and they are enabled by symbolic link to sites-enabled. Configuration files are backed up to Dropbox/machines/gw/config/nginx. Create symbolic links with "sudo ln -s /etc/nginx/sites-available/trac.yoursite.com trac.yoursite.com" command at sites-enabled folder. Start nginx with "sudo /etc/init.d/nginx restart"

Install bind. ​https://help.ubuntu.com/12.04/serverguide/dns-installation.html. I imported the zones from linode DNS manager. Zone config files are backed up to Dropbox/machines/gw/config/bind

Install mail with command "sudo apt-get install postfix". Select internet site and mail name as mail.yoursite.com. Forward all mail sent to here to sysadmin as in ​http://www.cyberciti.biz/faq/linux-unix-bsd-postfix-forward-email-to-another-account/.

Install fail2ban with "sudo apt-get install fail2ban".

Install tiger.

sudo apt-get install fail2ban
cd /etc/fail2ban
sudo cp jail.conf jail.local
sudo vi jail.local
//change destemail to [email protected]
//change action = %(action_)s to action = %(action_mw)s
//change ssh maxretry to 3
//save
sudo vi /etc/rsyslog.conf
//change $RepeatedMsgReduction off
//save
sudo /etc/init.d/rsyslog restart
sudo /etc/init.d/fail2ban reload

6. To Be Done

In order to get back to work as soon as possible, some of the features are ommitted. Here are some future features to be done;

• Create a software RAID on VM; with harddisks on datastore 1 and 2. Mount this new drive as separate partition on /apps/data folder. So, in case of catastrophe (hd failure), we recover VM backup from datastore 2 and data folder from RAID without any data loss.

• For complete high availability on catastrophe, duplicate the ESXi infrastructure on another dedicated server, arrange backups to cross datastore 2 (if possible), and setup the data partition NOT as software RAID but as iSCSI, so that distributed VMs can share data folder. You can postpone this until you have a lot of paying customers...
  • Aside from shared data folder complete high availability setup requires; failover IP, nginx round robin (or haproxy), jetty session share, pgpool, and each passive ESXi checks the active and calls hetzner api to set itself as IP failover. Active IP distributes the load to passive ESXi jetty servers. This would be the ideal case, but some down time is not bad compared to this overhead.
  • Another option could be to start an Amazon EC2 (or linode) instance in case of hw fail until fixed. This may be the least extra overhead option until we need new ESXi only for loadbalancing.