Managing ESXi host DNS using Ansible 2.0

With the release on Ansible 2.0 came several new VMware modules that leverage pyVmomi, one of which was vmware_dns_config. This modules runs directly against ESXi hosts, so if you have small lab with no vCenter (because reasons?) you can still use this module to ensure that the host(s) have the desired DNS servers. As with the previous “vsphere” modules which I demoed last year on the #vBrownBag DevOps series, this tasks runs locally but leverage the previously mentioned pyVmomi instead of pySphere.

As of this writing, the Ansible documentation lists vSphere 5.5 as the only supported version. I am currently running the vCenter Server Appliance (VCSA) 6 in my lab, along with ESXi 6 hosts, though be aware YMMV.

Before you get started, I had a bit of head banging going on trying to determine how to setup Ansible properly. I had been using pyVmomi 5.5.0 as well as 6.0.0 without much success, Larry Smith in his testing found that 5.5.0-2014.1.1 seems to work (best), so when you install make sure you run

sudo pip install pyvmomi==5.5.0-2014.1.1

Now, once that is working, just a nice basic playbook in order to change DNS which you can find in my test playbook repo on GitHub. Here is a “before screenshot with one of my hosts having the incorrect DNS server:

Incorrect DNS servers for VMware ESXi host

Incorrect DNS servers for VMware ESXi host

Now, after running the playbook you can see that it has been corrected.

Corrected DNS servers for VMware ESXi host

Corrected DNS servers for VMware ESXi host

Enabling Disaster Recovery with Ansible

I always enjoy sharing my “ah-ha” moments, that is when you start to do something – or in this case are already doing something, that helps with other projects. This is a brief story about how we used Ansible to enable disaster recovery without needing to replicate virtual machines, or buy additional software.

I was working at a startup that had legal requirements to maintain a disaster recovery plan, yes Frank we had disaster recovery at a startup! The disaster recovery plan, however was put together in support of previous versions of our software, and needed to be updated to support our new version. On top of the legal requirements, we had a new potential customer who wanted to see our disaster recovery plan in action – this customer would be on our new platform, so the old plan wasn’t an option!

It was a pretty standard application, nothing crazy or “web-scale” to the likes of a Netflix, there were several reverse proxies, application front ends, management systems, etc. There were certain application configuration files, and several auxiliary files used by the application on a day to day basis that were stored on the virtual machine running the application. While important, those configuration and auxiliary files could also be recovered from backups, but important nonetheless. The really important application data was stored and accessed by our application on a EMC Celerra NAS. The virtual machines running the application were also of the “pet” variety – tweaked, and updated, and patched since launch to get them working just right. We didn’t want to lose this information.

As a VMware shop, it seemed pretty obvious what we needed to do, buy VMware SRM and go on with life. I spent several weeks researching SRM, understanding how it would work within our environment and coming up with a solution for replicating virtual machines to the DR site. At the time, we only replicated our Veeam backups to DR, along with replication of our production Celerra to a VNX at the DR site. My manager and I were talking through what we would need to do with SRM in order to reconfigure the application servers in order to work properly at the DR site, it was getting hairy but obviously doable.

I’d like to pause here and rewind a bit. While we were planning our new DR strategy, and ensuring we could meet customer timelines, one of my co-workers was in the process of migrating from bash scripts to manage and deploy software to a configuration management tool; we had selected Ansible but I suspect you could achieve similar results with other tools. The Ansible implementation was going great, that is to say Ansible was very easy to deploy, we were just making sure that modules were being leveraged in the correct way rather than just making a big playbook with a bunch of shell or raw modules using the same commands as the bash script.

That’s when the light bulb light up! As we were talking through how we would use SRM to alter the configuration files for the application, we realized that Ansible was already managing the configuration of the application, the application configuration files stored on the virtual machine which would need to be edited to support the network at the DR site all of a sudden seemed a little bit less important. We still, however had this auxiliary files the application would access (think public keys for decryption information, etc) that were stored on the virtual machine. What should we do? Well we were already storing the actual application data on a NAS, which was already being replicated and available at the DR site; what if we moved that auxiliary type data into a share and mounted it so it could be accessed as well? Once replication was setup, all we needed to do now was have Ansible mount the file system so it could be accessed, we tested this theory out and sure enough it worked.

But, as many shops have done, there were hand made tweaks to the application servers to improve performance, could we get away with losing that information? As you might expected, we couldn’t, but we made the decision to work through performance issues and add those to the Ansible playbook so moving forward, there was nothing specific to a virtual machine in any environment that couldn’t be managed, controlled, and configured through Ansible. What we didn’t have the option of then, that you could do today is use a tool like ScriptRock to monitor and record the configuration of your servers, as well as monitor and alert on drift.

Our application could now be fully deployed in production, as well as any other environment we needed, including DR with just a basic virtual machine template and an Ansible playbook.

2016 Q1 Goals

It’s 2016, time to set some goals, and one of those goals will not be supporting podcasts EVERY weeknight of the month :) Though I do hope you enjoyed #Commitmas! Understanding where I have failed before I decided to set goals a quarter at a time, and give myself a hard deadline for those goals.  I’ve also added some optional items in case I get things done a bit quicker than expected. Feel free to book mark this page to see how I did, share your goals, or ask questions about what I am doing. As always, you can find me on Twitter @jfrappier.

Jan through March Goals
Understand Ansible 2.0 VMware Modules (End of Jan)

Free Code Camp HTML5 & CSS (Jan 15th) checkmark
Free Code Camp Responsive Design w Bootstrap (Jan 29th)
Free Code Camp jQuery (Feb 5th)
Free Code Camp JavaScript (Feb 19th)
Free Code Camp Object Oriented Programming (Feb 26th)

Optional Items I would like to learn

HashiCorp Terraform & Otto

Nginx Fundamentas via Pluralsight

April through June Goals (Tentative)
Contribute back to Ansible via documentation update and update VMware Cluster module to support EVC (End of June)

Free Code Camp Basic Algorithm Scripting (May 27th – estimate is 50 hours)

Ansible all the VMware things

Last year, about this time I gave a demo on the #vBrownBag DevOps series about using Ansible with vSphere. At the time, there was functionality that allowed you to define new virtual machines, or clones of templates in an Ansible playbook; a very handy feature especially since you should have some form of configuration management anyways.

Fast forward a year – Ansible 2.0 (currently in RC/pre-GA) has added several new VMware specific modules to help you manage your VMware environment, you can see a full list of all modules on the Ansible Docs site (one of my favorite documentation sites). In this post, I will give a brief overview of some of the modules before I dive in deeper to each one. *Note that since this is currently pre-GA, you will need to install from source to get the bits, other wise you will have access to all of these.

If I think about the typical deployment process, the first step after deploying vCenter would be to create a data center – Ansible has you covered with the vmware_datacenter module. Next I would create a cluster and enable the required options such as HA and/or DRS – yup Ansible has you covered there as well with vmware_cluster – though currently this module is missing the ability to enable EVC, something I hope will be added.

Next I would add hosts to my cluster (vmware_host), configure 1 or more vmkernel adapters (vmware_vmkernel), provide the vmkernel adapter with an IP addres (vmware_vmkernel_ip_config) and create a VDS (vmware_dvswitch), a standard switch for management (vmware_vswitch) and necessary port groups (vmware_portgroup for VSS and vmware_dvs_portgroup for DVS).

There are also options manage host DNS using vmware_dns_config – it’s always a DNS problem right? So why not have configuration management for your hosts DNS settings? And I can event get to a the shell of a VM to execute processes using vmware_vm_shell – maybe an agent to be installed or even a script that can write back to my Ansible control machines hosts file so I can easily manage those new VMs from Ansible!


Ansible 2.0 has added a ton of great new modules, not only for VMware admins but several cloud providers. Having the ability to manage the configuration of my vCenter servers makes them much more “lightbulb” like – if its bad, nuke it and redeploy, makes for a easy documentation or DR setup.

Ansible Playbook to install Varnish

If you need to install Varnish, here is a quick little Ansible playbook to get it installed. Updated your host file and user as needed. I am still reading up on Varnish, so haven’t got much past the install yet.

- hosts: varnish (update to match your hosts file)
  remote_user: virtxpert (update to match your user account
  sudo: yes

  - name: Install apt-transport & curl
    apt: name=apt-transport-https state=present
    apt: name=curl state=present

  - name: Get Varnish key
    apt_key: url= state=present

  - name: Add Varnish sources to sources.list.d
    register: aptrepo

  - name: Apt Update
    apt: update_cache=yes

  - name: Install Varnish
    apt: name=varnish state=present

LogEntries by Rapid7 – Free Syslog in the Cloud!

Some people turn off cookies, me… I love them. Every now and again I am reminded of something on eBay or Amazon I looked at, and more often than not I find a little gem like LogEntries by Rapid7.

LogEntries is a log management system, think along the lines of LogInsight and Splunk, but in the cloud. The first 5GB per day is completely free (as long as you are cool with your logs being shipped to Rapid7) and signup actually works (not like those companies who make you talk to a person first!!). Once you sign up, you chose how you want to send your logs, either via an OS agent, standard syslog tools like Rsyslog, Syslog-NG, or Snare or even from supported applications such as PHP or Ruby.


If you are looking for a logging solution, with little to no ramp up time, check out LogEntries.

Software developers are the next wave of utility workers

During the #vDM30in30 i’ve not been bashful with my blog posts that might upset some folks, VMware is just as complex as OpenStack I’d never run KVM for an SMB and now this. The premise, just as “IT” people have been relegated to being classified as a basic provider of utility, so to will the software developer.

In the early days of computers, IT workers walked around thinking they were the reason the company existed, and it was a sh!ty attitude. The reality, IT is a service provider to support what a business is trying to do though technology, just as marketing is there to support the business by marketing it properly. As technology has advanced, the complexity of the infrastructure has dropped to the point where it is often compared to being an electrician. Take for example Active Directory, years ago you made a career out of being an Active Directory administrator….today it’s almost just a checklist item as organizations try to crunch budgets in their IT departments (and by the way that is wrong, AD is complex, and powerful…if you are just using AD for basic authentication you are doing it wrong). Server hardware is also been reduced, if you are hanging the next 20  years of your career on creating LUNs and configuring VSANs (the FC switch kind not the VMware kind) then you are likely in for a rude awakening as hyperconvered and pre-architected solutions take over and it becomes a matter of rolling in a rack and powering it on.

So, my question…when will that happen with software? When will the “my 5th grader can setup a VNX” change to “my 5th grader can code this app”? We are pushing software development and software interaction earlier and earlier in schools. The reality is many business people do not understand the complexity and artistry that goes into writing good code, just as they don’t understand the complexity and artistry of a good cable or server design.

Before you go throwing IT people under the bus, look ahead at your field. If you don’t think business will be looking to make producing code cheaper, I think you will be joining the SAN engineer for some extended time off in the future.

Teaser – Learning VMware NSX by Ranjit Singh @rjapproves

I have been fortuneate enough in my day job to get hands on experience with VMware NSX, even before the bits were available to download I was supporting NSX via the Federation Enterprise Hybrid Cloud, as it is one of the core components. For people still looking to get a jump start on learning NSX, I wanted to give you a bit of a teaser for an upcoming book by Ranjit Singh – Learning VMware NSX.


Since I am a technical reviewer for this book, which is being published by Packt Publishing, I can’t give away to much but can tell you it is packed with step by step examples on how to get up and running quickly with NSX and understand the various components and how they interact. Keep an eye out on the Packt site, and I expect you’ll here more from me and @rjapproves when it is released!