Quick Ansible playbook for installing Sysdig

Been thinking about Sysdig, and how it can be used for troubleshooting. One thought I had was to capture events during an Ansible playbook run in the event there were any problems. Now I’m not sure how practical that is just yet, but the first task was getting Sysdig installed. Of course, that meant writing an Ansible playbook to do so (really should have been a role probably but baby steps).

You can find the sysdig.yml file for Ubuntu/Debian in my test playbooks repository on GitHubhttps://github.com/jfrappier/ansible-test-playbooks

Playbook is based on the directions from sysdig.org, and tested on Ubuntu 14.04. As always, I am still learning here but feel free to update as you see fit and take it all with a grain of salt.

---
- hosts: parent
  vars:
      package: sysdig
      sysdig_key_url: https://s3.amazonaws.com/download.draios.com
      sysdig_key: DRAIOS-GPG-KEY.public
      dl_dir: /downloads
      sysdig_repo: http://download.draios.com/stable/deb stable-$(ARCH)/
      linux_headers: linux-headers-{{ ansible_kernel }}
  remote_user: [ENTERUSER]
  sudo: yes

  tasks:
  - name: Validating download directory
    file: path={{ dl_dir }} state=directory

  - name: Download Sysdig public key
    get_url: url={{ sysdig_key_url }}/{{ sysdig_key }} dest={{ dl_dir }} validate_certs=no

  - name: Installing Sysdig public key
    apt_key: file={{ dl_dir }}/{{ sysdig_key }} state=present

  - name: Adding Sysdig apt repository
    apt_repository: repo='deb {{ sysdig_repo }}' state=present

  - name: Update apt repositories
    apt: update_cache=yes

  - name: Install Linux Headers
    apt: name={{ linux_headers }} state=present

  - name: Install Sysdig
    apt: name={{ package }} state=present

Modules used in this playbook:

  • http://docs.ansible.com/apt_repository_module.html
  • http://docs.ansible.com/apt_key_module.html
  • http://docs.ansible.com/apt_module.html
  • http://docs.ansible.com/file_module.html
  • http://docs.ansible.com/get_url_module.html

Using ansible-galaxy init to create roles

Ansible held a free online 2 hour introduction session, and while I’m not an expert I do feel I have a good handle on some of the items such as inventory files, and playbook formats. However there is always something to learn! One thing I took away from todays training was an ansible-galaxy command.

This command can save a lot of manual effort up front when creating roles. It will create the basic folder structure and files necessary for an Ansible role – something up until know I’ve been doing by hand. To use it is simple, just type the command followed by the role you are creating. For example if you were creating a role for PostgreSQL you would simply type:

ansible-galaxy init postgres

And this would create the folders such as handlers or tasks, and a main.yml file where appropriate. It was new to me, so thought I’d share!

vRealize Application Services Home Lab Upgrade

As I did in the previous post with vRealize Automation, it is now time to upgrade vRealize Application services, again based on KB2109760 this would be the second item to upgrade before upgrading vCenter with embedded SSO. Not that it is horribly difficult, but there is no management interface as we had with the vRealize Automation appliance so we will have to download the files, copy them to the appliance and start the upgrade.

Before you being, ensure you can log into the console of the application services virtual appliance as root and SSH as darwin_user. If you are unable to SSH as darwin_user follow the directions here to enable the darwin_user account. Now, download the VMware vRealize Automation Application Services 6.2.0 upgrade installer from downloads.vmware.com. Once the file has been download, copy the .tgz file to the appliance, for example if you are using Windows you might use WinSCP to copy the file. Once the file is on the system, SSH to the appliance and navigate to the directory you placed the file in. For example here you can see the tgz file in the 62-upg folder I created.

apps-upg-step1

Next, untar the file by running tar xvfz ApplicationServices-6.2.0.0-2299597_Upgrade_Installer.tgz (or the appropriate build number based on your download). Once all the files have been extracted, you should have an install.sh file ready to run (no need to chmod to be executable). Run the install as root by running sudo ./install.sh (or sudo -su, then ./install.sh as the VMware docs state)

apps-upg-step2

Type Y to start the upgrade and the rest is scripted for you. Once the installer finishes, restart the vRealize Automation appliance and Application Services appliance, when the appliance reboots, you should be able to log in at https://{appsservicesURL}:8443/darwin/org/{vratenant} – for example https://vxprt-apps01.vxprt.local:8443/darwin/org/vsphere.local

apps-upgrade-ui

You are now on Application Services 6.2 (as seen in the lower right corner in the above screenshot).

vRealize Automation Home Lab Upgrade

With new versions of vRealize Automation and vSphere dropping, and seemingly being stable it is time to upgrade the home lab. Since this is a home lab, and somewhat basic there are just a few steps from KB2109760 that needs to be followed:

  1. Upgrade vRA (Appliance >> IaaS)
  2. Upgrade Application Services
  3. Upgrade vCenter
  4. Upgrade ESXi

In this post, I will cover the first step in the process, upgrade vRealize Automation to 6.2.latest. First, I have shut down services on my IaaS server. Now log into the VMware vCAC Appliance management interface on port 5480 – in my case https://vxprt-vcac01.vxprt.local:5480 for example and click on the update tab. Now, click on Check Updates. As you can see here, I have an available updated from 6.1.1.0 to 6.2.1.0

vcac-upgrade

Now, as you might expect, click on Install Updates >> OK. The upgrade process will begin.

vcac-upgrade-starting

After a few minutes, you should be presented with a message that a reboot is required.

vcac-upgrade-complete

Click on the System tab, click the Reboot button, and click the Reboot button again; the system will reboot. Once the reboot completes, you should be able to log in and verify the version by clicking on the system tab. Notice anything different?

vra-branding

The updated product name; vRealize Automation is now displayed instead of vCAC Appliance and the version is 6.2.1.0. Once all the services have started, you should also be able to log into the vRealize Automation console and see the tenant information from the previous configuration.

vra-console

The next step is to upgrade the IaaS components. Again this should be straight forward in a lab because all of the components are on a single server, and not distributed. Log into the IaaS server as the service account used to run the IaaS components, if you followed along in my vDM 30-in-30 challenge you would have named it something along the lines of svc_vra_iaas. Open a web browser and grab the vRA 6.2 PreReq script Brian Graf has built over on GitHub (https://github.com/vtagion/Scripts/blob/master/vRA%206.2%20PreReq%20Automation%20Script.ps1). Save, open a PowerShell console as administrator and run the script.

vra-iaas-script-upgrade

Follow the prompts in the prereq script, typically I have selected option 2 – I have internet access and want to download and install it automatically.

 

vra-iaas-net-upgrading

Select option 2, 2 more times. When prompted, provide the service account for the IaaS components, in my case vxprt\svc_vra_iaas and the script should complete.

vra-script-complete

Now, navigate to the vRA appliance page. Click on the vRealize Automation IaaS installation page link, download and extract the zip file containing the database upgrade scripts. From a command prompt run the following command:

dbupgrade.exe -S {servername\instancename} -d {dbname} -E

On my server I am using the default SQL Express instance, so the instance name is not needed, and my DB name is vCAC so my command looks like this:

dbupgrade -S localhost -d vCAC -E

db-upgrade

If you are receiving any errors, make sure that Named Pipes is enabled.

sql-named-pipes

Now that the DB is upgraded, download the IaaS Installer file, do not rename the file, and run it. The upgrade is of the next, next, next variety.

  1. Click Next
  2. Accept the terms and click next
  3. Enter the root password for the vRA appliance, accept the certificate, and click Next
  4. Upgrade should be the only option that is available, click Next
  5. Enter the service account password, the server name, database name, and click Next
  6. Click the Upgrade button

If the computer gods are on your side, the installation should complete

iaas-upgrade-done

Click Next and Finish. If you flip back over to your vRA console, you should see all of the available tabs based on the user permission – in this case my iaasadmin user.

vra-portal

Up next is upgrading Application Services.

Home Lab – $1250 8-Core / 32GB / 750GB Flash / 2TB HDD 2015 Edition

11-139-022-TSIt was a bit over a year ago that I wrote about my 8-core home lab. I was asked if there were any updates to the build and I was curious to see how it stood up a year later. Happily for me, and anyone who has invested in this build, the same basic platform is still a solid option for your home lab. I have made a few tweaks below based on some new hardware being available. As I did last year, there was a focus on keeping cost down but having enough power to run a fully nested home lab.

With 32GB I have been able to run Windows 8.1 and VMware Workstation with 3x nested ESXi 5.5 hosts each with 8GB of RAM. One of those host runs the vCAC / vRA appliance, one runs the Application Services appliance, and the 3rd is used when provisioning virtual machines. In addition to the 3x nested hosts, I run a 5.5 VCSA at 4GB RAM, Windows 2012 R2 DC, Windows 2012 R2 vCAC / vRA IaaS with SQL Express on the same virtual machine, and CentOS 5.5 running Ansible in Workstation. With everything powered on I run at about 85% memory utilization and only push the CPU’s during provisioning processes.

The hardware…

CPU:  AMD FX8320 – This is the exact same processor as last year. It is an 8-core AMD processor that fully supports nested ESXi and 64-bit virtual machines running on the nested ESXi hosts.

Motherboard: ASRock 990FX Extreme6 – This is a new motherboard for 2015, versus the Asus I used last year (though that board is still available). The reason for the change, the ASRock Extreme6 supports up to 64GB of memory where as the Asus only supported 32GB. Now, having said that this build still uses 32GB because the 16GB memory modules are $190 each, compared to 4x 8GB (32GB total) modules being $210 TOTAL. This board has onboard RAID and has 5 6Gbps SATA ports.

Memory: G.SKILL Ripjaw X Series – Similar memory to what was used last year, just not in a full kit so pick up 4 of these.

Flash: Crucial MX200 – These were used instead of the Corsair Neutron drives I used last year, for no other reason than saving a few dollars to upgrade in other areas. The Neutron drives have been great for the last year, no problems to report so far. At $1250 you can pick up 3 of these if you like, or drop the price of your home lab.

HDD: Seagate Hybrid 1TB – I again opted for the hybrid drives for bulk / lower tier storage. I run most of my lab off these drives, configured in a RAID0. I opted for 2 of these.

NIC: Intel Dual-port 82575 – Because HCL, and wanted the possibility to install clean on baremetal. If you go the VMware Workstation route, you could skip this card potentially unless you would like more ports to get fancy with. You could again lower your cost here by going with a used card as I ended up doing last year like the HP7170 from Amazon.

Video: MSI …whatever – This is here because the motherboard doesn’t have on-board video. Buy a card based on your needs, I went cheap here because i don’t use the box for any sort of gaming. If you’ll have other uses, obviously look at your requirements.

Case: Corsair Air 540 – Case again is getting into personal preference area. The graphite 230T I used last year is perfectly capable. The Air 540 has 4 internal 2.5″ drive bays and 2 hot swap drive bays to support the 3x SSD and 2X HDD drives.

Power Supply: Rosewill RD600-M – This is the new version of the power supply used last year, which has been stable for me even through a faulty UPS.

Preparing Ubuntu template virtual machines

Bob Plankers has a great post over at lonelysysadmin.net for preparing CentOS based virtual machines for being a template. As I’ve started working with Ubuntu more I decided to take that list and Ubuntu-ize it (mostly from proding by Sarah Zelechoski – one of the smartest people I’ve ever had the privilege to work with…so many thank you’s). Anyways here is that guide… Ubuntu-ized.

Stop logging services (auditd and rsyslog):

service auditd stop
service rsyslog stop

Check for, and remove old kernels

Check your current kernel by running

uname -r

Then run

dpkg -l | grep linux-image-

If additional images are listed, remove them by running

apt-get autoremove linux-image-#.##.#

You can remove multiple images all on the same line just by listing them one after another.

Clean out apt-get

apt-get clean

Force the logs to rotate & remove old logs we don’t need

logrotate –f /etc/logrotate.conf
find /var/log -name "*.gz" -type f -delete

Truncate the audit logs (and other logs we want to keep placeholders for)

cat /dev/null > /var/log/audit/audit.log
cat /dev/null > /var/log/wtmp
cat /dev/null > /var/log/lastlog

Remove the udev persistent device rules

Well, saved a step here – there are rules which exclude creating files that match MAC addresses for VMware, Hyper-V, KVM, Xen, and virtualbox (see /lib/udev/rules.d/75-persistent-net-generator.rules). So long as your MAC matches this, nothing to clean up. Otherwise

rm -f /etc/udev/rules.d/70-persistent-net.rules

It will be recreated on the next boot, so any time you power on this VM (updates maybe?) you’ll need to delete this file again so it is not saved in the template.

Remove the traces of the template MAC address and UUIDs.

Here is another step you shouldn’t need to do, however you may want to check /etc/network/interfaces to verify

Clean /tmp out

rm -rf /tmp/*
rm -rf /var/tmp/*

Remove the SSH host keys

rm –rf /etc/ssh/*key*
rm –rf ~/.ssh/authorized_keys

Update network config

If you have set /etc/network/interfaces, make sure to reset for cloning purposes. For example as I wrote this it had a static IP address which I changed to DHCP before shutting down and converting to a template.

Remove hostname

If you have named your virtual machine anything other than localhost, and want the template to spin up with a generic name, versus say “ubuntu-template” remove entry from /etc/hostname

cat /dev/null > /etc/hostname

Remove the user’s shell history

If you have switched to root at any point, do this as root and individual user accounts

history -w
history -c

That should about do it, depending on where this template is going, make sure any ISOs attached to the CD-ROM or networks for the NIC’s are adjusted properly. While many of the steps were the same there were a few differences to be aware of. Anything else you like to clean out? Comment below please!

Hands on with Microsoft Visual Studio Code @code

As a Windows user I have been looking for a good markdown tool to write in, however most of the tools freely available have been mediocre at best. Enter Visual Studio Code, a (currently) free download from Microsoft that codesupports Windows, OSX, and Linux (OSX/open source gear heads take notice – write software cross platform!). You can download Code without any login from vistualstudio.com.

Once downloaded, it is a pretty a-typical install, no next, next, next – it just works! The UI takes a bit of poking around to get comfortable with, but after just a few minutes all seemed to be working as expected.

Below you can see an example of some markdown syntax in Code.

code-markdown

The toolbar at the top of the image

toolbar

allows you to change between split screen or single screen and, as I have done above show a preview of what you are writing. This is just a quick hands on, you can see how simple it is to get started. Now that I have found a tool that seems work properly in Windows, my next step is to find a tool for markdown presentations that is also easy to use (in Windows of course:) )

New free software from EMC to build your own SDS solution

**Disclaimer: I am an EMC employee, this post was not sponsored or in any way required by my employer, it is my experience getting to know this particular product.**

There were two software related announcements at EMC World this week which I found very exciting. Building on the free for no production use of RecoverPoint for Virtual Machines from VMworld 2014, EMC announced the same for ScaleIO. ScaleIO allows you build your own Hyperconverged Infrastructure solution (HCI). This is the same software used in the new VxRack from VCE which was also announced at EMC World.

CoprHDIn addition to ScaleIO, EMC also announced CoprHD which is an open source version of EMC ViPR (@coprhd). ViPR (which is also free for non production use) is a solution that allows you to manage multiple arrays and present those as virtual volumes to hosts. In addition to managing the arrays, it also provides a self-service and automation at the storage layer. EMC ViPR also supports ScaleIO, assuming this carries over to CoprHD you could deploy a fully managed, and automated storage solution on commodity hardware for test/dev or QA (I hope they publish more specific guidelines on just what they mean by “non-production”).

Last, but not least, the community version of the VNXe which you can use to provide full block and file servers on commodity hardware. The vVNX will later come in a supported ROBO and cloud edition.

My hope is that CoprHD, ScaleIO, and the community edition of the vVNX will lead to more solutions being open sourced and offered in a free to use model. CoprHD should be available on GitHub by June, ScaleIO by the end of May, whereas the vVNX is available now for download.

 

Yummy! – PowerCLI Cookbook Review by Phillip Sellers (@pbsellers)

The PowerCLI Cookbook by Phillip Sellers is an excellent resource for any skill level, whether you are a beginner or looking for a great reference to have with you.

PowerCLI Cookbook by Phillip Sellers

First and foremost, this book far exceeds what I expect out of a technology cookbook. If you step back and think about a (food) cookbook you get the recipe for what you are going to make (i.e. what you are going to do in PowerCLI) and the ingredients to make it (i.e. the cmdlets necessary to perform the task). Phillip took that a step further and began the cookbook with how to actually start the oven, or in this case a simple recipe to connect to vCenter and get started using PowerCLI.

The chapters in the book are laid out very well, starting with basic hosts related tasks, before moving on to vCenter, virtual machines, and other more complex scenarios – the build up in this format makes it excellent for those who are new to PowerCLI, or even VMware for that matter. Each recipie also has a “how it works” section where the components use are explained (no one has ever told me how food flavors work together!).

You could quite literally use the book to just about stand up a complete vSphere environment as all the major topics such as networking, datastores, clusters, and virtual machine management (including using PowerCLI to invoke in guest scrips) is covered.

**Disclaimer – I have a book published with Packt Publishing and spoke to Phillip before he decided to write the book. This book was provided to me by the author but the review was not read, or approved by Phillip, it is simply my opinion on the book and its contents.**

ViPR SRM Explore Reports and Topology Maps

**Disclaimer: I am an EMC employee, this post was not sponsored or in any way required by my employer, it is my experience getting to know this particular product.**

Up until now I went through a basic ViPR SRM installation, getting a basic single VM environment setup. What I want to show in this post is my favorite ViPR SRM feature – topology maps. To understand why these are useful, lets step back and give some scenarios:

You are the personal responsible for supporting the storage within your environment, you may support other things but ultimately when there is a storage related problem your name is called. An application own comes to you and says their application is slow, and that the network team said everything on their end is fine so its probably the storage. Great – now what?

  1. You come into a new organization – whether as an internal IT person or a var and you’ve inherited an environment cabled by 3 monkeys and a cat with no documentation – now what?

This is where topology maps can be very useful. The topology maps is that end-to-end visualization and monitoring component I mentioned in previous posts. I see from my virtual machine or even some applications such as SQL Server all the way through to the underlying storage, and drill down on each component. Let me shows you some examples.

To access the topology maps, click on Explore >> Hosts – small aside here – host could be any physical or virtual server in the environment discovered by ViPR SRM, not just ESXi hosts. So this could be an ESXi host, a virtual machine, or a physical host running its own OS.

vipr-srm-explore-reports-hosts

From this report, you can see a list of all the hosts in the environment, which for some could be a very extensive list. I should mention that the filter field is not a search field, so you cannot type the end of a machine name; for example maybe all your VM names end in OS type or some other identifier, you couldn’t just type W2K8 to find a server name myserver-w2k8, you would have to start with myserver, but would then see a list of all servers starting with that string. You can filter on any column that has the funnel icon, so for example I could filter on just physical hosts, or virtual machines by clicking the funnel icon in the host type column;

vipr-srm-filter

Using the example above, let’s say an application owner has complained about performance and you need to investigate to see if storage could be the problem. Filter on the host name, in this case I will pick on mhmbd078-W2K8, as you can see below I start typing that name and can select it from a the list or type it in full and hit enter to filter on that one host

vipr-srm-filter-hostname

 

Now I just see that specific host, in this case a virtual machine as you can see here with 16GB of memory and 4 vCPU:

vipr-srm-single-host-explore

This much information is available in just a few clicks, now there are many places you could get this information but as I continue to drill deeper, you will start to see just how much information we have at hand. With just what is available so far, you might be able to say to the application owner who issued the complain that there is not enough memory, for example maybe you know that this particular application needs 32GB of memory, so disk I/O could be a problem if the application and OS are constantly swapping to disk. But, maybe so far everything checks out, if I click on any of the text here, it will take me into the detail of that virtual machine.

Now, this is where it gets interesting; what you see below is the topology map for mbmbd078-w2k8, we can see the host, the datastore it is on, the host it is on, the VSANs it is connected to and the arrays connected to those VSANs. Also, notice to the right we have different reports related to the host, we can see attributes about the host which is show by default, you can also see:

  • Capacity information about the hosts local disks, in this case VMDKs and since it is a virtual machine, the datastore
  • Path details for the disks attached to the host
  • Related storage performance
  • Events related to the host

vipr-srm-topology-map

You can click on any element in the map to see details specific to that item, for example if you click on the datastore – DS_Bootcamp_D you can see reports about the datastore, or on the host – you guessed it, reports about the host. You may have also noticed the + icon next to some of the elements, this is because there are additional components, using VSAN0040 as an example, we can click on the + sign to see switches in that VSAN

vipr-srm-exapanded-element

Now I see two switches, each with their own + icon, I can keep drilling down and see ports on that switch as well. I can expand different elements and hover over different components to see how they are connected. For example I have expanded my host to see my HBAs, I can see that the particular HBA I am interested in is connected to VSAN mptb023 so I have expanded that as well and drilled down to see the switch ports. While I have some limited lab resolution available, you can see here that when I hover over the HBA from the host it highlights the path to the port on the switch – in this case fc1/6 (as shown by the blue highlighted line)

vipr-srm-show-details

This is just one specific report, and I have only skimmed the surface of the data available in this report. Imagine being able to show this to an application owner as you troubleshoot each component, and explain how/why any particular piece of the infrastructure supporting the application is, or isn’t doing what it is supposed to. For those folks who worked in a silo’d type group, I’d urge you not use this information to punt back over your wall to someone else, but rather be the person to start poking some pinholes in the silo, call up a virtualization, OS, or network person depending on what you might think the problem is and work with them, sharing knowledge and help the application owner be a happy customer. After all, even if you are “internal” IT – you are still providing a service to the business – they are you customers, treat them like it. Silos will only fall if someone starts poking holes, no reason it can’t be you.

If you haven’t done so, chat with your EMC rep (they can likey get you in touch with an SE who can help if you have any setup questions) and head over to support.emc.com to sign up for an account and download ViPR SRM which comes with a 30 day license.