Cody Bunch Some Random IT Guy - OpenStack, DevOps, Cloud, Things

Paris Summit - Day 1

Hooray Paris!

Well, maybe. Having only seen the inside of the conference center and hotel lobby of the hotel nextdoor, I didn’t get to do much summiting today. Rather, the day was spent prepping for tomorrows workshop.

I do hear that the #vBrownBags are doing extremely well with 19 individual sessions recorded (and streamed?) today, and a full schedule of 100-ish more this week.

Clone all the USB keys! - Method 3

Method 3 you ask? Wel, sure, I wrote on this around the time of OSCON earlier this year. However, for the OpenStack in Paris, we needed a larger axe. Thanks to Lars & Mr. Cloud Null, we managed to apply the appropriate amount of strategic violence:

cat ~/
set -e -o -v

for i in `jot 19 3`;
    do sudo diskutil erasedisk FAT32 PARIS /dev/disk${i}

for i in `jot 19 3`;
    do sudo diskutil partitionDisk /dev/disk${i} MBR MS-DOS PARIS 0b

for i in `jot 19 3`;
    do rsync --progress -az /Volumes/PARIS "/Volumes/PARIS $i/" &

First it formats all the disks, then it rsyncs all the things.

OpenStack Book Discounts During Summit

Starting now and running the week of the summit, the two best selling OpenStack books will be on super sale:

OpenStack Cookbook

Book : nieX72Mn7U

eBook: k1QxrwyMvD

Learning OpenStack Networking (Neutron)

Book : luyLRpSQ

eBook: IXQ1swn2

Paris Summit Preparation

As you might have guessed from the large number of ZeroVM posts recently, that something was up. If you are going to be at the OpenStack Summit in Paris, there will be a ZeroVM workshop given by my coworkers Egle, Lars, and I.

This session will be a 90-minute, into the deep end, building applications to work with ZeroVM.

The materials, if you want to play along at home, can be found here:

Still missing are the slides and video which I will update the post with after the show. See you in Paris.

Using Packer to Make Vagrant Boxes

Part of working on the 3rd edition of the OpenStack Cookbook required, among other things, a new release of Ubuntu that came pre-loaded with the Juno OpenStack packages. Not a problem! Excepting that there were no ready made images on Vagrant Cloud. Enter Packer.


What the deuce is packer? From the site:

Packer is a tool for creating identical machine images for multiple platforms from a single source configuration.

Said another way, it takes the ‘configuration is code’ and pushes it back to the golden images, to a degree. The idea is one can shrink the time to deploy if instead of starting with the most generic box and adding, you start somewhere in the middle, adding common things to a ‘golden’ image, and then deploying on top of that.

In the use-case for the OpenStack Cookbook? Well, we just needed a manila Ubuntu 14.10 box for Fusion and Virtualbox. As there are three of us on the project now, having a common starting ground is ideal.

Our Packer Setup

I’ll leave installing Packer as an exercise for the reader. The docs are pretty good in the regard.

For our build, we use the following packer.json:

  "builders": [
    {"type": "virtualbox-iso",
    "guest_os_type": "Ubuntu_64",
    "iso_url": "",
    "iso_checksum": "91bd1cfba65417bfa04567e4f64b5c55",
    "http_directory": "preseed",
    "ssh_username": "vagrant",
    "ssh_password": "vagrant",
    "output_directory": "ubuntu64_basebox_virtualbox",
    "shutdown_command": "echo 'shutdown -P now' >; echo 'vagrant'|sudo -S sh ''",
    "boot_command": [
      "/install/vmlinuz noapic ",
      "ks=http://:/preseed.cfg ",
      "debian-installer=en_US auto locale=en_US kbd-chooser/method=us ",
      "hostname= ",
      "fb=false debconf/frontend=noninteractive ",
      "keyboard-configuration/modelcode=SKIP keyboard-configuration/layout=USA ",
      "keyboard-configuration/variant=USA console-setup/ask_detect=false ",
      "initrd=/install/initrd.gz -- ",
    {"type": "vmware-iso",
    "guest_os_type": "ubuntu-64",
    "iso_url": "",
    "iso_checksum": "91bd1cfba65417bfa04567e4f64b5c55",
    "http_directory": "preseed",
    "ssh_username": "vagrant",
    "ssh_password": "vagrant",
    "output_directory": "ubuntu64_basebox_vmware",
    "shutdown_command": "echo 'shutdown -P now' >; echo 'vagrant'|sudo -S sh ''",
    "boot_command": [
      "/install/vmlinuz noapic ",
      "ks=http://:/preseed.cfg ",
      "debian-installer=en_US auto locale=en_US kbd-chooser/method=us ",
      "hostname= ",
      "fb=false debconf/frontend=noninteractive ",
      "keyboard-configuration/modelcode=SKIP keyboard-configuration/layout=USA ",
      "keyboard-configuration/variant=USA console-setup/ask_detect=false ",
      "initrd=/install/initrd.gz -- ",
  "provisioners": [{
    "type": "shell",
    "execute_command": "echo 'vagrant' | sudo -S sh ''",
    "inline": [
      "apt-get update -y",
      "apt-get install -y linux-headers-$(uname -r) build-essential dkms",
      "apt-get clean",
      "mount -o loop VBoxGuestAdditions.iso /media/cdrom",
      "sh /media/cdrom/",
      "umount /media/cdrom",
      "mkdir /home/vagrant/.ssh",
      "mkdir /root/.ssh",
      "wget -qO- >> ~/.ssh/authorized_keys",
      "echo 'vagrant ALL=NOPASSWD:ALL' > /tmp/vagrant",
      "chmod 0440 /tmp/vagrant",
      "mv /tmp/vagrant /etc/sudoers.d/"
  "post-processors": [
      "type": "vagrant",
      "override": {
        "virtualbox": {
          "output": ""
        "vmware": {
          "output": ""

This in turn calls a kickstart (yes kickstart on Ubuntu) file stored in the preseed folder. This looks like:



lang en_US.UTF-8
keyboard us

network --device eth0 --bootproto dhcp

timezone --utc America/Chicago

clearpart --all --initlabel
bootloader --location=mbr

part /boot --fstype=ext3 --size=256 --asprimary
part pv.01 --size=1024 --grow --asprimary
volgroup vg_root pv.01
logvol swap --fstype swap --name=lv_swap --vgname=vg_root --size=1024
logvol / --fstype=ext4 --name=lv_root --vgname=vg_root --size=1024 --grow

auth --enableshadow --enablemd5

# rootpw is vagrant
rootpw --iscrypted $1$dUDXSoA9$/bEOTiK9rmsVgccsYir8W0
user --disabled

firewall --disabled




apt-get update

apt-get upgrade -y linux-generic


useradd -m -s /bin/bash vagrant
echo vagrant:vagrant | chpasswd

mkdir -m 0700 -p /home/vagrant/.ssh

curl >> /home/vagrant/.ssh/authorized_keys

chmod 600 /home/vagrant/.ssh/authorized_keys
chown -R vagrant:vagrant /home/vagrant/.ssh

echo "vagrant ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers

sed -i 's/^PermitRootLogin without-password/PermitRootLogin yes/' /etc/ssh/sshd_config

apt-get clean

rm -rf /tmp/*

rm -f /var/log/wtmp /var/log/btmp

history -c

Once both files are in place and packer is installed, building the images is as simple as:

packer build packer.json

This will take a long time, but it does eventually get there, and will produce two .box files in the output folders you specified in the packer.json. From there, you can upload them somewhere and have vagrantcloud make them available, as we have under bunchc/utopic-x64.


In this post we looked at how to use to build Vagrant boxes for both VMware Workstation / Fusion and Virtualbox.

Changing Jenkins Home Folder

Everything Jenkins does happens in the context of the jenkins user by default. While that sounds obvious, it has some interesting implications if you aren’t careful.

The Problem

In my homelab, I’ve installed 14.04 onto a 16gb usb key. This is a hold over from the days when it was an ESXi lab, but, it works for the most part. However, for things like downloading vagrant boxes and the like, one needs to mount external storage. Enter the problem: Jenkins sets itself up in /var/lib/jenkins by default, which lives on my very limited usb stick.

This in turn means anytime Jenkins tried to ‘vagrant up’ a thing, it would fill my root partition trying to store all the temporary files vagrant used. Whoops.

Changing the Jenkins Home Directory

There are a number of posts out there on how to do this. None of them, however, addressed the vagrant issue in particular. To do that, one has to hit it with a larger bat.

First Change /etc/defaults

The first thing to do, is change how Jenkins thinks of itself. The most straight forward way I found was to change that in /etc/defaults/jenkins. This will be around line 23 or 25:

# jenkins home location

Then Change /etc/passwd

This is the step that was needed to fix the problem of Jenkins running vagrant and vagrant subsequently filling /:

Change: jenkins:x:105:114:Jenkins,,,:/var/lib/jenkins:/bin/bash

To this: jenkins:x:105:114:Jenkins,,,:/new/location:/bin/bash

Finally Restart Jenkins

Now to make sure all the things stick:

sudo /etc/init.d/jenkins restart


In this post we showed you how to change the Jenkins home directory, and why it’s also important to change it in /etc/passwd.

ZeroVM - A ZeroCloud Lab

At this point we’re pretty far down the ZeroVM rabbit hole. We’ve covered quite a bit of ground to get to this stage as well. Specific to this post, you will want to know more about ZeroCloud. You can do that here.

Regarding getting up to speed on ZeroVM as I am, you will find the following useful:

Building The ZeroCloud Lab

There are a few ways of going about this, however before we get started, realize that this is still early days and if you find this post a few months or a year or two down the road, it will likely be super out of date.

To get working with ZeroCloud, the most straight forward path is to follow the directions in the ZeroVM Docs here.

Once you’ve finished, you should have a functional ZeroCloud lab.

Experimenting in the ZeroCloud Lab

Having a lab up in running is one thing, but it’s rather boring thing by itself. So, let’s do something with it!

ZeroCloud Hello World Example

The first thing we’ll try to do, is a generic “Hello World” example, taken from here.

1. Log into the lab:

vagrant ssh

cd devstack/
source /vagrant/adminrc

zpm auth
export OS_AUTH_TOKEN=PKIZ_Zrz_Qa5NJm44FWeF7Wp...

Note: You will need to change the AUTH_TOKEN and STORAGE_URL values to match those returned by zpm auth.

2. Create our “Hello World” python file:

cat > example <<EOF
import sys
print("Hello from ZeroVM!")
print("sys.platform is '%s'" % sys.platform)

3. Now we run the example:

curl -i -X POST -H "X-Auth-Token: $OS_AUTH_TOKEN" \
  -H "X-Zerovm-Execute: 1.0" -H "Content-Type: application/python" \
  --data-binary @example $OS_STORAGE_URL

Ok, so what did we do then? In the first step we logged in to our lab, sourced a file that contains credentials for our environment, and then set some additional environment variables to store our auth token and swift url.

Next we created the hello-world example file. In the absence of a text editor we just dumped everything between “«EOF” and “EOF” into the file ‘example’. Note the first line in the file begins with #!file://. This is important as it lets the parser know what to do with said file, in this case, fire up python.

Finally, we use curl against our Swift install, letting it know to jump out to the ZeroVM middleware and that our application type is python. Additionally we specify our entire program as the payload in --data-binary


In this post, we showed you how to build a ZeroCloud lab and run an example “Hello World” application within it. If you would like to explore some more complex applications, be sure to explore the various talks and videos here.

ZeroVM - Getting Started with ZeroCloud

If you are getting up to speed on ZeroVM as I am, you will find the following useful:

Additionally, this post does a great job of explaining how ZeroVM fits when you start to approach middling to large (Read: Big Data) problems. Go read it now.

Getting Started with ZeroCloud

Now that you’ve read the introduction by Lars, and I trust that you have, we can start looking into the ‘what’ of ZeroCloud. In this post we’ll touch on it’s architecture and use case or two. In a subsequent post we’ll show you how to build your own small-scale lab for such an environment.

Note: Also go here, watch this, around the 16:30 mark, pay attention to the ‘t = data / Rate’ bits.

Note: This is where things in ZeroVM start to get really cool.

ZeroCloud Architecture

The best way to describe the what and how of ZeroCloud is with a few diagrams which we will then dive into. These are borrowed form slides in past presentations.

Highlevel ZeroCloud Architecture

This first diagram is a hugely simplified view of a swift environment, that is, if you have worked with OpenStack Swift it should look familiar enough. The “Proxy” nodes at the top are Swift Proxies, the Storage Nodes below are well, object storage nodes.

What changes for ZeroCloud is the addition of a few ZVM parts made available via Swift Middleware. In this case ZVM represents job brokers, some messaging, ZeroVM itself, as well as an ‘executor’ that handles the connecting of channels and the reading / writing of objects in Swift. This next diagram breaks that down further:

ZeroCloud Dataflow Architecture

In this diagram, which admittedly is an eyechart, each step in the ZeroVM Swift Middleware is shown. Working left to right:

1. The user initiates a job request via the Swift REST API by sending a post request with the code to be executed.

2. This hits the Swift Proxy node and is handed to the ZeroVM middleware, which, via a job scheduler finds where the closest replica of your data is, and sends the request along.

3. The job arrives at the executor who then fires up a net-new ZeroVM session for the job, opens the required IO Channels to your file* and if needed other ZeroVM sessions.

4. The ZeroVM job, finished with it’s work, sends the response back up to the Proxy Server who then serves the response back to the user.

Note: re: file in #3, in the default configuration this is a net new copy of the file rather than the actual object itself. This prevents accidental damage to the file.

ZeroCloud Use Cases

So what do you do with a deterministic, computable, distributed data store? Some things that jump to mind are things like “A Giant MapReduce Cluster”, similar to the one described here. As Lars describes, ZeroCloud can address one of the biggest MapReduce computational issues, that is data locality.

In addition to MapReduce type workloads, what else can one do? One could couple ElasticSeach and Log Stash, and indeed serve Kibana directly from Zero Cloud. Video trans-coding is another use case that I’ve seen.

ZeroVM/ZeroCloud then, works well when you have a data heavy workload that would benefit being attacked in a distributed fashion.


In this post we introduced you to ZeroCloud, the amalgamation of ZeroVM and OpenStack Swift. We dove into the architecture of ZeroCloud and introduced some use cases where having programmable distributed storage system is insanely useful.

ZeroVM - IO Operations in ZeroVM

If you are just joining these posts, you might want to take a few minutes and review these other posts on ZeroVM:

IO Operations in ZeroVM

IO in ZeroVM is an odd sort of beast. As you are not working in a container or a VM you do not inherit the unlimited read/write to arbirtrary places as you would have otherwise. Instead the abstraction unit for ZeroVM is the channel.


As the intro said, a channel is the unit of IO abstraction within ZeroVM. That means any network communication, any file write, any input from a pipe, or output to swift, each is an individual channel.

Additionally, channels have to be explicitly defined prior to launching a ZeroVM instance. In some ways, this is like specifying the VMDK for a traditional VM to boot from. The requirement to specify all channels before hand does provide a powerful security mechanism in that all IO is effectively sandboxed. An erroneous process can not cause damage outside of the narrow constraints defined within said channel.

Channels in ZeroVM are unique in other ways. Each channel, regardless of type, is presented to the instance as a file handle.

Channel Quotas

To keep with the security and isolation provided within ZeroVM, along with the explicit channel definition, one must also specify a quota for IO. You specify said quota with four bits of information: (1) Total reads; (2) Total writes; (3) Total Read Size; and (4) Total Write Size.

Quotas are handled on a ‘first one hit’ basis. So, if you have 10,000 reads to do, but said number of reads exceeds the total read size in bytes at read number 3,500, well, that’s it. No more reads for you. The inverse of this is also true, when you run out of the number of allowed reads or writes before you hit the size quota, you will also get a failure returned.

When you do hit this limit, however, ZeroVM tries to fail gracefully, in that you will still have all the data read or written up to that point. Below is an example of what happens when you exceed the number of available reads:

$ dd if=/dev/urandom of=/home/vagrant/file.txt bs=1048576 count=100

$ cat > /home/vagrant/ <<EOF
file = open('/dev/3.file.txt', 'r')

$ su - vagrant -c "zvsh --zvm-save-dir ~/ --zvm-image python.tar --zvm-image ~/file.txt python"

$ sudo sed -i "s|Channel = /home/vagrant/file.txt,/dev/3.file.txt,3,0,4294967296,4294967296,0,0|Channel = /home/vagrant/file.txt,/dev/3.file.txt,3,0,3,3,3,3|g" /home/vagrant/manifest.1
$ sudo sed -i "s|/home/vagrant/stdout.1|/dev/stdout|g" /home/vagrant/manifest.1
$ sudo sed -i "s|/home/vagrant/stderr.1|/dev/stderr|g" /home/vagrant/manifest.1

$ zerovm -t 2 manifest.1

Traceback (most recent call last):
  File "/dev/", line 2, in <module>
IOError: [Errno -122] Unknown error 4294967174

What is happening in that example is this:

1. Create a 100MB file full of random bits

2. Create ‘’ to read the first 5 characters in the file

3. Use zvsh to run inside ZeroVM… this should dump some garbage characters to the console.

4. Change the quota from ‘huge’ to 3

5. Adjust the manifest file to allow stderr and stdout to work properly

6. Run ZeroVM with the updated manifest & receive IO error

Specifying Channels in a Manifest File

So that example took me way too long to figure out. Mostly due to a few small things and one big one: Creating the manifest file properly. What follows is a quick way to create a manifest file as well as our example manifest and a breakdown thereof.

Creating a Manifest File

It turns out that the zvsh utility that is bundled with ZeroVM has the ability to save the manifest (and other runtime files) when you launch it. To generate the template manifest that is used and modified in this example, run the following command:

zvsh --zvm-save-dir ~/ --zvm-image python.tar --zvm-image ~/file.txt python

Breakdown of the Manifest File

The example file that follows was for the specific bits used in our example. It can, however, make for a good starting point for your own manifest files.

Node = 1
Version = 20130611
Timeout = 50
Memory = 4294967296,0
Program = /home/vagrant/boot.1
Channel = /dev/stdin,/dev/stdin,0,0,4294967296,4294967296,0,0
Channel = /dev/stdout,/dev/stdout,0,0,0,0,4294967296,4294967296
Channel = /dev/stderr,/dev/stderr,0,0,0,0,4294967296,4294967296
Channel = /home/vagrant/,/dev/,3,0,4294967296,4294967296,0,0
Channel = /home/vagrant/python.tar,/dev/2.python.tar,3,0,4294967296,4294967296,0,0
Channel = /home/vagrant/file.txt,/dev/3.file.txt,3,0,3,3,3,3
Channel = /home/vagrant/boot.1,/dev/self,3,0,4294967296,4294967296,0,0
Channel = /home/vagrant/nvram.1,/dev/nvram,3,0,4294967296,4294967296,4294967296,4294967296

Taken line by line:

1. Node an optional parameter specifying the node number.

2. Version mandatory. As manifest versions are incompatible. This tells both ZeroVM and you which version of the manifest you are using.

3. Timeout in seconds, is a mandatory parameter. This prevents long running ZeroVM jobs from consuming excess resources.

4. Memory Also mandatory, This is a comma separated value, where the first position is the 32-bit value representing memory. In our case, the max 4GB. The second number can be either 0 or 1 specifying the eTag used to checksum all data passing through ZeroVM

5. Program Specifies the ZeroVM cross compiled binary that we will run

6. Channel The channel definitions, these follow the format:

Channel = <uri>,<alias>,<type>,<etag>,<gets>,<get_size>,<puts>,<put_size>


In this post we covered how IO is handled in ZeroVM. We first explained the Channel concept, how quotas are handled within channels, and finally, how to add a channel or channels at ZeroVM runtime using a manifest.

ZeroVM Link Dump

Instead of curating my own links this time, my friend and colleague Carina C. Zona, ZeroVM’s community manager has gathered these as they relate to ZeroVM:

Posts by Lars

“ZeroVM Architecture and ZVM Runtime (ZRT)”

Ryan McKinney at University of Texas San Antonio Cloud & Big Data Laboratory [2014-08-05]

“Changing the World with ZeroVM and Swift”

Jakub Krajcovic at PyConAU OpenStack Miniconf [2014-08-01]

“Deep Dive into ZeroVM”

Van Lindberg at OSCON Expo [2014-07-23]

  • (materials unavailable)

“What Is ZeroVM?”

Carina C. Zona at OSCON Expo [2014-07-22]

“ZeroVM: Virtualization for the Cloud”

Lars Butler at EuroPython [2014-07-21]

“Introduction to ZeroVM”

Muharem Jrnjadovic at Open Cloud Day [2014-07]

“NoSQL - Computable Object Store with OpenStack Swift and ZeroVM”

(aka “Programmable Object Store with OpenStack Swift and ZeroVM”) Adrian Otto at BigDataCamp [2014-06-14]

“Using ZeroVM and Swift to Build a Compute Enabled Storage Platform”

Blake Yeager & Camuel Gilydov at OpenStack Summit [2014-05-14]

“Using ZeroVM and Swift to Build a Compute Enabled Storage Platform”

Blake Yeager at Open BigCloud Symposium [2014-05-08]

  • (materials unavailable)

“ZeroVM Background”

Prosunjit Biswas at University of Texas at San Antionio [UTSA] Institute of Cyber Security [2014-04-23]

“Process Virtualization with ZeroVM”

Jakub Krajcovi at [2014-02-26]

##”ZeroVM Zwift: OpenStack Platform” Camuel Gilydov and Constantine Peresypkin at OpenStack Summit [2013-04]

“Big Data on OpenStack”

Camuel Gilydov at OpenStack Isreal [2012-06]

“Containers or VMs: Which Virtualization Technology Works for You?”

Panel discussion with ZeroVM, Docker, and VMWare at while42 meetup [2014-03-04]