Cody Bunch Some Random IT Guy - OpenStack, DevOps, Cloud, Things

Currently Reading - Sept 2014

I’ve found overtime my reading habbits have changed from mostly tech books to a fair mix of things, with some things that suprised even me… business books. o.O? Here’s what’s currently on my list and in progress, first the boring ones:

Then on the more fun side:

MariaDB With Galera on Vagrant

Found myself in some training this last week using MariaDB, and being that I like to get a bit more hands on than most, using the class provided lab environment wasn’t going to cut it. This meant wrapping some scripting into a Vagrant environment so I could reliably reporduce the three node lab.

Getting Started

You’ll need Vagrant and Git. It’s also preferred that you have vagrant-cachier installed. You should have vagrant-cachier anyways, but alas, that is not for right now. Once you have these out of the way, do the following:

  1. git clone https://github.com/bunchc/mariadb-galera-vagrant.git
  2. cd mariadb-galera-vagrant
  3. vagrant up

Validating the Cluster

After a few minutes, you should be able to log into any of the nodes. Specifically, node-01 will be used to ‘bootstrap’ the cluster, the other two nodes will join from there.

To validate the cluster:

vagrant ssh mariadb-02
sudo su -
mysql -uroot
SHOW GLOBAL STATUS LIKE 'wsrep%';

You should be able to do the above from any of the nodes in the cluster.

Raspberry Pi as PXE Server

As I start to move from OpenStack Compute Cells in Devstack to OpenStack Compute Cells physicall, I needed to re-think my homelab some. It had been running some variation of vsphere and the OpenStack Cookbook work from various projects prior. Basically, it was a Hodor of a home lab.

Hodor!

Rather than lose a bit of hardware to foreman, or one of the new razor forks (I may still go this route later), I decided instead to pound a Raspberry Pi into service. To turn the rPI into a usable provisioning server, I did the following:

  1. Provision a 16gb card with Raspbian
  2. Beat Networking Into Submission
  3. Setup IP Tables for Nat
  4. Configure DHCP, PXE, TFTP, etc

We’ll skip step 1 as the folks at raspberrypi.org cover that pretty well already.

Beat Networking Into Submission

So the rPI does some auto-hotplugging bits that can cause you some issues if you try to use both wifi and ethernet at the same time. If you’re not expecting them, well… let’s just say I spent too long trying to solve before googling the problem. Here’s what my networking config files look like:

# cat /etc/network/interfaces
auto lo
iface lo inet loopback

auth eth0
allow-hotplug eth0
iface eth0 inet static
    address 172.16.0.1
    netmask 255.255.255.0

auto wlan0
allow-hotplug wlan0
iface wlan0 inet static
wpa-conf /etc/wpa_supplicant/wpa_supplicant.conf
    address 10.0.1.15
    netmask 255.255.255.0
    broadcast 10.0.1.255
    gateway 10.0.1.1

iface default inet dhcp

# cat /etc/default/ifplugd
INTERFACES="eth0"
HOTPLUG_INTERFACES="eth0"
ARGS="-q -f -u0 -d10 -w -I"
SUSPEND_ACTION="stop"

Once you have that, reboot the rPI and you should be good to go.

Setup IP Tables for NAT

This is also pretty well straight forward, but for some reason I end up googling it each time. Here it is for reference:

# NAT
iptables --table nat --append POSTROUTING --out-interface wlan0 -j MASQUERADE
iptables --append FORWARD --in-interface eth0 -j ACCEPT
iptables-save | sudo tee /etc/iptables.conf
iptables-restore < /etc/iptables.conf
sh -c "echo 1 > /proc/sys/net/ipv4/ip_forward"
sed -i "s/exit 0/iptables-restore < \/etc\/\iptables.conf \nexit 0/g" /etc/rc.local
sed -i "s/#net.ipv4.ip_forward=1/net.ipv4.ip_forward=1/g" /etc/sysctl.conf

Configure DHCP, PXE, TFTP, etc

This one is a bit more involved, but is all done via the following bash commands:

MY_IP=$(ifconfig eth0 | awk '/inet addr/ {split ($2,A,":"); print A[2]}')

# Install the things
sudo apt-get install -y dnsmasq nfs-kernel-server syslinux-common

# Setup some shell folders
sudo mkdir -p /tftpboot/images/ubuntu/14.04/amd64
sudo cp -r /usr/lib/syslinux/* /tftpboot/
sudo mkdir -p /tftpboot/pxelinux.cfg/Ubuntu
sudo cp /usr/lib/syslinux/vesamenu.c32 /tftpboot/

# Create a pxe.conf file
sudo cat > /tftpboot/pxelinux.cfg/pxe.conf <<EOF
MENU TITLE  PXE Server 
NOESCAPE 1
ALLOWOPTIONS 1
PROMPT 0
MENU WIDTH 80
MENU ROWS 14
MENU TABMSGROW 24
MENU MARGIN 10
MENU COLOR border               30;44      #ffffffff #00000000 std
EOF

# Create our PXE Menu
sudo cat > /tftpboot/pxelinux.cfg/default <<EOF
DEFAULT vesamenu.c32 
TIMEOUT 600
ONTIMEOUT BootLocal
PROMPT 0
MENU INCLUDE pxelinux.cfg/pxe.conf
NOESCAPE 1
LABEL BootLocal
        localboot 0
        TEXT HELP
        Boot to local hard disk
        ENDTEXT
MENU BEGIN Ubuntu
MENU TITLE Ubuntu 
        LABEL Previous
        MENU LABEL Previous Menu
        TEXT HELP
        Return to previous menu
        ENDTEXT
        MENU EXIT
        MENU SEPARATOR
        MENU INCLUDE Ubuntu/Ubuntu.menu
MENU END
EOF

sudo cat > /tftpboot/pxelinux.cfg/Ubuntu/Ubuntu.menu <<EOF
LABEL 2
        MENU LABEL Ubuntu 14.04 (64-bit)
        kernel tftp://$MY_IP/images/ubuntu/14.04/amd64/install/netboot/ubuntu-installer/amd64/linux
        append auto=true priority=critical vga=788 initrd=tftp://$MY_IP/images/ubuntu/14.04/amd64/install/netboot/ubuntu-installer/amd64/initrd.gz locale=en_US.UTF-8 kbd-chooser/method=us netcfg/choose_interface=auto url=tftp://172.16.11.250/preseed.cfg
        TEXT HELP
        Boot the Ubuntu 14.04 64-bit DVD
        ENDTEXT
EOF

# Configure dnsmasq for tftp & dhcp
sudo cat >> /etc/dnsmasq.conf <<EOF
server=$MY_IP@eth0
interface=eth0
no-dhcp-interface=wlan0
dhcp-range=172.16.0.10,172.16.0.253,12h
dhcp-boot=pxelinux.0
pxe-service=x86PC,"Booting from Network...",pxelinux
enable-tftp
tftp-root=/tftpboot
dhcp-boot=pxelinux.0,servername,$MY_IP
EOF
sudo service dnsmasq restart

# Get 14.04 and extract the needful
wget -O ~/ubuntu-14.04-server-amd64.iso http://mirror.anl.gov/pub/ubuntu-iso/CDs/trusty/ubuntu-14.04.1-server-amd64.iso

sudo mkdir /mnt/loop
sudo mount -o loop -t iso9660 ~/ubuntu-14.04-server-amd64.iso /mnt/loop
sudo cp -R /mnt/loop/* /tftpboot/images/ubuntu/14.04/amd64
sudo cp -R /mnt/loop/.disk /tftpboot/images/ubuntu/14.04/amd64
sudo umount /mnt/loop

# Get a generic preseed
wget -O /tftpboot/preseed.cfg https://help.ubuntu.com/lts/installation-guide/example-preseed.txt

# Setup NFS mounts
echo "
/tftpboot/images/ubuntu/    *(ro,sync,no_subtree_check)" | sudo tee -a /etc/exports

# Enable RPCBind, NFS, and restart them
update-rc.d rpcbind enable && update-rc.d nfs-common enable
service rpcbind start
service nfs-kernel-server restart

The commands themselves are commented pretty well. Generically what they do is install dnsmasq for dhcp & tftp. From there we download and extract all the things from the ISO that we’ll need, configure some menus, download a basic preseed, and restart some services.

Summary

In this post, we took a lowly Raspberry Pi and indentured it into some network servitude as a pxe / tftp server. We did this by installing and configuring dnsmasq for tftp and dhcp. Additionally we set up some fancy pxeboot menus and configured them to boot locally as a priority and to the network in times of need. Finally, we pulled down an Ubuntu 14.04 image and generic preseed file to use for automatic installs.

Resources

Vagrant and Hanlon (The other Razor Fork)

You may or may not know that Razor, the bare-metal lifecycle tool, has forked from its original bits. This is a good thing, but it’s still really early days on the new forks, so we’ll see ultimately what plays out.

The Puppet-Labs “Razor-Server” fork can be found here. If you want to work with it, or test it locally, you can use the bits written by Egle here.

Hanlon, the CSC fork here, is being perused by Tom McSweeney who helped get the original razor off the ground.

To help me work with it and contribute some, I’ve built a small Vagrant environment (largely basd on the Vagrant environment for ‘the other’ razor.).

Vagrant Up Hanlon

To get started, clone the repo and vagrant up, like so:

git clone https://github.com/bunchc/vagrant-hanlon.git
cd vagrant-hanlon
vagrant up

What this does during the vagrant up process is:

  • Install dnsmasq
  • Configure IPtables for NAT
  • Installs Mongodb
  • Installs Java
  • Installs RBENV
  • Installs both jruby and ruby 1.9.3
  • Downloads Hanlon
  • Starts Puma (to run Hanlon)
  • Pulls down a few images to add to Hanlon

Once you have Hanlon running, command wise it works very similar to the old Razor. That is:

hanlon node
hanlon policy
hanlon image

Summary

In this post, you cloned and started to work with Hanlon, one of the derivatives of the Razor bare-metal provisioning framework.

Updated Blog / Markdown workflow

A while ago I posted about my markdown workflow. While that workflow was decent and worked for a goodly little while, it left quite a bit to be desired. Specifically, I the posting process into Wordpress was cumbersome at best.

A New Blog Engine

Like I said above, working with Wordpress and markdown was cumbersome and then some. So it was time for something new. At first I was going to use similar to this, which is actually what is in use on the openstackcookbook.com site. It has the following benefits:

  • Posts are written in Markdown
  • Integrated with gh-pages
  • Straighforward publishing process git push origin master

It was almost what I needed. In the end I went with Hugo. What Hugo added was some pluggable themes and templates. Also the ability to run locally before pushing to github.

To setup Hugo on github, use this.

Current Workflow

My current workflow needed some help then. I kept the same sublime text plugins, that is:

  • MarkdownEditing
    • This has a number of really handy keyboard bindings. It also has some decent highlighting and what not.
  • Markdown Preview
    • This one allows me to go from Markdown into what it’ll look like on the web.
  • Markdown TOC
    • This lets me go from a basic set of sections and files into a more full fledged table of contents.

Additionally, to create a new post, I modified the Rakefile found here, to look like this:

require 'fileutils'
task :post do
    title = ENV['title'] || "new-post"
    tags = ENV['tags'] || ''
    make_img_dir = ENV['imgdir'] || false
    date = (ENV['date'] ? Time.parse(ENV['date']) : Time.now).strftime('%Y-%m-%d')
    filename = File.join('.', "#{Time.now.strftime('%Y-%m-%d')}-#{title.strip.gsub(' ', '-').gsub(/[^\w-]/, '')}.md")
    open(filename, 'w') do |post|
        post.puts "---"
        post.puts "title: \"#{title}\""
        post.puts "date: #{Time.now.strftime('%Y-%m-%d')}"
        post.puts "categories: "
        post.puts "---"
        post.puts "\nYour content here."
        if make_img_dir
            img_dir = File.basename(filename.chomp(File.extname(filename)))
            FileUtils.mkdir_p("../images/posts/#{img_dir}")
            post.puts "\n" * 5
            post.puts "[imgdir]: /images/posts/#{img_dir}/"
        end
    end
end

So creating a new post goes like this: bunchc: blog/content/posts$ rake post title=”Title Here”

In turn, that creates a YYYY-MM-DD-Title-Here.md file for editing in sublime. It also adds the metadata section at the top of the file for me:

---
title: "Updated Blog / Markdown workflow"
date: 2014-08-14
categories: 
---

From there, I write the file, save the file, and run ./deploy.sh from the Hugo installer linked earlier. That handles all the pushing and bits to git.

Summary

The gist of it is, Wordpress was a bit much and a bit heavy for what I needed. Hugo, Markdown, and GitHub Pages gave me a streamlined process that looks decent for posting.

Multi-Node Devstack with Neutron and Cells

OpenStack Cells allow you to break up Nova into smaller domains. In turn, this allows for a number of interesting things. Not the least of which is scale.

From the OpenStack.org docs: Note I’ve added the bold on some important points.

Cells functionality enables you to scale an OpenStack Compute cloud in a more distributed fashion without having to use complicated technologies like database and message queue clustering. It supports very large deployments.

When this functionality is enabled, the hosts in an OpenStack Compute cloud are partitioned into groups called cells. Cells are configured as a tree. The top-level cell should have a host that runs a nova-api service, but no nova-compute services. Each child cell should run all of the typical nova-* services in a regular Compute cloud except for nova-api. You can think of cells as a normal Compute deployment in that each cell has its own database server and message queue broker.

So, in addition to breaking down at physical boundaries or failure domains, you can break nova-compute into smaller chunks based on DB and MQ scaling limits.

However, not all of us have multi-geo hundred plus node compute labs to play with… so how do we test out this functionality before hand? Devstack!

Getting Started

tl;dr - git clone https://github.com/bunchc/devstack-cells.git; cd devstack-cells; vagrant up, and go to “Configuring and Creating Cells”

To get started, you’ll need the following:

  • virtualbox (fusion or workstation work just as well)
  • a minimum of two Ubuntu 14.04 VM with:
    • About 2GB ram
    • 2x networks
      • eth0 = NAT
      • eth1 = host only

Both Nodes

Once you have that taken care of, on each node we need to create a stack user:

sudo adduser --disabled-password --gecos "" stack
echo "stack ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers

Parent Node

Time to Download devstack:

sudo su - stack
git clone -b stable/icehouse https://github.com/openstack-dev/devstack 

Note the sudo command. Everything from this point is done as stack

Next, let’s make the local.conf file:

echo "
[[local|localrc]]
ADMIN_PASSWORD=$ADMIN_PASSWORD
DATABASE_PASSWORD=$ADMIN_PASSWORD
RABBIT_PASSWORD=$ADMIN_PASSWORD
SERVICE_PASSWORD=$ADMIN_PASSWORD
SERVICE_TOKEN=a682f596-76f3-11e3-b3b2-e716f9080d50

GIT_BASE=${GIT_BASE:-https://git.openstack.org}

# Cells!
ENABLED_SERVICES+=,n-cell,n-api-meta
DISABLED_SERVICE+=,n-cpu,n-net,n-sch

# Neutron - Networking Service
# If Neutron is not declared the old good nova-network will be used
ENABLED_SERVICES+=,q-svc,q-agt,q-dhcp,q-l3,q-meta,neutron,q-lbaas,q-vpn,q-fwaas

# Neutron Stuff
OVS_VLAN_RANGES=RegionOne:1:4000
OVS_ENABLE_TUNNELING=False

## Images
# 32bit image (~660MB)
IMAGE_URLS+=",http://fedorapeople.org/groups/heat/prebuilt-jeos-images/F19-i386-cfntools.qcow2"
# 64bit image (~640MB)
IMAGE_URLS+=",http://fedorapeople.org/groups/heat/prebuilt-jeos-images/F19-x86_64-cfntools.qcow2"
IMAGE_URLS+=",http://mirror.chpc.utah.edu/pub/fedora/linux/releases/20/Images/x86_64/Fedora-x86_64-20-20131211.1-sda.qcow2"
IMAGE_URLS+=",http://launchpad.net/cirros/trunk/0.3.0/+download/cirros-0.3.0-x86_64-uec.tar.gz"

# Output
LOGFILE=/opt/stack/logs/stack.sh.log
VERBOSE=True
LOG_COLOR=False
SCREEN_LOGDIR=/opt/stack/logs
" | tee -a /home/stack/devstack/local.conf

The important bit for cells in this file are:

ENABLED_SERVICES+=,n-cell,n-api-meta
DISABLED_SERVICE+=,n-cpu,n-net,n-sch

This enables the cell service, and the nova-api metadata service. It then disables compute, nova-networking, and the scheduler. Those tasks will be handled on the children.

With that out of the way run ./stack.sh and grab a coffee. When it completes, add the following to /etc/nova/nova.conf:

[cells]
enable=True
name=api
cell_type=api

Finally, restart the n-api serivice.

Child Node

Time to Download devstack:

sudo su - stack
git clone -b stable/icehouse https://github.com/openstack-dev/devstack 

Note the sudo command. Everything from this point is done as stack

Next, on the child node, we create a local.conf file

cd devstack
echo "
[[local|localrc]]
ADMIN_PASSWORD=$ADMIN_PASSWORD
DATABASE_PASSWORD=$ADMIN_PASSWORD
RABBIT_PASSWORD=$ADMIN_PASSWORD
SERVICE_PASSWORD=$ADMIN_PASSWORD
SERVICE_TOKEN=a682f596-76f3-11e3-b3b2-e716f9080d50

HOST_IP=<ipaddres of eth0>
SERVICE_HOST=<ip address of controllers eth0>
MYSQL_HOST=$SERVICE_HOST
RABBIT_HOST=$SERVICE_HOST
Q_HOST=$SERVICE_HOST
GLANCE_HOSTPORT=$SERVICE_HOST:9292

GIT_BASE=${GIT_BASE:-https://git.openstack.org}

# Cells!
ENABLED_SERVICES+=,n-cell
DISABLED_SERVICE+=,n-api,key,g-api

# Neutron - Networking Service
# If Neutron is not declared the old good nova-network will be used
ENABLED_SERVICES+=,q-svc,q-agt,q-dhcp,q-l3,q-meta,neutron,q-lbaas,q-vpn,q-fwaas

# Neutron Stuff
OVS_VLAN_RANGES=RegionOne:1:4000
OVS_ENABLE_TUNNELING=False

# Output
LOGFILE=/opt/stack/logs/stack.sh.log
VERBOSE=True
LOG_COLOR=False
SCREEN_LOGDIR=/opt/stack/logs
" | tee -a /home/stack/devstack/local.conf

The important bit for cells is:

ENABLED_SERVICES+=,n-cell
DISABLED_SERVICE+=,n-api,key,g-api

This enables the cell service and turns of keystone along with nova-api and glance-api services. Those in turn will be handled by the parent.

Next, run ./stack.sh and grab a coffee. When it completes, add the following to /etc/nova/nova.conf

[cells]
enable=True
name=cell1
cell_type=comput

Configuring and Creating Cells

Ok, what happened above was we got two nodes up and running and ready to go with Devstack and the prerequsite services for cells. Now we actually have to actuall make the cells. To do that:

Parent

nova-manage cell create --name=cell1 --cell_type=child --username=guest --password=password --hostname=<child ip> --port=5672 --virtual_host=/ --woffset=1.0 --wscale=1.0

Then in screen, find the n-cell-* services and restart them.

Child

nova-manage cell create --name=parent --cell_type=parent --username=guest --password=password --hostname=<parent ip> --port=5672 --virtual_host=/ --woffset=1.0 --wscale=1.0

Then in screen, find the n-cell-* services and restart them.

Conclusion

In this post, we created two virtual machines, installed Devstack while enabling nova-cells. Finally, we actually configured the cells to talk to one another. This post was largely an amalgamation of two other posts highlighted in the resources section

Resources

Linkdump Bare-Metal Edition

Here goes the first post on the Hugo platform on the github blog. Some links I had to close out as I consolidated memory to fire up a multi-devstack.