Cody Bunch Some Random IT Guy - OpenStack, DevOps, Cloud, Things

Updated Blog / Markdown workflow

A while ago I posted about my markdown workflow. While that workflow was decent and worked for a goodly little while, it left quite a bit to be desired. Specifically, I the posting process into Wordpress was cumbersome at best.

A New Blog Engine

Like I said above, working with Wordpress and markdown was cumbersome and then some. So it was time for something new. At first I was going to use similar to this, which is actually what is in use on the openstackcookbook.com site. It has the following benefits:

  • Posts are written in Markdown
  • Integrated with gh-pages
  • Straighforward publishing process git push origin master

It was almost what I needed. In the end I went with Hugo. What Hugo added was some pluggable themes and templates. Also the ability to run locally before pushing to github.

To setup Hugo on github, use this.

Current Workflow

My current workflow needed some help then. I kept the same sublime text plugins, that is:

  • MarkdownEditing
    • This has a number of really handy keyboard bindings. It also has some decent highlighting and what not.
  • Markdown Preview
    • This one allows me to go from Markdown into what it’ll look like on the web.
  • Markdown TOC
    • This lets me go from a basic set of sections and files into a more full fledged table of contents.

Additionally, to create a new post, I modified the Rakefile found here, to look like this:

require 'fileutils'
task :post do
    title = ENV['title'] || "new-post"
    tags = ENV['tags'] || ''
    make_img_dir = ENV['imgdir'] || false
    date = (ENV['date'] ? Time.parse(ENV['date']) : Time.now).strftime('%Y-%m-%d')
    filename = File.join('.', "#{Time.now.strftime('%Y-%m-%d')}-#{title.strip.gsub(' ', '-').gsub(/[^\w-]/, '')}.md")
    open(filename, 'w') do |post|
        post.puts "---"
        post.puts "title: \"#{title}\""
        post.puts "date: #{Time.now.strftime('%Y-%m-%d')}"
        post.puts "categories: "
        post.puts "---"
        post.puts "\nYour content here."
        if make_img_dir
            img_dir = File.basename(filename.chomp(File.extname(filename)))
            FileUtils.mkdir_p("../images/posts/#{img_dir}")
            post.puts "\n" * 5
            post.puts "[imgdir]: /images/posts/#{img_dir}/"
        end
    end
end

So creating a new post goes like this: bunchc: blog/content/posts$ rake post title=”Title Here”

In turn, that creates a YYYY-MM-DD-Title-Here.md file for editing in sublime. It also adds the metadata section at the top of the file for me:

---
title: "Updated Blog / Markdown workflow"
date: 2014-08-14
categories: 
---

From there, I write the file, save the file, and run ./deploy.sh from the Hugo installer linked earlier. That handles all the pushing and bits to git.

Summary

The gist of it is, Wordpress was a bit much and a bit heavy for what I needed. Hugo, Markdown, and GitHub Pages gave me a streamlined process that looks decent for posting.

Multi-Node Devstack with Neutron and Cells

OpenStack Cells allow you to break up Nova into smaller domains. In turn, this allows for a number of interesting things. Not the least of which is scale.

From the OpenStack.org docs: Note I’ve added the bold on some important points.

Cells functionality enables you to scale an OpenStack Compute cloud in a more distributed fashion without having to use complicated technologies like database and message queue clustering. It supports very large deployments.

When this functionality is enabled, the hosts in an OpenStack Compute cloud are partitioned into groups called cells. Cells are configured as a tree. The top-level cell should have a host that runs a nova-api service, but no nova-compute services. Each child cell should run all of the typical nova-* services in a regular Compute cloud except for nova-api. You can think of cells as a normal Compute deployment in that each cell has its own database server and message queue broker.

So, in addition to breaking down at physical boundaries or failure domains, you can break nova-compute into smaller chunks based on DB and MQ scaling limits.

However, not all of us have multi-geo hundred plus node compute labs to play with… so how do we test out this functionality before hand? Devstack!

Getting Started

tl;dr - git clone https://github.com/bunchc/devstack-cells.git; cd devstack-cells; vagrant up, and go to “Configuring and Creating Cells”

To get started, you’ll need the following:

  • virtualbox (fusion or workstation work just as well)
  • a minimum of two Ubuntu 14.04 VM with:
    • About 2GB ram
    • 2x networks
      • eth0 = NAT
      • eth1 = host only

Both Nodes

Once you have that taken care of, on each node we need to create a stack user:

sudo adduser --disabled-password --gecos "" stack
echo "stack ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers

Parent Node

Time to Download devstack:

sudo su - stack
git clone -b stable/icehouse https://github.com/openstack-dev/devstack 

Note the sudo command. Everything from this point is done as stack

Next, let’s make the local.conf file:

echo "
[[local|localrc]]
ADMIN_PASSWORD=$ADMIN_PASSWORD
DATABASE_PASSWORD=$ADMIN_PASSWORD
RABBIT_PASSWORD=$ADMIN_PASSWORD
SERVICE_PASSWORD=$ADMIN_PASSWORD
SERVICE_TOKEN=a682f596-76f3-11e3-b3b2-e716f9080d50

GIT_BASE=${GIT_BASE:-https://git.openstack.org}

# Cells!
ENABLED_SERVICES+=,n-cell,n-api-meta
DISABLED_SERVICE+=,n-cpu,n-net,n-sch

# Neutron - Networking Service
# If Neutron is not declared the old good nova-network will be used
ENABLED_SERVICES+=,q-svc,q-agt,q-dhcp,q-l3,q-meta,neutron,q-lbaas,q-vpn,q-fwaas

# Neutron Stuff
OVS_VLAN_RANGES=RegionOne:1:4000
OVS_ENABLE_TUNNELING=False

## Images
# 32bit image (~660MB)
IMAGE_URLS+=",http://fedorapeople.org/groups/heat/prebuilt-jeos-images/F19-i386-cfntools.qcow2"
# 64bit image (~640MB)
IMAGE_URLS+=",http://fedorapeople.org/groups/heat/prebuilt-jeos-images/F19-x86_64-cfntools.qcow2"
IMAGE_URLS+=",http://mirror.chpc.utah.edu/pub/fedora/linux/releases/20/Images/x86_64/Fedora-x86_64-20-20131211.1-sda.qcow2"
IMAGE_URLS+=",http://launchpad.net/cirros/trunk/0.3.0/+download/cirros-0.3.0-x86_64-uec.tar.gz"

# Output
LOGFILE=/opt/stack/logs/stack.sh.log
VERBOSE=True
LOG_COLOR=False
SCREEN_LOGDIR=/opt/stack/logs
" | tee -a /home/stack/devstack/local.conf

The important bit for cells in this file are:

ENABLED_SERVICES+=,n-cell,n-api-meta
DISABLED_SERVICE+=,n-cpu,n-net,n-sch

This enables the cell service, and the nova-api metadata service. It then disables compute, nova-networking, and the scheduler. Those tasks will be handled on the children.

With that out of the way run ./stack.sh and grab a coffee. When it completes, add the following to /etc/nova/nova.conf:

[cells]
enable=True
name=api
cell_type=api

Finally, restart the n-api serivice.

Child Node

Time to Download devstack:

sudo su - stack
git clone -b stable/icehouse https://github.com/openstack-dev/devstack 

Note the sudo command. Everything from this point is done as stack

Next, on the child node, we create a local.conf file

cd devstack
echo "
[[local|localrc]]
ADMIN_PASSWORD=$ADMIN_PASSWORD
DATABASE_PASSWORD=$ADMIN_PASSWORD
RABBIT_PASSWORD=$ADMIN_PASSWORD
SERVICE_PASSWORD=$ADMIN_PASSWORD
SERVICE_TOKEN=a682f596-76f3-11e3-b3b2-e716f9080d50

HOST_IP=<ipaddres of eth0>
SERVICE_HOST=<ip address of controllers eth0>
MYSQL_HOST=$SERVICE_HOST
RABBIT_HOST=$SERVICE_HOST
Q_HOST=$SERVICE_HOST
GLANCE_HOSTPORT=$SERVICE_HOST:9292

GIT_BASE=${GIT_BASE:-https://git.openstack.org}

# Cells!
ENABLED_SERVICES+=,n-cell
DISABLED_SERVICE+=,n-api,key,g-api

# Neutron - Networking Service
# If Neutron is not declared the old good nova-network will be used
ENABLED_SERVICES+=,q-svc,q-agt,q-dhcp,q-l3,q-meta,neutron,q-lbaas,q-vpn,q-fwaas

# Neutron Stuff
OVS_VLAN_RANGES=RegionOne:1:4000
OVS_ENABLE_TUNNELING=False

# Output
LOGFILE=/opt/stack/logs/stack.sh.log
VERBOSE=True
LOG_COLOR=False
SCREEN_LOGDIR=/opt/stack/logs
" | tee -a /home/stack/devstack/local.conf

The important bit for cells is:

ENABLED_SERVICES+=,n-cell
DISABLED_SERVICE+=,n-api,key,g-api

This enables the cell service and turns of keystone along with nova-api and glance-api services. Those in turn will be handled by the parent.

Next, run ./stack.sh and grab a coffee. When it completes, add the following to /etc/nova/nova.conf

[cells]
enable=True
name=cell1
cell_type=comput

Configuring and Creating Cells

Ok, what happened above was we got two nodes up and running and ready to go with Devstack and the prerequsite services for cells. Now we actually have to actuall make the cells. To do that:

Parent

nova-manage cell create --name=cell1 --cell_type=child --username=guest --password=password --hostname=<child ip> --port=5672 --virtual_host=/ --woffset=1.0 --wscale=1.0

Then in screen, find the n-cell-* services and restart them.

Child

nova-manage cell create --name=parent --cell_type=parent --username=guest --password=password --hostname=<parent ip> --port=5672 --virtual_host=/ --woffset=1.0 --wscale=1.0

Then in screen, find the n-cell-* services and restart them.

Conclusion

In this post, we created two virtual machines, installed Devstack while enabling nova-cells. Finally, we actually configured the cells to talk to one another. This post was largely an amalgamation of two other posts highlighted in the resources section

Resources

Linkdump Bare-Metal Edition

Here goes the first post on the Hugo platform on the github blog. Some links I had to close out as I consolidated memory to fire up a multi-devstack.