Cody Bunch Some Random IT Guy - OpenStack, DevOps, Cloud, Things

OpenStack Vancouver in Security

Having done some work around OpenStack and Security, and even spoken in a prior summit, I’m always interested to see how many InfoSec sessions get accepted and what the various topics are.

This time around I am really excited, not so much by the quantity, but by the content and direction folks are thinking. Take a look over some of these sessions:

The first two address, or begin to anyways, what happens to security as one moves to an agile / devops / deploy to production all the time style world. The next one addresses hardening and security OpenStack at the source code level.

Following that, a session addressing detection, which has long needed some vision. Finally, the last two by Rob Clark look like they’ll drive home the points made in the others.

There are plenty more security related sessions at this summit as well.

Update: Userdata Hardening Script

A while back I did a cleanse of the gists I had collected on github. One of the unfortunate side-effects of this, was that I lost some post content. Apologies.

The biggest one that went missing was the userdata script I use on Ubuntu servers. Here it is, updated, and with some additions for better firewalling / logging and the like. Shoot me a note on twitter with any questions or comments.

vDM Zombies and other late night ramblings

Ok, so I blame @discoposse (web) for this. That is, I’ve been drawn into the Interop Virtual Design Master challenge. What follows below are the late night ramblings of someone without enough caffeine or knowlege of what exactly is handy after a Zombie outbreak.

Intro

The prompts for this challenge, both parts 1 and 2 deal with how to bring infrastructure back online in a post zombie apocalypse world. As not much more had been specified other than a number of users and make the infrastructure defensible, we’re going to go off the deep end rather than with a traditional setup.

This ‘design’ is setup to be phased in as various parts of the infrastructure are brought online.

Assumptions

Someone did something dumb. Likely mixing both nano technology and bioengineered genetic weapons. Whatever the case, to say there was fecal matter and that damned fan would be an understatement. Everything is offline. Everything. The power grid is in shambles, most of the power plants have become hold out pockets of zombies, or have fallen in to various stages of ‘melt down’. Communications, which over the last several decades has come to depend largely on electronics, well, they’re turbo offline.

In both the Vegas and Anchorage hold out, there have been a few strong personalities that have stepped forward. We used to call them preppers, but, well, now they’re the folk with solar panels and food stock. There are also a number of first responders (fire fighters, police, and what’s left of the national guard) that have begun to reestablish some form of normal.

As the mission at this stage is to handle the intimidate situation dictates the use of pen/pencil & paper to keep records.

It is also assumed, that the zombie outbreak was middling to quick. Allowing for some preparation and notification for folks to arrive at the shelter facilities. About 3-7 days worth of food & water. Or, enough time to secure the means for producing more of the same.

User Persona

There are two basic user persona we’re designing for:

  • First responders / Preppers
  • Survivors

First Responder / Prepper

In this user class we have the ‘helpers’. These are the folks with the guns, food, shelter, and some solar panels. As users of our new infrastructure, they require the ability to communicate with one another over middling to long distances to support both the day to day operations of the camp as well as scouting missions and logistics.

Survivors

The survivors depend on our infrastructure both directly to receive news and to reach out to loved ones who may have survived.

Phase 1 - Food, Shelter, Support, Comms

Phase one begins with bringing the bases of operation online and providing basic services for those coming in and dealing with the outbreak.

Design Goals

Establish a base camp, bring basic services (food, etc) online, and support the communications and logistics missions to acquire further supplies.

Basic Design

This basic design is a rough outline of the first phase of this plan. That is, get a base-camp up to basic operational capacity: Shelter & Comms.

These however will be approached out of order, as based on the assumptions, survivors & helpers will have enough food & water supplies to bring comms back online first to further support the turn-up.

Note: Due to time constraints, we’ll not go into a full operational plan, contingencies, troubleshooting, and the rest that would accompany a full design. Any haste at this stage would result in the loss of life.

Need 1 - Comms

As we are dealing with first responders, their vehicles and persons are equipped with some form of radio communication. This, in addition to the solar / generators or other power producing equipment provided by the preppers will support the first effort at comms.

In this case, radios were chosen as they are readily available, designed for adverse conditions, and are relatively low power. Through the use of a relay network, that is people stationed at 75% of the effective radio distance to relay the message, we are able to provide robust communication back to base camp to organize parties to salvage food and water supplies from what bits of civilization that are now uninhabited. This communications network also has regular check-ins to protect against node failure and give early warning of zombie movements in the are.

This radio network has limited broadcast abilities due to power constraints. However, once daily, survivors are allowed to broadcast a prerecorded message to reach out to loved ones. Any messages received throughout the day are relayed by the network and radio operators to the survivor in question.

Need 2 - Shelter

Shelter! This is where the preppers shine. Between personal shipping container bunkers, extra tarps and enough ammo to clear out an old shopping mall’s worth of zombies. More specifically, this design requires the preppers to take over the supernap. While the existing comms infrastructure in the area isn’t hugely valuable at this stage, the size and design of the facility will allow for the shelter of people, recirculation of air, and be easily defensible.

The design requires the Alaska based preppers take over the 5th Avenue Mall in Anchorage. Once cleared & secured, the building has access to nearby parks and greenways to provide additional food and water.

On Food and Water

It needs restating that the first responders and preppers bringing these facilities online will both be able to procure as they secure, as well as once secure & comms operational, provide the survivors the means to organize into parties and procure & secure further.

Phase 2 - Defense and Expansion

Oh. No. They. Didn’t. sqrt(-shit)^2.

That is, folks are several months into what has to be one of humanities biggest challenges to date, and now there are zombie apologists. We need to defend our infrastructure whilst bringing other survivor colonies online.

Design Goals

Defend. Expand.

Basic Design

This is broken up by component. Comms, shelter, expansion.

Comms

First up, comms! - Turns out packet radios and our distributed radio network are fairly robust. However, they are subject to ‘meat’ problems. That is, when the zombies meat you, comms are off line. To address this challenge, with the limited technologist contingent onhand, the first order of business is training. That is, each technologiest will be assigned a party of 2 ‘grunts’ and 2 guards. The technologist and the grunts job is two fold: 1) establish more powerful antenna and automated radio relay stations; and 2) train the grunts enough to allow them to fan out and perform the network upgrade autonomously.

There is another design challenge that needs to be addressed coms wise. That is, the long distance comms between survivor outposts. The radio network is still power constrained, and with the zombie apologists in action, also highly variable. To overcome this, we suggest the training of homing pigeons and a robust (n+1 birds), encoded messaging protocol.

Shelter

This is broken into two bits:

  • Alaska
  • Las Vegas

In Alaska it is recommended that the encampment work to harden the 5th Avenue Mall. Specifically on the 6th Avenue street side we recommend the destruction of the two over-road people bridges, and the defense of all ingress / egress points except those critical with concrete of other available building material.

In Las Vegas, continue to employ the private security team (the dudes with M16’s at the Supernap). The approaches to each building are already Tier IV Gold certified, and fairly robust.

Expand

To expand to the next three survivor outposts, we suggest the following:

  • Salvage multiple trucks from the local area to include, as available:
    • School Bus (or charter bus)
    • Mitsubshi FUSO Flatbed trucks

In the schoolbus will be sent an number of first responders and trained survivors to establish the next base-camp. The flatbed cargo truck is to be loaded with supplies: gasoline, food, water, ammo, vehicle repair parts, radio equipment, and pigeons.

Each additional network will provide additional radio broadcast points, further hardening the radio network against failure and allowing the survivors a greater chance of reaching loved ones. Additionally, the increasd robustness of the network will allow the first responders to better communicate and organize supplies, scouting sorties, defense, and broadcast zombie movement.

VMware & Vagrant Performance Hacks

I believe I’ve posted on this before, however I think I deleted the gist that contained these bits of info.

Here are some Vagrantfile settings specific to VMware desktop products that will help you eek out a bit more performance.

# VMware Fusion / Workstation
config.vm.provider :vmware_fusion or config.vm.provider :vmware_workstation do |vmware, override|
override.vm.synced_folder ".", "/vagrant", type: "nfs"

# Fusion Performance Hacks
vmware.vmx["logging"] = "FALSE"
vmware.vmx["MemTrimRate"] = "0"
vmware.vmx["MemAllowAutoScaleDown"] = "FALSE"
vmware.vmx["mainMem.backing"] = "swap"
vmware.vmx["sched.mem.pshare.enable"] = "FALSE"
vmware.vmx["snapshot.disabled"] = "TRUE"
vmware.vmx["isolation.tools.unity.disable"] = "TRUE"
vmware.vmx["unity.allowCompostingInGuest"] = "FALSE"
vmware.vmx["unity.enableLaunchMenu"] = "FALSE"
vmware.vmx["unity.showBadges"] = "FALSE"
vmware.vmx["unity.showBorders"] = "FALSE"
vmware.vmx["unity.wasCapable"] = "FALSE"
vmware.vmx["vhv.enable"] = "TRUE"
end

The basics of what we’re doing is as follows:

  • Using NFS shared folders
  • Turning off logging
  • Tuning memory settings
  • Turning off unity

Hope this saves you some time.

SaltStack and Multi-Minions with Vagrant

Needed me a SaltStack test environment with a flexible number of minions. However, after a few quick searches there wasn’t much around that fit this bill. What follows here, then, is a Vagrantfile, minion.conf, and bash file to build this environment. In gist form:

References

http://humankeyboard.com/saltstack/2014/saltstack-virtualbox-vagrant.html

On Mentorship - Round 2

This started as an idea, a motion to put the user back into the user group. This is a great idea, and having taken on, and attempted to assist 5-ish folks in the first round, I learned a lot, and want to not only do this again, but want to make it a more frequent, ongoing sort of thing.

That is, beyond just mentoring someone short term, say over a few weeks, or until their next speaking engagement. Instead I’d like to help a bit longer term, and more broadly reaching. So here it goes:

The Program

Not sure if calling it a program right, but alas. Over the next quater or so, I’ll provide unlimited email support, and as much Skype / phone support as I can handle. At a minimum this means: 1 hour a week on Skype (or phone) to talk about where you’re at, what you’ve worked on, or are working on, etc. These are 1 - to - 1 mentorships, that is, you and I working on helping you get to that next step.

Class Q2 - 2015

Class (if we can call it a class, a huddle, a gander, etc?), not really sure what to call it, the name isn’t as important as what it represents. It represents a group of five individuals at some point along their career path that want some help doing that ‘next’ thing.

You

Looking to do that ‘next’ thing. Have at least an hour a week for a Skype (or G+, or regular old phone call) to chat. Have a willingness or the ability to find more time in a week to work on that ‘next’ thing.

That’s pretty much it. There are no qualifiers other than “Don’t be an asshole”.

Me

Me, I do things. I’ve published a few books, I’ve worked with others to help them get down that path. I’ve spoken at some events, and again, have helped some others down that path. I helped start a podcast, which in turn has helped some folks take that next step. I also want to do more.

Selection Process

To be fair, I’m not sure this will get even the requsite 5 responses. As this is the second round and things aren’t exactly formal yet, I think that’s all there is too it. Be one of the 5. If there are more than 5 signed up, we’ll figure it out from there. If there are A LOT more than 5, I’ll let y’all know how it works from there.

Application

Here it is.

That’s all there is to it. Disclaimer: It’s a Google form, who’s only recipient is me (insofar as these things can be guaranteed).

Link Dump - Docker & InfoSec

It’s that time again, wherein my borwser is eating all my memory and the tabs need to be closed.

That’s it this time around.

A Detour Into Camp Coffee

What? Camp who?

https://pbs.twimg.com/media/B91WL3nCQAAuegd.jpg:large

I like coffee. A lot. As spring approaches, I am also gearing up to head back outdoors. Camping and coffee don’t always get along, however. That is, you can make some really good coffee when doing “plop and drop” camping, but if you’re reducing the amount of kit you carry, your options start to get really limited.

With that in mind, I decided to take on the ‘Camp Coffee’ problem by well… trying all the coffee. For Science!

Camp Coffee Showdown

There were about 6 rounds involved in this, ranging from instant to stove top. Each produced hugely different results and what follows are my experiences with each. You’ll note the ‘stove top’ was used here, as conditions outside weren’t conducive to fire construction.

The rounds:

  • Taster’s Choice
  • Folgers Instant
  • Starbucks VIA
  • Percolator
  • Cowboy / Turkish
  • Mokka

Note: Before we get too deep into this, I left out some of the standard options, travel french press, aeropress, etc. While you can pack those, they’re also more or less known quantities / qualities.

Camp Coffee Round 1 - Taster’s Choice

https://pbs.twimg.com/media/B9QBP0bCUAA4E6M.jpg:large

https://pbs.twimg.com/media/B9QBPz1CUAAVQoT.jpg:large

Yes it’s instant coffee. It is also everything that is wrong in the world, in the universe, all bundled up into one little package of crystallized hate. I mean, I suppose it’s coffee, if you like aromatic gym socks and hints of industrial cleanser. This was the only one in the round up to make me spit and pour it out as fast as I could.

Quality: None. There was no quality here. Unless you are trying to get an oil stain out of your driveway I guess.

Trouble: All of it. All the trouble.

Camp Coffee Round 2 - Folgers Instant

https://pbs.twimg.com/media/B9QD9tACEAEUmlt.jpg:large

https://pbs.twimg.com/media/B9QD9sNCUAEoFrK.jpg:large

This stuff is magic. That is, after the old armpit socks from the last round, it was amazing how much like Folgers this tasted. Not sure if that is a compliment or not, but well, it was tolerable.

Quality: Only if I can’t find VIA

Trouble: None

Camp Coffee Round 3 - Starbucks VIA

https://pbs.twimg.com/media/B9QGar9CUAANM3I.jpg:large

https://pbs.twimg.com/media/B9QGarvCQAA9O5U.jpg:large

Nothing fancy here. Hot water, Coffee Powder, Stir. It is the stir part that will get you. Unlike the other two ‘instant’ coffees in the round up, this one uses a ‘micro-grind’ of sorts. Like cowboy below, don’t drink the last sip.

Quality: Decent, bordering on good

Trouble: None

Camp Coffee Round 4 - Percolator

https://pbs.twimg.com/media/B9QKdBgCAAEPYpw.jpg:large

https://pbs.twimg.com/media/B9QKdChCAAEidGd.jpg:large

The coffee snob in me is almost ashamed of having done a percolated pot. Yes, it’s an American coffee staple. Yes, it’s what I grew up on. Yes, it brings back the memory of that amazing trip to Bear Den campground in North Carolina where I brewed my parents a cup of coffee, and completely forgot the water. Apparently one can burn coffee.

The flip side of this is: I could totally see bringing a fire or stove top pot on a trip if I had a smaller one in my arsenal.

Quality: Alright

Trouble: Don’t forget the water.

Camp Coffee Round 5 - Cowboy / Turkish

https://pbs.twimg.com/media/B9QS9-JCUAIS70d.jpg:large

https://pbs.twimg.com/media/B9QS9__CYAEgs2L.jpg:large

https://pbs.twimg.com/media/B9QUlDhCIAEjnJR.jpg:large

So I call this Cowboy rather than Turkish, as well, they’re prepared almost the same: powdered coffee grounds, boiled in the water a few times. The differentiation here, is that Turkish generally calls for an almost equal amount of sugar to go with it. Brewing it was fast, but it was still a bit of trouble that is, having to schlep the grinder and the little Turkish pot thing. It was also gritty as heck. The slurry at the bottom should only be consumed if you need real ultimate power.

Quality: Good

Trouble: Medium

Camp Coffee Round 6 - Mokka

https://pbs.twimg.com/media/B9QVhy-CIAIf6qg.jpg

https://pbs.twimg.com/media/B9QVhyZCMAAJlzp.jpg

The stove top Mokka pot. This along with the percolator in round 4 is how I grew up drinking coffee. At my grandfather’s house it’d be called “black” coffee and brought out around the holidays or when he was playing cards out by the pool with his buddies. It was often mixed with Sambuca around the holidays. I knew what to expect on this one. It was added to get an idea of the time vs trouble. It was a lot trouble, btw. Good for a plop and drop setup, but not so much if you need mobility.

Quality: Great

Trouble: A lot

Camp Coffee Summary

Let it not be said there are not consequences to drinking 6 cups of coffee in less than an hour. With that, I’ll likely go this season with either an Aeropress, Cowboy, or Via depending on packing requirements.

Basic Server Hardening with Salt

Before we get started, take a gander at my last few posts on this, here and here.

The idea here is roughly the same. That is, build a small, basic ‘base’ profile, template, state, or whatever, that has some simple hardening bits applied. The idea being to give you a reasonable start in turn letting you apply additional layers down the road.

The reason you’d move away from doing this with OpenStack Orchestration (Heat) and into a config management tool is that it allows you to apply the same practices more generically. That is not everyone runs OpenStack, but lots of folks are moving to some flavor of configuration management tool, if they weren’t there already. You can then include these SaltStates in Heat or whatever orchestration tool of choice.

Basic Hardening with SaltStack

For the sake of not making this a huge-tastic blog post, we’ll skip the part where I explain the what and how of getting started with SaltStack. Many others have done better than I. What follows is my top.sls, secureserver.sls and sysctl.sys files.

top.sls

This file controls what Salt minions get what ‘states’. For securing all the servers, that’s pretty straight forward. I would like to call out, however, the ordering of the states:

base:
  '*':
    - sysctl
    - secureserver

This defines a ‘base’ state that will match for all Salt minions. (Minion is Salt for agent.) It then states to apply first the sysctl state, then the secureserver state.

sysctl.sls

There isn’t much fancy to this file. In fact, it’s more or less a YAML formatted version of what you’d throw into /etc/sysctl.conf. What the lines below do is turn off IP routing, ignore broadcasts, responses, and all manner of other fun icmp stuff.

net.ipv4.icmp_echo_ignore_broadcasts:
  sysctl.present:
    - value: 1

net.ipv4.icmp_ignore_bogus_error_responses:
  sysctl.present:
  - value: 1

net.ipv4.tcp_syncookies:
  sysctl.present:
    - value: 1

net.ipv4.conf.all.log_martians:
  sysctl.present:
  - value: 1

net.ipv4.conf.default.log_martians:
  sysctl.present:
    - value: 1

net.ipv4.conf.all.accept_source_route:
  sysctl.present:
    - value: 0

net.ipv4.conf.default.accept_source_route:
  sysctl.present:
    - value: 0

net.ipv4.conf.all.rp_filter:
  sysctl.present:
    - value: 1

net.ipv4.conf.default.rp_filter:
  sysctl.present:
    - value: 1

net.ipv4.conf.all.accept_redirects:
  sysctl.present:
    - value: 0

net.ipv4.conf.default.accept_redirects:
  sysctl.present:
    - value: 0

net.ipv4.conf.all.secure_redirects:
  sysctl.present:
    - value: 0

net.ipv4.conf.default.secure_redirects:
  sysctl.present:
    - value: 0

net.ipv4.ip_forward:
  sysctl.present:
    - value: 0

net.ipv4.conf.all.send_redirects:
    sysctl.present:
    - value: 0

net.ipv4.conf.default.send_redirects:
  sysctl.present:
    - value: 0

secureserver.sls

This file, unlike the sysctl bits, actually does some install and config bits. If you’ve read the post I mentioned at the beginning of this post, you’ll recognize the packages being installed.

The configure bits are two fold: 1) We configure logwatch using a file hosted on the Salt master; 2) We configure IP tables to allow ssh and deny all the other things:

iptables:
  pkg:
    - installed

denyhosts:
  pkg:
    - installed

psad:
  pkg:
    - installed

fail2ban:
  pkg:
    - installed

logwatch:
  pkg:
    - installed

aide:
  pkg:
    - installed

/etc/cron.daily/00logwatch:
  file:
    - managed
    - source: salt://cron.daily/00logwatch
    - require:
      - pkg: logwatch

ssh:
  iptables.append:
    - table: filter
    - chain: INPUT
    - jump: ACCEPT
    - match: state
    - connstate: NEW
    - dport: 22
    - proto: tcp
    - sport: 1025:65535
    - save: True

allow established:
  iptables.append:
    - table: filter
    - chain: INPUT
    - match: state
    - connstate: ESTABLISHED
    - jump: ACCEPT

default to reject:
  iptables.append:
    - table: filter
    - chain: INPUT
    - jump: REJECT

Summary

In this post we’ve covered how to do some very very basic hardening of your server using SaltStack. This likely wont work for all circumstances (like if you’re going to actually run nginx or apache, you need to add those ports accordingly).

Resources

Link Dump - Random Edition

Today’s link dump is brought to you by Google Chrome and my poor swap file.