Cody Bunch Some Random IT Guy - OpenStack, DevOps, Cloud, Things

Fixing cgroup memory on Raspbian Buster for Kernel 5.4.x

After running sudo apt -y full-upgrade my Raspberry Pi k3s cluster got upgrade from 4.x kernels to 5.x. Which turbo-broke k3s.

The Frror

After rebooting into the new kernel and k3s not working, I found the following error in both /var/log/syslog/ and journalctl -xeu k3s (or for the worker nodes journalctl -xeu k3s-agent)

level=fatal msg="failed to find memory cgroup, you may need to add \"cgroup_memory=1 cgroup_enable=memory\" to your linux cmdline (/boot/cmdline.txt on a Raspberry 
Pi)"

However, those values were totally in /boot/cmdline.txt:

$ cat /boot/cmdline.txt
console=serial0,115200 console=tty1 root=PARTUUID=dd5ac5d2-02 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait cgroup_enable=1 cgroup_memory=1 swapaccount=1 cgroup_enable=memory dwc_otg.lpm_enable=0 

What could it be then?

The Fix

This fix, of course, is to update the firmware on the pi via rpi-update

At the time of this writing, the fix was released 4 days ago according to this Github issue.

While you can run rpi-update by itself and answer its warning prompt, that can get tedious for more than one or two nodes. Since I was upgrading nine nodes, something more robust was needed. Something like the following command:

$ sudo PRUNE_MODULES=1 RPI_REBOOT=1 SKIP_WARNING=1 rpi-update

Or for those using Ansible to manage their fleet:

ansible -i swarm.yml nodes --become -m shell -a 'sudo PRUNE_MODULES=1 RPI_REBOOT=1 SKIP_WARNING=1 rpi-update'

Note: You should probably do at least one node the manual way to ensure the update works. The commands shown here can… accelerate your journey to serverless.

Random PowerShell/PowerCLI Snippets

Cleaning out a few browser tabs, which of course means putting things where I will remember them. This time, that is a bunch of random PowerShell and PowerCLI I’ve been using.

The Snippets

The following sections will state if the snippet is either [PowerShell] or [PowerCLI] and include a link to the site where I found said snippet. The PowerCLI snippets assume a connection to either vCenter or ESXi.

[PowerCLI] Get IP Address of VM

(Get-VM -Name test01).Guest.IPAddress

vsaiyan.info (@vSaiyanman)

[PowerShell] Create an Empty File (touch)

# New file
New-Item -ItemType file example.txt

# Update timestamp on a file
(gci example.txt).LastWriteTime = Get-Date

https://superuser.com/a/540935

[PowerCLI] Rescan All HBAs

# Rescans all HBAs on all hosts
Get-VMHost | Get-VMHostStorage -RescanAllHba

# Rescans all HBAs in a given cluster
Get-Cluster -Name "ProVMware" | Get-VMHost | Get-VMHostStorage -RescanAllHba

Note: The fact that I wrote this post about a decade ago still blows my mind.

vbrownbag.com

[PowerShell] Measure Execution Time (time 'someCommand')

Measure-Command { Get-VMHost | GetVMHostStorage -RescanAllHba }

Microsoft Docs

[PowerShell] Windows Event Logs

Get-EventLog -LogName Security -EntryType Error -Newest 5

Microsoft Docs

[PowerShell] Disable Windows Defender Firewall

Set-NetFirewallProfile -Profile Domain,Public,Private -Enabled False

[PowerShell] List / Add Windows Defender Firewall Rules

# List the rules. This produces A LOT of output
Get-NetFirewallRule

# Disables outbound telnet
New-NetFirewallRule -DisplayName "Block Outbound Telnet" -Direction Outbound -Program %SystemRoot%\System32\tlntsvr.exe Protocol TCP LocalPort 23 -Action Block PolicyStore domain.contoso.com\gpo_name

Microsoft Docs

Getting Started with Ansible and vSphere

My work has started to shift around some recently and I find myself returning to VMware products after a hiatus. In addition to running into a blog post I wrote a decade ago, it has been interesting to see how else VMware has kept up.

In particular, I have spent A LOT of time automating various things with Ansible, so discovering the extensive list of VMware modules. This post shows how to use the Ansible VMware modules to launch your first VM.

Getting Started

There are quite a few prerequisites to getting this going. Buckle up!

First, the requirements for your ‘Control Node’, where you will run Ansible from:

  • Ansible
  • Python
  • Pyvmomi
  • vSphere Automation Python SDK
  • Access to vSphere

Installing the prerequisites on Ubuntu 18.04

Note: While you can likely install these tools directly on Windows, I’ve found it easier to use Ubuntu by way of WSL2.

  1. Install Ansible:
root@9a45e927fc78:/# apt update -q
root@9a45e927fc78:/# apt install -y software-properties-common
root@9a45e927fc78:/# add-apt-repository --yes --update ppa:ansible/ansible
root@9a45e927fc78:/# apt install -y ansible
  1. Install / update Python
root@9a45e927fc78:/# apt install -y python-minimal python-pip
  1. Install Pyvmomi
root@9a45e927fc78:/# pip install pyvmomi
  1. Install the VMware Python SDK
root@9a45e927fc78:/# pip install --upgrade git+https://github.com/vmware/vsphere-automation-sdk-python.git

Creating a VM with Ansible

With everything we need installed, we can now make the VM with Ansible. We do that by creating an Ansible playbook, and then running it. An Ansible playbook is a YAML file that describes what you would like done. To create yours, open your favorite text editor and paste the following YAML into it.

# new-vm-playbook.yml
- name: Create New VM Playbook
  hosts: localhost
  gather_facts: no
  tasks:
  - name: Clone the template
    vmware_guest:
      hostname: "vcenter.codybunch.lab"
      username: "administrator@codybunch.lab"
      password: "ultra-Secret_P@%%w0rd"
      resource_pool: "Resource Pool to create VM in"
      datacenter: "Name of the Datacenter to create the VM in"
      folder: "/vm"
      cluster: "MyCluster"
      networks:
          - name: "Network 1"
      hardware:
          num_cpus: 2
          memory_mb: 4096
      validate_certs: False
      name: "new-vm-from-ansible"
      template: "centos-7-vsphere"
      datastore: "iscsi-datastore"
      state: poweredon
      wait_for_ip_address: yes
    register: new_vm

The great thing about describing the changes in this way, is that the file itself is rather readable. That said, we should go over what running this playbook will actually do.

The playbook tells Ansible to execute on the localhost (hosts: localhost). You can change this to be any host that can access vCenter and has pyVmomi installed. It then tells Ansible we would like to build a vm (vmware_guest) and supplies the details to make that happen. Make note of the folder: "/vm" setting which will place the VM at the root of the datacenter. As with the other variables, change this to fit your environment.

Once you have adjusted the playbook to suit your environment, it can be run as follows:

ansible-playbook new-vm-playbook.yml

Resources

Migrating to Hyper-V: Convert VMDK to VHDX

I have been on a desktop OS and tools journey for the last six or nine months. Long story short, my 2012 Retina MacBook Pro gave up the ghost in a most unpleasant way. I grabbed a System76 laptop, and gave Linux on the desktop a shot for a little while. Eventually I repaved that and moved to Windows 10 full time. As part of that, I enabled WSL 2, which breaks VMware desktop products. No biggie, one can convert VMs, right?

Well…

Requirements

It is assumed that you have the following:

  • Win10
  • WSL 2 [1]
  • Ubuntu (in WSL that is)

Note: You can get qemu-img as a native tool for Windows. I used the Linux version as I had a bit more experience with it.

Process

There are three basic parts of this process: Converting the VMDK, Importing into Hyper-V, and Installing integration services.

Converting from VMDK to VHDX

This stage will be done entirely within WSL/Ubuntu, so open a terminal and then use the following commands:

  1. Ensure you have qemu-img:
sudo apt update
sudo apt install qemu-utils
# --output truncated-- #
  1. Convert the image:
PathToVMDK="/mnt/c/Users/bunchc/vms/win7/win7.vmdk"
PathToVHDX="/mnt/c/Users/bunchc/vms/win7/win7.vhdx"

$ qemu-img convert $PathToVMDK -O vhdx $PathToVHDX -p

The command tells qemu-img to convert the VMDK specified in PathToVMDK with an output type (-O vhdx) of vhdx and store the resulting image at PathToVHDX. The -p tells qemu-img to display the progress as it converts. You will need to change the PathTo variables to reflect your environment.

Note: There are some tools that will do this in PowerShell [2] [3]. However, they can be rather flaky depending on the source VMDK. qemu-img was quite a bit more robust in this regard.

Importing into Hyper-V

Once the conversion is complete, you can import the VM into Hyper-V as you would any other vhdx. You will want to be careful when selecting which generation, that is, unless you installed the VM with UEFI, you will want to select ‘Generation 1’.

Once Imported, edit the settings of the VM.

When happy with the settings power the VM on, log in, uninstall VMware Tools and reboot, then prepare to install Hyper-V integration services.

Installing integration services

The Hyper-V integration services will generally be installed seamlessly via Windows Update. If this is not the case, or you are using an older version of Windows, download the integration tools [4], and then use the following command from an elevated command prompt to install them:

dism.exe /Online /Add-Package /PackagePath:C:\Users\bunchc\Desktop\windows6.x-hypervintegrationservices-x86.cab

Resources

The following pages were used to help make this post:

  1. Installing WSL-2 on Win10.
  2. Microsoft Virtual Machine Conversion Kit.
  3. Convert a VMware VM to Hyper-V.
  4. Download the integration services package.
  5. Installing Hyper-V integration services manually.

There Are Door Lights - LED Light Control for the Home Lab

There are door lights

With the holiday season over, I found I had an excess of USB powered fairy style LED lights that needed something to do. The garage that the home lab cabinet lives in is also darker than usual during these winter months. Like the Brady Bunch theme song then, these two must somehow come together… and now, THERE ARE DOOR LIGHTS.

Door Lights build goals

Drama aside, when I set out, the goal of the door lights build was:

  • Light up the inside of the homelab cabinet
  • Trigger said lighting when the cabinet doors open and close
  • As much as possible, use parts already on hand

Simple enough, right? What follows then is the documentation of said build, some action shots, and what might get done differently next time.

Hardware

Before we get into the parts list, remember that this build happened with parts on hand. Everything except the Pi Cobbler came from an Arduino starter kit. I do not recall the specific one, but most “IoT” starter kits will include these things.

Parts

  • Raspberry Pi 3 Model B Rev 1.2
  • Raspberry Pi Accessories (Case, Power Supply, SD card)
  • 1uF Capacitor
  • LDR Light Sensor
  • Breadboard
  • Adafruit Pi Cobbler Plus (https://www.adafruit.com/product/2029)

Build

The light sensor circuit I built, and instructions on how to do so came from here.

The finished product:

Finished light sensor circuit

Attached to the Raspberry Pi:

Light sensor attached to raspberry pi

Yes, yes, I need to dust…

Software

With the hardware build out of the way, the next step was to use a bit of software to pull tie everything together. Almost 100% of the difficulty I had getting the software to work is related to how the existing USB libraries handle switchable USB Hubs. You see, instead of messing with relays or other bits of circuitry, the Raspberry Pi 3 Model B (and others) has the ability to entirely shut off the onboard USB ports. YES! Or so I thought.

Raspberry Pi Setup

  • Raspbian Stretch
  • Docker

Docker? Yes, and not because everything needs to run in a container. Rather, because I can build and run ARM images for Docker on my laptop and consume the same images on the Raspberry Pi, which speeds up building said images. Moreover, because I was not 115% sure what packages, libraries, configurations, and any other changes were required, running in a container allowed me to keep the host OS on the Raspberry Pi clean. Which is a good thing, as that Pi also runs Pi-Hole and nobody wants a DNS outage.

The code

The code for this project is primairly two files:

  • door_sensor.py
  • hub_ctrl.py

door_sensor.py

An explanation for the code for door_sensor.py follows:

The first line imports the LightSensor class from the gpiozero Python library. This provides all the functions we will need to see if the sensor we wired up is getting light. If it is, that means that the cabinet doors are indeed open. The second line imports a modified version of hub_ctrl.py from the GNUK project. More on this file in a bit.

Line 4 tells Python that we have a light sensor on pin 4. Line 5 prints the current state to the console.

Lines 7 - 13 are where the fun happens. We start a loop and then wait for it to become light. When it does, we turn on the LED lights and then wait for it to get dark again.

hub_ctrl.py

Earlier in the post I mentioned how the USB libraries in Python do not seem to provide convenience methods for controlling switchable USB hubs. This led down a rabbit hole of USB protocol debugging with usbmon. That is, until I found this code from the GnuK project. The only bit that was missing was a nice way to call it from door_lights.py. To that end, I added a send_command function to said file, that wrapped up a bit of their existing code.

Docker

Here is the Dockerfile for this build:

This container is then launched like so:

docker run --privileged --rm -it \
    --device /dev/bus/usb \
    --name there-are-door-lights 
    there-are-door-lights python3 door_sensor.py

In Action

THERE.ARE.DOOR.LIGHTS.

door lights in action

OpenStack Summit Berlin - Day 1

Day 1

It Starts!

What follows is going to be a slightly expanded form of the live tweeting I will be doing all week. That is, the post will be lightly edited and written stream of consciousness style.

Keynotes

I was a little late to the keynotes today, so missed the beginning. That said, there was a lot going on, and all of it good.

  1. OpenStack Summit is now Open Infrastructure Summit
  2. Next one will be in Denver, CO the week of 29, April 2019
  3. Textiles is crazy interesting.

    OpenStack is the infrastructure on which Oerlikon produces 40,000 bobbins of thread a day. They’re using OpenStack infrastructure to back end containers, AI/ML (for making real-time decisions on production lines). This takes inputs from about 70k sensors per 120 machines.

  4. The theme has been the same as the last bunch of Summits. That is, OpenStack has matured as an infrastructure platform and is the infrastructure supporting more shiny workloads (AI/ML, Kubernetes, etc).
  5. Focus on Day 2 Operations. Infrastructure, at scale, is difficult.
  6. There is an Internet of Cows.
  7. OSP 14 is coming.

Sign all the books

We had a good turnout for the first book signing of the summit. We’ll be doing these every day at 11:00 and 16:00. We have 100 copies total for the OpenStack Cookbook and another 100 of James Denton’s Neutron book

OpenStack Cookbooks

Show floor

The marketplace, aka show floor gets a special call out for being busier than the last handful of summits. There is a good showing of OpenStack service providers along with the usual players (RedHat, Canonical, Cisco, Netapp, etc.). There are also new players, hyper.sh and Ambedded.

Lightning Talks

While sitting to write up this post, the folks from Roche gave a live demo of using Google Home to launch Virtual Machines on OpenStack. If you’re looking for the recording later, the title was “Ok Google, create me a VM, please”

I Accidentally Twitter

Toward the end of the day, I accidentally my Twitter account. That is, the account got locked due to unusual activity. That said, I managed to capture most of my day 1 sessions well and will get those posts up once the videos are available.

Random PowerShell tidbits

Random Useful PowerShell

I’ve needed to use a few PowerShell commands over and over a few times in the last week or two. So that I do not otherwise forget them again, here they are:

Networking

Change DNS address for all Ethernet interfaces

Get-NetAdapter -InterfaceAlias "Ethernet*" | Set-DnsClientServerAddress -ServerAddresses ("4.2.2.2")

Note: This will configure all interfaces that have “Ethernet” listed in the name. You can narrow this search by using Get-NetAdapter.

Set MTU

Get-NetIPInterface -InterfaceAlias "Ethernet*"| Set-NetIPInterface -NlMutBytes 1450

Note: This will configure all interfaces that have “Ethernet” listed in the name. You can narrow this search by using Get-NetIPInterface.

Disable firewall

Set-NetFirewallProfile -Profile '*' -Enabled False

Note: This is generally considered a bad idea. You can get a lot more specific with Set-NetFirewallProfile, and I suggest you do.

Host management

Rename a computer

Rename-Computer -NewName "windows-2016.orangutan.lab"

Enable WinRM

Enable-PSRemoting -SkipNetworkProfileCheck -Force

Note -SkipNetworkProfileCheck <- This saved me quite a bit of heartache. PSRemoting can be rather flaky, the -SkipNetworkProfileCheck flag made it somewhat less so.

Test WinRM to remote box

$so = New-PSSessionOption -SkipCACheck -SkipCNCheck

Enter-PSSession -Computername 10.127.16.74 -Credential "vagrant" -UseSSL -SessionOption $so
Get-IPAddress -IPAddress "10.*"

Change service startup to automatic

Get-Service -DisplayName "Cloudbase*" | Set-Service -StartupType Automatic

[PowerShell] Joining a domain

Recently my world has been centered more and more around Windows. Lately this is not a Bad Thing™. In fact, Windows Server Core and PowerShell have both come a LONG way. In the not so recent past, I wrote about how to set up Active Directory with PowerShell. In this post, I show you how to use PowerShell to join said domain.

Getting Started

The process that follows assumes you have:

  • An Active Directory domain.
  • A server to join to the domain.
  • Optional: Said server is Windows Server Core
  • A user account with access to said domain.
  • A local user account on the server to be joined.

How to do it

To join the server to the domain, we will:

  1. Set DNS to use the Domain Controller
Get-NetAdapter -InferfaceIndex
Set-DnsClientServerAddress -InterfaceIndex 2 `
    -ServerAddresses ("10.127.16.100")
  1. Optional: Rename the computer

This can be done in two ways, either rename & reboot, or rename as part of the join. I have found the two reboot process to work more consistently.

Rename and reboot

# If you have more work to do, remove the -Reboot 
# from the Rename-Computer command and use:
# Reboot-Computer -Force

Rename-Computer -NewName "app-01" -Reboot

Rename at Join

Add-Computer -DomainName "codybunch.local" `
    -NewName "app-01" `
    -LocalCredential 'Administrator' `
    -DomainCredential 'codybunch\Administrator' `
    -Restart -Force
  1. Join the domain

Last, we join the domain.

Note: Only the -DomainName parameter is required. If the others are left unspecified, you will be prompted.

Add-Computer -DomainName "codybunch.local" `
    -LocalCredential 'Administrator' `
    -DomainCredential 'codybunch\Administrator' `
    -Restart -Force

My cloudbase-init.conf file

Because finding a complete sample was harder than I want to admit, here is a copy of my current cloudbase-init file.

Original source here.

NSXpert in 72 hours? VMware NSX Resources

Asking for NSX Help

Community is a wonderful thing. Just today I needed to learn all I could about NSX. The goal was to become an NSX “Expert” by Monday. NSX is a bit too complex for that, but, when asked, the community responded with plenty of links and suggestions. What follows here are a pile of links and my rough plan of attack.

Video Resources

While I’m sure there are more out there, I plan to start with the #vBrownBag NSX series by Tim Davis (@ALDTD). There are 3 pretty intense videos in the series:

Also worth mentioning, the official VMware NSX YouTube Channel.

UPDATE 2018-06-09 @ 09:40

Written Resources

There are a huge number of guides here, but they are comprehensive.

You will need to search for NSX specific labs. Recommended was getting started and then working as best as you could along with the study guides linked prior.

In depth guides for getting up and going on VMware NSX

Validated Reference Designs for NSX

When all else fails check the docs.

When the docs fail, check the communities forums.

UPDATE 2018-06-09

A few more links came up overnight that should be shared.

A PDF Guide to architecting NSX solutions for service providers. This will help you wrap your head around some of the considerations for how to deploy NSX within the context of the VMware SDDC.

The ultimate HOL, is of course, setting it up yourself. This is a subset of the docs that walks you through an installation.

UPDATE 2018-06-09 @ 09:40 More Links!

UPDATE 2018-06-09 @ 12:21 Additional links from @williamwilloby

Classes

While I don’t think I’ll have time over the next few days to attend a class, VMware has quite a few available: https://mylearn.vmware.com/mgrReg/plan.cfm?plan=48389&ui=www_edu