Cody Bunch Some Random IT Guy - OpenStack, DevOps, Cloud, Things

[Lab] Automated Install of ESXi with

Earlier this year, Jake Robinson tweeted about doing a manual ESXi install, and based on some replies to it I shared that I automate ESXi host installation in my lab using I then promised a writeup and then procrastinated on getting that done. No longer, it is time!


You will need a number of things in order for this to work. Here is a list:

  • A TFTP server to serve ipxe files from
  • Ability to set options on your DHCP server
    • 66 (TFTP Server IP)
    • 67 (Network boot program)
  • An HTTP server to serve
    • & custom menus
    • ESXi installer files
  • Access to ESXi install media

Getting Started

There are a few moving parts here, that we’ll tackle in the following order:

  • Extracting ESXi files
  • Customizing ESXi boot.cfg
  • Customizing ESXi Kickstart
  • Installing
  • Adding a custom menu to

Extracting ESXi files

The first thing we need to do is make the ESXi install files available over http so your hosts can install them. The following steps cover this. It is also important to remember the file names are case sensitive, and some utilities will change them all to upper or lower case when extracting.

  1. Download the ESXi installer ISO.
  2. Mount the image (the steps on how to do this will vary by system).
  3. Copy all of the files to a folder on your webserver. e.g. web/VMware/ESXi/v7.0.0U2/
  4. Ensure the extracted files are lowercase.

Customizing ESXi boot.cfg

Next, make a copy of the boot.cfg file from the extracted ESXi installer. You will need to make the following modifications:

  • Change the prefix= line to point to the location where you extracted the installer to. e.g. prefix=http://boot.codybunch.lab/VMware/ESXi/v7.0.0U2
  • [Optional] If using a kickstart file, add that to the end of the prefix= line. ks=http://boot.codybunch.lab/VMware/ESXi/kickstart.cfg
  • Remove cdromBoot from the kernelopt= line.
    • Depending on your hardware, you may need to add other options here, e.g. allowLegacyCPU=TRUE
  • Remove the forward slash / from the file names in kernel= and modules= lines.
    • kernel=/b.b00 changes to kernel=b.b00
    • modules=/jumpstart.gz --- /useropts.gz ... changes to modules=jumpstart.gz --- useropts.gz ...

[Optional] Customizing ESXi Kickstart

Using a Kickstart file can automate the installation to where it is non interactive. My kickstart file follows. You can find various options for this file in the Resources section.


Installing can be done by following the self hosting instructions in the documentation. Paying special attention to the custom options section here.

In order to boot from, you need to set DHCP options 66 and 67 as discussed earlier. Specifically, set option 66 or next-server to the IP address of your TFTP server, and option 67 or boot-file to the The following diagram shows what this looks like for my Ubiquiti homelab setup:

Adding a custom menu to

Per the ‘Self Hosted Custom Options’ section in the docs, you can add custom menu structures by copying and editing their sample menu:

cp etc/netbootxyz/custom /etc/netbootxyz/custom

The menu file I use is as follows:

There are a few interesting bits I would like to call out:

  • Line 19 uses the sanboot option to stream the ESXi ISO to the host. This allows you to walk the installation as if you had booted directly from the ISO.
  • Line 23 loads the customized boot.cfg we created earlier in the post without loading a kickstart. This allows you to walk the installer by hand.
  • Line 27 loads the kickstart file, and performs the installation for you.
  • The lines (20, 24, 28) that read boot || goto custom_exit tell ipxe to attempt to boot the host, or (||) if that fails, to exit the custom menu and return to the main menu.

Summary allows for a good deal of flexibility when running a home lab. Both for rapidly changing the OS on your physical hosts as well as having installers available for any number of other operating systems and utilities available via network boot. You can also add custom options to install things like ESXi, as was shown in this post.

[Hyper-V] Windows 10 Guest OS Dynamic Disk Compaction

This morning I awoke to the Windows 10 VM I use for work having used almost all of my hard drive. This being both the blessing and the curse of using dynamic, or thin provisioned, disks, it was not unexpected. However, having not managed Hyper-V virtual machines before, compacting the disk was a new process for me. I have now documented it here in case you run into this as well.

Guest OS Operations

To reclaim the most space from a dynamic disk, you will want to perform a few optimizations within the virtual machine first. We do this by using the Windows defrag tool to move all free space to the end of the disk, and then use sdelete from sysinternals to zero out the free space.

  1. Defragment the disk
  2. Run sdelete

Defragment the disk

The Windows defrag tool is intelligent enough to know it is running inside a VM, and will attempt to move all off the free space to the end of the disk, enabling it to be compacted most effectively.

While you can defragment the disk using the GUI disk defragment tool, I find knowing the command line tools to do the same make automation easier down the road.

To defragment a disk in the virtual machine, open an Administrator command prompt or PowerShell, and run the following command:

C:\WINDOWS\system32> defrag C: /U /V /H

The command reads as follows: “Defragment the C: drive while displaying progress (/U) and verbose (/V) messages. Run the defragmentation process at normal priority (/H).”

C: is to specify the drive you would like to operate on. Change this if needed to match the dynamic drive in your VM. /U and /V display progress and verbose logging about what the defrag process is doing, and are optional if you are not running this interactively. /H raises the priority of the defrag process so it will finish quicker, and is also optional.

When finished, you will see output similar to the following:

Output of defragmentation process

Run sdelete

Running sdelete after a disk defrag may seem like an extra step, and if time is critical, it likely is. However, allowing sdelete to zero the free space that was moved to the end of the disk, ensures we will reclaim the most space during compaction.

From an elevated command shell, run the following:

C:\WINDOWS\system32> sdelete -nobanner -z c:

This tells sdelete to not display the startup banner and copyright message(-nobanner), and to zero (-z) the free space of c:. When complete the output will look similar to the following:

Output of sdelete free space cleanup

Once this is complete, shut down the virtual machine.

Host Operations

Since we are focusing on command line ways to do this, we’ll look at using both PowerShell and diskpart.

Compact VHD using PowerShell

To compact the VHD/VHDX using PowerShell, open an elevated shell and run the following commands:

  1. Mount the VHD readonly:
$VHDPath="E:\Path\To\VHD\Windows 10 x64-disk1-cl1.vhdx"
Mount-VHD -Path $VHDPath -ReadOnly
  1. Compact the VHD:
$VHDPath="E:\Path\To\VHD\Windows 10 x64-disk1-cl1.vhdx"
Optimize-VHD -Path $VHDPath -Mode Full
  1. Unmount the VHD:
$VHDPath="E:\Path\To\VHD\Windows 10 x64-disk1-cl1.vhdx"
Dismount-VHD -Path $VHDPath

Note: When these commands run successfully, progress is displayed but there is no output.

Compact VHD using diskpart

To compact the VHD/VHDX using diskpart, open an elevated shell and run the following commands:

  1. Open diskpart
C:\Users\bunchc> diskpart
  1. Select the disk
DISKPART> select vdisk file="E:\Path\To\VHD\Windows 10 x64-disk1-cl1.vhdx"

DiskPart successfully selected the virtual disk file.
  1. Attach disk readonly
DISKPART> attach vdisk readonly

  100 percent completed

DiskPart successfully attached the virtual disk file.
  1. Compact the disk
DISKPART> compact vdisk

  100 percent completed

DiskPart successfully compacted the virtual disk file.
  1. Detach the disk
DISKPART> detach vdisk

DiskPart successfully detached the virtual disk file.


You should now have a much smaller VHD/VHDX file. An interesting note in my case, is that when I used diskpart right after using PowerShell’s Optimize-VHD, I recovered an additional 1GB of disk space.


Coffee? Coffee! - My Coffee Recommendations

At the beginning of this COVID-19 pandemic, I was asked by a good friend for some coffee recommendations. I originally brain dumped all of those into a Twitter thread, but I needed a better spot to find them.

TL;DR: My goto is a portable Aeropress kit, local beans, and brewed following this guide.

This post may get rather long, so for ease of finding your way around, here is how to find the important bits:


One can go crazy when it comes to searching out beans. Like any good rabbit hole, finding “good”, “the best”, and so on quickly becomes an exercise in patience and budget. It is also highly individualized. My recommendations are what I’ve found to be reliably good, at least by my standards.

My primary recommendation for beans, if you are not going to roast your own, is to find and support a local roaster. This was important under normal circumstances and is even more so now. However, the in person approach may be more… complex in the short term.

Ruta Maya

Ruta Maya Coffee Bag

Ruta Maya, is sturdy, reliable, relatively cheap, and consistent. For these reasons, as well as it’s availability, make it a good daily driver coffee. If nothing else it is a wild step up from whatever you have currently sitting in the Maxwell House can you got from your grandfather.

Shotgun House

Shot Gun House Roasters

Shotgun House is local to me (for Texan values of local anyways). The shop is great, but even more so, I have yet to be disappointed in a coffee I’ve brewed from them.

Brown Coffee Co

Brown Coffee logo

My friends, I have saved the best for last. Brown Coffee Company is the coffee I aspire to when I roast my own beans. Like Shotgun House, they are local to me, and the coffee they produce is near perfection.


A gear selection is a bit harder. What follows here is what has become my ‘goto’ kit.

tl;dr - Here is the Amazon list of my goto coffee gear.


Aeropress coffee maker being pressed to produce a cup of coffee

Yes, the Aeropress. It is a coffee maker that punches way above it’s weight class in terms of producing a flavorful cup of coffee. The best part is, that when you roll out of bed at a time that should not have an o’clock associated with it, the Aeropress is simple enough to operate without being entirely awake.

Amazon Smile Link

I brew with it using the method documented here.

Porlex Mini Grinder

Porlex Mini Grinder

There are a number of good ceramic burr grinders out there. Portable, electric, and with costs that will boggle your budget. The Porlex mini grinder has become my goto for two reasons: reliability, and repeatability.

Reliability: I have been using said grinder almost daily for about 12 years. It has yet to fail me.

Repeatability: Not just for me, but for others. When I describe the coarseness of a grind using the ‘clicks’ I mention in my post on how to brew with the Aeropress, the grind itself is consistent over time, and on other Porlex mini grinders.

It also happens to fit almost perfectly inside the Aeropress when going portable.

Amazon Smile Link

Bonavita Kettle

Bonavita 0.5L Kettle

When traveling the 0.5L kettle is the way to go. It produces enough hot water to brew an entire Aeropress, and has a little extra you can use to cleanup afterwards. The 0.5L kettle also fits nicely in the go bag.

When at home I use the 1L version of the same.

Amazon Smile Link

Rothco Mechanics Tool Bag

Rothco Mechanics Tool Bag in olive green

This bag is not the most stylish, but it is sturdy, and just the right size to hold all of your coffee gear along with a bag or two of beans. It fits at the bottom of most laptop or carry on bags as well.

Amazon Smile Link

[WSL2] Mount vhdx to WSL2

Sometimes the best blog posts come from Bad Ideas™. Like this one:

Screenshot of a tweet that reads: "Why is my AppData folder 54GB? o.O Now to find out how badly Win10 breaks if I move it.

While the cleanup from the experimental moving of AppData is the subject of another post, one thing that didn’t come back properly was my default WSL distribution. This post shows how I mounted the ext4.vhdx to a different WSL distribution to export the data.

Mount additional vhdx to WSL2

Note: Do not use this for anything other than data recovery. Even then, there are likely better methods. Why? This method uses FUSE mounts which are s-l-o-w, and in general works against the seamlessness that WSL aims for.

In order to mount a VHD or VHDX file to WSL, you need to first install some tools, and, likely a Kernel.

$ sudo apt update
$ sudo apt install -y libguestfs-tools linux-image-generic

The kernel also needs to be accessible by our user:

$ sudo chmod 0777 /boot/initrd.img-5.4.0-51-generic
$ sudo chmod 0777 /boot/vmlinuz-5.4.0-51-generic

Once your tools are installed, have a look at the partitions and filesystems on the vhdx using virt-filesystems:

$ virt-filesystems -a /mnt/c/oldUbuntu/ext4.vhdx -l
Name      Type        VFS   Label  Size          Parent
/dev/sda  filesystem  ext4  -      274877906944  -   

Cool, as you see from the output, there is one disk /dev/sda with an ext4 filesystem on it. Next, we make a spot to mount it to, and then, mount it using guestmount:

$ cd ~
$ mkdir import
$ guestmount \
  --add /mnt/c/oldUbuntu/ext4.vhdx \
  --ro \
  -i ./import/

The guestmount command above adds (--add) an image (/mnt/c/oldUbuntu/ext4.vhdx), specifies we want it read-only (--ro), and to inspect it (-i) for filesystems to mount to our destination (./import/).

Note: This failed for me the first few times until I had a kernel installed in /boot that was readable by the user.

Now we can check our mounts and confirm that it is indeed mounted, and even browse the filesystem if we’d like:

$ mount
/dev/fuse on /home/bunchc/import type fuse (rw,nosuid,nodev,relatime,user_id=1000,group_id=1000)

$ ls import/
bin   dev  home  lib    lost+found  mnt  proc  run   snap  sys usr
boot  etc  init  lib64  media       opt  root  sbin  srv   tmp  var  


In this post we used guestmount to mount a vhdx into WSL2 for data recovery. While it is possible to do this for mounting and using secondary disk files, in that case you likely just want a Virtual Machine. It will be faster and more stable.


[Kubernetes] Fix: Node Password Rejected...

[Kubernetes] Fix: Node Password Rejected…

Node password rejected, duplicate hostname or contents of...

Judging by the error message, my Kubernetes cluster was having about the same Monday as I was.

What led here was trying to add a node back into the cluster after having rebuilt it.

Some steps…

To fix this, we need to uninstall the k3s-agent from the node in question, remove the local password file on said node, and finally, remove the node entry from the primary node.

On the node

On the node having the issue, we log in and uninstall the k3s agent and remove the local password file. As no workloads have been scheduled to the node, due to the “Node password rejected…” error, this should be non-impacting. The following commands show how to do this:

Uninstall k3s:

$ sudo /usr/local/bin/

Remove local password file:

$ sudo rm -f /etc/rancher/node/password

On the primary node

Log into primary node, and check for the node in /var/lib/rancher/k3s/server/cred/node-passwd:

# cat /var/lib/rancher/k3s/server/cred/node-passwd

# Count the number of entries
# cat /var/lib/rancher/k3s/server/cred/node-passwd | wc -l

Next we use sed to test removing the line:

# sed '/swarm-02/c\' /var/lib/rancher/k3s/server/cred/node-passwd

Looks good, but once more to be sure:

# sed '/swarm-02/c\' /var/lib/rancher/k3s/server/cred/node-passwd | wc -l

Now that we have verified our sed syntax, we can remove the line and restart the k3s service:

# sed -i '/swarm-02/c\' /var/lib/rancher/k3s/server/cred/node-passwd

# sudo systemctl restart k3s

Finally, we reinstall k3s on the node using k3sup from Alex Ellis:

$ k3sup join \
  --ip ${nodeIP} \
  --server-ip ${primaryNodeIP} \
  --user ${k3sUser} \
  --k3s-version "v1.18.9+k3s1"

Note: I tag the specific version of k3s to use as my cluster is running on Raspberry Pi Model 2 B, which are not quite strong enough to stay super current.

Once the installation is finished, you can check that the node has indeed joined the cluster happily:

kubectl get node
NAME       STATUS                     ROLES    AGE     VERSION
swarm-03   Ready                      <none>   72d     v1.18.9+k3s1
swarm-08   Ready                      <none>   72d     v1.18.9+k3s1
swarm-06   Ready                      <none>   72d     v1.18.9+k3s1
swarm-04   Ready                      <none>   72d     v1.18.9+k3s1
swarm-01   Ready,SchedulingDisabled   master   72d     v1.18.9+k3s1
swarm-02   Ready                      <none>   7m12s   v1.18.9+k3s1
swarm-09   Ready                      <none>   72d     v1.18.9+k3s1
swarm-05   Ready                      <none>   72d     v1.18.9+k3s1
swarm-07   Ready                      <none>   68d     v1.18.9+k3s1


You may also need to use the following commands to completely eject the node:

Note: These commands alone did not fix the “Node password rejected…” issue, as the kubectl delete node command did not clear out the node entry in /var/lib/rancher/k3s/server/cred/node-passwd

# Drain the node
$ kubectl drain [nodeName] --force --delete-local-data

# Delete the node
$ kubectl delete node [nodeName]


Visually Identify Raspberry Pi In A Cluster

Visually Identify Raspberry Pi

Raspberry Pi are great! Low power, do not use much space, and so on.

However, I run a cluster of about nine nodes, which makes finding a specific node almost as challenging as finding a random server in a datacenter.

Good news is, you can make one of the onboard LEDs flash to give you an idea of which node it is.

The following instructions assume you have both remote access (SSH) to your Raspberry Pi, as well as sudo or root permissions.

  1. Verify the current kernel module for LED0:

     $ cat /sys/class/leds/led0/trigger
     none rc-feedback kbd-scrolllock kbd-numlock kbd-capslock
     kbd-kanalock kbd-shiftlock  kbd-altgrlock kbd-ctrllock
     kbd-altlock kbd-shiftllock kbd-shiftrlock kbd-ctrlllock
     kbd-ctrlrlock timer oneshot heartbeat backlight gpio cpu
     cpu0 cpu1 cpu2 cpu3 default-on input panic actpwr [mmc0]
     rfkill-any rfkill-none
  2. Enable the ledtrig_heartbeat kernel module:

     $ sudo su -
     # modprobe ledtrig_heartbeat
  3. Set LED0 to blinking:

     # echo "heartbeat" > /sys/class/leds/led0/trigger
     # cat /sys/class/leds/led0/trigger
     none rc-feedback kbd-scrolllock kbd-numlock kbd-capslock
     kbd-kanalock kbd-shiftlock  kbd-altgrlock kbd-ctrllock
     kbd-altlock kbd-shiftllock kbd-shiftrlock kbd-ctrlllock
     kbd-ctrlrlock timer oneshot [heartbeat] backlight gpio cpu
     cpu0 cpu1 cpu2 cpu3 default-on input panic actpwr mmc0
     rfkill-any rfkill-none
  4. Reset back to SD card access:

     # echo "mmc0" > /sys/class/leds/led0/trigger
     # cat /sys/class/leds/led0/trigger
     none rc-feedback kbd-scrolllock kbd-numlock kbd-capslock
     kbd-kanalock kbd-shiftlock  kbd-altgrlock kbd-ctrllock
     kbd-altlock kbd-shiftllock kbd-shiftrlock kbd-ctrlllock
     kbd-ctrlrlock timer oneshot heartbeat backlight gpio cpu
     cpu0 cpu1 cpu2 cpu3 default-on input panic actpwr [mmc0]
     rfkill-any rfkill-none


The status LED (the flashy green one, or LED0) on a Raspberry Pi will flash on SD card access. To identify a specific node, we take advantage of the fact that LED0 is programmable, and set it to heartbeat, which makes it flash on and off in a steady pattern.

You can also configure the red power LED to heartbeat as well by changing led0 to led1 in the preceding commands.

Bonus - Enable and disable blinking with Ansible

The commands discussed before work for a single node, but, honestly, they can be hard to remember when you are in a pinch. Have no fear, the following is an Ansible playbook that will set them to blinking for you:

# Sets led0 on an rPI to heartbeat
# To allow for identification of a node in a cluster of rPI.

- name: Host Provisioning
  hosts: swarm
  become: true
  gather_facts: false
  become_method: sudo
    - vars/provision.yaml
    - name: Modprobe led heartbeat
        name: ledtrig_heartbeat
        state: present
    - name: Set LED0 to blink
      shell: "echo heartbeat > /sys/class/leds/led0/trigger"
        executable: /bin/bash
        - identify_me is defined
        - identify_me
    - name: Set LED0 to normal
      shell: "echo mmc0 > /sys/class/leds/led0/trigger"
        executable: /bin/bash
        - identify_me is not defined

Then run the playbook as follows:

# Enable heartbeat LED across the cluster
$ ansible-playbook -i swarm.yml -e "identify_me=true" identify-rpi.yml

# Disable it across the cluster
$ ansible-playbook -i swarm.yml identify-rpi.yml

# Enable heartbeat LED for a specific node
$ ansible-playbook -i swarm.yml -e "identify_me=true" --limit swarm-02 identify-rpi.yml


Faster Bulk File Copy to Synology (Tar over Netcat)

Faster Bulk File Copy to Synology

The thing about moving a large amount of data around your network, is well, it is going to take some time. Sometimes, lots and lots and lots of time. One of the ways I have overcome this in the past was to use tar and nc to basically ship the files as fast as they could be sent. This gets to be interesting when the Synology NAS does not ship with the nc or Netcat tool by default.

I am speed

Note: This method of bulk file moving does not even pretend to be secure. We are basically dumping raw data onto the network from one node and dumping it back to disk on the other. GOTTA GO FAST


The TL;DR for receiving:

  • SSH into the NAS and become root
  • Run synogear install
  • Move to the receiving folder: cd /volume1/Data/destination/
  • Run /var/packages/DiagnosisTool/target/tool/ncat -l -p 7000 | tar -xpf -

The TL;DR for sending:

  • SSH into the NAS and become root
  • Run synogear install
  • Move to the sending folder: cd /volume1/Data/destination/
  • Run tar -cf - * | /var/packages/DiagnosisTool/target/tool/ncat sendinghost 7000

Installing Ncat on Synology

The Synology NAS does not ship with the Netcat installed, so we need to get that installed first. Thankfully, Synology provides a set of “Diagnostic” tools (synogear) that come with ncat, which is an improved version of the standard nc. To install the synogear utilities from Synology. First connect to your NAS over SSH then, as root check to see if the tools are installed:

# synogear list

If the tools have not been installed before you will see the following output:

Tools are not installed yet. You can run this command to install it:
   synogear install

If the synogear tools have been installed before, but this is your first time loading them this session you will see the following:

Tools are installed but not loaded yet. You can run this command to load it:
   synogear install

Next is to load or install the tools and check again:

# synogear install
# synogear list
All tools:
addr2line              eu-readelf    ld               pkill             strings
addr2name              eu-size       ld.bfd           pmap              strip
ar                     eu-stack      ldd              ps                sysctl
arping                 eu-strings  pstree            sysstat
as                     eu-strip      lsof             pwdx              tcpdump_wrapper
autojump               eu-unstrip    ltrace           ranlib            tcpspray
capsh                  file          mpstat           rarpd             tcpspray6
c++filt                fio           name2addr        rdisc             tcptraceroute6
cifsiostat     ncat             rdisc6            telnet
clockdiff              free          ndisc6           readelf           tload
dig                    gcore         nethogs          rltraceroute6     tmux         gdb           nm               sa1               top
elfedit                gdbserver     nmap             sa2               tracepath
eu-addr2line           getcap        nping            sadc              traceroute6
eu-ar                  getpcaps      nslookup         sadf              tracert6
eu-elfcmp              gprof         objcopy          sar               uptime
eu-elfcompress         iftop         objdump          setcap            vmstat
eu-elflint             iostat       w
eu-findtextrel         iotop         pgrep            size              watch
eu-make-debug-archive  iperf         pidof            slabtop           zblacklist
eu-nm                  iperf3        pidstat          sockstat          zmap
eu-objdump             kill          ping     ztee
eu-ranlib              killall       ping6            strace

Sending and Receiving using tar and ncat (ncat, etc)

Unix like systems allow us to pipeline the output of one command to the input of another, which is what makes this possible. We take the output of the tar archiving utility and push it into netcat to move over the network.

Note: The following commands assume you have logged into your NAS over SSH and have loaded the synogear tools.


The command for sending looks like this:

tar -cf - * | /var/packages/DiagnosisTool/target/tool/ncat destinationHost 7000

Which breaks down like this:

  • Have tar create (-c) an archive file (f) to stdout (-) of all files (*)
  • Send the output to the ncat utility
  • Have ncat send the data to the remote host (destinationHost) over port 7000

Note: If successful this command does not produce output. You can add -v to the tar command to get some feedback. However, doing so produces a lot of noise and is not a great indicator of progress.


The command for receiving looks like this:

/var/packages/DiagnosisTool/target/tool/ncat -l -p 7000 | tar -xpf -

Which breaks down like this:

  • Run the ncat utility, telling it to listen (-l) on port 7000 (-p 7000)
  • Take whatever output comes over port 7000 and pass it to the tar program
  • Have tar extract (-x) what it receives, preserve (-p) permissions, from the ‘file’ (-f) stdin (-).

Note: If successful this command does not produce output. You can add -v to the tar command to get some feedback. However, doing so produces a lot of noise and is not a great indicator of progress.


One of the things I dislike about this method is there is not a good way to see progress. Depending on the source / destination you may be able to use the pv or Pipe Viewer utility. Like this:


tar -cf - * | pv | nc destinationHost 7000


ncat -l -p 7000 | pv | tar -xpf -

Both cases will produce output that indicates how much data has been sent, the duration, and current speed:

ncat -l -p 7000 | pv | tar -xpf -
19.08 MB 0:00:04 5.15 MB/s] [<=>                                                                        ]


Fixing cgroup memory on Raspbian Buster for Kernel 5.4.x

After running sudo apt -y full-upgrade my Raspberry Pi k3s cluster got upgrade from 4.x kernels to 5.x. Which turbo-broke k3s.

The Frror

After rebooting into the new kernel and k3s not working, I found the following error in both /var/log/syslog/ and journalctl -xeu k3s (or for the worker nodes journalctl -xeu k3s-agent)

level=fatal msg="failed to find memory cgroup, you may need to add \"cgroup_memory=1 cgroup_enable=memory\" to your linux cmdline (/boot/cmdline.txt on a Raspberry 

However, those values were totally in /boot/cmdline.txt:

$ cat /boot/cmdline.txt
console=serial0,115200 console=tty1 root=PARTUUID=dd5ac5d2-02 rootfstype=ext4 elevator=deadline rootwait cgroup_enable=1 cgroup_memory=1 swapaccount=1 cgroup_enable=memory dwc_otg.lpm_enable=0 

What could it be then?

The Fix

This fix, of course, is to update the firmware on the pi via rpi-update

At the time of this writing, the fix was released 4 days ago according to this Github issue.

While you can run rpi-update by itself and answer its warning prompt, that can get tedious for more than one or two nodes. Since I was upgrading nine nodes, something more robust was needed. Something like the following command:


Or for those using Ansible to manage their fleet:

ansible -i swarm.yml nodes --become -m shell -a 'sudo PRUNE_MODULES=1 RPI_REBOOT=1 SKIP_WARNING=1 rpi-update'

Note: You should probably do at least one node the manual way to ensure the update works. The commands shown here can… accelerate your journey to serverless.

Random PowerShell/PowerCLI Snippets

Cleaning out a few browser tabs, which of course means putting things where I will remember them. This time, that is a bunch of random PowerShell and PowerCLI I’ve been using.

The Snippets

The following sections will state if the snippet is either [PowerShell] or [PowerCLI] and include a link to the site where I found said snippet. The PowerCLI snippets assume a connection to either vCenter or ESXi.

[PowerCLI] Get IP Address of VM

(Get-VM -Name test01).Guest.IPAddress (@vSaiyanman)

[PowerShell] Create an Empty File (touch)

# New file
New-Item -ItemType file example.txt

# Update timestamp on a file
(gci example.txt).LastWriteTime = Get-Date

[PowerCLI] Rescan All HBAs

# Rescans all HBAs on all hosts
Get-VMHost | Get-VMHostStorage -RescanAllHba

# Rescans all HBAs in a given cluster
Get-Cluster -Name "ProVMware" | Get-VMHost | Get-VMHostStorage -RescanAllHba

Note: The fact that I wrote this post about a decade ago still blows my mind.

[PowerShell] Measure Execution Time (time 'someCommand')

Measure-Command { Get-VMHost | GetVMHostStorage -RescanAllHba }

Microsoft Docs

[PowerShell] Windows Event Logs

Get-EventLog -LogName Security -EntryType Error -Newest 5

Microsoft Docs

[PowerShell] Disable Windows Defender Firewall

Set-NetFirewallProfile -Profile Domain,Public,Private -Enabled False

[PowerShell] List / Add Windows Defender Firewall Rules

# List the rules. This produces A LOT of output

# Disables outbound telnet
New-NetFirewallRule -DisplayName "Block Outbound Telnet" -Direction Outbound -Program %SystemRoot%\System32\tlntsvr.exe Protocol TCP LocalPort 23 -Action Block PolicyStore\gpo_name

Microsoft Docs

Getting Started with Ansible and vSphere

My work has started to shift around some recently and I find myself returning to VMware products after a hiatus. In addition to running into a blog post I wrote a decade ago, it has been interesting to see how else VMware has kept up.

In particular, I have spent A LOT of time automating various things with Ansible, so discovering the extensive list of VMware modules. This post shows how to use the Ansible VMware modules to launch your first VM.

Getting Started

There are quite a few prerequisites to getting this going. Buckle up!

First, the requirements for your ‘Control Node’, where you will run Ansible from:

  • Ansible
  • Python
  • Pyvmomi
  • vSphere Automation Python SDK
  • Access to vSphere

Installing the prerequisites on Ubuntu 18.04

Note: While you can likely install these tools directly on Windows, I’ve found it easier to use Ubuntu by way of WSL2.

  1. Install Ansible:
root@9a45e927fc78:/# apt update -q
root@9a45e927fc78:/# apt install -y software-properties-common
root@9a45e927fc78:/# add-apt-repository --yes --update ppa:ansible/ansible
root@9a45e927fc78:/# apt install -y ansible
  1. Install / update Python
root@9a45e927fc78:/# apt install -y python-minimal python-pip
  1. Install Pyvmomi
root@9a45e927fc78:/# pip install pyvmomi
  1. Install the VMware Python SDK
root@9a45e927fc78:/# pip install --upgrade git+

Creating a VM with Ansible

With everything we need installed, we can now make the VM with Ansible. We do that by creating an Ansible playbook, and then running it. An Ansible playbook is a YAML file that describes what you would like done. To create yours, open your favorite text editor and paste the following YAML into it.

# new-vm-playbook.yml
- name: Create New VM Playbook
  hosts: localhost
  gather_facts: no
  - name: Clone the template
      hostname: "vcenter.codybunch.lab"
      username: "administrator@codybunch.lab"
      password: "ultra-Secret_P@%%w0rd"
      resource_pool: "Resource Pool to create VM in"
      datacenter: "Name of the Datacenter to create the VM in"
      folder: "/vm"
      cluster: "MyCluster"
          - name: "Network 1"
          num_cpus: 2
          memory_mb: 4096
      validate_certs: False
      name: "new-vm-from-ansible"
      template: "centos-7-vsphere"
      datastore: "iscsi-datastore"
      state: poweredon
      wait_for_ip_address: yes
    register: new_vm

The great thing about describing the changes in this way, is that the file itself is rather readable. That said, we should go over what running this playbook will actually do.

The playbook tells Ansible to execute on the localhost (hosts: localhost). You can change this to be any host that can access vCenter and has pyVmomi installed. It then tells Ansible we would like to build a vm (vmware_guest) and supplies the details to make that happen. Make note of the folder: "/vm" setting which will place the VM at the root of the datacenter. As with the other variables, change this to fit your environment.

Once you have adjusted the playbook to suit your environment, it can be run as follows:

ansible-playbook new-vm-playbook.yml