Introduction

Hello I'm Jimmy. This is a collection of notes and configurations that I've accumulated while setting up my development environment for day-to-day productivity.

I work with both Windows and macOS, and I rely heavily on Linux environments within containers or VMs. This site serves as a central repository for the things I learn while configuring these environments.

I'm making this information public in the hope that others might find it useful. Feel free to browse around, and I hope you find something that helps you in your own setup!

Windows

Windows has evolved into a powerful desktop environment, especially with the introduction of WSL2.

Essential Windows Productivity Tools

  • Powertoys: Utilities like FancyZones for window management, and Command Palette for quick app launching streamline my workflow.
  • Windows Terminal: A modern, customizable terminal that integrates seamlessly with WSL and support multiple profiles, themes, and keyboard shortcuts.
  • Hurl: Hurl is a utility that lets you choose which browser (like Firefox or Edge) to open when you click a link outside your browser.

Useful Resources

WSL2

Installation

Install wsl with the following command:

wsl.exe --install

Install FedoraLinux-42 distribution:

wsl.exe --install FedoraLinux-42

Keep your WSL installation updated by running:

PS E:\clones\homelabguide> wsl.exe --update
Checking for updates.
The most recent version of Windows Subsystem for Linux is already installed.

Configuration

Resource Allocation

Create or modify .wslconfig file to customize WSL2 resource allocation:

cd ~
notepad.exe .wslconfig

Here is a sample configuration:

# Settings apply across all Linux distros running on WSL 2
[wsl2]
memory=4GB
processors=4
swap=4GB
networkingMode=mirrored
dnsTunneling=true
autoProxy=true

[experimental]
autoMemoryReclaim=gradual

I'm not going into details why I configure with these settings, but you can read more about it here.

After making changes to .wslconfig, apply them by shutting down WSL:

wsl.exe --shutdown

Disable windows interop

I experience slowness when typing in the WSL terminal while the interop appendWindowsPath is enabled. You can disable it by configuring /etc/wsl.conf file:

[interop]
appendWindowsPath=false

Read more about it here

Accessing Your WSL Environment

Enter your Fedora Linux environment:

wsl.exe -d FedoraLinux-42

SSH Setup

Generate an SSH key for secure authentication:

ssh-keygen

To share your existing Windows SSH keys with WSL:

wsl -d FedoraLinux-42
cp -rv /mnt/c/Users/{replace_with_your_username}/.ssh/ ~
chmod 600 ~/.ssh/id_ed25519

Make sure to replace {replace_with_your_username} with your actual Windows username.

Key Principles

  • Always think Window and WSL as two separate operating systems.
  • Windows files are mounted inside WSL under /mnt/c, but this is actually a mounted Windows file system, not native Linux storage.
  • To avoid perfomance issues do not work with Windows files from Linux and avoid editing WSL files directly from Windows. If you need to sync files between the two environments, consider using a remote git repository or a dedicated synchronization tool like Mutagen.io, as these methods are more reliable for cross-platform workflows.

Useful Resources

Winget

To install winget, checkout this guide: https://learn.microsoft.com/en-us/windows/package-manager/winget/

Using Windows 10/11 IoT Enterprise LTSC user, without Microsoft Store, Follow this guide instead: https://learn.microsoft.com/en-us/windows/iot/iot-enterprise/deployment/install-winget-windows-iot

Installing apps with winget, First search for what you want:

PS C:\Users\Jimbo> winget search espanso
Name    Id              Version Source
---------------------------------------
Espanso Espanso.Espanso 2.2.3   winget

Then install it:

winget install --id Espanso.Espanso

Powertoys

Powertoys utilities that I find useful:

  1. Command Pallete
  2. FancyZones
  3. ZoomIt

Command Pallete lets you run apps by typing their name - similar to Spotlight on macOS.

FancyZones is help to create window layout. Press Win + Shift + ` to open the FancyZones editor.

I use 4 virtual desktops for different tasks:

  1. Terminal
  2. Code
  3. Browser
  4. Other

Switch between the virtual desktop using Ctrl + Win + Arrow left/right.

To jump to any open window from any desktop, use Command Pallete by typing < before the app name.

This setup is heavily inspired by DHH in the Omakub demo.

Windows Terminal

Configuration

configure, Catppuccin theme please follow the instruction in catppuccin

Change bell notification style to Flash taskbar, I find audible bell notification is annoying. Go to Settings > Defaults > Advanced > Bell notification style > and check Flash taskbar.

Add a new profile for a remote machine

Go to Settings > Open JSON file

in your editor of choice, add a new profile by adding this json object under the profiles.list array

{
    "commandline": "ssh jimbo@devbox",
    "name": "jimbo@devbox",
}

Install Nerdfont

Must install Nerd Fonts for the icons to work.

Syncthing

Install

install syncthing via winget

winget install --id Syncthing.Syncthing

Run syncthing with proxy

set ALL_PROXY=socks5://127.0.0.1:1080
set ALL_PROXY_NO_FALLBACK=1
syncthing serve --no-browser

macOS

For macOS, I often work on a company-issued laptop but still need a Linux-like environment for development. Here's my streamlined setup:

  • Lima-vm: My go-to for running Linux virtual machines on macOS. It's lightweight and supports x86_64 containers with the --rosetta flag, making it easy to run and manage Linux containers locally.
  • Wezterm: I use Wezterm because it reads your SSH config directly, allowing you to open a new tab and SSH into a remote machine instantly - no extra setup required. This makes managing multiple remote sessions fast and seamless.
  • Amethyst: A tiling window manager that helps keep my workspace organized and simple keyboard shortcusts.
  • Maccy: A clipboard manager that boosts productivity.

Lima-vm

Installation

Follow installation instructions for Macos.

Create a new VM with the following command:

limactl start template://docker --rosetta

--rosetta flag is required for working with x86_64 containers, if you don't need it, you can omit it.

Accessing the VM with the following command:

limactl shell docker

Tunnel Access

If you want to browse using a browser that's aware of the network in the lima-vm instance, you can use the following command:

limactl tunnel --socks-port 1080 default

Later in the browser (I'm using firefox), you can set the SOCKS proxy to socks://localhost:1080.

Wezterm

Grab the latest release from here

Here is my lua config file:

local wezterm = require("wezterm")

local config = wezterm.config_builder()

config.color_scheme = "catppuccin-latte"

config.font = wezterm.font("Jetbrains Mono")

config.keys = {
	{
		key = "Enter",
		mods = "ALT",
		action = wezterm.action.DisableDefaultAssignment,
	},
}

-- https://github.com/wez/wezterm/discussions/4728
local is_darwin <const> = wezterm.target_triple:find("darwin") ~= nil

if is_darwin then
	config.font_size = 16.0
else
	config.font_size = 14.0
end

return config

Amethyst

I keep all windows fullscreen using the Fullscreen layout and navigate between them with Ctrl + Cmd + Arrow right/left.

I only use these 2 layouts:

  1. Fullscreen
  2. Two Panes

Layouts

Fullscreen is my go-to most of the time.

Shortcuts

Some apps like system settings work better floating, so I add them to the float list.

The apps that I keep it the float list:

  1. Telegram
  2. Appstore
  3. Finder
  4. Maccy
  5. Logseg

This setup is heavily inspired by DHH in the Omakub demo.

Homelab

This section covers tools available in Linux, especially related to containers, Kubernetes, and self-hosted apps.

Podman

Installation

Install podman

sudo dnf install -y podman

Compatibility with Docker

Install podman-docker

sudo dnf install -y podman-docker

Install single binary docker-compose

curl -SL https://github.com/docker/compose/releases/download/v5.1.0/docker-compose-linux-x86_64 -o /usr/local/bin/docker-compose

Make it executable

sudo chmod +x /usr/local/bin/docker-compose

Enable podman socket

systemctl --user --now enable podman.socket

Find your podman socket address

podman info | rg remote -A2

Set DOCKER_HOST environment variable

export DOCKER_HOST=unix:///run/user/1000/podman/podman.sock

You'll need to set the DOCKER_HOST variable each time you open a new terminal, or just add it to your .bashrc file.

Notes on Docker Compose with WSL

If you're running docker-compose and find an error like netavark (exit code 1): nftables error: "nft" did not return successfully while applying ruleset it' likely because limitation in how nftables operates in WSL.

To fix this, switch podman to use iptables as firewall driver.

Install iptables

sudo dnf install -y iptables

Open or create /etc/containers/containers.conf, and add the following:

[network]
firewall_driver="iptables"

Running Podman in the Background

For containers you want to keep running, podman has systemd integration called Quadlet.

Example for a browserless container:

Create a systemd service file in ~/.config/containers/systemd/browserless.container

[Unit]
Description=Browserless

[Container]
Image=ghcr.io/browserless/chrome:v2.27.0
PublishPort=3000:3000
AutoUpdate=registry
PodmanArgs=--memory=1g --cpus=0.8

[Service]
Restart=always

[Install]
WantedBy=default.target

Reload systemd

systemctl --user daemon-reload

Start and enable the service

systemctl --user start browserless.service
systemctl --user enable browserless.service

Check status

systemctl --user status browserless.service

We have to enable the linger for our user to start the containers without the user being logged in:

loginctl enable-linger $USER

Podman Auto-Update

The AutoUpdate=registry option helps update images automatically, but you still need to run:

podman auto-update

You can set up a cronjob to run this command regularly.

Useful Resources

Multipass: Ubuntu VMs Made Simple

Overview

Multipass is a streamlined solution for creating and managing Ubuntu VMs locally. It offers a "just works" experience that's perfect for:

  • Isolating different project environments
  • Testing with different dependency versions (PHP, Node.js, Go, etc.)
  • DevOps learning and experimentation
  • Container operations with Podman
  • Ansible automation practice

Installation

sudo snap install multipass

Verify installation:

multipass version

Basic VM Operations

Create VM with Custom Resources

multipass launch --name dev-vm -c2 -m4G -d20G "24.10"

-c2: 2 CPU cores -m4G: 4GB RAM -d20G: 20GB disk "24.10": Ubuntu 24.10 (includes latest Podman with Quadlet support)

Create VM with SSH Key

  1. Create cloud-init.yaml:
ssh_authorized_keys:
  - YOUR_PUBLIC_KEY_HERE
  1. Launch with cloud-init:
multipass launch --name dev-vm -c2 -m4G -d20G --cloud-init cloud-init.yaml "24.10"

Connect to VM

Get VM IP address:

multipass list

SSH directly:

ssh ubuntu@VM_IP_ADDRESS

Or use built-in shell:

multipass shell vm1

File Sharing

Mount local directory to VM:

multipass mount /path/to/local/directory vm1:/path/in/vm

VM Management

  • List VMs: multipass list
  • Start VM: multipass start vm1
  • Stop VM: multipass stop vm1
  • Delete VM: multipass delete vm1
  • Permanently remove: multipass purge

Ansible: Automation for Your Lab

Ansible is my go-to tool for automating home lab setup and management. For a clean install, I recommend using uv to manage Python and Ansible dependencies—it keeps your environment tidy and repeatable.

To test your Ansible playbooks, use Multipass to spin up local VM. Set up SSH access for the root user using SSH keys only, and always disable password login for security.

If you want a lightweight web UI to manage and schedule your playbooks, try Semaphore. It’s easy to set up, supports cron-like scheduling, and uses a portable Bolt database.

Managing Python Environments with uv

uv is a modern Python package and environment manager that stands out for several reasons.

  • Speed: uv is significantly faster than traditional tools like pip and virtualenv.
  • Portability: uv is a single binary with no dependencies, making it easy to use across different systems.
  • Container/Docker friendly: It's portability and single-binary design make it ideal for packaging Python environment inside containers, simplifying Docker builds and reducing image size.

Installing uv

curl -LsSf https://astral.sh/uv/install.sh | sh

Initializing Ansible Project with uv

uv init
uv add ansible
uv sync

Centralized Execution with Semaphore UI

In production homelab environments, executing playbooks directly from developer workstations or ad-hoc agent shells introduces drift and risk. Instead, standardise on a central execution plane using Semaphore UI:

  • Single Execution Path: All playbook runs, checks, and applies run consistently from Semaphore against predefined inventories.
  • Strict Verification: Review playbook diffs (--check / dry run) in the UI before executing writes or state changes against bare-metal hosts.
  • Centralized Secrets: Inject credentials (SSH keys, vault passwords, environment variables) directly inside Semaphore environments rather than distributing them across local machines.

Nomad

Install Nomad with hashi-up (Single Node)

hashi-up nomad install \
  --ssh-target-addr 192.168.1.10 \
  --ssh-target-user ubuntu \
  --ssh-target-key ~/.ssh/id_ed25519 \
  --server

Replace the IP address, username, and SSH key path with your environment details.

Enable Podman Task Driver & Raw exec plugin in Nomad

Make sure you have Podman installed. Edit the nomad configuration file (/etc/nomad.d/nomad.hcl) and add the following:

# generated with hashi-up

datacenter = "dc1"
data_dir   = "/opt/nomad"
plugin_dir = "/opt/nomad/data/plugins"

server {
  enabled          = true
  bootstrap_expect = 1
}

plugin "nomad-driver-podman" {
  enabled = true
}

plugin "raw_exec" {
  config {
    enabled = true
  }
}

client {
  enabled = true
}

Restart Nomad to apply the changes.

sudo systemctl restart nomad

Check that the Podman and Raw exec driver is enabled:

nomad node status -self -short | grep Drivers
CSI Drivers     = <none>
Drivers         = exec,java,podman,qemu,raw_exec

Deploy Traefik with Nomad and Enable Nomad Service Discovery

Create a Nomad job file (e.g., traefik.nomad) with the following content:

job "traefik" {
  datacenters = ["dc1"]
  type        = "service"

  group "traefik" {
    count = 1

    network {
      port "http" {
        static = 8080
      }
      port "admin" {
        static = 8081
      }
    }

    service {
      name     = "traefik-http"
      provider = "nomad"
      port     = "http"
    }

    task "server" {
      driver = "podman"

      config {
        image = "docker.io/traefik:v2.11.2"
        ports = ["admin", "http"]
        args = [
          "--api.dashboard=true",
          "--api.insecure=true", # Do not expose to the internet!
          "--entrypoints.web.address=:${NOMAD_PORT_http}",
          "--entrypoints.traefik.address=:${NOMAD_PORT_admin}",
          "--providers.nomad=true",
          "--providers.nomad.endpoint.address=http://${NOMAD_IP_http}:4646",
          "--providers.nomad.exposedByDefault=false"
        ]
      }
    }
  }
}

Deploy the job:

nomad job run traefik.nomad

Deploy a Demo Web Application (with Nomad Service Discovery and Traefik Routing):

Create a Nomad job file (e.g., demo-webapp.nomad) with the following content:

job "demo-webapp" {
  datacenters = ["dc1"]

  group "demo" {
    count = 3

    network {
      port "http" {
        to = -1 # Dynamic port allocation
      }
    }

    service {
      name     = "demo-webapp"
      port     = "http"
      provider = "nomad"
      tags = [
        "traefik.enable=true",
        "traefik.http.routers.demo-webapp-http.rule=Host(`demo-webapp-192-168-1-10.sslip.io`)",
        "traefik.http.routers.demo-webapp-http.tls=false"
      ]

      check {
        type     = "http"
        path     = "/"
        interval = "2s"
        timeout  = "2s"
      }
    }

    task "server" {
      env {
        PORT   = "${NOMAD_PORT_http}"
        NODEIP = "${NOMAD_IP_http}"
      }

      driver = "podman"

      config {
        image = "docker.io/hashicorp/demo-webapp-lb-guide"
        ports = ["http"]
      }
    }
  }
}

Deploy the job:

nomad job run demo-webapp.nomad

Once deployed, Traefik will automatically discover the demo-webapp service via the Nomad provider and route traffic to all running instances. You can test load balancing by sending HTTP requests to the configured hostname and port.

curl http://demo-webapp-192-168-1-10.sslip.io:8080
# Output example:
# Welcome! You are on node 192.168.1.10:20190

Repeat the request several times to see responses from different containers, indicating load balancing is working.

Tilt

Tilt is a handy tool for local development with Kubernetes. To setup a local Kubernetes cluster checkout microk8s - just make sure to enable the host-access and registry addon.

I keep an example Tiltfile project that serves as my cheatsheets whenever I need to spin up a new project. You can find it here.

Useful Resources

Mutagen

Grab the latest binary from here

Make sure you've already set up your SSH config for your remote machine. Here's an example from my ~/.ssh/config:

Host devbox
    HostName 192.168.1.100
    User jimbo

Sync files from remote to local:

mutagen sync create \
    --name=music-localdev ./localdev \
    jimbo@devbox:~/clones/music/localdev --ignore-vcs

Now you can open the local folder with any editor you like.

Port forwarding:

mutagen forward create --name=mysql tcp:localhost:3306 devbox:tcp:localhost:3306

This lets you connect to the remote MySQL server using your local MySQL client.

Using it along with WSL

My quick tips for WSL:

  • Treat WSL and Windows as separate systems
  • Access WSL files from Windows through the /wsl network drive
  • Avoid using the /mnt path in WSL for Windows files - it's slow! Use git or Mutagen instead for better performance

Useful Resources

Kubernetes (mikrok8s)

Prefer ubuntu based host for mikrok8s, you can use lima-vm or wsl2 distro. you can follow the instructions here for installing mikrok8s.

kubectl config

Make sure ~/.kube exists

mkdir -p ~/.kube

Create the config file

microk8s config > ~/.kube/config

Execute cluster-info to verify the connection

kubectl cluster-info

Microk8s addon

Enable host-access addon for convenient way to access the host from inside the cluster.

microk8s enable host-access

Since tilt need a local registry to push the images, you can enable the local registry addon.

microk8s enable registry

later if you want to free some spaces from the registry, you can run the following command:

microk8s disable registry
microk8s disable hostpath-storage:destroy-storage
microk8s enable registry

Production Lessons & Troubleshooting

AppArmor / Container-Env Failure After Reboot

On certain Ubuntu installations or environments where systemd-detect-virt reports a container (for instance, if /.dockerenv is present), systemd skips mounting securityfs at boot. Consequently:

  • apparmor.service fails its assertion.
  • snapd.apparmor.service skips loading snap AppArmor profiles with Inside container environment without internal policy.
  • snap.microk8s.daemon-containerd and other MicroK8s daemons crashloop with missing profile snap.microk8s.microk8s or aa_is_enabled() failed unexpectedly.

Manual Recovery Steps:

If MicroK8s fails to start after a reboot due to this issue:

# 1. Mount securityfs
sudo mount -t securityfs securityfs /sys/kernel/security

# 2. Start AppArmor and parse snap profiles
sudo systemctl start apparmor.service
sudo apparmor_parser -r -W /var/lib/snapd/apparmor/profiles/

# 3. Reset failed state and restart MicroK8s daemons
sudo systemctl reset-failed snap.microk8s.daemon-*
sudo systemctl start snap.microk8s.daemon-containerd \
                     snap.microk8s.daemon-k8s-dqlite \
                     snap.microk8s.daemon-kubelite \
                     snap.microk8s.daemon-cluster-agent

Permanent Fix:

Use a systemd oneshot unit or an Ansible playbook to ensure securityfs is mounted and snap AppArmor profiles are loaded at boot prior to starting snap.microk8s.daemon-*.service.

Storage & Ingress Realities

  • Storage: While hostpath-storage is fine for simple local testing, production workloads needing persistence on dedicated directories (or multi-node portability) should use carefully scoped HostPath volumes, local path provisioners, or external replication (such as Litestream for SQLite).
  • Ingress: In practice, pair MicroK8s with Traefik or an Ingress/Gateway API controller and MetalLB for predictable IP allocation on your local network or Tailscale mesh.

Useful Resources

Flux

Bootstrap flux with existing git repository, image-reflector and image-automation controllers enabled for image update automation.

flux bootstrap git --url=ssh://git@<YOUR_GIT_REPOSITORY_URL> \
    --private-key-file=$HOME/.ssh/id_ed25519 \
    --branch=main \
    --path=clusters/homelab \
    --components-extra image-reflector-controller,image-automation-controller

Upgrading Flux

Update your flux cli into the latest version:

flux install \
    --components-extra="image-reflector-controller,image-automation-controller" \
    --export > ./clusters/homelab/flux-system/gotk-components.yaml

Reconcile Manually

flux reconcile ks flux-system --with-source

Production Lessons & Best Practices

CRD Dependencies & Multi-Stage Reconciliation

When deploying services that depend on custom CustomResourceDefinitions (e.g., Gateway API TCPRoute or HTTPRoute, Traefik IngressRoutes, or cert-manager resources), always ensure the CRD definitions are staged in an earlier Kustomization or dependency tree:

  • Use dependsOn: in Flux Kustomization manifests to enforce that CRDs and operators reconcile before application routes.
  • Avoid mixing CRD installations and their custom resources within the exact same unstaged kustomization to prevent reconciliation errors and race conditions during cluster bootstrap.

Secrets Management in GitOps

Never commit plaintext secrets to your Git repository:

  • SOPS + age: Standard pattern for encrypting secret files in Git. The cluster keeps the age private key in a Kubernetes Secret (sops-age), allowing Flux kustomize-controller to decrypt in-memory.
  • External Secret Stores (Infisical / Vault): When using token-based secret sync, ensure authorization credentials have fallback renewal mechanisms and token rotation procedures to prevent 401/403 checkout and synchronization failures.

Useful Resources

Incus

create incus storage

incus storage create sda-pool dir source=/path/to/storage

create a vm using that storage

incus launch images:ubuntu/24.04/cloud forgejo-runner-01 --vm \
  -c user.user-data="$(cat user-data)" \
  -c limits.cpu=2 \
  -c limits.memory=4GB \
  -s sda-pool

Fnox: Secret Management for Dotfiles & Workstations

Fnox is a lightweight CLI secret manager designed to handle local environment variables and secrets using age encryption. It pairs seamlessly with chezmoi for managing dotfiles without exposing credentials in plaintext repositories.

Why Fnox?

When managing dotfiles across multiple machines (Linux, WSL2, macOS, Windows), you frequently need access to API tokens, private keys, and environment variables. Storing these directly in dotfiles or shell startup scripts (.bashrc, .zshrc) risks accidental leaks to public Git remotes.

Fnox solves this by:

  1. Age-Encrypted Config Files: Configuration files (config.toml) contain secrets encrypted with an age public key. These files are completely safe to commit to version control.
  2. Local-Only Private Keys: Decryption relies on a local age private identity key that remains strictly on the host and is never committed to Git.
  3. In-Flight Secret Injection: Secrets can be injected directly into ephemeral processes (fnox exec -- <cmd>) without writing secrets to persistent shell history or disk.

Directory & File Structure

In a chezmoi-managed dotfiles repository, the encrypted secret configuration is tracked while private keys remain ignored:

FileIn Chezmoi SourceCommitted to Git?Purpose
config.tomlprivate_dot_config/fnox/config.tomlYes (encrypted values)Encrypted secret definitions
Private Key(Ignored / untracked)No (Strictly local-only)Age private identity key for decryption

Basic Workflow

1. Generating an Age Identity

Generate an age private key and obtain the recipient public key:

age-keygen
# Outputs:
# Public key: age1xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

2. Initializing Configuration

Configure your config.toml with the recipient public key:

[encryption]
recipient = "age1xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"

[secrets]
OPENAI_API_KEY = "age-encrypted-data..."
GITHUB_TOKEN = "age-encrypted-data..."

3. Setting and Retrieving Secrets

Add or update a secret:

fnox set OPENAI_API_KEY "sk-..."

Retrieve a single secret:

fnox get OPENAI_API_KEY

Process Secret Injection (fnox exec)

The most secure way to use secrets is injecting them on-demand into child processes rather than exporting them into long-lived shell sessions:

# Injects secrets defined in config.toml into the environment of the command
fnox exec -- my-tool --option

# Target a specific environment configuration file
fnox exec -c ~/.config/fnox/homelab.toml -- ansible-playbook site.yml

Combining with Chezmoi

When configuring tools like coding agents, CLI tools, or shell templates:

  • Keep the dotfile template clean and reference environment variables.
  • Wrap application launches or session commands in fnox exec:
# Example alias in .bashrc:
alias agent="fnox exec -c ~/.config/fnox/agent.toml -- agent-cli"

This ensures credentials remain encrypted at rest and are only decrypted into memory during the execution lifecycle.

Networking & Mesh

Modern homelabs rely heavily on mesh networks, overlay topologies, and secure tunnels to connect physical nodes, VMs, and mobile devices across different locations without opening public router ports.

In this section:

Headscale & Tailscale

Tailscale creates a secure WireGuard mesh between all your machines, phones, and servers. For full privacy and control, Headscale provides an open-source, self-hosted implementation of the Tailscale coordination server.

Why Headscale?

  • Zero Open Ports: Nodes establish direct WireGuard tunnels using DERP relays and NAT traversal (STUN/ICE), working cleanly behind CGNAT.
  • Full Ownership: Control user namespaces, pre-auth keys, ACLs, and node registrations on your own VPS.
  • Custom MagicDNS: Use custom domain names and internal IP addresses (such as 100.64.0.0/10) for your homelab services.

Kubernetes Ingress via Tailscale / Headscale

Instead of exposing services directly to the public internet or managing complex VPN gateways, you can run a dedicated Tailscale ingress proxy pod inside your cluster. The proxy joins your mesh and forwards all received mesh traffic directly to your cluster's ingress controller (such as Traefik or ingress-nginx).

[ Tailscale / Headscale Client ]
               |
               v (Tailnet WireGuard Traffic)
   [ tailscale-ingress Pod ]
               | (Kernel Forwarding via TS_DEST_IP)
               v
     [ Traefik Service IP ]
               |
      +--------+--------+
      |                 |
[ IngressRoute / HTTPRoute / Ingress ]
      |                 |
  [ App A ]         [ App B ]

Architecture & Key Settings

  1. State Persistence in Secrets (TS_KUBE_SECRET): The official Tailscale container automatically persists its node state and WireGuard private key inside a Kubernetes Secret. This ensures the node retains its IP address and registration across pod restarts.
  2. One-Time Authentication (TS_AUTH_ONCE=true): Uses an auth key only on first registration; subsequent boots rely on the saved secret.
  3. Transparent IP Forwarding (TS_DEST_IP): In kernel-networking mode, setting TS_DEST_IP instructs the container to forward all incoming connections to the target IP (the cluster IP of Traefik) without port translation.
  4. Recreate Strategy: Setting strategy: type: Recreate ensures that two pods never attempt to use the same Tailscale machine identity at the same time during updates.

Example Kubernetes Manifests

1. RBAC (ServiceAccount & Secret Permissions)

The proxy pod requires permission to create and read its state secret:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: tailscale-ingress
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: tailscale-ingress
  namespace: default
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["create"]
  - apiGroups: [""]
    resources: ["secrets"]
    resourceNames: ["tailscale-ingress-state"]
    verbs: ["get", "update", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: tailscale-ingress
  namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: tailscale-ingress
subjects:
  - kind: ServiceAccount
    name: tailscale-ingress
    namespace: default

2. Auth Secret

apiVersion: v1
kind: Secret
metadata:
  name: tailscale-ingress-auth
  namespace: default
type: Opaque
stringData:
  TS_AUTHKEY: "tskey-auth-xxxxxx" # Pre-auth key from Tailscale or Headscale

3. Ingress Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: tailscale-ingress
  namespace: default
  labels:
    app: tailscale-ingress
spec:
  replicas: 1
  strategy:
    type: Recreate # Prevents two pods claiming the same mesh identity concurrently
  selector:
    matchLabels:
      app: tailscale-ingress
  template:
    metadata:
      labels:
        app: tailscale-ingress
    spec:
      serviceAccountName: tailscale-ingress
      initContainers:
        # Enable kernel IP forwarding inside the pod network namespace
        - name: sysctler
          image: ghcr.io/tailscale/tailscale:v1.68.0
          command: ["/bin/sh", "-c"]
          args:
            - sysctl -w net.ipv4.ip_forward=1 net.ipv6.conf.all.forwarding=1
          securityContext:
            privileged: true
      containers:
        - name: tailscale
          image: ghcr.io/tailscale/tailscale:v1.68.0
          env:
            # Pre-auth key used on initial registration
            - name: TS_AUTHKEY
              valueFrom:
                secretKeyRef:
                  name: tailscale-ingress-auth
                  key: TS_AUTHKEY
                  optional: true
            - name: TS_AUTH_ONCE
              value: "true"
            # Kubernetes Secret where Tailscale node credentials & state are persisted
            - name: TS_KUBE_SECRET
              value: tailscale-ingress-state
            - name: TS_USERSPACE
              value: "false"
            # Target IP: Replace with your Traefik or Ingress controller ClusterIP
            - name: TS_DEST_IP
              value: "<TRAEFIK_SERVICE_CLUSTER_IP>"
            # Hostname visible on the Tailscale/Headscale network
            - name: TS_HOSTNAME
              value: "k8s-ingress"
            # Optional: Point to a self-hosted Headscale server
            # - name: TS_EXTRA_ARGS
            #   value: "--login-server=https://headscale.example.com --accept-dns=false"
            - name: TS_DEBUG_FIREWALL_MODE
              value: auto
            - name: TS_ENABLE_HEALTH_CHECK
              value: "true"
            - name: TS_LOCAL_ADDR_PORT
              value: "[::]:9002"
          readinessProbe:
            httpGet:
              path: /healthz
              port: 9002
            initialDelaySeconds: 5
            periodSeconds: 5
          securityContext:
            privileged: true
          resources:
            requests:
              cpu: 50m
              memory: 64Mi
            limits:
              cpu: 500m
              memory: 256Mi

FRP & STCP Tunnels

FRP (Fast Reverse Proxy) is a high-performance reverse proxy that can expose internal TCP and UDP services behind NAT or firewalls to the outside world.

STCP (Secret TCP) Pattern

While standard FRP proxies open a public port on the VPS (frps), the STCP (Secret TCP) mode keeps traffic private and encrypted:

[ Homelab Service ] <--- [ frpc (Server) ]
                                | (Encrypted STCP Tunnel)
                         [ frps (Public VPS) ]
                                |
[ Client Machine ]  <--- [ frpc (Visitor) ]

Key Advantages

  1. No Open Public Ports: The public VPS relay does not bind listening ports for the service. Traffic cannot be scanned or accessed by arbitrary internet clients.
  2. Mutual Secret Authentication: Only authorized clients running a matching visitor configuration with the shared secret key can establish a local socket connection to the target service.
  3. P2P Acceleration: FRP can negotiate peer-to-peer connections between client and server when possible, falling back to relaying through the VPS.

Blocky DNS

Blocky is a fast and lightweight DNS proxy and ad-blocker written in Go, designed for home networks and container environments.

Why Blocky for Homelab?

  • Ultra-low Footprint: Uses minimal memory and CPU, making it ideal for low-power edge nodes (e.g. Raspberry Pi).
  • Fast In-Memory / Redis Caching: Reduces upstream DNS lookup latency across the entire tailnet or LAN.
  • Custom Blacklists & Whitelists: Block ad and telemetry domains at the DNS level without needing full browser extensions.
  • Configurable Negative Caching: Fine-grained control over cacheTimeNegative prevents persistent lookup failures when newly added services or transient records are queried.

AI Runtimes & Gateways

Running autonomous agents and AI tooling in a homelab requires careful separation of concerns: unified API multiplexing, context management, and secure exposure without leaking keys or endpoints to the public internet.

In this section:

AI Model Gateways (CLIProxyAPI)

When managing multiple machines, container clusters, and local development environments that query LLM providers, pointing each client directly to external vendor APIs with individual keys quickly leads to token sprawl, rate-limit collisions, and maintenance headaches.

An AI Model Gateway like CLIProxyAPI acts as an internal proxy that normalizes endpoints, pools credentials, and manages routing rules.

Architecture Pattern

[ Local Dev / Agents / Clusters ]
               |
               v
     [ Internal AI Gateway ]  (CLIProxyAPI)
               |
   +-----------+-----------+
   |                       |
[ Public Providers ]  [ Local LLM Hosts ]
(OpenAI, Anthropic)   (Ollama, vLLM)

Dual-Tier Access Model

To balance security and flexibility across devices:

1. Private Mesh / Local Network (Zero Trust)

  • Expose the gateway only over internal networks or overlay meshes (e.g. Tailscale / Headscale).
  • Local agents and in-cluster pods communicate directly using internal cluster DNS or local visitor tunnels (such as FRP STCP on 127.0.0.1:<PORT>).
  • No public certificates or third-party edge authentication required inside the private perimeter.

2. External Edge with Service Tokens

  • If external tools (mobile devices, cloud webhooks) must reach the gateway, expose only specific API prefixes (e.g. /v1/*) behind an edge tunnel (such as Cloudflare Tunnels).
  • Protect public ingress with Service Tokens (e.g. Cloudflare Access service tokens) in addition to gateway bearer authentication.
  • Keep admin, control plane, and web configuration interfaces strictly restricted to internal networks.

Configuration Templating & Secret Injection via gomplate

Many application images (including cli-proxy-api) expect a static config.yaml file containing sensitive credentials (upstream provider API keys, bearer tokens, management passwords) rather than reading all configuration directly from environment variables.

In a GitOps workflow, committing static configuration files containing plain secrets to Git is an anti-pattern. A robust solution is to use a gomplate initContainer:

+------------------------------------------------------------+
| Kubernetes Pod                                             |
|                                                            |
|  [ ConfigMap ]            [ Kubernetes Secret ]            |
|  (config.yaml template    (SOPS/age encrypted in Git,      |
|   with {{ .Env.KEY }})     injected as env vars)           |
|         \                          /                       |
|          v                        v                        |
|     +----------------------------------+                   |
|     | initContainer: render-config     |                   |
|     | (Runs gomplate against env://)   |                   |
|     +----------------------------------+                   |
|                      |                                     |
|                      v                                     |
|              [ emptyDir Volume ]                           |
|             (/config/config.yaml)                          |
|                      |                                     |
|                      v                                     |
|     +----------------------------------+                   |
|     | Main Container: cli-proxy-api    |                   |
|     | (Mounts rendered /config)        |                   |
|     +----------------------------------+                   |
+------------------------------------------------------------+

Why this pattern works so well:

  1. Clean Separation: Non-sensitive configuration layout lives in standard, reviewable ConfigMap templates committed to Git, while secrets live separately in SOPS/age-encrypted files.
  2. Dynamic In-Memory Assembly: Secrets are injected into the initContainer via envFrom: secretRef and rendered to an ephemeral emptyDir volume using gomplate: 
    gomplate -d Env=env:// < /template/config.yaml > /rendered-config/config.yaml
  3. No Custom Images: Uses standard lightweight images (like alpine with gomplate) without needing to build custom wrapper container images for your applications.

Autonomous Agents & Context Management

Running long-lived autonomous agents (such as Hermes Agent) in Kubernetes or container environments introduces unique operational challenges: context bloat, persistent memory, dynamic plugin loading, and non-disruptive backups.

Architecture Pattern

A reliable deployment structure pairs the agent runtime with supporting sidecars and scheduled jobs:

+-------------------------------------------------------------+
| Kubernetes Pod                                              |
|                                                             |
|  +------------------+          +-------------------------+  |
|  | git-sync Sidecar | -------> | Shared Volume           |  |
|  | (Pulls plugins)  | (link)   | - /opt/data/.git-sync   |  |
|  +------------------+          | - /opt/data/plugins     |  |
|                                | - /opt/data/lcm.db      |  |
|  +------------------+          | - /opt/data/outputs/    |  |
|  | Agent Runtime    | <------- +-------------------------+  |
|  | (Hermes + LCM)   |                                       |
|  +------------------+                                       |
+-------------------------------------------------------------+
               |
               v (Scheduled CronJob)
    [ Safe SQLite Online Backup ]

Key Architectural Practices

1. git-sync Plugin Delivery Sidecar

Rather than rebuilding custom container images whenever agent plugins or extensions update:

  • Run a git-sync sidecar container to pull the plugin repository from Git continuously.
  • Path Isolation: Clone the repository into a hidden folder (e.g. /opt/data/.git-sync/my-plugin) and symlink the plugin into /opt/data/plugins/my-plugin. This prevents the agent's plugin discovery loader from picking up duplicate definitions from worktree internals.

2. Lossless Context Management (LCM)

Standard agent loops often discard history or rely on lossy prompt compaction once context limits approach. Structured context managers (like hermes-lcm) address this by:

  • DAG-based History: Storing messages as directed acyclic graphs in an embedded SQLite database.
  • Large Output Externalization: Writing large command outputs and tool responses to disk and inserting pointers/summaries into the prompt, keeping token usage under control.
  • Hybrid Vector Retrieval: Generating embeddings on message nodes for semantic recall across sessions.

3. Non-Locking Online SQLite Backups

SQLite databases used by active agent processes will corrupt or lock if copied with standard cp or file archivers while WAL (Write-Ahead Logging) is active.

  • Use SQLite's online backup API in a scheduled backup job: 
    sqlite3 /opt/data/lcm.db ".backup '/tmp/backup/lcm.db'"
  • Sync both the clean snapshot and externalized output payloads to your backup storage target or offsite bucket.