AI Model Gateways (CLIProxyAPI)

When managing multiple machines, container clusters, and local development environments that query LLM providers, pointing each client directly to external vendor APIs with individual keys quickly leads to token sprawl, rate-limit collisions, and maintenance headaches.

An AI Model Gateway like CLIProxyAPI acts as an internal proxy that normalizes endpoints, pools credentials, and manages routing rules.

Architecture Pattern

[ Local Dev / Agents / Clusters ]
               |
               v
     [ Internal AI Gateway ]  (CLIProxyAPI)
               |
   +-----------+-----------+
   |                       |
[ Public Providers ]  [ Local LLM Hosts ]
(OpenAI, Anthropic)   (Ollama, vLLM)

Dual-Tier Access Model

To balance security and flexibility across devices:

1. Private Mesh / Local Network (Zero Trust)

Expose the gateway only over internal networks or overlay meshes (e.g. Tailscale / Headscale).
Local agents and in-cluster pods communicate directly using internal cluster DNS or local visitor tunnels (such as FRP STCP on 127.0.0.1:<PORT>).
No public certificates or third-party edge authentication required inside the private perimeter.

2. External Edge with Service Tokens

If external tools (mobile devices, cloud webhooks) must reach the gateway, expose only specific API prefixes (e.g. /v1/*) behind an edge tunnel (such as Cloudflare Tunnels).
Protect public ingress with Service Tokens (e.g. Cloudflare Access service tokens) in addition to gateway bearer authentication.
Keep admin, control plane, and web configuration interfaces strictly restricted to internal networks.

Configuration Templating & Secret Injection via `gomplate`

Many application images (including cli-proxy-api) expect a static config.yaml file containing sensitive credentials (upstream provider API keys, bearer tokens, management passwords) rather than reading all configuration directly from environment variables.

In a GitOps workflow, committing static configuration files containing plain secrets to Git is an anti-pattern. A robust solution is to use a gomplate initContainer:

+------------------------------------------------------------+
| Kubernetes Pod                                             |
|                                                            |
|  [ ConfigMap ]            [ Kubernetes Secret ]            |
|  (config.yaml template    (SOPS/age encrypted in Git,      |
|   with {{ .Env.KEY }})     injected as env vars)           |
|         \                          /                       |
|          v                        v                        |
|     +----------------------------------+                   |
|     | initContainer: render-config     |                   |
|     | (Runs gomplate against env://)   |                   |
|     +----------------------------------+                   |
|                      |                                     |
|                      v                                     |
|              [ emptyDir Volume ]                           |
|             (/config/config.yaml)                          |
|                      |                                     |
|                      v                                     |
|     +----------------------------------+                   |
|     | Main Container: cli-proxy-api    |                   |
|     | (Mounts rendered /config)        |                   |
|     +----------------------------------+                   |
+------------------------------------------------------------+

Why this pattern works so well:

Clean Separation: Non-sensitive configuration layout lives in standard, reviewable ConfigMap templates committed to Git, while secrets live separately in SOPS/age-encrypted files.
Dynamic In-Memory Assembly: Secrets are injected into the initContainer via envFrom: secretRef and rendered to an ephemeral emptyDir volume using gomplate:
```
gomplate -d Env=env:// < /template/config.yaml > /rendered-config/config.yaml
```
No Custom Images: Uses standard lightweight images (like alpine with gomplate) without needing to build custom wrapper container images for your applications.

homelabguide