Securing Self-Hosted AI Stacks in 2026: A Defensive Playbook

Every scan of the public internet in 2026 turns up the same picture: thousands of self-hosted inference servers with no authentication, chat UIs holding conversation history and API keys, and GPU boxes running builds with published RCE chains. This guide is for developers who run Ollama, vLLM, Open WebUI, or anything like them on hardware they control. It works through the stack in layers: exposure, patching, authentication, network isolation, and prompt-level defense, with CVE numbers and dates you can check.

The exposure numbers should scare you

In January 2026, researchers at SentinelLABS and Censys published a joint study of publicly reachable Ollama servers: about 175,000 unique hosts across 130 countries and more than 4,000 autonomous systems over a 293-day scanning window, with a persistent backbone of roughly 23,000 hosts generating most of the observed activity. That was not a one-off finding. Cisco Talos had earlier identified more than 1,100 exposed Ollama instances with a single Shodan case study in September 2025, and in April 2026 Censys documented an active campaign sweeping cloud IP ranges for exposed ComfyUI instances, targeting more than 1,000 of them for cryptomining and proxy botnet duty.

None of these machines were hacked. Their owners set OLLAMA_HOST=0.0.0.0 to reach the API from another machine, skipped the firewall step, and moved on. Ollama ships with no authentication of its own, so a routable address is the entire attack. And the consequences stopped being theoretical: in June 2026, Sysdig's threat research team documented an attacker who stole access to a misconfigured, publicly exposed Ollama server and used it not for free tokens but to develop and test an automated offensive hacking tool. An exposed GPU box can be conscripted to attack other people.

Check yourself from outside your own network (a cheap VPS or a phone hotspot works):

curl -s --max-time 5 http://YOUR_PUBLIC_IP:11434/api/tags     # Ollama model list
curl -s --max-time 5 http://YOUR_PUBLIC_IP:8000/v1/models     # vLLM / OpenAI-compatible
curl -s --max-time 5 http://YOUR_PUBLIC_IP:8188/system_stats  # ComfyUI

If any of those return JSON, stop reading and go fix your firewall. The article will still be here.

Your attack surface is bigger than one port

A self-hosted AI stack in 2026 has four kinds of listeners, and each fails differently.

Inference APIs. Ollama on 11434, llama.cpp's llama-server on 8080, vLLM on 8000, Triton Inference Server on its HTTP and gRPC ports. Exposure here means stolen compute, stolen model files, and a pivot point inside your network, because half the value of an inference API to an attacker is that it sits on a machine that can reach your other machines.

Chat UIs. Open WebUI, LibreChat, and AnythingLLM hold conversation history, uploaded documents, RAG indexes, and stored provider API keys. An exposed chat UI is a data breach, not just resource theft.

Media pipelines. ComfyUI listens on 8188 and AUTOMATIC1111's Stable Diffusion WebUI on 7860. Both are routinely started with --listen so a laptop can reach the GPU box, and both have large plugin ecosystems that execute arbitrary code by design. The Pickai campaign in the next section shows exactly how that ends.

Desktop apps with quiet servers. LM Studio, KoboldCpp, and text-generation-webui can all serve an OpenAI-compatible API with a flag or checkbox. They are designed for localhost. Treat any remote-access toggle as a production deployment decision, because that is what it is.

Patch cadence is now a security control

Four incidents from the past year, all in software this audience actually runs:

• vLLM, CVE-2025-62164 (CVSS 8.8, advisory published November 2025). From version 0.10.2, the Completions API accepted user-supplied prompt embeddings and reconstructed them by passing a base64 payload to torch.load(). PyTorch 2.8.0 had disabled sparse tensor integrity checks by default, so a crafted tensor could trigger an out-of-bounds memory write during to_dense(): denial of service for certain, code execution plausibly; fixed in 0.11.1. The lesson is that an innocuous-looking API feature became a deserialization sink through a change in a dependency's defaults.
• NVIDIA Triton chain (August 2025). Wiz Research chained three bugs, CVE-2025-23320 leaking the name of a private shared memory region, then CVE-2025-23319 and CVE-2025-23334 abusing that name for out-of-bounds writes and reads, into full unauthenticated remote code execution against Triton. NVIDIA fixed the chain in release 25.07 in its August 2025 security bulletin.
• Open WebUI, CVE-2025-64496 (CVSS 7.3 per the project advisory, 8.0 per NVD; disclosed November 7, 2025, found by Cato CTRL). With the Direct Connections feature enabled in versions 0.6.34 and earlier, a malicious external model server could push server-sent "execute" events that the frontend executed as code. That means arbitrary JavaScript in the victim's browser, token theft, account takeover, and, chained with the Functions API, code execution on the backend. Fixed in 0.6.35.
• ComfyUI and Pickai (2025). QiAnXin's XLab tracked a C++ backdoor distributed through known ComfyUI vulnerabilities to roughly 700 servers, concentrated in Germany, the US, and China. Payloads were ELF binaries disguised as files like config.json and vim.json, and at one point the malware was served from the compromised website of a commercial AI vendor.

Three habits follow. Watch the GitHub security advisories for every inference server and UI you run. Pin versions so you know what you are running. Schedule upgrades like you schedule backups, because in this ecosystem a three-month-old build is legacy software with a public exploit writeup.

Layer one: turn on the auth your server already has

llama-server ships bearer-token auth. Use the file variant so the key stays out of process listings and shell history:

llama-server -m model.gguf --host 127.0.0.1 --port 8080 \
  --api-key-file /etc/llama/api.key
# or: LLAMA_API_KEY=... llama-server -m model.gguf

vLLM has an equivalent flag, but read the fine print in its own security docs: --api-key authenticates only routes under the /v1, /v2, and /inference prefixes. The /invocations endpoint exposes the same inference capability and is not covered, so the flag alone is not a perimeter. vLLM's documentation says plainly not to rely on it by itself and to deploy behind a reverse proxy:

vllm serve Qwen/Qwen2.5-7B-Instruct --api-key "$(cat /run/secrets/vllm-key)"

Ollama has no native authentication at all. Loopback binding plus a fronting proxy is the only sane configuration for anything beyond a single desktop.

Static keys are fine for one user on one box. They are also one shared secret with no identity, no budget, and no rotation story, which is why the next two layers exist.

Layer two: put identity in front

A reverse proxy with forward authentication, Caddy or nginx paired with Authelia or Authentik, gives you SSO in front of anything that speaks HTTP, including servers with no auth of their own. The pattern is always the same: backends bind to loopback or a private Docker network, the proxy terminates TLS, and every route denies by default until the identity provider says otherwise. Test it from outside, because a forward-auth misconfiguration fails open, not closed.

Among self-hosted chat frontends, Open WebUI is the most complete identity layer: three roles (Admin, User, and Pending, with new signups parked in Pending until approved), groups with additive granular permissions, per-model access controls, and SSO via OIDC or LDAP, with SCIM 2.0 provisioning available on top. One licensing note before you standardize on it: since v0.6.6 (April 2025) it uses a BSD-3-based license with a branding protection clause, and deployments above 50 users in a rolling 30-day period cannot remove the branding without an enterprise license. LibreChat and AnythingLLM both offer multi-user modes with their own authentication if that clause bothers you.

Whatever UI you pick, avoid the classic failure mode: a login page in front of an inference API that happily answers anyone who talks to it directly. The UI's auth protects the UI. The API needs its own layer.

Layer three: a gateway that issues its own keys

Once more than one person or app talks to your models, put an LLM gateway between clients and servers. LiteLLM is the default choice for self-hosters: it exposes a single OpenAI-compatible endpoint, fans out to your Ollama, vLLM, and llama-server backends, and mints virtual keys backed by Postgres. Keys carry budgets per key, team, org, and model, with daily and monthly resets, plus rate limits; requests stop at the cap, and a leaked key gets rotated without touching backend configuration. Portkey AI Gateway plays a similar role with routing, retries, and guardrail hooks if you want an alternative.

Network isolation: boring, cheap, effective

Authentication limits who can use a service. Isolation limits who can even reach it, and it would have prevented essentially all 175,000 exposures in the SentinelLABS dataset.

First, bind to loopback by default. 0.0.0.0 is a deliberate decision, not a convenience setting. Second, when you need remote access, use an overlay network: WireGuard, or a mesh built on it like Tailscale or NetBird, puts every service on a private interface reachable only by enrolled devices, with zero public ports.

Third, know the Docker trap. Publishing a port with -p 3000:8080 bypasses ufw entirely, because Docker writes its own NAT rules and routes traffic through the FORWARD chain while ufw's rules live on INPUT. Your firewall says closed; the internet says open. Either publish to loopback explicitly or put rules in the DOCKER-USER chain, which Docker reserves for exactly this:

services:
  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    ports:
      - "127.0.0.1:3000:8080"   # not "3000:8080", which punches through ufw

Fourth, filter egress on GPU boxes. The Pickai backdoor needed to reach command-and-control servers to matter. An inference server has no business initiating arbitrary outbound connections; allow the model registries you pull from and deny the rest, and a compromise becomes a stranded payload instead of a breach.

Prompt injection does not care about your firewall

The OWASP Top 10 for LLM Applications 2025 puts prompt injection at number one (LLM01), and the rest of the list reads like a self-hoster's postmortem file: sensitive information disclosure (LLM02), improper output handling (LLM05), excessive agency (LLM06), unbounded consumption (LLM10). Everything above hardens the pipe. This class of attack rides inside the data.

Direct injection is a user typing "ignore previous instructions." The variant that should worry you is indirect: instructions hidden in a web page your RAG pipeline scraped, a PDF a user uploaded, a README your coding agent read, or a tool description from an MCP server you installed last week. The moment your model can call tools, a successful injection stops being a weird answer and becomes an action taken with your credentials.

Guardrail tooling helps, layered. NeMo Guardrails, NVIDIA's open-source toolkit, defines programmable rails in its Colang DSL across input, dialog, retrieval, execution, and output stages, so you can reject suspicious prompts before the model sees them and catch policy violations after. Guardrails AI takes a validator approach, enforcing structured output contracts and composable checks on what comes back. Presidio detects and redacts PII before text hits your logs or leaves your network. LLM Guard and Meta's Prompt Guard classifiers are worth evaluating in the same slot.

Be honest about the ceiling: published evasion research keeps beating injection classifiers, so guardrails are filters, not fixes. The durable defenses are architectural. Give tools least privilege, require human confirmation for consequential actions, encode and sanitize model output before anything downstream executes it, keep secrets out of system prompts, and treat every model response as untrusted input to the rest of your system.

Attack your own stack before someone else does

You do not need a red team on payroll. garak, NVIDIA's Apache-2.0 LLM vulnerability scanner, ships dozens of probe modules covering prompt injection, jailbreaks, data leakage, and encoding tricks, and its REST generator can point at any endpoint returning JSON, which covers every server in this article:

python -m pip install -U garak
python -m garak --list_probes
python -m garak --target_type huggingface --target_name gpt2 --probes dan.Dan_11_0

PyRIT, from Microsoft's AI red team, orchestrates multi-turn attack strategies rather than firing single probes, which matters once you have memory, RAG, or agents in the loop. promptfoo folds red teaming into CI: describe your app, generate attacks mapped to the OWASP LLM Top 10, and fail the build when a new model or prompt regresses:

npx promptfoo@latest redteam setup
npx promptfoo@latest redteam run
npx promptfoo@latest redteam report

Run one of these before anything faces another human, and re-run after every model swap or guardrail change. Attackers rescan constantly; you should too.

Three moves this weekend

If you do nothing else, do these, in order.

Move one: kill the exposure. Run the curl checks from outside your network. Bind everything to loopback, put remote access on WireGuard or a mesh VPN, and audit docker ps for published ports that bypassed your firewall.

Move two: authenticate every listener. API keys on llama-server and vLLM, a forward-auth proxy in front of Ollama and the media UIs, and a gateway like LiteLLM the moment a second person or app shows up.

Move three: patch and subscribe. Update vLLM, Triton, Open WebUI, and ComfyUI past the fixes above, then subscribe to their advisories so the next CVE is a Tuesday chore instead of an incident.

Guardrails and red teaming come after, because they protect a stack that is otherwise already locked. Looking ahead, watch three things: agent and MCP tooling is growing attack surface faster than defenses are maturing, guardrail evasion research is outpacing classifiers, and inference servers keep collecting CVEs as security researchers finally give them the attention crypto miners gave them first. The good news is that every fix in this article is free, most take under an hour, and the 175,000 hosts in that scan prove the bar for being a harder target than the next server is very, very low.