The Best Self-Hosted AI Stack for Small Teams in 2026

Your team wants ChatGPT-grade tooling without sending company data to a third party. This is a reference architecture for a self-hosted AI stack that a team of three to fifteen people can actually run in 2026: one GPU box, an inference server, a chat UI, a RAG layer, single sign-on, and tracing, with honest alternatives at every layer. Everything here is open source or close to it, and everything speaks the OpenAI-compatible API, so each piece stays replaceable.

The reference architecture at a glance

The stack is seven components, and five of them are containers on one machine:

teammates --> Caddy or Traefik (TLS) --> Authentik (SSO, forward auth)
                     |
               Open WebUI (chat, workspaces, RAG)
                     |
               LiteLLM (virtual keys, budgets, fallbacks)
                /             \
         vLLM (LLM)       TEI (embeddings)
                     |
               Qdrant (vectors)      Langfuse (traces)

The picks, compressed: vLLM serves the model, Open WebUI is the interface, LiteLLM sits between them as the gateway, Qdrant holds vectors, TEI serves the embedding model, Authentik handles login, and Langfuse records every request. None of these is exotic. That is deliberate: the layer where you should be adventurous is the model, not the plumbing. Each section below explains the pick and names the alternative worth considering.

One box beats a small cluster

The biggest mistake small teams make is designing for imaginary scale. Ten people generate bursty, low-concurrency traffic: a few simultaneous chats, an editor plugin, a nightly batch job. A single modern inference server with continuous batching absorbs that without noticing. A cluster adds network storage, service discovery, GPU scheduling, and three new ways to be down on a Monday morning.

What to put in the box, by budget:

• 24 GB VRAM (used RTX 3090 or 4090 class). Enough for gpt-oss-20b with room for context, or a quantized Qwen3-30B-A3B. The right entry point if you are not yet sure the team will adopt the stack.
• 32 GB VRAM (RTX 5090). The awkward truth of mid-2026: the card lists at $1,999 but the memory shortage has kept real street prices for board partner cards well above that number. Worth it near list price, painful otherwise.
• 96 GB VRAM (RTX PRO 6000 Blackwell). The cheapest single card that runs 70B-class models at FP8, and it fits gpt-oss-120b, a model sized to run on a single 80 GB GPU. It shares the 5090's 1,792 GB/s memory bandwidth on the same GB202 silicon, adds ECC memory, and holds sustained load better. Expect workstation pricing, several times a 5090 at list. This is the card the reference build assumes.

Round the machine out with 64 to 128 GB of system RAM, a 2 TB NVMe drive for weights, and plain Docker Compose. Skip Kubernetes: for one box it buys you nothing but YAML.

When you genuinely outgrow one machine, or you inherit a drawer of mismatched GPUs, GPUStack (Apache 2.0) is the sane middle ground. It manages a mixed GPU cluster and schedules vLLM, SGLang, and TensorRT-LLM workers for you without dragging you into a full MLOps platform.

Inference: vLLM by default

vLLM is the boring, correct choice, and boring is what you want in the layer everything else depends on. It is Apache 2.0, actively shipped (the v0.27 line landed in August 2026), and its PagedAttention memory management plus continuous batching mean one GPU serves the whole team concurrently instead of queueing requests one at a time. It exposes an OpenAI-compatible endpoint, so every UI and library in this article connects to it without adapters.

docker run --gpus all --ipc=host -p 8000:8000 \
  vllm/vllm-openai:latest \
  --model openai/gpt-oss-20b \
  --max-model-len 32768

Weaknesses, honestly: it is NVIDIA-first (ROCm support exists but is a rougher road), the flag surface is enormous, and it wants VRAM headroom for KV cache that a maxed-out consumer card cannot always spare. If a model barely fits, serving it well is a different question from loading it.

The alternatives:

• SGLang is vLLM's real rival, not a niche option. Public benchmarks have it ahead of vLLM in raw throughput on small and mid-size dense models, and the gap grows to multiples on prefix-heavy traffic (agents, RAG, long multi-turn chats) where its RadixAttention prefix cache shines, while shrinking toward zero on unique-prompt batch work. If your workload is mostly agents hammering shared system prompts, pick SGLang and do not look back.
• llama.cpp (MIT, over 125,000 GitHub stars as of August 2026) is the portability king: CUDA, Metal, ROCm, and Vulkan backends, GGUF quantization from 1.5-bit to 8-bit, and a built-in llama-server with OpenAI-compatible /v1 endpoints. It is the answer for Macs, CPU-only boxes, and odd hardware. It is not the answer for ten concurrent users on one big GPU.
• Ollama (MIT) wraps local inference in the friendliest model management anywhere, and it is exactly right for week one and for individual dev laptops. But it is tuned as a personal tool first, convenience over throughput, and its defaults are not built for a shared server under concurrent load. Start here, then graduate to vLLM when the second teammate shows up.

The models to pin, August 2026 edition

Model churn is constant. The architecture should not care. Pick two or three models, pin exact versions in config, and change them deliberately after evaluation, not because a leaderboard moved.

• gpt-oss-20b (Apache 2.0) is the default general model. OpenAI's open-weight release ships in MXFP4 quantization and runs in about 16 GB of memory, which is exactly why the 24 GB hardware tier works with context to spare.
• gpt-oss-120b steps up reasoning quality substantially and was sized for a single 80 GB GPU; on a 96 GB card it runs comfortably. If you bought the RTX PRO 6000, this is why.
• Qwen3-30B-A3B-Instruct-2507 (Apache 2.0) is the efficiency play: a mixture-of-experts model with 30.5B total parameters but only 3.3B active per token, and 262K native context. It answers like a much larger model while decoding at small-model speed, and the Qwen3-Coder-30B-A3B variant is the natural pair for editor integrations.
• Gemma 3 27B adds image understanding if you need multimodal, but it ships under Google's Gemma terms rather than an OSI-approved license; read them before building a product on it.
• For retrieval, Qwen3-Embedding (0.6B, 4B, and 8B sizes, Apache 2.0) is the current open default; the 8B model took the top MTEB multilingual score (70.58) when it landed, and the 0.6B is the one you actually serve on a busy box.

Chat UI: Open WebUI, with eyes open

Open WebUI wins on completeness. The v0.11 release from late July 2026 reorganized the whole interface, and underneath it you get multi-user accounts with role-based access, per-model permissions, shared workspaces, document collections with built-in RAG, tool calling, and connections to anything OpenAI-compatible. It is the closest thing self-hosting has to a default answer.

Know the license before you standardize on it. Since v0.6.6 in April 2025 it is BSD-3 based with an added branding clause: deployments over 50 users must keep Open WebUI branding unless they hold an enterprise license, while deployments of 50 or fewer may rebrand freely. For a small team this changes nothing in practice, but it is not plain BSD, and your future self should know that before the company grows.

LibreChat is the runner-up and becomes the pick when SSO is a day-one requirement: LDAP and OIDC ship natively, and v0.8.7 (June 2026) includes agents, MCP support, code interpreter, and per-user model permissions. The cost is heavier plumbing, MongoDB for history plus Meilisearch for search, and a config file that rewards patience.

AnythingLLM trades flexibility for approachability: workspace-scoped RAG in a single package that nontechnical teammates understand in five minutes. If your team is mostly not developers, it is a legitimate pick over both of the above.

RAG: built-in first, a real engine when it hurts

Do not start with a dedicated RAG platform. Open WebUI's knowledge collections handle the first few hundred documents fine, and you will learn what your team actually asks before committing to heavier machinery. The upgrade trigger is specific and you will recognize it: scanned PDFs and tables coming back mangled, teammates asking where an answer came from, or documents that must respect permission boundaries.

Then choose by problem, not by feature list:

• RAGFlow if the problem is the documents. Its layout-aware parsing handles tables, scans, and complex PDFs better than most open alternatives, retrieval is hybrid with reranking, and answers cite back to the exact region of the source page. Development pace is aggressive: the August 19, 2026 v0.27 release added an agentic retrieval framework and dataset-level knowledge compilation into wiki, graph, and timeline views. The price is operational weight; it is a multi-container deployment that amounts to its own small platform.
• Onyx if the problem is scattered knowledge. The MIT-licensed community edition connects to more than 70 sources (Slack, Google Drive, Confluence, GitHub, Salesforce), and OIDC and SAML sign-in work on the standard images. Several connectors can mirror per-document permissions from the source tool into retrieval; check the plan matrix for which governance features sit behind paid tiers before promising them to anyone. It is the honest answer when the real request is "search everything we have", not "chat with this folder".

Building a custom pipeline instead, for an internal product rather than a chat tool? Three pieces. Docling converts PDFs and office documents into clean structured text. TEI, Hugging Face's Rust-based embedding server, serves the embedding model with dynamic batching and supports the Qwen3-Embedding and EmbeddingGemma architectures among many others. Qdrant (Apache 2.0, on its v1.19 line as of August 2026) stores the vectors and, critically, runs dense, sparse, and filtered retrieval in a single query, which is what production RAG actually needs. If Postgres already runs in your stack, pgvector is the lower-ops call until you need hybrid search at scale, and many teams never do.

The gateway: LiteLLM turns a server into a platform

A bare vLLM endpoint is a shared secret and a prayer. Putting the LiteLLM proxy in front of it turns it into a platform: a virtual API key per teammate, spend tracking per key, rate limits, and fallback routing, all from one YAML file. The project passed 50,000 GitHub stars by mid-2026, and it has become the default way self-hosters expose models to their own tools.

model_list:
  - model_name: team-default
    litellm_params:
      model: openai/gpt-oss-20b
      api_base: http://vllm:8000/v1
      api_key: none
  - model_name: escalation
    litellm_params:
      model: anthropic/claude-sonnet-4-5
router_settings:
  fallbacks:
    - team-default: ["escalation"]

The fallback pattern is the killer feature for small teams: local first for privacy and cost, cloud overflow for the rare job that needs a frontier model or for the afternoon the GPU box is down. Your tools all point at one URL and never know the difference.

Weaknesses: the configuration surface is vast, releases are frequent, and upgrades occasionally move things. Pin the Docker tag and read release notes before bumping.

Auth: SSO once, forward auth for everything else

vLLM and TEI have no real authentication story of their own. Never expose them beyond the Docker network. The front door is a reverse proxy, Caddy or Traefik, terminating TLS, with Authentik behind it as the identity provider. Authentik runs in 2 CPU cores and 2 GB of RAM, speaks OIDC, SAML, and LDAP, and gives Open WebUI or LibreChat proper single sign-on. Its forward auth mode is the underrated part: it puts a login page in front of services that ship without one, which in this stack means Qdrant's dashboard and anything else you would rather not firewall-and-hope.

Authelia is the lighter alternative if Authentik feels like too much identity provider; Keycloak is the heavier enterprise one. And none of this has to be your first move: a team of three can skip public exposure entirely and reach the box over WireGuard or Tailscale, which is both simpler and safer than a half-configured public deployment.

Traces from day one: Langfuse

You cannot tune what you cannot see, and with LLMs you cannot even bill what you cannot see. Langfuse is the observability pick: an MIT-licensed core (with a small enterprise-licensed folder, the same pattern Onyx uses), self-hosted with Docker Compose, and a v3 architecture that moved trace analytics onto ClickHouse so the dashboard stays fast past your first million spans. The repository passed 33,000 GitHub stars in 2026, and self-hosting is a first-class deployment path, not a demo tier.

The integration cost is nearly zero because LiteLLM ships a native Langfuse callback: set two environment variables on the gateway and every request from every teammate lands as a trace with model, latency, token counts, and cost attached. When you later tune the RAG pipeline, Langfuse's datasets and evaluation features are the difference between vibes and measurements.

Choosing your version of this stack

Three honest configurations:

1. The trial, 1 to 3 people, one 24 GB card: Ollama plus Open WebUI, no gateway, no SSO, Tailscale for access. An afternoon of work, and it proves whether the team will actually use the thing before you spend GPU money.
2. The reference, 3 to 15 people, 32 to 96 GB: vLLM, TEI, LiteLLM, Open WebUI, Qdrant, Authentik, and Langfuse under one Docker Compose file on one box. A weekend to stand up properly, an hour a week to run.
3. The knowledge team: the reference, with Onyx replacing built-in RAG (or RAGFlow if your pain is document parsing rather than connectors), and SSO from day one, because connector-based retrieval without real permissions is a data leak with a search bar.

Through the rest of 2026, watch three things. vLLM and SGLang keep converging on each other's features, so the inference choice matters a little less every quarter. Small-activation MoE models like the Qwen3 30B-A3B line keep making 24 GB cards punch above their weight. And GPU street pricing remains hostage to the memory shortage, which means the used 3090 market stays the best value in self-hosting. The architecture above survives all three trends, and that is the point: boring interfaces, pinned versions, and every component individually replaceable.