club-3090

Recipes for serving LLMs locally on RTX 3090s (and other NVIDIA cards): working compose configs, patches and measured numbers, across vLLM, SGLang, llama.cpp, ik_llama and exllamav3. Pick a model for your card count, launch it with one command, and get an OpenAI-compatible API.

📣 The newest slugs, numbers and caveats are posted in Announcements first. The repo docs catch up afterwards, so when a page here and an announcement disagree, the announcement is newer. The card pages below link each slug to its announcement thread.

Pick your path

Every measured result lives in BENCHMARKS.md. Terms like TPS, KV cache and MTP are explained in the GLOSSARY; engines in INFERENCE_ENGINES.md; quant names in QUANTIZATION.md. The full docs index is docs/README.md.

Quick start

Needs Linux (on Windows, set up WSL2 first), Docker with the NVIDIA Container Toolkit, and room on disk for the weights.

git clone https://github.com/noonghunna/club-3090.git && cd club-3090

bash scripts/setup.sh           # pick a model; downloads and SHA-verifies the weights
bash scripts/launch.sh          # pick a config for your GPUs; boots it and runs a quick check

# the launcher prints a test request for what you booted; for the Qwen3.6-27B dual default:
curl -sf http://localhost:8010/v1/chat/completions -H "Content-Type: application/json" \
  -d '{"model":"qwen3.6-27b","messages":[{"role":"user","content":"Capital of France?"}],"max_tokens":200}'

bash scripts/switch.sh --list   # every config this machine can run (--all adds retired ones)
bash scripts/switch.sh    # switch to another one (🧪 slugs need --force)
bash scripts/update.sh          # later: pull the latest stack and re-run setup
bash scripts/settings.sh show   # your saved settings (model dir, default pins, …) and where each comes from

Settings you save (the weights dir, your default config) live in ~/.config/club-3090/, shared by every checkout: where settings live. Upgrading from before v0.12.0? What moved, and how to move yours.

Prefer a screen to the CLI? c3 is a terminal cockpit for the same flow: browse the catalog, serve with one key, watch GPUs and containers, run health checks. Install it with uv pip install -e tools/serve-cockpit and run c3; details in tools/serve-cockpit/.

Other hardware

The configs are hardware-class aware, and contributors have run them on 4090s, 5090s, A-series cards and more: Can I use a 4090? · Can I use a 5090? · HARDWARE.md (card classes, power limits, NVLink, PCIe).

More than chat

Club 3090 AI Studio adds open-weight image, video and audio generation alongside a chat model, driven from Open WebUI. The image bundle is one command: bash scripts/setup-image-studio.sh.

Where things live

models///compose///.yml   one compose per slug
models///patches/                                     vendored fixes for that engine
scripts/          setup, launch, switch, verify, bench, quality, report, update
docs/             the guides linked above
tools/            the c3 cockpit, kv-calc, chart and docs generators
BENCHMARKS.md     every measured row · CHANGELOG.md  what changed, dated

Engines and hardware are documented once, under docs/; everything specific to a model (quants, quirks, recipes) sits under models//.

Community

• 💬 Discord — casual chat, hardware questions, share what you're running. Use for synchronous Q&A.
• 📋 GitHub Discussions — async, searchable. Best for cross-rig benchmark drops, "should I tune X" type threads, and anything you want others to find via search.
• 🐛 GitHub Issues — bug reports, regression repros, concrete asks. Triage ladder in FAQ before filing.

Community projects

Projects in the club-3090 ecosystem maintained outside this repo:

• VykosX/club-3090-server — single-file installer adding a server-management layer on top of club-3090: browser admin panel on :8008/admin, OpenAI-compatible reverse proxy on :8009 with multi-backend routing, GPU-aware multi-instance orchestration, fan/power controls, audit logs, and per-user API auth/quota. Headless Arch + Debian/Ubuntu friendly. Started 2026-05-05, AGPL-3.0; see discussion #108 for the announcement and current WIP status. Not yet officially adopted — listed here as a community pointer until it converges on a stable surface area.

If you've built something that integrates with club-3090 and you'd like a pointer added here, open a discussion.

Credits

The stack stands on a lot of shoulders:

• Qwen team (@Alibaba_Qwen) — for the base models and the MTP head architecture
• Lorbus — for the AutoRound INT4 quant with preserved BF16 mtp.fc (the model this whole stack runs on)
• Sandermage — root-caused vllm#40880 (MTP × TurboQuant cudagraph capture) and shipped the fix that made spec-decode work on consumer Ampere. The patch tree it shipped in is no longer used here, but the diagnosis stands and is upstream at vllm#40914.
• vibhavagarwal5 — TurboQuant landing PR + tracking issue #40069
• vLLM project — the engine + active maintenance
• llama.cpp — the alternative engine path
• Luce z-lab — DFlash N=5 draft model for Qwen3.6-27B
• Intel AutoRound — quantization framework
• @paulp83 — the Blackwell reference rig. First full gate on native FP4 (Qwen3.6-27B NVFP4 TP=2 @262K: 168.0 / 215.7 decode, 91% NIAH at 240K, soak PASS), first NVFP4 validation on the 35B-A3B MoE, first Blackwell DiffusionGemma and Nemotron-75B runs, and the only mixed-architecture numbers anyone has contributed (5090 + 3090 Ti in one box) — a rig class we could not otherwise characterise.
• @henrykrinkle01 — raised the bar on how we report quality, not just what we measure. The Results Card v2 quality table — per-pack dispersion and p50/p95 latency — is his format, adopted wholesale from #770. Also the first cross-rig 8-pack on Qwen3.6-27B and still the only one running both reasoning modes on one rig, one day, with repeats, the finding that cli-40 variance is mode-dependent, and the custom-all-reduce-ON measurement on a working-P2P rig.
• All cross-rig contributors — @ampersandru, @walmis, @3dluvr, and the Reddit / X local-LLM community for benchmark data and bug reports.

License

Apache 2.0. Do what you want with it. If you get better numbers on your rig — open an issue. If you add a new model with working configs — open a PR.