07 · Basic AI starter
You just bought a local AI powerhouse. Here's how to use it.
No, really. The 5090's 32GB of VRAM is the single most expensive ingredient in AI right now — and it's already sitting in your new machine. Everything below runs private and offline on this PC — no subscriptions, no API keys, no data leaving your house. This section assumes you've never touched AI tooling — start at the top and work down.
⚖️ Full disclosure — I still pay for online subscriptions, and you probably should too. State-of-the-art models are advancing faster than local hardware can keep up, and their performance is simply far beyond anything that fits in 32GB. Great online LLMs are also cheap now: GLM (z.ai), MiniMax, Moonshot, DeepSeek and the like cost less than you'd think — and I happily use Claude Opus and frontier GPT models too, just not at home. Two things to keep honest: local AI is NOT cheaper — the machine and the electricity are real costs — and the only thing that's genuinely cheaper and better locally is video generation. Learn local AI because it's private, offline, and yours — not because it beats the cloud.
Reality check
Local AI is not free — here's the bill
"It runs on my machine" hides real costs. Before you cancel any subscriptions, count these:
| Cost factor | What it actually runs | Rough reality |
|---|
| Hardware | The machine itself | $3,900+ up front. Amortized over 3 years, that's ~$108/month if AI were its only job |
|---|
| Electricity | ~575W GPU + ~150W system under load | ~0.7 kWh/hour of heavy inference. Rates vary a lot by region — at $0.17/kWh (example only): ~$0.12/hr — roughly $3.60/mo at 1hr/day, ~$29/mo at 8hr/day. Check your own utility rate |
|---|
| Idle & standby | PC left on for on-demand access | Even idling adds a few watts-round-the-clock to the bill |
|---|
| Your time | Setup, model shopping, troubleshooting | Free in dollars, expensive in evenings — that's the real currency of local AI |
|---|
Compare honestly: a cheap online plan (GLM, DeepSeek, MiniMax class) can cost less per month than the electricity alone — with far more capable models. Local makes sense for privacy, offline use, video generation, and the joy of tinkering — not for saving money.
The 60-second concept
What "local AI" actually means
- An AI model is just a big file. ChatGPT runs on a giant file hosted on OpenAI's servers. Local AI means downloading a similar — smaller — file and running it on your GPU instead.
- VRAM is the gate. The model has to fit in GPU memory. Cloud GPUs with 32GB+ rent for dollars per hour; consumer cards mostly ship 8–16GB. Yours has 32GB — workstation territory.
- "Quantized" means compressed. Models circulate in compressed forms (Q4, Q5, Q8…) that trade a little quality for a fraction of the size. That's the practical format for home use.
- Why bother when ChatGPT exists? Privacy (legal docs, code, personal data never leave your machine), no per-token cost, no rate limits, works offline — and it's genuinely fun to tinker with.
The cheat sheet
What fits in 32GB of VRAM
Rough sizes for quantized (Q4) text models, plus the image-model classes:
| Model class | VRAM needed | Fits the 5090? | What it's like |
|---|
| 7–9B · small | ~5–6 GB | ✅ easily | Fast, capable chat and summaries — a great first model |
|---|
| 13–14B · medium | ~8–10 GB | ✅ easily | The speed-vs-smarts sweet spot for daily use |
|---|
| 27–32B · large | ~17–20 GB | ✅ comfortably | Genuinely smart coding & writing assistants — where 32GB shines |
|---|
| 70B · extra large | ~40+ GB | ⚠️ partial offload | Spills into system RAM — runs, but slower |
|---|
| Image · SDXL class | ~7–10 GB | ✅ easily | Full-resolution image generation |
|---|
| Image · Flux class | ~24 GB | ✅ with headroom | Top-tier quality — barely fits 24GB cards; yours has spare |
|---|
Rule of thumb: a 16GB card caps out around the 27–32B class. Your 32GB card runs that class plus a long context window — exactly what coding and document work want.
Step 01 · Tonight
Your first hour: Ollama ↗
Ollama is the lowest-friction way to run AI locally — if you can install an app and type one command, you can do this.
- Install it. Download from ollama.com and run the installer. That's the whole setup — it handles all the model-file plumbing.
- Pull your first model. Open a terminal (PowerShell on Windows) and type the magic incantation below. First run downloads a few GB, then drops you straight into a chat.
- Chat. Ask it things. It's entirely local — unplug your ethernet and it still works.
- Try a bigger one. When the small model feels limiting, jump up a weight class. Everything up to the 32B class fits your card with room to spare.
- Add a nicer UI when ready. The terminal is fine to start; when you want a ChatGPT-style interface, any "Ollama frontend" (OpenWebUI and friends) connects automatically.
ollama run llama3.2 # your first local AI — that's it
Already a developer? Continue (VS Code), Cline (VS Code), and aider (CLI) auto-detect a running Ollama and use it as a free, private coding-assistant backend. For a ChatGPT-style web UI on top, see OpenWebUI. Even Claude Code can drive local models via the ANTHROPIC_BASE_URL env var.
Step 02 · Next weekend
- LM Studio is Ollama with a full GUI — a polished chat interface, zero terminal required.
- Browse models inside the app. Search HuggingFace, see file sizes before downloading, one-click run. Perfect for window-shopping the model zoo.
- Real knobs. Context length, GPU offload layers, samplers, KV-cache quantization — the settings that matter once you know what they do.
- Built-in local server. Exposes an OpenAI-compatible API, so almost any AI app or script can talk to your local models.
- Multi-GPU support — relevant if a second 5090 ever shows up next to the first.
Step 03 · The deep end
vLLM ↗ — when you're serving, not chatting
- vLLM is production-grade inference: continuous batching and PagedAttention squeeze maximum throughput from a GPU.
- Built for serving many requests at once — self-hosting an API for your apps, batch-processing documents, powering a shared assistant.
- Linux-first. Runs on Windows via WSL2, but it's a Linux tool at heart: terminal, config files, and the expectation you know what you're doing. Hence step 03, not step 01.
Side quest · Images & video
ComfyUI ↗: make pictures (and clips)
ComfyUI is the de facto standard for local image and video generation — node-based visual workflows instead of code.
- Node-based workflow: wire boxes together (load model → describe image → generate). No coding, but a real learning curve — budget an evening of tutorials before it clicks.
- The 5090 runs everything current at full resolution: SDXL (the workhorse), Flux (top-tier quality), Z-Image (Alibaba Tongyi), and LTX-Video for open-source video generation.
- Why VRAM matters here too: Flux-class models barely fit a 24GB card. Yours fits them with headroom — bigger batches, higher resolutions, longer clips.
Recap
Which tool, when
| Your goal | Use | Effort |
|---|
| Try local AI for the first time, tonight | Ollama | 10 minutes |
|---|
| A polished chat UI + model browsing | LM Studio | an afternoon |
|---|
| Free private coding assistant | Ollama + Continue / Cline / aider | an afternoon |
|---|
| Self-host an API / max throughput | vLLM | a weekend · Linux |
|---|
| Image & video generation | ComfyUI | a weekend · tutorials |
|---|
The whole stack — every tool above, every model you'd realistically run — costs $0 in software on the machine you already bought. Just keep the ledger honest: hardware and electricity mean local is not the cheap option, and cloud subscriptions still win on raw capability. Where local genuinely wins: privacy, offline access, tinkering — and video generation, which is cheaper and better on your own GPU.
