Self-Hosting AI Agents: Why I Moved Off APIs (and the Stack I Run)

Gartner predicts that by 2028, 33% of enterprise software will ship with agentic AI baked in. Almost all of it will run on rented GPUs, shipping your prompts and your customers' data to a third party on every single call. I moved my agents off APIs last year. Not for ideology. For three reasons I can put numbers on: cost, latency, and data custody. If you build agents, self-hosting at least part of your stack is now a real option, and this is the exact setup I run.

Self-hosting AI agents: the real cost of "pay per token"

The pitch for hosted APIs is seductive: no hardware, no ops, scale to zero. The catch shows up on the bill. Every token your agent reads or writes is metered, and the meter runs whether the task was worth it or not. At prototype volume this is pennies. At production volume, with an agent that loops through tool calls and re-reads context on every step, it is a line item that grows with your success.

Self-hosting flips the economics. A used NVIDIA RTX 3090 runs about $700 on the secondhand market and carries 24 GB of VRAM. Left running 24/7 at full load, it draws roughly 350 watts. Do the math: 0.35 kW times 720 hours a month is 252 kWh, and at a residential rate around $0.15/kWh that is about $38 a month in electricity. One upfront check, one recurring bill smaller than most people's phone plan.

The break-even is simple. If your agents spend more than roughly $100 a month on API inference, the hardware pays for itself inside a year, and after that the marginal cost is $38 a month forever. Stanford's 2025 AI Index put U.S. private AI investment at $109.1 billion in 2024, nearly 12 times China's $9.3 billion. The money is clearly flowing to build. Most of it is being spent renting compute that could run in a closet.

The reason nobody puts on the slide: data custody

Cost gets the meeting. Data custody is why I stayed. Every prompt you send to a hosted API is a copy that leaves your perimeter. For a personal project that is fine. For an agent processing customer emails, contracts, or internal docs, it is a standing exposure. Self-hosting keeps the context on the box you control. No vendor retention policy to read, no subprocessor list to audit, no surprise that your prompts quietly became someone else's training data. The model runs, the context stays, and when the task is done you delete the file. That alone was worth the $700.

The self-hosted stack I actually run

I am not running a datacenter. I am running one GPU, one model server, and an agent loop. Here is the breakdown.

Pick the model by VRAM, not by hype

The model you can run is decided by one number: how much VRAM you have. A 4-bit quantized 8-billion-parameter model needs about 4.7 GB of memory. That fits on a single 24 GB card with room to spare for context. Step up to a 70B model at 4-bit and you need roughly 40 GB, which means either two 3090s or a Mac Studio with an M2 Ultra and its 192 GB of unified memory.

The open models are good enough for most agent work now. Meta shipped Llama 3.1 (8B, 70B, 405B) in July 2024. Alibaba's Qwen2.5 landed in September 2024, and its 7B and 14B variants are quietly excellent for tool-calling agents. Mistral's 7B has been the default local workhorse since late 2023. None of these are GPT-4-class on the hardest reasoning, but for the 90% of agent tasks that are extraction, routing, and structured tool use, they are more than enough.

The runtime: Ollama, and why

I use Ollama. It is an open-source runner that pulls GGUF model files from Hugging Face and serves them behind an OpenAI-compatible API. The entire setup is one command:

ollama run llama3.1

From your agent's point of view, nothing changes. You point the base_url at localhost instead of the hosted endpoint, and the rest of your code stays the same. On a 3090, an 8B model serves around 35 tokens per second, and time-to-first-token is measured in milliseconds instead of the network round-trip you eat on every hosted call. For an agent that makes ten tool calls per task, that latency difference is the gap between a snappy assistant and a patient one.

The glue: an agent loop and MCP

The model is the brain. The agent loop is the behavior. I wire tool access through MCP, the same protocol I broke down when I explained why agents cannot use tools safely without it. MCP gives the agent a checked, schema-defined way to call functions; self-hosting gives it a private place to think. Together they close the loop: the agent reasons on your hardware, calls tools through a verified interface, and never ships your context to a vendor.

Where self-hosting still loses

I am not going to sell you a closet as a replacement for the cloud. Three places where renting still wins.

Frontier reasoning. If your agent hits a problem that needs GPT-4-class or Claude Opus-level thinking, a 70B local model will stall or hallucinate where the frontier model would not. For the hard 10% of calls, keep an API key warm.

You own the uptime. A dead GPU is a dead agent. There is no status page to watch and no SLA to hide behind. You need monitoring, you need a restart policy, and you need to know what "the agent is slow" looks like when it is actually "the VRAM filled up."

Quantization costs quality. Running at 4-bit is what makes local inference possible, but it is not free. Some tasks that an FP16 model nails will drift at Q4. Test on your own data before you trust it on a customer-facing path.

A decision rule for self-hosted vs API agents

Stop treating this as a religion. Use a rule:

• Self-host the high-volume, repetitive, data-sensitive, latency-tolerant work. That is the 90% that is commodity inference.
• Rent the frontier reasoning, the burst you cannot size, and the zero-ops path you do not want to staff.
• Run hybrid by default. Local 8B for the routine loop, API for the calls that actually need the ceiling.

How to start self-hosting your AI agent (3 steps)

1. Get 24 GB of VRAM. A used RTX 3090 at around $700 is the cheapest real entry. A Mac with 64 GB unified memory works too if you already own one.
2. Run one command. ollama run llama3.1, then repoint your agent's base_url to localhost. Your existing code runs unmodified.
3. Wrap tool access in MCP and keep a kill switch. The model is yours; the tools should be too, and you should be one command away from pulling the plug if something misbehaves.

The point is not to reject the cloud. It is to stop defaulting to it. Own the 90% that is commodity inference. Rent the 10% that is frontier. That is the actual engineering call, and it is a lot cheaper than the bill you are currently paying for the privilege of renting your own thoughts back.