Is hosting an open-source inference engine profitable?.

TL;DR: H200 loses money even when fully busy. B300 makes money once it's 35% busy. Which GPU card you rent matters more than which inference engine you run.

A common claim on X and LinkedIn is that inference is an easy business: demand is so high that anyone who can rent GPUs can sell tokens at a profit, with no technology of their own. This post checks that claim against public data.

Two views of the same market

Let's be fair about the technology first. Open-source inference engines are mature now. vLLM and SGLang are production-grade, and DeepSeek just open-sourced DeepInfer, the engine behind its own API (DeepInfer deployment Q&A). The vLLM team reproduced DeepInfer's decode throughput within 6% on the same hardware (vLLM, Oct 23, 2026). And even so, I know several companies that hand-wrote their own engines and got excellent results. Friends at inference providers confirm it: a hand-rolled engine can reach that level too. So the engine is a solved problem, whichever way you get one.

Then there's the other camp: it's a price war, margins are 10% at best and often negative. They point to OpenRouter, where 29 providers sell the same DeepSeek V4.1 Flash and the cheapest lists input at $0.056 per million against ~$0.30 for most, below cost by any math (Jeff Xiong, Oct 23, 2026; OpenRouter provider listing). OpenRouter's own data says the quiet part: V4's token volume roughly doubled in the first half of 2026 while its share of spend barely moved (OpenRouter Insights, Jul 2026).

Both sides cite real numbers. The "DeepSeek makes $38 a GPU-hour" camp (DeepInfer deployment Q&A, Oct 2026) and the "everyone on OpenRouter is losing money" camp are measuring different things.

So in this post I use only public data: GPU rental prices, measured throughput of one model on each card, the precision each card can actually run, and what the market actually pays per token. One model, one traffic pattern, one speed promise, one price, one rent. Only the card changes. The question is simple: if you buy or rent cards and host a model, do you make money?

The short answer

Yes, it can be profitable, but only on Blackwell, only if your cards are busy more than a third of the time, and only if you don't race to the bottom on price. An 8×B300 node at 50–70% utilization runs a 31–51% gross margin. H200 does not break even at any utilization.

Methodology

The setup below is held fixed throughout. Changing any part of it changes the answer, so it is worth stating precisely.

Model: DeepSeek V4.1 Flash. It's the #1 open-weight model on OpenRouter by a wide margin, roughly 30% of all tokens in the week of Oct 19–25 (OpenRouter rankings), and it's the only model with public third-party measurements on all three cards.

Traffic: SemiAnalysis's AgentX workload: 393 real Claude Code sessions replayed, median input 142k tokens, median output 444 tokens, ~95% of input is cache hits (AgentX methodology). This is what OpenRouter traffic actually looks like in 2026: the top 10 apps are all coding agents (OpenRouter apps), and agentic traffic is 7x human traffic with ~87% of tokens served from cache (OpenRouter, "Cache is Crucial", Oct 8, 2026). Chatbots are a small share now.

Speed promise: 100 output tokens/s per user. The median OpenRouter provider does ~35 (SemiAnalysis, InferenceX v2), so this is strict, but it's the one level where all three cards have numbers.

Throughput: best public open-source engine result on InferenceX for each card (H200, B300, GB300 NVL72), vLLM / SGLang / Dynamo, FP4 weights everywhere. H200 runs FP4 weights through dequant kernels; Blackwell runs native FP4.

Price: what OpenRouter providers actually collect on this model today: $0.013 per million input tokens (blended across cached and uncached) and $0.57 per million output (OpenRouter, V4.1 Flash 1017 providers). With a 99/1 input/output mix that's $0.019 per million total tokens. Not list price. Not DeepSeek's off-peak price. What the market pays.

Rent: lowest publicly visible on-demand price. H200 $3.99, B300 $7.40 (Hyperstack). GB300 NVL72 has no public rack price, so I used the SAVRN index at $15.21 (SAVRN GPU price index).

Utilization: tokens you actually sold ÷ tokens the benchmark says you could sell. Not GPU busy time. A new provider with no direct customers, relying on OpenRouter routing alone, lands at 10–30%. DeepInfra puts typical self-hosted inference at 10–20% (DeepInfra, Aug 2026), which matches what new providers have told us.

One formula: gross margin per GPU-hour = tokens/s × 3600 × price × utilization − rent. No bandwidth, no people, no spare capacity. If it doesn't work here, it won't work with those added.

H200 vs B300 vs GB300 NVL72: which GPU is profitable for inference?

At the price the market pays today, H200 cannot cover its rent, an 8×B300 node breaks even at 35% utilization, and a GB300 NVL72 rack breaks even at anywhere from 24% to 84% depending on how it is configured.

Throughput and rent on their own don't compare across cards. Divide one by the other and you get the number that does: how much rent you burn per million tokens at full load. The market pays $0.0188 per million, so that's the line to beat.

The last column summarizes the article in a single number. The percentage in brackets is also the break-even utilization: if rent eats 35% of the price at full load, you need to be 35% busy to cover it.

Then what each card earns. Dollars per GPU are not comparable across cards that cost 2–4x different rents, so the fair yardstick is gross margin: 1 − rent ÷ revenue. I use 50% as the bar: below that, bandwidth, people, spare capacity and the occasional idle week eat what's left.

In dollars, for the two that work: a B300 clears about $2,400 per GPU per month at 50% busy and $5,500 at 70%; a rack-tuned GB300 clears $12,500 and $22,000.

Three results stand out.

H200 can't cover its own rent at 100% utilization. Full load earns $1.6/hour against $3.99 rent. This is not a pricing problem or a tuning problem. Hopper has no FP4 compute units, so a model built around FP4 runs at 1/13 the throughput of a B300 while costing half as much. Software doesn't fix a 13x gap.

B300 is the practical starting point. An 8-GPU node breaks even at 35% and is clearly profitable above 50%. Note that 35% is already above where most new providers land, so this only works if you bring 1–2 direct customers with you. OpenRouter routing alone won't get you there.

GB300 NVL72 is the best and the worst option, depending on who tuned it. Same rack, same model: the standard InferenceX config needs 84% utilization to break even; the vLLM team's rack-level setup with prefill/decode disaggregation and Wide-EP needs 24% (vLLM, Oct 23, 2026). A 3.5x spread from configuration alone. If you can't reproduce that tuning, you're better off with B300s.

What utilization do you need to break even?

An 8×B300 node needs 35% utilization, a GB300 NVL72 rack needs 24–84% depending on tuning, and H200 would need more than 100%. A new provider relying on OpenRouter routing alone typically reaches 10–30%.

Look at where the grey band sits. That's where you'll be in month one. Only one bar lives there, and it's the one that requires a full rack and an engineering team that can reproduce a vLLM blog post.

The slope is the card's earning power; the intercept is the rent. H200 is a flat line below zero. B300 is the only line that is both cheap enough to start and steep enough to matter.

Why do estimates of inference margins differ so widely?

Estimates of this business differ widely because the hardware ranking never changes (H200 < B300 ≈ GB300), but three business variables swing the break-even for the same B300 node from 20% to 93%:

How did GPU rental prices change in 2026?

Of the three variables, rent is the one you control least, and in 2026 it went the wrong way. GPUs are supposed to depreciate. This year every generation got more expensive to rent, including the A100 from 2020: H200 on-demand rent is up 163% since January, B200 up 122%, H100 up 36% (a16z, Charts of the Week, Oct 23, 2026, using Ornn's Compute Price Index). Every one of them now rents above a six-year straight-line depreciation curve, the H200 by 143%.

The timing tells you why. H100 and B200 rents fell to about 77% of their January level in February and March, then snapped back in April, the same month agentic token volume on OpenRouter started its vertical climb (OpenRouter rankings). Rent follows token demand with a lag of weeks, and token prices follow competition with no lag at all. That is the squeeze: your cost went up 2x while your price went down, and both moves were set by other people.

Two practical consequences. First, the rents in this post are the cheap end of the market, not the middle: Ornn's executed-transaction index on Oct 25 has H200 at $6.30 and B200 at $9.70 per GPU-hour against the $3.99 and $7.40 Hyperstack list prices I used (data.ornn.com). At index rents, the B300 break-even moves from 35% toward 50%. Second, a 3–6 month reserved contract signed now is a bet that token demand keeps outrunning Blackwell supply. In 2026 that bet has paid; it is still a bet.

Take all three "good" and a B300 breaks even at 10%. Take all three "bad" and it loses money at 100%. The most optimistic estimates assume the good column; the most pessimistic assume the bad column. Each is correct for its own assumptions.

What inference provider pricing pages don't show

If these margins look thinner than what some providers appear to earn, it is because successful providers rarely sell exactly what the listing describes. The following practices are worth knowing before comparing prices on OpenRouter.

The model name is the same. The weights are not. Everyone lists "DeepSeek V4.1 Flash" or "GLM 5.3", but the cheapest endpoints run 4-bit weights, the expensive ones run 8-bit. On GLM-5.2, DeepInfra, Nebius and Parasail serve FP4, Novita and SiliconFlow serve FP8 (Artificial Analysis, GLM-5.2 providers), and NVIDIA quotes DeepInfra's cost per million tokens dropping from 20¢ on Hopper to 5¢ on NVFP4 (NVIDIA, Feb 2026). That 4x is where the "30% below list" prices come from. OpenRouter shows the precision only in a small tag on the endpoint, and about 95% of endpoints don't declare one at all. Baseten does it in the open: on GLM 5.3 their FP4 endpoint sells at list price and their FP8 endpoint is the "Fast" tier at 1.5x (OpenRouter, GLM 5.3). Same model, two products.

Quantization is not free, but it's not the main way quality gets cut either. On paper FP4 is nearly lossless now: NVIDIA's NVFP4 build of V4.1 Flash moves every benchmark by less than 1.5 points against DeepSeek's own checkpoint (GPQA Diamond 91.3 vs 91.0, Terminal-Bench 2.1 82.2 vs 81.6; NVIDIA release, Sep 16, 2026). Real users notice anyway: one team quantized GLM-5.2 to NVFP4, saw identical benchmark scores, and rolled it back after coding customers reported the chain of thought collapsing (umans.ai, Jul 2026). But the bigger cuts are cheaper than quantization. When Artificial Analysis scored the same models across providers in August, the most restrictive GLM-5.2 endpoints scored half the reference on scientific reasoning because of output-token caps, some gpt-oss-120b endpoints scored 22% on tool calling against 37% for the reference, and the worst endpoints produced roughly half the reference's reasoning tokens (Artificial Analysis, Endpoint Accuracy Index, Aug 4, 2026). Every one of those cuts raises tokens per second, which raises revenue per GPU-hour, which is why they happen. The labs have started policing it: Moonshot open-sourced a Kimi Vendor Verifier in April (GIGAZINE, Apr 21, 2026), and on Kimi K3 the providers that submitted now sit within about 4 points of the official API (Kimi-Vendor-Verifier results). The ones that don't submit are the ones to worry about.

Cache pricing is a routing weapon, not just a discount. OpenRouter pins a session to the same provider for 10 minutes only if that provider's cache-read price is below its input price (OpenRouter, sticky routing). So a low cache price buys you the follow-up requests, which are the cheap ones to serve. Providers who don't understand this get every request re-routed by price and watch their own cache hit rate collapse. The flip side: if you promise a cache discount but your KV memory can't hold the sessions, you pay full prefill cost on tokens you billed at 2%.

Price is the ranking algorithm. OpenRouter's default routing weights providers by the inverse square of price, so a provider at $1 gets roughly 9x the traffic of one at $3 (OpenRouter, provider routing). New providers also sit in an "insufficient data" tier until they've served 100 general and 200 tool-call requests, and Auto Exacto re-scores every endpoint every five minutes on tool-call success and throughput (OpenRouter, Auto Exacto). Free endpoints are the extreme version: MiniMax M3.1 Flash (free) and Nemotron 3 Ultra (free) are in the top 10 by tokens while their paid versions are a rounding error (OpenRouter rankings). That's subsidized traffic to buy ranking and data, not demand.

The benchmark counts tokens you don't get paid for. AgentX throughput is "total tokens" including cache hits. DeepSeek's 155 trillion tokens a day and $38 per GPU-hour are at off-peak prices with roughly 96% of input cached (DeepInfer deployment Q&A). Do the division and the real price is about 1 cent per million tokens. Anyone who quotes "25,000 tokens/s × $0.60" is using the output price, which applies to only 1% of the tokens.

The token business is often not the business. Look at what the incumbents actually sell: Together's and Fireworks' pricing pages lead with dedicated endpoints and GPU clusters by the hour, and Fireworks' on-demand H100 went from $4 to $8 an hour in a year (Fireworks pricing; Together pricing). The cheap per-token API is what brings customers in. The money is made on dedicated endpoints, priority tiers at 1.25–1.5x, region surcharges (Fireworks charges 1.5x for US-only), batch discounts at 50% and paid cache retention (DeepInfra, Aug 2026). If your plan is "list on OpenRouter and sell tokens", you are competing on the one product the big providers use as advertising.

The same practices, summarized. The right-hand column is how you catch it as a buyer, or what you'd have to decide as a seller.

Two common misconceptions about inference economics

Cached tokens are cheap to buy, not free to serve. 95% of the tokens flowing through your box are cache hits billed at $0.003–0.006 per million. They still occupy HBM and still trigger partial recompute. DeepSeek's $38/GPU-hour number comes from a 128+ GPU disaggregated deployment with ~100 PiB of SSD holding agent sessions for three days (DeepInfer deployment Q&A). That is not something 8 GPUs replicate.

You can't fill a rack from OpenRouter alone. A GB300 NVL72 at 70% utilization has to sell ~1.5 trillion tokens a day. That's 20–40% of everything OpenRouter routes to this model (~4–7 trillion a day, per OpenRouter rankings). You will not get that from OpenRouter's routing. An 8×B300 node needs ~1% of it, which is roughly what a mid-tier provider actually gets. What limits you is how many tokens you can sell, not how many you can produce.

And as I said last week: this is increasingly an SEO business. OpenRouter allocates traffic on price, latency and uptime (OpenRouter provider routing), so everyone optimizes for the algorithm. Which means the price column above drifts toward "bad" over time unless you have customers who pay you directly.

So, is it profitable?

The engine is not the problem. vLLM is within 6% of DeepSeek's own DeepInfer on the same hardware (vLLM, Oct 23, 2026). The open-source stack is done.

The problem is everything around it: the card (Blackwell or nothing), the configuration (a 3.5x spread on the same rack), the price you can actually hold, and the traffic you can actually attract. Small teams can make money on 8×B300 at 50%+ utilization with a couple of direct customers. They will not make money racing to the floor on OpenRouter, and they will not make money on H200 no matter what they do.

The three numbers that decide profitability

Beyond the benchmarks, profitability comes down to three numbers nobody can look up for you:

1. Your throughput, not SemiAnalysis's. Replay a real agent trace on the exact checkpoint you'll serve, at 50 and 100 tok/s per user. First attempts land at 50–70% of the published number.
2. Your rent, not the list price. Get a signed quote. For GB300 it has to be a full NVL72 rack with the NVLink fabric, or it's a different machine with the same name.
3. Your price, not the model's. One week of real customers tells you what you actually collect per million tokens, how much of it is cache, and how many tokens a day you can really sell.

Put those three numbers into the formula above and you have your own answer. This post only shows you what the answer depends on.

So here's my answer to the question in the title.

The engine is free. The model is free. The benchmarks are public. Anyone can get all three, which is why 30+ providers are selling the same model at 30+ different prices. None of them has an advantage the others can't copy next week.

What isn't free: Blackwell at a reasonable rent, an engineer who can reproduce a vLLM blog post, and customers who pay you directly instead of through a ranking algorithm. Have all three, and an 8×B300 node pays for itself at 35% utilization and is solidly profitable above 50%. With only two of the three, you are effectively subsidizing your customers' workloads.

Hosting an open-source inference engine produces tokens, not profit. Getting paid for those tokens is a separate business, and most providers on OpenRouter are not really in it.

Frequently asked questions

Can you make money hosting DeepSeek V4.1 Flash on OpenRouter?

Only with the right hardware and enough traffic. At the price OpenRouter providers actually collect, about $0.019 per million total tokens, an 8×B300 node breaks even at 35% utilization and runs a 31–51% gross margin at 50–70% utilization. New providers that rely on OpenRouter routing alone typically land at 10–30% utilization, below break-even, so a few direct customers are needed.

Why does H200 lose money serving DeepSeek V4.1 Flash?

Hopper has no native FP4 compute, so a model built around FP4 runs at about 1/13 the throughput of a B300 while renting for about half as much. At full load an H200 earns about $1.60 per GPU-hour against $3.99 rent, so it cannot break even at any utilization.

What GPU utilization does an inference provider need to break even?

In this analysis, 35% for an 8×B300 node, 24% for a rack-tuned GB300 NVL72 and 84% for a GB300 NVL72 on the standard configuration. With realistic changes in price, rent and throughput, the B300 break-even ranges from 20% to 93%.

Does the choice of open-source inference engine matter for profitability?

Less than the GPU. vLLM reproduces the decode throughput of DeepSeek's own DeepInfer engine within 6% on the same hardware. The GPU generation, the configuration, the price you can hold and the traffic you can attract matter far more.

Is DeepSeek's $38 per GPU-hour figure representative?

No. It comes from a disaggregated deployment of 128+ GPUs with about 100 PiB of SSD for caching, at off-peak prices with about 96% of input tokens cached. Spread over all tokens served, that is about 1 cent per million tokens, and an 8-GPU deployment cannot replicate the setup.