Engineering notes
Practical tradeoffs in running and using open models.
What a million tokens actually costs
A million input tokens and a million output tokens are different products. With Qwen3 32B, 800,000 input and 200,000 output tokens cost $0.16 at the listed rates. The same total with an 80% output mix costs $0.19.
Read the guideEstimate consumption before you migrate
Select a representative week and group requests by workflow. Preserve token counts and timings without keeping sensitive content in analytics.
Read the guideLatency and throughput answer different questions
TTFT includes queuing and prefill. It determines how long a user waits before seeing output.
Read the guideQuantization is a tradeoff, not a free upgrade
Lower precision reduces memory use and can improve throughput. Whether quality changes depends on the model, quantization method and task.
Read the guideSelf-hosting versus API: do the arithmetic
Dedicated monthly cost divided by effective per-million-token API cost gives the volume crossover before engineering and reliability costs. An H100 at $1,690 and an effective $0.16 per million tokens gives about 10.56 billion tokens per month. This does not prove a single H100 can serve that workload.
Read the guide