Posts Tagged "serving"

At each decoding step, a sparse-attention model reads a sliver of a session's history. Adaptive HiSparse keeps the rest in host memory, lends it GPU space when there is room, and takes the space back when new requests need it. On DeepSeek-V4.1-Flash at 500K tokens of context, that is 20 to 27% more output from the same GPUs at high load.

Last October, ten billion lifetime tokens earned developers a public tribute at OpenAI DevDay. This June, one Umans AI user ran nearly eleven billion tokens in a single day. What that says about token demand, compute supply, and the serving efficiency that has to close the gap between them.

Put the exact same model on GPUs and its token production cost can vary by multiples depending on how you serve it. A walk through the economics of a token: the frontier, the dollars it turns into, the levers that move it, and the claim our measurements let us make.