Batch vs real-time inference
Updated
Real-time inference answers each API request as it arrives, in seconds, while batch inference takes a file of many requests and returns all the results within a set window, such as 24 hours.
Both run the same model. Batch gives up a fast answer in exchange for a simpler way to send large amounts of work, and often a lower price.
How they differ
| Real-time | Batch | |
|---|---|---|
| Answer arrives | In seconds, streamed if you want | Within the window, for example 24 hours |
| How you send work | One HTTP request per call | A JSONL file, one request per line |
| Retries and rate limits | Your code handles them | The provider works through the file |
| Results | In the response | An output file matched by custom_id, plus an error file |
| Good for | Chat, coding agents, anything a person waits on | Evaluations, classification, summaries, backfills |
Why batch is often cheaper
A provider has to keep capacity ready for real-time peaks. Batch work can run in the gaps, so many providers price it lower. The OpenAI Batch API, whose file format most batch APIs follow, documents a 24-hour completion window, up to 50,000 requests per batch and a 50% discount against its synchronous API.
Check the batch price per model before moving work. On OpenScale, real-time requests already bill below list price (DeepSeek V4.1 Flash is 50% under its list price), and a batch discount is set per model off the list price. Where a model has none, real-time is the cheaper path and batch is worth it for the workflow alone.
The batch workflow
{"custom_id": "doc-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "deepseek/deepseek-v4.1-flash", "messages": [{"role": "user", "content": "Summarize: ..."}]}}- Upload the JSONL file to
POST /v1/fileswithpurpose=batch. - Create the batch with
POST /v1/batches, naming the file and"completion_window": "24h". - Poll
GET /v1/batches/{id}until it completes, then downloadoutput_file_idanderror_file_id.
OpenScale's Batch API follows this OpenAI-compatible format, one model per batch, and checks every line when the batch is created, so a bad line fails the batch before anything runs. The API reference has the details.
How to choose
- A person or an agent is waiting on the answer: real-time.
- The work is a list you can prepare in advance and nobody needs the result this minute: batch.
- The list is small (under a few hundred requests): real-time with a simple concurrency limit is often simpler.
Questions
Is the output of batch inference different from real-time?
No. It is the same model with the same parameters. Only the timing and the way you send requests change.
Can I stream batch results?
No. Batch requests run without streaming, and results arrive as a file.