Serverless
Your own model on dedicated GPUs: templates, the image contract and how GPU time is billed.
Serverless
Run your own model on dedicated GPUs and call it like any OpenScale model, with your API keys: send the endpoint's model id as model to https://api.openscale.so/v1/chat/completions. Create an endpoint in the dashboard under Serverless: pick the GPU (RTX 5090, RTX PRO 6000 or H100 80GB), how many pods it may run and the most you pay per GPU-hour. You pay the GPU price for each second a GPU of the endpoint runs, from the moment its pod starts until it stops; the calls themselves are not billed per token. A new pod takes about 15 minutes to start serving; keep pods warm (min pods, or warm pods by time of day) to answer without that wait.
Each endpoint has a monthly cap on its GPU cost. When the month's GPU cost reaches the cap, or your balance runs out, the endpoint pauses its pods and we email you; raise the cap or add credits, then resume it.
Start from a template
Set LM_MODEL to the Hugging Face model to serve, build the image, push it to your registry and use it as the endpoint's image. vLLM:
FROM vllm/vllm-openai:v0.33.0@sha256:d825baf9ce4b1b6bfc5ecc7b79880c7e6748c65a72d697ce3d47607aafa84c6e
COPY --from=us-docker.pkg.dev/openscale-prod-e73caa/serverless-templates/contract:v1 /opt/serving /opt/serving
ENV LM_ENGINE=vllm \
LM_MODEL=Qwen/Qwen3-8B
ENTRYPOINT ["bash", "/opt/serving/boot.sh"]SGLang:
FROM lmsysorg/sglang:v0.5.23@sha256:f832b5c39559c9caa900b22ad37c76fcce66470676855c42052102c537812f1f
COPY --from=us-docker.pkg.dev/openscale-prod-e73caa/serverless-templates/contract:v1 /opt/serving /opt/serving
ENV LM_ENGINE=sglang \
LM_MODEL=Qwen/Qwen3-8B
ENTRYPOINT ["bash", "/opt/serving/boot.sh"]LM_ARGS passes extra engine flags, for example ENV LM_ARGS="--max-model-len 32768". For a gated model, set your own HF_TOKEN in the image and keep the image private: give the endpoint the registry's user and token.
The image contract
Any image can run on an endpoint when it does the following:
- At start the pod runs
/opt/serving/boot.sh(the image's own entrypoint does not run). It starts your server and keeps it running. The pod's GPUs have a CUDA 13 driver. - Port 8000 serves: -
GET /health:200with{"upstream": true}once the model answers,503with{"upstream": false}while it loads. No key. -GET /lm/metrics: JSON within_flight(calls in progress),limit_N(calls it takes at once) andupstream_up(1when serving). Scaling reads it. -POST /v1/chat/completions: the OpenAI format, streaming withusageon the last chunk, for the model named by theLM_SERVED_MODELenvironment variable. - Every request except
/healthcarriesAuthorization: Bearer <LLM_API_KEY>, from theLLM_API_KEYenvironment variable; refuse any other with401.
The templates' /opt/serving (from us-docker.pkg.dev/openscale-prod-e73caa/serverless-templates/contract:v1) does all of this in front of vLLM or SGLang: copy it into your own image to get the same.
More of the API reference
- API reference: base URLs, authentication and a first request.
- Chat Completions: POST /v1/chat/completions: the OpenAI Chat Completions format, streamed or not, with reasoning and tools.
- Responses: POST /v1/responses: the OpenAI Responses format with stored state, background runs and enforced tool rules.
- Conversations: Keep the turns of a Responses conversation on our side and continue it by id.
- Prompts: Save instructions, messages and settings once, then use them by id in the Responses API.
- Anthropic Messages: POST /anthropic/v1/messages: the Anthropic Messages format, for Claude Code and Anthropic SDKs.
- Models and pricing: Model ids, prices per 1M tokens, context windows, output limits, versions and the live catalog endpoint.
- Batch: Upload a JSONL file of chat completions, create a batch and collect the results when it finishes.
- Errors: The error shape, every status code and code, and which ones to retry.
- Rate limits: Requests and tokens per minute, how a 429 tells you when to retry, and how to get more.
- Data retention: What is stored, for how long, how to delete it, and zero data retention.
- Interactive reference: try every endpoint in the browser.
- OpenAPI document: the whole API as OpenAPI 3.1.
This page as Markdown: /docs/serverless.md