Serverless

Your own model on dedicated GPUs: templates, the image contract and how GPU time is billed.

Serverless

Run your own model on dedicated GPUs and call it like any OpenScale model, with your API keys: send the endpoint's model id as model to https://api.openscale.so/v1/chat/completions. Create an endpoint in the dashboard under Serverless: pick the GPU (RTX 5090, RTX PRO 6000 or H100 80GB), how many pods it may run and the most you pay per GPU-hour. You pay the GPU price for each second a GPU of the endpoint runs, from the moment its pod starts until it stops; the calls themselves are not billed per token. A new pod takes about 15 minutes to start serving; keep pods warm (min pods, or warm pods by time of day) to answer without that wait.

Each endpoint has a monthly cap on its GPU cost. When the month's GPU cost reaches the cap, or your balance runs out, the endpoint pauses its pods and we email you; raise the cap or add credits, then resume it.

Start from a template

Set LM_MODEL to the Hugging Face model to serve, build the image, push it to your registry and use it as the endpoint's image. vLLM:

dockerfile
FROM vllm/vllm-openai:v0.33.0@sha256:d825baf9ce4b1b6bfc5ecc7b79880c7e6748c65a72d697ce3d47607aafa84c6e
COPY --from=us-docker.pkg.dev/openscale-prod-e73caa/serverless-templates/contract:v1 /opt/serving /opt/serving
ENV LM_ENGINE=vllm \
    LM_MODEL=Qwen/Qwen3-8B
ENTRYPOINT ["bash", "/opt/serving/boot.sh"]

SGLang:

dockerfile
FROM lmsysorg/sglang:v0.5.23@sha256:f832b5c39559c9caa900b22ad37c76fcce66470676855c42052102c537812f1f
COPY --from=us-docker.pkg.dev/openscale-prod-e73caa/serverless-templates/contract:v1 /opt/serving /opt/serving
ENV LM_ENGINE=sglang \
    LM_MODEL=Qwen/Qwen3-8B
ENTRYPOINT ["bash", "/opt/serving/boot.sh"]

LM_ARGS passes extra engine flags, for example ENV LM_ARGS="--max-model-len 32768". For a gated model, set your own HF_TOKEN in the image and keep the image private: give the endpoint the registry's user and token.

The image contract

Any image can run on an endpoint when it does the following:

  • At start the pod runs /opt/serving/boot.sh (the image's own entrypoint does not run). It starts your server and keeps it running. The pod's GPUs have a CUDA 13 driver.
  • Port 8000 serves: - GET /health: 200 with {"upstream": true} once the model answers, 503 with {"upstream": false} while it loads. No key. - GET /lm/metrics: JSON with in_flight (calls in progress), limit_N (calls it takes at once) and upstream_up (1 when serving). Scaling reads it. - POST /v1/chat/completions: the OpenAI format, streaming with usage on the last chunk, for the model named by the LM_SERVED_MODEL environment variable.
  • Every request except /health carries Authorization: Bearer <LLM_API_KEY>, from the LLM_API_KEY environment variable; refuse any other with 401.

The templates' /opt/serving (from us-docker.pkg.dev/openscale-prod-e73caa/serverless-templates/contract:v1) does all of this in front of vLLM or SGLang: copy it into your own image to get the same.

More of the API reference

  • API reference: base URLs, authentication and a first request.
  • Chat Completions: POST /v1/chat/completions: the OpenAI Chat Completions format, streamed or not, with reasoning and tools.
  • Responses: POST /v1/responses: the OpenAI Responses format with stored state, background runs and enforced tool rules.
  • Conversations: Keep the turns of a Responses conversation on our side and continue it by id.
  • Prompts: Save instructions, messages and settings once, then use them by id in the Responses API.
  • Anthropic Messages: POST /anthropic/v1/messages: the Anthropic Messages format, for Claude Code and Anthropic SDKs.
  • Models and pricing: Model ids, prices per 1M tokens, context windows, output limits, versions and the live catalog endpoint.
  • Batch: Upload a JSONL file of chat completions, create a batch and collect the results when it finishes.
  • Errors: The error shape, every status code and code, and which ones to retry.
  • Rate limits: Requests and tokens per minute, how a 429 tells you when to retry, and how to get more.
  • Data retention: What is stored, for how long, how to delete it, and zero data retention.
  • Interactive reference: try every endpoint in the browser.
  • OpenAPI document: the whole API as OpenAPI 3.1.

This page as Markdown: /docs/serverless.md