# Serverless

Your own model on dedicated GPUs: templates, the image contract and how GPU time is billed.

## Serverless

Run your own model on dedicated GPUs and call it like any OpenScale model, with your API keys:
send the endpoint's model id as `model` to `https://api.openscale.so/v1/chat/completions`.
Create an endpoint in the dashboard under Serverless: pick the GPU (RTX 5090, RTX PRO 6000 or
H100 80GB), how many pods it may run and the most you pay per GPU-hour. You pay the GPU price
for each second a GPU of the endpoint runs, from the moment its pod starts until it stops; the
calls themselves are not billed per token. A new pod takes about 15 minutes to start
serving; keep pods warm (min pods, or warm pods by time of day) to answer without that wait.

Each endpoint has a monthly cap on its GPU cost. When the month's GPU cost reaches the cap, or
your balance runs out, the endpoint pauses its pods and we email you; raise the cap or add
credits, then resume it.

### Start from a template

Set `LM_MODEL` to the Hugging Face model to serve, build the image, push it to your registry
and use it as the endpoint's image. vLLM:

```dockerfile
FROM vllm/vllm-openai:v0.33.0@sha256:d825baf9ce4b1b6bfc5ecc7b79880c7e6748c65a72d697ce3d47607aafa84c6e
COPY --from=us-docker.pkg.dev/openscale-prod-e73caa/serverless-templates/contract:v1 /opt/serving /opt/serving
ENV LM_ENGINE=vllm \
    LM_MODEL=Qwen/Qwen3-8B
ENTRYPOINT ["bash", "/opt/serving/boot.sh"]
```

SGLang:

```dockerfile
FROM lmsysorg/sglang:v0.5.23@sha256:f832b5c39559c9caa900b22ad37c76fcce66470676855c42052102c537812f1f
COPY --from=us-docker.pkg.dev/openscale-prod-e73caa/serverless-templates/contract:v1 /opt/serving /opt/serving
ENV LM_ENGINE=sglang \
    LM_MODEL=Qwen/Qwen3-8B
ENTRYPOINT ["bash", "/opt/serving/boot.sh"]
```

`LM_ARGS` passes extra engine flags, for example `ENV LM_ARGS="--max-model-len 32768"`. For a
gated model, set your own `HF_TOKEN` in the image and keep the image private: give the
endpoint the registry's user and token.

### The image contract

Any image can run on an endpoint when it does the following:

- At start the pod runs `/opt/serving/boot.sh` (the image's own entrypoint does not run). It
  starts your server and keeps it running. The pod's GPUs have a CUDA 13 driver.
- Port 8000 serves:
  - `GET /health`: `200` with `{"upstream": true}` once the model answers, `503` with
    `{"upstream": false}` while it loads. No key.
  - `GET /lm/metrics`: JSON with `in_flight` (calls in progress), `limit_N` (calls it takes
    at once) and `upstream_up` (`1` when serving). Scaling reads it.
  - `POST /v1/chat/completions`: the OpenAI format, streaming with `usage` on the last chunk,
    for the model named by the `LM_SERVED_MODEL` environment variable.
- Every request except `/health` carries `Authorization: Bearer <LLM_API_KEY>`, from the
  `LLM_API_KEY` environment variable; refuse any other with `401`.

The templates' `/opt/serving` (from `us-docker.pkg.dev/openscale-prod-e73caa/serverless-templates/contract:v1`) does all of this in front of vLLM
or SGLang: copy it into your own image to get the same.

## More of the API reference

- [API reference](https://openscale.so/docs): base URLs, authentication and a first request.
- [Chat Completions](https://openscale.so/docs/chat-completions): POST /v1/chat/completions: the OpenAI Chat Completions format, streamed or not, with reasoning and tools.
- [Responses](https://openscale.so/docs/responses): POST /v1/responses: the OpenAI Responses format with stored state, background runs and enforced tool rules.
- [Conversations](https://openscale.so/docs/conversations): Keep the turns of a Responses conversation on our side and continue it by id.
- [Prompts](https://openscale.so/docs/prompts): Save instructions, messages and settings once, then use them by id in the Responses API.
- [Anthropic Messages](https://openscale.so/docs/messages): POST /anthropic/v1/messages: the Anthropic Messages format, for Claude Code and Anthropic SDKs.
- [Models and pricing](https://openscale.so/docs/models): Model ids, prices per 1M tokens, context windows, output limits, versions and the live catalog endpoint.
- [Batch](https://openscale.so/docs/batch): Upload a JSONL file of chat completions, create a batch and collect the results when it finishes.
- [Errors](https://openscale.so/docs/errors): The error shape, every status code and code, and which ones to retry.
- [Rate limits](https://openscale.so/docs/rate-limits): Requests and tokens per minute, how a 429 tells you when to retry, and how to get more.
- [Data retention](https://openscale.so/docs/data-retention): What is stored, for how long, how to delete it, and zero data retention.
- [Interactive reference](https://openscale.so/docs/reference): try every endpoint in the browser.
- [OpenAPI document](https://openscale.so/openapi.json): the whole API as OpenAPI 3.1.
