API Documentation

A normal request has no deadline: if capacity is busy it waits, and you only find out that the answer is late. A fill-or-kill (FOK) request states a time to live (TTL). We either start processing it on our own engine within the TTL, or reject it immediately with HTTP 429, so your code can go elsewhere while the answer is still worth having. A FOK request is never queued waiting for a slot, and it is never forwarded to another provider as a substitute for starting it ourselves.

It is opt-in per request; requests that do not ask for it are unaffected. Use it when a fast no is worth more than a slow yes:

• agents with a per-turn time budget that can fall back to another model or provider;
• latency-sensitive interactive calls (autocomplete, routing, classification) where a late answer is useless;
• routers that fan out across providers and want an immediate, machine-readable refusal.

The TTL is a promise about when generation starts, not when it finishes. Total latency still depends on prompt and output length.

Request: body form

Add a fill_or_kill object to a normal /v1/chat/completions body. It is removed by the gateway before the request reaches the model.

POST /v1/chat/completions

{
  "model": "google/gemma-4-26B-A4B-it",
  "messages": [{"role": "user", "content": "Summarise this ticket in one line."}],
  "max_tokens": 200,
  "fill_or_kill": {
    "ttl_ms": 2000,
    "bid": {"type": "request", "amount_usd": 0.02}
  }
}

Per-million-token bid (any subset of the three rates; at least one is required):

"bid": {
  "type": "tokens",
  "input_usd_per_mtok": 0.50,
  "cached_usd_per_mtok": 0.10,
  "output_usd_per_mtok": 1.50
}

type can be omitted when only one shape's fields are present. If both a request price and per-token rates are present without a type, the request is rejected with 400 rather than guessed at, since the two shapes differ by about six orders of magnitude. The only supported currency is usd.

Absolute deadline instead of a TTL:

"fill_or_kill": {
  "deadline": 1791043200500,
  "bid": 0.02
}

Request: header form

For SDKs or proxies that cannot add body fields, the same request can be expressed with headers. Headers are only read when the body carries no fill_or_kill object.

A bid header is required, exactly as in the body form.

cURL (body form)

curl -sS -i "https://inference.raindragon.ai/v1/chat/completions" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "model": "google/gemma-4-26B-A4B-it",
    "messages": [{"role": "user", "content": "Summarise this ticket in one line."}],
    "max_tokens": 200,
    "fill_or_kill": {"ttl_ms": 2000, "bid": {"type": "request", "amount_usd": 0.02}}
  }'

cURL (header form)

curl -sS -i "https://inference.raindragon.ai/v1/chat/completions" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "X-Fill-Or-Kill: 1" \
  -H "X-Fok-Ttl-Ms: 2000" \
  -H "X-Fok-Bid-Usd: 0.02" \
  -d '{
    "model": "google/gemma-4-26B-A4B-it",
    "messages": [{"role": "user", "content": "Summarise this ticket in one line."}],
    "max_tokens": 200
  }'

Python (OpenAI SDK)

Use extra_body or extra_headers. Set max_retries=0 on the client (or per call with client.with_options(max_retries=0)): the OpenAI SDK automatically retries 429 responses with backoff, which turns an immediate rejection into a slow one.

import openai
from openai import OpenAI

client = OpenAI(
    base_url="https://inference.raindragon.ai/v1",
    api_key="YOUR_API_KEY",
    max_retries=0,  # IMPORTANT: the SDK retries 429s by default; a FOK reject
                    # is an answer, not a transient error.
)

try:
    resp = client.chat.completions.create(
        model="google/gemma-4-26B-A4B-it",
        messages=[{"role": "user", "content": "Summarise this ticket in one line."}],
        max_tokens=200,
        # Option A: body field
        extra_body={"fill_or_kill": {
            "ttl_ms": 2000,
            "bid": {"type": "request", "amount_usd": 0.02},
        }},
        # Option B (instead of A): headers
        # extra_headers={"X-Fill-Or-Kill": "1",
        #                "X-Fok-Ttl-Ms": "2000",
        #                "X-Fok-Bid-Usd": "0.02"},
    )
    print(resp.choices[0].message.content)
except openai.RateLimitError as e:
    body = e.body if isinstance(e.body, dict) else {}
    detail = body.get("detail", body)
    if isinstance(detail, dict) and detail.get("type") == "fill_or_kill_rejected":
        # Not started within the TTL. Nothing ran, nothing was billed.
        print("rejected:", detail["reason"], "est. start ms:", detail["estimated_start_ms"])
        # ...send it somewhere else, or retry later on your own schedule.
    else:
        raise  # an ordinary rate limit, not a fill-or-kill rejection

# Need the X-Fill-Or-Kill / X-Fok-Request-Id response headers? Use with_raw_response:
raw = client.chat.completions.with_raw_response.create(
    model="google/gemma-4-26B-A4B-it",
    messages=[{"role": "user", "content": "Hi"}],
    extra_body={"fill_or_kill": {"ttl_ms": 2000, "bid": 0.02}},
)
print(raw.headers.get("X-Fill-Or-Kill"))   # "accepted"
resp = raw.parse()

Response: accepted

A normal chat completion, with two extra response headers. If X-Fill-Or-Kill: accepted is absent, the request was served as an ordinary (non-FOK) request, for example because the feature is switched off.

HTTP/1.1 200 OK
X-Fill-Or-Kill: accepted
X-Fok-Request-Id: 3f2c9a0e8b7d4c1fa6e5d4c3b2a19087

{ ...a normal chat.completion... }

With "stream": true, the first event on the stream is an acknowledgement chunk with an empty delta and an x_fok object, sent as soon as the request is admitted. Standard SDKs treat it like any empty chunk (so delta.content is None/null on it); read x_fok from the raw event if you want it. The model's output follows afterwards, so do not expect text immediately after the acknowledgement.

data: {"id": "fok-3f2c9a0e8b7d", "object": "chat.completion.chunk", "created": 1791043200,
       "model": "google/gemma-4-26B-A4B-it",
       "choices": [{"index": 0, "delta": {}, "finish_reason": null}],
       "x_fok": {
         "accepted": true,
         "status": "generating",
         "request_id": "3f2c9a0e8b7d4c1fa6e5d4c3b2a19087",
         "ttl_ms": 2000,
         "estimated_start_ms": 0,
         "bid": {"type": "request", "amount_usd": 0.02, "currency": "usd"},
         "reason": "running 3, queue 0, est start 0ms within 2000ms",
         "billing": "metered_rate_not_bid",
         "preemptable": true
       }}

data: {...normal chunks...}

An accepted FOK request runs at lower scheduling priority than ordinary traffic ("preemptable": true). If the engine becomes full while it is generating, it may be paused and recomputed rather than an ordinary request. You still get the full answer, but time after the start can be longer than for a normal request. In rare cases a failure after generation has started can be completed by a fallback provider; the start promise has already been kept at that point.

Response: rejected (HTTP 429)

Returned immediately, before the request reaches the model. A rejected request is not billed. There is no Retry-After header; use estimated_start_ms (our estimate of how long you would have waited, or null if unknown) to decide whether and when to retry. Identify a FOK rejection by detail.type == "fill_or_kill_rejected"; other 429s (account rate limits, server overload) have different bodies.

HTTP/1.1 429 Too Many Requests
Content-Type: application/json

{
  "detail": {
    "message": "Fill-or-kill request rejected: cannot start within the requested TTL.",
    "type": "fill_or_kill_rejected",
    "code": "capacity",
    "request_id": "3f2c9a0e8b7d4c1fa6e5d4c3b2a19087",
    "ttl_ms": 2000,
    "reason": "queue depth 5 > 2",
    "estimated_start_ms": 3120,
    "engine": {"running": 18.0, "waiting": 5.0, "ttft_mean_s": 1.9}
  },
  "request_id": "req_..."
}

detail.request_id is the FOK decision id (the same value as X-Fok-Request-Id on an accept); the outer request_id is the gateway's request id. reason is human-readable and may change; branch on type and code. Reasons you may see include: the deadline already passed, recent time-to-first-token too high, queue too deep, too many running requests, estimated start later than your TTL, the prompt does not fit our engine (context length or free KV cache), the engine is unhealthy, or being outbid (see below).

Response: invalid request (HTTP 400)

A fill_or_kill block we cannot read is an error, never silently served as a normal request:

HTTP/1.1 400 Bad Request

{
  "detail": {
    "message": "bid carries both a request price and per-token rates; set bid.type to say which one applies",
    "type": "invalid_request_error",
    "param": "fill_or_kill",
    "code": "invalid_fill_or_kill"
  },
  "request_id": "req_..."
}

Causes include:

• fill_or_kill is not an object (including true), or carries no bid;
• a TTL, deadline or bid value that is not a finite, non-negative number;
• a bid with both a request price and per-token rates and no type, or a type that contradicts the fields present;
• bid.type other than request or tokens;
• a currency other than usd.

How the bid works today

Accepted FOK requests are billed at our normal metered rates, not at your bid. The bid is a statement of what the request is worth to you, not a price quote: bidding high does not raise your bill, and bidding low (or zero) does not lower it. Every bid is recorded alongside the decision.

When there is spare capacity, which is the usual case, the bid has no effect at all: admission depends only on load and your TTL. When capacity is contested (requests are queueing, the engine is close to its running limit, or the KV cache is nearly full), FOK requests that pass the load checks enter a short auction:

• your request is held for up to 250 ms (never more than half of your remaining TTL, minus a 150 ms reserve) so that other FOK bids arriving in that window can compete;
• bids are ranked by our estimate of the value of your bid per unit of engine resource the request would use (from your bid, the prompt length and predicted output length), not by the raw bid amount, so a large prompt needs a proportionally larger bid;
• the top-ranked bids take the available slots (one slot when the signal is queueing; otherwise up to four, capped by the headroom left below the fill-or-kill limit on running requests, so a request that passed admission always gets at least one); the rest are rejected with 429 and a reason beginning outbid.

The auction only decides which FOK requests start. It does not change what an accepted request is charged, and it never affects ordinary (non-FOK) traffic. If there is no time left for an auction, or anything in it fails, the request is admitted on the load checks alone.

Limits

The returned ttl_ms (in the 429 body and in x_fok) is the TTL after clamping, so you can see whether yours was adjusted. If our load metrics cannot be read at all, FOK requests are admitted rather than rejected.