Production best practices

Timeouts and cold starts, retries, API key handling, privacy and monitoring for Krun in production.

Timeouts and cold starts

Krun One runs on serverless GPUs that scale to zero when there is no traffic. After a period without requests, the next request starts a model worker first. This cold start can take tens of seconds. While workers are warm, requests are much faster.

Krun does not promise a fixed end-to-end latency. Plan for both cases:

• Set a client timeout of about 70 seconds. The API gives the model backend up to 60 seconds, including a cold start, and needs a few seconds more to respond. The Python SDK uses 70 seconds by default.
• Don’t block a user-facing request on a cold start if you can avoid it. For interactive flows, consider a fallback path, such as a default route, when a decision takes too long for your UX.
• Expect 504 UPSTREAM_TIMEOUT occasionally when a cold start takes longer than the API’s deadline. Retrying usually reaches the worker that has started in the meantime.

Python

from krun import Krun

client = Krun(timeout=70.0)  # default; per call: client.decide(..., timeout=30.0)

Retries

Retry only errors that are transient:

Retry	Don’t retry automatically
502 INFERENCE_FAILED	400 INVALID_REQUEST, 400 INVALID_OPTIONS: fix the request
503 UPSTREAM_UNAVAILABLE	401 UNAUTHORIZED: fix the API key
504 UPSTREAM_TIMEOUT	404 NOT_FOUND
Connection errors	413 PAYLOAD_TOO_LARGE: reduce the request
	429 QUOTA_EXCEEDED: the monthly quota is exhausted
	429 RATE_LIMITED: wait until Retry-After instead of retrying right away

Use exponential backoff with jitter, and honor the Retry-After header when it is present. Keep the number of retries small: the API already retries its model backend internally.

The Python SDK does this for you. decide() and models() retry once by default after connection errors and 502, 503 or 504, waiting for Retry-After (up to 10 seconds) or 0.5 s, 1 s, 2 s and so on. Set max_retries to change it. The SDK never retries its own timeout, and never retries feedback().

A retried decision can at worst be counted twice for usage. It has no other side effects.

Protect your API key

• Call Krun only from your server. Never put a Krun API key in browser, mobile or desktop client code.
• Load the key from an environment variable or a secret manager, as KRUN_API_KEY. Don’t commit it.
• Use separate keys per service or environment, so you can revoke one without affecting the others.
• If a key leaks, ask Krun to revoke it and issue a new one.

See Authentication.

Privacy

The API does not store or log the content of your requests: the context, question ids, option ids, descriptions, answers and probabilities are not persisted. For each request, Krun records usage metadata such as the request id, model, task type, status, input tokens, the number of questions and options, and latency.

Feedback stores what you send in expected_decision and metadata. Don’t put personal data in them.

Log request ids

Log the X-Request-ID (result.request_id in Python) next to each decision in your system. You need it to send feedback, and it lets Krun find the request if you contact support. Error responses include it too, in error.request_id.

Monitor decisions

• Track the abstention rate per question. A rising rate can mean your traffic changed or an option is missing.
• Log confidence and send feedback, so you can check accuracy on your own data.
• Watch for 429 responses and the X-RateLimit-Remaining header. See Rate limits and quotas.

Keep the option set stable

Probabilities are relative to the options you send. Changing option ids, descriptions or the number of options changes the answers. Version your option sets and compare accuracy before and after a change.