Troubleshoot

Find what you see in the tables below, and what to do about it. Start with what the server says about itself:

# 200 when the engine is up, 503 when it has died, and 000 while nothing answers yet (the server is still starting)
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8000/health
# the registered tasks, and whether the server applies them
curl -s http://127.0.0.1:8000/v1/tasks

Run against the CPU stand-in at v0.35.0 on 2026-10-26. It printed:

200
{"enabled":true,"head_engine":true,"fingerprint":"{\"desnake_labels\": true, \"hide_index_keys\": true, \"model\": \"Qwen3-0.6B-Base\", \"pad_to\": \"block\", \"pad_where\": \"front\", \"prompt_format\": {\"slot\": \"prefill\", \"tail\": \"spaced\", \"variants\": \"single\"}, \"served_name\": \"decisio-qwen3.6-35b-a3b-letters\"}","store":{"saved_to":{"backend":"dir","path":"decisio-tasks"},"readout":{"loaded":1,"applied":1,"other_stack":0},"abstention":{"loaded":0,"applied":0,"other_model":0,"other_layout":0},"at_start":{"saved_to":{"backend":"dir","path":"decisio-tasks"},"from_dir":{"readout":0,"abstention":0},"from_file":{"readout":0,"abstention":0},"usable":0}},"tasks":[{"format":"decisio-task/1","id":"ticket-routing","key":"090e8cdb70d68b239d07a0dda892976980e4bc669a722b37a24ecfd58fa94368","type":"choice","options":["billing","refund","payment_failed","login","password_reset","account_closure","bug","feature_request","performance","data_export","integration","device_delivery"],"fingerprint":"{\"desnake_labels\": true, \"hide_index_keys\": true, \"model\": \"Qwen3-0.6B-Base\", \"pad_to\": \"block\", \"pad_where\": \"front\", \"prompt_format\": {\"slot\": \"prefill\", \"tail\": \"spaced\", \"variants\": \"single\"}, \"served_name\": \"decisio-qwen3.6-35b-a3b-letters\"}","calibration":{"format":"decisio-task-prior/1","task_id":"ticket-routing","fingerprint":"{\"desnake_labels\": true, \"hide_index_keys\": true, \"model\": \"Qwen3-0.6B-Base\", \"pad_to\": \"block\", \"pad_where\": \"front\", \"prompt_format\": {\"slot\": \"prefill\", \"tail\": \"spaced\", \"variants\": \"single\"}, \"served_name\": \"decisio-qwen3.6-35b-a3b-letters\"}","K":12,"n_examples":120,"bias":[0.180587,0.001957,0.253523,0.005244,-0.166792,0.01471,0.279972,-0.293541,-0.106625,-0.175816,0.183685,-0.176904],"applied":true,"cv_logloss_plain":2.42554,"cv_logloss_fitted":2.35643,"cv_acc_plain":0.2667,"cv_acc_fitted":0.2917,"cv_t":-4.231,"lambda":0.1,"reason":"cross-validated gain","inputs_sha256":"d1e93377654df574e4826f636b9aac4693cc94059a06f0b97b05f15b2fb17f5a"},"head":{"format":"decisio-intent-head/1","task_id":"ticket-routing","fingerprint":"{\"desnake_labels\": true, \"hide_index_keys\": true, \"model\": \"Qwen3-0.6B-Base\", \"pad_to\": \"block\", \"pad_where\": \"front\", \"prompt_format\": {\"slot\": \"prefill\", \"tail\": \"spaced\", \"variants\": \"single\"}, \"served_name\": \"decisio-qwen3.6-35b-a3b-letters\"}","K":12,"options":["billing","refund","payment_failed","login","password_reset","account_closure","bug","feature_request","performance","data_export","integration","device_delivery"],"options_sha256":"c1cc9a5d741840dd23eeb3146aa445f21e69a31bdc98652e24b614499129229e","n_examples":120,"min_per_option":10,"applied":true,"cv_logloss":{"none":2.42554,"100.0":2.42214,"10.0":2.39179,"1.0":2.11252,"0.1":1.00911,"0.01":0.45439,"0.001":0.28943,"0.0001":0.23452},"temperature":0.766382,"temperature_cv":{"n":120,"log_loss":{"without":0.23452,"with":0.22105},"ece":{"without":0.01955,"with":0.02235}},"lambda":0.0001,"reason":"cross-validated gain","A":{"shape":[1024,12]},"c":{"shape":[12]},"inputs_sha256":"d8f21b2c08ffec9c307d318889fd63aa48d81fbdca4ebed8b527ae4c6683b728"},"n_examples":120,"per_option_min":10}]}

1. It does not start

What you see	Why	What to do
Connection refused right after the start	The server is still starting: the first start downloads the weights and loads the model.	Wait until GET /health answers.
vLLM's ValueError: To serve at least one request with the model's max seq len, or decisio's "holds a context of N tokens, less than the 32,768"	The card is too small for the model at that text length. The container warns first when the card has less than 90 GB.	Start a smaller model (--base gemma-4-12b), lower the length (--max-model-len 16384) or raise --gpu-memory-utilization; or run on a Mac.
decisio: no NVIDIA GPU is visible to this container, and exit code 1	The container cannot see the card, or the driver does not support CUDA 13.0.	Update the driver (R580 or newer; nvidia-smi shows the CUDA version it supports), install the NVIDIA Container Toolkit, and start with --gpus all.
uvicorn's address already in use, and exit code 1, after the model has loaded	Something else holds port 8000; some GPU rental templates run a proxy on it.	Start on another port (--port 8001, or DECISIO_PORT in the container), or find the process: ss -ltnp | grep :8000 on Linux, lsof -nP -iTCP:8000 -sTCP:LISTEN on a Mac.
OSError: We couldn't connect to 'https://huggingface.co'	The download failed and the files are not in the cache.	Start again: finished files stay in the cache. Check the cache's disk (~/.cache/huggingface/hub, $HF_HOME, or /data/hf in the container). A mirror must serve the pinned revision. Set HF_HUB_OFFLINE=1 only once the files are cached.
The server will not start and mentions VLLM_USE_DEEP_GEMM	The server refuses to start when it is set to anything other than 0.	Unset it, or pass --allow-deep-gemm if you mean it.
The server refuses a model repository	The repository's settings differ from the installed release's; the message names what differs and the release it was made for.	Use the repository revision made for your release, or install that release.

2. A request fails

Every error comes with a detail that names what is wrong: read it first.

What you see	Why	What to do
422 on a request	The request is malformed, or the prompt is longer than the length the server serves (/health reports it); the detail names what is wrong.	Read the detail; shorten the text or the question and send it again. The server stays up.
422 on a choice	It failed validation, for example with more than 255 options.	Fix what the detail names, such as cutting the options to 255 or fewer.
415 on a request	The body was not sent as Content-Type: application/json.	Send the header on every POST.
503 on every request, and /health answers 503	The engine died; the server is about to exit with code 70.	Let your supervisor start a new server; read the reason in the body.

3. A task does not apply

What you see	Why	What to do
WARNING: tasks fitted under another model or rendering are not applied at the start	The tasks were taught under another model, prompt or yes/no rendering.	Teach them again from their examples.
No x-decisio-tasks header on a question you taught	The question matched no task: its type, options or order differ from the registration.	Ask it exactly as registered; renaming, adding or reordering an option makes a different task.
Every answer of a task says certified: false	Nothing was certified (nothing_certified), the serving setup changed (serving_stack_changed), or traffic shifted (traffic_changed).	Read certified_reason and GET /v1/tasks/{id}; certify again on new labelled examples.

4. An answer surprises you

What you see	Why	What to do
An answer differs from an earlier one	A request repeats exactly on a freshly started server; asked again on one that has already answered it, it can move slightly, and across restarts it is not promised.	Compare answers from the same server and session before treating a difference as a fault.
The first request after a start is slow	It warms the model; a long text seen for the first time also takes longer on the larger models.	Send a few requests first, and raise a client's timeout for long texts.

What next

• Operate: what to watch, and what happens when the engine dies.
• Requirements and limits: what each machine needs, and what repeats.
• The HTTP API: every status and field.