# Troubleshoot

> What you see, why it happens, and what to do, from a server that will not start to an answer that surprises you.

Source: https://opendecisio.com/docs/troubleshooting/

Find what you see in the tables below, and what to do about it. Start with what the server says about itself:

 

```
# 200 when the engine is up, 503 when it has died, and 000 while nothing answers yet (the server is still starting)
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8000/health
# the registered tasks, and whether the server applies them
curl -s http://127.0.0.1:8000/v1/tasks
```

 

Run against the CPU stand-in at v0.35.0 on 2026-10-26. It printed:

```
200
{"enabled":true,"head_engine":true,"fingerprint":"{\"desnake_labels\": true, \"hide_index_keys\": true, \"model\": \"Qwen3-0.6B-Base\", \"pad_to\": \"block\", \"pad_where\": \"front\", \"prompt_format\": {\"slot\": \"prefill\", \"tail\": \"spaced\", \"variants\": \"single\"}, \"served_name\": \"decisio-qwen3.6-35b-a3b-letters\"}","store":{"saved_to":{"backend":"dir","path":"decisio-tasks"},"readout":{"loaded":1,"applied":1,"other_stack":0},"abstention":{"loaded":0,"applied":0,"other_model":0,"other_layout":0},"at_start":{"saved_to":{"backend":"dir","path":"decisio-tasks"},"from_dir":{"readout":0,"abstention":0},"from_file":{"readout":0,"abstention":0},"usable":0}},"tasks":[{"format":"decisio-task/1","id":"ticket-routing","key":"090e8cdb70d68b239d07a0dda892976980e4bc669a722b37a24ecfd58fa94368","type":"choice","options":["billing","refund","payment_failed","login","password_reset","account_closure","bug","feature_request","performance","data_export","integration","device_delivery"],"fingerprint":"{\"desnake_labels\": true, \"hide_index_keys\": true, \"model\": \"Qwen3-0.6B-Base\", \"pad_to\": \"block\", \"pad_where\": \"front\", \"prompt_format\": {\"slot\": \"prefill\", \"tail\": \"spaced\", \"variants\": \"single\"}, \"served_name\": \"decisio-qwen3.6-35b-a3b-letters\"}","calibration":{"format":"decisio-task-prior/1","task_id":"ticket-routing","fingerprint":"{\"desnake_labels\": true, \"hide_index_keys\": true, \"model\": \"Qwen3-0.6B-Base\", \"pad_to\": \"block\", \"pad_where\": \"front\", \"prompt_format\": {\"slot\": \"prefill\", \"tail\": \"spaced\", \"variants\": \"single\"}, \"served_name\": \"decisio-qwen3.6-35b-a3b-letters\"}","K":12,"n_examples":120,"bias":[0.180587,0.001957,0.253523,0.005244,-0.166792,0.01471,0.279972,-0.293541,-0.106625,-0.175816,0.183685,-0.176904],"applied":true,"cv_logloss_plain":2.42554,"cv_logloss_fitted":2.35643,"cv_acc_plain":0.2667,"cv_acc_fitted":0.2917,"cv_t":-4.231,"lambda":0.1,"reason":"cross-validated gain","inputs_sha256":"d1e93377654df574e4826f636b9aac4693cc94059a06f0b97b05f15b2fb17f5a"},"head":{"format":"decisio-intent-head/1","task_id":"ticket-routing","fingerprint":"{\"desnake_labels\": true, \"hide_index_keys\": true, \"model\": \"Qwen3-0.6B-Base\", \"pad_to\": \"block\", \"pad_where\": \"front\", \"prompt_format\": {\"slot\": \"prefill\", \"tail\": \"spaced\", \"variants\": \"single\"}, \"served_name\": \"decisio-qwen3.6-35b-a3b-letters\"}","K":12,"options":["billing","refund","payment_failed","login","password_reset","account_closure","bug","feature_request","performance","data_export","integration","device_delivery"],"options_sha256":"c1cc9a5d741840dd23eeb3146aa445f21e69a31bdc98652e24b614499129229e","n_examples":120,"min_per_option":10,"applied":true,"cv_logloss":{"none":2.42554,"100.0":2.42214,"10.0":2.39179,"1.0":2.11252,"0.1":1.00911,"0.01":0.45439,"0.001":0.28943,"0.0001":0.23452},"temperature":0.766382,"temperature_cv":{"n":120,"log_loss":{"without":0.23452,"with":0.22105},"ece":{"without":0.01955,"with":0.02235}},"lambda":0.0001,"reason":"cross-validated gain","A":{"shape":[1024,12]},"c":{"shape":[12]},"inputs_sha256":"d8f21b2c08ffec9c307d318889fd63aa48d81fbdca4ebed8b527ae4c6683b728"},"n_examples":120,"per_option_min":10}]}
```

 

## 1 It does not start

 

| What you see | Why | What to do |
| --- | --- | --- |
| Connection refused right after the start | The server is still starting: the first start downloads the weights and loads the model. | Wait until `GET /health` answers. |
| vLLM's `ValueError: To serve at least one request with the model's max seq len`, or decisio's "holds a context of N tokens, less than the 32,768" | The card is too small for the model at that text length. The container warns first when the card has less than 90 GB. | Start a smaller model (`--base gemma-4-12b`), lower the length (`--max-model-len 16384`) or raise `--gpu-memory-utilization`; or run on a Mac. |
| `decisio: no NVIDIA GPU is visible to this container`, and exit code 1 | The container cannot see the card, or the driver does not support CUDA 13.0. | Update the driver (R580 or newer; `nvidia-smi` shows the CUDA version it supports), install the NVIDIA Container Toolkit, and start with `--gpus all`. |
| uvicorn's `address already in use`, and exit code 1, after the model has loaded | Something else holds port 8000; some GPU rental templates run a proxy on it. | Start on another port (`--port 8001`, or `DECISIO_PORT` in the container), or find the process: `ss -ltnp \| grep :8000` on Linux, `lsof -nP -iTCP:8000 -sTCP:LISTEN` on a Mac. |
| `OSError: We couldn't connect to 'https://huggingface.co'` | The download failed and the files are not in the cache. | Start again: finished files stay in the cache. Check the cache's disk (`~/.cache/huggingface/hub`, `$HF_HOME`, or `/data/hf` in the container). A mirror must serve the pinned revision. Set `HF_HUB_OFFLINE=1` only once the files are cached. |
| The server will not start and mentions `VLLM_USE_DEEP_GEMM` | The server refuses to start when it is set to anything other than 0. | Unset it, or pass `--allow-deep-gemm` if you mean it. |
| The server refuses a model repository | The repository's settings differ from the installed release's; the message names what differs and the release it was made for. | Use the repository revision made for your release, or install that release. |

 

## 2 A request fails

 

Every error comes with a `detail` that names what is wrong: read it first.

 

| What you see | Why | What to do |
| --- | --- | --- |
| 422 on a request | The request is malformed, or the prompt is longer than the length the server serves (`/health` reports it); the `detail` names what is wrong. | Read the `detail`; shorten the text or the question and send it again. The server stays up. |
| 422 on a choice | It failed validation, for example with more than 255 options. | Fix what the `detail` names, such as cutting the options to 255 or fewer. |
| 415 on a request | The body was not sent as `Content-Type: application/json`. | Send the header on every POST. |
| 503 on every request, and `/health` answers 503 | The engine died; the server is about to exit with code 70. | Let your supervisor start a new server; read the reason in the body. |

 

## 3 A task does not apply

 

| What you see | Why | What to do |
| --- | --- | --- |
| `WARNING: tasks fitted under another model or rendering are not applied` at the start | The tasks were taught under another model, prompt or yes/no rendering. | Teach them again from their examples. |
| No `x-decisio-tasks` header on a question you taught | The question matched no task: its type, options or order differ from the registration. | Ask it exactly as registered; renaming, adding or reordering an option makes a different task. |
| Every answer of a task says `certified: false` | Nothing was certified (`nothing_certified`), the serving setup changed (`serving_stack_changed`), or traffic shifted (`traffic_changed`). | Read `certified_reason` and `GET /v1/tasks/{id}`; certify again on new labelled examples. |

 

## 4 An answer surprises you

 

| What you see | Why | What to do |
| --- | --- | --- |
| An answer differs from an earlier one | A request repeats exactly on a freshly started server; asked again on one that has already answered it, it can move slightly, and across restarts it is not promised. | Compare answers from the same server and session before treating a difference as a fault. |
| The first request after a start is slow | It warms the model; a long text seen for the first time also takes longer on the larger models. | Send a few requests first, and raise a client's timeout for long texts. |

 

## What next

 

- [Operate](https://opendecisio.com/docs/operating/): what to watch, and what happens when the engine dies.
- [Requirements and limits](https://opendecisio.com/docs/limits/): what each machine needs, and what repeats.
- [The HTTP API](https://opendecisio.com/docs/api/): every status and field.