NVIDIA NIM Operational Traps

Failure modes that cost real time running NVIDIA-NIM containers, extracted from two GPU hosts so they are findable without knowing which machine produced them. Every figure here was measured, not read from documentation.

The unifying shape: a NIM reports healthy and fails at the one thing it exists for. Health endpoints return 200, curl exits 0, and the answer is simply absent. usage.prompt_tokens on a real call is the only honest test.

The cache directory must be owned before the container starts

Symptom: the container restarts forever, ~15 s apart, never becomes ready.

nimlib.exceptions.ManifestDownloadError:
  Error downloading manifest: I/O error Permission denied (os error 13)

Cause: NIMs run as uid 1000, and a cache directory created by root is unwritable. --restart unless-stopped then retries against a directory it has no rights to, forever.

Fix — create and own it before docker run:

install -d -o 1000 -g 1000 /path/to/nim-cache

Do not assume the uid. The name varies between images (nvs, nim); only the uid is stable. Check it:

docker run --rm --entrypoint id <image>

There is no universal correct mode for a bind mount — it is whatever uid the container’s process runs as. On one host Ollama’s volume works fine at 0755 root:root because that container runs as root, while the NIM beside it needs uid 1000 to own its cache.

If a daemon does no UID remapping — check docker info for a userns entry in SecurityOptions; absent means none — container uid 1000 is literally host uid 1000. The container owns those files, so owner bits alone suffice and 0777 is never necessary.

Tightening a cache safely while a download is running:

chown -R 1000:1000 <cache> && chmod -R u=rwX,g=rX,o= <cache>

• chown must precede chmod. In that order the process owns the tree throughout. Reverse it and the top directory is still root:root when the mode tightens — the container loses access mid-write.
• Capital X sets execute on directories and on files that already had it, so weights do not come out executable. Replaces the two-pass find -type d / -type f.
• The parent directory does not need to be traversable. A bind mount grafts the directory into the container’s mount namespace, so path resolution inside runs through the container path — the host path above the mount point is invisible and its permissions are never consulted.

Two crash loops look identical from outside

Both present as a container restarting forever. docker ps shows the same thing and --restart unless-stopped hides both. The log line is the only discriminator:

docker logs <container> 2>&1 | grep -iE "permission denied|kv cache"

The default context does not fit a small card

A model’s default max_seq_len is sized for datacenter GPUs. On a 24 GB card the arithmetic runs out:

Set NIM_MAX_MODEL_LEN explicitly. The engine names both ceilings itself when launched uncapped, so launch once without a cap and read the number out of the log rather than guessing.

The same model on a bigger card has no such constraint — an L40S reported 20.91 GiB of KV available and a 148,128 ceiling against the L4’s 1.85 GiB and 13,488. This is a card constraint, not a model defect, and it is exactly why a launcher script that works at one site fails at another.

Run list-model-profiles before letting weights download

NIM logs Detected 0 compatible profile(s) and then falls back to a generic selector. That line means there is no profile because the model does not fit this card — treat it as a stop signal, not a warning to ride past.

Ignoring it cost ~1 hour and ~20 GB on a hybrid Mamba MoE model that then OOM’d: 23.23 GiB in use on a 23.53 GiB card, needing 610 MiB more. MoE weights are large regardless of a modest hidden_size.

The check reads the manifest only and costs seconds. Selecting a model for “already on disk” over “actually fits” is the error it prevents.

⚠️ list-model-profiles is not always free. For one vision model it pulled 6.55 GB. Running it with --rm and no cache mount discards that download.

Video input consumes essentially the whole context

Measured at ~484 tokens per frame on a vision NIM at 4 fps, against a 12,288-token budget:

• Frame count is the cost, not clip length. Ten frames from a 60-second video cost the same as ten from a five-second one. Long clips are fine on a small card; dense sampling is not.
• The lever is per-request: media_io_kwargs.num_frames. The NIM prints the example in its own startup banner.
• ⚠️ The server-side fps env var does nothing. NIM_MEDIA_IO_KWARGS='{"video": {"fps": 2.0}}' is accepted — it appears in the vLLM engine args — and prompt_tokens comes back byte-identical. It looks like it worked, which is worse than failing outright.

Too many frames fails two different ways and both are quiet. A prompt that fits but crowds out the answer returns HTTP 200 with finish_reason: length; a prompt that exceeds max_model_len returns HTTP 400 with no choices[] at all — and curl exits 0 either way. A script that does not check both reports success on both.

The plateau, and why “fewer frames” is not universally right

Past a certain count, frame cost stops rising. Measured independently on two hosts, matching to within one token:

The cap is a total pixel budget in video_preprocessor_config.json, and the arithmetic is exact:

25,165,824 px ÷ 256 (patch 16²) ÷ 4 (merge 2²) ÷ 2 (temporal) = 12,288 tokens

So more frames are nearly free — they buy temporal resolution inside a fixed budget. What actually binds on a small card is max_model_len, which in one case was coincidentally also 12,288, leaving 17 tokens to answer with and creating the appearance of a 24-frame ceiling that was never a frame limit.

⚠️ A number that reproduces across two sites reads like confirmation, and the advice can still invert. The same measurements support “always pass --num-frames 10” on a 12,288-context endpoint and “dense sampling is free” on a 131,072-context one. What binds is a host property neither measurement mentions.

🔴 Tool calling: the environment variables exist, are correctly named, and are ignored

Tested end to end on NIM 2.0.9 / vLLM 0.25.1 with nemotron-3-nano, 2026-08-07. It could not be made to work. Recorded because every surface says it should.

The request fails with:

"auto" tool choice requires --enable-auto-tool-choice and --tool-call-parser to be set

Setting them does not clear it. All three variables were set on the container and verified present:

And nimlib/nim_inference_sdk/vllm.py maps all three to engine args — enable_auto_tool_choice, tool_call_parser, tool_parser_plugin are all in its arg table. The engine still receives none of them, and the startup log contains zero lines mentioning tool.

What to check before spending a restart cycle

• Is the parser name even present in the build? qwen3_coder — the correct parser format for a model emitting <tool_call><function=…><parameter=…> — does not exist anywhere in this vLLM package. grep -rl qwen3_coder returns nothing. A parser name from documentation or a search result is not a parser name your build ships.
• Enumerate what is actually registered. In this build the only one is nemotron_json, and it arrives as a plugin file shipped inside another model’s snapshot (nvidia-nemotron-nano-9b-v2/.../nemotron_toolcall_parser_no_streaming.py), not as part of vLLM. vLLM 0.25.1 in this NIM ships no tool parsers of its own — vllm/tool_parsers/ registry is empty at import.
• Match the parser to what the model actually emits, read from that model’s own chat_template.jinja, not the first one find returns. nemotron-3-nano emits XML (<function=, <parameter=), so nemotron_json would not parse it even if it were wired through.
• ⚠️ A pinned NIM_MODEL_PROFILE is the prime suspect for the variables being ignored — the profile carries an engine configuration that may take precedence. Untested: unpinning changes which weights load.

This is the same shape as the NIM_MEDIA_IO_KWARGS fps variable elsewhere in this note — accepted, plausible, and silently without effect. The lesson generalises: on a NIM, confirm a setting reached the engine (startup log or an API probe) rather than confirming it reached the container. docker inspect showing your variable proves nothing about the engine.

Things that look wrong afterward and are not

• find -perm -0002 reporting dozens of world-writable entries — every one is a symlink, which is always lrwxrwxrwx on Linux because the mode bits are never consulted. The HuggingFace cache layout is built from symlinks. Always pass -type d or -type f, or the count is meaningless.
• Per-profile warnings naming GPUs the host does not have (b300, gh200-144gb, h100-nvl) — that is the manifest enumerating every profile, not a fault. Look beneath them for the single real error.
• A container umask of 0022 makes new files world-readable, never world-writable. A tightened cache does not silently revert.

Diagnosing a crash loop: read mtimes, not live state

A live permissions check ran three minutes after a fix and reported everything healthy — cache mode, ownership, and a successful write test as the container’s uid. All true, all useless.

Passing permissions at the moment you look are not evidence about permissions at the moment the process failed. The fix leaves a timestamp: here the cache mtime was eight seconds after the container’s last death. On a crash loop, read the mtimes of the paths named in the error before trusting a live probe.

Related

• NVIDIA NIM documentation — what NIMs are
• Diagnosing Tool Calling on an NVIDIA NIM — the tool-calling trap above, worked end to end