Diagnosing Tool Calling on an NVIDIA NIM

A method for finding out why a NIM refuses to call tools, and for deciding in about ten minutes whether a given NIM build can do it at all. Written after a full diagnostic pass on NIM 2.0.9 / vLLM 0.25.1, which ended in a negative result — the method is the point, and it works whether the answer is yes or no.

Sanitized-shareable. Hosts, paths and model names in the body are placeholders. Real specifics for this estate are in the appendix.

Why this is hard, and why it is not obvious

Almost everyone assumes “tool calling” is one feature the model either has or does not have. It is not. It is two independent mechanisms that must agree, and they live in different places:

1. The model produces ordinary text in a particular convention when it decides to call a tool.
2. The server runs a parser that recognises that convention and converts it into the structured tool_calls JSON your agent framework expects.

If either half is missing or they disagree, you get a model that is perfectly capable of calling tools and an API that never reports one. The model is not broken. The server is not broken. They are simply not introduced to each other.

The closest familiar analogy is scanf. The model is a program writing to stdout in some format. The parser is your scanf format string. A correct program and a correct scanf still produce garbage if the format string does not match what is actually being printed. Nobody would call that “the program does not support output.”

This is why Ollama appears to “just work” and a NIM does not — see Why Ollama is different near the end. It is a packaging difference, not a capability difference, and it explains why an agent demo that runs on Ollama stalls the moment you point it at a NIM.

Part 1 — The three things that must line up

Number 3 is the dangerous one. The first two fail loudly with an HTTP error. The third succeeds — you get a normal-looking response, finish_reason: stop, and the tool call sitting in the message body as plain text that your agent framework ignores. An agent built on that silently does nothing and reports success.

On a NIM you do not set those flags directly. The NIM is a wrapper: it reads environment variables and builds the server’s command line for you. So there is a fourth thing that must hold, and it is the one that bit us:

4. The NIM must actually translate your environment variable into the server flag. docker inspect showing your variable proves the container received it. It proves nothing about whether the inference engine did.

Part 2 — The diagnostic procedure

Seven steps, in order. Each one either produces an answer or eliminates a cause. Do not skip to changing settings — steps 1–4 are all read-only and take a few minutes.

Step 1 — Reproduce the failure, and classify it

Send one request with a tool definition. Use the smallest possible tool.

curl -s localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "<model-id>",
    "messages": [{"role": "user",
      "content": "What is the weather in Phoenix? Use the tool."}],
    "tools": [{"type":"function","function":{
      "name":"get_weather",
      "description":"Get current weather for a city",
      "parameters":{"type":"object",
        "properties":{"city":{"type":"string"}},
        "required":["city"]}}}],
    "tool_choice": "auto",
    "max_tokens": 400
  }'

Three outcomes, three different problems:

⚠️ Give it enough max_tokens. Reasoning models spend tokens thinking before they emit the call. A tight limit produces a truncated response that looks like a refusal. If finish_reason is length, raise the limit and retry before concluding anything.

Step 2 — Read the error literally, then check the variables

The error names two flags. It means both, not either — enabling tool choice without naming a parser produces the identical message, which misleads you into thinking your setting did not take.

Find the variable names from the NIM itself rather than from a blog post:

docker exec <nim-container> /opt/nim/.venv/bin/python -c \
  "import nimlib; print([n for n in dir(nimlib) if 'TOOL' in n])"

Then read the value each name maps to. On the build tested, these were the real names:

• NIM_ENABLE_AUTO_TOOL_CHOICE
• NIM_TOOL_CALL_PARSER
• NIM_TOOL_PARSER_PLUGIN — for loading a parser that is not built in

Set them, restart the container, and re-run Step 1. If the error persists with all three set, you are in the fourth failure mode — jump to Step 6.

Step 3 — Find out what the model actually emits

This is the step people skip, and it decides everything. You are looking for the model’s own chat template, which is the file that tells the model how to format a tool call.

docker exec <nim-container> sh -c \
  'find /opt/nim/.cache -path "*<your-model-name>*" -name chat_template.jinja'

🔴 Do not use find … | head -1. A NIM cache can hold several models, and you will read the wrong one and draw a confident wrong conclusion. Filter by your model’s directory name. This exact mistake cost a wrong parser recommendation during the pass this document is based on.

Then look at the format inside it:

docker exec <nim-container> sh -c \
  'grep -ohE "<function=|<parameter=|<tool_call>" \
     <path-to-template> | sort | uniq -c'

The two families you will see:

Both begin with <tool_call>. That is why the token alone is not enough to identify the format, and why reading only the token list leads you to the wrong parser. Read the template body.

Step 4 — Find out what parsers the build actually has

A parser name from documentation, a model card, or a search result is not a parser name your build ships. Verify:

# every parser name registered anywhere in the image
docker exec <nim-container> sh -c \
  'grep -rhoE "register_module\\(\\[?\"[a-z0-9_]+\"" \
     /opt/nim 2>/dev/null' \
  | grep -oE '"[a-zA-Z0-9_]+"' | tr -d '"' | sort -u

# and confirm a specific candidate exists at all
docker exec <nim-container> sh -c \
  'grep -rl "<candidate-name>" /opt/nim 2>/dev/null | head'

If the second command returns nothing, that parser is not in your build. Setting it will not error — it will simply do nothing, which is indistinguishable from the setting being ignored.

Parsers can also arrive as plugin files inside a model’s snapshot directory, not as part of the engine:

docker exec <nim-container> sh -c \
  'find /opt/nim/.cache -iname "*toolcall*parser*.py" -o -iname "*tool_parser*.py"'

If you find one, open it — the name it registers is the string you pass:

docker exec <nim-container> sh -c 'grep -n "register_module" <plugin-path>'
# -> @ToolParserManager.register_module("nemotron_json")

Load it with NIM_TOOL_PARSER_PLUGIN=<path-to-that-file>.

Step 5 — Decide whether a match exists

Put Step 3 and Step 4 side by side.

• Match found → set the three variables, restart, re-run Step 1.
• No match → stop. No amount of configuration will help. The build does not contain a parser for what this model emits. Your options are a different NIM version, a different model, or writing a parser plugin.

Recording a “no match” is a real result. It is the difference between “we could not get it working” and “this build cannot do it, here is the evidence” — and only the second one is actionable with a vendor.

Step 6 — Confirm the setting reached the engine, not just the container

This is the check that separates a configuration problem from a product problem, and it is the one nobody runs.

# the container has your variable...
docker inspect <nim-container> \
  --format '{{range .Config.Env}}{{println .}}{{end}}' | grep -i tool

# ...but did the engine hear about it?
docker logs <nim-container> 2>&1 | grep -ic tool

If the container has the variables and the startup log contains zero lines mentioning tool, the NIM did not pass them through. That is not your mistake to fix by trying more values.

The parallel from ordinary systems work: exporting a variable in a wrapper script proves nothing about the child process’s argv. You check ps -eo args, not env. The startup log is the NIM’s equivalent of ps — it is where you see what the engine was really launched with.

Step 7 — 🔴 The ten-second test that settles it: does the engine-args class even have the fields?

Run this first if you only run one thing. It answers the whole question without a restart, without touching the running service, and without guessing at parser names:

docker exec <nim-container> /opt/nim/.venv/bin/python -c "
import nimlib.llm.arg_utils as a, dataclasses
print([f.name for f in dataclasses.fields(a.NimAsyncEngineArgs) if 'tool' in f.name])"

An empty list means this NIM build cannot do tool calling by configuration, at all. Not with a different parser, not with a plugin, not with a different profile.

Why this is decisive. The NIM reads your environment variables into a dictionary, then applies them to an engine-arguments object by attribute name. The loop is literally this:

for field_name, env_value in env_defaults.items():
    if hasattr(engine_args, field_name):
        setattr(engine_args, field_name, env_value)
    else:
        logger.warning(f"Field '{field_name}' not found in NimAsyncEngineArgs")

If the class does not define the field, the value is dropped and a warning is logged. Your variable was read, understood, and thrown away. Nothing downstream ever sees it.

In C terms: the wrapper parses your --flag, looks it up in a table of known options, finds no entry, prints a warning to stderr that nobody reads, and calls the child without it. The flag was never malformed. The table was incomplete.

You can also catch this without any Python — the warnings appear in the container’s own startup output and in list-model-profiles:

WARNING arg_utils.py] Field 'enable_auto_tool_choice' not found in NimAsyncEngineArgs
WARNING arg_utils.py] Field 'tool_call_parser'       not found in NimAsyncEngineArgs
WARNING arg_utils.py] Field 'tool_parser_plugin'     not found in NimAsyncEngineArgs

⚠️ These are WARNING, not ERROR, and they scroll past in a wall of startup logging. They are the single most important lines in the output and they look like noise.

Step 8 — Only if Step 7 passes: consider the pinned profile

If the fields do exist and it still fails, a pinned NIM_MODEL_PROFILE carries its own engine configuration that may take precedence.

docker exec <nim-container> sh -c 'list-model-profiles'

🔴 Read the output before unpinning — it can be actively dangerous. On the box tested, the profile list showed Compatible with system and runnable: <None>, because the only GPU was already 92% occupied by the running model. The only alternatives to the pinned fp8 profile were bf16 at 63 GB against a 46 GB card. Unpinning would have left the NIM unable to select any profile and failing to start — trading a working service with no tool calling for no service at all.

Check Free GPUs and the memory figure in each profile name (vllm-fp8-tp1-pp1-**34.0** = 34 GB) before changing this.

Part 3 — What this method found (NIM 2.0.9 / vLLM 0.25.1)

A clean negative, established in about forty minutes and three container restarts.

The verdict

enable_auto_tool_choice, tool_call_parser and tool_parser_plugin are ABSENT from the engine-arguments class in NIM 2.0.9. So is chat_template. reasoning_parser and max_model_len are present, which is why those environment variables work — the mechanism is sound, the table is simply incomplete.

This is not a configuration error, a parser-selection error, or a model problem. The plumbing is half-built: the variable names are defined in nimlib, the argument table in nim_inference_sdk/vllm.py references them, and the dataclass they would be written to has no such fields. Every surface says the feature exists except the one that matters.

Three consequences, worth separating because they have different remedies:

• No amount of configuration will enable tool calling on this build. Not a different parser, not a plugin, not a different profile. Stop trying values.
• The parser for this model’s format is separately missing — qwen3_coder is not in the image. So even once the wiring is fixed, a matching parser has to arrive too.
• The remedy is a NIM version, not a setting. This is a clean, reproducible vendor report: version, model, the three absent fields, the warning lines, and the code path that drops them.

Part 4 — Why Ollama is different

Not because Ollama is more capable. Because it does not make the parser your problem.

Ollama ships the template and the parser as a matched pair and picks them for you. The NIM exposes them as independent knobs — more flexible, and it means a working combination has to be assembled rather than assumed.

So “I have to use Ollama for agent demos” is a reasonable engineering position, not a workaround. The honest framing for an audience: tool calling requires the serving layer to parse the model’s output, and the two must be matched. Some stacks match them for you; some make it configurable, and configurable means it can be misconfigured.

Part 5 — The ten-minute checklist for a new NIM version

Run this before building anything on a NIM’s tool calling.

1. 🔴 Run the ten-second field check (Step 7) FIRST. Empty list → this build cannot do it. Stop; everything below is wasted effort.
2. Send one tool request. Populated tool_calls? Done.
3. Read the model’s own chat template. JSON-in-tag, or XML function/parameter?
4. List the parsers the image actually contains. Does one match?
5. If no match → stop. Record the version, the model, the format, and the parser list. That is a vendor conversation, not a config problem.
6. If a match → set the three variables and restart.
7. Re-test. Still failing? Check the startup log for tool. Zero lines means the NIM did not wire it through.
8. Only then consider the model profile — and read list-model-profiles first. If Free GPUs is <None> or the alternatives exceed your card, unpinning breaks the service instead of fixing it.

The single most useful habit: after any change, verify at the engine, not the container. docker inspect tells you what you asked for. The startup log and an actual API probe tell you what you got.