Misc. bug: llama-server throws "Unsupported param: tools"

[hsm207]

Name and Version

version: 4369 (21ae3b9)
built with Apple clang version 15.0.0 (clang-1500.3.9.4) for arm64-apple-darwin23.6.0

Operating systems

Mac

Which llama.cpp modules do you know to be affected?

llama-server

Problem description & steps to reproduce

llama-server throws:

{"code":500,"message":"Unsupported param: tools","type":"server_error"}

if the request has "tools" parameter.

To reproduce:

1. Start llama-server e.g.

./llama-server \
-m andyywl_Qwen2.5-1.5B-Instruct.Q6_K.gguf_Qwen2.5-1.5B-Instruct.Q6_K.gguf
--port 8080

2. Make a request to the server e.g.

curl http://localhost:8080/v1/chat/completions
-H "Content-Type: application/json"
-H "Authorization: Bearer NOT_APPLICABLE_FOR_LOCAL_MODELS"
-X POST
-d '{
"messages": [
{
"content": "You are a helpful AI assistant. Solve tasks using your tools. Reply with TERMINATE when the task has been completed.",
"role": "system"
},
{
"content": "What is the weather in New York?",
"role": "user",
"name": "user"
}
],
"model": null,
"stream": false,
"tools": [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "",
"parameters": {
"type": "object",
"properties": {
"city": {
"description": "city",
"title": "City",
"type": "string"
}
},
"required": [
"city"
]
}
}
}
]
}'

First Bad Commit

No response

Relevant log output

srv          init: initializing slots, n_slots = 1
slot         init: id  0 | task -1 | new slot n_ctx_slot = 4096
main: model loaded
main: chat template, built_in: 1, chat_example: '<|im_start|>system
You are a helpful assistant<|im_end|>
<|im_start|>user
Hello<|im_end|>
<|im_start|>assistant
Hi there<|im_end|>
<|im_start|>user
How are you?<|im_end|>
<|im_start|>assistant
'
main: server is listening on http://127.0.0.1:8080 - starting the main loop
srv  update_slots: all slots are idle
got exception: {"code":500,"message":"Unsupported param: tools","type":"server_error"}
request: POST /v1/chat/completions 127.0.0.1 500

[okigan]

Running into same issue

[ra1995]

+1, Facing the same issue with the latest build
Using OpenAI API style endpoint with llama-server and qwen 2.5 with tool calling support

request: POST /v1/chat/completions 127.0.0.1 200
got exception: {"code":500,"message":"Unsupported param: tools","type":"server_error"}

[serhatay]

+1, same problem with the latest code. Built with Vulkan support. It's never worked for me.

[vadx64]

Same problem, when llama.cpp server will support tools?

[ra1995]

Did anyone try initializing llama-server using the --jinja param?

[sdesrozis]

--jinja is mandatory, otherwise the server raises:

{"error":{"code":500,"message":"tools param requires --jinja flag","type":"server_error"}}

The answer (llama3.2 but anyway) is:

{"choices":[{"finish_reason":"stop","index":0,"message":{"content":"{\"name\": \"get_current_weather\", \"parameters\": {\"location\": \"New York, USA\", \"format\": \"celsius\"}}","role":"assistant"}}], ...

It sounds that the model can use the schema but the response is not following the openai toolcalling api.

Using ollama, the answer has the good format:

{"choices":[{"index":0,"message":{"role":"assistant","content":"","tool_calls":[{"id":"call_ndym13dn","index":0,"type":"function","function":{"name":"get_current_weather","arguments":"{\"format\":\"celsius\",\"location\":\"New York, USA\"}"}}]},"finish_reason":"tool_calls"}], ...

See this discussion #6429
See this PR #5695

HTH