Repository navigation
latest model runner seems to have a few issues #969
Description
Activity
I encountered the same problem in version v1.2.3 of docker/model-runner; with this version, my GPU is not recognized. When I revert to version v1.2.2, the problem does not occur.
Log of version v1.2.2 where the problem does not occur:
[docker-model-runner] 2026-06-13T16:24:12.674331695Z time=2026-06-13T16:24:12.674Z level=INFO msg=LLAMA_SERVER_PATH path=/app [docker-model-runner] 2026-06-13T16:24:12.674568648Z time=2026-06-13T16:24:12.674Z level=INFO msg="Successfully initialized store" component=model-manager [docker-model-runner] 2026-06-13T16:24:12.674839178Z time=2026-06-13T16:24:12.674Z level=INFO msg="Logs endpoint enabled at /logs" dir=/tmp/model-runner-logs [docker-model-runner] 2026-06-13T16:24:12.674849500Z time=2026-06-13T16:24:12.674Z level=INFO msg="Metrics endpoint enabled at /metrics" [docker-model-runner] 2026-06-13T16:24:12.674873015Z time=2026-06-13T16:24:12.674Z level=INFO msg="Listening on TCP port" port=12434 [docker-model-runner] 2026-06-13T16:24:12.714996169Z time=2026-06-13T16:24:12.714Z level=INFO msg="Failed to ensure latest llama.cpp" error="llama.cpp auto-updated is disabled" [docker-model-runner] 2026-06-13T16:24:12.765029881Z time=2026-06-13T16:24:12.764Z level=INFO msg="installed llama-server" gpuSupport=true [docker-model-runner] 2026-06-13T16:24:12.765045034Z time=2026-06-13T16:24:12.764Z level=WARN msg="Backend installation failed for" backend=mlx error="MLX is only available on macOS ARM64" [docker-model-runner] 2026-06-13T16:24:12.765048996Z time=2026-06-13T16:24:12.765Z level=WARN msg="Backend installation failed for" backend=vllm error="vLLM binary not found" [docker-model-runner] 2026-06-13T16:24:12.765425672Z time=2026-06-13T16:24:12.765Z level=WARN msg="Backend installation failed for" backend=sglang error="python3 not found in PATH" [docker-model-runner] 2026-06-13T16:24:35.657633400Z time=2026-06-13T16:24:35.657Z level=INFO msg="getting model by reference" component=model-manager reference=llama3.2 [docker-model-runner] 2026-06-13T16:24:35.657979791Z time=2026-06-13T16:24:35.657Z level=INFO msg="getting model by reference" component=model-manager reference=llama3.2 [docker-model-runner] 2026-06-13T16:24:35.658072057Z time=2026-06-13T16:24:35.658Z level=INFO msg="Loading backend runner" backend=llama.cpp model=sha256:436bb282b41968a83638482999980267ca8d7e8b5574604460efa9efff11cf59 mode=completion [docker-model-runner] 2026-06-13T16:24:35.658094888Z time=2026-06-13T16:24:35.658Z level=INFO msg="Listing available models" component=model-manager [docker-model-runner] 2026-06-13T16:24:35.658149999Z time=2026-06-13T16:24:35.658Z level=INFO msg="successfully listed models" component=model-manager count=1 [docker-model-runner] 2026-06-13T16:24:35.658470210Z time=2026-06-13T16:24:35.658Z level=INFO msg="getting model by reference" component=model-manager reference=sha256:436bb282b41968a83638482999980267ca8d7e8b5574604460efa9efff11cf59 [docker-model-runner] 2026-06-13T16:24:35.658488450Z time=2026-06-13T16:24:35.658Z level=INFO msg="Listing available models" component=model-manager [docker-model-runner] 2026-06-13T16:24:35.658753541Z time=2026-06-13T16:24:35.658Z level=INFO msg="successfully listed models" component=model-manager count=1 [docker-model-runner] 2026-06-13T16:24:35.659262864Z time=2026-06-13T16:24:35.659Z level=INFO msg="Listing available models" component=model-manager [docker-model-runner] 2026-06-13T16:24:35.659859999Z time=2026-06-13T16:24:35.659Z level=INFO msg="successfully listed models" component=model-manager count=1 [docker-model-runner] 2026-06-13T16:24:35.660288029Z time=2026-06-13T16:24:35.660Z level=INFO msg="backend args" backend=llama.cpp args="[-ngl 999 --metrics --model /models/bundles/sha256/436bb282b41968a83638482999980267ca8d7e8b5574604460efa9efff11cf59/model/model.gguf --host inference-runner-0.sock --jinja]" [docker-model-runner] 2026-06-13T16:24:35.699405410Z time=2026-06-13T16:24:35.699Z level=INFO msg="warn: LLAMA_ARG_HOST environment variable is set, but will be overwritten by command line argument --host" [docker-model-runner] 2026-06-13T16:24:35.699553843Z time=2026-06-13T16:24:35.699Z level=INFO msg="0.00.035.440 I log_info: verbosity = 3 (adjust with the `-lv N` CLI arg)" [docker-model-runner] 2026-06-13T16:24:35.699562465Z time=2026-06-13T16:24:35.699Z level=INFO msg="0.00.035.442 I device_info:" [docker-model-runner] 2026-06-13T16:24:35.699708863Z time=2026-06-13T16:24:35.699Z level=INFO msg="0.00.035.620 I - Vulkan0 : AMD Radeon RX 6600 (RADV NAVI23) (8176 MiB, 7183 MiB free)" [docker-model-runner] 2026-06-13T16:24:35.699896553Z time=2026-06-13T16:24:35.699Z level=INFO msg="0.00.035.788 I - Vulkan1 : Intel(R) UHD Graphics 730 (ADL-S GT1) (23179 MiB, 20861 MiB free)" [docker-model-runner] 2026-06-13T16:24:35.699899719Z time=2026-06-13T16:24:35.699Z level=INFO msg="0.00.035.791 I - CPU : 12th Gen Intel(R) Core(TM) i5-12400 (30905 MiB, 30905 MiB free)" [docker-model-runner] 2026-06-13T16:24:35.699921020Z time=2026-06-13T16:24:35.699Z level=INFO msg="0.00.035.814 I system_info: n_threads = 6 (n_threads_batch = 6) / 12 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX_VNNI = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | " [docker-model-runner] 2026-06-13T16:24:35.699927223Z time=2026-06-13T16:24:35.699Z level=INFO msg="0.00.035.817 I srv llama_server: n_parallel is set to auto, using n_parallel = 4 and kv_unified = true"Log from version v1.2.3 where the GPU is not identified:
[docker-model-runner] 2026-06-13T16:25:54.788825004Z time=2026-06-13T16:25:54.788Z level=INFO msg=LLAMA_SERVER_PATH path=/app [docker-model-runner] 2026-06-13T16:25:54.789035227Z time=2026-06-13T16:25:54.789Z level=INFO msg="Successfully initialized store" component=model-manager [docker-model-runner] 2026-06-13T16:25:54.789319541Z time=2026-06-13T16:25:54.789Z level=INFO msg="Logs endpoint enabled at /logs" dir=/tmp/model-runner-logs [docker-model-runner] 2026-06-13T16:25:54.789325452Z time=2026-06-13T16:25:54.789Z level=INFO msg="Metrics endpoint enabled at /metrics" [docker-model-runner] 2026-06-13T16:25:54.789329168Z time=2026-06-13T16:25:54.789Z level=INFO msg="Listening on TCP port" port=12434 [docker-model-runner] 2026-06-13T16:25:54.817909339Z time=2026-06-13T16:25:54.817Z level=WARN msg="sglang package not found. Install with: uv pip install sglang" [docker-model-runner] 2026-06-13T16:25:54.817922163Z time=2026-06-13T16:25:54.817Z level=WARN msg="Backend installation failed for" backend=sglang error="sglang package not installed" [docker-model-runner] 2026-06-13T16:25:54.828696908Z time=2026-06-13T16:25:54.828Z level=INFO msg="Failed to ensure latest llama.cpp" error="llama.cpp auto-updated is disabled" [docker-model-runner] 2026-06-13T16:25:54.864013913Z time=2026-06-13T16:25:54.863Z level=INFO msg="installed llama-server" gpuSupport=false [docker-model-runner] 2026-06-13T16:25:54.864037570Z time=2026-06-13T16:25:54.864Z level=WARN msg="Backend installation failed for" backend=mlx error="MLX is only available on macOS ARM64" [docker-model-runner] 2026-06-13T16:25:54.864040777Z time=2026-06-13T16:25:54.864Z level=WARN msg="Backend installation failed for" backend=vllm error="vLLM binary not found" [docker-model-runner] 2026-06-13T16:26:09.565882101Z time=2026-06-13T16:26:09.565Z level=INFO msg="getting model by reference" component=model-manager reference=llama3.2 [docker-model-runner] 2026-06-13T16:26:09.567585698Z time=2026-06-13T16:26:09.566Z level=INFO msg="getting model by reference" component=model-manager reference=llama3.2 [docker-model-runner] 2026-06-13T16:26:09.567619798Z time=2026-06-13T16:26:09.566Z level=INFO msg="Loading backend runner" backend=llama.cpp model=sha256:436bb282b41968a83638482999980267ca8d7e8b5574604460efa9efff11cf59 mode=completion [docker-model-runner] 2026-06-13T16:26:09.567626753Z time=2026-06-13T16:26:09.567Z level=INFO msg="Listing available models" component=model-manager [docker-model-runner] 2026-06-13T16:26:09.567631174Z time=2026-06-13T16:26:09.567Z level=INFO msg="successfully listed models" component=model-manager count=1 [docker-model-runner] 2026-06-13T16:26:09.567984597Z time=2026-06-13T16:26:09.567Z level=INFO msg="getting model by reference" component=model-manager reference=sha256:436bb282b41968a83638482999980267ca8d7e8b5574604460efa9efff11cf59 [docker-model-runner] 2026-06-13T16:26:09.568007671Z time=2026-06-13T16:26:09.567Z level=INFO msg="Listing available models" component=model-manager [docker-model-runner] 2026-06-13T16:26:09.568148621Z time=2026-06-13T16:26:09.568Z level=INFO msg="successfully listed models" component=model-manager count=1 [docker-model-runner] 2026-06-13T16:26:09.568744893Z time=2026-06-13T16:26:09.568Z level=INFO msg="Listing available models" component=model-manager [docker-model-runner] 2026-06-13T16:26:09.568762743Z time=2026-06-13T16:26:09.568Z level=INFO msg="successfully listed models" component=model-manager count=1 [docker-model-runner] 2026-06-13T16:26:09.569698662Z time=2026-06-13T16:26:09.569Z level=INFO msg="backend args" backend=llama.cpp args="[-ngl 999 --metrics --model /models/bundles/sha256/436bb282b41968a83638482999980267ca8d7e8b5574604460efa9efff11cf59/model/model.gguf --host inference-runner-0.sock --jinja]" [docker-model-runner] 2026-06-13T16:26:09.596382506Z time=2026-06-13T16:26:09.596Z level=INFO msg="warning: no usable GPU found, --gpu-layers option will be ignored" [docker-model-runner] 2026-06-13T16:26:09.596398765Z time=2026-06-13T16:26:09.596Z level=INFO msg="warning: one possible reason is that llama.cpp was compiled without GPU support" [docker-model-runner] 2026-06-13T16:26:09.596408616Z time=2026-06-13T16:26:09.596Z level=INFO msg="warning: consult docs/build.md for compilation instructions" [docker-model-runner] 2026-06-13T16:26:09.596416098Z time=2026-06-13T16:26:09.596Z level=INFO msg="warn: LLAMA_ARG_HOST environment variable is set, but will be overwritten by command line argument --host" [docker-model-runner] 2026-06-13T16:26:09.596638074Z time=2026-06-13T16:26:09.596Z level=INFO msg="0.00.018.263 I log_info: verbosity = 3 (adjust with the `-lv N` CLI arg)" [docker-model-runner] 2026-06-13T16:26:09.596651784Z time=2026-06-13T16:26:09.596Z level=INFO msg="0.00.018.266 I device_info:" [docker-model-runner] 2026-06-13T16:26:09.596657551Z time=2026-06-13T16:26:09.596Z level=INFO msg="0.00.018.275 I - CPU : 12th Gen Intel(R) Core(TM) i5-12400 (30905 MiB, 30905 MiB free)" [docker-model-runner] 2026-06-13T16:26:09.596667634Z time=2026-06-13T16:26:09.596Z level=INFO msg="0.00.018.302 I system_info: n_threads = 6 (n_threads_batch = 6) / 12 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX_VNNI = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | " [docker-model-runner] 2026-06-13T16:26:09.596672676Z time=2026-06-13T16:26:09.596Z level=INFO msg="0.00.018.308 I srv llama_server: n_parallel is set to auto, using n_parallel = 4 and kv_unified = true"If you need more information I can provide it; I can simulate the problem by changing the image version.
@danielsoares02 English not Portuguese please... Although I'd love to practise my Portuguese, we have to standardise on English
@ericcurtin, translated into English for easier reading.
Sorry for any grammatical errors, English is not my native language.
This bundles a few unrelated problems, please split into separate issues with repro details. The ROCm detection regression looks real and worth a fix on its own. Model-runner is being deprecated in favor of llmman, please open these against docker/llmman instead.
1st) "creating temporary files seems to be broken"
2nd) the default "auto" option for the --gpu doesn't detect rocm anymore and switches to "none" aka cpu decoding
3rd) when using --gpu rocm --host 0.0.0.0 is ignored ( instead 2 mappings are done -> 127.0.0.1 and 172 ...... )