I installed it with
but got this error
doas pkg install py312-vllmbut got this error
Code:
$ vllm serve mistralai/Devstral-Small-2-24B-Instruct-2512 \
--config-format hf \
--load-format safetensors \
--tool-call-parser mistral \
--enable-auto-tool-choice \
--tensor-parallel-size 2 \
--port 8080
/usr/local/lib/python3.12/site-packages/torch/cuda/__init__.py:68: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
import pynvml # type: ignore[import]
W1025 18:19:00.233000 42680 site-packages/torch/_opaque_base.py:6] torch._opaque_base is deprecated, use torch._custom_class_base instead
W1025 18:19:00.234000 42680 site-packages/torch/_library/opaque_object.py:288] register_opaque_type is deprecated, use register_custom_class instead
W1025 18:19:00.234000 42680 site-packages/torch/_library/opaque_object.py:211] typ='value' is deprecated, use typ='constant' instead
(APIServer pid=42680) INFO 10-25 18:19:06 [utils.py:299]
(APIServer pid=42680) INFO 10-25 18:19:06 [utils.py:299] █ █ █▄ ▄█
(APIServer pid=42680) INFO 10-25 18:19:06 [utils.py:299] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.20.2
(APIServer pid=42680) INFO 10-25 18:19:06 [utils.py:299] █▄█▀ █ █ █ █ model mistralai/Devstral-Small-2-24B-Instruct-2512
(APIServer pid=42680) INFO 10-25 18:19:06 [utils.py:299] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀
(APIServer pid=42680) INFO 10-25 18:19:06 [utils.py:299]
(APIServer pid=42680) INFO 10-25 18:19:06 [utils.py:233] non-default args: {'model_tag': 'mistralai/Devstral-Small-2-24B-Instruct-2512', 'enable_auto_tool_choice': True, 'tool_call_parser': 'mistral', 'port': 8080, 'model': 'mistralai/Devstral-Small-2-24B-Instruct-2512', 'config_format': 'hf', 'load_format': 'safetensors', 'tensor_parallel_size': 2}
(APIServer pid=42680) Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
config.json: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.82k/1.82k [00:00<00:00, 939kB/s]
(APIServer pid=42680) Unrecognized keys in `rope_parameters` for 'rope_type'='yarn': {'llama_4_scaling_beta'}
(APIServer pid=42680) Unrecognized keys in `rope_parameters` for 'rope_type'='yarn': {'llama_4_scaling_beta'}
processor_config.json: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 976/976 [00:00<00:00, 1.06MB/s]
(APIServer pid=42680) INFO 10-25 18:19:08 [model.py:555] Resolved architecture: Mistral3ForConditionalGeneration
(APIServer pid=42680) INFO 10-25 18:19:08 [model.py:1680] Using max model len 393216
(APIServer pid=42680) INFO 10-25 18:19:08 [importing.py:68] Triton not installed or not compatible; certain GPU-related functions will not be available.
(APIServer pid=42680) WARNING 10-25 18:19:08 [cpu.py:424] Failed to import from vllm._C_AVX2: ModuleNotFoundError("No module named 'vllm._C_AVX2'")
(APIServer pid=42680) INFO 10-25 18:19:09 [nixl_utils.py:20] Setting UCX_RCACHE_MAX_UNRELEASED to '1024' to avoid a rare memory leak in UCX when using NIXL.
(APIServer pid=42680) WARNING 10-25 18:19:09 [nixl_utils.py:34] NIXL is not available
(APIServer pid=42680) WARNING 10-25 18:19:09 [nixl_utils.py:44] NIXL agent config is not available
(APIServer pid=42680) WARNING 10-25 18:19:09 [cpu.py:424] Failed to import from vllm._C_AVX2: ModuleNotFoundError("No module named 'vllm._C_AVX2'")
(APIServer pid=42680) WARNING 10-25 18:19:09 [cpu.py:424] Failed to import from vllm._C_AVX2: ModuleNotFoundError("No module named 'vllm._C_AVX2'")
(APIServer pid=42680) WARNING 10-25 18:19:09 [cpu.py:424] Failed to import from vllm._C_AVX2: ModuleNotFoundError("No module named 'vllm._C_AVX2'")
(APIServer pid=42680) INFO 10-25 18:19:09 [scheduler.py:239] Chunked prefill is enabled with max_num_batched_tokens=4096.
(APIServer pid=42680) INFO 10-25 18:19:09 [vllm.py:840] Asynchronous scheduling is enabled.
(APIServer pid=42680) INFO 10-25 18:19:09 [kernel.py:205] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'])
params.json: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.19k/1.19k [00:00<00:00, 866kB/s]
consolidated.safetensors.index.json: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 101k/101k [00:00<00:00, 32.7MB/s]
Parse safetensors files: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:01<00:00, 1.29it/s]
(APIServer pid=42680) INFO 10-25 18:19:12 [config.py:303] Inferred from consolidated*.safetensors files torch.bfloat16 dtype.
(APIServer pid=42680) Traceback (most recent call last):
(APIServer pid=42680) File "/usr/local/bin/vllm", line 8, in <module>
(APIServer pid=42680) sys.exit(main())
(APIServer pid=42680) ^^^^^^
(APIServer pid=42680) File "/usr/local/lib/python3.12/site-packages/vllm/entrypoints/cli/main.py", line 92, in main
(APIServer pid=42680) args.dispatch_function(args)
(APIServer pid=42680) File "/usr/local/lib/python3.12/site-packages/vllm/entrypoints/cli/serve.py", line 122, in cmd
(APIServer pid=42680) uvloop.run(run_server(args))
(APIServer pid=42680) File "/usr/local/lib/python3.12/site-packages/uvloop/__init__.py", line 96, in run
(APIServer pid=42680) return __asyncio.run(
(APIServer pid=42680) ^^^^^^^^^^^^^^
(APIServer pid=42680) File "/usr/local/lib/python3.12/asyncio/runners.py", line 195, in run
(APIServer pid=42680) return runner.run(main)
(APIServer pid=42680) ^^^^^^^^^^^^^^^^
(APIServer pid=42680) File "/usr/local/lib/python3.12/asyncio/runners.py", line 118, in run
(APIServer pid=42680) return self._loop.run_until_complete(task)
(APIServer pid=42680) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=42680) File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
(APIServer pid=42680) File "/usr/local/lib/python3.12/site-packages/uvloop/__init__.py", line 48, in wrapper
(APIServer pid=42680) return await main
(APIServer pid=42680) ^^^^^^^^^^
(APIServer pid=42680) File "/usr/local/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 678, in run_server
(APIServer pid=42680) await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)
(APIServer pid=42680) File "/usr/local/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 692, in run_server_worker
(APIServer pid=42680) async with build_async_engine_client(
(APIServer pid=42680) ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=42680) File "/usr/local/lib/python3.12/contextlib.py", line 210, in __aenter__
(APIServer pid=42680) return await anext(self.gen)
(APIServer pid=42680) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=42680) File "/usr/local/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 100, in build_async_engine_client
(APIServer pid=42680) async with build_async_engine_client_from_engine_args(
(APIServer pid=42680) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=42680) File "/usr/local/lib/python3.12/contextlib.py", line 210, in __aenter__
(APIServer pid=42680) return await anext(self.gen)
(APIServer pid=42680) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=42680) File "/usr/local/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 136, in build_async_engine_client_from_engine_args
(APIServer pid=42680) async_llm = AsyncLLM.from_vllm_config(
(APIServer pid=42680) ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=42680) File "/usr/local/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 217, in from_vllm_config
(APIServer pid=42680) return cls(
(APIServer pid=42680) ^^^^
(APIServer pid=42680) File "/usr/local/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 132, in __init__
(APIServer pid=42680) self.renderer = renderer = renderer_from_config(self.vllm_config)
(APIServer pid=42680) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=42680) File "/usr/local/lib/python3.12/site-packages/vllm/renderers/registry.py", line 84, in renderer_from_config
(APIServer pid=42680) tokenizer = cached_tokenizer_from_config(model_config, **kwargs)
(APIServer pid=42680) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=42680) File "/usr/local/lib/python3.12/site-packages/vllm/tokenizers/registry.py", line 261, in cached_tokenizer_from_config
(APIServer pid=42680) return cached_get_tokenizer(
(APIServer pid=42680) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=42680) File "/usr/local/lib/python3.12/site-packages/vllm/tokenizers/registry.py", line 222, in get_tokenizer
(APIServer pid=42680) config = get_config(
(APIServer pid=42680) ^^^^^^^^^^^
(APIServer pid=42680) File "/usr/local/lib/python3.12/site-packages/vllm/transformers_utils/config.py", line 706, in get_config
(APIServer pid=42680) config_dict, config = config_parser.parse(
(APIServer pid=42680) ^^^^^^^^^^^^^^^^^^^^
(APIServer pid=42680) File "/usr/local/lib/python3.12/site-packages/vllm/transformers_utils/config.py", line 308, in parse
(APIServer pid=42680) config = adapt_config_dict(config_dict, defaults=hf_config_dict)
(APIServer pid=42680) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=42680) File "/usr/local/lib/python3.12/site-packages/vllm/transformers_utils/configs/mistral.py", line 84, in adapt_config_dict
(APIServer pid=42680) config_dict = _remap_mistral_vision_args(config_dict)
(APIServer pid=42680) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=42680) File "/usr/local/lib/python3.12/site-packages/vllm/transformers_utils/configs/mistral.py", line 108, in _remap_mistral_vision_args
(APIServer pid=42680) "text_config": PretrainedConfig.from_dict(config),
(APIServer pid=42680) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=42680) File "/usr/local/lib/python3.12/site-packages/transformers/configuration_utils.py", line 744, in from_dict
(APIServer pid=42680) logger.info(f"Model config {config}")
(APIServer pid=42680) ^^^^^^^^
(APIServer pid=42680) File "/usr/local/lib/python3.12/site-packages/transformers/configuration_utils.py", line 827, in __repr__
(APIServer pid=42680) return f"{self.__class__.__name__} {self.to_json_string()}"
(APIServer pid=42680) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=42680) File "/usr/local/lib/python3.12/site-packages/transformers/configuration_utils.py", line 945, in to_json_string
(APIServer pid=42680) return json.dumps(config_dict, indent=2, sort_keys=True) + "\n"
(APIServer pid=42680) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=42680) File "/usr/local/lib/python3.12/json/__init__.py", line 238, in dumps
(APIServer pid=42680) **kw).encode(obj)
(APIServer pid=42680) ^^^^^^^^^^^
(APIServer pid=42680) File "/usr/local/lib/python3.12/json/encoder.py", line 202, in encode
(APIServer pid=42680) chunks = list(chunks)
(APIServer pid=42680) ^^^^^^^^^^^^
(APIServer pid=42680) File "/usr/local/lib/python3.12/json/encoder.py", line 432, in _iterencode
(APIServer pid=42680) yield from _iterencode_dict(o, _current_indent_level)
(APIServer pid=42680) File "/usr/local/lib/python3.12/json/encoder.py", line 406, in _iterencode_dict
(APIServer pid=42680) yield from chunks
(APIServer pid=42680) File "/usr/local/lib/python3.12/json/encoder.py", line 439, in _iterencode
(APIServer pid=42680) o = _default(o)
(APIServer pid=42680) ^^^^^^^^^^^
(APIServer pid=42680) File "/usr/local/lib/python3.12/json/encoder.py", line 180, in default
(APIServer pid=42680) raise TypeError(f'Object of type {o.__class__.__name__} '
(APIServer pid=42680) TypeError: Object of type set is not JSON serializable