ayo/ipex-llm
1
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

add --blocksize to doc and script (#12187)

Browse source
This commit is contained in:
Shaojun Liu 2024-10-12 09:17:42 +08:00 committed by GitHub
parent 6ffaec66a2
commit 49eb20613a
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
2 changed files with 2 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

1
docker/llm/serving/xpu/docker/start-vllm-service.sh
View file

@ -19,6 +19,7 @@ python -m ipex_llm.vllm.xpu.entrypoints.openai.api_server \
--port 8000 \ --port 8000 \
--model $model \ --model $model \
--trust-remote-code \ --trust-remote-code \
--block-size 8 \
--gpu-memory-utilization 0.9 \ --gpu-memory-utilization 0.9 \
--device xpu \ --device xpu \
--dtype float16 \ --dtype float16 \

1
docs/mddocs/DockerGuides/vllm_docker_quickstart.md
View file

@ -103,6 +103,7 @@ Before performing benchmark or starting the service, you can refer to this [sect
|`--max-model-len`| Model context length. If unspecified, will be automatically derived from the model config.| |`--max-model-len`| Model context length. If unspecified, will be automatically derived from the model config.|
|`--max-num-batched-token`| Maximum number of batched tokens per iteration.| |`--max-num-batched-token`| Maximum number of batched tokens per iteration.|
|`--max-num-seq`| Maximum number of sequences per iteration. Default: 256| |`--max-num-seq`| Maximum number of sequences per iteration. Default: 256|
|`--block-size`| vLLM block size. Set to 8 to achieve a performance boost.|
#### Single card serving #### Single card serving