ayo/ipex-llm
1
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

Update README.md (#12877)

Browse source
This commit is contained in:
Guancheng Fu 2025-02-24 09:59:17 +08:00 committed by GitHub
parent 10400abfb7
commit 02ec313eab
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
1 changed files with 7 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

7
python/llm/example/GPU/vLLM-Serving/README.md
View file

@ -241,3 +241,10 @@ llm = LLM(model="DeepSeek-R1-Distill-Qwen-7B", # Unquantized model path on disk
When finish executing, the low-bit model has been saved at `/llm/fp8-model-path`. When finish executing, the low-bit model has been saved at `/llm/fp8-model-path`.
Later we can use the option `--low-bit-model-path /llm/fp8-model-path` to use the low-bit model. Later we can use the option `--low-bit-model-path /llm/fp8-model-path` to use the low-bit model.
### 5. Known issues
#### Runtime memory
If runtime memory is a concern, you can set --swap-space 0.5 to reduce memory consumption during execution. The default value for --swap-space is 4, which means that by default, the system reserves 4GB of memory for use when GPU memory is insufficient.