ayo/ipex-llm
1
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

fix vLLM/docker issues (#11348)

Browse source 
* fix

* fix

* ffix
This commit is contained in:
Guancheng Fu 2024-06-18 16:23:53 +08:00 committed by GitHub
parent 83082e5cc7
commit c9b4cadd81
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
3 changed files with 7 additions and 5 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

3
docker/llm/serving/xpu/docker/Dockerfile
View file

@ -21,7 +21,8 @@ RUN apt-get update && \
pip install outlines==0.0.34 --no-deps && \ pip install outlines==0.0.34 --no-deps && \
pip install interegular cloudpickle diskcache joblib lark nest-asyncio numba scipy && \ pip install interegular cloudpickle diskcache joblib lark nest-asyncio numba scipy && \
# For Qwen series models support # For Qwen series models support
pip install transformers_stream_generator einops tiktoken pip install transformers_stream_generator einops tiktoken && \
pip uninstall -y deepspeed
COPY ./vllm_offline_inference.py /llm/ COPY ./vllm_offline_inference.py /llm/
COPY ./payload-1024.lua /llm/ COPY ./payload-1024.lua /llm/

2
docker/llm/serving/xpu/docker/benchmark_vllm_throughput.py
View file

@ -345,7 +345,7 @@ if __name__ == "__main__":
parser.add_argument( parser.add_argument(
"--load-in-low-bit", "--load-in-low-bit",
type=str, type=str,
choices=["sym_int4", "fp8", "fp16"], choices=["sym_int4", "fp8", "fp8_e4m3", "fp16", "fp6"],
default="sym_int4", default="sym_int4",
help="Low-bit format quantization with IPEX-LLM") help="Low-bit format quantization with IPEX-LLM")

5
docs/readthedocs/source/doc/LLM/Quickstart/vLLM_quickstart.md
View file

@ -58,7 +58,7 @@ To run offline inference using vLLM for a quick impression, use the following ex
.. note:: .. note::
Please modify the MODEL_PATH in offline_inference.py to use your chosen model. Please modify the MODEL_PATH in offline_inference.py to use your chosen model.
You can try modify load_in_low_bit to different values in **[sym_int4, fp8, fp16]** to use different quantization dtype. You can try modify load_in_low_bit to different values in **[sym_int4, fp6, fp8, fp8_e4m3, fp16]** to use different quantization dtype.
``` ```
```bash ```bash
@ -111,6 +111,7 @@ served_model_name="YOUR_MODEL_NAME"
# --max-model-len, --max-num-batched-tokens, --max-num-seqs # --max-model-len, --max-num-batched-tokens, --max-num-seqs
# to acquire the best performance # to acquire the best performance
# Change value --load-in-low-bit to [fp6, fp8, fp8_e4m3, fp16] to use different low-bit formats
python -m ipex_llm.vllm.xpu.entrypoints.openai.api_server \ python -m ipex_llm.vllm.xpu.entrypoints.openai.api_server \
--served-model-name $served_model_name \ --served-model-name $served_model_name \
--port 8000 \ --port 8000 \
@ -245,7 +246,7 @@ wget https://raw.githubusercontent.com/intel-analytics/ipex-llm/main/docker/llm/
export MODEL="YOUR_MODEL" export MODEL="YOUR_MODEL"
# You can change load-in-low-bit from values in [sym_int4, fp8, fp16] # You can change load-in-low-bit from values in [sym_int4, fp6, fp8, fp8_e4m3, fp16]
python3 ./benchmark_throughput.py \ python3 ./benchmark_throughput.py \
--backend vllm \ --backend vllm \