ayo/ipex-llm
1
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

Modify readme for bigdl-llm-serving-cpu (#9105)

Browse source
This commit is contained in:
Guancheng Fu 2023-10-09 09:56:09 +08:00 committed by GitHub
parent 2756f9c20d
commit df8df751c4
1 changed files with 2 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
docker/llm/serving/cpu/kubernetes/README.md
View file

@ -15,6 +15,8 @@ After downloading the model, please change name from `vicuna-7b-v1.5` to `vicuna
You can download the model from [here](https://huggingface.co/lmsys/vicuna-7b-v1.5). You can download the model from [here](https://huggingface.co/lmsys/vicuna-7b-v1.5).
For ChatGLM models, users do not need to add `bigdl` into model path. We have already used the `BigDL-LLM` backend for this model.
### Kubernetes config ### Kubernetes config
We recommend to setup your kubernetes cluster before deployment. Mostly importantly, please set `cpu-management-policy` to `static` by using this [tutorial](https://kubernetes.io/docs/tasks/administer-cluster/cpu-management-policies/). Also, it would be great to also set the `topology management policy` to `single-numa-node`. We recommend to setup your kubernetes cluster before deployment. Mostly importantly, please set `cpu-management-policy` to `static` by using this [tutorial](https://kubernetes.io/docs/tasks/administer-cluster/cpu-management-policies/). Also, it would be great to also set the `topology management policy` to `single-numa-node`.