ayo/ipex-llm
1
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

Disable lm head (#11972)

Browse source
This commit is contained in:
binbin Deng 2024-08-30 11:05:18 +08:00 committed by GitHub
parent 7d103417b8
commit cd077881f1
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
2 changed files with 4 additions and 3 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

4
python/llm/example/NPU/HF-Transformers-AutoModels/LLM/README.md
View file

@ -145,8 +145,8 @@ python minicpm.py --disable-transpose-value-cache
python minicpm.py --repo-id-or-model-path openbmb/MiniCPM-2B-sft-bf16 --disable-transpose-value-cache
```
#### High CPU Utilization
You can reduce CPU utilization by setting the environment variable with `set IPEX_LLM_CPU_LM_HEAD=0`.
#### Better Performance with High CPU Utilization
You could enable optimization by setting the environment variable with `set IPEX_LLM_CPU_LM_HEAD=1` for better performance. But this will cause high CPU utilization.
### Sample Output

3
python/llm/src/ipex_llm/transformers/npu_models/convert_mp.py
View file

@ -43,7 +43,8 @@ def optimize_llm_pre(model: torch.nn.Module, qtype):
model.apply(pre_compute_inv_freq)
# lm_head to cpu optimization
if os.environ.get("IPEX_LLM_CPU_LM_HEAD", "1") != "0":
if os.environ.get("IPEX_LLM_CPU_LM_HEAD", "0") != "0":
# disable the optimization by default
from ipex_llm.transformers.low_bit_linear import SYM_INT4, SYM_INT8
if qtype == "sym_int4_rtn":
lm_qtype = SYM_INT4