ayo/ipex-llm
1
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

LLM: fix the performance drop of starcoder (#9889)

Browse source 
* LLM: fix the performance drop of starcoder

* small fix

* small fix
This commit is contained in:
WeiguangHan 2024-01-12 09:14:15 +08:00 committed by GitHub
parent d9cf55bce9
commit 0e69bfe6b0
1 changed files with 3 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

3
python/llm/dev/benchmark/all-in-one/run.py
View file

@ -375,8 +375,11 @@ def run_transformer_int4_gpu(repo_id,
model = model.to('xpu')
else:
if 'starcoder' in repo_id:
# Load starcoder-15.5b model in bf16 format to avoid CPU OOM.
model = AutoModelForCausalLM.from_pretrained(model_path, optimize_model=True, load_in_low_bit=low_bit,
trust_remote_code=True, use_cache=True, torch_dtype=torch.bfloat16).eval()
# Convert the low-bit model back to fp32 for performance considerations.
model = model.float()
else:
model = AutoModelForCausalLM.from_pretrained(model_path, optimize_model=True, load_in_low_bit=low_bit,
trust_remote_code=True, use_cache=True).eval()