ayo/ipex-llm
1
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

[LLM] vLLM: Delete last_kv_cache before prefilling (#9619)

Browse source 
Remove last_kv_cache before prefilling to reduce peak memory usage.
This commit is contained in:
Xiangyu Tian 2023-12-07 11:32:33 +08:00 committed by GitHub
parent 48b85593b3
commit deee65785c
1 changed files with 2 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
python/llm/src/bigdl/llm/vllm/model_executor/models/bigdl_llama.py
View file

@ -186,6 +186,8 @@ class BigDLLlamaForCausalLM(BigDLModelForCausalLM):
"use_cache": True, "use_cache": True,
# "return_dict": True, # "return_dict": True,
} }
if self.last_kv_cache:
del self.last_kv_cache
# pdb.set_trace() # pdb.set_trace()
if self.device.type == 'xpu': if self.device.type == 'xpu':