ayo/ipex-llm
1
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

LLM: fix chatglm kv cache (#9215)

Browse source
This commit is contained in:
Ruonan Wang 2023-10-18 19:09:53 +08:00 committed by GitHub
parent 0765f94770
commit 942d6418e7
1 changed files with 2 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
python/llm/src/bigdl/llm/transformers/models/chatglm.py
View file

@ -77,6 +77,8 @@ def attention_fn(
device=device) device=device)
new_cache_k[:] = cache_k new_cache_k[:] = cache_k
new_cache_v[:] = cache_v new_cache_v[:] = cache_v
cache_k = new_cache_k
cache_v = new_cache_v
key_layer, value_layer = append_kv_cache(cache_k, cache_v, key_layer, value_layer) key_layer, value_layer = append_kv_cache(cache_k, cache_v, key_layer, value_layer)
elif use_cache: elif use_cache: