ayo/ipex-llm
1
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

fix chatglm (#10540)

Browse source
This commit is contained in:
Yishuo Wang 2024-03-26 16:01:00 +08:00 committed by GitHub
parent 2ecd737474
commit 69a28d6b4c
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
1 changed files with 1 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
python/llm/src/ipex_llm/transformers/models/chatglm2.py
View file

@ -184,7 +184,7 @@ def chatglm2_model_forward(
def chatglm2_attention_forward(
self, hidden_states, attention_mask, rotary_pos_emb, kv_cache=None, use_cache=True
):
if use_quantize_kv_cache(self.query_key_value, hidden_states):
if use_quantize_kv_cache(self.query_key_value, hidden_states.transpose(0, 1)):
forward_function = chatglm2_quantized_attention_forward_8eb45c
else:
forward_function = chatglm2_attention_forward_8eb45c