ayo/ipex-llm
1
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

[NPU]Qwen2 groupwise performance opt (#12299)

Browse source 
* qwen2 gw performance opt

* remove debug
This commit is contained in:
Yina Chen 2024-10-30 11:40:21 +02:00 committed by GitHub
parent 41b8064554
commit 0763268e4c
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
1 changed files with 4 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

3
python/llm/src/ipex_llm/transformers/npu_models/qwen2_mp.py
View file

@ -229,6 +229,9 @@ class LowBitQwenMultiDecoderlayer(LLMBaseNNFactory):
new_value_states = self.convert_to_fp16(curr_key_values[i][1]) new_value_states = self.convert_to_fp16(curr_key_values[i][1])
print(f"{mode} start compiling") print(f"{mode} start compiling")
if group_size != 0 and (mode == "prefill" or num_layers == 2):
self.compile(npu_dpu_groups=6)
else:
self.compile() self.compile()
print(f"{mode} end compiling") print(f"{mode} end compiling")