ayo/ipex-llm
1
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

Add lookup generate in load_low_bit (#12243)

Browse source 
* add lookup generate in load_low_bit

* update comment
This commit is contained in:
Yina Chen 2024-10-22 10:51:52 +03:00 committed by GitHub
parent d8c1287335
commit ec465fbcd7
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
1 changed files with 7 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

7
python/llm/src/ipex_llm/transformers/model.py
View file

@ -797,6 +797,13 @@ class _BaseAutoModelClass:
model)
torch.distributed.barrier()
# add lookup_generate to loaded model
from .lookup import lookup_generate
import types
model.lookup_generate = types.MethodType(lookup_generate, model)
if model.config.model_type == "minicpmv" and hasattr(model, 'llm'):
model.llm.lookup_generate = types.MethodType(lookup_generate, model.llm)
return model