ayo/ipex-llm
1
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

Skip rescale rwkv linear when load_low_bit (#10164)

Browse source 
* rwkv_ld
This commit is contained in:
Zhao Changmin 2024-02-19 15:56:42 +08:00 committed by GitHub
parent 3e2af5ec0a
commit f8730e8dc1
1 changed files with 4 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

4
python/llm/src/bigdl/llm/transformers/model.py
View file

@ -561,6 +561,10 @@ class _BaseAutoModelClass:
else: else:
model = model_class(config, *model_args, **kwargs) model = model_class(config, *model_args, **kwargs)
# rwkv model linear layers has been rescaled
if model.config.model_type == "rwkv":
model.layers_are_rescaled = True
# Loading args may differ based on their usage # Loading args may differ based on their usage
quant_device = "meta" if bigdl_lcmu_enabled else "cpu" quant_device = "meta" if bigdl_lcmu_enabled else "cpu"
model = ggml_convert_low_bit(model, qtype, optimize_model, device=quant_device, model = ggml_convert_low_bit(model, qtype, optimize_model, device=quant_device,