ayo/ipex-llm
1
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

correct Readme GPU example and API docstring (#9225)

Browse source 
* update readme to correct GPU usage

* update from_pretrained supported low bit options

* fix stype check
This commit is contained in:
Chen, Zhentao 2023-10-19 16:08:47 +08:00 committed by GitHub
parent f87f67ee1c
commit 5850241423
2 changed files with 6 additions and 4 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

1
python/llm/README.md
View file

@ -127,6 +127,7 @@ You may apply INT4 optimizations to any Hugging Face *Transformers* model on Int
```python
#load Hugging Face Transformers model with INT4 optimizations
from bigdl.llm.transformers import AutoModelForCausalLM
import intel_extension_for_pytorch
model = AutoModelForCausalLM.from_pretrained('/path/to/model/', load_in_4bit=True)
#run the optimized model on Intel GPU

9
python/llm/src/bigdl/llm/transformers/model.py
View file

@ -60,9 +60,9 @@ class _BaseAutoModelClass:
:param load_in_4bit: boolean value, True means load linear's weight to symmetric int 4.
Default to be False.
:param load_in_low_bit: str value, options are sym_int4, asym_int4, sym_int5, asym_int5
, sym_int8 or fp16. sym_int4 means symmetric int 4, asym_int4 means
asymmetric int 4, etc. Relevant low bit optimizations will
be applied to the model.
, sym_int8, nf3, nf4 or fp16. sym_int4 means symmetric int 4,
asym_int4 means asymmetric int 4, nf4 means 4-bit NormalFloat, etc.
Relevant low bit optimizations will be applied to the model.
:param optimize_model: boolean value, Whether to further optimize the low_bit llm model.
Default to be True.
:param modules_to_not_convert: list of str value, modules (nn.Module) that are skipped when
@ -106,7 +106,8 @@ class _BaseAutoModelClass:
from .convert import ggml_convert_low_bit
invalidInputError(q_k in ggml_tensor_qtype,
f"Unknown load_in_low_bit value: {q_k}, expected:"
f" sym_int4, asym_int4, sym_int5, asym_int5, sym_int8 or fp16.")
f" sym_int4, asym_int4, sym_int5, asym_int5, sym_int8, nf3, nf4 "
"or fp16.")
qtype = ggml_tensor_qtype[q_k]
# In case it needs a second try,