ayo/ipex-llm
1
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1
2261 commits 1 branch 0 tags 37 MiB
Commit graph

939 commits

Author SHA1 Message Date
Wang, Jian4
093e6f8f73 LLM: Add qwen CPU speculative example (#9985) 
* init from gpu

* update for cpu

* update

* update

* fix xpu readme

* update

* update example prompt

* update prompt and add 72b

* update

* update
 2024-01-25 17:01:34 +08:00
Yishuo Wang
bf65548d29 Add quantize kv cache support for chaglm2/3 (#9996) 2024-01-25 16:55:59 +08:00
Chen, Zhentao
86055d76d5 fix optimize_model not working (#9995) 2024-01-25 16:39:05 +08:00
Wang, Jian4
9bff84e6fd LLM: Convert draft_model kv_cache from bf16 to fp32 (#9964) 
* convert bf16 to fp32

* update

* change when init

* init first and cut off after

* init and exchange

* update python type

* update

* fix bug

* update

* update
 2024-01-25 11:20:27 +08:00
Yina Chen
99ff6cf048 Update gpu spec decoding baichuan2 example dependency (#9990) 
* add dependency

* update

* update
 2024-01-25 11:05:04 +08:00
Yina Chen
27338540c3 Fix repetition_penalty not activated issue (#9989) 2024-01-25 10:40:41 +08:00
Jason Dai
3bc3d0bbcd Update self-speculative readme (#9986) 2024-01-24 22:37:32 +08:00
Yuwen Hu
b27e5a27b9 Remove the check for meta device in _replace_with_low_bit_linear (#9984) 2024-01-24 18:15:39 +08:00
Ruonan Wang
d4f65a6033 LLM: add mistral speculative example (#9976) 
* add mistral example

* update
 2024-01-24 17:35:15 +08:00
Yina Chen
b176cad75a LLM: Add baichuan2 gpu spec example (#9973) 
* add baichuan2 gpu spec example

* update readme & example

* remove print

* fix typo

* meet comments

* revert

* update
 2024-01-24 16:40:16 +08:00
Jinyi Wan
ec2d9de0ea Fix README.md for solar (#9957) 2024-01-24 15:50:54 +08:00
Mingyu Wei
bc9cff51a8 LLM GPU Example Update for Windows Support (#9902) 
* Update README in LLM GPU Examples

* Update reference of Intel GPU

* add cpu_embedding=True in comment

* small fixes

* update GPU/README.md and add explanation for cpu_embedding=True

* address comments

* fix small typos

* add backtick for cpu_embedding=True

* remove extra backtick in the doc

* add period mark

* update readme
 2024-01-24 13:42:27 +08:00
Chen, Zhentao
e0db44dcb6 fix unexpected keyword argument 'device' (#9982) 
* add device for chatglm3 only

* add comment for this change

* fix style

* fix style

* fix style again..

* finally fixed style
 2024-01-24 13:20:46 +08:00
Mingyu Wei
50a851e3b3 LLM: separate arc ut for disable XMX (#9953) 
* separate test_optimize_model api with disabled xmx

* delete test_optimize_model in test_transformers_api.py

* set env variable in .sh/ put back test_optimize_model

* unset env variable

* remove env setting in .py

* address errors in action

* remove import ipex

* lower tolerance
 2024-01-23 19:04:47 +08:00
Yuwen Hu
8d28aa8e2b [LLM] Fix the model.device problem when cpu_embedding=True (#9971) 
* Overwrite the device attribute for CPUPinnedParam

* Expose cpu_embedding=True for Linux users

* Fix python style
 2024-01-23 18:51:11 +08:00
Yishuo Wang
f82782cd3b fix starcoder (#9975) 2024-01-23 17:24:53 +08:00
WeiguangHan
be5836bee1 LLM: fix outlier value (#9945) 
* fix outlier value

* small fix
 2024-01-23 17:04:13 +08:00
Yishuo Wang
2c8a9aaf0d fix qwen causal mask when quantize_kv_cache=True (#9968) 2024-01-23 16:34:05 +08:00
Yina Chen
5aa4b32c1b LLM: Add qwen spec gpu example (#9965) 
* add qwen spec gpu example

* update readme

---------

Co-authored-by: rnwang04 <ruonan1.wang@intel.com>
 2024-01-23 15:59:43 +08:00
Yina Chen
36c665667d Add logits processor & qwen eos stop in speculative decoding (#9963) 
* add logits processor & qwen eos

* fix style

* fix

* fix

* fix style

* fix style

* support transformers 4.31

* fix style

* fix style

---------

Co-authored-by: rnwang04 <ruonan1.wang@intel.com>
 2024-01-23 15:57:28 +08:00
Ruonan Wang
60b35db1f1 LLM: add chatglm3 speculative decoding example (#9966) 
* add chatglm3 example

* update

* fix
 2024-01-23 15:54:12 +08:00
Xin Qiu
da4687c917 fix fp16 (#9970) 2024-01-23 15:53:32 +08:00
Chen, Zhentao
301425e377 harness tests on pvc multiple xpus (#9908) 
* add run_multi_llb.py

* update readme

* add job hint
 2024-01-23 13:20:37 +08:00
Ruonan Wang
27b19106f3 LLM: add readme for speculative decoding gpu examples (#9961) 
* add readme

* add readme

* meet code review
 2024-01-23 12:54:19 +08:00
Chen, Zhentao
39219b7e9a add default device meta when lcmu enabled (#9941) 2024-01-23 11:00:49 +08:00
Xin Qiu
dacf680294 add fused rotary pos emb for qwen (#9956) 
* add fused rotary pos emb for qwen

* update
 2024-01-23 10:37:56 +08:00
Ruonan Wang
7b1d9ad7c0 LLM: limit esimd sdp usage for k_len < 8 (#9959) 
* update

* fix
 2024-01-23 09:28:23 +08:00
Ruonan Wang
3e601f9a5d LLM: Support speculative decoding in bigdl-llm (#9951) 
* first commit

* fix error, add llama example

* hidden print

* update api usage

* change to api v3

* update

* meet code review

* meet code review, fix style

* add reference, fix style

* fix style

* fix first token time
 2024-01-22 19:14:56 +08:00
Cheen Hau, 俊豪
947b1e27b7 Add readme for Whisper Test (#9944) 
* Fix local data path

* Remove non-essential files

* Add readme

* Minor fixes to script

* Bugfix, refactor

* Add references to original source. Bugfixes.

* Reviewer comments

* Properly print and explain output

* Move files to dev/benchmark

* Fixes
 2024-01-22 15:11:33 +08:00
Xin Qiu
6fb3f40f7e fix error for benchmark_util.py running on cpu (#9949) 2024-01-22 10:14:40 +08:00
Heyang Sun
fb91c97fe8 support for Baichuan/Baichuan2 13B Chat running speculative decoding (#9921) 
* support for Baichuan/Baichuan2 13B Chat running speculative decoding

* fix stype
 2024-01-22 09:11:44 +08:00
Xin Qiu
97f0cd8975 optimize Decilm 7b (#9922) 
* optimize deci

* update

* decilm attension forward
 2024-01-19 17:31:13 +08:00
Wang, Jian4
bcaeb05272 Update optimize qwen (#9943) 
* update for n tokens input

* fix dtype

* update
 2024-01-19 16:54:59 +08:00
binbin Deng
db8e90796a LLM: add avg token latency information and benchmark guide of autotp (#9940) 2024-01-19 15:09:57 +08:00
Ruonan Wang
bf37b3a670 LLM: optimize CPU speculative decoding of chatglm3 (#9928) 
* update

* fix style

* meet code review
 2024-01-19 14:10:22 +08:00
Shaojun Liu
967714bac8 gguf memory optimization for mixtral (#9939) 2024-01-19 11:13:15 +08:00
Xin Qiu
610b5226be move reserved memory to benchmark_utils.py (#9907) 
* move reserved memory to benchmark_utils.py

* meet code review
 2024-01-19 09:44:30 +08:00
Lilac09
7032a2ad73 Optimize gguf load memory for mistral (#9923) 
* optimize gguf load for mistral

* fix output of gguf mistral

* reset
 2024-01-19 09:14:39 +08:00
Shaojun Liu
9a46f019d7 gguf memory optimization for baichuan (#9937) 2024-01-19 09:11:02 +08:00
Guancheng Fu
2e1448f08e [Serving] Add vllm_worker to fastchat serving framework (#9934) 
* add worker

* finish

* finish

* add license

* add more comments
 2024-01-18 21:33:36 +08:00
Chen, Zhentao
a8c866c32b add ppl benchmark (#9914) 
* add ppl benchmark

* add license

* add readme

* add dataset argument

* add dataset usage

* fixed low bit args

* correct result

* fix terminal display

* fix ppl update

* enable fp16 fp32 bf16

* format the desc

* fix model_kwargs

* add more readme
 2024-01-18 17:54:28 +08:00
WeiguangHan
100e0a87e5 LLM: add compressed chatglm3 model (#9892) 
* LLM: add compressed chatglm3 model

* small fix

* revert github action
 2024-01-18 17:48:15 +08:00
Yuwen Hu
9e2ac5291b Add rwkv v4 back for igpu perf test 32-512 (#9938) 2024-01-18 17:15:28 +08:00
Yishuo Wang
7bbb98abb6 Disable fused layer norm when using XMX to fix mpt UT (#9933) 2024-01-18 16:22:12 +08:00
Wang, Jian4
1fc9dfa265 LLM: Update for Qwen n tokens inputs (#9931) 
* update for n tokens inputs

* update style

* update
 2024-01-18 15:56:29 +08:00
Heyang Sun
5184f400f9 Fix Mixtral GGUF Wrong Output Issue (#9930) 
* Fix Mixtral GGUF Wrong Output Issue

* fix style

* fix style
 2024-01-18 14:11:27 +08:00
Yishuo Wang
453df868c9 add rwkv v5 attention kernel (#9927) 2024-01-18 10:16:29 +08:00
Ruonan Wang
054952f82f LLM: Fix rope of chatglm3 to support speculative decoding on CPU (#9926) 2024-01-18 09:28:10 +08:00
Ziteng Zhang
18cd1f1432 [LLM]Solve the problem of calling bmm operator in BF16Linear (#9924) 
* Solve the problem of calling bmm operator in BF16Linear
 2024-01-17 18:08:35 +08:00
Yina Chen
98b86f83d4 Support fast rope for training (#9745) 
* init

* init

* fix style

* add test and fix

* address comment

* update

* merge upstream main
 2024-01-17 15:51:38 +08:00
Yuwen Hu
0c498a7b64 Add llama2-13b to igpu perf test (#9920) 2024-01-17 14:58:45 +08:00
Ruonan Wang
b059a32fff LLM: add benchmark api for bigdl-llm fp16 on GPU (#9919) 
* add bmk for bigdl fp16

* fix
 2024-01-17 14:24:35 +08:00
Ruonan Wang
427f75000b LLM: fix sdp of chatglm3 (#9917) 
* fix

* fix

* fix
 2024-01-17 13:37:28 +08:00
Yishuo Wang
94767da7cf optimize rwkv v4 first token performance (#9912) 2024-01-17 09:27:41 +08:00
Cengguang Zhang
511cbcf773 LLM: add Ceval benchmark test. (#9872) 
* init ceval benchmark test.

* upload dataset.

* add other tests.

* add qwen evaluator.

* fix qwen evaluator style.

* fix qwen evaluator style.

* update qwen evaluator.

* add llama evaluator.

* update eval

* fix typo.

* fix

* fix typo.

* fix llama evaluator.

* fix bug.

* fix style.

* delete dataset.

* fix style.

* fix style.

* add README.md and fix typo.

* fix comments.

* remove run scripts
 2024-01-16 19:14:26 +08:00
Shaojun Liu
b909c5c9c2 GGUF load memory optimization (#9913) 
* block-wise

* convert linear for module

* revert

* Fix PEP8 checks Error
 2024-01-16 18:54:39 +08:00
Yuwen Hu
8643b62521 [LLM] Support longer context in iGPU perf tests (2048-256) (#9910) 2024-01-16 17:48:37 +08:00
Xin Qiu
dee32f7d15 copy fused rms norm's reuslt to avoid <unk> (#9909) 2024-01-16 16:54:08 +08:00
Ruonan Wang
8d7326ae03 LLM: fix chatglm3 sdp to support speculative decoding (#9900) 
* fix chatglm3

* fix

* update

* meet code review

* fix
 2024-01-16 11:29:13 +08:00
Guancheng Fu
9f34da7cdb Update PVC XMX condition (#9901) 
* update pvc xmx condition

* update condition

* update conditon
 2024-01-15 15:42:15 +08:00
Yishuo Wang
6637860ddf change xmx condition (#9896) 2024-01-12 19:51:48 +08:00
WeiguangHan
0e69bfe6b0 LLM: fix the performance drop of starcoder (#9889) 
* LLM: fix the performance drop of starcoder

* small fix

* small fix
 2024-01-12 09:14:15 +08:00
Ruonan Wang
d9cf55bce9 LLM: fix MLP check of mixtral (#9891) 2024-01-11 18:01:59 +08:00
Ziteng Zhang
4f4ce73f31 [LLM] Add transformer_autocast_bf16 into all-in-one (#9890) 
* Add transformer_autocast_bf16 into all-in-one
 2024-01-11 17:51:07 +08:00
Ziteng Zhang
4af88a67b9 support chatglm3 with bf16 (#9888) 
* support chatglm3 with bigdl-bf16
 2024-01-11 16:45:21 +08:00
Yuwen Hu
0aef35a965 [LLM] Improve LLM doc regarding windows gpu related info (#9880) 
* Improve runtime configuration for windows

* Add python 310/311 supports for wheel downloading

* Add troubleshooting for windows gpu

* Remove manually import ipex due to auto importer

* Add info regarding cpu_embedding=True on iGPU

* More info for Windows users

* Small updates to API docs

* Python style fix

* Remove tip for loading from saved optimize_model for now

* Updated based on comments

* Update win info for multi-intel gpus selection

* Small fix

* Small fix
 2024-01-11 14:37:16 +08:00
Jinyi Wan
07485eff5a Add SOLAR-10.7B to README (#9869) 2024-01-11 14:28:41 +08:00
WeiguangHan
33fd1f9c76 LLM: fix input length logic for run_transformer_int4_gpu (#9864) 
* LLM: fix input length logic for run_transformer_int4_gpu

* small fix

* small fix

* small fix
 2024-01-10 18:20:14 +08:00
Ruonan Wang
53531ae4ee LLM: support qkv fusion for fp8e5 (#9878) 
* update

* add mistral

* meet code review
 2024-01-10 17:50:00 +08:00
Lilac09
cb32b985ec add mistral and chatglm support to vllm (#9879) 
* add mistral and chatglm support to vllm

* add mistral and chatglm support to vllm
 2024-01-10 15:38:42 +08:00
ZehuaCao
e76d984164 [LLM] Support llm-awq vicuna-7b-1.5 on arc (#9874) 
* support llm-awq vicuna-7b-1.5 on arc

* support llm-awq vicuna-7b-1.5 on arc
 2024-01-10 14:28:39 +08:00
Ruonan Wang
3e05c9e11b LLM: update esimd sdp kernel (#9871) 2024-01-09 18:10:01 +08:00
Yuwen Hu
023679459e [LLM] Small fixes for finetune related examples and UTs (#9870) 2024-01-09 18:05:03 +08:00
Cheen Hau, 俊豪
b2aa267f50 Enhance LLM GPU installation document (#9828) 
* Improve gpu install doc

* Add troubleshooting - setvars.sh not done properly.

* Further improvements

* 2024.x.x -> 2024.0

* Fixes

* Fix Install BigDL-LLM From Wheel : bigdl-llm[xpu_2.0]

* Remove "export USE_XETLA=OFF" for Max GPU
 2024-01-09 16:30:50 +08:00
Yuwen Hu
23fc888abe Update llm gpu xpu default related info to PyTorch 2.1 (#9866) 2024-01-09 15:38:47 +08:00
Yishuo Wang
36496d60ac only use quantize kv cache on MTL (#9862) 2024-01-09 13:24:02 +08:00
ZehuaCao
146076bdb5 Support llm-awq backend (#9856) 
* Support for LLM-AWQ Backend

* fix

* Update README.md

* Add awqconfig

* modify init

* update

* support llm-awq

* fix style

* fix style

* update

* fix AwqBackendPackingMethod not found error

* fix style

* update README

* fix style

---------

Co-authored-by: Uxito-Ada <414416158@qq.com>
Co-authored-by: Heyang Sun <60865256+Uxito-Ada@users.noreply.github.com>
Co-authored-by: cyita <yitastudy@gmail.com>
 2024-01-09 13:07:32 +08:00
Ruonan Wang
fea6f16057 LLM: add mlp fusion for fp8e5 and update related check (#9860) 
* update mlp fusion

* fix style

* update
 2024-01-09 09:56:32 +08:00
binbin Deng
294fd32787 LLM: update DeepSpeed AutoTP example with GPU memory optimization (#9823) 2024-01-09 09:22:49 +08:00
Yuwen Hu
5ba1dc38d4 [LLM] Change default Linux GPU install option to PyTorch 2.1 (#9858) 
* Update default xpu to ipex 2.1

* Update related install ut support correspondingly

* Add arc ut tests for both ipex 2.0 and 2.1

* Small fix

* Diable ipex 2.1 test for now as oneapi 2024.0 has not beed installed on the test machine

* Update document for default PyTorch 2.1

* Small fix

* Small fix

* Small doc fixes

* Small fixes
 2024-01-08 17:16:17 +08:00
Mingyu Wei
ed81baa35e LLM: Use default typing-extension in LangChain examples (#9857) 
* remove typing extension downgrade in readme; minor fixes of code

* fix typos in README

* change default question of docqa.py
 2024-01-08 16:50:55 +08:00
Jiao Wang
3b6372ab12 Fix Llama transformers 4.36 support (#9852) 
* supoort 4.36

* style

* update

* update

* update

* fix merge

* update
 2024-01-08 00:32:23 -08:00
Chen, Zhentao
1b585b0d40 set fp8 default as e5m2 (#9859) 2024-01-08 15:53:57 +08:00
Ruonan Wang
dc995006cc LLM: add flash attention for mistral / mixtral (#9846) 
* add flash attention for mistral

* update

* add flash attn for mixtral

* fix style
 2024-01-08 09:51:34 +08:00
Yishuo Wang
afaa871144 [LLM] support quantize kv cache to fp8 (#9812) 2024-01-08 09:28:20 +08:00
Jiao Wang
248ae7fad2 LLama optimize_model to support transformers 4.36 (#9818) 
* supoort 4.36

* style

* update

* update

* update
 2024-01-05 11:30:18 -08:00
Ruonan Wang
a60bda3324 LLM: update check for deepspeed (#9838) 2024-01-05 16:44:10 +08:00
Ruonan Wang
16433dd959 LLM: fix first token judgement of flash attention (#9841) 
* fix flash attention

* meet code review

* fix
 2024-01-05 13:49:37 +08:00
Yina Chen
f919f5792a fix kv cache out of bound (#9827) 2024-01-05 12:38:57 +08:00
Ruonan Wang
5df31db773 LLM: fix accuracy issue of chatglm3 (#9830) 
* add attn mask for first token

* fix

* fix

* change attn calculation

* fix

* fix

* fix style

* fix style
 2024-01-05 10:52:05 +08:00
Jinyi Wan
3147ebe63d Add cpu and gpu examples for SOLAR-10.7B (#9821) 2024-01-05 09:50:28 +08:00
WeiguangHan
ad6b182916 LLM: change the color of peak diff (#9836) 2024-01-04 19:30:32 +08:00
Xiangyu Tian
38c05be1c0 [LLM] Fix dtype mismatch in Baichuan2-13b (#9834) 2024-01-04 15:34:42 +08:00
Ruonan Wang
8504a2bbca LLM: update qlora alpaca example to change lora usage (#9835) 
* update example

* fix style
 2024-01-04 15:22:20 +08:00
Ziteng Zhang
05b681fa85 [LLM] IPEX auto importer set on by default (#9832) 
* Set BIGDL_IMPORT_IPEX default to True

* Remove import intel_extension_for_pytorch as ipex from GPU example
 2024-01-04 13:33:29 +08:00
Wang, Jian4
4ceefc9b18 LLM: Support bitsandbytes config on qlora finetune (#9715) 
* test support bitsandbytesconfig

* update style

* update cpu example

* update example

* update readme

* update unit test

* use bfloat16

* update logic

* use int4

* set defalut bnb_4bit_use_double_quant

* update

* update example

* update model.py

* update

* support lora example
 2024-01-04 11:23:16 +08:00
WeiguangHan
9a14465560 LLM: add peak diff (#9789) 
* add peak diff

* small fix

* revert yml file
 2024-01-03 18:18:19 +08:00
Mingyu Wei
f4eb5da42d disable arc ut (#9825) 2024-01-03 18:10:34 +08:00
Ruonan Wang
20e9742fa0 LLM: fix chatglm3 issue (#9820) 
* fix chatglm3 issue

* small update
 2024-01-03 16:15:55 +08:00
Wang, Jian4
a54cd767b1 LLM: Add gguf falcon (#9801) 
* init falcon

* update convert.py

* update style
 2024-01-03 14:49:02 +08:00
Yuwen Hu
668c2095b1 Remove unnecessary warning when installing llm (#9815) 2024-01-03 10:30:05 +08:00
dingbaorong
f5752ead36 Add whisper test (#9808) 
* add whisper benchmark code

* add librispeech_asr.py

* add bigdl license
 2024-01-02 16:36:05 +08:00
binbin Deng
6584539c91 LLM: fix installation of codellama (#9813) 2024-01-02 14:32:50 +08:00
Kai Huang
4d01069302 Temp remove baichuan2-13b 1k from arc perf test (#9810) 2023-12-29 12:54:13 +08:00
dingbaorong
a2e668a61d fix arc ut test (#9736) 2023-12-28 16:55:34 +08:00
Qiyuan Gong
f0f9d45eac [LLM] IPEX import support bigdl-core-xe-21 (#9769) 
Add support for bigdl-core-xe-21.
 2023-12-28 15:23:58 +08:00
dingbaorong
a8baf68865 fix csv_to_html (#9802) 2023-12-28 14:58:51 +08:00
Guancheng Fu
5857a38321 [vLLM] Add option to adjust KV_CACHE_ALLOC_BLOCK_LENGTH (#9782) 
* add option kv_cache_block

* change var name
 2023-12-28 14:41:47 +08:00
Ruonan Wang
99bddd3ab4 LLM: better FP16 support for Intel GPUs (#9791) 
* initial support

* fix

* fix style

* fix

* limi esimd usage condition

* refactor code

* fix style

* small fix

* meet code review

* small fix
 2023-12-28 13:30:13 +08:00
Yishuo Wang
7d9f6c6efc fix cpuinfo error (#9793) 2023-12-28 09:23:44 +08:00
Wang, Jian4
7ed9538b9f LLM: support gguf mpt (#9773) 
* add gguf mpt

* update
 2023-12-28 09:22:39 +08:00
Cengguang Zhang
d299f108d0 update falcon attention forward. (#9796) 2023-12-28 09:11:59 +08:00
Shaojun Liu
a5e5c3daec set warm_up: 3 num_trials: 50 for cpu stress test (#9799) 2023-12-28 08:55:43 +08:00
dingbaorong
f6bb4ab313 Arc stress test (#9795) 
* add arc stress test

* triger ci

* triger CI

* triger ci

* disable ci
 2023-12-27 21:02:41 +08:00
Kai Huang
40eaf76ae3 Add baichuan2-13b to Arc perf (#9794) 
* add baichuan2-13b

* fix indent

* revert
 2023-12-27 19:38:53 +08:00
Shaojun Liu
6c75c689ea bigdl-llm stress test for stable version (#9781) 
* 1k-512 2k-512 baseline

* add cpu stress test

* update yaml name

* update

* update

* clean up

* test

* update

* update

* update

* test

* update
 2023-12-27 15:40:53 +08:00
dingbaorong
5cfb4c4f5b Arc stable version performance regression test (#9785) 
* add arc stable version regression test

* empty gpu mem between different models

* triger ci

* comment spr test

* triger ci

* address kai's comments and disable ci

* merge fp8 and int4

* disable ci
 2023-12-27 11:01:56 +08:00
binbin Deng
40edb7b5d7 LLM: fix get environment variables setting (#9787) 2023-12-27 09:11:37 +08:00
Kai Huang
689889482c Reduce max_cache_pos to reduce Baichuan2-13B memory (#9694) 
* optimize baichuan2 memory

* fix

* style

* fp16 mask

* disable fp16

* fix style

* empty cache

* revert empty cache
 2023-12-26 19:51:25 +08:00
Jason Dai
361781bcd0 Update readme (#9788) 2023-12-26 19:46:11 +08:00
Yuwen Hu
c38e18f2ff [LLM] Migrate iGPU perf tests to new machine (#9784) 
* Move 1024 test just after 32-32 test; and enable all model for 1024-128

* Make sure python output encoding in utf-8 so that redirect to txt can always be success

* Upload results to ftp

* Small fix
 2023-12-26 19:15:57 +08:00
WeiguangHan
c05d7e1532 LLM: add star_corder_15.5b model (#9772) 
* LLM: add star_corder_15.5b model

* revert llm_performance_tests.yml
 2023-12-26 18:55:56 +08:00
Ziteng Zhang
44b4a0c9c5 [LLM] Correct prompt format of Yi, Llama2 and Qwen in generate.py (#9786) 
* correct prompt format of Yi

* correct prompt format of llama2 in cpu generate.py

* correct prompt format of Qwen in GPU example
 2023-12-26 16:57:55 +08:00
Xiangyu Tian
0ea842231e [LLM] vLLM: Add api_server entrypoint (#9783) 
Add vllm.entrypoints.api_server for benchmark_serving.py in vllm.
 2023-12-26 16:03:57 +08:00
dingbaorong
64d05e581c add peak gpu mem stats in transformer_int4_gpu (#9766) 
* add peak gpu mem stats in transformer_int4_gpu

* address weiguang's comments
 2023-12-26 15:38:28 +08:00
Ziteng Zhang
87b4100054 [LLM] Support Yi model in chat.py (#9778) 
* Suppot Yi model

* code style& add reference link
 2023-12-26 10:03:39 +08:00
Ruonan Wang
11d883301b LLM: fix wrong batch output caused by flash attention (#9780) 
* fix

* meet code review

* move batch size check to the beginning

* move qlen check inside function

* meet code review
 2023-12-26 09:41:27 +08:00
Heyang Sun
66e286a73d Support for Mixtral AWQ (#9775) 
* Support for Mixtral AWQ

* Update README.md

* Update README.md

* Update awq_config.py

* Update README.md

* Update README.md
 2023-12-25 16:08:09 +08:00
Ruonan Wang
1917bbe626 LLM: fix BF16Linear related training & inference issue (#9755) 
* fix bf16 related issue

* fix

* update based on comment & add arc lora script

* update readme

* update based on comment

* update based on comment

* update

* force to bf16

* fix style

* move check input dtype into function

* update convert

* meet code review

* meet code review

* update merged model to support new training_mode api

* fix typo
 2023-12-25 14:49:30 +08:00
Xiangyu Tian
30dab36f76 [LLM] vLLM: Fix kv cache init (#9771) 
Fix kv cache init
 2023-12-25 14:17:06 +08:00
Yina Chen
449b387125 Support relora in bigdl-llm (#9687) 
* init

* fix style

* update

* support resume & update readme

* update

* update

* remove important

* add training mode

* meet comments
 2023-12-25 14:04:28 +08:00
Shaojun Liu
b6222404b8 bigdl-llm stable version: let the perf test fail if the difference between perf and baseline is greater than 5% (#9750) 
* test

* test

* test

* update

* revert
 2023-12-25 13:47:11 +08:00
Ziteng Zhang
986f65cea9 [LLM] Add trust_remote_code for local renamed model in bigdl_llm_model.py (#9762) 2023-12-25 11:31:14 +08:00
Yishuo Wang
be13b162fe add codeshell example (#9743) 2023-12-25 10:54:01 +08:00
Guancheng Fu
daf536fb2d vLLM: Apply attention optimizations for selective batching (#9758) 
* fuse_rope for prefil

* apply kv_cache optimizations

* apply fast_decoding_path

* Re-enable kv_cache optimizations for prefill

* reduce KV_CACHE_ALLOC_BLOCK for selective_batching
 2023-12-25 10:29:31 +08:00
binbin Deng
ed8ed76d4f LLM: update deepspeed autotp usage (#9733) 2023-12-25 09:41:14 +08:00
Yuwen Hu
02436c6cce [LLM] Enable more long context in-out pairs for iGPU perf tests (#9765) 
* Add test for 1024-128 and enable more tests for 512-64

* Fix date in results csv name to the time when the performance is triggered

* Small fix

* Small fix

* further fixes
 2023-12-22 18:18:23 +08:00
Chen, Zhentao
7fd7c37e1b Enable fp8e5 harness (#9761) 
* fix precision format like fp8e5

* match fp8_e5m2
 2023-12-22 16:59:48 +08:00
Qiyuan Gong
4c487313f2 Revert "[LLM] IPEX auto importer turn on by default for XPU (#9730)" (#9759) 
This reverts commit 0284801fbd.
 2023-12-22 16:38:24 +08:00
Qiyuan Gong
0284801fbd [LLM] IPEX auto importer turn on by default for XPU (#9730) 
* Set BIGDL_IMPORT_IPEX default to true, i.e., auto import IPEX for XPU.
* Remove import intel_extension_for_pytorch as ipex from GPU example.
* Add support for bigdl-core-xe-21.
 2023-12-22 16:20:32 +08:00
Chen, Zhentao
86a69e289c fix harness runner label of manual trigger (#9754) 
* fix runner

* update golden
 2023-12-22 15:09:22 +08:00
Guancheng Fu
fdf93c9267 Implement selective batching for vLLM (#9659) 
* add control to load hf model

* finish initial version of selective_batching

* temp

* finish

* Remove print statement

* fix error

* Apply yang's optimization

* a version that works

* We need to check kv_cache passed in, this could be an error. TODO: add fast decoding path

* format

* temp solution: not batching prefill requests

* a version that works for prefill batching

* format

* a solid version: works normally

* a temp version

* Solid version: remove redundant functions

* fix format

* format

* solid: add option to enable selective_batching

* remove logic for using transformer models

* format

* format

* solid: enable argument VLLM_ENABLE_SELECTIVE_BATCHING

* format

* finish

* format
 2023-12-22 13:45:46 +08:00
Ruonan Wang
2f36769208 LLM: bigdl-llm lora support & lora example (#9740) 
* lora support and single card example

* support multi-card, refactor code

* fix model id and style

* remove torch patch, add two new class for bf16, update example

* fix style

* change to training_mode

* small fix

* add more info in help

* fixstyle, update readme

* fix ut

* fix ut

* Handling compatibility issues with default LoraConfig
 2023-12-22 11:05:39 +08:00
SONG Ge
ba0b939579 [LLM] Support transformers-v4.36.0 on mistral model (#9744) 
* add support transformers-v4.36.0 on mistral model

* python/llm/src/bigdl/llm/transformers/models/mistral.py

* make the redundant implementation as utils

* fix code style

* fix

* fix style

* update with utils enough_kv_room
 2023-12-22 09:59:27 +08:00
Xin Qiu
e36111e713 mixstral fused qkv and rope (#9724) 
* mixstral fused qkv and rope

* fix and clean

* fix style

* update

* update

* fix

* update

* fix
 2023-12-22 09:26:35 +08:00
Jiao Wang
e4f6e43675 safetenor to false (#9728) 2023-12-21 14:41:51 -08:00
Shaojun Liu
bb52239e0a bigdl-llm stable version release & test (#9732) 
* stable version test

* trigger spr test

* update

* trigger

* test

* test

* test

* test

* test

* refine

* release linux first
 2023-12-21 22:55:33 +08:00
WeiguangHan
d4d2ccdd9d LLM: remove startcorder-15.5b (#9748) 2023-12-21 18:52:52 +08:00
WeiguangHan
474c099559 LLM: using separate threads to do inference (#9727) 
* using separate threads to do inference

* resolve some comments

* resolve some comments

* revert llm_performance_tests.yml file
 2023-12-21 17:56:43 +08:00
Yishuo Wang
426660b88e simplify qwen attention (#9747) 2023-12-21 17:53:29 +08:00
Wang, Jian4
984697afe2 LLM: Add bloom gguf support (#9734) 
* init

* update bloom add merges

* update

* update readme

* update for llama error

* update
 2023-12-21 14:06:25 +08:00
Heyang Sun
df775cf316 fix python style (#9742) 
* fix python style

* fix

* fix
 2023-12-21 11:25:05 +08:00
Chen, Zhentao
b06a3146c8 Fix 70b oom (#9738) 
* add default value to bigdl llm

* fix model oom
 2023-12-21 10:40:52 +08:00
Xin Qiu
6c3e698bf1 mistral decoding_fast_path and fused mlp (#9714) 
* mistral decoding_fast_path and fused mlp

* meet code review
 2023-12-21 10:11:37 +08:00
Heyang Sun
d157f623b6 Load Mixtral gguf in a block-wise way (#9725) 
* Load Mixtral gguf in a block-wise way

* refine
 2023-12-21 10:03:23 +08:00
WeiguangHan
34bb804189 LLM: check csv and its corresponding yaml file (#9702) 
* LLM: check csv and its corresponding yaml file

* run PR arc perf test

* modify the name of some variables

* execute the check results script in right place

* use cp to replace mv command

* resolve some comments

* resolve more comments

* revert the llm_performance_test.yaml file
 2023-12-21 09:54:33 +08:00
Zhao Changmin
4bda975a3e LLM: Align lowbit model config (#9735) 
* align lowbit model config
 2023-12-21 09:48:58 +08:00
Wang, Jian4
e1e921f425 LLM: gguf other model using dtype (#9729) 2023-12-21 09:33:40 +08:00
Yishuo Wang
13ea6330bd optimize qwen rope (#9737) 2023-12-20 17:34:34 +08:00
Ziteng Zhang
4c032a433e [LLM] Add glibc checker (#9624) 
* Add glibc checker
* Add env BIGDL_GLIBC_CHECK to control glibc checker. The default is false, i.e., don't check.
 2023-12-20 16:52:43 +08:00
Yina Chen
cd652a1710 Support fp8 e5m2 on arc (#9711) 
* init

* fix style

* update

* fix style

* update
 2023-12-20 16:26:17 +08:00
Yishuo Wang
e54c428d30 add bf16/fp16 fuse mlp support (#9726) 2023-12-20 10:40:45 +08:00
Heyang Sun
612651cb5d fix typo (#9723) 2023-12-20 09:41:59 +08:00
WeiguangHan
3aa8b66bc3 LLM: remove starcoder-15.5b model temporarily (#9720) 2023-12-19 20:14:46 +08:00
Yishuo Wang
522cf5ed82 [LLM] Improve chatglm2/3 rest token performance with long context (#9716) 2023-12-19 17:29:38 +08:00
Yishuo Wang
f2e6abb563 fix mlp batch size check (#9718) 2023-12-19 14:22:22 +08:00
Heyang Sun
1fa7793fc0 Load Mixtral GGUF Model (#9690) 
* Load Mixtral GGUF Model

* refactor

* fix empty tensor when to cpu

* update gpu and cpu readmes

* add dtype when set tensor into module
 2023-12-19 13:54:38 +08:00
Qiyuan Gong
d0a3095b97 [LLM] IPEX auto importer (#9706) 
* IPEX auto importer and get_ipex_version.
* Add BIGDL_IMPORT_IPEX to control auto import, default is false.
 2023-12-19 13:39:38 +08:00
Yang Wang
f4fb58d99c fusing qkv project and rope (#9612) 
* Try fusing qkv project and rope

* add fused mlp

* fuse append cache

* fix style and clean up code

* clean up
 2023-12-18 16:45:00 -08:00
Kai Huang
4c112ee70c Rename qwen in model name for arc perf test (#9712) 2023-12-18 20:34:31 +08:00
Cengguang Zhang
4d22add4af LLM: fix qwen efficiency issue in perf-test. 2023-12-18 18:32:54 +08:00
Ruonan Wang
8ed89557e5 LLM: add mlp optimization of mixtral (#9709) 2023-12-18 16:59:52 +08:00
Chen, Zhentao
b3647507c0 Fix harness workflow (#9704) 
* error when larger than 0.001

* fix env setup

* fix typo

* fix typo
 2023-12-18 15:42:10 +08:00
binbin Deng
12df70953e LLM: add resume_from_checkpoint related section (#9705) 2023-12-18 12:27:02 +08:00
Xin Qiu
320110d158 handle empty fused norm result (#9688) 
* handle empty fused norm result

* remove fast_rms_norm

* fix style
 2023-12-18 09:56:11 +08:00
Ziteng Zhang
67cc155771 [LLM] Correct chat format of llama and add llama_stream_chat in chat.py 
* correct chat format of llama
* add llama_stream_chat
 2023-12-15 16:36:46 +08:00
Ziteng Zhang
0d41b7ba7b [LLM] Correct chat format & add stop words for chatglm3 in chat.py 
* correct chat format of chatglm3
* correct stop words of chatglm3
 2023-12-15 16:35:17 +08:00
Ziteng Zhang
d57efd8eb9 [LM] Add stop_word for Qwen model and correct qwen chat format in chat.py (#9642) 
* add stop words list for qwen

* change qwen chat format
 2023-12-15 14:53:58 +08:00
SONG Ge
d5b81af7bd Support mixtral attention optimization on transformers-v4.36.0 (#9674) 
* add example code to support mistral/mixtral attention on transformers v4.36.0

* update

* style fix

* add update for seen-tokens

* support mixtral

* rm mistral change

* small fix

* add more comments and remove use_cache part

---------

Co-authored-by: plusbang <binbin1.deng@intel.com>
 2023-12-15 14:30:23 +08:00
Cengguang Zhang
adbef56001 LLM: update qwen attention forward. (#9695) 
* feat: update qwen attention forward.

* fix: style.
 2023-12-15 14:06:15 +08:00
Wang, Jian4
b8437a1c1e LLM: Add gguf mistral model support (#9691) 
* add mistral support

* need to upgrade transformers version

* update
 2023-12-15 13:37:39 +08:00
Wang, Jian4
496bb2e845 LLM: Support load BaiChuan model family gguf model (#9685) 
* support baichuan model family gguf model

* update gguf generate.py

* add verify models

* add support model_family

* update

* update style

* update type

* update readme

* update

* remove support model_family
 2023-12-15 13:34:33 +08:00
Lilac09
3afed99216 fix path issue (#9696) 2023-12-15 11:21:49 +08:00
Jason Dai
37f509bb95 Update readme (#9692) 2023-12-14 19:50:21 +08:00
WeiguangHan
1f0245039d LLM: check the final csv results for arc perf test (#9684) 
* LLM: check the final csv results for arc perf test

* delete useless python script

* change threshold

* revert the llm_performance_tests.yml
 2023-12-14 19:46:08 +08:00
Yishuo Wang
9a330bfc2b fix fuse mlp when using q5_0 or fp8 (#9689) 2023-12-14 16:16:05 +08:00
Yuwen Hu
82ac2dbf55 [LLM] Small fixes for win igpu test for ipex 2.1 (#9686) 
* Fixes to install for igpu performance tests

* Small update for core performance tests model lists
 2023-12-14 15:39:51 +08:00
WeiguangHan
3e8d198b57 LLM: add eval func (#9662) 
* Add eval func

* add left eval
 2023-12-14 14:59:02 +08:00
Ziteng Zhang
21c7503a42 [LLM] Correct prompt format of Qwen in generate.py (#9678) 
* Change qwen prompt format to chatml
 2023-12-14 14:01:30 +08:00
Qiyuan Gong
223c9622f7 [LLM] Mixtral CPU examples (#9673) 
* Mixtral CPU PyTorch and hugging face examples, based on #9661 and #9671
 2023-12-14 10:35:11 +08:00
Xin Qiu
5e46e0e5af fix baichuan2-7b 1st token performance regression on xpu (#9683) 
* fix baichuan2-7b 1st token performance regression

* add comments

* fix style
 2023-12-14 09:58:32 +08:00
ZehuaCao
877229f3be [LLM]Add Yi-34B-AWQ to verified AWQ model. (#9676) 
* verfiy Yi-34B-AWQ

* update
 2023-12-14 09:55:47 +08:00
binbin Deng
68a4be762f remove disco mixtral, update oneapi version (#9671) 2023-12-13 23:24:59 +08:00
Ruonan Wang
1456d30765 LLM: add dot to option name in setup (#9682) 2023-12-13 20:57:27 +08:00
Yuwen Hu
cbdd49f229 [LLM] win igpu performance for ipex 2.1 and oneapi 2024.0 (#9679) 
* Change igpu win tests for ipex 2.1 and oneapi 2024.0

* Qwen model repo id updates; updates model list for 512-64

* Add .eval for win igpu all-in-one benchmark for best performance
 2023-12-13 18:52:29 +08:00
Mingyu Wei
16febc949c [LLM] Add exclude option in all-in-one performance test (#9632) 
* add exclude option in all-in-one perf test

* update arc-perf-test.yaml

* Exclude in_out_pairs in main function

* fix some bugs

* address Kai's comments

* define excludes at the beginning

* add bloomz:2048 to exclude
 2023-12-13 18:13:06 +08:00
Ruonan Wang
9b9cd51de1 LLM: update setup to provide new install option to support ipex 2.1 & oneapi 2024 (#9647) 
* update setup

* default to 2.0 now

* meet code review
 2023-12-13 17:31:56 +08:00
Yishuo Wang
09ca540f9b use fuse mlp in qwen (#9672) 2023-12-13 17:20:08 +08:00
Ruonan Wang
c7741c4e84 LLM: update moe block convert to optimize rest token latency of Mixtral (#9669) 
* update moe block convert

* further accelerate final_hidden_states

* fix style

* fix style
 2023-12-13 16:17:06 +08:00
ZehuaCao
503880809c verfiy codeLlama (#9668) 2023-12-13 15:39:31 +08:00