Commit graph

307 commits

Author SHA1 Message Date
Chen, Zhentao
f315c7f93a Move harness nightly related files to llm/test folder (#10209)
* move harness nightly files to test folder

* change workflow file path accordingly

* use arc01 when pr

* fix path

* fix fp16 csv path
2024-02-23 11:12:36 +08:00
Ovo233
60e11b6739 LLM: Add mlp layer unit tests (#10200)
* add mlp layer unit tests

* add download baichuan-13b

* exclude llama for now

* install additional packages

* rename bash file

* switch to Baichuan2

* delete attention related code

* fix name errors in yml file
2024-02-22 13:44:45 +08:00
hxsz1997
5b387bb71a Change the nightly test time of ppl and harness (#10198)
* remove include and language option, select the corresponding dataset based on the model name in Run

* change the nightly test time

* change the nightly test time of harness and ppl
2024-02-21 17:39:33 +08:00
yb-peng
b1a97b71a9 Harness eval: Add is_last parameter and fix logical operator in highlight_vals (#10192)
* Add is_last parameter and fix logical operator in highlight_vals

* Add script to update HTML files in parent folder

* Add running update_html_in_parent_folder.py in summarize step

* Add licence info

* Remove update_html_in_parent_folder.py in Summarize the results for pull request
2024-02-21 14:45:32 +08:00
Chen, Zhentao
39d37bd042 upgrade harness package version in workflow (#10188)
* upgrade harness

* update readme
2024-02-21 11:21:30 +08:00
yb-peng
de3dc609ee Modify harness evaluation workflow (#10174)
* Modify table head in harness

* Specify the file path of fp16.csv

* change run to run nightly and run pr to debug

* Modify the way to get fp16.csv to downloading from github

* Change the method to calculate diff in html table

* Change the method to calculate diff in html table

* Re-arrange job order

* Re-arrange job order

* Change limit

* Change fp16.csv  path

* Change highlight rules

* Change limit
2024-02-20 18:55:43 +08:00
hxsz1997
b55fd00fb1 remove include and language option, select the corresponding dataset based on the model name in Run (#10181) 2024-02-20 17:34:52 +08:00
hxsz1997
6e10d98a8d Fix some typos (#10175)
* add llm-ppl workflow

* update the DATASET_DIR

* test multiple precisions

* modify nightly test

* match the updated ppl code

* add matrix.include

* fix the include error

* update the include

* add more model

* update the precision of include

* update nightly time and add more models

* fix the workflow_dispatch description, change default model of pr and modify the env

* modify workflow_dispatch language options

* modify options

* modify language options

* modeify workflow_dispatch type

* modify type

* modify the type of language

* change seq_len type

* fix some typos

* revert changes to stress_test.txt
2024-02-20 14:14:53 +08:00
Yuxuan Xia
71875ebc24 Fix the C-Eval nightly test trigger time (#10172)
* Add c-eval workflow and modify running files

* Modify the chatglm evaluator file

* Modify the ceval workflow for triggering test

* Modify the ceval workflow file

* Modify the ceval workflow file

* Modify ceval workflow

* Adjust the ceval dataset download

* Add ceval workflow dependencies

* Modify ceval workflow dataset download

* Add ceval test dependencies

* Add ceval test dependencies

* Correct the result print

* Fix the nightly test trigger time
2024-02-20 09:53:59 +08:00
WeiguangHan
6c09aed90d LLM: add qwen_1.5_7b model for arc perf test (#10166)
* LLM: add qwen_1.5_7b model for arc perf test

* small fix

* revert some codes
2024-02-19 17:21:00 +08:00
Yuxuan Xia
209122559a Add Ceval workflow and modify the result printing (#10140)
* Add c-eval workflow and modify running files

* Modify the chatglm evaluator file

* Modify the ceval workflow for triggering test

* Modify the ceval workflow file

* Modify the ceval workflow file

* Modify ceval workflow

* Adjust the ceval dataset download

* Add ceval workflow dependencies

* Modify ceval workflow dataset download

* Add ceval test dependencies

* Add ceval test dependencies

* Correct the result print
2024-02-19 17:06:53 +08:00
yb-peng
50fa004ba5 Specify the version of pandas in harness evaluation workflow (#10159)
* Specify the version of pandas in harness evaluation workflow

* Specify the version of pandas in harness evaluation workflow
2024-02-19 16:27:08 +08:00
Shaojun Liu
7a3a20cf5b Fix: GitHub-owned GitHubAction not pinned by hash (#10152) 2024-02-18 16:49:28 +08:00
Shaojun Liu
c3daacec6d Fix Token Permission issues (#10151)
Co-authored-by: Your Name <Your Email>
2024-02-18 13:23:54 +08:00
yb-peng
b7c5104d98 remove limit in harness run (#10139) 2024-02-09 11:20:53 +08:00
yb-peng
b4dc33def6 In harness-evaluation workflow, add statistical tables (#10118)
* chnage storage

* fix typo

* change label

* change label to arc03

* change needs in the last step

* add generate csv in harness/make_table_results.py

* modify needs in the last job

* add csv to html

* mfix path issue in llm-harness-summary-nightly

* modify output_path

* modify args in make_table_results.py

* modify make table command in summary

* change pr env label

* remove irrelevant code in summary; add set output path step; add limit in harness run

* re-organize code structure

* modify limit in run harness

* modify csv_to_html input path

* modify needs in summary-nightly
2024-02-08 19:01:05 +08:00
Shaojun Liu
c2378a9546 Fix code scanning issues (#10129)
* Fix code scanning issues

* update oneccl_bind_pt link

* update

* update

---------

Co-authored-by: Your Name <Your Email>
2024-02-08 17:19:44 +08:00
Yuwen Hu
81ed65fbe7 [LLM] Add qwen1.5-7B in iGPU perf (#10127)
* Add qwen1.5 test config yaml with transformers 4.37.0

* Update for yaml file
2024-02-07 22:31:20 +08:00
hxsz1997
b4c327ea78 Llm ppl workflow bug fix (#10128)
* add llm-ppl workflow

* update the DATASET_DIR

* test multiple precisions

* modify nightly test

* match the updated ppl code

* add matrix.include

* fix the include error

* update the include

* add more model

* update the precision of include

* update nightly time and add more models

* fix the workflow_dispatch description, change default model of pr and modify the env

* modify workflow_dispatch language options

* modify options

* modify language options

* modeify workflow_dispatch type

* modify type

* modify the type of language

* change seq_len type
2024-02-07 18:48:14 +08:00
hxsz1997
76bd792ff1 Fix llm ppl workflow workflow_dispatch bugs (#10125)
* add llm-ppl workflow

* update the DATASET_DIR

* test multiple precisions

* modify nightly test

* match the updated ppl code

* add matrix.include

* fix the include error

* update the include

* add more model

* update the precision of include

* update nightly time and add more models

* fix the workflow_dispatch description, change default model of pr and modify the env

* modify workflow_dispatch language options

* modify options

* modify language options
2024-02-07 17:41:44 +08:00
Shaojun Liu
9f5a86f9db fix OpenSSF Token-Permissions issues (#10121)
Co-authored-by: Your Name <Your Email>
2024-02-07 16:51:10 +08:00
hxsz1997
1710ecb990 Add llm-ppl workflow (#10074)
* add llm-ppl workflow

* update the DATASET_DIR

* test multiple precisions

* modify nightly test

* match the updated ppl code

* add matrix.include

* fix the include error

* update the include

* add more model

* update the precision of include

* update nightly time and add more models

* fix the workflow_dispatch description, change default model of pr and modify the env
2024-02-07 16:29:57 +08:00
Shaojun Liu
5e9710cec4 Update threshold for cpu stable version tests (#10108)
* update threshold

* update

* test

* update

* update

* revert

* revert

---------

Co-authored-by: Your Name <Your Email>
2024-02-07 11:21:23 +08:00
pengyb2001
f63eba6c5a change pr test machine 2024-02-06 23:35:18 +08:00
pengyb2001
e627727b4b change download path 2024-02-06 21:12:51 +08:00
pengyb2001
2c4e610743 remove irrelevant code 2024-02-06 20:12:10 +08:00
pengyb2001
d11ef0d117 remove retry in llm install part 2024-02-06 14:25:26 +08:00
pengyb2001
94723bb0b1 add retry in run llm install part;test arc05 with llama2 2024-02-06 14:09:14 +08:00
pengyb2001
2c75b5b981 remove mistral in pr job 2024-02-06 13:51:57 +08:00
pengyb2001
5edefe7d8e remove nightly summary job 2024-02-06 13:50:38 +08:00
pengyb2001
bc92dbf7be remove stableml;change schedule;change storage method 2024-02-06 11:20:37 +08:00
dingbaorong
36c9442c6d Arc Stable version test (#10087)
* add batch_size in stable version test

* add batch_size in excludes

* add excludes for batch_size

* fix ci

* triger regression test

* fix xpu version

* disable ci

* address kai's comment

---------

Co-authored-by: Ariadne <wyn2000330@126.com>
2024-02-06 10:23:50 +08:00
Yuwen Hu
136f042f84 [LLM] Make sure python 310-311 tests only happen for nightly tests (#10081)
* Make sure python 310-311 tests only happen for nightly tests

* Use default runner for setup-python-version

* Small fixes
2024-02-04 16:14:48 +08:00
Jin Qiao
f9a468a2c7 LLM: conditionally choose python version for unit test (#10062)
* conditional python version

* retry

* temporary skip llm-cpp-build

* apply on llm-unit-test-on-arc

* fix

* add llm-cpp-build dependency

* use GITHUB_OUTPUT instead of set-output

* check nightly build

* fix quote

* fix quote

* add llm-cpp-build dependency

* test nightly build

* test pull request
2024-02-04 13:37:34 +08:00
yb-peng
738275761d In llm-harness-evaluation, add new models and change schedule to nightly (#10072)
* add new models and change schedule to nightly

* correct syntax error

* modify env set up and job

* change label and schedule time

* change schedule time

* change label
2024-02-04 13:12:09 +08:00
Shaojun Liu
698f84648c split stable version tests (#10076)
Co-authored-by: Your Name <Your Email>
2024-02-04 11:08:12 +08:00
WeiguangHan
d2d3f6b091 LLM: ensure the result of daily arc perf test (#10016)
* ensure the result of daily arc perf test

* small fix

* small fix

* small fix

* small fix

* small fix

* small fix

* small fix

* small fix

* small fix

* small fix

* concat more csvs

* small fix

* revert some files
2024-01-31 18:26:21 +08:00
Yuwen Hu
863c3f94d0 [LLM] Change nightly perf to install from pypi (#10027)
* Change to install from pypi and have a check to make sure the installed bigdl-llm version is as expected

* Make sure result date is the same as tested bigdl-llm version

* Small fixes

* Small fix

* Small fixes

* Small fix

* Small fixes

* Small updates
2024-01-30 18:15:44 +08:00
Yuwen Hu
68d78fb57e [LLM] Small improvement to iGPU perf test #9915)
- Avoid delete csv if there is something wrong with concating csv
2024-01-17 11:21:58 +08:00
Yuwen Hu
8643b62521 [LLM] Support longer context in iGPU perf tests (2048-256) (#9910) 2024-01-16 17:48:37 +08:00
Yuwen Hu
023679459e [LLM] Small fixes for finetune related examples and UTs (#9870) 2024-01-09 18:05:03 +08:00
Yuwen Hu
aebed4b7bc Enable llm gpu tests for PyTorch 2.1 (#9863) 2024-01-09 16:29:02 +08:00
Yuwen Hu
5ba1dc38d4 [LLM] Change default Linux GPU install option to PyTorch 2.1 (#9858)
* Update default xpu to ipex 2.1

* Update related install ut support correspondingly

* Add arc ut tests for both ipex 2.0 and 2.1

* Small fix

* Diable ipex 2.1 test for now as oneapi 2024.0 has not beed installed on the test machine

* Update document for default PyTorch 2.1

* Small fix

* Small fix

* Small doc fixes

* Small fixes
2024-01-08 17:16:17 +08:00
Chen, Zhentao
cad5c2f516 fixed harness deps version (#9854)
* fixed harness deps version

* fix typo
2024-01-08 15:22:42 +08:00
WeiguangHan
4269a585b2 LLM: arc perf test using ipex2.1 (#9837)
* LLM: upgrade to ipex_2.1 for arc perf test

* revert llm_performance_tests.yml
2024-01-05 18:12:19 +08:00
Yuwen Hu
f25d23dfbf [LLM] Add support for PyTorch 2.1 install in UT for GPU (#9845)
* Add support for ipex 2.1 install in UT and fix perf test

* Small fix
2024-01-05 16:13:18 +08:00
Yuwen Hu
ad4a6b5096 Fix langchain UT by not downgrading typing-extension (#9842) 2024-01-05 13:38:04 +08:00
Wang, Jian4
4ceefc9b18 LLM: Support bitsandbytes config on qlora finetune (#9715)
* test support bitsandbytesconfig

* update style

* update cpu example

* update example

* update readme

* update unit test

* use bfloat16

* update logic

* use int4

* set defalut bnb_4bit_use_double_quant

* update

* update example

* update model.py

* update

* support lora example
2024-01-04 11:23:16 +08:00
dingbaorong
f6bb4ab313 Arc stress test (#9795)
* add arc stress test

* triger ci

* triger CI

* triger ci

* disable ci
2023-12-27 21:02:41 +08:00
Yuwen Hu
dfe28c58bb Small upload fix for igpu-perf test (#9792) 2023-12-27 15:50:58 +08:00