Commit graph

78 commits

Author SHA1 Message Date
Xu, Shuo
64cfed602d
Add new models to benchmark (#11505)
* Add new models to benchmark

* remove Qwen/Qwen-VL-Chat to pass the validation

---------

Co-authored-by: ATMxsp01 <shou.xu@intel.com>
2024-07-08 10:35:55 +08:00
Yuwen Hu
8f376e5192
Change igpu perf to mainly test int4+fp16 (#11513) 2024-07-05 17:12:33 +08:00
Shaojun Liu
932ef78131
Update Workflow Inputs, Runner, and PR Validation Process (#11501)
* update check-artifact runner label to Shire

* update github.event.inputs to inputs

* update PR template
2024-07-03 16:49:54 +08:00
Yuwen Hu
4e32c92979
Further fix for triggering perf test from commit (#11493)
* Further fix for triggering perf test from commit

* Small fix
2024-07-02 18:56:53 +08:00
Yuwen Hu
986b10e397
Further fix for performance tests triggered by pr (#11488) 2024-07-02 15:29:42 +08:00
Yuwen Hu
bb6953c19e
Support pr validate perf test (#11486)
* Support triggering performance tests through commits

* Small fix

* Small fix

* Small fixes
2024-07-02 15:20:42 +08:00
Yuwen Hu
ca24794dd0
Fixes for performance test triggering (#11481) 2024-07-01 18:39:54 +08:00
Yuwen Hu
6bdc562f4c
Enable triggering nightly tests/performance tests from another repo (#11480)
* Enable triggering from another workflow for nightly tests and example tests

* Enable triggering from another workflow for nightly performance tests
2024-07-01 17:45:42 +08:00
Yuwen Hu
75f836f288
Add extra warmup for THUDM/glm-4-9b-chat in igpu-performance test (#11417) 2024-06-24 18:08:05 +08:00
Shaojun Liu
5e823ef2ce
Fix nightly arc perf (#11404)
* pip install pytest for arc perf test

* trigger test
2024-06-24 15:58:41 +08:00
Wenjing Margaret Mao
c0e86c523a
Add qwen-moe batch1 to nightly perf (#11369)
* add moe

* reduce 437 models

* rename

* fix syntax

* add moe check result

* add 430 + 437

* all modes

* 4-37-4 exclud

* revert & comment

---------

Co-authored-by: Yishuo Wang <yishuo.wang@intel.com>
2024-06-20 14:17:41 +08:00
Wenjing Margaret Mao
b2f62a8561
Add batch 4 perf test (#11355)
* copy files to this branch

* add tasks

* comment one model

* change the model to test the 4.36

* only test batch-4

* typo

* typo

* typo

* typo

* typo

* typo

* add 4.37-batch4

* change the file name

* revet yaml file

* no print

* add batch4 task

* revert

---------

Co-authored-by: Yishuo Wang <yishuo.wang@intel.com>
2024-06-20 09:48:52 +08:00
Yuwen Hu
a2a5890b48
Make manually-triggered perf test able to choose which test to run (#11324) 2024-06-17 10:23:13 +08:00
Yuwen Hu
1978f63f6b
Fix igpu performance guide regarding html generation (#11328) 2024-06-17 10:21:30 +08:00
Wenjing Margaret Mao
b61f6e3ab1
Add update_parent_folder for nightly_perf_test (#11287)
* add update_parent_folder and change the workflow file

* add update_parent_folder and change the workflow file

* move to pr mode and comment the test

* use one model per comfig

* revert

---------

Co-authored-by: Yishuo Wang <yishuo.wang@intel.com>
2024-06-12 17:58:13 +08:00
Wenjing Margaret Mao
70b17c87be
Merge multiple batches (#11264)
* add merge steps

* move to pr mode

* remove build + add merge.py

* add tohtml and change cp

* change test_batch folder path

* change merge_temp path

* change to html folder

* revert

* change place

* revert 437

* revert space

---------

Co-authored-by: Yishuo Wang <yishuo.wang@intel.com>
2024-06-07 18:38:45 +08:00
Wenjing Margaret Mao
c825a7e1e9
change the workflow file to test ftp (#11241)
* change the workflow to test ftp

* comment some models

* revert file

---------

Co-authored-by: Yishuo Wang <yishuo.wang@intel.com>
2024-06-06 16:53:19 +08:00
Wenjing Margaret Mao
231b968aba
Modify the check_results.py to support batch 2&4 (#11133)
* add batch 2&4 and exclude to perf_test

* modify the perf-test&437 yaml

* modify llm_performance_test.yml

* remove batch 4

* modify check_results.py to support batch 2&4

* change the batch_size format

* remove genxir

* add str(batch_size)

* change actual_test_casese in check_results file to support batch_size

* change html highlight

* less models to test html and html_path

* delete the moe model

* split batch html

* split

* use installing from pypi

* use installing from pypi - batch2

* revert cpp

* revert cpp

* merge two jobs into one, test batch_size in one job

* merge two jobs into one, test batch_size in one job

* change file directory in workflow

* try catch deal with odd file without batch_size

* modify pandas version

* change the dir

* organize the code

* organize the code

* remove Qwen-MOE

* modify based on feedback

* modify based on feedback

* modify based on second round of feedback

* modify based on second round of feedback + change run-arc.sh mode

* modify based on second round of feedback + revert config

* modify based on second round of feedback + revert config

* modify based on second round of feedback + remove comments

* modify based on second round of feedback + remove comments

* modify based on second round of feedback + revert arc-perf-test

* modify based on third round of feedback

* change error type

* change error type

* modify check_results.html

* split batch into two folders

* add all models

* move csv_name

* revert pr test

* revert pr test

---------

Co-authored-by: Yishuo Wang <yishuo.wang@intel.com>
2024-06-05 15:04:55 +08:00
Yuwen Hu
9f8074c653
Add extra warmup for chatglm3-6b in igpu-performance test (#11197)
* Add extra warmup for chatglm3-6b to record more stable performance (int4+fp32)

* Small updates
2024-06-04 14:06:09 +08:00
Yina Chen
b6b70d1ba0
Divide core-xe packages (#11131)
* temp

* add batch

* fix style

* update package name

* fix style

* add workflow

* use temp version to run uts

* trigger performance test

* trigger win igpu perf

* revert workflow & setup
2024-05-28 12:00:18 +08:00
Jiao Wang
0a06a6e1d4
Update tests for transformers 4.36 (#10858)
* update unit test

* update

* update

* update

* update

* update

* fix gpu attention test

* update

* update

* update

* update

* update

* update

* update example test

* replace replit code

* update

* update

* update

* update

* set safe_serialization false

* perf test

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* delete

* update

* update

* update

* update

* update

* update

* revert

* update
2024-05-24 10:26:38 +08:00
Yuwen Hu
b3027e2d60
Update for cpu install option in performance tests (#11060) 2024-05-17 10:33:43 +08:00
Yuwen Hu
8010af700f
Update igpu performance test to use pypi installed oneAPI (#11010) 2024-05-14 14:05:33 +08:00
Yuwen Hu
c936ba3b64
Small fix for supporting workflow dispatch in nightly perf (#10908) 2024-04-29 13:25:14 +08:00
Yuwen Hu
94b4e96fa6
Small updates for workflow-dispatch triggered nightly perf (#10902)
* Small fix for workflow-dispatch triggerd nightly perf

* Small fix
2024-04-28 11:27:20 +08:00
Yuwen Hu
7c290d3f92
Add workflow dispatch trigger to nightly perf (#10900) 2024-04-28 09:54:30 +08:00
Shaojun Liu
e10040b7f1
upgrade to python 3.11 (#10695) 2024-04-09 17:04:42 +08:00
Shaojun Liu
db7c5cb78f
update model path for spr perf test (#10687)
* update model path for spr perf test

* revert
2024-04-08 10:21:56 +08:00
Shaojun Liu
d18dbfb097
update spr perf test (#10644) 2024-04-03 15:53:55 +08:00
Cheen Hau, 俊豪
1c5eb14128
Update pip install to use --extra-index-url for ipex package (#10557)
* Change to 'pip install .. --extra-index-url' for readthedocs

* Change to 'pip install .. --extra-index-url' for examples

* Change to 'pip install .. --extra-index-url' for remaining files

* Fix URL for ipex

* Add links for ipex US and CN servers

* Update ipex cpu url

* remove readme

* Update for github actions

* Update for dockerfiles
2024-03-28 09:56:23 +08:00
Shaojun Liu
c563b41491
add nightly_build workflow (#10533)
* add nightly_build workflow

* add create-job-status-badge action

* update

* update

* update

* update setup.py

* release

* revert
2024-03-26 12:47:38 +08:00
Shaojun Liu
93e6804bfe
update nightly test (#10520)
* trigger nightly test

* trigger perf test

* update bigdl-llm to ipex-llm

* revert
2024-03-25 18:22:05 +08:00
Yuwen Hu
1579ee4421 [LLM] Add nightly igpu perf test for INT4+FP16 1024-128 (#10496) 2024-03-21 16:07:06 +08:00
Yuwen Hu
d45e577d8c [LLM] Test load_low_bit in iGPU perf test on Windows (#10313) 2024-03-04 18:03:57 +08:00
Shaojun Liu
bab2ee5f9e update nightly spr perf test (#10178)
* update nightly spr perf test

* update

* update runner lable

* update

* update

* update folder

* revert
2024-03-04 13:46:33 +08:00
Yuwen Hu
d85f7c78df Small fix for better trail (#10256) 2024-02-27 20:00:40 +08:00
WeiguangHan
6c09aed90d LLM: add qwen_1.5_7b model for arc perf test (#10166)
* LLM: add qwen_1.5_7b model for arc perf test

* small fix

* revert some codes
2024-02-19 17:21:00 +08:00
Shaojun Liu
7a3a20cf5b Fix: GitHub-owned GitHubAction not pinned by hash (#10152) 2024-02-18 16:49:28 +08:00
Shaojun Liu
c2378a9546 Fix code scanning issues (#10129)
* Fix code scanning issues

* update oneccl_bind_pt link

* update

* update

---------

Co-authored-by: Your Name <Your Email>
2024-02-08 17:19:44 +08:00
Yuwen Hu
81ed65fbe7 [LLM] Add qwen1.5-7B in iGPU perf (#10127)
* Add qwen1.5 test config yaml with transformers 4.37.0

* Update for yaml file
2024-02-07 22:31:20 +08:00
WeiguangHan
d2d3f6b091 LLM: ensure the result of daily arc perf test (#10016)
* ensure the result of daily arc perf test

* small fix

* small fix

* small fix

* small fix

* small fix

* small fix

* small fix

* small fix

* small fix

* small fix

* concat more csvs

* small fix

* revert some files
2024-01-31 18:26:21 +08:00
Yuwen Hu
863c3f94d0 [LLM] Change nightly perf to install from pypi (#10027)
* Change to install from pypi and have a check to make sure the installed bigdl-llm version is as expected

* Make sure result date is the same as tested bigdl-llm version

* Small fixes

* Small fix

* Small fixes

* Small fix

* Small fixes

* Small updates
2024-01-30 18:15:44 +08:00
Yuwen Hu
68d78fb57e [LLM] Small improvement to iGPU perf test #9915)
- Avoid delete csv if there is something wrong with concating csv
2024-01-17 11:21:58 +08:00
Yuwen Hu
8643b62521 [LLM] Support longer context in iGPU perf tests (2048-256) (#9910) 2024-01-16 17:48:37 +08:00
WeiguangHan
4269a585b2 LLM: arc perf test using ipex2.1 (#9837)
* LLM: upgrade to ipex_2.1 for arc perf test

* revert llm_performance_tests.yml
2024-01-05 18:12:19 +08:00
Yuwen Hu
dfe28c58bb Small upload fix for igpu-perf test (#9792) 2023-12-27 15:50:58 +08:00
Yuwen Hu
c38e18f2ff [LLM] Migrate iGPU perf tests to new machine (#9784)
* Move 1024 test just after 32-32 test; and enable all model for 1024-128

* Make sure python output encoding in utf-8 so that redirect to txt can always be success

* Upload results to ftp

* Small fix
2023-12-26 19:15:57 +08:00
Yuwen Hu
02436c6cce [LLM] Enable more long context in-out pairs for iGPU perf tests (#9765)
* Add test for 1024-128 and enable more tests for 512-64

* Fix date in results csv name to the time when the performance is triggered

* Small fix

* Small fix

* further fixes
2023-12-22 18:18:23 +08:00
Yuwen Hu
1c8c104bb8 [LLM] Small fixes for igpu win perf tests (#9756) 2023-12-22 15:51:03 +08:00
WeiguangHan
2d1bf20309 LLM: small fix llm_performance_tests.html (#9753)
* LLM: small fix llm_performance_tests.html

* reslove some comments

* revert the llm_performance_test.yaml
2023-12-22 13:55:01 +08:00