dingbaorong 5a2ce421af add cpu and gpu examples of flan-t5 (#9171 )

* add cpu and gpu examples of flan-t5

* address yuwen's comments
* Add explanation  why we add modules to not convert
* Refine prompt and add a translation example
* Add a empty line at the end of files

* add examples of flan-t5 using optimize_mdoel api

* address bin's comments

* address binbin's comments

* add flan-t5 in readme

2023-10-24 15:24:01 +08:00

2.5 KiB

Raw Blame History

BigDL-LLM INT4 Optimization for Large Language Model on Intel GPUs

You can use optimize_model API to accelerate general PyTorch models on Intel GPUs. This directory contains example scripts to help you quickly get started using BigDL-LLM to run some popular open-source models in the community. Each model has its own dedicated folder, where you can find detailed instructions on how to install and run it.

Verified models

Model	Example
Mistral	link
LLaMA 2	link
ChatGLM2	link
Baichuan	link
Baichuan2	link
Replit	link
StarCoder	link
Dolly v1	link
Dolly v2	link
Flan-t5	link

Verified Hardware Platforms

Intel Arc™ A-Series Graphics
Intel Data Center GPU Flex Series
Intel Data Center GPU Max Series

Recommended Requirements

To apply Intel GPU acceleration, there’re several steps for tools installation and environment preparation.

Step 1, only Linux system is supported now, Ubuntu 22.04 is prefered.

Step 2, please refer to our driver installation for general purpose GPU capabilities.

Note

: IPEX 2.0.110+xpu requires Intel GPU Driver version is Stable 647.21.

Step 3, you also need to download and install Intel® oneAPI Base Toolkit. OneMKL and DPC++ compiler are needed, others are optional.

Note

: IPEX 2.0.110+xpu requires Intel® oneAPI Base Toolkit's version >= 2023.2.0.

Best Known Configuration on Linux

For better performance, it is recommended to set environment variables on Linux:

export USE_XETLA=OFF
export SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS=1

2.5 KiB Raw Blame History Unescape Escape