ayo/ipex-llm
1
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1
ipex-llm/python/llm/example/transformers/transformers_int4
History
Ruonan Wang e9aa2bd890 LLM: reduce GPU 1st token latency and update example (#8763) 
* reduce 1st token latency

* update example

* fix

* fix style

* update readme of gpu benchmark
2023-08-16 18:01:23 +08:00
..
baichuan
chatglm LLM: first update descriptions for ChatGLM transformers int4 example (#8646) 2023-08-02 11:00:56 +08:00
chatglm2 LLM: add chat & stream chat example for ChatGLM2 transformers int4 (#8636) 2023-08-01 14:57:45 +08:00
dolly_v1
dolly_v2 [LLM] Add more transformers int4 example (Dolly v2) (#8571) 2023-07-19 18:20:16 +08:00
falcon
GPU LLM: reduce GPU 1st token latency and update example (#8763) 2023-08-16 18:01:23 +08:00
internlm
llama2 [LLM] Add more transformers int4 example (Llama 2) (#8602) 2023-07-25 09:21:12 +08:00
moss
mpt
phoenix
qwen LLM: add Qwen transformers int4 example (#8699) 2023-08-08 11:23:09 +08:00
redpajama
starcoder [LLM] Add more transformers int4 example (starcoder) (#8540) 2023-07-17 14:41:19 +08:00
vicuna
whisper LLM: add whisper example for arc transformer int4 (#8749) 2023-08-14 17:05:48 +08:00
README.md LLM: add Qwen transformers int4 example (#8699) 2023-08-08 11:23:09 +08:00

README.md

BigDL-LLM Transformers INT4 Optimization for Large Language Model

You can use BigDL-LLM to run any Huggingface Transformer models with INT4 optimizations on either servers or laptops. This directory contains example scripts to help you quickly get started using BigDL-LLM to run some popular open-source models in the community. Each model has its own dedicated folder, where you can find detailed instructions on how to install and run it.

Verified models

Model Example
LLaMA link
LLaMA 2 link
MPT link
Falcon link
ChatGLM link
ChatGLM2 link
MOSS link
Baichuan link
Dolly-v1 link
Dolly-v2 link
RedPajama link
Phoenix link
StarCoder link
InternLM link
Whisper link
Qwen link

Recommended Requirements

To run the examples, we recommend using Intel® Xeon® processors (server), or >= 12th Gen Intel® Core™ processor (client).

For OS, BigDL-LLM supports Ubuntu 20.04 or later, CentOS 7 or later, and Windows 10/11.

Best Known Configuration on Linux

For better performance, it is recommended to set environment variables on Linux with the help of BigDL-Nano:

pip install bigdl-nano
source bigdl-nano-init