History

Heyang Sun 177273c1a4 IPEX Speculative Support for Baichuan2 7B (#10112 ) * IPEX Speculative Support for Baichuan2 7B * fix license problems * refine		2024-02-19 09:12:57 +08:00
..
baichuan2	IPEX Speculative Support for Baichuan2 7B (#10112 )	2024-02-19 09:12:57 +08:00
llama2	LLM : Add llama ipex optimized (#10046 )	2024-01-31 10:38:46 +08:00
mistral	Mistral CPU example of speculative decoding (#10024 )	2024-02-01 10:52:32 +08:00
qwen	LLM: Add qwen CPU speculative example (#9985 )	2024-01-25 17:01:34 +08:00
vicuna	Vicuna CPU example of speculative decoding (#10018 )	2024-01-31 11:23:50 +08:00
README.md	LLama2 CPU example of speculative decoding (#9962 )	2024-01-31 09:45:20 +08:00

README.md

Self-Speculative Decoding for Large Language Model BF16 Inference using BigDL-LLM on Intel CPUs

You can use BigDL-LLM to run BF16 inference for any Huggingface Transformer model with self-speculative decoding on Intel CPUs. This directory contains example scripts to help you quickly get started to run some popular open-source models using self-speculative decoding. Each model has its own dedicated folder, where you can find detailed instructions on how to install and run it.

Verified Hardware Platforms

Intel Xeon SPR server

Recommended Requirements

To run these examples with BigDL-LLM, we have some recommended requirements for your machine, please refer to here for more information. Make sure you have installed bigdl-llm before:

pip install --pre --upgrade bigdl-llm[all]

Moreover, install IPEX 2.1.0, which can be done through pip install intel_extension_for_pytorch==2.1.0.