History

ZehuaCao 36eb8b2e96 Add llama3 speculative example (#10856 ) * Initial llama3 speculative example * update README * update README * update README		2024-04-23 17:03:54 +08:00
..
baichuan2
chatglm3
llama2
llama3	Add llama3 speculative example (#10856 )	2024-04-23 17:03:54 +08:00
mistral
mixtral
qwen
starcoder
vicuna
ziya
README.md

README.md

Self-Speculative Decoding for Large Language Model BF16 Inference using IPEX-LLM on Intel CPUs

You can use IPEX-LLM to run BF16 inference for any Huggingface Transformer model with self-speculative decoding on Intel CPUs. This directory contains example scripts to help you quickly get started to run some popular open-source models using self-speculative decoding. Each model has its own dedicated folder, where you can find detailed instructions on how to install and run it.

Verified Hardware Platforms

Intel Xeon SPR server

Recommended Requirements

To run these examples with IPEX-LLM, we have some recommended requirements for your machine, please refer to here for more information. Make sure you have installed ipex-llm before:

pip install --pre --upgrade ipex-llm[all]

Moreover, install IPEX 2.1.0, which can be done through pip install intel_extension_for_pytorch==2.1.0.