36a9e88104
* Speculative Starcoder on CPU * enable kv-cache pre-allocation * refine codes * refine * fix style * fix style * fix style * refine * refine * Update speculative.py * Update gptbigcode.py * fix style * Update speculative.py * enable mixed-datatype layernorm on top of torch API * adaptive dtype * Update README.md |
||
|---|---|---|
| .. | ||
| baichuan2 | ||
| chatglm3 | ||
| llama2 | ||
| mistral | ||
| qwen | ||
| starcoder | ||
| vicuna | ||
| ziya | ||
| README.md | ||
Self-Speculative Decoding for Large Language Model BF16 Inference using BigDL-LLM on Intel CPUs
You can use BigDL-LLM to run BF16 inference for any Huggingface Transformer model with self-speculative decoding on Intel CPUs. This directory contains example scripts to help you quickly get started to run some popular open-source models using self-speculative decoding. Each model has its own dedicated folder, where you can find detailed instructions on how to install and run it.
Verified Hardware Platforms
- Intel Xeon SPR server
Recommended Requirements
To run these examples with BigDL-LLM, we have some recommended requirements for your machine, please refer to here for more information. Make sure you have installed bigdl-llm before:
pip install --pre --upgrade bigdl-llm[all]
Moreover, install IPEX 2.1.0, which can be done through pip install intel_extension_for_pytorch==2.1.0.