ayo/ipex-llm
1
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1
ipex-llm/python/llm/example/CPU/Speculative-Decoding
History
Jiao Wang 93146b9433
Reconstruct Speculative Decoding example directory (#11136) 
* update

* update

* update
2024-05-29 13:15:27 -07:00
..
EAGLE Reconstruct Speculative Decoding example directory (#11136) 2024-05-29 13:15:27 -07:00
Self-Speculation Reconstruct Speculative Decoding example directory (#11136) 2024-05-29 13:15:27 -07:00
README.md Reconstruct Speculative Decoding example directory (#11136) 2024-05-29 13:15:27 -07:00

README.md

Speculative-Decoding Examples on Intel CPU

This folder contains examples of running Speculative-Decoding Examples with IPEX-LLM on Intel CPU:

  • Self-Speculation: running BF16 inference for Huggingface Transformer model with self-speculative decoding with IPEX-LLM on Intel CPUs
  • EAGLE: running speculative sampling using EAGLE (Extrapolation Algorithm for Greater Language-model Efficiency) with IPEX-LLM on Intel CPUs