ayo/ipex-llm
1
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

transformer int4 and native int4's benchmark script for 32 256 1k 2k input (#8871)

Browse source 
* transformer

* move

* update

* add header

* update all-in-one

* clean up
This commit is contained in:
Xin Qiu 2023-09-07 09:49:55 +08:00 committed by GitHub
parent bfc71fbc15
commit 5d9942a3ca
7 changed files with 176 additions and 50 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

7
python/llm/dev/benchmark/all-in-one/config.yaml
View file

@ -1 +1,6 @@
model: /path/to/model repo_id:
- 'THUDM/chatglm2-6b'
- 'meta-llama/Llama-2-7b-chat-hf'
local_model_hub: 'path to your local model hub'
warm_up: 1
num_trials: 3

16
python/llm/dev/benchmark/all-in-one/prompt/1024.txt Normal file
View file

@ -0,0 +1,16 @@
In the year 2048, the world was a very different place from what it had been just two decades before. The pace of technological progress had quickened to an almost unimaginable degree, and the changes that had swept through society as a result were nothing short of revolutionary.
In many ways, the year 2048 represented the culmination of a long and tumultuous journey that humanity had been on since the dawn of civilization. The great leaps forward in science and technology that had occurred over the course of the previous century had laid the groundwork for a future that was beyond anything anyone could have imagined.
One of the most striking aspects of life in 2048 was the degree to which technology had become an integral part of nearly every aspect of daily existence. From the moment people woke up in the morning until they went to bed at night, they were surrounded by devices and systems that were powered by advanced artificial intelligence and machine learning algorithms.
In fact, it was hard to find anything in people's lives that wasn't touched by technology in some way. Every aspect of society had been transformed, from the way people communicated with one another to the way they worked, played, and even socialized. And as the years went on, it seemed as though there was no limit to what technology could achieve.
Despite all of these advances, however, not everyone was happy with the state of the world in 2048. Some people saw the increasing reliance on technology as a sign that humanity was losing touch with its own humanity, and they worried about the implications of this for the future.
Others were more pragmatic, recognizing that while technology had brought many benefits, it also posed new challenges and risks that needed to be addressed. As a result, there was a growing movement of people who were working to ensure that the advances of technology were used in ways that were safe, ethical, and beneficial for everyone.
One person who was at the forefront of this movement was a young woman named Maya. Maya was a brilliant and ambitious researcher who had dedicated her life to understanding the implications of emerging technologies like artificial intelligence and biotechnology. She was deeply concerned about the potential risks and unintended consequences of these technologies, and she worked tirelessly to raise awareness about the need for responsible innovation.
Maya's work had earned her a reputation as one of the most influential voices in the field of technology and ethics, and she was widely respected for her deep understanding of the issues and her ability to communicate complex ideas in ways that were accessible and engaging. She was also known for her passionate and inspiring speeches, which often left her audiences with a sense of purpose and determination to make the world a better place through their own efforts.
One day, Maya received an invitation to speak at a major conference on technology and ethics, which was being held in a large convention center in the heart of the city. The conference was expected to attract thousands of people from all over the world, and there was a great deal of excitement and anticipation about what Maya would say.
As she prepared for her speech, Maya knew that she had a big responsibility on her shoulders. She felt a deep sense of obligation to use her platform to inspire others to take action and make a difference in the world, and she was determined to do everything in her power to live up to this responsibility.
When the day of the conference arrived, Maya was filled with a mixture of excitement and nerves. She spent hours rehearsing her speech and fine-tuning her ideas, making sure that she had everything just right. Finally, after what felt like an eternity, it was time for her to take the stage.
As she stepped up to the podium, Maya could feel the energy of the crowd surging around her. She took a deep breath and began to speak, her voice strong and clear as she outlined the challenges and opportunities facing society in the age of technology. She spoke passionately about the need for responsible innovation and the importance of considering the ethical implications of our actions, and she inspired many people in the audience to take up this cause and make a difference in their own lives.
Overall, Maya's speech was a resounding success, and she received countless messages of gratitude and appreciation from those who had heard her speak. She knew that there was still much work to be done, but she felt hopeful about the future and the role that technology could play in creating a better world for all.
As Maya left the stage and made her way back to her seat, she couldn't help but feel a sense of pride and accomplishment at what she had just accomplished. She knew that her words had the power to inspire others and make a real difference in the world, and she was grateful for the opportunity to have played a part in this important work.
For Maya, the future was full of promise and possibility, and she was determined to continue doing everything in her power to help create a brighter, more ethical world for everyone.
As she

41
python/llm/dev/benchmark/all-in-one/prompt/2048.txt Normal file
View file

@ -0,0 +1,41 @@
The sun was setting over the horizon, casting long shadows across the dusty ground of the town square. The last rays of light streamed through the gaps between the buildings, illuminating the cobblestones and the people milling about. A group of children played a rough game of tag, their laughter filling the air.
In the center of the square stood a lone figure, a man with a tired face and weary eyes. He was tall and broad shouldered, but his posture was slumped and his head hung low. Despite the warmth of the day, he wore a thick woolen coat that seemed too heavy for the weather.
The man looked out at the crowd, his gaze sweeping over the faces of the townspeople as they went about their business. His eyes settled on a young woman standing on the edge of the square, watching him with curiosity. She was pretty, with chestnut hair pulled back into a ponytail and bright green eyes that sparkled in the fading light.
The man felt a sudden jolt of recognition, as if he had seen her before. But he couldn't remember where or when. He tried to shake off the feeling, but it lingered like a ghostly presence.
Suddenly, the woman's eyes widened in alarm, and she began to run towards him. Her movements were urgent and panicked, as if she was trying to escape something or someone. The man watched her go, his confusion growing with each step.
He turned to look around the square, but there was no sign of anyone else. It was as if the woman had appeared out of nowhere, and now she was gone just as suddenly.
The man rubbed his temples, feeling a mounting sense of unease. He tried to make sense of what he had just seen, but it was like trying to grasp smoke in his hands. He shook his head, frustrated with himself for being so easily spooked.
Just then, a voice called out from behind him. "Hey there! You look lost."
The man turned to see a young man standing behind him, a friendly smile on his face. He was tall and lean, with tousled blond hair that seemed to glow in the fading light. His eyes were bright and curious, as if he was eager to know everything about the world around him.
The man hesitated for a moment, unsure of how to respond. He wasn't used to talking to strangers, especially ones who looked so young and innocent. But there was something about the boy that made him feel comfortable, as if he had known him all his life.
"I'm not lost," he said finally, his voice gruff but friendly. "Just a little confused, I guess."
The boy grinned. "Well, I can help with that! My name is Jake, by the way. What's your name?"
The man hesitated for a moment, then introduced himself as Michael. They chatted for a few minutes, exchanging small talk and pleasantries. But even as they spoke, the man couldn't shake the feeling that something was amiss. It was as if he had forgotten something important, something that he needed to remember before it was too late.
He excused himself from Jake, promising to come back later and chat some more. As he walked away, he felt a growing sense of unease. Something wasn't right, and he needed to figure out what it was before it was too late.
[CHAPTER 4:
THE LIBRARY](9781441125608_epub_itb-ch4.xhtml)
The man returned to the library, his mind still racing with thoughts of the mysterious boy and the strange feeling that had been nagging at him all day. He wandered through the shelves, scanning the titles of books and flipping through their pages, searching for something that might help him remember what he had forgotten.
It wasn't until he stumbled upon a book on ancient myths and legends that he felt a spark of recognition. As he read through the stories of gods and monsters, he began to recall fragments of memories from his own life. Memories of strange symbols carved into walls, of dreams filled with images of a dark forest and a mysterious figure.
He realized with a start that these were not just random memories, but pieces of a larger puzzle that had been scattered throughout his life. He felt a sudden urgency to put them together before they faded away completely.
As he continued to read through the book, he began to notice patterns and connections between the myths and his own memories. The symbols he had seen as a child were not just random carvings, but part of an ancient language that held the key to unlocking the secrets of his past.
With newfound determination, the man decided to use every resource at his disposal to solve the mystery of his life. He began to scour books and artifacts, piecing together fragments of information until they formed a clear picture of what had really happened to him.
The truth was stranger than he could have ever imagined, and it led him on a journey that would take him deep into the heart of the enigmatic city where he had lived his entire life. The deeper he delved, the more he realized that his past was connected to a dark conspiracy that had been hidden from him for his own protection.
As he uncovered the truth, he also discovered that he was not alone in his quest. There were others who shared his memories and knew the secrets of the city's hidden history. They banded together to form a small community, each contributing their unique talents to uncover the truth about their past lives.
Together, they began to piece together a stunning picture of a world that had been lost for centuries, a world where magic and technology had coexisted in perfect harmony. They discovered artifacts and documents that revealed the true extent of the city's power and influence, and how it had affected the course of human history.
But with this newfound knowledge came danger and intrigue, as powerful forces sought to keep the secrets of the past buried forever. The man and his companions found themselves in a desperate race against time, trying to uncover everything they could about their past lives before it was too late.
As the truth came into focus, the man realized that he had been given a second chance at life, a chance to make amends for the mistakes of his past and to ensure that the truth would never be forgotten again. He knew that his journey was far from over, but he was ready to face whatever lay ahead, armed with the knowledge that he had spent a lifetime accumulating.
[CHAPTER 5:
THE MYSTERIOUS FIGURE](9781441125608_epub_itb-ch5.xhtml)
The man's heart raced as he stumbled upon a clue that might lead him to the mysterious figure from his memories. He had been searching for weeks, scouring the city's forgotten corners and reading every book he could find that might hold a clue. And finally, he had found something.
It was a small, hand-drawn map that seemed to show the location of an underground chamber deep beneath the city. The map was old and faded, but the man could make out several landmarks that matched up with his own memories of the dark forest. He felt a thrill of excitement at the thought that he might finally uncover the truth about the figure he had seen so many years ago.
The man quickly gathered a team of fellow seekers and set out to find the chamber. They followed the map through a labyrinth of tunnels and passageways, each step echoing off the walls in the eerie darkness. The air was thick with the smell of damp earth and ancient stone, and the man could feel the weight of history bearing down on him as they went deeper into the underground maze.
Finally, after what felt like an eternity, they reached their destination: a small chamber filled with strange symbols and markings that seemed to be connected to the mysterious figure from his memories. The man's heart raced as he began to decipher the code, piecing together words and phrases that had been lost for centuries.
As he worked, a sudden noise interrupted him. He spun around, his hand already reaching for the hilt of his hidden blade, ready for whatever danger might be lurking in the shadows. But to his surprise, he saw a figure stepping out from behind a stack of crates, their face obscured by a hooded cloak.
The man's heart sank as he realized that this was not the mysterious figure from his memories, but someone else entirely. Still, he kept his wits about him and addressed the stranger cautiously.
"Who are you?" he asked, his voice low and steady. "What do you know about the one I'm looking for?"
The stranger hesitated for a moment before speaking in a hushed tone. "I might be able to help," they said. "But you have to promise me one thing: you must never reveal what you're about to hear to anyone. Not even your closest allies."
The man nodded, his mind racing with the possibilities of what he was about to learn. He knew that this stranger held the key to unlocking the secrets of his past, and he was ready to do whatever it took to get the answers he had been seeking for so long.
[CHAPTER 6:
THE DARK FOREST](9781441125608_epub_itb-ch6.xhtml)
The man awoke with a start, the dream still fresh in his mind. He could see it all so clearly, as if it were etched into his memory like a brand into his skin. The dark forest stretched out before him, the thick canopy of leaves overhead. It was then that he heard the rustling of leaves and the soft earth beneath him.
He stood there, amidst the trees, his senses strained with the sounds of the birds chirping and the wind whispering through the branches above. He looked down at the forest, dappled sunlight filtering through the leaves, casting shadows across the ground as it seemed to beaten gold it cast a warm light

4
python/llm/dev/benchmark/all-in-one/prompt/256.txt Normal file
View file

@ -0,0 +1,4 @@
Once upon a time, there was a young girl named Samantha who lived with her parents in a small town. Samantha had always dreamed of traveling the world and experiencing new cultures and adventures. But as much as she yearned for something more than what her town could offer, it felt out of reach for a girl like her.
One day, while browsing through the pages of a travel magazine, Samantha came across an advertisement that seemed too good to be true. It was an invitation to travel to a faraway land and experience all the adventures she had ever dreamed of. The only catch was that she needed to attend a special briefing beforehand in order to ensure her safety during the trip.
Samantha quickly scribbled down the information on the back of the advertisement and resolved to attend the briefing at the end of the week. As time passed, Samantha became more and more excited about the prospect of traveling abroad. She even began putting together a packing list, imagining all the things she would need for her adventure.
Finally, the day of the briefing arrived. Samantha made sure to arrive early so that she could go through security clearance before anyone else did. As she waited in line, she couldn't help but

1
python/llm/dev/benchmark/all-in-one/prompt/32.txt Normal file
View file

@ -0,0 +1 @@
Once upon a time, there existed a little girl who liked to have adventures. She wanted to go to places and meet new people, and have fun.

50
python/llm/dev/benchmark/all-in-one/prompt/8192.txt Normal file
View file

@ -0,0 +1,50 @@
The sun was setting over the horizon, casting long shadows across the dusty ground of the town square. The last rays of light streamed through the gaps between the buildings, illuminating the cobblestones and the people milling about. A group of children played a rough game of tag, their laughter filling the air.
In the center of the square stood a lone figure, a man with a tired face and weary eyes. He was tall and broad shouldered, but his posture was slumped and his head hung low. Despite the warmth of the day, he wore a thick woolen coat that seemed too heavy for the weather.
The man looked out at the crowd, his gaze sweeping over the faces of the townspeople as they went about their business. His eyes settled on a young woman standing on the edge of the square, watching him with curiosity. She was pretty, with chestnut hair pulled back into a ponytail and bright green eyes that sparkled in the fading light.
The man felt a sudden jolt of recognition, as if he had seen her before. But he couldn't remember where or when. He tried to shake off the feeling, but it lingered like a ghostly presence.
Suddenly, the woman's eyes widened in alarm, and she began to run towards him. Her movements were urgent and panicked, as if she was trying to escape something or someone. The man watched her go, his confusion growing with each step.
He turned to look around the square, but there was no sign of anyone else. It was as if the woman had appeared out of nowhere, and now she was gone just as suddenly.
The man rubbed his temples, feeling a mounting sense of unease. He tried to make sense of what he had just seen, but it was like trying to grasp smoke in his hands. He shook his head, frustrated with himself for being so easily spooked.
Just then, a voice called out from behind him. "Hey there! You look lost."
The man turned to see a young man standing behind him, a friendly smile on his face. He was tall and lean, with tousled blond hair that seemed to glow in the fading light. His eyes were bright and curious, as if he was eager to know everything about the world around him.
The man hesitated for a moment, unsure of how to respond. He wasn't used to talking to strangers, especially ones who looked so young and innocent. But there was something about the boy that made him feel comfortable, as if he had known him all his life.
"I'm not lost," he said finally, his voice gruff but friendly. "Just a little confused, I guess."
The boy grinned. "Well, I can help with that! My name is Jake, by the way. What's your name?"
The man hesitated for a moment, then introduced himself as Michael. They chatted for a few minutes, exchanging small talk and pleasantries. But even as they spoke, the man couldn't shake the feeling that something was amiss. It was as if he had forgotten something important, something that he needed to remember before it was too late.
He excused himself from Jake, promising to come back later and chat some more. As he walked away, he felt a growing sense of unease. Something wasn't right, and he needed to figure out what it was before it was too late.
[CHAPTER 4:
THE LIBRARY](9781441125608_epub_itb-ch4.xhtml)
The man returned to the library, his mind still racing with thoughts of the mysterious boy and the strange feeling that had been nagging at him all day. He wandered through the shelves, scanning the titles of books and flipping through their pages, searching for something that might help him remember what he had forgotten.
It wasn't until he stumbled upon a book on ancient myths and legends that he felt a spark of recognition. As he read through the stories of gods and monsters, he began to recall fragments of memories from his own life. Memories of strange symbols carved into walls, of dreams filled with images of a dark forest and a mysterious figure.
He realized with a start that these were not just random memories, but pieces of a larger puzzle that had been scattered throughout his life. He felt a sudden urgency to put them together before they faded away completely.
As he continued to read through the book, he began to notice patterns and connections between the myths and his own memories. The symbols he had seen as a child were not just random carvings, but part of an ancient language that held the key to unlocking the secrets of his past.
With newfound determination, the man decided to use every resource at his disposal to solve the mystery of his life. He began to scour books and artifacts, piecing together fragments of information until they formed a clear picture of what had really happened to him.
The truth was stranger than he could have ever imagined, and it led him on a journey that would take him deep into the heart of the enigmatic city where he had lived his entire life. The deeper he delved, the more he realized that his past was connected to a dark conspiracy that had been hidden from him for his own protection.
As he uncovered the truth, he also discovered that he was not alone in his quest. There were others who shared his memories and knew the secrets of the city's hidden history. They banded together to form a small community, each contributing their unique talents to uncover the truth about their past lives.
Together, they began to piece together a stunning picture of a world that had been lost for centuries, a world where magic and technology had coexisted in perfect harmony. They discovered artifacts and documents that revealed the true extent of the city's power and influence, and how it had affected the course of human history.
But with this newfound knowledge came danger and intrigue, as powerful forces sought to keep the secrets of the past buried forever. The man and his companions found themselves in a desperate race against time, trying to uncover everything they could about their past lives before it was too late.
As the truth came into focus, the man realized that he had been given a second chance at life, a chance to make amends for the mistakes of his past and to ensure that the truth would never be forgotten again. He knew that his journey was far from over, but he was ready to face whatever lay ahead, armed with the knowledge that he had spent a lifetime accumulating.
[CHAPTER 5:
THE MYSTERIOUS FIGURE](9781441125608_epub_itb-ch5.xhtml)
The man's heart raced as he stumbled upon a clue that might lead him to the mysterious figure from his memories. He had been searching for weeks, scouring the city's forgotten corners and reading every book he could find that might hold a clue. And finally, he had found something.
It was a small, hand-drawn map that seemed to show the location of an underground chamber deep beneath the city. The map was old and faded, but the man could make out several landmarks that matched up with his own memories of the dark forest. He felt a thrill of excitement at the thought that he might finally uncover the truth about the figure he had seen so many years ago.
The man quickly gathered a team of fellow seekers and set out to find the chamber. They followed the map through a labyrinth of tunnels and passageways, each step echoing off the walls in the eerie darkness. The air was thick with the smell of damp earth and ancient stone, and the man could feel the weight of history bearing down on him as they went deeper into the underground maze.
Finally, after what felt like an eternity, they reached their destination: a small chamber filled with strange symbols and markings that seemed to be connected to the mysterious figure from his memories. The man's heart raced as he began to decipher the code, piecing together words and phrases that had been lost for centuries.
As he worked, a sudden noise interrupted him. He spun around, his hand already reaching for the hilt of his hidden blade, ready for whatever danger might be lurking in the shadows. But to his surprise, he saw a figure stepping out from behind a stack of crates, their face obscured by a hooded cloak.
The man's heart sank as he realized that this was not the mysterious figure from his memories, but someone else entirely. Still, he kept his wits about him and addressed the stranger cautiously.
"Who are you?" he asked, his voice low and steady. "What do you know about the one I'm looking for?"
The stranger hesitated for a moment before speaking in a hushed tone. "I might be able to help," they said. "But you have to promise me one thing: you must never reveal what you're about to hear to anyone. Not even your closest allies."
The man nodded, his mind racing with the possibilities of what he was about to learn. He knew that this stranger held the key to unlocking the secrets of his past, and he was ready to do whatever it took to get the answers he had been seeking for so long.
[CHAPTER 6:
THE DARK FOREST](9781441125608_epub_itb-ch6.xhtml)
The man awoke with a start, the dream still fresh in his mind. He could see it all so clearly, as if it were etched into his memory like a brand into his skin. The dark forest stretched out before him, the thick canopy of leaves overhead. It was then that he heard the rustling of leaves and the soft earth beneath him.
He stood there, amidst the trees, his senses strained with the sounds of the birds chirping and the wind whispering through the branches above. He looked down at the forest, dappled sunlight filtering through the leaves, casting shadows across the ground as it seemed to beaten gold it cast a warm light
He took a step forward, his foot sinking into the soft earth. He felt a sudden jolt of recognition, as if he had been here before. But he couldn't remember when or why. He tried to shake off the feeling, but it lingered like a ghostly presence.
He walked deeper into the forest, the trees growing taller and closer together. The air grew thick with the scent of damp earth and decaying leaves. He could hear the distant sound of running water, the rush of a river or stream. He followed the sound, his heart pounding in his chest.
As he walked, the trees grew taller and closer together, until he was surrounded by a canopy of leaves that blocked out most of the light. He felt a sudden sense of unease, as if he was being watched. He tried to shake off the feeling, but it lingered like a ghostly presence.
He heard the sound of footsteps behind him, heavy and deliberate. He spun around, his hand already reaching for the hilt of his sword. But there was no one there. He was alone in the forest, surrounded by the silence of the trees.
He took a deep breath, trying to calm himself down. He knew that he was being foolish, that there was no one there. But the feeling persisted, as if he was being watched by an unseen force. He tried to shake off the feeling, but it lingered like a ghostly presence.
He walked on, his senses strained with the sounds of the forest. He could hear the rustling of leaves, the chirping of birds, and the distant sound of running water. But he could not shake off the feeling that he was being watched. He knew that he was alone in the forest, but he could not shake off the feeling that he was being followed.
He walked on, his heart pounding in his chest. He knew that he was being foolish, that there was no one there. But the feeling persisted, as if he was being watched by an unseen force. He tried to shake off the feeling, but it lingered like a ghostly presence.
He walked on, his senses strained with the sounds of the forest. He could hear the rustling of leaves, the chirping of birds, and the distant sound of running water. But he could not shake off the feeling that he was being watched. He knew that he was alone in the forest, but he could not shake off the feeling that he was being followed.
[CHAPTER 7:

107
python/llm/dev/benchmark/all-in-one/run.py
View file

@ -22,7 +22,8 @@ import argparse
from bigdl.llm.transformers import AutoModel, AutoModelForCausalLM from bigdl.llm.transformers import AutoModel, AutoModelForCausalLM
from transformers import AutoTokenizer from transformers import AutoTokenizer
import numpy as np
from datetime import date
import os import os
current_dir = os.path.dirname(os.path.realpath(__file__)) current_dir = os.path.dirname(os.path.realpath(__file__))
@ -31,69 +32,77 @@ import sys
sys.path.append(benchmark_util_path) sys.path.append(benchmark_util_path)
from benchmark_util import BenchmarkWrapper from benchmark_util import BenchmarkWrapper
input_str_32 = "Once upon a time, there existed a little girl who liked to have adventures. She wanted to go to places and meet new people, and have fun"
input_str_1024 = "It is done, and submitted. You can play 'Survival of the Tastiest' on Android, and on the web. Playing on the web works, but you have to simulate multiple touch for table moving and that can be a bit confusing. There is a lot I'd like to talk about. I will go through every topic, insted of making the typical what went right/wrong list. Concept Working over the theme was probably one of the hardest tasks which I had to face. Originally, I had an idea of what kind of game I wanted to develop, gameplay wise - something with a lot of enemies/actors, simple graphics, maybe set in space, controlled from a top-down view. I was confident that I could fit any theme around it. In the end, the problem with a theme like 'Evolution' in a game is that evolution is unassisted. It happens through several seemingly random mutations over time, with the most apt permutation surviving. This genetic car simulator is, in my opinion, a great example of actual evolution of a species facing a challenge. But is it a game? In a game, you need to control something to reach an objective. That control goes against what evolution is supposed to be like. If you allow the user to pick how to evolve something, it's not evolution anymore - it's the equivalent of intelligent design, the fable invented by creationists to combat the idea of evolution. Being agnostic and a Pastafarian, that's not something that rubbed me the right way. Hence, my biggest dillema when deciding what to create was not with what I wanted to create, but with what I did not. I didn't want to create an 'intelligent design' simulator and wrongly call it evolution. This is a problem, of course, every other contestant also had to face. And judging by the entries submitted, not many managed to work around it. I'd say the only real solution was through the use of artificial selection, somehow. So far, I haven't seen any entry using this at its core gameplay. Alas, this is just a fun competition and after a while I decided not to be as strict with the game idea, and allowed myself to pick whatever I thought would work out. My initial idea was to create something where humanity tried to evolve to a next level, but had some kind of foe trying to stop them from doing so. I kind of had this image of human souls flying in space towards a monolith or a space baby (all based in 2001: A Space Odyssey of course) but I couldn't think of compelling (read: serious) mechanics for that. Borgs were my next inspiration, as their whole hypothesis fit pretty well into the evolution theme. But how to make it work? Are you the borg, or fighting the Borg? The third and final idea came to me through my girlfriend, who somehow gave me the idea of making something about the evolution of Pasta. The more I thought about it the more it sounded like it would work, so I decided to go with it. Conversations with my inspiring co-worker Roushey (who also created the 'Mechanical Underdogs' signature logo for my intros) further matured the concept, as it involved into the idea of having individual pieces of pasta flying around and trying to evolve until they became all-powerful. A secondary idea here was that the game would work to explain how the Flying Spaghetti Monster came to exist - by evolving from a normal dinner table. So the idea evolved more or less into this: you are sitting a table. You have your own plate, with is your 'base'. There are 5 other guests at the table, each with their own plate. Your plate can spawn little pieces of pasta. You do so by 'ordering' them through a menu. Some pastas are better than others; some are faster, some are stronger. They have varying 'costs', which are debited from your credits (you start with a number of credits). Once spawned, your pastas start flying around. Their instinct is to fly to other plates, in order to conquer them (the objective of the game is having your pasta conquer all the plates on the table). But they are really autonomous, so after being spawned, you have no control over your pasta (think DotA or LoL creeps). Your pasta doesn't like other people's pasta, so if they meet, they shoot sauce at each other until one dies. You get credits for other pastas your own pasta kill. Once a pasta is in vicinity of a plate"
results = [] results = []
def run_model(model_name, repo_id_or_model_path):
first, rest = run_pipeline(repo_id_or_model_path, '32/32') def run_model(repo_id, local_model_hub=None, warm_up=1, num_trials=3):
results.append([model_name, first, rest, '32/32']) # TODO: make a parameter
first, rest = run_pipeline(repo_id_or_model_path, '1024/128') in_out_pairs = ['32-32', '1024-128']
results.append([model_name, first, rest, '1024/128']) result = run_transformer_int4(repo_id, local_model_hub, in_out_pairs, warm_up, num_trials)
for in_out_pair in in_out_pairs:
results.append([repo_id,
np.mean(result[in_out_pair], axis=0)[0],
np.mean(result[in_out_pair], axis=0)[1],
np.mean(result[in_out_pair], axis=0)[2],
in_out_pair])
def run_transformer_int4(repo_id,
def run_pipeline(repo_id_or_model_path, in_out_pair): local_model_hub,
if in_out_pair == '32/32': in_out_pairs,
prompt, n_predict = input_str_32, 32 warm_up,
num_trials,
device='cpu'):
if local_model_hub:
repo_model_name = repo_id.split("/")[1]
model_path = local_model_hub + "/" + repo_model_name
else: else:
prompt, n_predict = input_str_1024, 128 model_path = repo_id
model_path = repo_id_or_model_path
# Load model in 4 bit, # Load model in 4 bit,
# which convert the relevant layers in the model into INT4 format # which convert the relevant layers in the model into INT4 format
model = AutoModelForCausalLM.from_pretrained(model_path, st = time.perf_counter()
load_in_4bit=True, if repo_id in ['THUDM/chatglm-6b', 'THUDM/chatglm2-6b']:
trust_remote_code=True) model = AutoModel.from_pretrained(model_path, load_in_4bit=True, trust_remote_code=True, torch_dtype='auto')
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
else:
model = AutoModelForCausalLM.from_pretrained(model_path, load_in_4bit=True)
tokenizer = AutoTokenizer.from_pretrained(model_path)
end = time.perf_counter()
print(">> loading of model costs {}s".format(end - st))
model = BenchmarkWrapper(model) model = BenchmarkWrapper(model)
# Load tokenizer result = {}
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
# Generate predicted tokens
with torch.inference_mode(): with torch.inference_mode():
input_ids = tokenizer.encode(prompt, return_tensors="pt") for in_out in in_out_pairs:
st = time.time() in_out_len = in_out.split("-")
# if your selected model is capable of utilizing previous key/value attentions in_len = int(in_out_len[0])
# to enhance decoding speed, but has `"use_cache": false` in its model config, out_len = int(in_out_len[1])
# it is important to set `use_cache=True` explicitly in the `generate` function input_str = open(f"prompt/{in_len}.txt", 'r').read()
# to obtain optimal performance with BigDL-LLM INT4 optimizations # As different tokenizer has different encodings,
output = model.generate(input_ids, # slice the input_ids to ensure the prompt length is required length.
max_new_tokens=n_predict) input_ids = tokenizer.encode(input_str, return_tensors="pt")
end = time.time() input_ids = input_ids[:, :in_len]
output_str = tokenizer.decode(output[0], skip_special_tokens=True) result[in_out] = []
print(f'Inference time: {end-st} s') for i in range(num_trials + warm_up):
print('-'*20, 'Prompt', '-'*20) st = time.perf_counter()
print(prompt) output_ids = model.generate(input_ids, do_sample=False, max_new_tokens=out_len)
print('-'*20, 'Output', '-'*20) end = time.perf_counter()
print(output_str) print("model generate cost: " + str(end - st))
output = tokenizer.batch_decode(output_ids)
return round(model.first_cost * 1000, 2), round(model.rest_cost_mean * 1000, 2) print(output[0])
if i >= warm_up:
result[in_out].append([model.first_cost, model.rest_cost_mean, model.encoder_time])
return result
if __name__ == '__main__': if __name__ == '__main__':
from omegaconf import OmegaConf from omegaconf import OmegaConf
conf = OmegaConf.load(f'{current_dir}/config.yaml') conf = OmegaConf.load(f'{current_dir}/config.yaml')
today = date.today()
import pandas as pd import pandas as pd
for model in conf.keys(): for model in conf.repo_id:
run_model(model, conf[model]) run_model(model, conf['local_model_hub'], conf['warm_up'], conf['num_trials'])
df = pd.DataFrame(results, columns=['model', '1st token avg latency (ms/token)', '2+ avg latency (ms/token)', 'input/output tokens']) df = pd.DataFrame(results, columns=['model', '1st token avg latency (s)', '2+ avg latency (s/token)', 'encoder time (s)', 'input/output tokens'])
df.to_csv(f'{current_dir}/results.csv') df.to_csv(f'{current_dir}/results-{today}.csv')