Add task in allinone (#11226)

* add task

* update prompt

* modify typos

* add more cases in summarize

* Make the summarize & QA prompt preprocessing as a util function

python/llm/dev/benchmark/all-in-one/config.yaml

@@ -39,3 +39,4 @@ use_fp16_torch_dtype: True # whether use fp16 for non-linear layer (only availab
 n_gpu: 2 # number of GPUs to use (only available now for "pipeline_parallel_gpu" test_api)
 lookahead: 3
 max_matching_ngram_size: 2
+task: 'continuation' # when test_api is "transformer_int4_fp16_lookahead_gpu", task could be 'QA', 'continuation' or 'summarize' 

python/llm/dev/benchmark/all-in-one/prompt/QA/orca_497.txt

@@ -0,0 +1,101 @@
+<s>[INST] <<SYS>>
+You are an AI assistant. User will you give you a task. Your goal is to complete the task as faithfully as you can. While performing the task think step-by-step and justify your steps.
+<</SYS>>
+
+News article:
+
+(Carolyn Kaster/AP Photo) 
+  
+ Weary of waiting for an economic recovery worth its name, a frustrated American public has sent Barack Obama's job approval rating to a career low - with a majority in the latest ABC News/Washington Post poll favoring a Republican Congress to act as a check on his policies. 
+  
+ Registered voters by 53-39 percent in the national survey say they'd rather see the Republicans in control of Congress as a counterbalance to Obama's policies than a Democratic-led Congress to help support him. It was similar in fall 2010, when the Republicans took control of the House of Representatives and gained six Senate seats. 
+  
+ See PDF with full results and charts here. 
+  
+ Obama's job approval rating, after a slight winter rebound, has lost 5 points among all adults since March, to 41 percent, the lowest of his presidency by a single point. Fifty-two percent disapprove, with "strong" disapproval exceeding strong approval by 17 percentage points. He's lost ground in particular among some of his core support groups. 
+  
+ Economic discontent remains the driving element in political views in this survey, produced for ABC by Langer Research Associates. Americans rate the condition of the economy negatively by 71-29 percent - the least bad since November 2007, but still dismal by any measure. Only 28 percent think the economy's improving, down by 9 points since just before Obama won his second term. He gets just 42 percent approval for handling it. 
+  
+ Economic views are strongly related to political preferences. Among people who see the economy improving, 65 percent prefer Democratic control of Congress, while among those who see the economy as stagnant or worsening, 62 percent favor Republican control. Notably, economic views are linked with preferences for control of Congress regardless of people's partisan affiliation. 
+  
+ The results suggest the corrosive effects of the long downturn on the president's popularity: Among those who say the economy is in bad shape, Obama's overall approval rating has lost 20 points since February 2012, from 46 percent then to 26 percent now. 
+  
+ The president faces other challenges. While he's hailed insurance exchange sign-ups as a marker of the Affordable Care Act's success, the program and his rating for handling it have lost ground, both down from their levels late last month after the Healthcare.gov website was stabilized. The law gets 44 percent support, down 5 points; Obama has just 37 percent approval for its implementation, down 7. 
+  
+ One reason is that the law seems to have opened an avenue for public ire about health care costs to be directed at the administration. Six in 10 blame the ACA for increasing costs nationally, and 47 percent think it's caused their own health care expenses to rise. Regardless of whether or how much those costs would have risen otherwise, Obamacare is taking a heavy dose of the blame. 
+  
+ Separately, a current issue on the world stage offers no respite for Obama: Given continued tensions over Ukraine, just 34 percent of Americans approve of how he's handling that situation, 8 points fewer than early last month. Forty-six percent disapprove, with two in 10 withholding judgment. 
+  
+ DISCONTENT/MIDTERMS - With these and other problems - but chiefly the economy - the public by more than 2-1, 66-30 percent, says the country's headed seriously off on the wrong track. That's about where it's been lately, and more negative than a year ago. 
+  
+ General anti-incumbency results: Just 22 percent of Americans say they're inclined to re-elect their representative in Congress, unchanged from last month as the fewest in ABC/Post polls dating back 25 years. 
+  
+ Another outcome is risk for the president's party, in punishment for his handling of the helm. A single point divides Democratic and Republican candidates for the House in preference among registered voters, 45-44 percent. Among those who say they're certain to vote (with Republicans more apt to show up in midterms), that goes to 44-49 percent. 
+  
+ Independents, a sometimes swing-voting group, favor Republican House candidates by 55-32 percent (among those who say they're certain to vote). And, as with views on control of Congress, perceptions of the economy correlate with congressional vote preference, regardless of partisanship. 
+  
+ ISSUES - None of this means the GOP is home free. A robust improvement in the economy could change the equation. (As many, at least, say it's currently holding steady, 35 percent, as think it's getting worse, 36 percent.) And even as the brunt of economic unhappiness falls on the president, the public divides essentially evenly on which party they trust more to handle the economy - suggesting that the Republicans have yet to present a broadly appealing alternative. 
+  
+ In another example, for all of Obamacare's controversies, the Democrats hold a slight 8-point edge in trust to handle health care, again indicating that the Republicans have yet to seize the opportunity to present a compelling solution of their own. Indeed, the Democrats have a 6-point lead in trust to handle "the main problems the nation faces" - although, as with all others, that narrows among likely voters, in this case to 37-40 percent, a numerical (but not significant) GOP edge. 
+  
+ The Republicans have a 9-point advantage in trust to handle the federal deficit - an improvement for the party from last month. Similarly, Americans by a 7-point margin trust the Republicans over Obama to find the right mix of spending to cut and federal programs to maintain. The president had an 11-point lead on that question just after the partial government shutdown last fall. 
+  
+ The Democrats push back with two results that they're likely to stress as the November election draws closer: One is a broad, 20-point advantage, 52-32 percent, in trust over the Republicans to help the middle class (but again, this narrows among likely voters). The other is an even wider, 30-point lead, 55-25 percent, in trust to handle issues of particular concern to women. 
+  
+ The Republicans have some vulnerability in other areas, as well. Americans say the Democratic Party comes closer than the GOP to their positions on climate change, by 18 points; whether or not to raise the minimum wage, by 16 points; gay marriage, by 14 points; and the issue of abortion, by 8 points. On one remaining issue, gun control, the Republicans have a slight, 5-point edge. 
+  
+ HEALTH CARE - Obamacare, for its part, is a subject the Republicans have sought to turn to their advantage in the midterm elections, and the poll results show ample opportunity. 
+  
+ Costs are a particular target. As noted, 47 percent of Americans feel that their health care costs are rising as a result of the ACA; 58 percent say the same about the overall costs of health care nationally. Just 8 and 11 percent, respectively, say the law has decreased these costs. If there's a case to be made that costs would have risen anyway - or that they would have risen faster absent the ACA - it's yet to resonate with large segments of the population. 
+  
+ Other assessments also are critical. The public by a 20-point margin, 44-24 percent, is more apt to say the law has made the overall health care system worse rather than better (although the number who say it's made things better is up by 5 points from December). The rest, 29 percent, see no change. Americans by 29-14 percent likewise say the ACA has made their own care worse rather than better, with more, 53 percent, reporting no impact. 
+  
+ Despite the website's improvements, half say the law's implementation is going worse than they expected when it began, vs. 41 percent better - another sign of the persistent antipathy that's dogged Obamacare from the start. 
+  
+ The poll also shows both the striking partisan division on Obamacare and the extent to which, on several questions, independents side more with Republicans on the issue. Thirty-eight percent of Democrats, for instance, say the ACA has increased health care costs nationally; that soars to 67 percent of independents and 73 percent of Republicans. And while 47 percent of Democrats think it's made the health care system better, just 6 and 16 percent of Republicans and independents, respectively, agree. 
+  
+ OBAMA/GROUPS - Divisions among groups remain especially stark in terms of Obama's ratings; further, as noted, he's lost ground in some of his core support groups. The president's approval rating since early March has lost 14 points among liberals, 12 points among people with postgraduate degrees, 10 points among urban residents, 9 points among Democrats and 7 points among those with incomes less than $50,000. He's lost 9 points among independents as well. 
+  
+ With 41 percent approval overall (his previous low was 42 percent last November and the same in October 2011), Obama's at new lows among nonwhites (61-34 percent, approve-disapprove) and liberals (63-31 percent), and matches his lows among moderates (46-48 percent) and independents (33-59 percent). His rating among Democrats, 74-22 percent, is a single point from its low. 
+  
+ Other results also mark the extent of the difficulties facing Obama and his party alike. A form of statistical analysis called regression finds that, as noted above, views on the economy correlate both with congressional vote preference, and views on which party should control Congress, independently of partisan affiliation. That suggests that the Democrats are in serious need of a positive shift in economic views. 
+  
+ That may be hard to accomplish. While 50 percent of Democrats say the economy's in good shape, that plummets not only among Republicans but independents as well, to 12 and 22 percent, respectively. And while 46 percent of Democrats see improvement in the economy, again just 22 percent of independents, and 15 percent of Republicans, agree. 
+  
+ Preferences on which party controls Congress may reflect a general inclination in favor of divided government - and don't always predict outcomes, as in 2002, when more registered voters preferred Democratic control yet the GOP held its ground. It's striking, nonetheless, that this poll finds Republican control favored not only in the 2012 red states, by 56-36 percent, but also by 51-41 percent in the blue states that backed Obama fewer than two years ago. 
+  
+ METHODOLOGY - This ABC News/Washington Post poll was conducted by telephone April 24-27, 2014, in English and Spanish, among a random national sample of 1,000 adults, including landline and cell-phone-only respondents. Results have a margin of sampling error of 3.5 points, including design effect. Partisan divisions are 32-21-38 percent, Democrats-Republicans-independents. 
+  
+ The survey was produced for ABC News by Langer Research Associates of New York, N.Y., with sampling, data collection and tabulation by Abt-SRBI of New York, N.Y. ||||| President Obama’s approval rating fell to 41 percent, down from 46 percent through the first three months of the year and the lowest of his presidency in Washington Post-ABC News polls. (Charles Dharapak/AP) 
+  
+ Democrats face serious obstacles as they look to the November elections, with President Obama’s approval rating at a new low and a majority of voters saying they prefer a Congress in Republican hands to check the president’s agenda, according to a new Washington Post-ABC News poll. 
+  
+ Obama’s approval rating fell to 41 percent, down from 46 percent through the first three months of the year and the lowest of his presidency in Post-ABC News polls. Just 42 percent approve of his handling of the economy, 37 percent approve of how he is handling the implementation of the Affordable Care Act and 34 percent approve of his handling of the situation involving Ukraine and Russia. 
+  
+ Obama’s low rating could be a significant drag on Democratic candidates this fall — past elections suggest that when approval ratings are as low as Obama’s, the president’s party is almost certain to suffer at the ballot box in November. 
+  
+ Republicans are favored to maintain control of the House, with the focus now on whether they can take control of the Senate. One key question about November is who will vote. Turnout in midterm elections is always lower than in presidential elections, and at this point, key elements of the Republican coalition — namely white voters and older voters — say they are more certain to cast ballots this fall than are younger voters and minorities, two groups that Democrats and Obama relied on in 2008 and 2012. 
+  
+ Democrats are not without assets as the midterm election campaigns intensify. Americans trust Democrats over Republicans by 40 to 34 percent to handle the country’s main problems. By significant margins, Americans see Democrats as better for the middle class and on women’s issues. Americans favor the Democrats’ positions on raising the minimum wage, same-sex marriage and on the broad issue of dealing with global climate change. 
+  
+ View Graphic Obama receives low marks as Democrats face midterm turnout challenge 
+  
+ Led by Obama, Democrats have sought to use many of these issues to draw contrasts with Republicans, both nationally and in states with the most competitive races. As yet, however, there is little evidence that those assets outweigh either the normal midterm disadvantages of the party that holds the White House or the dissatisfaction with the general direction of the country and Obama’s leadership generally. 
+  
+ The Affordable Care Act is expected to be a major issue in the midterm elections. Obama recently urged Democrats to defend the law energetically, particularly after the administration announced that 8 million people signed up for it during the initial enrollment period. Republicans are confident that opposition to the new law will energize their supporters. 
+  
+ The Post-ABC poll found that 44 percent say they support the law while 48 percent say they oppose it, which is about where it was at the end of last year and in January. Half of all Americans also say they think implementation is worse than expected. 
+  
+ Last month, a Post-ABC poll found 49 percent of Americans saying they supported the new law compared with 48 percent who opposed it. That finding was more positive for the administration than most other polls at the time. Democrats saw it as a possible leading indicator of a shift in public opinion, but that has not materialized. 
+  
+ A 58 percent majority say the new law is causing higher costs overall, and 47 percent say it will make the health-care system worse. While a majority say the quality of the health care they receive will remain the same, a plurality expect it to result in higher personal costs for that care. 
+  
+ A number of Democratic strategists are urging their candidates to campaign on a message that calls for continued implementation of the law, with some fixes. These strategists say that message is more popular than the “repeal and replace” theme of the Republicans. A separate poll Tuesday from the Kaiser Family Foundation finds nearly six in 10 want Congress to improve the law rather than repeal it and replace it with something new. 
+  
+ Democrats are hoping to put Republicans on the defensive on the question of “what next” for the Affordable Care Act. Republicans say they remain confident that the health-care issue will help them more in November. 
+  
+ Pessimism about the economy also persists, with more than seven in 10 describing the economy in negative terms. Public attitudes about the future of the economy are anything but rosy. Just 28 percent say they think the economy is getting better, while 36 percent say it is getting worse and 35 percent say it’s staying the same. 
+  
+ Americans express continued discontent about the country’s direction, with two-thirds saying things are on the wrong track. Asked whether each party’s incumbents deserve relection, at least six in 10 say they do not. 
+  
+ Among registered voters, 45 percent intend to vote for the Democratic candidate in House elections this fall, and 44 percent for the Republican candidate. Based on past elections, that close margin is troubling news for Democrats.. What is a shorter version of the above article? [/INST]

python/llm/dev/benchmark/all-in-one/prompt/QA/orca_776.txt

@@ -0,0 +1,6 @@
+<s>[INST] <<SYS>>
+You are an AI assistant that follows instruction extremely well. Help as much as you can.
+<</SYS>>
+
+This article: At Birmingham, Oliphant's team had reached a different conclusion. Oliphant had delegated the task to two German refugee scientists, Rudolf Peierls and Otto Frisch, who could not work on Oliphant's radar project because they were enemy aliens and therefore lacked the necessary security clearance. Francis Perrin had calculated the critical mass of uranium to be about 40 tonnes (39 long tons; 44 short tons). He reckoned that if a neutron reflector were placed around it, this might be reduced to 12 tonnes (12 long tons; 13 short tons). Peierls attempted to simplify the problem by using the fast neutrons produced by fission, thus omitting consideration of moderator. He too calculated the critical mass of a sphere of uranium in a theoretical paper written in 1939 to be "of the order of tons".Peierls knew the importance of the size of the critical mass that would allow a chain reaction to take place and its practical significance. In the interior of a critical mass sphere, neutrons are spontaneously produced by the fissionable material. A very small portion of these neutrons are colliding with other nuclei, while a larger portion of the neutrons are escaping through the surface of the sphere. Peierls calculated the equilibrium of the system, where the number of neutrons being produced equalled the number escaping.Niels Bohr had theorised that the rare uranium-235 isotope, which makes up only about 0.7% of natural uranium, was primarily responsible for fission with fast neutrons, although this was not yet universally accepted. Frisch and Peierls were thus able to revise their initial estimate of critical mass needed for nuclear fission in uranium to be substantially less than previously assumed. They estimated a metallic sphere of uranium-235 with a radius of 2.1 centimetres (0.83 in) could suff. What were the full names of the two people who calculated the critical mass of uranium?, what is it ?
+Answer: [/INST]

python/llm/dev/benchmark/all-in-one/prompt/QA/orca_99.txt

@@ -0,0 +1,5 @@
+<s>[INST] <<SYS>>
+You are an AI assistant. You will be given a task. You must generate a detailed and long answer.
+<</SYS>>
+
+You could go directly into the confessional (provided there's no one else in there or waiting outside), but sometimes it's nice to take a minute in the pew by yourself beforehand. You have this beautiful church probably almost all to yourself. Can you feel its energy resonating through you? Can you feel the majesty of the Lord's kingdom and how you're a part of it? Take a moment to kneel and pray with your head down and hands clasped together. Reflect on your faith and how you feel currently. Think about how you've been responding to God's call and how you've been living in the light of his love. When the priest is ready for you, of course. You'll probably see him there by his lonesome or someone else walk out just before you. Sit down either across from him or behind the screen -- it's totally up to you whether or not you prefer to remain anonymous. He won't treat you any differently either way. Make the sign of the cross upon his prompt, saying, "Bless me, Father, for I have sinned.  It has been (blank) since my last confession." This is your standard, traditional phrasing. However, if you just sit down and say hello, that's fine, too. The priest knows what he's doing. The Byzantine Rite is a bit different. The priest may sit to your side and put his epitrachelion on your head. He may then also do the Prayer of Absolution. But the idea remains the exact same -- just go wherever he takes you. Once you sit down and you've made the sign of the cross, just sit back and follow the priest's lead. He'll ask you how long it's been since your last confession (if you don't voluntarily offer that information), how you are feeling, maybe how your faith is going, and then ask you what sins you would like to talk about with him and God. It's just a casual conversation! Do not fret. There is absolutely zero pressure on your part. Again, as long as you come there with the intention of leaving with a clean heart, you're more than welcome in the church. There is no wrong way to go about confession! This part is intimidating, but think about it this way: the priest you're talking to has probably heard just about everything before. Whatever you have to say will not blow his mind. So when he asks, start rattling them off, from the most serious to the least. If he asks any questions, answer them, but do not feel the need to go into detail. A simple, "I did so and so," will suffice. Your priest is going to be very understanding. If you don't remember the exact timeframe, that's fine. If you don't remember your motivation, that's fine. All your priest cares about is that you're being as honest as possible and that your heart is in the right place. He'll talk you through everything, possibly asking about your intentions, but mainly just letting you know that God loves you, sin and all. If he has any ideas to bring you closer to God, he may suggest them at this juncture. He's there to help, after all. He will then ask you to make an Act of Contrition. That goes like this:  My God, I am sorry for my sins with all my heart.In choosing to do wrong and failing to do good,I have sinned against You whom I should loveabove all things. I firmly intend, with your help,to do penance, to sin no more, andto avoid whatever leads me to sin.Our Savior Jesus Christ suffered and died for us.In his name, my God, have mercy  (If you are a Roman Catholic, your act of contrition will go like this: Oh my God, I am very sorry for having offended thee. I detest all of my sins because of thy just punishment. But most of all, because they offend you, my God, who is all good and deserving of all my love. I firmly resolve with the help of thy grace, to sin no more, and to avoid the near occasion of sin. Amen. Don't worry! It won't be anything huge. You may even walk away just having to say a few meaningful prayers. Take the absolution to heart --. What is a one-sentence summary of the following article? [/INST]

python/llm/dev/benchmark/all-in-one/prompt/summarize/cnn_239.txt

@@ -0,0 +1 @@
+(CNN)Jason Rezaian has sat in jail in Iran for nearly nine months. The Washington Post's bureau chief in Tehran was arrested in July on unspecified allegations. It took more than four months for a judge to hear charges against him. They remained publicly undisclosed until last week. The Iranian-American will be tried soon on espionage, Tehran's chief justice said. He is accused of economic spying, the Post reported, citing Iranian state media. The Washington Post did not mince words on the allegation. "Any charges of that sort would be absurd, the product of fertile and twisted imaginations," the paper said in a statement. The State Department also reacted with term "absurd" after hearing of reports in Iran's press about the charges. "If the reports are true, these charges are absurd, should be immediately dismissed and Jason should be immediately freed so that he can return to his family," the State Department official said. Since officers picked up Rezaian and his wife, Yeganeh Salehi, on July 22 at their home, the Post, the State Department and Rezaian's family have protested and called for his release. Salehi was released on bail in October. Rezaian was denied bail. And for months, he was denied access to proper legal representation, his family has said. Boxing great Muhammad Ali, also an American Muslim, appealed to Tehran last month to give Rezaian full access to legal representation and free him on bail. "To my knowledge, Jason is a man of peace and great faith, a man whose dedication and respect for the Iranian people is evident in his work," Ali said in a religiously worded statement. The journalist has also not been allowed to see visitors aside from his wife and has endured long interrogations, family members have said. In December, after a 10-hour hearing, Rezaian signed a paper to acknowledge that he understood the charges against him, the Post reported. Iran's human rights chief, Mohammad Javad Larijani, told news outlet France 24 last year that he hoped Rezaian's case would come to a positive conclusion. He said, "Let us hope that this fiasco will end on good terms." More on detained Americans . CNN's Sara Mazloumsaki and Azadeh Ansari contributed to this report.

python/llm/dev/benchmark/all-in-one/prompt/summarize/cnn_615.txt

@@ -0,0 +1 @@
+(CNN)You might call her a watchdog, because this Boston area Doberman really has a thing for timepieces. She recently ate three. Last Thursday, Jeff Courcelle came home from work to find 5-year-old Mocha -- a pure bred fawn-colored Doberman pincher -- hovering over a pile of screws, metal pieces, three watch heads and some chewed leather straps. "My husband, who's the most calm person that I know, called me up and said, 'I'm not quite sure if I should panic,' " said Courcelle's wife, Michele Parkinson. The 80-pound Doberman, whom her owners describe as "more goofy than scary," had pulled down a basket of wrist wear from a shelf in their bedroom and eaten nearly all the contents. Parkinson knew that Mocha wouldn't be able to pass all that leather on her own. The couple took her to the MSPCA's Angell Animal Medical Center, a 24-hour emergency and specialty hospital, where a veterinarian performed a 3-hour endoscopy to explore the contents of her belly. Mocha was a repeat offender: Just last summer, she got very sick and had to have emergency stomach surgery after a piece of plastic from an orange juice container perforated her intestine. She had 28 staples down her belly and 10 inches of intestine removed, Parkinson said. Fortunately this time, the jewelry remains were still in Mocha's belly and had not made their way into the digestive tract. The X-ray, however, was disturbing. Parkinson and her husband were just expecting to see a couple metal pieces. "It just looked like a Christmas tree and I almost threw up," Parkinson said. The veterinarian removed "about a pound of leather straps and metal pieces and detritus" during the endoscopy, and let nature take its course for the remaining pieces, MSPCA spokesman Rob Halpin said. As of Friday, Mocha was no worse for wear. The hospital sees dozens of cases each week of dogs ingesting foreign objects, and is trained to look for the symptoms of blockages -- typically lethargy, not eating and vomiting, Halpin said. They once saw a golden retriever who had stopped eating and found 43 pacifiers in her belly. (Apparently she was taking them from babies at the park.) And there was the 100-pound bull mastiff who ate his owner's brie that was set out for a party -- along with the cheese knife. The night Mocha stayed in the hospital, a nervous Parkinson stayed awake reading stories about dogs ingesting watches and other objects. She found one article about a Newfoundland whose owner knew something was awry only when he heard an alarm go off from his dog's belly. "We've taken every imaginable thing that could fit down the gullet of a dog out with surgery," Halpin said. "There's some evolutionary traits that some dogs have that lead them to eat first and think later ... and some of them are so food motivated that anything with a scent could be associated with food, and they go for it." Mocha likes to suck on fleece blankets and has been known to eat rubber ear buds or hair elastics, but nothing like a pile of jewelry, Parkinson said. Her breeder wondered if the dog was acting out of anxiety. That day, Parkinson had left Mocha in a different apartment the couple owns that the dog wasn't as used to. The breeder told Parkinson that Dobermans are particularly known to get anxious and do these sort of things. "She had a dog that actually consumed her whole dog bed," Parkinson said. From now on, Parkinson said she will put Mocha in a crate if the dog will be staying somewhere new. Follow-up X-rays Monday on Mocha showed a few pieces of metal left, "but they were moving along" and the vet expected her to pass them naturally. Parkinson said Friday Mocha was "her playful, energetic, curious Doberman self." But now that she thinks of it, the timing of this whole incident is a little suspicious. "My husband was all excited about the new Apple watch, but couldn't justify a reason to purchase it since he owned three watches," she said. "I am convinced that he and Mocha joined forces here to destroy all of his current watches in order to make room for Apple's new watch."

python/llm/dev/benchmark/all-in-one/prompt/summarize/cnn_64.txt

@@ -0,0 +1 @@
+(CNN)Authorities identified and charged a man Monday in connection with the discovery of human remains in a duffel bag in Cambridge, Massachusetts, over the weekend. Carlos Colina, 32, was arraigned on charges of assault and battery causing serious bodily injury and improper disposal of a body, the Middlesex District Attorney's Office said in a statement. "This was a gruesome discovery," said District Attorney Marian Ryan. "Detectives are continuing to analyze evidence and awaiting information from the Office of the Chief Medical Examiner so that we may determine if additional charges are warranted." Police were notified Saturday morning about a suspicious item along a walkway in Cambridge. Officers arrived at the scene, opened a duffel bag and found human remains. After that discovery, police say, a surveillance video led them to an apartment building, where more body parts were discovered in a common area. That location is near the Cambridge Police Department headquarters. The remains at both locations belonged to the same victim, identified Monday as Jonathan Camilien, 26. Camilien and Colina knew each other, according to authorities. The next scheduled hearing in the case is set for April 14. CNN's Andreas Preuss contributed to this report.

python/llm/dev/benchmark/all-in-one/run.py

@@ -70,6 +70,42 @@ def run_model_in_thread(model, in_out, tokenizer, result, warm_up, num_beams, in
                 result[in_out].append([model.first_cost, model.rest_cost_mean, model.encoder_time,
                                         actual_in_len, actual_out_len, load_time, model.peak_memory])
 
+def preprocess_prompt(tokenizer, in_len, task):
+    if task == 'summarize':
+        if in_len == 512:
+            input_str = open(f"prompt/summarize/cnn_239.txt", 'r').read()
+        elif in_len == 1024:
+            input_str = open(f"prompt/summarize/cnn_615.txt", 'r').read()
+        elif in_len == 2048:
+            input_str = open(f"prompt/summarize/cnn_824.txt", 'r').read()
+        elif in_len <= 256:
+            input_str = open(f"prompt/summarize/cnn_64.txt", 'r').read()
+        else:
+            input_str = open(f"prompt/summarize/cnn_5618.txt", 'r').read()
+        question = "Can you please summarize this article?"
+        prompt_format = "[INST] Article:```{}``` \n\n Question: {} \n\n [/INST]"
+        special_tokens_len = len(tokenizer.encode(prompt_format.format("", question), add_special_tokens=False))
+        max_article_len = in_len - special_tokens_len
+        article_ids = tokenizer.encode(input_str, add_special_tokens=False)
+        if len(article_ids) > max_article_len:
+            article_ids = article_ids[:max_article_len]
+        truncated_article_text = tokenizer.decode(article_ids, skip_special_tokens=True)
+        final_prompt = prompt_format.format(truncated_article_text, question)
+        input_ids = tokenizer.encode(final_prompt, return_tensors="pt", truncation=True, max_length=in_len)
+    elif task == 'QA':
+        if in_len == 512:
+            input_str = open(f"prompt/QA/orca_776.txt", 'r').read()
+        elif in_len == 1024:
+            input_str = open(f"prompt/QA/orca_99.txt", 'r').read()
+        elif in_len == 2048:
+            input_str = open(f"prompt/QA/orca_401.txt", 'r').read()
+        elif in_len == 4096:
+            input_str = open(f"prompt/QA/orca_497.txt", 'r').read()
+        else:
+            raise ValueError("No corresponding prompt available now, will be added later.")          
+        input_ids = tokenizer.encode(input_str, return_tensors="pt")    
+    return input_ids
+
 def run_model(repo_id, test_api, in_out_pairs, local_model_hub=None, warm_up=1, num_trials=3, num_beams=1, low_bit='sym_int4', cpu_embedding=False, batch_size=1, streaming=False, use_fp16_torch_dtype=False, n_gpu=2):
     # TODO: make a parameter
     result= {}
@@ -463,28 +499,22 @@ def run_transformer_int4_gpu(repo_id,
             in_out_len = in_out.split("-")
             in_len = int(in_out_len[0])
             out_len = int(in_out_len[1])
-            # As different tokenizer has different encodings,
-            # in_len.txt maybe shorter than we need,
-            # use much longer context to make sure input length
-            test_length = min(in_len*2, 8192)
-            while test_length not in [32, 256, 1024, 2048, 8192] and test_length < 8192:
-                test_length = test_length * 2
-            # For the sequence length not in [32, 256, 1024, 2048, 8192], it will be truncated from 8192.txt.
-            test_length = min(test_length, 8192)
-            input_str = open(f"prompt/{test_length}.txt", 'r').read()
-            if lookahead:
-                question = "Can you please summarize this article?"
-                question_tokens = tokenizer.encode(question, return_tensors="pt")
-                max_article_len = in_len - question_tokens.size(1)
-                article_ids = tokenizer.encode(input_str, return_tensors="pt")
-                if article_ids.size(1) > max_article_len:
-                    article_ids = article_ids[:, :max_article_len]
-                input_ids = torch.cat((article_ids, question_tokens), dim=1)
-            else:
+            if not lookahead or conf['task'] == 'continuation':
+                # As different tokenizer has different encodings,
+                # in_len.txt maybe shorter than we need,
+                # use much longer context to make sure input length
+                test_length = min(in_len*2, 8192)
+                while test_length not in [32, 256, 1024, 2048, 8192] and test_length < 8192:
+                    test_length = test_length * 2
+                # For the sequence length not in [32, 256, 1024, 2048, 8192], it will be truncated from 8192.txt.
+                test_length = min(test_length, 8192)
+                input_str = open(f"prompt/continuation/{test_length}.txt", 'r').read()
                 # As different tokenizer has different encodings,
                 # slice the input_ids to ensure the prompt length is required length.
                 input_ids = tokenizer.encode(input_str, return_tensors="pt")
                 input_ids = input_ids[:, :in_len]
+            elif conf['task'] == 'summarize' or conf['task'] == 'QA':
+                input_ids = preprocess_prompt(tokenizer, in_len, conf['task'])
             true_str = tokenizer.batch_decode(input_ids)[0]
             input_list = [true_str] * batch_size
             input_ids = tokenizer(input_list, return_tensors="pt").input_ids.to('xpu')
@@ -1827,6 +1857,7 @@ def run_pipeline_parallel_gpu(repo_id,
 
 if __name__ == '__main__':
     from omegaconf import OmegaConf
+    global conf
     conf = OmegaConf.load(f'{current_dir}/config.yaml')
     today = date.today()
     if 'exclude' in conf: