Source: BIRD Mini-Dev (SQLite). Methodology: docs/03_eval_methodology.md.
| Configuration | Model | n | EA | Simple | Moderate | Challenging | Validity | Recall@k | Empty % | P50 latency | P95 latency |
|---|---|---|---|---|---|---|---|---|---|---|---|
| C_dense_cards | codestral-latest | 200 | 55.0% | 68.7% | 50.5% | 41.2% | 100.0% | 100.0% | 5.0% | 1460 ms | 3941 ms |
| C_dense_cards | codestral-latest | 200 | 56.0% | 71.6% | 50.5% | 41.2% | 100.0% | 100.0% | 2.5% | 1361 ms | 4051 ms |
| C_dense_cards | codestral-latest | 200 | 55.5% | 70.1% | 50.5% | 41.2% | 100.0% | 100.0% | 2.5% | 21 ms | 561 ms |
| F_self_consistency | codestral-latest | 200 | 0.0% | 0.0% | 0.0% | 0.0% | 100.0% | 0.0% | 0.0% | 29820 ms | 30107 ms |
| F_self_consistency | codestral-latest | 200 | 60.0% | 71.6% | 56.6% | 47.1% | 100.0% | 100.0% | 3.0% | 6861 ms | 15615 ms |
| F_self_consistency | codestral-latest | 200 | 60.0% | 71.6% | 56.6% | 47.1% | 100.0% | 100.0% | 3.0% | 71 ms | 2154 ms |
Model: codestral-latest · n=200 · EA=55.0% · Validity=100.0% · Recall@k=100.0%
| qid | db | diff | match | recall | err | lat ms | tokens | question |
|---|---|---|---|---|---|---|---|---|
| 5 | california_schools | simple | ✓ | ✓ | 2216 | 6516 | How many schools with an average score in Math greater than 400 in the SAT test are exclusively virtual? | |
| 25 | california_schools | moderate | ✗ | ✓ | 1304 | 6598 | Name schools in Riverside which the average of average math score for SAT is grater than 400, what is the funding type o | |
| 32 | california_schools | moderate | ✗ | ✓ | execution_failed | 1428 | 6363 | What is the eligible free or reduced price meal rate for the top 5 schools in grades 1-12 with the highest free or reduc |
| 36 | california_schools | challenging | ✓ | ✓ | 1766 | 6766 | Under whose administration is the school with the highest number of students scoring 1500 or more on the SAT? Indicate t | |
| 37 | california_schools | moderate | ✗ | ✓ | 2137 | 6651 | What is the complete address of the school with the lowest excellence rate? Indicate the Street, City, Zip and State. | |
| 39 | california_schools | simple | ✓ | ✓ | 1539 | 6687 | What is the average number of test takers from Fresno schools that opened between 1/1/1980 and 12/31/1980? | |
| 48 | california_schools | moderate | ✓ | ✓ | 2187 | 6623 | What is the ratio of merged Unified School District schools in Orange County to merged Elementary School District school | |
| 50 | california_schools | simple | ✗ | ✓ | 2092 | 6606 | What is the postal street address for the school with the 7th highest Math average? Indicate the school's name. | |
| 77 | california_schools | moderate | ✗ | ✓ | empty_result | 1379 | 6666 | Which schools served a grade span of Kindergarten to 9th grade in the county of Los Angeles and what is its Percent (%) |
| 92 | financial | simple | ✓ | ✓ | 2490 | 4670 | List out the no. of districts that have female average salary is more than 6000 but less than 10000? | |
| 98 | financial | moderate | ✗ | ✓ | empty_result | 2291 | 4719 | Among the accounts who have approved loan date in 1997, list out the accounts that have the lowest approved amount and c |
| 99 | financial | moderate | ✗ | ✓ | empty_result | 3497 | 4687 | Among the accounts who have loan validity more than 12 months, list out the accounts that have the highest approved amou |
| 112 | financial | simple | ✓ | ✓ | 1529 | 4677 | For the female client who was born in 1976/1/29, which district did she opened her account? | |
| 115 | financial | challenging | ✗ | ✓ | execution_failed | 1620 | 4442 | For the branch which located in the south Bohemia with biggest number of inhabitants, what is the percentage of the male |
| 118 | financial | moderate | ✓ | ✓ | 1294 | 4714 | For loan amount less than USD100,000, what is the percentage of accounts that is still running with no issue. | |
| 120 | financial | moderate | ✓ | ✓ | 12128 | 5027 | From Year 1995 to 2000, who are the accounts holders from 'east Bohemia'. State the account ID the frequency of statemen | |
| 125 | financial | challenging | ✗ | ✓ | 1765 | 5064 | For loans contracts which are still running where client are in debt, list the district of the and the state the percent | |
| 138 | financial | moderate | ✓ | ✓ | 1950 | 4656 | In the branch where the second-highest number of crimes were committed in 1995 occurred, how many male clients are there | |
| 159 | financial | simple | ✗ | ✓ | 1951 | 4815 | List all the withdrawals in cash transactions that the client with the id 3356 makes. | |
| 168 | financial | moderate | ✓ | ✓ | 2119 | 4699 | What percentage of clients who opened their accounts in the district with an average salary of over 10000 are women? | |
| 169 | financial | challenging | ✗ | ✓ | 1878 | 4887 | What was the growth rate of the total amount of loans across all accounts for a male client between 1996 and 1997? | |
| 173 | financial | challenging | ✗ | ✓ | 1412 | 4764 | How often does account number 3 request an account statement to be released? What was the aim of debiting 3539 in total? | |
| 189 | financial | moderate | ✗ | ✓ | empty_result | 2138 | 4683 | Name the account numbers of female clients who are oldest and have lowest average salary? |
| 192 | financial | moderate | ✓ | ✓ | 1373 | 4709 | What is the average amount of loan which are still on running contract with statement issuance after each transaction? | |
| 194 | financial | moderate | ✓ | ✓ | 1457 | 4645 | Provide the IDs and age of the client with high level credit card, which is eligible for loans. | |
| 207 | toxicology | challenging | ✗ | ✓ | 1293 | 2803 | What elements are in a double type bond? | |
| 208 | toxicology | moderate | ✓ | ✓ | 1331 | 2781 | Which type of label is the most numerous in atoms with hydrogen? | |
| 219 | toxicology | challenging | ✗ | ✓ | execution_failed | 1297 | 2589 | What is the percentage of carcinogenic molecules in triple type bonds? |
| 227 | toxicology | simple | ✓ | ✓ | 1162 | 2818 | What is the percentage of molecules that are carcinogenic? Please provide your answer as a percentage with three decimal | |
| 230 | toxicology | challenging | ✓ | ✓ | 1168 | 2793 | What are the elements of the toxicology and label of molecule TR060? | |
| 232 | toxicology | moderate | ✗ | ✓ | execution_failed | 1202 | 2564 | Please list top three molecules that have single bonds between two atoms and are not carcinogenic in alphabetical order. |
| 236 | toxicology | moderate | ✓ | ✓ | 1218 | 2844 | What are the bond type and the atoms of the bond ID of TR001_6_9? | |
| 239 | toxicology | simple | ✓ | ✓ | 1156 | 2768 | How many connections does the atom 19 have? | |
| 253 | toxicology | challenging | ✓ | ✓ | 1421 | 2791 | List the elements of all the triple bonds. | |
| 260 | toxicology | moderate | ✓ | ✓ | 1487 | 2850 | Calculate the total atoms with triple-bond molecules containing the element phosphorus or bromine. | |
| 268 | toxicology | challenging | ✓ | ✓ | 1345 | 2845 | What are the elements for bond id TR001_10_11? | |
| 273 | toxicology | moderate | ✗ | ✓ | 1428 | 2860 | What is the percentage of element chlorine in carcinogenic molecules? | |
| 282 | toxicology | challenging | ✓ | ✓ | 1567 | 2923 | What is the ratio of Hydrogen elements in molecule ID TR006? List the ratio with its label. | |
| 327 | toxicology | moderate | ✓ | ✓ | 1940 | 2868 | Which non-carcinogenic molecules consisted more than 5 atoms? | |
| 347 | card_games | moderate | ✗ | ✓ | 2539 | 9057 | Find all cards illustrated by Stephen Daniel and describe the text of the ruling of these cards. State if these cards ha | |
| 349 | card_games | moderate | ✗ | ✓ | 2171 | 8718 | Name the card and artist with the most ruling information. Also state if the card is a promotional printing. | |
| 352 | card_games | moderate | ✗ | ✓ | 5859 | 8697 | Calculate the percentage of the cards availabe in Chinese Simplified. | |
| 356 | card_games | simple | ✓ | ✓ | 1204 | 8539 | How many cards have infinite power? | |
| 358 | card_games | simple | ✓ | ✓ | 1325 | 8603 | What is the border color of card "Ancestor's Chosen"? | |
| 366 | card_games | simple | ✗ | ✓ | 2775 | 8665 | What is the rule of playing card "Benalish Knight"? | |
| 377 | card_games | simple | ✗ | ✓ | 1368 | 8612 | How many cards with original type of "Summon - Angel" have subtype other than "Angel"? | |
| 391 | card_games | moderate | ✓ | ✓ | 3264 | 8732 | Among the Artifact cards, which are black color and comes with foreign languague translation? | |
| 407 | card_games | moderate | ✗ | ✓ | 2770 | 8726 | Lists all types of cards in German. | |
| 408 | card_games | moderate | ✗ | ✓ | 1492 | 8625 | How many unknown power cards contain info about the triggered ability | |
| 412 | card_games | moderate | ✗ | ✓ | empty_result | 2196 | 8705 | What is the foreign name of the card in French of type Creature, normal layout and black border color, by artist Matthew |
| 414 | card_games | simple | ✓ | ✓ | 1395 | 8693 | What language is the set of 180 cards that belongs to the Ravnica block translated into? | |
| 427 | card_games | moderate | ✓ | ✓ | 1312 | 8720 | What languages are available in the set known as Archenemy on the magic card market and having the code ARC? | |
| 459 | card_games | moderate | ✗ | ✓ | 1346 | 8705 | Which card costs more converted mana, "Serra Angel" or "Shrine Keeper"? | |
| 466 | card_games | moderate | ✓ | ✓ | 1491 | 8709 | Among the cards in the set "Hauptset Zehnte Edition", how many of them are designed by Adam Rex? | |
| 472 | card_games | moderate | ✓ | ✓ | 1438 | 8692 | Among the sets in the block "Ice Age", how many of them have an Italian translation? | |
| 484 | card_games | moderate | ✗ | ✓ | 16527 | 8814 | Please list the Italian names of the cards in the set Coldsnap with the highest converted mana cost. | |
| 486 | card_games | moderate | ✓ | ✓ | 7929 | 8817 | What is the percentage of the cards with a converted mana cost of 7 in the set Coldsnap? | |
| 518 | card_games | moderate | ✗ | ✓ | 61711 | 8781 | Which of the play format has the highest number of banned status? Indicate the play format and the names of all the card | |
| 531 | codebase_community | simple | ✗ | ✓ | 1842 | 5788 | Which user has a higher reputation, Harlan or Jarrod Dixon? | |
| 557 | codebase_community | moderate | ✗ | ✓ | 2050 | 6511 | Among the posts with a score of over 5, what is the percentage of them being owned by an elder user? | |
| 563 | codebase_community | moderate | ✗ | ✓ | empty_result | 1524 | 6549 | User No.3025 gave a comment at 20:29:39 on 2014/4/23 to a post, how many favorite counts did that post get? |
| 571 | codebase_community | moderate | ✗ | ✓ | 2547 | 6421 | For the user No.24, how many times is the number of his/her posts compared to his/her votes? | |
| 584 | codebase_community | moderate | ✗ | ✓ | 2321 | 6567 | Write all the comments left by users who edited the post titled 'Why square the difference instead of taking the absolut | |
| 595 | codebase_community | moderate | ✗ | ✓ | 3147 | 6470 | Which user have only one post history per post and having at least 1000 views? | |
| 634 | codebase_community | challenging | ✗ | ✓ | empty_result | 11847 | 6380 | Among posts by Harvey Motulsky and Noah Snyder, which one has higher popularity? |
| 669 | codebase_community | simple | ✓ | ✓ | 1316 | 5790 | When did 'chl' cast its first vote in a post? | |
| 671 | codebase_community | simple | ✓ | ✓ | 1246 | 5798 | What is the display name of the user who acquired the first Autobiographer badge? | |
| 672 | codebase_community | moderate | ✗ | ✓ | 1656 | 6379 | Among the users located in United Kingdom, how many users whose post have a total favorite amount of 4 or more? | |
| 694 | codebase_community | moderate | ✗ | ✓ | 2133 | 6655 | Provide the text of the latest 10 comments to the post with the title 'Analysing wind data with R' and the display name | |
| 707 | codebase_community | moderate | ✗ | ✓ | execution_failed | 1883 | 6171 | Among the posts with views ranging from 100 to 150, what is the comment with the highest score? |
| 716 | codebase_community | moderate | ✗ | ✓ | 2311 | 6502 | Among the comments with scores between 5 to 10, what is the percentage of the users with 0 up votes? | |
| 723 | superhero | moderate | ✓ | ✓ | 1645 | 3655 | Among the superheroes with blue eyes, how many of them have the super power of "Agility"? | |
| 730 | superhero | challenging | ✓ | ✓ | 3227 | 3684 | List the superheroes from Marvel Comics who have the super power of 'Super Strength'. | |
| 736 | superhero | moderate | ✓ | ✓ | 1332 | 3548 | Who is the dumbest superhero? | |
| 737 | superhero | simple | ✓ | ✓ | 992 | 3466 | What is Copycat's race? | |
| 738 | superhero | simple | ✓ | ✓ | 1323 | 3628 | Which superheroes have a durability attribute value of less than 50? | |
| 743 | superhero | challenging | ✗ | ✓ | 2197 | 3729 | What is the percentage of superheroes who act in their own self-interest or make decisions based on their own moral code | |
| 747 | superhero | simple | ✓ | ✓ | 987 | 3466 | What is the total number of superheroes without full name? | |
| 750 | superhero | simple | ✓ | ✓ | 1266 | 3531 | What is the average weight of all female superheroes? | |
| 751 | superhero | moderate | ✓ | ✓ | 1613 | 3618 | List down at least five superpowers of male superheroes. | |
| 753 | superhero | moderate | ✓ | ✓ | 1269 | 3653 | Among the superheroes with height from 170 to 190, list the names of the superheroes with no eye color. | |
| 765 | superhero | simple | ✓ | ✓ | 1272 | 3502 | How many heroes have stealth power? | |
| 773 | superhero | challenging | ✓ | ✓ | 1262 | 3612 | Which superhero has the same eyes, hair and skin colour? Indicate the publisher of the superhero. | |
| 775 | superhero | challenging | ✓ | ✓ | 1887 | 3647 | What is the percentage of blue female superheroes among all female superheroes? | |
| 781 | superhero | simple | ✓ | ✓ | 1741 | 3566 | Provide the heights of the heroes whose eye colours are amber. | |
| 785 | superhero | simple | ✓ | ✓ | 1174 | 3537 | Describe the names of neutral alignment superheroes. | |
| 791 | superhero | simple | ✗ | ✓ | 910 | 3491 | Calculate the average height for all superhero. | |
| 794 | superhero | moderate | ✓ | ✓ | 1454 | 3546 | Which hero was the fastest? | |
| 798 | superhero | moderate | ✓ | ✓ | 1200 | 3592 | What is the publisher for Hawkman, Karate Kid and Speedy? | |
| 800 | superhero | moderate | ✓ | ✓ | 1900 | 3624 | Calculate the percentage of superheroes with blue eyes. | |
| 806 | superhero | simple | ✓ | ✓ | 1235 | 3472 | Provide the eye colour of the superhero who has Karen Beecher-Duncan as their full name. | |
| 819 | superhero | challenging | ✓ | ✓ | 1629 | 3824 | In superheroes with missing weight data, calculate the difference between the number of superheroes with blue eyes and n | |
| 825 | superhero | moderate | ✓ | ✓ | 1459 | 3567 | Identify the gender of the superhero who has the ability of Phoenix Force. | |
| 847 | formula_1 | simple | ✗ | ✓ | 1446 | 6665 | What is the surname of the driver with the best lap time in race number 19 in the second qualifying period? | |
| 859 | formula_1 | simple | ✓ | ✓ | 1305 | 6656 | What's Bruno Senna's Q1 result in the qualifying race No. 354? | |
| 861 | formula_1 | simple | ✗ | ✓ | empty_result | 1352 | 6657 | What is his number of the driver who finished 0:01:54 in the Q3 of qualifying race No.903? |
| 862 | formula_1 | simple | ✓ | ✓ | 1360 | 6647 | For the Bahrain Grand Prix in 2007, how many drivers not finished the game? | |
| 865 | formula_1 | moderate | ✗ | ✓ | 1866 | 6705 | For all the drivers who finished the game in race No. 592, who is the oldest? | |
| 866 | formula_1 | moderate | ✗ | ✓ | 1089 | 6754 | Who was the player that got the lap time of 0:01:27 in the race No. 161? Show his introduction website. | |
| 875 | formula_1 | simple | ✓ | ✓ | 1219 | 6598 | Show me the season page of year when the race No. 901 took place. | |
| 877 | formula_1 | moderate | ✓ | ✓ | 1283 | 6640 | For all the drivers who finished the game in race No. 872, who is the youngest? | |
| 879 | formula_1 | moderate | ✓ | ✓ | 1259 | 6585 | For the driver who set the fastest lap speed, what is his nationality? | |
| 881 | formula_1 | moderate | ✗ | ✓ | 2073 | 6763 | For the drivers who took part in the race in 1983/7/16, what's their race completion rate? | |
| 894 | formula_1 | moderate | ✗ | ✓ | 1511 | 6674 | What is the best lap time recorded? List the driver and race with such recorded lap time. | |
| 896 | formula_1 | challenging | ✗ | ✓ | 5037 | 6777 | Calculate the percentage whereby Hamilton was not at the 1st track of the the f1 circuit since 2010. | |
| 897 | formula_1 | moderate | ✓ | ✓ | 1376 | 6705 | Name the driver with the most winning. Mention his nationality and what is his maximum point scores. | |
| 898 | formula_1 | simple | ✗ | ✓ | execution_failed | 1257 | 6382 | How old is the youngest Japanese driver? What is his name? |
| 902 | formula_1 | simple | ✗ | ✓ | 1238 | 6713 | Which race was Alex Yoong in when he was in track number less than 20? | |
| 904 | formula_1 | moderate | ✗ | ✓ | 2271 | 6526 | State the race and year of race in which Michael Schumacher had his fastest lap. | |
| 909 | formula_1 | moderate | ✓ | ✓ | 1274 | 6706 | Among all European Grand Prix races, what is the percentage of the races were hosted in Germany? | |
| 912 | formula_1 | simple | ✓ | ✓ | 1007 | 6304 | What's the reference name of Marina Bay Street Circuit? | |
| 915 | formula_1 | simple | ✗ | ✓ | 1342 | 6608 | Which country is the oldest driver from? | |
| 930 | formula_1 | simple | ✗ | ✓ | execution_failed | 1620 | 6424 | In which Formula_1 race did Lewis Hamilton rank the highest? |
| 945 | formula_1 | simple | ✓ | ✓ | 1085 | 6273 | How many circuits are there in Adelaide, Australia? | |
| 950 | formula_1 | simple | ✗ | ✓ | 1288 | 6628 | Please list the constructor names with 0 points at race 291. | |
| 959 | formula_1 | simple | ✗ | ✓ | 1454 | 6679 | What is the fastest lap number of the champion in 2009? | |
| 971 | formula_1 | simple | ✓ | ✓ | 10164 | 6549 | Please state the reference name of the oldest German driver. | |
| 981 | formula_1 | moderate | ✓ | ✓ | 2510 | 6832 | On what year did the youngest driver had his first qualifying race? Also state the name, date and time of the race. | |
| 988 | formula_1 | challenging | ✗ | ✓ | 2720 | 6782 | List down top 3 German drivers who has the shortest average pit stop duration and were born between 1980-1985. | |
| 989 | formula_1 | moderate | ✗ | ✓ | 2135 | 6702 | Who is the champion of the Canadian Grand Prix in 2008? Indicate his finish time. | |
| 990 | formula_1 | challenging | ✓ | ✓ | 1492 | 6727 | What is the constructor reference name of the champion in the 2009 Singapore Grand Prix? Please give its website. | |
| 1028 | european_football_2 | challenging | ✗ | ✓ | 1695 | 12013 | In Scotland Premier League, which away team won the most during the 2010 season? | |
| 1029 | european_football_2 | moderate | ✗ | ✓ | 1297 | 11866 | What are the speed in which attacks are put together of the top 4 teams with the highest build Up Play Speed? | |
| 1030 | european_football_2 | moderate | ✓ | ✓ | 1467 | 11847 | Give the name of the league had the most matches end as draw in the 2016 season? | |
| 1035 | european_football_2 | simple | ✓ | ✓ | 1043 | 11735 | Give the team_fifa_api_id of teams with more than 50 but less than 60 build-up play speed. | |
| 1036 | european_football_2 | challenging | ✗ | ✓ | 2323 | 11911 | List the long name of teams with above-average build-up play passing in 2012. | |
| 1037 | european_football_2 | challenging | ✗ | ✓ | 2746 | 11949 | Calculate the percentage of players who prefer left foot, who were born between 1987 and 1992. | |
| 1039 | european_football_2 | simple | ✓ | ✓ | 2056 | 11784 | Find the average number of long-shot done by Ahmed Samir Farag. | |
| 1042 | european_football_2 | challenging | ✓ | ✓ | 2218 | 12018 | List the name of leagues in which the average goals by the home team is higher than the away team in the 2009/2010 seaso | |
| 1057 | european_football_2 | moderate | ✓ | ✓ | 1514 | 11864 | Calculate the average home team goal in the 2010/2011 season in the country of Poland. | |
| 1078 | european_football_2 | simple | ✓ | ✓ | 1323 | 11628 | Which player is older, Aaron Lennon or Abdelaziz Barrada? | |
| 1088 | european_football_2 | moderate | ✓ | ✓ | 1619 | 11886 | Please list the names of the players whose volley score and dribbling score are over 70. | |
| 1094 | european_football_2 | challenging | ✗ | ✓ | execution_failed | 1775 | 11607 | How much higher in percentage is Ariel Borysiuk's overall rating than that of Paulin Puel? |
| 1103 | european_football_2 | moderate | ✓ | ✓ | 1526 | 11836 | What was the overall rating for Aaron Mooy on 2016/2/4? | |
| 1110 | european_football_2 | moderate | ✓ | ✓ | 1402 | 11788 | Tell the build Up play passing class for "FC Lorient" on 2010/2/22. | |
| 1116 | european_football_2 | simple | ✓ | ✓ | 1182 | 11689 | List down most tallest players' name. | |
| 1122 | european_football_2 | simple | ✓ | ✓ | 1623 | 11663 | State the name of the most strongest player. | |
| 1130 | european_football_2 | moderate | ✓ | ✓ | 1170 | 11795 | What are the short name of team who played safe while creating chance of passing? | |
| 1133 | european_football_2 | simple | ✓ | ✓ | 1079 | 11659 | How many football players born after the 1990s have the first name "Aaron"? | |
| 1141 | european_football_2 | moderate | ✓ | ✓ | 1837 | 11710 | Does the KSV Cercle Brugge team have a slow, balanced or fast speed class? | |
| 1144 | european_football_2 | simple | ✗ | ✓ | 1354 | 11810 | Please state the finishing rate and curve score of the player who has the heaviest weight. | |
| 1146 | european_football_2 | moderate | ✓ | ✓ | 3900 | 11772 | Please provide the full name of the away team that scored the most goals. | |
| 1147 | european_football_2 | simple | ✓ | ✓ | 1517 | 11638 | Please name one player whose overall strength is the greatest. | |
| 1152 | thrombosis_prediction | moderate | ✓ | ✓ | 1346 | 5074 | What is the ratio of outpatient to inpatient followed up treatment among all the 'SLE' diagnosed patient? | |
| 1156 | thrombosis_prediction | moderate | ✗ | ✓ | execution_failed | 1175 | 4668 | State the ID and age of patient with positive degree of coagulation. |
| 1157 | thrombosis_prediction | simple | ✓ | ✓ | 1158 | 4958 | For patients with severe degree of thrombosis, list their ID, sex and disease the patient is diagnosed with. | |
| 1168 | thrombosis_prediction | challenging | ✗ | ✓ | execution_failed | 1291 | 4725 | The oldest SJS patient's medical laboratory work was completed on what date, and what age was the patient when they init |
| 1185 | thrombosis_prediction | challenging | ✗ | ✓ | 2094 | 5334 | For the patient who was born on 1959/2/18, what is the decrease rate for his/her total cholesterol from November to Dece | |
| 1198 | thrombosis_prediction | simple | ✓ | ✓ | 962 | 4846 | How many female patients were given an APS diagnosis? | |
| 1205 | thrombosis_prediction | moderate | ✗ | ✓ | 1889 | 5003 | Was the patient with the number 57266's uric acid within a normal range? | |
| 1208 | thrombosis_prediction | moderate | ✓ | ✓ | 1183 | 5037 | Provide IDs for male patients with ALT glutamic pylvic transaminase (GPT) that have history of ALT glutamic pylvic trans | |
| 1220 | thrombosis_prediction | simple | ✓ | ✓ | 1185 | 5069 | Provide all ID, sex and birthday of patients whose urea nitrogen (UN) just within the borderline of passing? | |
| 1227 | thrombosis_prediction | moderate | ✗ | ✓ | execution_failed | 1202 | 4698 | What is the average age of the male patient with high cholesterol? |
| 1232 | thrombosis_prediction | challenging | ✓ | ✓ | 1460 | 5188 | Provide ID, sex and age of patient who has blood glucose (GLU) not within normal range but with total cholesterol(T-CHO) | |
| 1235 | thrombosis_prediction | moderate | ✗ | ✓ | execution_failed | 1323 | 4696 | What are the patient's diagnosis for those who has lower red blood blood cell? State their ID and age. |
| 1247 | thrombosis_prediction | challenging | ✗ | ✓ | 1900 | 5056 | Among the male patients who have a normal level of white blood cells, how many of them have an abnormal fibrinogen level | |
| 1251 | thrombosis_prediction | simple | ✗ | ✓ | 1068 | 4877 | How many patients with an Ig G higher than normal? | |
| 1252 | thrombosis_prediction | moderate | ✗ | ✓ | 1205 | 4959 | Among the patients with a normal Ig G level, how many of them have symptoms? | |
| 1254 | thrombosis_prediction | moderate | ✗ | ✓ | execution_failed | 1317 | 4725 | How many patients with a normal Ig A level came to the hospital after 1990/1/1? |
| 1255 | thrombosis_prediction | moderate | ✗ | ✓ | 1361 | 4953 | For the patients with an abnormal Ig M level, what is the most common disease they are diagnosed with? | |
| 1257 | thrombosis_prediction | challenging | ✗ | ✓ | 1236 | 4988 | Among the patients whose creatinine level is abnormal, how many of them aren't 70 yet? | |
| 1275 | thrombosis_prediction | moderate | ✗ | ✓ | execution_failed | 1289 | 4726 | Among the patients who has a normal level of anti-centromere and a normal level of anti-SSB, how many of them are male? |
| 1281 | thrombosis_prediction | moderate | ✗ | ✓ | 1804 | 4980 | Among the patients who have an abnormal level of glutamic oxaloacetic transaminase, when was the youngest of them born? | |
| 1302 | thrombosis_prediction | challenging | ✗ | ✓ | execution_failed | 1099 | 4695 | For the patients with a normal range of creatinine phosphokinase, how many of them have a positive measure of degree of |
| 1312 | student_club | simple | ✓ | ✓ | 1109 | 4833 | What's Angela Sanders's major? | |
| 1340 | student_club | moderate | ✓ | ✓ | 2082 | 5042 | Calculate the difference of the total amount spent in all events by the Student_Club in year 2019 and 2020. | |
| 1344 | student_club | simple | ✓ | ✓ | 1223 | 4891 | What was the notes of the fundraising on 2019/9/14? | |
| 1352 | student_club | moderate | ✓ | ✓ | 1621 | 4863 | For all the club members from "Business" major, how many of them wear medium size t-shirt? | |
| 1356 | student_club | simple | ✓ | ✓ | 1126 | 4797 | Which department was the President of the club in? | |
| 1376 | student_club | moderate | ✓ | ✓ | 9878 | 4883 | Among all the closed events, which event has the highest spend-to-budget ratio? | |
| 1378 | student_club | simple | ✓ | ✓ | 1006 | 4293 | What is the highest amount of budget spend for an event? | |
| 1380 | student_club | simple | ✓ | ✓ | 1119 | 4568 | What is the total amount of money spent for food? | |
| 1387 | student_club | moderate | ✓ | ✓ | 1432 | 4954 | Which student has been entrusted to manage the budget for the Yearly Kickoff? | |
| 1390 | student_club | moderate | ✓ | ✓ | 2344 | 4493 | Based on the total cost for all event, what is the percentage of cost for Yearly Kickoff event? | |
| 1399 | student_club | moderate | ✗ | ✓ | 1476 | 4902 | Did Maya Mclean attend the 'Women's Soccer' event? | |
| 1403 | student_club | moderate | ✓ | ✓ | 1231 | 4891 | Indicate the name of the closed event whose cost has exceeded the budget the most. | |
| 1404 | student_club | moderate | ✗ | ✓ | execution_failed | 1423 | 4640 | Identify the type of expenses and their total value approved for 'October Meeting' event. |
| 1409 | student_club | simple | ✓ | ✓ | 1149 | 4853 | Mention the total expense used on 8/20/2019. | |
| 1410 | student_club | simple | ✓ | ✓ | 1398 | 4928 | List out the full name and total cost that member id "rec4BLdZHS2Blfp4v" incurred? | |
| 1411 | student_club | simple | ✓ | ✓ | 1182 | 4887 | State what kind of expenses that Sacha Harrison incurred? | |
| 1422 | student_club | simple | ✗ | ✓ | 1052 | 4814 | State the category of events were held at MU 215. | |
| 1464 | student_club | challenging | ✓ | ✓ | 4725 | 4935 | Write the full names of students who received funds on the date of 9/9/2019 and include the amount received. | |
| 1472 | debit_card_specializing | moderate | ✗ | ✓ | 1574 | 3259 | In 2012, who had the least consumption in LAM? | |
| 1473 | debit_card_specializing | moderate | ✓ | ✓ | 1641 | 3311 | What was the average monthly consumption of customers in SME for the year 2013? | |
| 1476 | debit_card_specializing | challenging | ✓ | ✓ | 2199 | 3475 | What was the difference in gas consumption between CZK-paying customers and EUR-paying customers in 2012? | |
| 1479 | debit_card_specializing | moderate | ✓ | ✓ | 1979 | 3192 | Which year recorded the most consumption of gas paid in CZK? | |
| 1480 | debit_card_specializing | moderate | ✗ | ✓ | 1964 | 3285 | What was the gas consumption peak month for SME customers in 2013? | |
| 1484 | debit_card_specializing | simple | ✓ | ✓ | 1316 | 3267 | How many more "discount" gas stations does the Czech Republic have compared to Slovakia? | |
| 1486 | debit_card_specializing | simple | ✗ | ✓ | 1343 | 3254 | Is it true that more SMEs pay in Czech koruna than in euros? If so, how many more? | |
| 1493 | debit_card_specializing | simple | ✓ | ✓ | 1533 | 3327 | In February 2012, what percentage of customers consumed more than 528.3? | |
| 1500 | debit_card_specializing | simple | ✗ | ✓ | empty_result | 1706 | 3215 | Please list the product description of the products consumed in September, 2013. |
| 1501 | debit_card_specializing | moderate | ✓ | ✓ | 1537 | 3286 | Please list the countries of the gas stations with transactions taken place in June, 2013. | |
| 1506 | debit_card_specializing | moderate | ✓ | ✓ | 1351 | 3238 | Please list the product descriptions of the transactions taken place in the gas stations in the Czech Republic. | |
| 1515 | debit_card_specializing | simple | ✓ | ✓ | 1318 | 3225 | What segment did the customer have at 2012/8/23 21:20:00? | |
| 1521 | debit_card_specializing | moderate | ✓ | ✓ | 1361 | 3396 | For all the transactions happened during 8:00-9:00 in 2012/8/26, how many happened in CZE? | |
| 1525 | debit_card_specializing | simple | ✗ | ✓ | 1546 | 3286 | What is the percentage of the customers who used EUR in 2012/8/25? | |
| 1526 | debit_card_specializing | challenging | ✗ | ✓ | empty_result | 2671 | 3476 | For the customer who paid 634.8 in 2012/8/25, what was the consumption decrease rate from Year 2012 to 2013? |
| 1528 | debit_card_specializing | simple | ✗ | ✓ | 2037 | 3188 | What is the percentage of "premium" against the overall segment in Country = "SVK"? | |
| 1529 | debit_card_specializing | moderate | ✗ | ✓ | 1175 | 3166 | What is the amount spent by customer "38508" at the gas stations? How much had the customer spent in January 2012? | |
| 1531 | debit_card_specializing | moderate | ✗ | ✓ | 1505 | 3305 | Who is the top spending customer and how much is the average price per single item purchased by this customer? What curr |
Model: codestral-latest · n=200 · EA=56.0% · Validity=100.0% · Recall@k=100.0%
| qid | db | diff | match | recall | err | lat ms | tokens | question |
|---|---|---|---|---|---|---|---|---|
| 5 | california_schools | simple | ✓ | ✓ | 245 | 6331 | How many schools with an average score in Math greater than 400 in the SAT test are exclusively virtual? | |
| 25 | california_schools | moderate | ✗ | ✓ | 22 | 6403 | Name schools in Riverside which the average of average math score for SAT is grater than 400, what is the funding type o | |
| 32 | california_schools | moderate | ✗ | ✓ | execution_failed | 19 | 6167 | What is the eligible free or reduced price meal rate for the top 5 schools in grades 1-12 with the highest free or reduc |
| 36 | california_schools | challenging | ✓ | ✓ | 9 | 6572 | Under whose administration is the school with the highest number of students scoring 1500 or more on the SAT? Indicate t | |
| 37 | california_schools | moderate | ✗ | ✓ | 12 | 6462 | What is the complete address of the school with the lowest excellence rate? Indicate the Street, City, Zip and State. | |
| 39 | california_schools | simple | ✓ | ✓ | 26 | 6492 | What is the average number of test takers from Fresno schools that opened between 1/1/1980 and 12/31/1980? | |
| 48 | california_schools | moderate | ✓ | ✓ | 24 | 6429 | What is the ratio of merged Unified School District schools in Orange County to merged Elementary School District school | |
| 50 | california_schools | simple | ✗ | ✓ | 12 | 6364 | What is the postal street address for the school with the 7th highest Math average? Indicate the school's name. | |
| 77 | california_schools | moderate | ✗ | ✓ | empty_result | 15 | 6471 | Which schools served a grade span of Kindergarten to 9th grade in the county of Los Angeles and what is its Percent (%) |
| 92 | financial | simple | ✓ | ✓ | 12 | 4497 | List out the no. of districts that have female average salary is more than 6000 but less than 10000? | |
| 98 | financial | moderate | ✓ | ✓ | 9 | 4492 | Among the accounts who have approved loan date in 1997, list out the accounts that have the lowest approved amount and c | |
| 99 | financial | moderate | ✓ | ✓ | 8 | 4483 | Among the accounts who have loan validity more than 12 months, list out the accounts that have the highest approved amou | |
| 112 | financial | simple | ✓ | ✓ | 11 | 4482 | For the female client who was born in 1976/1/29, which district did she opened her account? | |
| 115 | financial | challenging | ✗ | ✓ | 12 | 4564 | For the branch which located in the south Bohemia with biggest number of inhabitants, what is the percentage of the male | |
| 118 | financial | moderate | ✗ | ✓ | 12 | 4523 | For loan amount less than USD100,000, what is the percentage of accounts that is still running with no issue. | |
| 120 | financial | moderate | ✓ | ✓ | 18 | 4802 | From Year 1995 to 2000, who are the accounts holders from 'east Bohemia'. State the account ID the frequency of statemen | |
| 125 | financial | challenging | ✗ | ✓ | 10280 | 4891 | For loans contracts which are still running where client are in debt, list the district of the and the state the percent | |
| 138 | financial | moderate | ✓ | ✓ | 3311 | 4451 | In the branch where the second-highest number of crimes were committed in 1995 occurred, how many male clients are there | |
| 159 | financial | simple | ✗ | ✓ | 2852 | 4621 | List all the withdrawals in cash transactions that the client with the id 3356 makes. | |
| 168 | financial | moderate | ✗ | ✓ | 1982 | 4470 | What percentage of clients who opened their accounts in the district with an average salary of over 10000 are women? | |
| 169 | financial | challenging | ✗ | ✓ | 2776 | 4743 | What was the growth rate of the total amount of loans across all accounts for a male client between 1996 and 1997? | |
| 173 | financial | challenging | ✗ | ✓ | 1341 | 4596 | How often does account number 3 request an account statement to be released? What was the aim of debiting 3539 in total? | |
| 189 | financial | moderate | ✓ | ✓ | 2053 | 4458 | Name the account numbers of female clients who are oldest and have lowest average salary? | |
| 192 | financial | moderate | ✓ | ✓ | 11 | 4515 | What is the average amount of loan which are still on running contract with statement issuance after each transaction? | |
| 194 | financial | moderate | ✓ | ✓ | 1432 | 4446 | Provide the IDs and age of the client with high level credit card, which is eligible for loans. | |
| 207 | toxicology | challenging | ✗ | ✓ | 1633 | 2608 | What elements are in a double type bond? | |
| 208 | toxicology | moderate | ✓ | ✓ | 1474 | 2586 | Which type of label is the most numerous in atoms with hydrogen? | |
| 219 | toxicology | challenging | ✗ | ✓ | execution_failed | 1475 | 2379 | What is the percentage of carcinogenic molecules in triple type bonds? |
| 227 | toxicology | simple | ✓ | ✓ | 1140 | 2623 | What is the percentage of molecules that are carcinogenic? Please provide your answer as a percentage with three decimal | |
| 230 | toxicology | challenging | ✓ | ✓ | 1110 | 2598 | What are the elements of the toxicology and label of molecule TR060? | |
| 232 | toxicology | moderate | ✗ | ✓ | execution_failed | 1587 | 2377 | Please list top three molecules that have single bonds between two atoms and are not carcinogenic in alphabetical order. |
| 236 | toxicology | moderate | ✓ | ✓ | 1276 | 2652 | What are the bond type and the atoms of the bond ID of TR001_6_9? | |
| 239 | toxicology | simple | ✓ | ✓ | 1502 | 2568 | How many connections does the atom 19 have? | |
| 253 | toxicology | challenging | ✓ | ✓ | 4575 | 2596 | List the elements of all the triple bonds. | |
| 260 | toxicology | moderate | ✓ | ✓ | 15 | 2656 | Calculate the total atoms with triple-bond molecules containing the element phosphorus or bromine. | |
| 268 | toxicology | challenging | ✓ | ✓ | 10 | 2620 | What are the elements for bond id TR001_10_11? | |
| 273 | toxicology | moderate | ✗ | ✓ | 1831 | 2678 | What is the percentage of element chlorine in carcinogenic molecules? | |
| 282 | toxicology | challenging | ✓ | ✓ | 1211 | 2728 | What is the ratio of Hydrogen elements in molecule ID TR006? List the ratio with its label. | |
| 327 | toxicology | moderate | ✗ | ✓ | execution_failed | 1252 | 2353 | Which non-carcinogenic molecules consisted more than 5 atoms? |
| 347 | card_games | moderate | ✗ | ✓ | 3852 | 8735 | Find all cards illustrated by Stephen Daniel and describe the text of the ruling of these cards. State if these cards ha | |
| 349 | card_games | moderate | ✗ | ✓ | 2118 | 8529 | Name the card and artist with the most ruling information. Also state if the card is a promotional printing. | |
| 352 | card_games | moderate | ✗ | ✓ | 3113 | 8510 | Calculate the percentage of the cards availabe in Chinese Simplified. | |
| 356 | card_games | simple | ✓ | ✓ | 1387 | 8344 | How many cards have infinite power? | |
| 358 | card_games | simple | ✓ | ✓ | 1882 | 8400 | What is the border color of card "Ancestor's Chosen"? | |
| 366 | card_games | simple | ✗ | ✓ | 2613 | 8470 | What is the rule of playing card "Benalish Knight"? | |
| 377 | card_games | simple | ✗ | ✓ | 1287 | 8417 | How many cards with original type of "Summon - Angel" have subtype other than "Angel"? | |
| 391 | card_games | moderate | ✓ | ✓ | 4044 | 8521 | Among the Artifact cards, which are black color and comes with foreign languague translation? | |
| 407 | card_games | moderate | ✗ | ✓ | 1102 | 8528 | Lists all types of cards in German. | |
| 408 | card_games | moderate | ✗ | ✓ | 1450 | 8421 | How many unknown power cards contain info about the triggered ability | |
| 412 | card_games | moderate | ✗ | ✓ | empty_result | 1596 | 8498 | What is the foreign name of the card in French of type Creature, normal layout and black border color, by artist Matthew |
| 414 | card_games | simple | ✓ | ✓ | 1333 | 8498 | What language is the set of 180 cards that belongs to the Ravnica block translated into? | |
| 427 | card_games | moderate | ✓ | ✓ | 2003 | 8525 | What languages are available in the set known as Archenemy on the magic card market and having the code ARC? | |
| 459 | card_games | moderate | ✗ | ✓ | 1292 | 8510 | Which card costs more converted mana, "Serra Angel" or "Shrine Keeper"? | |
| 466 | card_games | moderate | ✓ | ✓ | 1464 | 8514 | Among the cards in the set "Hauptset Zehnte Edition", how many of them are designed by Adam Rex? | |
| 472 | card_games | moderate | ✓ | ✓ | 1370 | 8496 | Among the sets in the block "Ice Age", how many of them have an Italian translation? | |
| 484 | card_games | moderate | ✗ | ✓ | 1975 | 8541 | Please list the Italian names of the cards in the set Coldsnap with the highest converted mana cost. | |
| 486 | card_games | moderate | ✓ | ✓ | 2146 | 8619 | What is the percentage of the cards with a converted mana cost of 7 in the set Coldsnap? | |
| 518 | card_games | moderate | ✗ | ✓ | 61695 | 8576 | Which of the play format has the highest number of banned status? Indicate the play format and the names of all the card | |
| 531 | codebase_community | simple | ✗ | ✓ | 1694 | 5608 | Which user has a higher reputation, Harlan or Jarrod Dixon? | |
| 557 | codebase_community | moderate | ✗ | ✓ | 2455 | 6324 | Among the posts with a score of over 5, what is the percentage of them being owned by an elder user? | |
| 563 | codebase_community | moderate | ✗ | ✓ | empty_result | 1880 | 6353 | User No.3025 gave a comment at 20:29:39 on 2014/4/23 to a post, how many favorite counts did that post get? |
| 571 | codebase_community | moderate | ✗ | ✓ | 2099 | 6238 | For the user No.24, how many times is the number of his/her posts compared to his/her votes? | |
| 584 | codebase_community | moderate | ✗ | ✓ | 2607 | 6375 | Write all the comments left by users who edited the post titled 'Why square the difference instead of taking the absolut | |
| 595 | codebase_community | moderate | ✗ | ✓ | 2503 | 6271 | Which user have only one post history per post and having at least 1000 views? | |
| 634 | codebase_community | challenging | ✗ | ✓ | execution_failed | 1593 | 5928 | Among posts by Harvey Motulsky and Noah Snyder, which one has higher popularity? |
| 669 | codebase_community | simple | ✓ | ✓ | 1250 | 5591 | When did 'chl' cast its first vote in a post? | |
| 671 | codebase_community | simple | ✓ | ✓ | 1935 | 5603 | What is the display name of the user who acquired the first Autobiographer badge? | |
| 672 | codebase_community | moderate | ✗ | ✓ | 1748 | 6184 | Among the users located in United Kingdom, how many users whose post have a total favorite amount of 4 or more? | |
| 694 | codebase_community | moderate | ✗ | ✓ | 1643 | 6459 | Provide the text of the latest 10 comments to the post with the title 'Analysing wind data with R' and the display name | |
| 707 | codebase_community | moderate | ✓ | ✓ | 1669 | 6352 | Among the posts with views ranging from 100 to 150, what is the comment with the highest score? | |
| 716 | codebase_community | moderate | ✗ | ✓ | 1569 | 6308 | Among the comments with scores between 5 to 10, what is the percentage of the users with 0 up votes? | |
| 723 | superhero | moderate | ✓ | ✓ | 1518 | 3458 | Among the superheroes with blue eyes, how many of them have the super power of "Agility"? | |
| 730 | superhero | challenging | ✓ | ✓ | 1484 | 3489 | List the superheroes from Marvel Comics who have the super power of 'Super Strength'. | |
| 736 | superhero | moderate | ✓ | ✓ | 1162 | 3347 | Who is the dumbest superhero? | |
| 737 | superhero | simple | ✓ | ✓ | 1041 | 3271 | What is Copycat's race? | |
| 738 | superhero | simple | ✓ | ✓ | 1525 | 3429 | Which superheroes have a durability attribute value of less than 50? | |
| 743 | superhero | challenging | ✗ | ✓ | 4354 | 3511 | What is the percentage of superheroes who act in their own self-interest or make decisions based on their own moral code | |
| 747 | superhero | simple | ✓ | ✓ | 11 | 3271 | What is the total number of superheroes without full name? | |
| 750 | superhero | simple | ✓ | ✓ | 8 | 3334 | What is the average weight of all female superheroes? | |
| 751 | superhero | moderate | ✓ | ✓ | 1269 | 3423 | List down at least five superpowers of male superheroes. | |
| 753 | superhero | moderate | ✓ | ✓ | 1109 | 3458 | Among the superheroes with height from 170 to 190, list the names of the superheroes with no eye color. | |
| 765 | superhero | simple | ✓ | ✓ | 1075 | 3307 | How many heroes have stealth power? | |
| 773 | superhero | challenging | ✓ | ✓ | 9921 | 3417 | Which superhero has the same eyes, hair and skin colour? Indicate the publisher of the superhero. | |
| 775 | superhero | challenging | ✓ | ✓ | 5549 | 3519 | What is the percentage of blue female superheroes among all female superheroes? | |
| 781 | superhero | simple | ✓ | ✓ | 10 | 3373 | Provide the heights of the heroes whose eye colours are amber. | |
| 785 | superhero | simple | ✓ | ✓ | 7 | 3341 | Describe the names of neutral alignment superheroes. | |
| 791 | superhero | simple | ✗ | ✓ | 890 | 3293 | Calculate the average height for all superhero. | |
| 794 | superhero | moderate | ✓ | ✓ | 1176 | 3353 | Which hero was the fastest? | |
| 798 | superhero | moderate | ✓ | ✓ | 1104 | 3397 | What is the publisher for Hawkman, Karate Kid and Speedy? | |
| 800 | superhero | moderate | ✓ | ✓ | 1632 | 3430 | Calculate the percentage of superheroes with blue eyes. | |
| 806 | superhero | simple | ✓ | ✓ | 1150 | 3277 | Provide the eye colour of the superhero who has Karen Beecher-Duncan as their full name. | |
| 819 | superhero | challenging | ✓ | ✓ | 1975 | 3551 | In superheroes with missing weight data, calculate the difference between the number of superheroes with blue eyes and n | |
| 825 | superhero | moderate | ✓ | ✓ | 1300 | 3373 | Identify the gender of the superhero who has the ability of Phoenix Force. | |
| 847 | formula_1 | simple | ✗ | ✓ | 2143 | 6460 | What is the surname of the driver with the best lap time in race number 19 in the second qualifying period? | |
| 859 | formula_1 | simple | ✓ | ✓ | 1422 | 6495 | What's Bruno Senna's Q1 result in the qualifying race No. 354? | |
| 861 | formula_1 | simple | ✗ | ✓ | empty_result | 1361 | 6452 | What is his number of the driver who finished 0:01:54 in the Q3 of qualifying race No.903? |
| 862 | formula_1 | simple | ✓ | ✓ | 1293 | 6447 | For the Bahrain Grand Prix in 2007, how many drivers not finished the game? | |
| 865 | formula_1 | moderate | ✓ | ✓ | 1493 | 6510 | For all the drivers who finished the game in race No. 592, who is the oldest? | |
| 866 | formula_1 | moderate | ✗ | ✓ | 1259 | 6559 | Who was the player that got the lap time of 0:01:27 in the race No. 161? Show his introduction website. | |
| 875 | formula_1 | simple | ✓ | ✓ | 1176 | 6403 | Show me the season page of year when the race No. 901 took place. | |
| 877 | formula_1 | moderate | ✓ | ✓ | 1355 | 6459 | For all the drivers who finished the game in race No. 872, who is the youngest? | |
| 879 | formula_1 | moderate | ✓ | ✓ | 1337 | 6405 | For the driver who set the fastest lap speed, what is his nationality? | |
| 881 | formula_1 | moderate | ✗ | ✓ | 1988 | 6552 | For the drivers who took part in the race in 1983/7/16, what's their race completion rate? | |
| 894 | formula_1 | moderate | ✗ | ✓ | 2182 | 6518 | What is the best lap time recorded? List the driver and race with such recorded lap time. | |
| 896 | formula_1 | challenging | ✗ | ✓ | 1578 | 6579 | Calculate the percentage whereby Hamilton was not at the 1st track of the the f1 circuit since 2010. | |
| 897 | formula_1 | moderate | ✓ | ✓ | 1307 | 6511 | Name the driver with the most winning. Mention his nationality and what is his maximum point scores. | |
| 898 | formula_1 | simple | ✗ | ✓ | execution_failed | 1235 | 6188 | How old is the youngest Japanese driver? What is his name? |
| 902 | formula_1 | simple | ✗ | ✓ | 1355 | 6518 | Which race was Alex Yoong in when he was in track number less than 20? | |
| 904 | formula_1 | moderate | ✗ | ✓ | 2364 | 6291 | State the race and year of race in which Michael Schumacher had his fastest lap. | |
| 909 | formula_1 | moderate | ✗ | ✓ | 1976 | 6512 | Among all European Grand Prix races, what is the percentage of the races were hosted in Germany? | |
| 912 | formula_1 | simple | ✓ | ✓ | 976 | 6108 | What's the reference name of Marina Bay Street Circuit? | |
| 915 | formula_1 | simple | ✗ | ✓ | 2007 | 6422 | Which country is the oldest driver from? | |
| 930 | formula_1 | simple | ✗ | ✓ | 1360 | 6472 | In which Formula_1 race did Lewis Hamilton rank the highest? | |
| 945 | formula_1 | simple | ✓ | ✓ | 1037 | 6078 | How many circuits are there in Adelaide, Australia? | |
| 950 | formula_1 | simple | ✗ | ✓ | 1201 | 6433 | Please list the constructor names with 0 points at race 291. | |
| 959 | formula_1 | simple | ✗ | ✓ | 2146 | 6509 | What is the fastest lap number of the champion in 2009? | |
| 971 | formula_1 | simple | ✓ | ✓ | 1113 | 6328 | Please state the reference name of the oldest German driver. | |
| 981 | formula_1 | moderate | ✓ | ✓ | 1552 | 6555 | On what year did the youngest driver had his first qualifying race? Also state the name, date and time of the race. | |
| 988 | formula_1 | challenging | ✗ | ✓ | 4196 | 6435 | List down top 3 German drivers who has the shortest average pit stop duration and were born between 1980-1985. | |
| 989 | formula_1 | moderate | ✗ | ✓ | 14 | 6501 | Who is the champion of the Canadian Grand Prix in 2008? Indicate his finish time. | |
| 990 | formula_1 | challenging | ✓ | ✓ | 1373 | 6522 | What is the constructor reference name of the champion in the 2009 Singapore Grand Prix? Please give its website. | |
| 1028 | european_football_2 | challenging | ✗ | ✓ | 1815 | 11825 | In Scotland Premier League, which away team won the most during the 2010 season? | |
| 1029 | european_football_2 | moderate | ✗ | ✓ | 1520 | 11678 | What are the speed in which attacks are put together of the top 4 teams with the highest build Up Play Speed? | |
| 1030 | european_football_2 | moderate | ✓ | ✓ | 1561 | 11668 | Give the name of the league had the most matches end as draw in the 2016 season? | |
| 1035 | european_football_2 | simple | ✓ | ✓ | 1293 | 11549 | Give the team_fifa_api_id of teams with more than 50 but less than 60 build-up play speed. | |
| 1036 | european_football_2 | challenging | ✗ | ✓ | 1704 | 11735 | List the long name of teams with above-average build-up play passing in 2012. | |
| 1037 | european_football_2 | challenging | ✗ | ✓ | 2731 | 11786 | Calculate the percentage of players who prefer left foot, who were born between 1987 and 1992. | |
| 1039 | european_football_2 | simple | ✓ | ✓ | 2128 | 11583 | Find the average number of long-shot done by Ahmed Samir Farag. | |
| 1042 | european_football_2 | challenging | ✓ | ✓ | 1565 | 11823 | List the name of leagues in which the average goals by the home team is higher than the away team in the 2009/2010 seaso | |
| 1057 | european_football_2 | moderate | ✓ | ✓ | 1545 | 11669 | Calculate the average home team goal in the 2010/2011 season in the country of Poland. | |
| 1078 | european_football_2 | simple | ✓ | ✓ | 1287 | 11421 | Which player is older, Aaron Lennon or Abdelaziz Barrada? | |
| 1088 | european_football_2 | moderate | ✓ | ✓ | 1563 | 11690 | Please list the names of the players whose volley score and dribbling score are over 70. | |
| 1094 | european_football_2 | challenging | ✗ | ✓ | execution_failed | 2085 | 11469 | How much higher in percentage is Ariel Borysiuk's overall rating than that of Paulin Puel? |
| 1103 | european_football_2 | moderate | ✓ | ✓ | 1320 | 11633 | What was the overall rating for Aaron Mooy on 2016/2/4? | |
| 1110 | european_football_2 | moderate | ✓ | ✓ | 1491 | 11581 | Tell the build Up play passing class for "FC Lorient" on 2010/2/22. | |
| 1116 | european_football_2 | simple | ✓ | ✓ | 1185 | 11494 | List down most tallest players' name. | |
| 1122 | european_football_2 | simple | ✓ | ✓ | 10452 | 11468 | State the name of the most strongest player. | |
| 1130 | european_football_2 | moderate | ✓ | ✓ | 1252 | 11605 | What are the short name of team who played safe while creating chance of passing? | |
| 1133 | european_football_2 | simple | ✓ | ✓ | 1216 | 11464 | How many football players born after the 1990s have the first name "Aaron"? | |
| 1141 | european_football_2 | moderate | ✓ | ✓ | 1328 | 11505 | Does the KSV Cercle Brugge team have a slow, balanced or fast speed class? | |
| 1144 | european_football_2 | simple | ✗ | ✓ | 1259 | 11606 | Please state the finishing rate and curve score of the player who has the heaviest weight. | |
| 1146 | european_football_2 | moderate | ✓ | ✓ | 1452 | 11564 | Please provide the full name of the away team that scored the most goals. | |
| 1147 | european_football_2 | simple | ✓ | ✓ | 2180 | 11482 | Please name one player whose overall strength is the greatest. | |
| 1152 | thrombosis_prediction | moderate | ✓ | ✓ | 1309 | 4877 | What is the ratio of outpatient to inpatient followed up treatment among all the 'SLE' diagnosed patient? | |
| 1156 | thrombosis_prediction | moderate | ✗ | ✓ | execution_failed | 1218 | 4482 | State the ID and age of patient with positive degree of coagulation. |
| 1157 | thrombosis_prediction | simple | ✓ | ✓ | 1694 | 4761 | For patients with severe degree of thrombosis, list their ID, sex and disease the patient is diagnosed with. | |
| 1168 | thrombosis_prediction | challenging | ✗ | ✓ | execution_failed | 1438 | 4527 | The oldest SJS patient's medical laboratory work was completed on what date, and what age was the patient when they init |
| 1185 | thrombosis_prediction | challenging | ✗ | ✓ | 1654 | 5188 | For the patient who was born on 1959/2/18, what is the decrease rate for his/her total cholesterol from November to Dece | |
| 1198 | thrombosis_prediction | simple | ✓ | ✓ | 972 | 4651 | How many female patients were given an APS diagnosis? | |
| 1205 | thrombosis_prediction | moderate | ✗ | ✓ | 1938 | 4861 | Was the patient with the number 57266's uric acid within a normal range? | |
| 1208 | thrombosis_prediction | moderate | ✓ | ✓ | 1072 | 4841 | Provide IDs for male patients with ALT glutamic pylvic transaminase (GPT) that have history of ALT glutamic pylvic trans | |
| 1220 | thrombosis_prediction | simple | ✓ | ✓ | 1185 | 4874 | Provide all ID, sex and birthday of patients whose urea nitrogen (UN) just within the borderline of passing? | |
| 1227 | thrombosis_prediction | moderate | ✗ | ✓ | execution_failed | 1162 | 4503 | What is the average age of the male patient with high cholesterol? |
| 1232 | thrombosis_prediction | challenging | ✓ | ✓ | 1377 | 4993 | Provide ID, sex and age of patient who has blood glucose (GLU) not within normal range but with total cholesterol(T-CHO) | |
| 1235 | thrombosis_prediction | moderate | ✗ | ✓ | execution_failed | 1226 | 4487 | What are the patient's diagnosis for those who has lower red blood blood cell? State their ID and age. |
| 1247 | thrombosis_prediction | challenging | ✗ | ✓ | 1300 | 4861 | Among the male patients who have a normal level of white blood cells, how many of them have an abnormal fibrinogen level | |
| 1251 | thrombosis_prediction | simple | ✗ | ✓ | 1038 | 4681 | How many patients with an Ig G higher than normal? | |
| 1252 | thrombosis_prediction | moderate | ✗ | ✓ | 1750 | 4759 | Among the patients with a normal Ig G level, how many of them have symptoms? | |
| 1254 | thrombosis_prediction | moderate | ✗ | ✓ | execution_failed | 1257 | 4528 | How many patients with a normal Ig A level came to the hospital after 1990/1/1? |
| 1255 | thrombosis_prediction | moderate | ✗ | ✓ | execution_failed | 1114 | 4475 | For the patients with an abnormal Ig M level, what is the most common disease they are diagnosed with? |
| 1257 | thrombosis_prediction | challenging | ✗ | ✓ | 1344 | 4793 | Among the patients whose creatinine level is abnormal, how many of them aren't 70 yet? | |
| 1275 | thrombosis_prediction | moderate | ✗ | ✓ | execution_failed | 1267 | 4537 | Among the patients who has a normal level of anti-centromere and a normal level of anti-SSB, how many of them are male? |
| 1281 | thrombosis_prediction | moderate | ✓ | ✓ | 1666 | 4762 | Among the patients who have an abnormal level of glutamic oxaloacetic transaminase, when was the youngest of them born? | |
| 1302 | thrombosis_prediction | challenging | ✗ | ✓ | execution_failed | 1140 | 4500 | For the patients with a normal range of creatinine phosphokinase, how many of them have a positive measure of degree of |
| 1312 | student_club | simple | ✓ | ✓ | 1188 | 4633 | What's Angela Sanders's major? | |
| 1340 | student_club | moderate | ✗ | ✓ | execution_failed | 1531 | 4504 | Calculate the difference of the total amount spent in all events by the Student_Club in year 2019 and 2020. |
| 1344 | student_club | simple | ✓ | ✓ | 1167 | 4696 | What was the notes of the fundraising on 2019/9/14? | |
| 1352 | student_club | moderate | ✓ | ✓ | 1093 | 4668 | For all the club members from "Business" major, how many of them wear medium size t-shirt? | |
| 1356 | student_club | simple | ✓ | ✓ | 1088 | 4602 | Which department was the President of the club in? | |
| 1376 | student_club | moderate | ✓ | ✓ | 1184 | 4686 | Among all the closed events, which event has the highest spend-to-budget ratio? | |
| 1378 | student_club | simple | ✓ | ✓ | 939 | 4097 | What is the highest amount of budget spend for an event? | |
| 1380 | student_club | simple | ✓ | ✓ | 1125 | 4373 | What is the total amount of money spent for food? | |
| 1387 | student_club | moderate | ✓ | ✓ | 2009 | 4755 | Which student has been entrusted to manage the budget for the Yearly Kickoff? | |
| 1390 | student_club | moderate | ✗ | ✓ | 1376 | 4250 | Based on the total cost for all event, what is the percentage of cost for Yearly Kickoff event? | |
| 1399 | student_club | moderate | ✗ | ✓ | 1926 | 4729 | Did Maya Mclean attend the 'Women's Soccer' event? | |
| 1403 | student_club | moderate | ✓ | ✓ | 1292 | 4694 | Indicate the name of the closed event whose cost has exceeded the budget the most. | |
| 1404 | student_club | moderate | ✗ | ✓ | execution_failed | 1353 | 4444 | Identify the type of expenses and their total value approved for 'October Meeting' event. |
| 1409 | student_club | simple | ✓ | ✓ | 1089 | 4658 | Mention the total expense used on 8/20/2019. | |
| 1410 | student_club | simple | ✓ | ✓ | 1340 | 4728 | List out the full name and total cost that member id "rec4BLdZHS2Blfp4v" incurred? | |
| 1411 | student_club | simple | ✓ | ✓ | 1146 | 4690 | State what kind of expenses that Sacha Harrison incurred? | |
| 1422 | student_club | simple | ✗ | ✓ | 1053 | 4617 | State the category of events were held at MU 215. | |
| 1464 | student_club | challenging | ✓ | ✓ | 1198 | 4740 | Write the full names of students who received funds on the date of 9/9/2019 and include the amount received. | |
| 1472 | debit_card_specializing | moderate | ✗ | ✓ | 1876 | 3066 | In 2012, who had the least consumption in LAM? | |
| 1473 | debit_card_specializing | moderate | ✓ | ✓ | 1654 | 3115 | What was the average monthly consumption of customers in SME for the year 2013? | |
| 1476 | debit_card_specializing | challenging | ✓ | ✓ | 9933 | 3220 | What was the difference in gas consumption between CZK-paying customers and EUR-paying customers in 2012? | |
| 1479 | debit_card_specializing | moderate | ✓ | ✓ | 317 | 2997 | Which year recorded the most consumption of gas paid in CZK? | |
| 1480 | debit_card_specializing | moderate | ✗ | ✓ | 276 | 3089 | What was the gas consumption peak month for SME customers in 2013? | |
| 1484 | debit_card_specializing | simple | ✓ | ✓ | 10 | 3072 | How many more "discount" gas stations does the Czech Republic have compared to Slovakia? | |
| 1486 | debit_card_specializing | simple | ✗ | ✓ | 1385 | 3061 | Is it true that more SMEs pay in Czech koruna than in euros? If so, how many more? | |
| 1493 | debit_card_specializing | simple | ✓ | ✓ | 13595 | 3132 | In February 2012, what percentage of customers consumed more than 528.3? | |
| 1500 | debit_card_specializing | simple | ✓ | ✓ | 688 | 3105 | Please list the product description of the products consumed in September, 2013. | |
| 1501 | debit_card_specializing | moderate | ✓ | ✓ | 1359 | 3091 | Please list the countries of the gas stations with transactions taken place in June, 2013. | |
| 1506 | debit_card_specializing | moderate | ✓ | ✓ | 1947 | 3041 | Please list the product descriptions of the transactions taken place in the gas stations in the Czech Republic. | |
| 1515 | debit_card_specializing | simple | ✓ | ✓ | 1358 | 3030 | What segment did the customer have at 2012/8/23 21:20:00? | |
| 1521 | debit_card_specializing | moderate | ✓ | ✓ | 1370 | 3199 | For all the transactions happened during 8:00-9:00 in 2012/8/26, how many happened in CZE? | |
| 1525 | debit_card_specializing | simple | ✗ | ✓ | 2115 | 3091 | What is the percentage of the customers who used EUR in 2012/8/25? | |
| 1526 | debit_card_specializing | challenging | ✗ | ✓ | empty_result | 1779 | 3289 | For the customer who paid 634.8 in 2012/8/25, what was the consumption decrease rate from Year 2012 to 2013? |
| 1528 | debit_card_specializing | simple | ✓ | ✓ | 1714 | 2956 | What is the percentage of "premium" against the overall segment in Country = "SVK"? | |
| 1529 | debit_card_specializing | moderate | ✗ | ✓ | 1990 | 2971 | What is the amount spent by customer "38508" at the gas stations? How much had the customer spent in January 2012? | |
| 1531 | debit_card_specializing | moderate | ✗ | ✓ | 2673 | 3074 | Who is the top spending customer and how much is the average price per single item purchased by this customer? What curr |
Model: codestral-latest · n=200 · EA=55.5% · Validity=100.0% · Recall@k=100.0%
| qid | db | diff | match | recall | err | lat ms | tokens | question |
|---|---|---|---|---|---|---|---|---|
| 5 | california_schools | simple | ✓ | ✓ | 79 | 6331 | How many schools with an average score in Math greater than 400 in the SAT test are exclusively virtual? | |
| 25 | california_schools | moderate | ✗ | ✓ | 21 | 6403 | Name schools in Riverside which the average of average math score for SAT is grater than 400, what is the funding type o | |
| 32 | california_schools | moderate | ✗ | ✓ | execution_failed | 16 | 6167 | What is the eligible free or reduced price meal rate for the top 5 schools in grades 1-12 with the highest free or reduc |
| 36 | california_schools | challenging | ✓ | ✓ | 10 | 6572 | Under whose administration is the school with the highest number of students scoring 1500 or more on the SAT? Indicate t | |
| 37 | california_schools | moderate | ✗ | ✓ | 13 | 6462 | What is the complete address of the school with the lowest excellence rate? Indicate the Street, City, Zip and State. | |
| 39 | california_schools | simple | ✓ | ✓ | 23 | 6492 | What is the average number of test takers from Fresno schools that opened between 1/1/1980 and 12/31/1980? | |
| 48 | california_schools | moderate | ✓ | ✓ | 20 | 6429 | What is the ratio of merged Unified School District schools in Orange County to merged Elementary School District school | |
| 50 | california_schools | simple | ✗ | ✓ | 10 | 6364 | What is the postal street address for the school with the 7th highest Math average? Indicate the school's name. | |
| 77 | california_schools | moderate | ✗ | ✓ | empty_result | 15 | 6471 | Which schools served a grade span of Kindergarten to 9th grade in the county of Los Angeles and what is its Percent (%) |
| 92 | financial | simple | ✓ | ✓ | 11 | 4497 | List out the no. of districts that have female average salary is more than 6000 but less than 10000? | |
| 98 | financial | moderate | ✓ | ✓ | 9 | 4492 | Among the accounts who have approved loan date in 1997, list out the accounts that have the lowest approved amount and c | |
| 99 | financial | moderate | ✓ | ✓ | 8 | 4483 | Among the accounts who have loan validity more than 12 months, list out the accounts that have the highest approved amou | |
| 112 | financial | simple | ✓ | ✓ | 9 | 4482 | For the female client who was born in 1976/1/29, which district did she opened her account? | |
| 115 | financial | challenging | ✗ | ✓ | 11 | 4564 | For the branch which located in the south Bohemia with biggest number of inhabitants, what is the percentage of the male | |
| 118 | financial | moderate | ✗ | ✓ | 9 | 4523 | For loan amount less than USD100,000, what is the percentage of accounts that is still running with no issue. | |
| 120 | financial | moderate | ✓ | ✓ | 13 | 4802 | From Year 1995 to 2000, who are the accounts holders from 'east Bohemia'. State the account ID the frequency of statemen | |
| 125 | financial | challenging | ✗ | ✓ | 11 | 4902 | For loans contracts which are still running where client are in debt, list the district of the and the state the percent | |
| 138 | financial | moderate | ✓ | ✓ | 9 | 4451 | In the branch where the second-highest number of crimes were committed in 1995 occurred, how many male clients are there | |
| 159 | financial | simple | ✗ | ✓ | 274 | 4621 | List all the withdrawals in cash transactions that the client with the id 3356 makes. | |
| 168 | financial | moderate | ✗ | ✓ | 15 | 4538 | What percentage of clients who opened their accounts in the district with an average salary of over 10000 are women? | |
| 169 | financial | challenging | ✗ | ✓ | 13 | 4743 | What was the growth rate of the total amount of loans across all accounts for a male client between 1996 and 1997? | |
| 173 | financial | challenging | ✗ | ✓ | 71 | 4596 | How often does account number 3 request an account statement to be released? What was the aim of debiting 3539 in total? | |
| 189 | financial | moderate | ✓ | ✓ | 12 | 4458 | Name the account numbers of female clients who are oldest and have lowest average salary? | |
| 192 | financial | moderate | ✓ | ✓ | 10 | 4515 | What is the average amount of loan which are still on running contract with statement issuance after each transaction? | |
| 194 | financial | moderate | ✓ | ✓ | 9 | 4443 | Provide the IDs and age of the client with high level credit card, which is eligible for loans. | |
| 207 | toxicology | challenging | ✗ | ✓ | 47 | 2608 | What elements are in a double type bond? | |
| 208 | toxicology | moderate | ✓ | ✓ | 22 | 2586 | Which type of label is the most numerous in atoms with hydrogen? | |
| 219 | toxicology | challenging | ✗ | ✓ | execution_failed | 25 | 2379 | What is the percentage of carcinogenic molecules in triple type bonds? |
| 227 | toxicology | simple | ✓ | ✓ | 13 | 2623 | What is the percentage of molecules that are carcinogenic? Please provide your answer as a percentage with three decimal | |
| 230 | toxicology | challenging | ✓ | ✓ | 12 | 2598 | What are the elements of the toxicology and label of molecule TR060? | |
| 232 | toxicology | moderate | ✗ | ✓ | execution_failed | 14 | 2377 | Please list top three molecules that have single bonds between two atoms and are not carcinogenic in alphabetical order. |
| 236 | toxicology | moderate | ✓ | ✓ | 16 | 2652 | What are the bond type and the atoms of the bond ID of TR001_6_9? | |
| 239 | toxicology | simple | ✓ | ✓ | 23 | 2568 | How many connections does the atom 19 have? | |
| 253 | toxicology | challenging | ✓ | ✓ | 39 | 2596 | List the elements of all the triple bonds. | |
| 260 | toxicology | moderate | ✓ | ✓ | 37 | 2656 | Calculate the total atoms with triple-bond molecules containing the element phosphorus or bromine. | |
| 268 | toxicology | challenging | ✓ | ✓ | 21 | 2620 | What are the elements for bond id TR001_10_11? | |
| 273 | toxicology | moderate | ✗ | ✓ | 24 | 2678 | What is the percentage of element chlorine in carcinogenic molecules? | |
| 282 | toxicology | challenging | ✓ | ✓ | 21 | 2728 | What is the ratio of Hydrogen elements in molecule ID TR006? List the ratio with its label. | |
| 327 | toxicology | moderate | ✗ | ✓ | execution_failed | 21 | 2353 | Which non-carcinogenic molecules consisted more than 5 atoms? |
| 347 | card_games | moderate | ✗ | ✓ | 516 | 9056 | Find all cards illustrated by Stephen Daniel and describe the text of the ruling of these cards. State if these cards ha | |
| 349 | card_games | moderate | ✗ | ✓ | 700 | 8529 | Name the card and artist with the most ruling information. Also state if the card is a promotional printing. | |
| 352 | card_games | moderate | ✗ | ✓ | 927 | 8510 | Calculate the percentage of the cards availabe in Chinese Simplified. | |
| 356 | card_games | simple | ✓ | ✓ | 191 | 8344 | How many cards have infinite power? | |
| 358 | card_games | simple | ✓ | ✓ | 194 | 8403 | What is the border color of card "Ancestor's Chosen"? | |
| 366 | card_games | simple | ✗ | ✓ | 1255 | 8471 | What is the rule of playing card "Benalish Knight"? | |
| 377 | card_games | simple | ✗ | ✓ | 203 | 8417 | How many cards with original type of "Summon - Angel" have subtype other than "Angel"? | |
| 391 | card_games | moderate | ✓ | ✓ | 995 | 8521 | Among the Artifact cards, which are black color and comes with foreign languague translation? | |
| 407 | card_games | moderate | ✗ | ✓ | 681 | 8528 | Lists all types of cards in German. | |
| 408 | card_games | moderate | ✗ | ✓ | 293 | 8421 | How many unknown power cards contain info about the triggered ability | |
| 412 | card_games | moderate | ✗ | ✓ | empty_result | 539 | 8498 | What is the foreign name of the card in French of type Creature, normal layout and black border color, by artist Matthew |
| 414 | card_games | simple | ✓ | ✓ | 39 | 8481 | What language is the set of 180 cards that belongs to the Ravnica block translated into? | |
| 427 | card_games | moderate | ✓ | ✓ | 34 | 8524 | What languages are available in the set known as Archenemy on the magic card market and having the code ARC? | |
| 459 | card_games | moderate | ✗ | ✓ | 209 | 8510 | Which card costs more converted mana, "Serra Angel" or "Shrine Keeper"? | |
| 466 | card_games | moderate | ✓ | ✓ | 249 | 8514 | Among the cards in the set "Hauptset Zehnte Edition", how many of them are designed by Adam Rex? | |
| 472 | card_games | moderate | ✓ | ✓ | 46 | 8496 | Among the sets in the block "Ice Age", how many of them have an Italian translation? | |
| 484 | card_games | moderate | ✗ | ✓ | 638 | 8540 | Please list the Italian names of the cards in the set Coldsnap with the highest converted mana cost. | |
| 486 | card_games | moderate | ✓ | ✓ | 331 | 8619 | What is the percentage of the cards with a converted mana cost of 7 in the set Coldsnap? | |
| 518 | card_games | moderate | ✗ | ✓ | 60179 | 8582 | Which of the play format has the highest number of banned status? Indicate the play format and the names of all the card | |
| 531 | codebase_community | simple | ✗ | ✓ | 47 | 5598 | Which user has a higher reputation, Harlan or Jarrod Dixon? | |
| 557 | codebase_community | moderate | ✗ | ✓ | 558 | 6297 | Among the posts with a score of over 5, what is the percentage of them being owned by an elder user? | |
| 563 | codebase_community | moderate | ✗ | ✓ | empty_result | 168 | 6357 | User No.3025 gave a comment at 20:29:39 on 2014/4/23 to a post, how many favorite counts did that post get? |
| 571 | codebase_community | moderate | ✗ | ✓ | 367 | 6239 | For the user No.24, how many times is the number of his/her posts compared to his/her votes? | |
| 584 | codebase_community | moderate | ✗ | ✓ | 995 | 6372 | Write all the comments left by users who edited the post titled 'Why square the difference instead of taking the absolut | |
| 595 | codebase_community | moderate | ✗ | ✓ | 1234 | 6271 | Which user have only one post history per post and having at least 1000 views? | |
| 634 | codebase_community | challenging | ✗ | ✓ | execution_failed | 476 | 5928 | Among posts by Harvey Motulsky and Noah Snyder, which one has higher popularity? |
| 669 | codebase_community | simple | ✓ | ✓ | 24 | 5591 | When did 'chl' cast its first vote in a post? | |
| 671 | codebase_community | simple | ✓ | ✓ | 42 | 5606 | What is the display name of the user who acquired the first Autobiographer badge? | |
| 672 | codebase_community | moderate | ✗ | ✓ | 282 | 6184 | Among the users located in United Kingdom, how many users whose post have a total favorite amount of 4 or more? | |
| 694 | codebase_community | moderate | ✗ | ✓ | 612 | 6459 | Provide the text of the latest 10 comments to the post with the title 'Analysing wind data with R' and the display name | |
| 707 | codebase_community | moderate | ✓ | ✓ | 502 | 6352 | Among the posts with views ranging from 100 to 150, what is the comment with the highest score? | |
| 716 | codebase_community | moderate | ✗ | ✓ | 119 | 6308 | Among the comments with scores between 5 to 10, what is the percentage of the users with 0 up votes? | |
| 723 | superhero | moderate | ✓ | ✓ | 17 | 3458 | Among the superheroes with blue eyes, how many of them have the super power of "Agility"? | |
| 730 | superhero | challenging | ✓ | ✓ | 167 | 3489 | List the superheroes from Marvel Comics who have the super power of 'Super Strength'. | |
| 736 | superhero | moderate | ✓ | ✓ | 16 | 3347 | Who is the dumbest superhero? | |
| 737 | superhero | simple | ✓ | ✓ | 9 | 3271 | What is Copycat's race? | |
| 738 | superhero | simple | ✓ | ✓ | 167 | 3429 | Which superheroes have a durability attribute value of less than 50? | |
| 743 | superhero | challenging | ✗ | ✓ | 24 | 3511 | What is the percentage of superheroes who act in their own self-interest or make decisions based on their own moral code | |
| 747 | superhero | simple | ✓ | ✓ | 17 | 3271 | What is the total number of superheroes without full name? | |
| 750 | superhero | simple | ✓ | ✓ | 16 | 3334 | What is the average weight of all female superheroes? | |
| 751 | superhero | moderate | ✓ | ✓ | 21 | 3423 | List down at least five superpowers of male superheroes. | |
| 753 | superhero | moderate | ✓ | ✓ | 17 | 3458 | Among the superheroes with height from 170 to 190, list the names of the superheroes with no eye color. | |
| 765 | superhero | simple | ✓ | ✓ | 20 | 3307 | How many heroes have stealth power? | |
| 773 | superhero | challenging | ✓ | ✓ | 17 | 3413 | Which superhero has the same eyes, hair and skin colour? Indicate the publisher of the superhero. | |
| 775 | superhero | challenging | ✓ | ✓ | 15 | 3519 | What is the percentage of blue female superheroes among all female superheroes? | |
| 781 | superhero | simple | ✓ | ✓ | 11 | 3373 | Provide the heights of the heroes whose eye colours are amber. | |
| 785 | superhero | simple | ✓ | ✓ | 11 | 3341 | Describe the names of neutral alignment superheroes. | |
| 791 | superhero | simple | ✗ | ✓ | 12 | 3293 | Calculate the average height for all superhero. | |
| 794 | superhero | moderate | ✓ | ✓ | 17 | 3353 | Which hero was the fastest? | |
| 798 | superhero | moderate | ✓ | ✓ | 11 | 3397 | What is the publisher for Hawkman, Karate Kid and Speedy? | |
| 800 | superhero | moderate | ✓ | ✓ | 14 | 3430 | Calculate the percentage of superheroes with blue eyes. | |
| 806 | superhero | simple | ✓ | ✓ | 12 | 3277 | Provide the eye colour of the superhero who has Karen Beecher-Duncan as their full name. | |
| 819 | superhero | challenging | ✓ | ✓ | 11 | 3551 | In superheroes with missing weight data, calculate the difference between the number of superheroes with blue eyes and n | |
| 825 | superhero | moderate | ✓ | ✓ | 14 | 3369 | Identify the gender of the superhero who has the ability of Phoenix Force. | |
| 847 | formula_1 | simple | ✗ | ✓ | 14 | 6471 | What is the surname of the driver with the best lap time in race number 19 in the second qualifying period? | |
| 859 | formula_1 | simple | ✓ | ✓ | 13 | 6495 | What's Bruno Senna's Q1 result in the qualifying race No. 354? | |
| 861 | formula_1 | simple | ✗ | ✓ | empty_result | 13 | 6452 | What is his number of the driver who finished 0:01:54 in the Q3 of qualifying race No.903? |
| 862 | formula_1 | simple | ✓ | ✓ | 19 | 6452 | For the Bahrain Grand Prix in 2007, how many drivers not finished the game? | |
| 865 | formula_1 | moderate | ✓ | ✓ | 21 | 6499 | For all the drivers who finished the game in race No. 592, who is the oldest? | |
| 866 | formula_1 | moderate | ✗ | ✓ | 22 | 6559 | Who was the player that got the lap time of 0:01:27 in the race No. 161? Show his introduction website. | |
| 875 | formula_1 | simple | ✓ | ✓ | 14 | 6403 | Show me the season page of year when the race No. 901 took place. | |
| 877 | formula_1 | moderate | ✓ | ✓ | 21 | 6458 | For all the drivers who finished the game in race No. 872, who is the youngest? | |
| 879 | formula_1 | moderate | ✓ | ✓ | 25 | 6405 | For the driver who set the fastest lap speed, what is his nationality? | |
| 881 | formula_1 | moderate | ✗ | ✓ | execution_failed | 21 | 6227 | For the drivers who took part in the race in 1983/7/16, what's their race completion rate? |
| 894 | formula_1 | moderate | ✗ | ✓ | 161 | 6518 | What is the best lap time recorded? List the driver and race with such recorded lap time. | |
| 896 | formula_1 | challenging | ✗ | ✓ | 32 | 6579 | Calculate the percentage whereby Hamilton was not at the 1st track of the the f1 circuit since 2010. | |
| 897 | formula_1 | moderate | ✓ | ✓ | 28 | 6511 | Name the driver with the most winning. Mention his nationality and what is his maximum point scores. | |
| 898 | formula_1 | simple | ✗ | ✓ | execution_failed | 12 | 6188 | How old is the youngest Japanese driver? What is his name? |
| 902 | formula_1 | simple | ✗ | ✓ | 21 | 6518 | Which race was Alex Yoong in when he was in track number less than 20? | |
| 904 | formula_1 | moderate | ✗ | ✓ | 510 | 6291 | State the race and year of race in which Michael Schumacher had his fastest lap. | |
| 909 | formula_1 | moderate | ✗ | ✓ | 17 | 6512 | Among all European Grand Prix races, what is the percentage of the races were hosted in Germany? | |
| 912 | formula_1 | simple | ✓ | ✓ | 13 | 6108 | What's the reference name of Marina Bay Street Circuit? | |
| 915 | formula_1 | simple | ✗ | ✓ | 21 | 6422 | Which country is the oldest driver from? | |
| 930 | formula_1 | simple | ✗ | ✓ | 24 | 6472 | In which Formula_1 race did Lewis Hamilton rank the highest? | |
| 945 | formula_1 | simple | ✓ | ✓ | 15 | 6078 | How many circuits are there in Adelaide, Australia? | |
| 950 | formula_1 | simple | ✗ | ✓ | 13 | 6433 | Please list the constructor names with 0 points at race 291. | |
| 959 | formula_1 | simple | ✗ | ✓ | 27 | 6509 | What is the fastest lap number of the champion in 2009? | |
| 971 | formula_1 | simple | ✓ | ✓ | 14 | 6328 | Please state the reference name of the oldest German driver. | |
| 981 | formula_1 | moderate | ✓ | ✓ | 19 | 6555 | On what year did the youngest driver had his first qualifying race? Also state the name, date and time of the race. | |
| 988 | formula_1 | challenging | ✗ | ✓ | 26 | 6435 | List down top 3 German drivers who has the shortest average pit stop duration and were born between 1980-1985. | |
| 989 | formula_1 | moderate | ✗ | ✓ | 24 | 6501 | Who is the champion of the Canadian Grand Prix in 2008? Indicate his finish time. | |
| 990 | formula_1 | challenging | ✓ | ✓ | 31 | 6522 | What is the constructor reference name of the champion in the 2009 Singapore Grand Prix? Please give its website. | |
| 1028 | european_football_2 | challenging | ✗ | ✓ | 186 | 11825 | In Scotland Premier League, which away team won the most during the 2010 season? | |
| 1029 | european_football_2 | moderate | ✗ | ✓ | 25 | 11671 | What are the speed in which attacks are put together of the top 4 teams with the highest build Up Play Speed? | |
| 1030 | european_football_2 | moderate | ✓ | ✓ | 140 | 11669 | Give the name of the league had the most matches end as draw in the 2016 season? | |
| 1035 | european_football_2 | simple | ✓ | ✓ | 15 | 11540 | Give the team_fifa_api_id of teams with more than 50 but less than 60 build-up play speed. | |
| 1036 | european_football_2 | challenging | ✗ | ✓ | execution_failed | 14 | 11348 | List the long name of teams with above-average build-up play passing in 2012. |
| 1037 | european_football_2 | challenging | ✗ | ✓ | 265 | 11786 | Calculate the percentage of players who prefer left foot, who were born between 1987 and 1992. | |
| 1039 | european_football_2 | simple | ✓ | ✓ | 274 | 11583 | Find the average number of long-shot done by Ahmed Samir Farag. | |
| 1042 | european_football_2 | challenging | ✓ | ✓ | 179 | 11823 | List the name of leagues in which the average goals by the home team is higher than the away team in the 2009/2010 seaso | |
| 1057 | european_football_2 | moderate | ✓ | ✓ | 149 | 11669 | Calculate the average home team goal in the 2010/2011 season in the country of Poland. | |
| 1078 | european_football_2 | simple | ✓ | ✓ | 27 | 11421 | Which player is older, Aaron Lennon or Abdelaziz Barrada? | |
| 1088 | european_football_2 | moderate | ✓ | ✓ | 195 | 11690 | Please list the names of the players whose volley score and dribbling score are over 70. | |
| 1094 | european_football_2 | challenging | ✗ | ✓ | execution_failed | 193 | 11469 | How much higher in percentage is Ariel Borysiuk's overall rating than that of Paulin Puel? |
| 1103 | european_football_2 | moderate | ✓ | ✓ | 189 | 11633 | What was the overall rating for Aaron Mooy on 2016/2/4? | |
| 1110 | european_football_2 | moderate | ✓ | ✓ | 25 | 11581 | Tell the build Up play passing class for "FC Lorient" on 2010/2/22. | |
| 1116 | european_football_2 | simple | ✓ | ✓ | 31 | 11494 | List down most tallest players' name. | |
| 1122 | european_football_2 | simple | ✓ | ✓ | 339 | 11468 | State the name of the most strongest player. | |
| 1130 | european_football_2 | moderate | ✓ | ✓ | 22 | 11605 | What are the short name of team who played safe while creating chance of passing? | |
| 1133 | european_football_2 | simple | ✓ | ✓ | 27 | 11464 | How many football players born after the 1990s have the first name "Aaron"? | |
| 1141 | european_football_2 | moderate | ✓ | ✓ | 20 | 11505 | Does the KSV Cercle Brugge team have a slow, balanced or fast speed class? | |
| 1144 | european_football_2 | simple | ✗ | ✓ | 214 | 11606 | Please state the finishing rate and curve score of the player who has the heaviest weight. | |
| 1146 | european_football_2 | moderate | ✓ | ✓ | 145 | 11564 | Please provide the full name of the away team that scored the most goals. | |
| 1147 | european_football_2 | simple | ✓ | ✓ | 256 | 11482 | Please name one player whose overall strength is the greatest. | |
| 1152 | thrombosis_prediction | moderate | ✓ | ✓ | 15 | 4879 | What is the ratio of outpatient to inpatient followed up treatment among all the 'SLE' diagnosed patient? | |
| 1156 | thrombosis_prediction | moderate | ✗ | ✓ | execution_failed | 10 | 4482 | State the ID and age of patient with positive degree of coagulation. |
| 1157 | thrombosis_prediction | simple | ✗ | ✓ | 10 | 4769 | For patients with severe degree of thrombosis, list their ID, sex and disease the patient is diagnosed with. | |
| 1168 | thrombosis_prediction | challenging | ✗ | ✓ | execution_failed | 23 | 4540 | The oldest SJS patient's medical laboratory work was completed on what date, and what age was the patient when they init |
| 1185 | thrombosis_prediction | challenging | ✗ | ✓ | 25 | 5197 | For the patient who was born on 1959/2/18, what is the decrease rate for his/her total cholesterol from November to Dece | |
| 1198 | thrombosis_prediction | simple | ✓ | ✓ | 18 | 4648 | How many female patients were given an APS diagnosis? | |
| 1205 | thrombosis_prediction | moderate | ✗ | ✓ | 15 | 4854 | Was the patient with the number 57266's uric acid within a normal range? | |
| 1208 | thrombosis_prediction | moderate | ✓ | ✓ | 21 | 4841 | Provide IDs for male patients with ALT glutamic pylvic transaminase (GPT) that have history of ALT glutamic pylvic trans | |
| 1220 | thrombosis_prediction | simple | ✓ | ✓ | 23 | 4874 | Provide all ID, sex and birthday of patients whose urea nitrogen (UN) just within the borderline of passing? | |
| 1227 | thrombosis_prediction | moderate | ✗ | ✓ | execution_failed | 12 | 4508 | What is the average age of the male patient with high cholesterol? |
| 1232 | thrombosis_prediction | challenging | ✓ | ✓ | 17 | 4971 | Provide ID, sex and age of patient who has blood glucose (GLU) not within normal range but with total cholesterol(T-CHO) | |
| 1235 | thrombosis_prediction | moderate | ✗ | ✓ | execution_failed | 17 | 4504 | What are the patient's diagnosis for those who has lower red blood blood cell? State their ID and age. |
| 1247 | thrombosis_prediction | challenging | ✗ | ✓ | 19 | 4863 | Among the male patients who have a normal level of white blood cells, how many of them have an abnormal fibrinogen level | |
| 1251 | thrombosis_prediction | simple | ✗ | ✓ | 16 | 4681 | How many patients with an Ig G higher than normal? | |
| 1252 | thrombosis_prediction | moderate | ✗ | ✓ | 12 | 4778 | Among the patients with a normal Ig G level, how many of them have symptoms? | |
| 1254 | thrombosis_prediction | moderate | ✗ | ✓ | execution_failed | 14 | 4528 | How many patients with a normal Ig A level came to the hospital after 1990/1/1? |
| 1255 | thrombosis_prediction | moderate | ✗ | ✓ | 20 | 4758 | For the patients with an abnormal Ig M level, what is the most common disease they are diagnosed with? | |
| 1257 | thrombosis_prediction | challenging | ✗ | ✓ | 17 | 4787 | Among the patients whose creatinine level is abnormal, how many of them aren't 70 yet? | |
| 1275 | thrombosis_prediction | moderate | ✗ | ✓ | execution_failed | 13 | 4532 | Among the patients who has a normal level of anti-centromere and a normal level of anti-SSB, how many of them are male? |
| 1281 | thrombosis_prediction | moderate | ✓ | ✓ | 16 | 4764 | Among the patients who have an abnormal level of glutamic oxaloacetic transaminase, when was the youngest of them born? | |
| 1302 | thrombosis_prediction | challenging | ✗ | ✓ | execution_failed | 15 | 4500 | For the patients with a normal range of creatinine phosphokinase, how many of them have a positive measure of degree of |
| 1312 | student_club | simple | ✓ | ✓ | 11 | 4636 | What's Angela Sanders's major? | |
| 1340 | student_club | moderate | ✗ | ✓ | execution_failed | 13 | 4506 | Calculate the difference of the total amount spent in all events by the Student_Club in year 2019 and 2020. |
| 1344 | student_club | simple | ✓ | ✓ | 11 | 4696 | What was the notes of the fundraising on 2019/9/14? | |
| 1352 | student_club | moderate | ✓ | ✓ | 22 | 4668 | For all the club members from "Business" major, how many of them wear medium size t-shirt? | |
| 1356 | student_club | simple | ✓ | ✓ | 19 | 4602 | Which department was the President of the club in? | |
| 1376 | student_club | moderate | ✓ | ✓ | 18 | 4686 | Among all the closed events, which event has the highest spend-to-budget ratio? | |
| 1378 | student_club | simple | ✓ | ✓ | 16 | 4098 | What is the highest amount of budget spend for an event? | |
| 1380 | student_club | simple | ✓ | ✓ | 15 | 4370 | What is the total amount of money spent for food? | |
| 1387 | student_club | moderate | ✓ | ✓ | 19 | 4755 | Which student has been entrusted to manage the budget for the Yearly Kickoff? | |
| 1390 | student_club | moderate | ✗ | ✓ | 16 | 4250 | Based on the total cost for all event, what is the percentage of cost for Yearly Kickoff event? | |
| 1399 | student_club | moderate | ✗ | ✓ | 13 | 4731 | Did Maya Mclean attend the 'Women's Soccer' event? | |
| 1403 | student_club | moderate | ✓ | ✓ | 20 | 4694 | Indicate the name of the closed event whose cost has exceeded the budget the most. | |
| 1404 | student_club | moderate | ✗ | ✓ | 16 | 4789 | Identify the type of expenses and their total value approved for 'October Meeting' event. | |
| 1409 | student_club | simple | ✓ | ✓ | 9 | 4658 | Mention the total expense used on 8/20/2019. | |
| 1410 | student_club | simple | ✓ | ✓ | 10 | 4707 | List out the full name and total cost that member id "rec4BLdZHS2Blfp4v" incurred? | |
| 1411 | student_club | simple | ✓ | ✓ | 11 | 4690 | State what kind of expenses that Sacha Harrison incurred? | |
| 1422 | student_club | simple | ✗ | ✓ | 11 | 4617 | State the category of events were held at MU 215. | |
| 1464 | student_club | challenging | ✓ | ✓ | 11 | 4749 | Write the full names of students who received funds on the date of 9/9/2019 and include the amount received. | |
| 1472 | debit_card_specializing | moderate | ✗ | ✓ | 214 | 3061 | In 2012, who had the least consumption in LAM? | |
| 1473 | debit_card_specializing | moderate | ✓ | ✓ | 323 | 3125 | What was the average monthly consumption of customers in SME for the year 2013? | |
| 1476 | debit_card_specializing | challenging | ✓ | ✓ | 321 | 3220 | What was the difference in gas consumption between CZK-paying customers and EUR-paying customers in 2012? | |
| 1479 | debit_card_specializing | moderate | ✓ | ✓ | 512 | 2997 | Which year recorded the most consumption of gas paid in CZK? | |
| 1480 | debit_card_specializing | moderate | ✗ | ✓ | 406 | 3089 | What was the gas consumption peak month for SME customers in 2013? | |
| 1484 | debit_card_specializing | simple | ✓ | ✓ | 18 | 3072 | How many more "discount" gas stations does the Czech Republic have compared to Slovakia? | |
| 1486 | debit_card_specializing | simple | ✗ | ✓ | 27 | 3061 | Is it true that more SMEs pay in Czech koruna than in euros? If so, how many more? | |
| 1493 | debit_card_specializing | simple | ✓ | ✓ | 159 | 3132 | In February 2012, what percentage of customers consumed more than 528.3? | |
| 1500 | debit_card_specializing | simple | ✓ | ✓ | 21 | 3105 | Please list the product description of the products consumed in September, 2013. | |
| 1501 | debit_card_specializing | moderate | ✓ | ✓ | 14 | 3086 | Please list the countries of the gas stations with transactions taken place in June, 2013. | |
| 1506 | debit_card_specializing | moderate | ✓ | ✓ | 17 | 3044 | Please list the product descriptions of the transactions taken place in the gas stations in the Czech Republic. | |
| 1515 | debit_card_specializing | simple | ✓ | ✓ | 19 | 3030 | What segment did the customer have at 2012/8/23 21:20:00? | |
| 1521 | debit_card_specializing | moderate | ✓ | ✓ | 11 | 3199 | For all the transactions happened during 8:00-9:00 in 2012/8/26, how many happened in CZE? | |
| 1525 | debit_card_specializing | simple | ✗ | ✓ | 14 | 3091 | What is the percentage of the customers who used EUR in 2012/8/25? | |
| 1526 | debit_card_specializing | challenging | ✗ | ✓ | empty_result | 68 | 3289 | For the customer who paid 634.8 in 2012/8/25, what was the consumption decrease rate from Year 2012 to 2013? |
| 1528 | debit_card_specializing | simple | ✓ | ✓ | 20 | 2962 | What is the percentage of "premium" against the overall segment in Country = "SVK"? | |
| 1529 | debit_card_specializing | moderate | ✗ | ✓ | 50 | 2971 | What is the amount spent by customer "38508" at the gas stations? How much had the customer spent in January 2012? | |
| 1531 | debit_card_specializing | moderate | ✗ | ✓ | 65 | 3074 | Who is the top spending customer and how much is the average price per single item purchased by this customer? What curr |
Model: codestral-latest · n=200 · EA=0.0% · Validity=100.0% · Recall@k=0.0%
| qid | db | diff | match | recall | err | lat ms | tokens | question |
|---|---|---|---|---|---|---|---|---|
| 5 | california_schools | simple | ✗ | ✗ | pipeline_exception | 6878 | 0 | How many schools with an average score in Math greater than 400 in the SAT test are exclusively virtual? |
| 25 | california_schools | moderate | ✗ | ✗ | pipeline_exception | 5880 | 0 | Name schools in Riverside which the average of average math score for SAT is grater than 400, what is the funding type o |
| 32 | california_schools | moderate | ✗ | ✗ | pipeline_exception | 21415 | 0 | What is the eligible free or reduced price meal rate for the top 5 schools in grades 1-12 with the highest free or reduc |
| 36 | california_schools | challenging | ✗ | ✗ | pipeline_exception | 29948 | 0 | Under whose administration is the school with the highest number of students scoring 1500 or more on the SAT? Indicate t |
| 37 | california_schools | moderate | ✗ | ✗ | pipeline_exception | 29970 | 0 | What is the complete address of the school with the lowest excellence rate? Indicate the Street, City, Zip and State. |
| 39 | california_schools | simple | ✗ | ✗ | pipeline_exception | 29719 | 0 | What is the average number of test takers from Fresno schools that opened between 1/1/1980 and 12/31/1980? |
| 48 | california_schools | moderate | ✗ | ✗ | pipeline_exception | 29692 | 0 | What is the ratio of merged Unified School District schools in Orange County to merged Elementary School District school |
| 50 | california_schools | simple | ✗ | ✗ | pipeline_exception | 30004 | 0 | What is the postal street address for the school with the 7th highest Math average? Indicate the school's name. |
| 77 | california_schools | moderate | ✗ | ✗ | pipeline_exception | 30086 | 0 | Which schools served a grade span of Kindergarten to 9th grade in the county of Los Angeles and what is its Percent (%) |
| 92 | financial | simple | ✗ | ✗ | pipeline_exception | 29848 | 0 | List out the no. of districts that have female average salary is more than 6000 but less than 10000? |
| 98 | financial | moderate | ✗ | ✗ | pipeline_exception | 29787 | 0 | Among the accounts who have approved loan date in 1997, list out the accounts that have the lowest approved amount and c |
| 99 | financial | moderate | ✗ | ✗ | pipeline_exception | 29663 | 0 | Among the accounts who have loan validity more than 12 months, list out the accounts that have the highest approved amou |
| 112 | financial | simple | ✗ | ✗ | pipeline_exception | 29685 | 0 | For the female client who was born in 1976/1/29, which district did she opened her account? |
| 115 | financial | challenging | ✗ | ✗ | pipeline_exception | 29736 | 0 | For the branch which located in the south Bohemia with biggest number of inhabitants, what is the percentage of the male |
| 118 | financial | moderate | ✗ | ✗ | pipeline_exception | 29624 | 0 | For loan amount less than USD100,000, what is the percentage of accounts that is still running with no issue. |
| 120 | financial | moderate | ✗ | ✗ | pipeline_exception | 29655 | 0 | From Year 1995 to 2000, who are the accounts holders from 'east Bohemia'. State the account ID the frequency of statemen |
| 125 | financial | challenging | ✗ | ✗ | pipeline_exception | 29887 | 0 | For loans contracts which are still running where client are in debt, list the district of the and the state the percent |
| 138 | financial | moderate | ✗ | ✗ | pipeline_exception | 29762 | 0 | In the branch where the second-highest number of crimes were committed in 1995 occurred, how many male clients are there |
| 159 | financial | simple | ✗ | ✗ | pipeline_exception | 29918 | 0 | List all the withdrawals in cash transactions that the client with the id 3356 makes. |
| 168 | financial | moderate | ✗ | ✗ | pipeline_exception | 29689 | 0 | What percentage of clients who opened their accounts in the district with an average salary of over 10000 are women? |
| 169 | financial | challenging | ✗ | ✗ | pipeline_exception | 29819 | 0 | What was the growth rate of the total amount of loans across all accounts for a male client between 1996 and 1997? |
| 173 | financial | challenging | ✗ | ✗ | pipeline_exception | 29817 | 0 | How often does account number 3 request an account statement to be released? What was the aim of debiting 3539 in total? |
| 189 | financial | moderate | ✗ | ✗ | pipeline_exception | 29509 | 0 | Name the account numbers of female clients who are oldest and have lowest average salary? |
| 192 | financial | moderate | ✗ | ✗ | pipeline_exception | 30036 | 0 | What is the average amount of loan which are still on running contract with statement issuance after each transaction? |
| 194 | financial | moderate | ✗ | ✗ | pipeline_exception | 29971 | 0 | Provide the IDs and age of the client with high level credit card, which is eligible for loans. |
| 207 | toxicology | challenging | ✗ | ✗ | pipeline_exception | 29785 | 0 | What elements are in a double type bond? |
| 208 | toxicology | moderate | ✗ | ✗ | pipeline_exception | 29825 | 0 | Which type of label is the most numerous in atoms with hydrogen? |
| 219 | toxicology | challenging | ✗ | ✗ | pipeline_exception | 29693 | 0 | What is the percentage of carcinogenic molecules in triple type bonds? |
| 227 | toxicology | simple | ✗ | ✗ | pipeline_exception | 30053 | 0 | What is the percentage of molecules that are carcinogenic? Please provide your answer as a percentage with three decimal |
| 230 | toxicology | challenging | ✗ | ✗ | pipeline_exception | 29531 | 0 | What are the elements of the toxicology and label of molecule TR060? |
| 232 | toxicology | moderate | ✗ | ✗ | pipeline_exception | 30066 | 0 | Please list top three molecules that have single bonds between two atoms and are not carcinogenic in alphabetical order. |
| 236 | toxicology | moderate | ✗ | ✗ | pipeline_exception | 29745 | 0 | What are the bond type and the atoms of the bond ID of TR001_6_9? |
| 239 | toxicology | simple | ✗ | ✗ | pipeline_exception | 29629 | 0 | How many connections does the atom 19 have? |
| 253 | toxicology | challenging | ✗ | ✗ | pipeline_exception | 29667 | 0 | List the elements of all the triple bonds. |
| 260 | toxicology | moderate | ✗ | ✗ | pipeline_exception | 29654 | 0 | Calculate the total atoms with triple-bond molecules containing the element phosphorus or bromine. |
| 268 | toxicology | challenging | ✗ | ✗ | pipeline_exception | 29790 | 0 | What are the elements for bond id TR001_10_11? |
| 273 | toxicology | moderate | ✗ | ✗ | pipeline_exception | 29825 | 0 | What is the percentage of element chlorine in carcinogenic molecules? |
| 282 | toxicology | challenging | ✗ | ✗ | pipeline_exception | 29488 | 0 | What is the ratio of Hydrogen elements in molecule ID TR006? List the ratio with its label. |
| 327 | toxicology | moderate | ✗ | ✗ | pipeline_exception | 30043 | 0 | Which non-carcinogenic molecules consisted more than 5 atoms? |
| 347 | card_games | moderate | ✗ | ✗ | pipeline_exception | 29786 | 0 | Find all cards illustrated by Stephen Daniel and describe the text of the ruling of these cards. State if these cards ha |
| 349 | card_games | moderate | ✗ | ✗ | pipeline_exception | 29753 | 0 | Name the card and artist with the most ruling information. Also state if the card is a promotional printing. |
| 352 | card_games | moderate | ✗ | ✗ | pipeline_exception | 29843 | 0 | Calculate the percentage of the cards availabe in Chinese Simplified. |
| 356 | card_games | simple | ✗ | ✗ | pipeline_exception | 29858 | 0 | How many cards have infinite power? |
| 358 | card_games | simple | ✗ | ✗ | pipeline_exception | 29830 | 0 | What is the border color of card "Ancestor's Chosen"? |
| 366 | card_games | simple | ✗ | ✗ | pipeline_exception | 29798 | 0 | What is the rule of playing card "Benalish Knight"? |
| 377 | card_games | simple | ✗ | ✗ | pipeline_exception | 29617 | 0 | How many cards with original type of "Summon - Angel" have subtype other than "Angel"? |
| 391 | card_games | moderate | ✗ | ✗ | pipeline_exception | 29854 | 0 | Among the Artifact cards, which are black color and comes with foreign languague translation? |
| 407 | card_games | moderate | ✗ | ✗ | pipeline_exception | 29821 | 0 | Lists all types of cards in German. |
| 408 | card_games | moderate | ✗ | ✗ | pipeline_exception | 29931 | 0 | How many unknown power cards contain info about the triggered ability |
| 412 | card_games | moderate | ✗ | ✗ | pipeline_exception | 29790 | 0 | What is the foreign name of the card in French of type Creature, normal layout and black border color, by artist Matthew |
| 414 | card_games | simple | ✗ | ✗ | pipeline_exception | 29824 | 0 | What language is the set of 180 cards that belongs to the Ravnica block translated into? |
| 427 | card_games | moderate | ✗ | ✗ | pipeline_exception | 29517 | 0 | What languages are available in the set known as Archenemy on the magic card market and having the code ARC? |
| 459 | card_games | moderate | ✗ | ✗ | pipeline_exception | 29723 | 0 | Which card costs more converted mana, "Serra Angel" or "Shrine Keeper"? |
| 466 | card_games | moderate | ✗ | ✗ | pipeline_exception | 29720 | 0 | Among the cards in the set "Hauptset Zehnte Edition", how many of them are designed by Adam Rex? |
| 472 | card_games | moderate | ✗ | ✗ | pipeline_exception | 29561 | 0 | Among the sets in the block "Ice Age", how many of them have an Italian translation? |
| 484 | card_games | moderate | ✗ | ✗ | pipeline_exception | 29776 | 0 | Please list the Italian names of the cards in the set Coldsnap with the highest converted mana cost. |
| 486 | card_games | moderate | ✗ | ✗ | pipeline_exception | 29826 | 0 | What is the percentage of the cards with a converted mana cost of 7 in the set Coldsnap? |
| 518 | card_games | moderate | ✗ | ✗ | pipeline_exception | 29683 | 0 | Which of the play format has the highest number of banned status? Indicate the play format and the names of all the card |
| 531 | codebase_community | simple | ✗ | ✗ | pipeline_exception | 29654 | 0 | Which user has a higher reputation, Harlan or Jarrod Dixon? |
| 557 | codebase_community | moderate | ✗ | ✗ | pipeline_exception | 29936 | 0 | Among the posts with a score of over 5, what is the percentage of them being owned by an elder user? |
| 563 | codebase_community | moderate | ✗ | ✗ | pipeline_exception | 29944 | 0 | User No.3025 gave a comment at 20:29:39 on 2014/4/23 to a post, how many favorite counts did that post get? |
| 571 | codebase_community | moderate | ✗ | ✗ | pipeline_exception | 29878 | 0 | For the user No.24, how many times is the number of his/her posts compared to his/her votes? |
| 584 | codebase_community | moderate | ✗ | ✗ | pipeline_exception | 29798 | 0 | Write all the comments left by users who edited the post titled 'Why square the difference instead of taking the absolut |
| 595 | codebase_community | moderate | ✗ | ✗ | pipeline_exception | 29815 | 0 | Which user have only one post history per post and having at least 1000 views? |
| 634 | codebase_community | challenging | ✗ | ✗ | pipeline_exception | 29549 | 0 | Among posts by Harvey Motulsky and Noah Snyder, which one has higher popularity? |
| 669 | codebase_community | simple | ✗ | ✗ | pipeline_exception | 29817 | 0 | When did 'chl' cast its first vote in a post? |
| 671 | codebase_community | simple | ✗ | ✗ | pipeline_exception | 29867 | 0 | What is the display name of the user who acquired the first Autobiographer badge? |
| 672 | codebase_community | moderate | ✗ | ✗ | pipeline_exception | 29833 | 0 | Among the users located in United Kingdom, how many users whose post have a total favorite amount of 4 or more? |
| 694 | codebase_community | moderate | ✗ | ✗ | pipeline_exception | 29741 | 0 | Provide the text of the latest 10 comments to the post with the title 'Analysing wind data with R' and the display name |
| 707 | codebase_community | moderate | ✗ | ✗ | pipeline_exception | 29750 | 0 | Among the posts with views ranging from 100 to 150, what is the comment with the highest score? |
| 716 | codebase_community | moderate | ✗ | ✗ | pipeline_exception | 29814 | 0 | Among the comments with scores between 5 to 10, what is the percentage of the users with 0 up votes? |
| 723 | superhero | moderate | ✗ | ✗ | pipeline_exception | 29871 | 0 | Among the superheroes with blue eyes, how many of them have the super power of "Agility"? |
| 730 | superhero | challenging | ✗ | ✗ | pipeline_exception | 30333 | 0 | List the superheroes from Marvel Comics who have the super power of 'Super Strength'. |
| 736 | superhero | moderate | ✗ | ✗ | pipeline_exception | 29645 | 0 | Who is the dumbest superhero? |
| 737 | superhero | simple | ✗ | ✗ | pipeline_exception | 29631 | 0 | What is Copycat's race? |
| 738 | superhero | simple | ✗ | ✗ | pipeline_exception | 29754 | 0 | Which superheroes have a durability attribute value of less than 50? |
| 743 | superhero | challenging | ✗ | ✗ | pipeline_exception | 29822 | 0 | What is the percentage of superheroes who act in their own self-interest or make decisions based on their own moral code |
| 747 | superhero | simple | ✗ | ✗ | pipeline_exception | 29760 | 0 | What is the total number of superheroes without full name? |
| 750 | superhero | simple | ✗ | ✗ | pipeline_exception | 29962 | 0 | What is the average weight of all female superheroes? |
| 751 | superhero | moderate | ✗ | ✗ | pipeline_exception | 29933 | 0 | List down at least five superpowers of male superheroes. |
| 753 | superhero | moderate | ✗ | ✗ | pipeline_exception | 29908 | 0 | Among the superheroes with height from 170 to 190, list the names of the superheroes with no eye color. |
| 765 | superhero | simple | ✗ | ✗ | pipeline_exception | 30003 | 0 | How many heroes have stealth power? |
| 773 | superhero | challenging | ✗ | ✗ | pipeline_exception | 29538 | 0 | Which superhero has the same eyes, hair and skin colour? Indicate the publisher of the superhero. |
| 775 | superhero | challenging | ✗ | ✗ | pipeline_exception | 29838 | 0 | What is the percentage of blue female superheroes among all female superheroes? |
| 781 | superhero | simple | ✗ | ✗ | pipeline_exception | 29593 | 0 | Provide the heights of the heroes whose eye colours are amber. |
| 785 | superhero | simple | ✗ | ✗ | pipeline_exception | 30044 | 0 | Describe the names of neutral alignment superheroes. |
| 791 | superhero | simple | ✗ | ✗ | pipeline_exception | 29541 | 0 | Calculate the average height for all superhero. |
| 794 | superhero | moderate | ✗ | ✗ | pipeline_exception | 29691 | 0 | Which hero was the fastest? |
| 798 | superhero | moderate | ✗ | ✗ | pipeline_exception | 29942 | 0 | What is the publisher for Hawkman, Karate Kid and Speedy? |
| 800 | superhero | moderate | ✗ | ✗ | pipeline_exception | 29716 | 0 | Calculate the percentage of superheroes with blue eyes. |
| 806 | superhero | simple | ✗ | ✗ | pipeline_exception | 29849 | 0 | Provide the eye colour of the superhero who has Karen Beecher-Duncan as their full name. |
| 819 | superhero | challenging | ✗ | ✗ | pipeline_exception | 29687 | 0 | In superheroes with missing weight data, calculate the difference between the number of superheroes with blue eyes and n |
| 825 | superhero | moderate | ✗ | ✗ | pipeline_exception | 29946 | 0 | Identify the gender of the superhero who has the ability of Phoenix Force. |
| 847 | formula_1 | simple | ✗ | ✗ | pipeline_exception | 29880 | 0 | What is the surname of the driver with the best lap time in race number 19 in the second qualifying period? |
| 859 | formula_1 | simple | ✗ | ✗ | pipeline_exception | 29876 | 0 | What's Bruno Senna's Q1 result in the qualifying race No. 354? |
| 861 | formula_1 | simple | ✗ | ✗ | pipeline_exception | 29544 | 0 | What is his number of the driver who finished 0:01:54 in the Q3 of qualifying race No.903? |
| 862 | formula_1 | simple | ✗ | ✗ | pipeline_exception | 29994 | 0 | For the Bahrain Grand Prix in 2007, how many drivers not finished the game? |
| 865 | formula_1 | moderate | ✗ | ✗ | pipeline_exception | 29968 | 0 | For all the drivers who finished the game in race No. 592, who is the oldest? |
| 866 | formula_1 | moderate | ✗ | ✗ | pipeline_exception | 30107 | 0 | Who was the player that got the lap time of 0:01:27 in the race No. 161? Show his introduction website. |
| 875 | formula_1 | simple | ✗ | ✗ | pipeline_exception | 30203 | 0 | Show me the season page of year when the race No. 901 took place. |
| 877 | formula_1 | moderate | ✗ | ✗ | pipeline_exception | 29914 | 0 | For all the drivers who finished the game in race No. 872, who is the youngest? |
| 879 | formula_1 | moderate | ✗ | ✗ | pipeline_exception | 29836 | 0 | For the driver who set the fastest lap speed, what is his nationality? |
| 881 | formula_1 | moderate | ✗ | ✗ | pipeline_exception | 29819 | 0 | For the drivers who took part in the race in 1983/7/16, what's their race completion rate? |
| 894 | formula_1 | moderate | ✗ | ✗ | pipeline_exception | 29873 | 0 | What is the best lap time recorded? List the driver and race with such recorded lap time. |
| 896 | formula_1 | challenging | ✗ | ✗ | pipeline_exception | 29683 | 0 | Calculate the percentage whereby Hamilton was not at the 1st track of the the f1 circuit since 2010. |
| 897 | formula_1 | moderate | ✗ | ✗ | pipeline_exception | 29376 | 0 | Name the driver with the most winning. Mention his nationality and what is his maximum point scores. |
| 898 | formula_1 | simple | ✗ | ✗ | pipeline_exception | 29822 | 0 | How old is the youngest Japanese driver? What is his name? |
| 902 | formula_1 | simple | ✗ | ✗ | pipeline_exception | 30072 | 0 | Which race was Alex Yoong in when he was in track number less than 20? |
| 904 | formula_1 | moderate | ✗ | ✗ | pipeline_exception | 29874 | 0 | State the race and year of race in which Michael Schumacher had his fastest lap. |
| 909 | formula_1 | moderate | ✗ | ✗ | pipeline_exception | 29780 | 0 | Among all European Grand Prix races, what is the percentage of the races were hosted in Germany? |
| 912 | formula_1 | simple | ✗ | ✗ | pipeline_exception | 29867 | 0 | What's the reference name of Marina Bay Street Circuit? |
| 915 | formula_1 | simple | ✗ | ✗ | pipeline_exception | 29562 | 0 | Which country is the oldest driver from? |
| 930 | formula_1 | simple | ✗ | ✗ | pipeline_exception | 29507 | 0 | In which Formula_1 race did Lewis Hamilton rank the highest? |
| 945 | formula_1 | simple | ✗ | ✗ | pipeline_exception | 29717 | 0 | How many circuits are there in Adelaide, Australia? |
| 950 | formula_1 | simple | ✗ | ✗ | pipeline_exception | 30000 | 0 | Please list the constructor names with 0 points at race 291. |
| 959 | formula_1 | simple | ✗ | ✗ | pipeline_exception | 30130 | 0 | What is the fastest lap number of the champion in 2009? |
| 971 | formula_1 | simple | ✗ | ✗ | pipeline_exception | 29900 | 0 | Please state the reference name of the oldest German driver. |
| 981 | formula_1 | moderate | ✗ | ✗ | pipeline_exception | 29958 | 0 | On what year did the youngest driver had his first qualifying race? Also state the name, date and time of the race. |
| 988 | formula_1 | challenging | ✗ | ✗ | pipeline_exception | 29869 | 0 | List down top 3 German drivers who has the shortest average pit stop duration and were born between 1980-1985. |
| 989 | formula_1 | moderate | ✗ | ✗ | pipeline_exception | 29589 | 0 | Who is the champion of the Canadian Grand Prix in 2008? Indicate his finish time. |
| 990 | formula_1 | challenging | ✗ | ✗ | pipeline_exception | 29996 | 0 | What is the constructor reference name of the champion in the 2009 Singapore Grand Prix? Please give its website. |
| 1028 | european_football_2 | challenging | ✗ | ✗ | pipeline_exception | 29987 | 0 | In Scotland Premier League, which away team won the most during the 2010 season? |
| 1029 | european_football_2 | moderate | ✗ | ✗ | pipeline_exception | 29806 | 0 | What are the speed in which attacks are put together of the top 4 teams with the highest build Up Play Speed? |
| 1030 | european_football_2 | moderate | ✗ | ✗ | pipeline_exception | 30158 | 0 | Give the name of the league had the most matches end as draw in the 2016 season? |
| 1035 | european_football_2 | simple | ✗ | ✗ | pipeline_exception | 29773 | 0 | Give the team_fifa_api_id of teams with more than 50 but less than 60 build-up play speed. |
| 1036 | european_football_2 | challenging | ✗ | ✗ | pipeline_exception | 29913 | 0 | List the long name of teams with above-average build-up play passing in 2012. |
| 1037 | european_football_2 | challenging | ✗ | ✗ | pipeline_exception | 29522 | 0 | Calculate the percentage of players who prefer left foot, who were born between 1987 and 1992. |
| 1039 | european_football_2 | simple | ✗ | ✗ | pipeline_exception | 29536 | 0 | Find the average number of long-shot done by Ahmed Samir Farag. |
| 1042 | european_football_2 | challenging | ✗ | ✗ | pipeline_exception | 29582 | 0 | List the name of leagues in which the average goals by the home team is higher than the away team in the 2009/2010 seaso |
| 1057 | european_football_2 | moderate | ✗ | ✗ | pipeline_exception | 29720 | 0 | Calculate the average home team goal in the 2010/2011 season in the country of Poland. |
| 1078 | european_football_2 | simple | ✗ | ✗ | pipeline_exception | 29954 | 0 | Which player is older, Aaron Lennon or Abdelaziz Barrada? |
| 1088 | european_football_2 | moderate | ✗ | ✗ | pipeline_exception | 29990 | 0 | Please list the names of the players whose volley score and dribbling score are over 70. |
| 1094 | european_football_2 | challenging | ✗ | ✗ | pipeline_exception | 29750 | 0 | How much higher in percentage is Ariel Borysiuk's overall rating than that of Paulin Puel? |
| 1103 | european_football_2 | moderate | ✗ | ✗ | pipeline_exception | 29921 | 0 | What was the overall rating for Aaron Mooy on 2016/2/4? |
| 1110 | european_football_2 | moderate | ✗ | ✗ | pipeline_exception | 29720 | 0 | Tell the build Up play passing class for "FC Lorient" on 2010/2/22. |
| 1116 | european_football_2 | simple | ✗ | ✗ | pipeline_exception | 29824 | 0 | List down most tallest players' name. |
| 1122 | european_football_2 | simple | ✗ | ✗ | pipeline_exception | 30133 | 0 | State the name of the most strongest player. |
| 1130 | european_football_2 | moderate | ✗ | ✗ | pipeline_exception | 29689 | 0 | What are the short name of team who played safe while creating chance of passing? |
| 1133 | european_football_2 | simple | ✗ | ✗ | pipeline_exception | 29801 | 0 | How many football players born after the 1990s have the first name "Aaron"? |
| 1141 | european_football_2 | moderate | ✗ | ✗ | pipeline_exception | 30071 | 0 | Does the KSV Cercle Brugge team have a slow, balanced or fast speed class? |
| 1144 | european_football_2 | simple | ✗ | ✗ | pipeline_exception | 29941 | 0 | Please state the finishing rate and curve score of the player who has the heaviest weight. |
| 1146 | european_football_2 | moderate | ✗ | ✗ | pipeline_exception | 29852 | 0 | Please provide the full name of the away team that scored the most goals. |
| 1147 | european_football_2 | simple | ✗ | ✗ | pipeline_exception | 29823 | 0 | Please name one player whose overall strength is the greatest. |
| 1152 | thrombosis_prediction | moderate | ✗ | ✗ | pipeline_exception | 29986 | 0 | What is the ratio of outpatient to inpatient followed up treatment among all the 'SLE' diagnosed patient? |
| 1156 | thrombosis_prediction | moderate | ✗ | ✗ | pipeline_exception | 29679 | 0 | State the ID and age of patient with positive degree of coagulation. |
| 1157 | thrombosis_prediction | simple | ✗ | ✗ | pipeline_exception | 29824 | 0 | For patients with severe degree of thrombosis, list their ID, sex and disease the patient is diagnosed with. |
| 1168 | thrombosis_prediction | challenging | ✗ | ✗ | pipeline_exception | 29926 | 0 | The oldest SJS patient's medical laboratory work was completed on what date, and what age was the patient when they init |
| 1185 | thrombosis_prediction | challenging | ✗ | ✗ | pipeline_exception | 29703 | 0 | For the patient who was born on 1959/2/18, what is the decrease rate for his/her total cholesterol from November to Dece |
| 1198 | thrombosis_prediction | simple | ✗ | ✗ | pipeline_exception | 29761 | 0 | How many female patients were given an APS diagnosis? |
| 1205 | thrombosis_prediction | moderate | ✗ | ✗ | pipeline_exception | 29702 | 0 | Was the patient with the number 57266's uric acid within a normal range? |
| 1208 | thrombosis_prediction | moderate | ✗ | ✗ | pipeline_exception | 29680 | 0 | Provide IDs for male patients with ALT glutamic pylvic transaminase (GPT) that have history of ALT glutamic pylvic trans |
| 1220 | thrombosis_prediction | simple | ✗ | ✗ | pipeline_exception | 30030 | 0 | Provide all ID, sex and birthday of patients whose urea nitrogen (UN) just within the borderline of passing? |
| 1227 | thrombosis_prediction | moderate | ✗ | ✗ | pipeline_exception | 29908 | 0 | What is the average age of the male patient with high cholesterol? |
| 1232 | thrombosis_prediction | challenging | ✗ | ✗ | pipeline_exception | 30078 | 0 | Provide ID, sex and age of patient who has blood glucose (GLU) not within normal range but with total cholesterol(T-CHO) |
| 1235 | thrombosis_prediction | moderate | ✗ | ✗ | pipeline_exception | 29778 | 0 | What are the patient's diagnosis for those who has lower red blood blood cell? State their ID and age. |
| 1247 | thrombosis_prediction | challenging | ✗ | ✗ | pipeline_exception | 29705 | 0 | Among the male patients who have a normal level of white blood cells, how many of them have an abnormal fibrinogen level |
| 1251 | thrombosis_prediction | simple | ✗ | ✗ | pipeline_exception | 29864 | 0 | How many patients with an Ig G higher than normal? |
| 1252 | thrombosis_prediction | moderate | ✗ | ✗ | pipeline_exception | 29767 | 0 | Among the patients with a normal Ig G level, how many of them have symptoms? |
| 1254 | thrombosis_prediction | moderate | ✗ | ✗ | pipeline_exception | 29809 | 0 | How many patients with a normal Ig A level came to the hospital after 1990/1/1? |
| 1255 | thrombosis_prediction | moderate | ✗ | ✗ | pipeline_exception | 29891 | 0 | For the patients with an abnormal Ig M level, what is the most common disease they are diagnosed with? |
| 1257 | thrombosis_prediction | challenging | ✗ | ✗ | pipeline_exception | 29478 | 0 | Among the patients whose creatinine level is abnormal, how many of them aren't 70 yet? |
| 1275 | thrombosis_prediction | moderate | ✗ | ✗ | pipeline_exception | 29755 | 0 | Among the patients who has a normal level of anti-centromere and a normal level of anti-SSB, how many of them are male? |
| 1281 | thrombosis_prediction | moderate | ✗ | ✗ | pipeline_exception | 30109 | 0 | Among the patients who have an abnormal level of glutamic oxaloacetic transaminase, when was the youngest of them born? |
| 1302 | thrombosis_prediction | challenging | ✗ | ✗ | pipeline_exception | 29845 | 0 | For the patients with a normal range of creatinine phosphokinase, how many of them have a positive measure of degree of |
| 1312 | student_club | simple | ✗ | ✗ | pipeline_exception | 30027 | 0 | What's Angela Sanders's major? |
| 1340 | student_club | moderate | ✗ | ✗ | pipeline_exception | 29664 | 0 | Calculate the difference of the total amount spent in all events by the Student_Club in year 2019 and 2020. |
| 1344 | student_club | simple | ✗ | ✗ | pipeline_exception | 29746 | 0 | What was the notes of the fundraising on 2019/9/14? |
| 1352 | student_club | moderate | ✗ | ✗ | pipeline_exception | 29844 | 0 | For all the club members from "Business" major, how many of them wear medium size t-shirt? |
| 1356 | student_club | simple | ✗ | ✗ | pipeline_exception | 29732 | 0 | Which department was the President of the club in? |
| 1376 | student_club | moderate | ✗ | ✗ | pipeline_exception | 29953 | 0 | Among all the closed events, which event has the highest spend-to-budget ratio? |
| 1378 | student_club | simple | ✗ | ✗ | pipeline_exception | 29751 | 0 | What is the highest amount of budget spend for an event? |
| 1380 | student_club | simple | ✗ | ✗ | pipeline_exception | 29601 | 0 | What is the total amount of money spent for food? |
| 1387 | student_club | moderate | ✗ | ✗ | pipeline_exception | 30032 | 0 | Which student has been entrusted to manage the budget for the Yearly Kickoff? |
| 1390 | student_club | moderate | ✗ | ✗ | pipeline_exception | 30182 | 0 | Based on the total cost for all event, what is the percentage of cost for Yearly Kickoff event? |
| 1399 | student_club | moderate | ✗ | ✗ | pipeline_exception | 29929 | 0 | Did Maya Mclean attend the 'Women's Soccer' event? |
| 1403 | student_club | moderate | ✗ | ✗ | pipeline_exception | 30162 | 0 | Indicate the name of the closed event whose cost has exceeded the budget the most. |
| 1404 | student_club | moderate | ✗ | ✗ | pipeline_exception | 29917 | 0 | Identify the type of expenses and their total value approved for 'October Meeting' event. |
| 1409 | student_club | simple | ✗ | ✗ | pipeline_exception | 29959 | 0 | Mention the total expense used on 8/20/2019. |
| 1410 | student_club | simple | ✗ | ✗ | pipeline_exception | 29682 | 0 | List out the full name and total cost that member id "rec4BLdZHS2Blfp4v" incurred? |
| 1411 | student_club | simple | ✗ | ✗ | pipeline_exception | 30329 | 0 | State what kind of expenses that Sacha Harrison incurred? |
| 1422 | student_club | simple | ✗ | ✗ | pipeline_exception | 30113 | 0 | State the category of events were held at MU 215. |
| 1464 | student_club | challenging | ✗ | ✗ | pipeline_exception | 29834 | 0 | Write the full names of students who received funds on the date of 9/9/2019 and include the amount received. |
| 1472 | debit_card_specializing | moderate | ✗ | ✗ | pipeline_exception | 29867 | 0 | In 2012, who had the least consumption in LAM? |
| 1473 | debit_card_specializing | moderate | ✗ | ✗ | pipeline_exception | 29906 | 0 | What was the average monthly consumption of customers in SME for the year 2013? |
| 1476 | debit_card_specializing | challenging | ✗ | ✗ | pipeline_exception | 29476 | 0 | What was the difference in gas consumption between CZK-paying customers and EUR-paying customers in 2012? |
| 1479 | debit_card_specializing | moderate | ✗ | ✗ | pipeline_exception | 29576 | 0 | Which year recorded the most consumption of gas paid in CZK? |
| 1480 | debit_card_specializing | moderate | ✗ | ✗ | pipeline_exception | 29892 | 0 | What was the gas consumption peak month for SME customers in 2013? |
| 1484 | debit_card_specializing | simple | ✗ | ✗ | pipeline_exception | 29690 | 0 | How many more "discount" gas stations does the Czech Republic have compared to Slovakia? |
| 1486 | debit_card_specializing | simple | ✗ | ✗ | pipeline_exception | 29671 | 0 | Is it true that more SMEs pay in Czech koruna than in euros? If so, how many more? |
| 1493 | debit_card_specializing | simple | ✗ | ✗ | pipeline_exception | 29850 | 0 | In February 2012, what percentage of customers consumed more than 528.3? |
| 1500 | debit_card_specializing | simple | ✗ | ✗ | pipeline_exception | 29876 | 0 | Please list the product description of the products consumed in September, 2013. |
| 1501 | debit_card_specializing | moderate | ✗ | ✗ | pipeline_exception | 29330 | 0 | Please list the countries of the gas stations with transactions taken place in June, 2013. |
| 1506 | debit_card_specializing | moderate | ✗ | ✗ | pipeline_exception | 29626 | 0 | Please list the product descriptions of the transactions taken place in the gas stations in the Czech Republic. |
| 1515 | debit_card_specializing | simple | ✗ | ✗ | pipeline_exception | 29678 | 0 | What segment did the customer have at 2012/8/23 21:20:00? |
| 1521 | debit_card_specializing | moderate | ✗ | ✗ | pipeline_exception | 29953 | 0 | For all the transactions happened during 8:00-9:00 in 2012/8/26, how many happened in CZE? |
| 1525 | debit_card_specializing | simple | ✗ | ✗ | pipeline_exception | 30037 | 0 | What is the percentage of the customers who used EUR in 2012/8/25? |
| 1526 | debit_card_specializing | challenging | ✗ | ✗ | pipeline_exception | 29722 | 0 | For the customer who paid 634.8 in 2012/8/25, what was the consumption decrease rate from Year 2012 to 2013? |
| 1528 | debit_card_specializing | simple | ✗ | ✗ | pipeline_exception | 29500 | 0 | What is the percentage of "premium" against the overall segment in Country = "SVK"? |
| 1529 | debit_card_specializing | moderate | ✗ | ✗ | pipeline_exception | 29710 | 0 | What is the amount spent by customer "38508" at the gas stations? How much had the customer spent in January 2012? |
| 1531 | debit_card_specializing | moderate | ✗ | ✗ | pipeline_exception | 29937 | 0 | Who is the top spending customer and how much is the average price per single item purchased by this customer? What curr |
Model: codestral-latest · n=200 · EA=60.0% · Validity=100.0% · Recall@k=100.0%
| qid | db | diff | match | recall | err | lat ms | tokens | question |
|---|---|---|---|---|---|---|---|---|
| 5 | california_schools | simple | ✓ | ✓ | 285 | 25146 | How many schools with an average score in Math greater than 400 in the SAT test are exclusively virtual? | |
| 25 | california_schools | moderate | ✗ | ✓ | 158 | 25609 | Name schools in Riverside which the average of average math score for SAT is grater than 400, what is the funding type o | |
| 32 | california_schools | moderate | ✓ | ✓ | 109 | 26026 | What is the eligible free or reduced price meal rate for the top 5 schools in grades 1-12 with the highest free or reduc | |
| 36 | california_schools | challenging | ✓ | ✓ | 77 | 25890 | Under whose administration is the school with the highest number of students scoring 1500 or more on the SAT? Indicate t | |
| 37 | california_schools | moderate | ✗ | ✓ | 107 | 25966 | What is the complete address of the school with the lowest excellence rate? Indicate the Street, City, Zip and State. | |
| 39 | california_schools | simple | ✓ | ✓ | 173 | 25966 | What is the average number of test takers from Fresno schools that opened between 1/1/1980 and 12/31/1980? | |
| 48 | california_schools | moderate | ✓ | ✓ | 8048 | 25777 | What is the ratio of merged Unified School District schools in Orange County to merged Elementary School District school | |
| 50 | california_schools | simple | ✗ | ✓ | 8147 | 25226 | What is the postal street address for the school with the 7th highest Math average? Indicate the school's name. | |
| 77 | california_schools | moderate | ✗ | ✓ | empty_result | 6745 | 25588 | Which schools served a grade span of Kindergarten to 9th grade in the county of Los Angeles and what is its Percent (%) |
| 92 | financial | simple | ✓ | ✓ | 6809 | 17880 | List out the no. of districts that have female average salary is more than 6000 but less than 10000? | |
| 98 | financial | moderate | ✓ | ✓ | 6672 | 17987 | Among the accounts who have approved loan date in 1997, list out the accounts that have the lowest approved amount and c | |
| 99 | financial | moderate | ✓ | ✓ | 7132 | 17956 | Among the accounts who have loan validity more than 12 months, list out the accounts that have the highest approved amou | |
| 112 | financial | simple | ✓ | ✓ | 7201 | 17964 | For the female client who was born in 1976/1/29, which district did she opened her account? | |
| 115 | financial | challenging | ✗ | ✓ | 15884 | 17929 | For the branch which located in the south Bohemia with biggest number of inhabitants, what is the percentage of the male | |
| 118 | financial | moderate | ✓ | ✓ | 9954 | 18079 | For loan amount less than USD100,000, what is the percentage of accounts that is still running with no issue. | |
| 120 | financial | moderate | ✓ | ✓ | 7921 | 19284 | From Year 1995 to 2000, who are the accounts holders from 'east Bohemia'. State the account ID the frequency of statemen | |
| 125 | financial | challenging | ✗ | ✓ | 11554 | 19486 | For loans contracts which are still running where client are in debt, list the district of the and the state the percent | |
| 138 | financial | moderate | ✓ | ✓ | 7720 | 17925 | In the branch where the second-highest number of crimes were committed in 1995 occurred, how many male clients are there | |
| 159 | financial | simple | ✗ | ✓ | 8133 | 18315 | List all the withdrawals in cash transactions that the client with the id 3356 makes. | |
| 168 | financial | moderate | ✗ | ✓ | 9746 | 18392 | What percentage of clients who opened their accounts in the district with an average salary of over 10000 are women? | |
| 169 | financial | challenging | ✗ | ✓ | 9742 | 17540 | What was the growth rate of the total amount of loans across all accounts for a male client between 1996 and 1997? | |
| 173 | financial | challenging | ✗ | ✓ | 171265 | 18269 | How often does account number 3 request an account statement to be released? What was the aim of debiting 3539 in total? | |
| 189 | financial | moderate | ✓ | ✓ | 8769 | 17277 | Name the account numbers of female clients who are oldest and have lowest average salary? | |
| 192 | financial | moderate | ✓ | ✓ | 6327 | 18049 | What is the average amount of loan which are still on running contract with statement issuance after each transaction? | |
| 194 | financial | moderate | ✓ | ✓ | 14498 | 17780 | Provide the IDs and age of the client with high level credit card, which is eligible for loans. | |
| 207 | toxicology | challenging | ✗ | ✓ | 5849 | 10430 | What elements are in a double type bond? | |
| 208 | toxicology | moderate | ✓ | ✓ | 6048 | 10335 | Which type of label is the most numerous in atoms with hydrogen? | |
| 219 | toxicology | challenging | ✗ | ✓ | 6331 | 9852 | What is the percentage of carcinogenic molecules in triple type bonds? | |
| 227 | toxicology | simple | ✓ | ✓ | 5683 | 10507 | What is the percentage of molecules that are carcinogenic? Please provide your answer as a percentage with three decimal | |
| 230 | toxicology | challenging | ✓ | ✓ | 6702 | 10372 | What are the elements of the toxicology and label of molecule TR060? | |
| 232 | toxicology | moderate | ✓ | ✓ | 7308 | 10081 | Please list top three molecules that have single bonds between two atoms and are not carcinogenic in alphabetical order. | |
| 236 | toxicology | moderate | ✓ | ✓ | 6123 | 10601 | What are the bond type and the atoms of the bond ID of TR001_6_9? | |
| 239 | toxicology | simple | ✓ | ✓ | 5274 | 10266 | How many connections does the atom 19 have? | |
| 253 | toxicology | challenging | ✓ | ✓ | 5811 | 10394 | List the elements of all the triple bonds. | |
| 260 | toxicology | moderate | ✓ | ✓ | 7197 | 10625 | Calculate the total atoms with triple-bond molecules containing the element phosphorus or bromine. | |
| 268 | toxicology | challenging | ✓ | ✓ | 6213 | 10594 | What are the elements for bond id TR001_10_11? | |
| 273 | toxicology | moderate | ✗ | ✓ | 6496 | 10726 | What is the percentage of element chlorine in carcinogenic molecules? | |
| 282 | toxicology | challenging | ✓ | ✓ | 6722 | 10927 | What is the ratio of Hydrogen elements in molecule ID TR006? List the ratio with its label. | |
| 327 | toxicology | moderate | ✓ | ✓ | 14915 | 10057 | Which non-carcinogenic molecules consisted more than 5 atoms? | |
| 347 | card_games | moderate | ✗ | ✓ | 10411 | 35032 | Find all cards illustrated by Stephen Daniel and describe the text of the ruling of these cards. State if these cards ha | |
| 349 | card_games | moderate | ✗ | ✓ | 9713 | 34117 | Name the card and artist with the most ruling information. Also state if the card is a promotional printing. | |
| 352 | card_games | moderate | ✗ | ✓ | 14031 | 33988 | Calculate the percentage of the cards availabe in Chinese Simplified. | |
| 356 | card_games | simple | ✓ | ✓ | 5011 | 33362 | How many cards have infinite power? | |
| 358 | card_games | simple | ✓ | ✓ | 5557 | 33595 | What is the border color of card "Ancestor's Chosen"? | |
| 366 | card_games | simple | ✗ | ✓ | 10875 | 33836 | What is the rule of playing card "Benalish Knight"? | |
| 377 | card_games | simple | ✗ | ✓ | 6157 | 33659 | How many cards with original type of "Summon - Angel" have subtype other than "Angel"? | |
| 391 | card_games | moderate | ✓ | ✓ | 8917 | 34118 | Among the Artifact cards, which are black color and comes with foreign languague translation? | |
| 407 | card_games | moderate | ✗ | ✓ | 9033 | 34045 | Lists all types of cards in German. | |
| 408 | card_games | moderate | ✗ | ✓ | 15915 | 33709 | How many unknown power cards contain info about the triggered ability | |
| 412 | card_games | moderate | ✗ | ✓ | empty_result | 12170 | 34015 | What is the foreign name of the card in French of type Creature, normal layout and black border color, by artist Matthew |
| 414 | card_games | simple | ✓ | ✓ | 6536 | 33965 | What language is the set of 180 cards that belongs to the Ravnica block translated into? | |
| 427 | card_games | moderate | ✓ | ✓ | 6969 | 34050 | What languages are available in the set known as Archenemy on the magic card market and having the code ARC? | |
| 459 | card_games | moderate | ✗ | ✓ | 5601 | 34078 | Which card costs more converted mana, "Serra Angel" or "Shrine Keeper"? | |
| 466 | card_games | moderate | ✓ | ✓ | 7846 | 34065 | Among the cards in the set "Hauptset Zehnte Edition", how many of them are designed by Adam Rex? | |
| 472 | card_games | moderate | ✓ | ✓ | 7099 | 33993 | Among the sets in the block "Ice Age", how many of them have an Italian translation? | |
| 484 | card_games | moderate | ✗ | ✓ | 11598 | 34175 | Please list the Italian names of the cards in the set Coldsnap with the highest converted mana cost. | |
| 486 | card_games | moderate | ✓ | ✓ | 11902 | 34449 | What is the percentage of the cards with a converted mana cost of 7 in the set Coldsnap? | |
| 518 | card_games | moderate | ✗ | ✓ | 78508 | 34246 | Which of the play format has the highest number of banned status? Indicate the play format and the names of all the card | |
| 531 | codebase_community | simple | ✗ | ✓ | 6164 | 22417 | Which user has a higher reputation, Harlan or Jarrod Dixon? | |
| 557 | codebase_community | moderate | ✗ | ✓ | 13297 | 25221 | Among the posts with a score of over 5, what is the percentage of them being owned by an elder user? | |
| 563 | codebase_community | moderate | ✗ | ✓ | empty_result | 18261 | 25385 | User No.3025 gave a comment at 20:29:39 on 2014/4/23 to a post, how many favorite counts did that post get? |
| 571 | codebase_community | moderate | ✗ | ✓ | 7909 | 24948 | For the user No.24, how many times is the number of his/her posts compared to his/her votes? | |
| 584 | codebase_community | moderate | ✗ | ✓ | 9117 | 25490 | Write all the comments left by users who edited the post titled 'Why square the difference instead of taking the absolut | |
| 595 | codebase_community | moderate | ✗ | ✓ | 15603 | 25092 | Which user have only one post history per post and having at least 1000 views? | |
| 634 | codebase_community | challenging | ✗ | ✓ | empty_result | 7478 | 23941 | Among posts by Harvey Motulsky and Noah Snyder, which one has higher popularity? |
| 669 | codebase_community | simple | ✓ | ✓ | 5465 | 22382 | When did 'chl' cast its first vote in a post? | |
| 671 | codebase_community | simple | ✓ | ✓ | 6248 | 22429 | What is the display name of the user who acquired the first Autobiographer badge? | |
| 672 | codebase_community | moderate | ✗ | ✓ | 6339 | 24733 | Among the users located in United Kingdom, how many users whose post have a total favorite amount of 4 or more? | |
| 694 | codebase_community | moderate | ✗ | ✓ | 7555 | 25837 | Provide the text of the latest 10 comments to the post with the title 'Analysing wind data with R' and the display name | |
| 707 | codebase_community | moderate | ✓ | ✓ | 7637 | 25007 | Among the posts with views ranging from 100 to 150, what is the comment with the highest score? | |
| 716 | codebase_community | moderate | ✗ | ✓ | 15972 | 25241 | Among the comments with scores between 5 to 10, what is the percentage of the users with 0 up votes? | |
| 723 | superhero | moderate | ✓ | ✓ | 7181 | 13818 | Among the superheroes with blue eyes, how many of them have the super power of "Agility"? | |
| 730 | superhero | challenging | ✓ | ✓ | 6764 | 13950 | List the superheroes from Marvel Comics who have the super power of 'Super Strength'. | |
| 736 | superhero | moderate | ✓ | ✓ | 5534 | 13385 | Who is the dumbest superhero? | |
| 737 | superhero | simple | ✓ | ✓ | 4806 | 13084 | What is Copycat's race? | |
| 738 | superhero | simple | ✓ | ✓ | 6627 | 13726 | Which superheroes have a durability attribute value of less than 50? | |
| 743 | superhero | challenging | ✗ | ✓ | 12249 | 14060 | What is the percentage of superheroes who act in their own self-interest or make decisions based on their own moral code | |
| 747 | superhero | simple | ✓ | ✓ | 4930 | 13084 | What is the total number of superheroes without full name? | |
| 750 | superhero | simple | ✓ | ✓ | 4333 | 13336 | What is the average weight of all female superheroes? | |
| 751 | superhero | moderate | ✓ | ✓ | 7078 | 13694 | List down at least five superpowers of male superheroes. | |
| 753 | superhero | moderate | ✓ | ✓ | 6180 | 13852 | Among the superheroes with height from 170 to 190, list the names of the superheroes with no eye color. | |
| 765 | superhero | simple | ✓ | ✓ | 5036 | 13232 | How many heroes have stealth power? | |
| 773 | superhero | challenging | ✓ | ✓ | 6498 | 13651 | Which superhero has the same eyes, hair and skin colour? Indicate the publisher of the superhero. | |
| 775 | superhero | challenging | ✓ | ✓ | 16745 | 13916 | What is the percentage of blue female superheroes among all female superheroes? | |
| 781 | superhero | simple | ✓ | ✓ | 5461 | 13484 | Provide the heights of the heroes whose eye colours are amber. | |
| 785 | superhero | simple | ✓ | ✓ | 8047 | 13362 | Describe the names of neutral alignment superheroes. | |
| 791 | superhero | simple | ✗ | ✓ | 4037 | 13169 | Calculate the average height for all superhero. | |
| 794 | superhero | moderate | ✓ | ✓ | 5475 | 13431 | Which hero was the fastest? | |
| 798 | superhero | moderate | ✓ | ✓ | 6194 | 13574 | What is the publisher for Hawkman, Karate Kid and Speedy? | |
| 800 | superhero | moderate | ✓ | ✓ | 5607 | 13722 | Calculate the percentage of superheroes with blue eyes. | |
| 806 | superhero | simple | ✓ | ✓ | 6122 | 13102 | Provide the eye colour of the superhero who has Karen Beecher-Duncan as their full name. | |
| 819 | superhero | challenging | ✓ | ✓ | 7088 | 14359 | In superheroes with missing weight data, calculate the difference between the number of superheroes with blue eyes and n | |
| 825 | superhero | moderate | ✓ | ✓ | 12497 | 13496 | Identify the gender of the superhero who has the ability of Phoenix Force. | |
| 847 | formula_1 | simple | ✗ | ✓ | 6725 | 25832 | What is the surname of the driver with the best lap time in race number 19 in the second qualifying period? | |
| 859 | formula_1 | simple | ✓ | ✓ | 13727 | 25842 | What's Bruno Senna's Q1 result in the qualifying race No. 354? | |
| 861 | formula_1 | simple | ✗ | ✓ | empty_result | 5946 | 25822 | What is his number of the driver who finished 0:01:54 in the Q3 of qualifying race No.903? |
| 862 | formula_1 | simple | ✓ | ✓ | 5699 | 25806 | For the Bahrain Grand Prix in 2007, how many drivers not finished the game? | |
| 865 | formula_1 | moderate | ✓ | ✓ | 6824 | 26052 | For all the drivers who finished the game in race No. 592, who is the oldest? | |
| 866 | formula_1 | moderate | ✗ | ✓ | 4977 | 26236 | Who was the player that got the lap time of 0:01:27 in the race No. 161? Show his introduction website. | |
| 875 | formula_1 | simple | ✓ | ✓ | 4528 | 25609 | Show me the season page of year when the race No. 901 took place. | |
| 877 | formula_1 | moderate | ✓ | ✓ | 5914 | 25781 | For all the drivers who finished the game in race No. 872, who is the youngest? | |
| 879 | formula_1 | moderate | ✓ | ✓ | 5724 | 25591 | For the driver who set the fastest lap speed, what is his nationality? | |
| 881 | formula_1 | moderate | ✗ | ✓ | 9173 | 25936 | For the drivers who took part in the race in 1983/7/16, what's their race completion rate? | |
| 894 | formula_1 | moderate | ✗ | ✓ | 8166 | 26043 | What is the best lap time recorded? List the driver and race with such recorded lap time. | |
| 896 | formula_1 | challenging | ✗ | ✓ | 7598 | 26407 | Calculate the percentage whereby Hamilton was not at the 1st track of the the f1 circuit since 2010. | |
| 897 | formula_1 | moderate | ✓ | ✓ | 6960 | 26014 | Name the driver with the most winning. Mention his nationality and what is his maximum point scores. | |
| 898 | formula_1 | simple | ✗ | ✓ | 6772 | 25025 | How old is the youngest Japanese driver? What is his name? | |
| 902 | formula_1 | simple | ✗ | ✓ | 13596 | 26072 | Which race was Alex Yoong in when he was in track number less than 20? | |
| 904 | formula_1 | moderate | ✗ | ✓ | 8069 | 25052 | State the race and year of race in which Michael Schumacher had his fastest lap. | |
| 909 | formula_1 | moderate | ✓ | ✓ | 6998 | 26090 | Among all European Grand Prix races, what is the percentage of the races were hosted in Germany? | |
| 912 | formula_1 | simple | ✓ | ✓ | 5162 | 24431 | What's the reference name of Marina Bay Street Circuit? | |
| 915 | formula_1 | simple | ✗ | ✓ | 6863 | 25413 | Which country is the oldest driver from? | |
| 930 | formula_1 | simple | ✗ | ✓ | 6949 | 25880 | In which Formula_1 race did Lewis Hamilton rank the highest? | |
| 945 | formula_1 | simple | ✓ | ✓ | 4541 | 24320 | How many circuits are there in Adelaide, Australia? | |
| 950 | formula_1 | simple | ✗ | ✓ | 5670 | 25737 | Please list the constructor names with 0 points at race 291. | |
| 959 | formula_1 | simple | ✗ | ✓ | 9469 | 26084 | What is the fastest lap number of the champion in 2009? | |
| 971 | formula_1 | simple | ✓ | ✓ | 4878 | 25317 | Please state the reference name of the oldest German driver. | |
| 981 | formula_1 | moderate | ✓ | ✓ | 7587 | 26210 | On what year did the youngest driver had his first qualifying race? Also state the name, date and time of the race. | |
| 988 | formula_1 | challenging | ✗ | ✓ | 9530 | 25856 | List down top 3 German drivers who has the shortest average pit stop duration and were born between 1980-1985. | |
| 989 | formula_1 | moderate | ✗ | ✓ | 17035 | 26031 | Who is the champion of the Canadian Grand Prix in 2008? Indicate his finish time. | |
| 990 | formula_1 | challenging | ✓ | ✓ | 7233 | 26117 | What is the constructor reference name of the champion in the 2009 Singapore Grand Prix? Please give its website. | |
| 1028 | european_football_2 | challenging | ✗ | ✓ | 13124 | 47377 | In Scotland Premier League, which away team won the most during the 2010 season? | |
| 1029 | european_football_2 | moderate | ✗ | ✓ | 8030 | 46713 | What are the speed in which attacks are put together of the top 4 teams with the highest build Up Play Speed? | |
| 1030 | european_football_2 | moderate | ✓ | ✓ | 10580 | 46648 | Give the name of the league had the most matches end as draw in the 2016 season? | |
| 1035 | european_football_2 | simple | ✓ | ✓ | 6884 | 46305 | Give the team_fifa_api_id of teams with more than 50 but less than 60 build-up play speed. | |
| 1036 | european_football_2 | challenging | ✗ | ✓ | 9346 | 46552 | List the long name of teams with above-average build-up play passing in 2012. | |
| 1037 | european_football_2 | challenging | ✗ | ✓ | 11890 | 47170 | Calculate the percentage of players who prefer left foot, who were born between 1987 and 1992. | |
| 1039 | european_football_2 | simple | ✓ | ✓ | 7311 | 46345 | Find the average number of long-shot done by Ahmed Samir Farag. | |
| 1042 | european_football_2 | challenging | ✓ | ✓ | 6629 | 47326 | List the name of leagues in which the average goals by the home team is higher than the away team in the 2009/2010 seaso | |
| 1057 | european_football_2 | moderate | ✓ | ✓ | 15845 | 46711 | Calculate the average home team goal in the 2010/2011 season in the country of Poland. | |
| 1078 | european_football_2 | simple | ✓ | ✓ | 5152 | 45682 | Which player is older, Aaron Lennon or Abdelaziz Barrada? | |
| 1088 | european_football_2 | moderate | ✗ | ✓ | 6704 | 46689 | Please list the names of the players whose volley score and dribbling score are over 70. | |
| 1094 | european_football_2 | challenging | ✗ | ✓ | 9329 | 46135 | How much higher in percentage is Ariel Borysiuk's overall rating than that of Paulin Puel? | |
| 1103 | european_football_2 | moderate | ✓ | ✓ | 7314 | 46552 | What was the overall rating for Aaron Mooy on 2016/2/4? | |
| 1110 | european_football_2 | moderate | ✓ | ✓ | 6447 | 46298 | Tell the build Up play passing class for "FC Lorient" on 2010/2/22. | |
| 1116 | european_football_2 | simple | ✓ | ✓ | 5168 | 45980 | List down most tallest players' name. | |
| 1122 | european_football_2 | simple | ✓ | ✓ | 6880 | 45871 | State the name of the most strongest player. | |
| 1130 | european_football_2 | moderate | ✓ | ✓ | 5850 | 46396 | What are the short name of team who played safe while creating chance of passing? | |
| 1133 | european_football_2 | simple | ✓ | ✓ | 4535 | 45856 | How many football players born after the 1990s have the first name "Aaron"? | |
| 1141 | european_football_2 | moderate | ✓ | ✓ | 9311 | 46050 | Does the KSV Cercle Brugge team have a slow, balanced or fast speed class? | |
| 1144 | european_football_2 | simple | ✗ | ✓ | 6977 | 46444 | Please state the finishing rate and curve score of the player who has the heaviest weight. | |
| 1146 | european_football_2 | moderate | ✓ | ✓ | 14492 | 46315 | Please provide the full name of the away team that scored the most goals. | |
| 1147 | european_football_2 | simple | ✓ | ✓ | 6612 | 45868 | Please name one player whose overall strength is the greatest. | |
| 1152 | thrombosis_prediction | moderate | ✓ | ✓ | 7450 | 19497 | What is the ratio of outpatient to inpatient followed up treatment among all the 'SLE' diagnosed patient? | |
| 1156 | thrombosis_prediction | moderate | ✗ | ✓ | execution_failed | 7895 | 17882 | State the ID and age of patient with positive degree of coagulation. |
| 1157 | thrombosis_prediction | simple | ✓ | ✓ | 5447 | 19099 | For patients with severe degree of thrombosis, list their ID, sex and disease the patient is diagnosed with. | |
| 1168 | thrombosis_prediction | challenging | ✗ | ✓ | execution_failed | 5889 | 18151 | The oldest SJS patient's medical laboratory work was completed on what date, and what age was the patient when they init |
| 1185 | thrombosis_prediction | challenging | ✓ | ✓ | 10281 | 20740 | For the patient who was born on 1959/2/18, what is the decrease rate for his/her total cholesterol from November to Dece | |
| 1198 | thrombosis_prediction | simple | ✓ | ✓ | 4514 | 18594 | How many female patients were given an APS diagnosis? | |
| 1205 | thrombosis_prediction | moderate | ✗ | ✓ | 7868 | 19383 | Was the patient with the number 57266's uric acid within a normal range? | |
| 1208 | thrombosis_prediction | moderate | ✓ | ✓ | 4864 | 19359 | Provide IDs for male patients with ALT glutamic pylvic transaminase (GPT) that have history of ALT glutamic pylvic trans | |
| 1220 | thrombosis_prediction | simple | ✓ | ✓ | 5676 | 19496 | Provide all ID, sex and birthday of patients whose urea nitrogen (UN) just within the borderline of passing? | |
| 1227 | thrombosis_prediction | moderate | ✗ | ✓ | execution_failed | 5617 | 18018 | What is the average age of the male patient with high cholesterol? |
| 1232 | thrombosis_prediction | challenging | ✓ | ✓ | 6749 | 19087 | Provide ID, sex and age of patient who has blood glucose (GLU) not within normal range but with total cholesterol(T-CHO) | |
| 1235 | thrombosis_prediction | moderate | ✗ | ✓ | 15207 | 18391 | What are the patient's diagnosis for those who has lower red blood blood cell? State their ID and age. | |
| 1247 | thrombosis_prediction | challenging | ✗ | ✓ | 6060 | 19444 | Among the male patients who have a normal level of white blood cells, how many of them have an abnormal fibrinogen level | |
| 1251 | thrombosis_prediction | simple | ✗ | ✓ | 4336 | 18724 | How many patients with an Ig G higher than normal? | |
| 1252 | thrombosis_prediction | moderate | ✗ | ✓ | 5053 | 18785 | Among the patients with a normal Ig G level, how many of them have symptoms? | |
| 1254 | thrombosis_prediction | moderate | ✗ | ✓ | execution_failed | 5899 | 18120 | How many patients with a normal Ig A level came to the hospital after 1990/1/1? |
| 1255 | thrombosis_prediction | moderate | ✗ | ✓ | 5231 | 19032 | For the patients with an abnormal Ig M level, what is the most common disease they are diagnosed with? | |
| 1257 | thrombosis_prediction | challenging | ✓ | ✓ | 5511 | 18886 | Among the patients whose creatinine level is abnormal, how many of them aren't 70 yet? | |
| 1275 | thrombosis_prediction | moderate | ✗ | ✓ | execution_failed | 5802 | 18133 | Among the patients who has a normal level of anti-centromere and a normal level of anti-SSB, how many of them are male? |
| 1281 | thrombosis_prediction | moderate | ✓ | ✓ | 4404 | 19057 | Among the patients who have an abnormal level of glutamic oxaloacetic transaminase, when was the youngest of them born? | |
| 1302 | thrombosis_prediction | challenging | ✗ | ✓ | execution_failed | 4907 | 18000 | For the patients with a normal range of creatinine phosphokinase, how many of them have a positive measure of degree of |
| 1312 | student_club | simple | ✓ | ✓ | 6818 | 18532 | What's Angela Sanders's major? | |
| 1340 | student_club | moderate | ✓ | ✓ | 7925 | 19140 | Calculate the difference of the total amount spent in all events by the Student_Club in year 2019 and 2020. | |
| 1344 | student_club | simple | ✓ | ✓ | 8545 | 18775 | What was the notes of the fundraising on 2019/9/14? | |
| 1352 | student_club | moderate | ✓ | ✓ | 4787 | 18680 | For all the club members from "Business" major, how many of them wear medium size t-shirt? | |
| 1356 | student_club | simple | ✓ | ✓ | 5069 | 18408 | Which department was the President of the club in? | |
| 1376 | student_club | moderate | ✓ | ✓ | 17365 | 18773 | Among all the closed events, which event has the highest spend-to-budget ratio? | |
| 1378 | student_club | simple | ✓ | ✓ | 5348 | 16411 | What is the highest amount of budget spend for an event? | |
| 1380 | student_club | simple | ✓ | ✓ | 4825 | 17458 | What is the total amount of money spent for food? | |
| 1387 | student_club | moderate | ✓ | ✓ | 7063 | 18700 | Which student has been entrusted to manage the budget for the Yearly Kickoff? | |
| 1390 | student_club | moderate | ✓ | ✓ | 6780 | 17109 | Based on the total cost for all event, what is the percentage of cost for Yearly Kickoff event? | |
| 1399 | student_club | moderate | ✗ | ✓ | 5333 | 18827 | Did Maya Mclean attend the 'Women's Soccer' event? | |
| 1403 | student_club | moderate | ✓ | ✓ | 5629 | 18768 | Indicate the name of the closed event whose cost has exceeded the budget the most. | |
| 1404 | student_club | moderate | ✗ | ✓ | 6228 | 18769 | Identify the type of expenses and their total value approved for 'October Meeting' event. | |
| 1409 | student_club | simple | ✓ | ✓ | 4896 | 18654 | Mention the total expense used on 8/20/2019. | |
| 1410 | student_club | simple | ✓ | ✓ | 7107 | 18889 | List out the full name and total cost that member id "rec4BLdZHS2Blfp4v" incurred? | |
| 1411 | student_club | simple | ✓ | ✓ | 8464 | 18760 | State what kind of expenses that Sacha Harrison incurred? | |
| 1422 | student_club | simple | ✗ | ✓ | 4530 | 18461 | State the category of events were held at MU 215. | |
| 1464 | student_club | challenging | ✓ | ✓ | 6489 | 18983 | Write the full names of students who received funds on the date of 9/9/2019 and include the amount received. | |
| 1472 | debit_card_specializing | moderate | ✗ | ✓ | 14592 | 11949 | In 2012, who had the least consumption in LAM? | |
| 1473 | debit_card_specializing | moderate | ✓ | ✓ | 11068 | 12468 | What was the average monthly consumption of customers in SME for the year 2013? | |
| 1476 | debit_card_specializing | challenging | ✓ | ✓ | 9303 | 12901 | What was the difference in gas consumption between CZK-paying customers and EUR-paying customers in 2012? | |
| 1479 | debit_card_specializing | moderate | ✓ | ✓ | 12292 | 11974 | Which year recorded the most consumption of gas paid in CZK? | |
| 1480 | debit_card_specializing | moderate | ✗ | ✓ | 8242 | 12354 | What was the gas consumption peak month for SME customers in 2013? | |
| 1484 | debit_card_specializing | simple | ✓ | ✓ | 5520 | 12321 | How many more "discount" gas stations does the Czech Republic have compared to Slovakia? | |
| 1486 | debit_card_specializing | simple | ✗ | ✓ | 6272 | 12230 | Is it true that more SMEs pay in Czech koruna than in euros? If so, how many more? | |
| 1493 | debit_card_specializing | simple | ✓ | ✓ | 7440 | 12546 | In February 2012, what percentage of customers consumed more than 528.3? | |
| 1500 | debit_card_specializing | simple | ✓ | ✓ | 6442 | 12403 | Please list the product description of the products consumed in September, 2013. | |
| 1501 | debit_card_specializing | moderate | ✓ | ✓ | 7513 | 12359 | Please list the countries of the gas stations with transactions taken place in June, 2013. | |
| 1506 | debit_card_specializing | moderate | ✓ | ✓ | 9547 | 12157 | Please list the product descriptions of the transactions taken place in the gas stations in the Czech Republic. | |
| 1515 | debit_card_specializing | simple | ✓ | ✓ | 15001 | 12109 | What segment did the customer have at 2012/8/23 21:20:00? | |
| 1521 | debit_card_specializing | moderate | ✓ | ✓ | 9208 | 12840 | For all the transactions happened during 8:00-9:00 in 2012/8/26, how many happened in CZE? | |
| 1525 | debit_card_specializing | simple | ✗ | ✓ | 6160 | 12330 | What is the percentage of the customers who used EUR in 2012/8/25? | |
| 1526 | debit_card_specializing | challenging | ✗ | ✓ | empty_result | 11851 | 13256 | For the customer who paid 634.8 in 2012/8/25, what was the consumption decrease rate from Year 2012 to 2013? |
| 1528 | debit_card_specializing | simple | ✓ | ✓ | 6549 | 11856 | What is the percentage of "premium" against the overall segment in Country = "SVK"? | |
| 1529 | debit_card_specializing | moderate | ✗ | ✓ | 6859 | 11857 | What is the amount spent by customer "38508" at the gas stations? How much had the customer spent in January 2012? | |
| 1531 | debit_card_specializing | moderate | ✗ | ✓ | 8090 | 12379 | Who is the top spending customer and how much is the average price per single item purchased by this customer? What curr |
Model: codestral-latest · n=200 · EA=60.0% · Validity=100.0% · Recall@k=100.0%
| qid | db | diff | match | recall | err | lat ms | tokens | question |
|---|---|---|---|---|---|---|---|---|
| 5 | california_schools | simple | ✓ | ✓ | 244 | 25146 | How many schools with an average score in Math greater than 400 in the SAT test are exclusively virtual? | |
| 25 | california_schools | moderate | ✗ | ✓ | 152 | 25609 | Name schools in Riverside which the average of average math score for SAT is grater than 400, what is the funding type o | |
| 32 | california_schools | moderate | ✓ | ✓ | 114 | 26026 | What is the eligible free or reduced price meal rate for the top 5 schools in grades 1-12 with the highest free or reduc | |
| 36 | california_schools | challenging | ✓ | ✓ | 71 | 25890 | Under whose administration is the school with the highest number of students scoring 1500 or more on the SAT? Indicate t | |
| 37 | california_schools | moderate | ✗ | ✓ | 70 | 25966 | What is the complete address of the school with the lowest excellence rate? Indicate the Street, City, Zip and State. | |
| 39 | california_schools | simple | ✓ | ✓ | 123 | 25966 | What is the average number of test takers from Fresno schools that opened between 1/1/1980 and 12/31/1980? | |
| 48 | california_schools | moderate | ✓ | ✓ | 94 | 25777 | What is the ratio of merged Unified School District schools in Orange County to merged Elementary School District school | |
| 50 | california_schools | simple | ✗ | ✓ | 2356 | 25226 | What is the postal street address for the school with the 7th highest Math average? Indicate the school's name. | |
| 77 | california_schools | moderate | ✗ | ✓ | empty_result | 76 | 25588 | Which schools served a grade span of Kindergarten to 9th grade in the county of Los Angeles and what is its Percent (%) |
| 92 | financial | simple | ✓ | ✓ | 65 | 17880 | List out the no. of districts that have female average salary is more than 6000 but less than 10000? | |
| 98 | financial | moderate | ✓ | ✓ | 53 | 17987 | Among the accounts who have approved loan date in 1997, list out the accounts that have the lowest approved amount and c | |
| 99 | financial | moderate | ✓ | ✓ | 51 | 17956 | Among the accounts who have loan validity more than 12 months, list out the accounts that have the highest approved amou | |
| 112 | financial | simple | ✓ | ✓ | 60 | 17964 | For the female client who was born in 1976/1/29, which district did she opened her account? | |
| 115 | financial | challenging | ✗ | ✓ | 1328 | 17929 | For the branch which located in the south Bohemia with biggest number of inhabitants, what is the percentage of the male | |
| 118 | financial | moderate | ✓ | ✓ | 66 | 18079 | For loan amount less than USD100,000, what is the percentage of accounts that is still running with no issue. | |
| 120 | financial | moderate | ✓ | ✓ | 1618 | 19284 | From Year 1995 to 2000, who are the accounts holders from 'east Bohemia'. State the account ID the frequency of statemen | |
| 125 | financial | challenging | ✗ | ✓ | 1525 | 19486 | For loans contracts which are still running where client are in debt, list the district of the and the state the percent | |
| 138 | financial | moderate | ✓ | ✓ | 59 | 17925 | In the branch where the second-highest number of crimes were committed in 1995 occurred, how many male clients are there | |
| 159 | financial | simple | ✗ | ✓ | 2919 | 18315 | List all the withdrawals in cash transactions that the client with the id 3356 makes. | |
| 168 | financial | moderate | ✗ | ✓ | 1953 | 18392 | What percentage of clients who opened their accounts in the district with an average salary of over 10000 are women? | |
| 169 | financial | challenging | ✗ | ✓ | 74 | 17540 | What was the growth rate of the total amount of loans across all accounts for a male client between 1996 and 1997? | |
| 173 | financial | challenging | ✗ | ✓ | 421 | 18269 | How often does account number 3 request an account statement to be released? What was the aim of debiting 3539 in total? | |
| 189 | financial | moderate | ✓ | ✓ | 70 | 17277 | Name the account numbers of female clients who are oldest and have lowest average salary? | |
| 192 | financial | moderate | ✓ | ✓ | 56 | 18049 | What is the average amount of loan which are still on running contract with statement issuance after each transaction? | |
| 194 | financial | moderate | ✓ | ✓ | 52 | 17780 | Provide the IDs and age of the client with high level credit card, which is eligible for loans. | |
| 207 | toxicology | challenging | ✗ | ✓ | 182 | 10430 | What elements are in a double type bond? | |
| 208 | toxicology | moderate | ✓ | ✓ | 60 | 10335 | Which type of label is the most numerous in atoms with hydrogen? | |
| 219 | toxicology | challenging | ✗ | ✓ | 57 | 9852 | What is the percentage of carcinogenic molecules in triple type bonds? | |
| 227 | toxicology | simple | ✓ | ✓ | 55 | 10507 | What is the percentage of molecules that are carcinogenic? Please provide your answer as a percentage with three decimal | |
| 230 | toxicology | challenging | ✓ | ✓ | 54 | 10372 | What are the elements of the toxicology and label of molecule TR060? | |
| 232 | toxicology | moderate | ✓ | ✓ | 51 | 10081 | Please list top three molecules that have single bonds between two atoms and are not carcinogenic in alphabetical order. | |
| 236 | toxicology | moderate | ✓ | ✓ | 45 | 10601 | What are the bond type and the atoms of the bond ID of TR001_6_9? | |
| 239 | toxicology | simple | ✓ | ✓ | 53 | 10266 | How many connections does the atom 19 have? | |
| 253 | toxicology | challenging | ✓ | ✓ | 78 | 10394 | List the elements of all the triple bonds. | |
| 260 | toxicology | moderate | ✓ | ✓ | 94 | 10625 | Calculate the total atoms with triple-bond molecules containing the element phosphorus or bromine. | |
| 268 | toxicology | challenging | ✓ | ✓ | 61 | 10594 | What are the elements for bond id TR001_10_11? | |
| 273 | toxicology | moderate | ✗ | ✓ | 73 | 10726 | What is the percentage of element chlorine in carcinogenic molecules? | |
| 282 | toxicology | challenging | ✓ | ✓ | 64 | 10927 | What is the ratio of Hydrogen elements in molecule ID TR006? List the ratio with its label. | |
| 327 | toxicology | moderate | ✓ | ✓ | 78 | 10057 | Which non-carcinogenic molecules consisted more than 5 atoms? | |
| 347 | card_games | moderate | ✗ | ✓ | 3286 | 35032 | Find all cards illustrated by Stephen Daniel and describe the text of the ruling of these cards. State if these cards ha | |
| 349 | card_games | moderate | ✗ | ✓ | 2143 | 34117 | Name the card and artist with the most ruling information. Also state if the card is a promotional printing. | |
| 352 | card_games | moderate | ✗ | ✓ | 3174 | 33988 | Calculate the percentage of the cards availabe in Chinese Simplified. | |
| 356 | card_games | simple | ✓ | ✓ | 531 | 33362 | How many cards have infinite power? | |
| 358 | card_games | simple | ✓ | ✓ | 499 | 33595 | What is the border color of card "Ancestor's Chosen"? | |
| 366 | card_games | simple | ✗ | ✓ | 5604 | 33836 | What is the rule of playing card "Benalish Knight"? | |
| 377 | card_games | simple | ✗ | ✓ | 499 | 33659 | How many cards with original type of "Summon - Angel" have subtype other than "Angel"? | |
| 391 | card_games | moderate | ✓ | ✓ | 2432 | 34118 | Among the Artifact cards, which are black color and comes with foreign languague translation? | |
| 407 | card_games | moderate | ✗ | ✓ | 2831 | 34045 | Lists all types of cards in German. | |
| 408 | card_games | moderate | ✗ | ✓ | 789 | 33709 | How many unknown power cards contain info about the triggered ability | |
| 412 | card_games | moderate | ✗ | ✓ | empty_result | 1979 | 34015 | What is the foreign name of the card in French of type Creature, normal layout and black border color, by artist Matthew |
| 414 | card_games | simple | ✓ | ✓ | 1091 | 33965 | What language is the set of 180 cards that belongs to the Ravnica block translated into? | |
| 427 | card_games | moderate | ✓ | ✓ | 109 | 34050 | What languages are available in the set known as Archenemy on the magic card market and having the code ARC? | |
| 459 | card_games | moderate | ✗ | ✓ | 651 | 34078 | Which card costs more converted mana, "Serra Angel" or "Shrine Keeper"? | |
| 466 | card_games | moderate | ✓ | ✓ | 609 | 34065 | Among the cards in the set "Hauptset Zehnte Edition", how many of them are designed by Adam Rex? | |
| 472 | card_games | moderate | ✓ | ✓ | 121 | 33993 | Among the sets in the block "Ice Age", how many of them have an Italian translation? | |
| 484 | card_games | moderate | ✗ | ✓ | 2036 | 34175 | Please list the Italian names of the cards in the set Coldsnap with the highest converted mana cost. | |
| 486 | card_games | moderate | ✓ | ✓ | 1040 | 34449 | What is the percentage of the cards with a converted mana cost of 7 in the set Coldsnap? | |
| 518 | card_games | moderate | ✗ | ✓ | 60770 | 34246 | Which of the play format has the highest number of banned status? Indicate the play format and the names of all the card | |
| 531 | codebase_community | simple | ✗ | ✓ | 149 | 22417 | Which user has a higher reputation, Harlan or Jarrod Dixon? | |
| 557 | codebase_community | moderate | ✗ | ✓ | 1385 | 25221 | Among the posts with a score of over 5, what is the percentage of them being owned by an elder user? | |
| 563 | codebase_community | moderate | ✗ | ✓ | empty_result | 450 | 25385 | User No.3025 gave a comment at 20:29:39 on 2014/4/23 to a post, how many favorite counts did that post get? |
| 571 | codebase_community | moderate | ✗ | ✓ | 928 | 24948 | For the user No.24, how many times is the number of his/her posts compared to his/her votes? | |
| 584 | codebase_community | moderate | ✗ | ✓ | 2024 | 25490 | Write all the comments left by users who edited the post titled 'Why square the difference instead of taking the absolut | |
| 595 | codebase_community | moderate | ✗ | ✓ | 5762 | 25092 | Which user have only one post history per post and having at least 1000 views? | |
| 634 | codebase_community | challenging | ✗ | ✓ | empty_result | 673 | 23941 | Among posts by Harvey Motulsky and Noah Snyder, which one has higher popularity? |
| 669 | codebase_community | simple | ✓ | ✓ | 68 | 22382 | When did 'chl' cast its first vote in a post? | |
| 671 | codebase_community | simple | ✓ | ✓ | 260 | 22429 | What is the display name of the user who acquired the first Autobiographer badge? | |
| 672 | codebase_community | moderate | ✗ | ✓ | 775 | 24733 | Among the users located in United Kingdom, how many users whose post have a total favorite amount of 4 or more? | |
| 694 | codebase_community | moderate | ✗ | ✓ | 1525 | 25837 | Provide the text of the latest 10 comments to the post with the title 'Analysing wind data with R' and the display name | |
| 707 | codebase_community | moderate | ✓ | ✓ | 1078 | 25007 | Among the posts with views ranging from 100 to 150, what is the comment with the highest score? | |
| 716 | codebase_community | moderate | ✗ | ✓ | 340 | 25241 | Among the comments with scores between 5 to 10, what is the percentage of the users with 0 up votes? | |
| 723 | superhero | moderate | ✓ | ✓ | 67 | 13818 | Among the superheroes with blue eyes, how many of them have the super power of "Agility"? | |
| 730 | superhero | challenging | ✓ | ✓ | 229 | 13950 | List the superheroes from Marvel Comics who have the super power of 'Super Strength'. | |
| 736 | superhero | moderate | ✓ | ✓ | 67 | 13385 | Who is the dumbest superhero? | |
| 737 | superhero | simple | ✓ | ✓ | 59 | 13084 | What is Copycat's race? | |
| 738 | superhero | simple | ✓ | ✓ | 199 | 13726 | Which superheroes have a durability attribute value of less than 50? | |
| 743 | superhero | challenging | ✗ | ✓ | 61 | 14060 | What is the percentage of superheroes who act in their own self-interest or make decisions based on their own moral code | |
| 747 | superhero | simple | ✓ | ✓ | 64 | 13084 | What is the total number of superheroes without full name? | |
| 750 | superhero | simple | ✓ | ✓ | 62 | 13336 | What is the average weight of all female superheroes? | |
| 751 | superhero | moderate | ✓ | ✓ | 58 | 13694 | List down at least five superpowers of male superheroes. | |
| 753 | superhero | moderate | ✓ | ✓ | 55 | 13852 | Among the superheroes with height from 170 to 190, list the names of the superheroes with no eye color. | |
| 765 | superhero | simple | ✓ | ✓ | 61 | 13232 | How many heroes have stealth power? | |
| 773 | superhero | challenging | ✓ | ✓ | 66 | 13651 | Which superhero has the same eyes, hair and skin colour? Indicate the publisher of the superhero. | |
| 775 | superhero | challenging | ✓ | ✓ | 62 | 13916 | What is the percentage of blue female superheroes among all female superheroes? | |
| 781 | superhero | simple | ✓ | ✓ | 48 | 13484 | Provide the heights of the heroes whose eye colours are amber. | |
| 785 | superhero | simple | ✓ | ✓ | 63 | 13362 | Describe the names of neutral alignment superheroes. | |
| 791 | superhero | simple | ✗ | ✓ | 55 | 13169 | Calculate the average height for all superhero. | |
| 794 | superhero | moderate | ✓ | ✓ | 68 | 13431 | Which hero was the fastest? | |
| 798 | superhero | moderate | ✓ | ✓ | 60 | 13574 | What is the publisher for Hawkman, Karate Kid and Speedy? | |
| 800 | superhero | moderate | ✓ | ✓ | 58 | 13722 | Calculate the percentage of superheroes with blue eyes. | |
| 806 | superhero | simple | ✓ | ✓ | 48 | 13102 | Provide the eye colour of the superhero who has Karen Beecher-Duncan as their full name. | |
| 819 | superhero | challenging | ✓ | ✓ | 65 | 14359 | In superheroes with missing weight data, calculate the difference between the number of superheroes with blue eyes and n | |
| 825 | superhero | moderate | ✓ | ✓ | 77 | 13496 | Identify the gender of the superhero who has the ability of Phoenix Force. | |
| 847 | formula_1 | simple | ✗ | ✓ | 63 | 25832 | What is the surname of the driver with the best lap time in race number 19 in the second qualifying period? | |
| 859 | formula_1 | simple | ✓ | ✓ | 58 | 25842 | What's Bruno Senna's Q1 result in the qualifying race No. 354? | |
| 861 | formula_1 | simple | ✗ | ✓ | empty_result | 54 | 25822 | What is his number of the driver who finished 0:01:54 in the Q3 of qualifying race No.903? |
| 862 | formula_1 | simple | ✓ | ✓ | 85 | 25806 | For the Bahrain Grand Prix in 2007, how many drivers not finished the game? | |
| 865 | formula_1 | moderate | ✓ | ✓ | 63 | 26052 | For all the drivers who finished the game in race No. 592, who is the oldest? | |
| 866 | formula_1 | moderate | ✗ | ✓ | 59 | 26236 | Who was the player that got the lap time of 0:01:27 in the race No. 161? Show his introduction website. | |
| 875 | formula_1 | simple | ✓ | ✓ | 56 | 25609 | Show me the season page of year when the race No. 901 took place. | |
| 877 | formula_1 | moderate | ✓ | ✓ | 65 | 25781 | For all the drivers who finished the game in race No. 872, who is the youngest? | |
| 879 | formula_1 | moderate | ✓ | ✓ | 92 | 25591 | For the driver who set the fastest lap speed, what is his nationality? | |
| 881 | formula_1 | moderate | ✗ | ✓ | 85 | 25936 | For the drivers who took part in the race in 1983/7/16, what's their race completion rate? | |
| 894 | formula_1 | moderate | ✗ | ✓ | 430 | 26043 | What is the best lap time recorded? List the driver and race with such recorded lap time. | |
| 896 | formula_1 | challenging | ✗ | ✓ | 95 | 26407 | Calculate the percentage whereby Hamilton was not at the 1st track of the the f1 circuit since 2010. | |
| 897 | formula_1 | moderate | ✓ | ✓ | 114 | 26014 | Name the driver with the most winning. Mention his nationality and what is his maximum point scores. | |
| 898 | formula_1 | simple | ✗ | ✓ | 51 | 25025 | How old is the youngest Japanese driver? What is his name? | |
| 902 | formula_1 | simple | ✗ | ✓ | 93 | 26072 | Which race was Alex Yoong in when he was in track number less than 20? | |
| 904 | formula_1 | moderate | ✗ | ✓ | 1822 | 25052 | State the race and year of race in which Michael Schumacher had his fastest lap. | |
| 909 | formula_1 | moderate | ✓ | ✓ | 61 | 26090 | Among all European Grand Prix races, what is the percentage of the races were hosted in Germany? | |
| 912 | formula_1 | simple | ✓ | ✓ | 47 | 24431 | What's the reference name of Marina Bay Street Circuit? | |
| 915 | formula_1 | simple | ✗ | ✓ | 1235 | 25413 | Which country is the oldest driver from? | |
| 930 | formula_1 | simple | ✗ | ✓ | 115 | 25880 | In which Formula_1 race did Lewis Hamilton rank the highest? | |
| 945 | formula_1 | simple | ✓ | ✓ | 65 | 24320 | How many circuits are there in Adelaide, Australia? | |
| 950 | formula_1 | simple | ✗ | ✓ | 65 | 25737 | Please list the constructor names with 0 points at race 291. | |
| 959 | formula_1 | simple | ✗ | ✓ | 1451 | 26084 | What is the fastest lap number of the champion in 2009? | |
| 971 | formula_1 | simple | ✓ | ✓ | 70 | 25317 | Please state the reference name of the oldest German driver. | |
| 981 | formula_1 | moderate | ✓ | ✓ | 76 | 26210 | On what year did the youngest driver had his first qualifying race? Also state the name, date and time of the race. | |
| 988 | formula_1 | challenging | ✗ | ✓ | 1441 | 25856 | List down top 3 German drivers who has the shortest average pit stop duration and were born between 1980-1985. | |
| 989 | formula_1 | moderate | ✗ | ✓ | 85 | 26031 | Who is the champion of the Canadian Grand Prix in 2008? Indicate his finish time. | |
| 990 | formula_1 | challenging | ✓ | ✓ | 81 | 26117 | What is the constructor reference name of the champion in the 2009 Singapore Grand Prix? Please give its website. | |
| 1028 | european_football_2 | challenging | ✗ | ✓ | 431 | 47377 | In Scotland Premier League, which away team won the most during the 2010 season? | |
| 1029 | european_football_2 | moderate | ✗ | ✓ | 64 | 46713 | What are the speed in which attacks are put together of the top 4 teams with the highest build Up Play Speed? | |
| 1030 | european_football_2 | moderate | ✓ | ✓ | 385 | 46648 | Give the name of the league had the most matches end as draw in the 2016 season? | |
| 1035 | european_football_2 | simple | ✓ | ✓ | 78 | 46305 | Give the team_fifa_api_id of teams with more than 50 but less than 60 build-up play speed. | |
| 1036 | european_football_2 | challenging | ✗ | ✓ | 1421 | 46552 | List the long name of teams with above-average build-up play passing in 2012. | |
| 1037 | european_football_2 | challenging | ✗ | ✓ | 2455 | 47170 | Calculate the percentage of players who prefer left foot, who were born between 1987 and 1992. | |
| 1039 | european_football_2 | simple | ✓ | ✓ | 506 | 46345 | Find the average number of long-shot done by Ahmed Samir Farag. | |
| 1042 | european_football_2 | challenging | ✓ | ✓ | 400 | 47326 | List the name of leagues in which the average goals by the home team is higher than the away team in the 2009/2010 seaso | |
| 1057 | european_football_2 | moderate | ✓ | ✓ | 319 | 46711 | Calculate the average home team goal in the 2010/2011 season in the country of Poland. | |
| 1078 | european_football_2 | simple | ✓ | ✓ | 67 | 45682 | Which player is older, Aaron Lennon or Abdelaziz Barrada? | |
| 1088 | european_football_2 | moderate | ✗ | ✓ | 769 | 46689 | Please list the names of the players whose volley score and dribbling score are over 70. | |
| 1094 | european_football_2 | challenging | ✗ | ✓ | 337 | 46135 | How much higher in percentage is Ariel Borysiuk's overall rating than that of Paulin Puel? | |
| 1103 | european_football_2 | moderate | ✓ | ✓ | 488 | 46552 | What was the overall rating for Aaron Mooy on 2016/2/4? | |
| 1110 | european_football_2 | moderate | ✓ | ✓ | 51 | 46298 | Tell the build Up play passing class for "FC Lorient" on 2010/2/22. | |
| 1116 | european_football_2 | simple | ✓ | ✓ | 82 | 45980 | List down most tallest players' name. | |
| 1122 | european_football_2 | simple | ✓ | ✓ | 892 | 45871 | State the name of the most strongest player. | |
| 1130 | european_football_2 | moderate | ✓ | ✓ | 72 | 46396 | What are the short name of team who played safe while creating chance of passing? | |
| 1133 | european_football_2 | simple | ✓ | ✓ | 76 | 45856 | How many football players born after the 1990s have the first name "Aaron"? | |
| 1141 | european_football_2 | moderate | ✓ | ✓ | 52 | 46050 | Does the KSV Cercle Brugge team have a slow, balanced or fast speed class? | |
| 1144 | european_football_2 | simple | ✗ | ✓ | 669 | 46444 | Please state the finishing rate and curve score of the player who has the heaviest weight. | |
| 1146 | european_football_2 | moderate | ✓ | ✓ | 444 | 46315 | Please provide the full name of the away team that scored the most goals. | |
| 1147 | european_football_2 | simple | ✓ | ✓ | 764 | 45868 | Please name one player whose overall strength is the greatest. | |
| 1152 | thrombosis_prediction | moderate | ✓ | ✓ | 49 | 19497 | What is the ratio of outpatient to inpatient followed up treatment among all the 'SLE' diagnosed patient? | |
| 1156 | thrombosis_prediction | moderate | ✗ | ✓ | execution_failed | 39 | 17882 | State the ID and age of patient with positive degree of coagulation. |
| 1157 | thrombosis_prediction | simple | ✓ | ✓ | 1474 | 19099 | For patients with severe degree of thrombosis, list their ID, sex and disease the patient is diagnosed with. | |
| 1168 | thrombosis_prediction | challenging | ✗ | ✓ | execution_failed | 46 | 18151 | The oldest SJS patient's medical laboratory work was completed on what date, and what age was the patient when they init |
| 1185 | thrombosis_prediction | challenging | ✓ | ✓ | 1748 | 20740 | For the patient who was born on 1959/2/18, what is the decrease rate for his/her total cholesterol from November to Dece | |
| 1198 | thrombosis_prediction | simple | ✓ | ✓ | 69 | 18594 | How many female patients were given an APS diagnosis? | |
| 1205 | thrombosis_prediction | moderate | ✗ | ✓ | 1606 | 19383 | Was the patient with the number 57266's uric acid within a normal range? | |
| 1208 | thrombosis_prediction | moderate | ✓ | ✓ | 57 | 19359 | Provide IDs for male patients with ALT glutamic pylvic transaminase (GPT) that have history of ALT glutamic pylvic trans | |
| 1220 | thrombosis_prediction | simple | ✓ | ✓ | 59 | 19496 | Provide all ID, sex and birthday of patients whose urea nitrogen (UN) just within the borderline of passing? | |
| 1227 | thrombosis_prediction | moderate | ✗ | ✓ | execution_failed | 39 | 18018 | What is the average age of the male patient with high cholesterol? |
| 1232 | thrombosis_prediction | challenging | ✓ | ✓ | 56 | 19087 | Provide ID, sex and age of patient who has blood glucose (GLU) not within normal range but with total cholesterol(T-CHO) | |
| 1235 | thrombosis_prediction | moderate | ✗ | ✓ | 69 | 18391 | What are the patient's diagnosis for those who has lower red blood blood cell? State their ID and age. | |
| 1247 | thrombosis_prediction | challenging | ✗ | ✓ | 59 | 19444 | Among the male patients who have a normal level of white blood cells, how many of them have an abnormal fibrinogen level | |
| 1251 | thrombosis_prediction | simple | ✗ | ✓ | 70 | 18724 | How many patients with an Ig G higher than normal? | |
| 1252 | thrombosis_prediction | moderate | ✗ | ✓ | 42 | 18785 | Among the patients with a normal Ig G level, how many of them have symptoms? | |
| 1254 | thrombosis_prediction | moderate | ✗ | ✓ | execution_failed | 42 | 18120 | How many patients with a normal Ig A level came to the hospital after 1990/1/1? |
| 1255 | thrombosis_prediction | moderate | ✗ | ✓ | 62 | 19032 | For the patients with an abnormal Ig M level, what is the most common disease they are diagnosed with? | |
| 1257 | thrombosis_prediction | challenging | ✓ | ✓ | 1273 | 18886 | Among the patients whose creatinine level is abnormal, how many of them aren't 70 yet? | |
| 1275 | thrombosis_prediction | moderate | ✗ | ✓ | execution_failed | 56 | 18133 | Among the patients who has a normal level of anti-centromere and a normal level of anti-SSB, how many of them are male? |
| 1281 | thrombosis_prediction | moderate | ✓ | ✓ | 63 | 19057 | Among the patients who have an abnormal level of glutamic oxaloacetic transaminase, when was the youngest of them born? | |
| 1302 | thrombosis_prediction | challenging | ✗ | ✓ | execution_failed | 48 | 18000 | For the patients with a normal range of creatinine phosphokinase, how many of them have a positive measure of degree of |
| 1312 | student_club | simple | ✓ | ✓ | 63 | 18532 | What's Angela Sanders's major? | |
| 1340 | student_club | moderate | ✓ | ✓ | 65 | 19140 | Calculate the difference of the total amount spent in all events by the Student_Club in year 2019 and 2020. | |
| 1344 | student_club | simple | ✓ | ✓ | 52 | 18775 | What was the notes of the fundraising on 2019/9/14? | |
| 1352 | student_club | moderate | ✓ | ✓ | 50 | 18680 | For all the club members from "Business" major, how many of them wear medium size t-shirt? | |
| 1356 | student_club | simple | ✓ | ✓ | 49 | 18408 | Which department was the President of the club in? | |
| 1376 | student_club | moderate | ✓ | ✓ | 56 | 18773 | Among all the closed events, which event has the highest spend-to-budget ratio? | |
| 1378 | student_club | simple | ✓ | ✓ | 69 | 16411 | What is the highest amount of budget spend for an event? | |
| 1380 | student_club | simple | ✓ | ✓ | 59 | 17458 | What is the total amount of money spent for food? | |
| 1387 | student_club | moderate | ✓ | ✓ | 53 | 18700 | Which student has been entrusted to manage the budget for the Yearly Kickoff? | |
| 1390 | student_club | moderate | ✓ | ✓ | 54 | 17109 | Based on the total cost for all event, what is the percentage of cost for Yearly Kickoff event? | |
| 1399 | student_club | moderate | ✗ | ✓ | 57 | 18827 | Did Maya Mclean attend the 'Women's Soccer' event? | |
| 1403 | student_club | moderate | ✓ | ✓ | 69 | 18768 | Indicate the name of the closed event whose cost has exceeded the budget the most. | |
| 1404 | student_club | moderate | ✗ | ✓ | 66 | 18769 | Identify the type of expenses and their total value approved for 'October Meeting' event. | |
| 1409 | student_club | simple | ✓ | ✓ | 60 | 18654 | Mention the total expense used on 8/20/2019. | |
| 1410 | student_club | simple | ✓ | ✓ | 57 | 18889 | List out the full name and total cost that member id "rec4BLdZHS2Blfp4v" incurred? | |
| 1411 | student_club | simple | ✓ | ✓ | 46 | 18760 | State what kind of expenses that Sacha Harrison incurred? | |
| 1422 | student_club | simple | ✗ | ✓ | 44 | 18461 | State the category of events were held at MU 215. | |
| 1464 | student_club | challenging | ✓ | ✓ | 47 | 18983 | Write the full names of students who received funds on the date of 9/9/2019 and include the amount received. | |
| 1472 | debit_card_specializing | moderate | ✗ | ✓ | 605 | 11949 | In 2012, who had the least consumption in LAM? | |
| 1473 | debit_card_specializing | moderate | ✓ | ✓ | 1171 | 12468 | What was the average monthly consumption of customers in SME for the year 2013? | |
| 1476 | debit_card_specializing | challenging | ✓ | ✓ | 942 | 12901 | What was the difference in gas consumption between CZK-paying customers and EUR-paying customers in 2012? | |
| 1479 | debit_card_specializing | moderate | ✓ | ✓ | 1229 | 11974 | Which year recorded the most consumption of gas paid in CZK? | |
| 1480 | debit_card_specializing | moderate | ✗ | ✓ | 1279 | 12354 | What was the gas consumption peak month for SME customers in 2013? | |
| 1484 | debit_card_specializing | simple | ✓ | ✓ | 53 | 12321 | How many more "discount" gas stations does the Czech Republic have compared to Slovakia? | |
| 1486 | debit_card_specializing | simple | ✗ | ✓ | 53 | 12230 | Is it true that more SMEs pay in Czech koruna than in euros? If so, how many more? | |
| 1493 | debit_card_specializing | simple | ✓ | ✓ | 419 | 12546 | In February 2012, what percentage of customers consumed more than 528.3? | |
| 1500 | debit_card_specializing | simple | ✓ | ✓ | 74 | 12403 | Please list the product description of the products consumed in September, 2013. | |
| 1501 | debit_card_specializing | moderate | ✓ | ✓ | 59 | 12359 | Please list the countries of the gas stations with transactions taken place in June, 2013. | |
| 1506 | debit_card_specializing | moderate | ✓ | ✓ | 63 | 12157 | Please list the product descriptions of the transactions taken place in the gas stations in the Czech Republic. | |
| 1515 | debit_card_specializing | simple | ✓ | ✓ | 41 | 12109 | What segment did the customer have at 2012/8/23 21:20:00? | |
| 1521 | debit_card_specializing | moderate | ✓ | ✓ | 43 | 12840 | For all the transactions happened during 8:00-9:00 in 2012/8/26, how many happened in CZE? | |
| 1525 | debit_card_specializing | simple | ✗ | ✓ | 49 | 12330 | What is the percentage of the customers who used EUR in 2012/8/25? | |
| 1526 | debit_card_specializing | challenging | ✗ | ✓ | empty_result | 81 | 13256 | For the customer who paid 634.8 in 2012/8/25, what was the consumption decrease rate from Year 2012 to 2013? |
| 1528 | debit_card_specializing | simple | ✓ | ✓ | 64 | 11856 | What is the percentage of "premium" against the overall segment in Country = "SVK"? | |
| 1529 | debit_card_specializing | moderate | ✗ | ✓ | 99 | 11857 | What is the amount spent by customer "38508" at the gas stations? How much had the customer spent in January 2012? | |
| 1531 | debit_card_specializing | moderate | ✗ | ✓ | 1947 | 12379 | Who is the top spending customer and how much is the average price per single item purchased by this customer? What curr |
Source: BIRD Mini-Dev (SQLite). Methodology: docs/03_eval_methodology.md.
| Configuration | Model | n | EA | Simple | Moderate | Challenging | Validity | Recall@k | Empty % | P50 latency | P95 latency |
|---|---|---|---|---|---|---|---|---|---|---|---|
| C_dense_cards | z-ai/glm-4.5-air:free | 5 | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 100.0% | 0.0% | 40977 ms | 358146 ms |
| C_dense_cards | deepseek/deepseek-v4-flash:free | 20 | 0.0% | 0.0% | 0.0% | 0.0% | 95.0% | 5.0% | 0.0% | 2077 ms | 5058 ms |
Model: z-ai/glm-4.5-air:free · n=5 · EA=0.0% · Validity=0.0% · Recall@k=100.0%
| qid | db | diff | match | recall | err | lat ms | tokens | question |
|---|---|---|---|---|---|---|---|---|
| 50 | california_schools | simple | ✗ | ✓ | invalid_sql | 433948 | 6395 | What is the postal street address for the school with the 7th highest Math average? Indicate the school's name. |
| 236 | toxicology | moderate | ✗ | ✓ | invalid_sql | 34832 | 3157 | What are the bond type and the atoms of the bond ID of TR001_6_9? |
| 260 | toxicology | moderate | ✗ | ✓ | invalid_sql | 32067 | 3172 | Calculate the total atoms with triple-bond molecules containing the element phosphorus or bromine. |
| 414 | card_games | simple | ✗ | ✓ | invalid_sql | 54940 | 8346 | What language is the set of 180 cards that belongs to the Ravnica block translated into? |
| 1029 | european_football_2 | moderate | ✗ | ✓ | invalid_sql | 40977 | 11160 | What are the speed in which attacks are put together of the top 4 teams with the highest build Up Play Speed? |
Model: deepseek/deepseek-v4-flash:free · n=20 · EA=0.0% · Validity=95.0% · Recall@k=5.0%
| qid | db | diff | match | recall | err | lat ms | tokens | question |
|---|---|---|---|---|---|---|---|---|
| 50 | california_schools | simple | ✗ | ✗ | pipeline_exception | 4493 | 0 | What is the postal street address for the school with the 7th highest Math average? Indicate the school's name. |
| 173 | financial | challenging | ✗ | ✓ | invalid_sql | 15807 | 4864 | How often does account number 3 request an account statement to be released? What was the aim of debiting 3539 in total? |
| 236 | toxicology | moderate | ✗ | ✗ | pipeline_exception | 2923 | 0 | What are the bond type and the atoms of the bond ID of TR001_6_9? |
| 260 | toxicology | moderate | ✗ | ✗ | pipeline_exception | 3109 | 0 | Calculate the total atoms with triple-bond molecules containing the element phosphorus or bromine. |
| 407 | card_games | moderate | ✗ | ✗ | pipeline_exception | 2928 | 0 | Lists all types of cards in German. |
| 408 | card_games | moderate | ✗ | ✗ | pipeline_exception | 2851 | 0 | How many unknown power cards contain info about the triggered ability |
| 414 | card_games | simple | ✗ | ✗ | pipeline_exception | 2396 | 0 | What language is the set of 180 cards that belongs to the Ravnica block translated into? |
| 571 | codebase_community | moderate | ✗ | ✗ | pipeline_exception | 1845 | 0 | For the user No.24, how many times is the number of his/her posts compared to his/her votes? |
| 634 | codebase_community | challenging | ✗ | ✗ | pipeline_exception | 1858 | 0 | Among posts by Harvey Motulsky and Noah Snyder, which one has higher popularity? |
| 672 | codebase_community | moderate | ✗ | ✗ | pipeline_exception | 2150 | 0 | Among the users located in United Kingdom, how many users whose post have a total favorite amount of 4 or more? |
| 896 | formula_1 | challenging | ✗ | ✗ | pipeline_exception | 1901 | 0 | Calculate the percentage whereby Hamilton was not at the 1st track of the the f1 circuit since 2010. |
| 971 | formula_1 | simple | ✗ | ✗ | pipeline_exception | 1919 | 0 | Please state the reference name of the oldest German driver. |
| 1029 | european_football_2 | moderate | ✗ | ✗ | pipeline_exception | 2036 | 0 | What are the speed in which attacks are put together of the top 4 teams with the highest build Up Play Speed? |
| 1094 | european_football_2 | challenging | ✗ | ✗ | pipeline_exception | 1876 | 0 | How much higher in percentage is Ariel Borysiuk's overall rating than that of Paulin Puel? |
| 1232 | thrombosis_prediction | challenging | ✗ | ✗ | pipeline_exception | 1953 | 0 | Provide ID, sex and age of patient who has blood glucose (GLU) not within normal range but with total cholesterol(T-CHO) |
| 1254 | thrombosis_prediction | moderate | ✗ | ✗ | pipeline_exception | 1922 | 0 | How many patients with a normal Ig A level came to the hospital after 1990/1/1? |
| 1387 | student_club | moderate | ✗ | ✗ | pipeline_exception | 2532 | 0 | Which student has been entrusted to manage the budget for the Yearly Kickoff? |
| 1506 | debit_card_specializing | moderate | ✗ | ✗ | pipeline_exception | 2087 | 0 | Please list the product descriptions of the transactions taken place in the gas stations in the Czech Republic. |
| 1525 | debit_card_specializing | simple | ✗ | ✗ | pipeline_exception | 2067 | 0 | What is the percentage of the customers who used EUR in 2012/8/25? |
| 1528 | debit_card_specializing | simple | ✗ | ✗ | pipeline_exception | 1864 | 0 | What is the percentage of "premium" against the overall segment in Country = "SVK"? |
Source: BIRD Mini-Dev (SQLite). Methodology: docs/03_eval_methodology.md.
| Configuration | Model | n | EA | Simple | Moderate | Challenging | Validity | Recall@k | Empty % | P50 latency | P95 latency |
|---|---|---|---|---|---|---|---|---|---|---|---|
| C_dense_cards | codestral-latest | 200 | 56.5% | 70.1% | 52.5% | 41.2% | 100.0% | 100.0% | 2.5% | 26 ms | 842 ms |
| C_dense_cards | llama3.1:8b | 5 | 0.0% | 0.0% | 0.0% | 0.0% | 100.0% | 0.0% | 0.0% | 47070 ms | 47377 ms |
Model: codestral-latest · n=200 · EA=56.5% · Validity=100.0% · Recall@k=100.0%
| qid | db | diff | match | recall | err | lat ms | tokens | question |
|---|---|---|---|---|---|---|---|---|
| 5 | california_schools | simple | ✓ | ✓ | 126 | 6355 | How many schools with an average score in Math greater than 400 in the SAT test are exclusively virtual? | |
| 25 | california_schools | moderate | ✗ | ✓ | 25 | 6450 | Name schools in Riverside which the average of average math score for SAT is grater than 400, what is the funding type o | |
| 32 | california_schools | moderate | ✗ | ✓ | 21 | 6650 | What is the eligible free or reduced price meal rate for the top 5 schools in grades 1-12 with the highest free or reduc | |
| 36 | california_schools | challenging | ✓ | ✓ | 12 | 6595 | Under whose administration is the school with the highest number of students scoring 1500 or more on the SAT? Indicate t | |
| 37 | california_schools | moderate | ✗ | ✓ | 13 | 6477 | What is the complete address of the school with the lowest excellence rate? Indicate the Street, City, Zip and State. | |
| 39 | california_schools | simple | ✓ | ✓ | 23 | 6530 | What is the average number of test takers from Fresno schools that opened between 1/1/1980 and 12/31/1980? | |
| 48 | california_schools | moderate | ✓ | ✓ | 20 | 6470 | What is the ratio of merged Unified School District schools in Orange County to merged Elementary School District school | |
| 50 | california_schools | simple | ✗ | ✓ | 11 | 6383 | What is the postal street address for the school with the 7th highest Math average? Indicate the school's name. | |
| 77 | california_schools | moderate | ✗ | ✓ | empty_result | 16 | 6504 | Which schools served a grade span of Kindergarten to 9th grade in the county of Los Angeles and what is its Percent (%) |
| 92 | financial | simple | ✓ | ✓ | 13 | 4538 | List out the no. of districts that have female average salary is more than 6000 but less than 10000? | |
| 98 | financial | moderate | ✓ | ✓ | 11 | 4558 | Among the accounts who have approved loan date in 1997, list out the accounts that have the lowest approved amount and c | |
| 99 | financial | moderate | ✓ | ✓ | 9 | 4549 | Among the accounts who have loan validity more than 12 months, list out the accounts that have the highest approved amou | |
| 112 | financial | simple | ✓ | ✓ | 12 | 4551 | For the female client who was born in 1976/1/29, which district did she opened her account? | |
| 115 | financial | challenging | ✗ | ✓ | 14 | 4606 | For the branch which located in the south Bohemia with biggest number of inhabitants, what is the percentage of the male | |
| 118 | financial | moderate | ✓ | ✓ | 17 | 4568 | For loan amount less than USD100,000, what is the percentage of accounts that is still running with no issue. | |
| 120 | financial | moderate | ✗ | ✓ | 29 | 4881 | From Year 1995 to 2000, who are the accounts holders from 'east Bohemia'. State the account ID the frequency of statemen | |
| 125 | financial | challenging | ✗ | ✓ | execution_failed | 16 | 4382 | For loans contracts which are still running where client are in debt, list the district of the and the state the percent |
| 138 | financial | moderate | ✓ | ✓ | 22 | 4526 | In the branch where the second-highest number of crimes were committed in 1995 occurred, how many male clients are there | |
| 159 | financial | simple | ✗ | ✓ | 622 | 4668 | List all the withdrawals in cash transactions that the client with the id 3356 makes. | |
| 168 | financial | moderate | ✗ | ✓ | 28 | 4539 | What percentage of clients who opened their accounts in the district with an average salary of over 10000 are women? | |
| 169 | financial | challenging | ✗ | ✓ | 36 | 4783 | What was the growth rate of the total amount of loans across all accounts for a male client between 1996 and 1997? | |
| 173 | financial | challenging | ✗ | ✓ | 165 | 4663 | How often does account number 3 request an account statement to be released? What was the aim of debiting 3539 in total? | |
| 189 | financial | moderate | ✗ | ✓ | execution_failed | 24 | 4247 | Name the account numbers of female clients who are oldest and have lowest average salary? |
| 192 | financial | moderate | ✓ | ✓ | 18 | 4582 | What is the average amount of loan which are still on running contract with statement issuance after each transaction? | |
| 194 | financial | moderate | ✓ | ✓ | 24 | 4514 | Provide the IDs and age of the client with high level credit card, which is eligible for loans. | |
| 207 | toxicology | challenging | ✗ | ✓ | 87 | 2669 | What elements are in a double type bond? | |
| 208 | toxicology | moderate | ✓ | ✓ | 27 | 2641 | Which type of label is the most numerous in atoms with hydrogen? | |
| 219 | toxicology | challenging | ✗ | ✓ | execution_failed | 25 | 2438 | What is the percentage of carcinogenic molecules in triple type bonds? |
| 227 | toxicology | simple | ✓ | ✓ | 14 | 2682 | What is the percentage of molecules that are carcinogenic? Please provide your answer as a percentage with three decimal | |
| 230 | toxicology | challenging | ✓ | ✓ | 14 | 2648 | What are the elements of the toxicology and label of molecule TR060? | |
| 232 | toxicology | moderate | ✗ | ✓ | execution_failed | 23 | 2420 | Please list top three molecules that have single bonds between two atoms and are not carcinogenic in alphabetical order. |
| 236 | toxicology | moderate | ✓ | ✓ | 23 | 2704 | What are the bond type and the atoms of the bond ID of TR001_6_9? | |
| 239 | toxicology | simple | ✓ | ✓ | 24 | 2621 | How many connections does the atom 19 have? | |
| 253 | toxicology | challenging | ✓ | ✓ | 48 | 2634 | List the elements of all the triple bonds. | |
| 260 | toxicology | moderate | ✓ | ✓ | 36 | 2718 | Calculate the total atoms with triple-bond molecules containing the element phosphorus or bromine. | |
| 268 | toxicology | challenging | ✓ | ✓ | 26 | 2705 | What are the elements for bond id TR001_10_11? | |
| 273 | toxicology | moderate | ✗ | ✓ | 28 | 2723 | What is the percentage of element chlorine in carcinogenic molecules? | |
| 282 | toxicology | challenging | ✓ | ✓ | 20 | 2780 | What is the ratio of Hydrogen elements in molecule ID TR006? List the ratio with its label. | |
| 327 | toxicology | moderate | ✓ | ✓ | 30 | 2728 | Which non-carcinogenic molecules consisted more than 5 atoms? | |
| 347 | card_games | moderate | ✗ | ✓ | 646 | 8906 | Find all cards illustrated by Stephen Daniel and describe the text of the ruling of these cards. State if these cards ha | |
| 349 | card_games | moderate | ✗ | ✓ | 795 | 8562 | Name the card and artist with the most ruling information. Also state if the card is a promotional printing. | |
| 352 | card_games | moderate | ✗ | ✓ | 1063 | 8545 | Calculate the percentage of the cards availabe in Chinese Simplified. | |
| 356 | card_games | simple | ✓ | ✓ | 234 | 8379 | How many cards have infinite power? | |
| 358 | card_games | simple | ✓ | ✓ | 251 | 8434 | What is the border color of card "Ancestor's Chosen"? | |
| 366 | card_games | simple | ✗ | ✓ | 1415 | 8502 | What is the rule of playing card "Benalish Knight"? | |
| 377 | card_games | simple | ✗ | ✓ | 234 | 8446 | How many cards with original type of "Summon - Angel" have subtype other than "Angel"? | |
| 391 | card_games | moderate | ✓ | ✓ | 1028 | 8571 | Among the Artifact cards, which are black color and comes with foreign languague translation? | |
| 407 | card_games | moderate | ✗ | ✓ | 603 | 8566 | Lists all types of cards in German. | |
| 408 | card_games | moderate | ✗ | ✓ | 392 | 8463 | How many unknown power cards contain info about the triggered ability | |
| 412 | card_games | moderate | ✗ | ✓ | 871 | 8620 | What is the foreign name of the card in French of type Creature, normal layout and black border color, by artist Matthew | |
| 414 | card_games | simple | ✓ | ✓ | 57 | 8539 | What language is the set of 180 cards that belongs to the Ravnica block translated into? | |
| 427 | card_games | moderate | ✓ | ✓ | 55 | 8565 | What languages are available in the set known as Archenemy on the magic card market and having the code ARC? | |
| 459 | card_games | moderate | ✗ | ✓ | 234 | 8544 | Which card costs more converted mana, "Serra Angel" or "Shrine Keeper"? | |
| 466 | card_games | moderate | ✓ | ✓ | 233 | 8548 | Among the cards in the set "Hauptset Zehnte Edition", how many of them are designed by Adam Rex? | |
| 472 | card_games | moderate | ✓ | ✓ | 54 | 8530 | Among the sets in the block "Ice Age", how many of them have an Italian translation? | |
| 484 | card_games | moderate | ✗ | ✓ | 840 | 8575 | Please list the Italian names of the cards in the set Coldsnap with the highest converted mana cost. | |
| 486 | card_games | moderate | ✓ | ✓ | 328 | 8651 | What is the percentage of the cards with a converted mana cost of 7 in the set Coldsnap? | |
| 518 | card_games | moderate | ✗ | ✓ | 60248 | 8609 | Which of the play format has the highest number of banned status? Indicate the play format and the names of all the card | |
| 531 | codebase_community | simple | ✗ | ✓ | 34 | 5676 | Which user has a higher reputation, Harlan or Jarrod Dixon? | |
| 557 | codebase_community | moderate | ✗ | ✓ | 277 | 6420 | Among the posts with a score of over 5, what is the percentage of them being owned by an elder user? | |
| 563 | codebase_community | moderate | ✗ | ✓ | empty_result | 88 | 6458 | User No.3025 gave a comment at 20:29:39 on 2014/4/23 to a post, how many favorite counts did that post get? |
| 571 | codebase_community | moderate | ✗ | ✓ | 198 | 6349 | For the user No.24, how many times is the number of his/her posts compared to his/her votes? | |
| 584 | codebase_community | moderate | ✗ | ✓ | 523 | 6483 | Write all the comments left by users who edited the post titled 'Why square the difference instead of taking the absolut | |
| 595 | codebase_community | moderate | ✗ | ✓ | 777 | 6384 | Which user have only one post history per post and having at least 1000 views? | |
| 634 | codebase_community | challenging | ✗ | ✓ | empty_result | 372 | 6305 | Among posts by Harvey Motulsky and Noah Snyder, which one has higher popularity? |
| 669 | codebase_community | simple | ✓ | ✓ | 15 | 5678 | When did 'chl' cast its first vote in a post? | |
| 671 | codebase_community | simple | ✓ | ✓ | 33 | 5691 | What is the display name of the user who acquired the first Autobiographer badge? | |
| 672 | codebase_community | moderate | ✗ | ✓ | 174 | 6291 | Among the users located in United Kingdom, how many users whose post have a total favorite amount of 4 or more? | |
| 694 | codebase_community | moderate | ✗ | ✓ | 360 | 6569 | Provide the text of the latest 10 comments to the post with the title 'Analysing wind data with R' and the display name | |
| 707 | codebase_community | moderate | ✓ | ✓ | 303 | 6462 | Among the posts with views ranging from 100 to 150, what is the comment with the highest score? | |
| 716 | codebase_community | moderate | ✗ | ✓ | 83 | 6420 | Among the comments with scores between 5 to 10, what is the percentage of the users with 0 up votes? | |
| 723 | superhero | moderate | ✓ | ✓ | 15 | 3572 | Among the superheroes with blue eyes, how many of them have the super power of "Agility"? | |
| 730 | superhero | challenging | ✓ | ✓ | 110 | 3599 | List the superheroes from Marvel Comics who have the super power of 'Super Strength'. | |
| 736 | superhero | moderate | ✓ | ✓ | 10 | 3445 | Who is the dumbest superhero? | |
| 737 | superhero | simple | ✓ | ✓ | 8 | 3381 | What is Copycat's race? | |
| 738 | superhero | simple | ✓ | ✓ | 92 | 3545 | Which superheroes have a durability attribute value of less than 50? | |
| 743 | superhero | challenging | ✗ | ✓ | 12 | 3624 | What is the percentage of superheroes who act in their own self-interest or make decisions based on their own moral code | |
| 747 | superhero | simple | ✓ | ✓ | 8 | 3381 | What is the total number of superheroes without full name? | |
| 750 | superhero | simple | ✓ | ✓ | 8 | 3444 | What is the average weight of all female superheroes? | |
| 751 | superhero | moderate | ✓ | ✓ | 9 | 3529 | List down at least five superpowers of male superheroes. | |
| 753 | superhero | moderate | ✓ | ✓ | 8 | 3583 | Among the superheroes with height from 170 to 190, list the names of the superheroes with no eye color. | |
| 765 | superhero | simple | ✓ | ✓ | 9 | 3426 | How many heroes have stealth power? | |
| 773 | superhero | challenging | ✓ | ✓ | 8 | 3524 | Which superhero has the same eyes, hair and skin colour? Indicate the publisher of the superhero. | |
| 775 | superhero | challenging | ✓ | ✓ | 10 | 3629 | What is the percentage of blue female superheroes among all female superheroes? | |
| 781 | superhero | simple | ✓ | ✓ | 10 | 3487 | Provide the heights of the heroes whose eye colours are amber. | |
| 785 | superhero | simple | ✓ | ✓ | 9 | 3452 | Describe the names of neutral alignment superheroes. | |
| 791 | superhero | simple | ✗ | ✓ | 8 | 3402 | Calculate the average height for all superhero. | |
| 794 | superhero | moderate | ✓ | ✓ | 10 | 3453 | Which hero was the fastest? | |
| 798 | superhero | moderate | ✓ | ✓ | 8 | 3503 | What is the publisher for Hawkman, Karate Kid and Speedy? | |
| 800 | superhero | moderate | ✓ | ✓ | 12 | 3548 | Calculate the percentage of superheroes with blue eyes. | |
| 806 | superhero | simple | ✓ | ✓ | 8 | 3379 | Provide the eye colour of the superhero who has Karen Beecher-Duncan as their full name. | |
| 819 | superhero | challenging | ✓ | ✓ | 9 | 3677 | In superheroes with missing weight data, calculate the difference between the number of superheroes with blue eyes and n | |
| 825 | superhero | moderate | ✓ | ✓ | 13 | 3498 | Identify the gender of the superhero who has the ability of Phoenix Force. | |
| 847 | formula_1 | simple | ✗ | ✓ | 16 | 6661 | What is the surname of the driver with the best lap time in race number 19 in the second qualifying period? | |
| 859 | formula_1 | simple | ✓ | ✓ | 16 | 6659 | What's Bruno Senna's Q1 result in the qualifying race No. 354? | |
| 861 | formula_1 | simple | ✗ | ✓ | empty_result | 13 | 6661 | What is his number of the driver who finished 0:01:54 in the Q3 of qualifying race No.903? |
| 862 | formula_1 | simple | ✓ | ✓ | 17 | 6650 | For the Bahrain Grand Prix in 2007, how many drivers not finished the game? | |
| 865 | formula_1 | moderate | ✗ | ✓ | 13 | 6708 | For all the drivers who finished the game in race No. 592, who is the oldest? | |
| 866 | formula_1 | moderate | ✗ | ✓ | 9 | 6757 | Who was the player that got the lap time of 0:01:27 in the race No. 161? Show his introduction website. | |
| 875 | formula_1 | simple | ✓ | ✓ | 8 | 6603 | Show me the season page of year when the race No. 901 took place. | |
| 877 | formula_1 | moderate | ✓ | ✓ | 12 | 6656 | For all the drivers who finished the game in race No. 872, who is the youngest? | |
| 879 | formula_1 | moderate | ✓ | ✓ | 19 | 6602 | For the driver who set the fastest lap speed, what is his nationality? | |
| 881 | formula_1 | moderate | ✓ | ✓ | 17 | 6770 | For the drivers who took part in the race in 1983/7/16, what's their race completion rate? | |
| 894 | formula_1 | moderate | ✗ | ✓ | 18969 | 6717 | What is the best lap time recorded? List the driver and race with such recorded lap time. | |
| 896 | formula_1 | challenging | ✗ | ✓ | 4459 | 6780 | Calculate the percentage whereby Hamilton was not at the 1st track of the the f1 circuit since 2010. | |
| 897 | formula_1 | moderate | ✓ | ✓ | 6092 | 6707 | Name the driver with the most winning. Mention his nationality and what is his maximum point scores. | |
| 898 | formula_1 | simple | ✗ | ✓ | execution_failed | 10 | 6386 | How old is the youngest Japanese driver? What is his name? |
| 902 | formula_1 | simple | ✗ | ✓ | 21 | 6717 | Which race was Alex Yoong in when he was in track number less than 20? | |
| 904 | formula_1 | moderate | ✗ | ✓ | 4318 | 6487 | State the race and year of race in which Michael Schumacher had his fastest lap. | |
| 909 | formula_1 | moderate | ✓ | ✓ | 31 | 6750 | Among all European Grand Prix races, what is the percentage of the races were hosted in Germany? | |
| 912 | formula_1 | simple | ✓ | ✓ | 24 | 6306 | What's the reference name of Marina Bay Street Circuit? | |
| 915 | formula_1 | simple | ✗ | ✓ | 44 | 6614 | Which country is the oldest driver from? | |
| 930 | formula_1 | simple | ✗ | ✓ | 33 | 6652 | In which Formula_1 race did Lewis Hamilton rank the highest? | |
| 945 | formula_1 | simple | ✓ | ✓ | 19 | 6277 | How many circuits are there in Adelaide, Australia? | |
| 950 | formula_1 | simple | ✗ | ✓ | 21 | 6632 | Please list the constructor names with 0 points at race 291. | |
| 959 | formula_1 | simple | ✗ | ✓ | 34 | 6710 | What is the fastest lap number of the champion in 2009? | |
| 971 | formula_1 | simple | ✓ | ✓ | 2318 | 6527 | Please state the reference name of the oldest German driver. | |
| 981 | formula_1 | moderate | ✓ | ✓ | 21 | 6748 | On what year did the youngest driver had his first qualifying race? Also state the name, date and time of the race. | |
| 988 | formula_1 | challenging | ✗ | ✓ | 16 | 6641 | List down top 3 German drivers who has the shortest average pit stop duration and were born between 1980-1985. | |
| 989 | formula_1 | moderate | ✗ | ✓ | 36 | 6699 | Who is the champion of the Canadian Grand Prix in 2008? Indicate his finish time. | |
| 990 | formula_1 | challenging | ✓ | ✓ | 35 | 6733 | What is the constructor reference name of the champion in the 2009 Singapore Grand Prix? Please give its website. | |
| 1028 | european_football_2 | challenging | ✗ | ✓ | 176 | 12183 | In Scotland Premier League, which away team won the most during the 2010 season? | |
| 1029 | european_football_2 | moderate | ✗ | ✓ | 31 | 12055 | What are the speed in which attacks are put together of the top 4 teams with the highest build Up Play Speed? | |
| 1030 | european_football_2 | moderate | ✓ | ✓ | 245 | 12015 | Give the name of the league had the most matches end as draw in the 2016 season? | |
| 1035 | european_football_2 | simple | ✓ | ✓ | 30 | 11969 | Give the team_fifa_api_id of teams with more than 50 but less than 60 build-up play speed. | |
| 1036 | european_football_2 | challenging | ✗ | ✓ | execution_failed | 31 | 11699 | List the long name of teams with above-average build-up play passing in 2012. |
| 1037 | european_football_2 | challenging | ✗ | ✓ | 424 | 12142 | Calculate the percentage of players who prefer left foot, who were born between 1987 and 1992. | |
| 1039 | european_football_2 | simple | ✓ | ✓ | 300 | 11940 | Find the average number of long-shot done by Ahmed Samir Farag. | |
| 1042 | european_football_2 | challenging | ✓ | ✓ | 250 | 12195 | List the name of leagues in which the average goals by the home team is higher than the away team in the 2009/2010 seaso | |
| 1057 | european_football_2 | moderate | ✓ | ✓ | 216 | 12032 | Calculate the average home team goal in the 2010/2011 season in the country of Poland. | |
| 1078 | european_football_2 | simple | ✓ | ✓ | 33 | 11796 | Which player is older, Aaron Lennon or Abdelaziz Barrada? | |
| 1088 | european_football_2 | moderate | ✗ | ✓ | 249 | 12029 | Please list the names of the players whose volley score and dribbling score are over 70. | |
| 1094 | european_football_2 | challenging | ✗ | ✓ | execution_failed | 240 | 11795 | How much higher in percentage is Ariel Borysiuk's overall rating than that of Paulin Puel? |
| 1103 | european_football_2 | moderate | ✓ | ✓ | 267 | 11996 | What was the overall rating for Aaron Mooy on 2016/2/4? | |
| 1110 | european_football_2 | moderate | ✓ | ✓ | 22 | 11932 | Tell the build Up play passing class for "FC Lorient" on 2010/2/22. | |
| 1116 | european_football_2 | simple | ✓ | ✓ | 38 | 11857 | List down most tallest players' name. | |
| 1122 | european_football_2 | simple | ✓ | ✓ | 338 | 11792 | State the name of the most strongest player. | |
| 1130 | european_football_2 | moderate | ✓ | ✓ | 23 | 11959 | What are the short name of team who played safe while creating chance of passing? | |
| 1133 | european_football_2 | simple | ✓ | ✓ | 29 | 11827 | How many football players born after the 1990s have the first name "Aaron"? | |
| 1141 | european_football_2 | moderate | ✓ | ✓ | 25 | 11878 | Does the KSV Cercle Brugge team have a slow, balanced or fast speed class? | |
| 1144 | european_football_2 | simple | ✗ | ✓ | 201 | 11970 | Please state the finishing rate and curve score of the player who has the heaviest weight. | |
| 1146 | european_football_2 | moderate | ✓ | ✓ | 200 | 11938 | Please provide the full name of the away team that scored the most goals. | |
| 1147 | european_football_2 | simple | ✓ | ✓ | 318 | 11791 | Please name one player whose overall strength is the greatest. | |
| 1152 | thrombosis_prediction | moderate | ✓ | ✓ | 22 | 4895 | What is the ratio of outpatient to inpatient followed up treatment among all the 'SLE' diagnosed patient? | |
| 1156 | thrombosis_prediction | moderate | ✗ | ✓ | execution_failed | 15 | 4489 | State the ID and age of patient with positive degree of coagulation. |
| 1157 | thrombosis_prediction | simple | ✗ | ✓ | 16 | 4787 | For patients with severe degree of thrombosis, list their ID, sex and disease the patient is diagnosed with. | |
| 1168 | thrombosis_prediction | challenging | ✗ | ✓ | execution_failed | 19 | 4548 | The oldest SJS patient's medical laboratory work was completed on what date, and what age was the patient when they init |
| 1185 | thrombosis_prediction | challenging | ✗ | ✓ | 20 | 5245 | For the patient who was born on 1959/2/18, what is the decrease rate for his/her total cholesterol from November to Dece | |
| 1198 | thrombosis_prediction | simple | ✓ | ✓ | 25 | 4666 | How many female patients were given an APS diagnosis? | |
| 1205 | thrombosis_prediction | moderate | ✗ | ✓ | 25 | 4854 | Was the patient with the number 57266's uric acid within a normal range? | |
| 1208 | thrombosis_prediction | moderate | ✓ | ✓ | 27 | 4863 | Provide IDs for male patients with ALT glutamic pylvic transaminase (GPT) that have history of ALT glutamic pylvic trans | |
| 1220 | thrombosis_prediction | simple | ✓ | ✓ | 33 | 4892 | Provide all ID, sex and birthday of patients whose urea nitrogen (UN) just within the borderline of passing? | |
| 1227 | thrombosis_prediction | moderate | ✗ | ✓ | execution_failed | 26 | 4523 | What is the average age of the male patient with high cholesterol? |
| 1232 | thrombosis_prediction | challenging | ✓ | ✓ | 35 | 5013 | Provide ID, sex and age of patient who has blood glucose (GLU) not within normal range but with total cholesterol(T-CHO) | |
| 1235 | thrombosis_prediction | moderate | ✗ | ✓ | execution_failed | 23 | 4521 | What are the patient's diagnosis for those who has lower red blood blood cell? State their ID and age. |
| 1247 | thrombosis_prediction | challenging | ✗ | ✓ | 28 | 4879 | Among the male patients who have a normal level of white blood cells, how many of them have an abnormal fibrinogen level | |
| 1251 | thrombosis_prediction | simple | ✗ | ✓ | 27 | 4702 | How many patients with an Ig G higher than normal? | |
| 1252 | thrombosis_prediction | moderate | ✗ | ✓ | 17 | 4791 | Among the patients with a normal Ig G level, how many of them have symptoms? | |
| 1254 | thrombosis_prediction | moderate | ✗ | ✓ | execution_failed | 25 | 4547 | How many patients with a normal Ig A level came to the hospital after 1990/1/1? |
| 1255 | thrombosis_prediction | moderate | ✗ | ✓ | 26 | 4806 | For the patients with an abnormal Ig M level, what is the most common disease they are diagnosed with? | |
| 1257 | thrombosis_prediction | challenging | ✗ | ✓ | 35 | 4815 | Among the patients whose creatinine level is abnormal, how many of them aren't 70 yet? | |
| 1275 | thrombosis_prediction | moderate | ✗ | ✓ | execution_failed | 27 | 4554 | Among the patients who has a normal level of anti-centromere and a normal level of anti-SSB, how many of them are male? |
| 1281 | thrombosis_prediction | moderate | ✓ | ✓ | 36 | 4783 | Among the patients who have an abnormal level of glutamic oxaloacetic transaminase, when was the youngest of them born? | |
| 1302 | thrombosis_prediction | challenging | ✗ | ✓ | execution_failed | 19 | 4517 | For the patients with a normal range of creatinine phosphokinase, how many of them have a positive measure of degree of |
| 1312 | student_club | simple | ✓ | ✓ | 18 | 4710 | What's Angela Sanders's major? | |
| 1340 | student_club | moderate | ✓ | ✓ | 21 | 4946 | Calculate the difference of the total amount spent in all events by the Student_Club in year 2019 and 2020. | |
| 1344 | student_club | simple | ✓ | ✓ | 23 | 4770 | What was the notes of the fundraising on 2019/9/14? | |
| 1352 | student_club | moderate | ✓ | ✓ | 23 | 4753 | For all the club members from "Business" major, how many of them wear medium size t-shirt? | |
| 1356 | student_club | simple | ✓ | ✓ | 23 | 4687 | Which department was the President of the club in? | |
| 1376 | student_club | moderate | ✓ | ✓ | 25 | 4768 | Among all the closed events, which event has the highest spend-to-budget ratio? | |
| 1378 | student_club | simple | ✓ | ✓ | 20 | 4186 | What is the highest amount of budget spend for an event? | |
| 1380 | student_club | simple | ✓ | ✓ | 23 | 4455 | What is the total amount of money spent for food? | |
| 1387 | student_club | moderate | ✓ | ✓ | 27 | 4840 | Which student has been entrusted to manage the budget for the Yearly Kickoff? | |
| 1390 | student_club | moderate | ✓ | ✓ | 26 | 4376 | Based on the total cost for all event, what is the percentage of cost for Yearly Kickoff event? | |
| 1399 | student_club | moderate | ✗ | ✓ | 26 | 4791 | Did Maya Mclean attend the 'Women's Soccer' event? | |
| 1403 | student_club | moderate | ✓ | ✓ | 21 | 4780 | Indicate the name of the closed event whose cost has exceeded the budget the most. | |
| 1404 | student_club | moderate | ✗ | ✓ | 25 | 4857 | Identify the type of expenses and their total value approved for 'October Meeting' event. | |
| 1409 | student_club | simple | ✓ | ✓ | 17 | 4744 | Mention the total expense used on 8/20/2019. | |
| 1410 | student_club | simple | ✓ | ✓ | 16 | 4792 | List out the full name and total cost that member id "rec4BLdZHS2Blfp4v" incurred? | |
| 1411 | student_club | simple | ✓ | ✓ | 18 | 4775 | State what kind of expenses that Sacha Harrison incurred? | |
| 1422 | student_club | simple | ✗ | ✓ | 18 | 4700 | State the category of events were held at MU 215. | |
| 1464 | student_club | challenging | ✓ | ✓ | 23 | 4836 | Write the full names of students who received funds on the date of 9/9/2019 and include the amount received. | |
| 1472 | debit_card_specializing | moderate | ✗ | ✓ | 400 | 3075 | In 2012, who had the least consumption in LAM? | |
| 1473 | debit_card_specializing | moderate | ✓ | ✓ | 467 | 3137 | What was the average monthly consumption of customers in SME for the year 2013? | |
| 1476 | debit_card_specializing | challenging | ✓ | ✓ | 462 | 3286 | What was the difference in gas consumption between CZK-paying customers and EUR-paying customers in 2012? | |
| 1479 | debit_card_specializing | moderate | ✓ | ✓ | 660 | 3010 | Which year recorded the most consumption of gas paid in CZK? | |
| 1480 | debit_card_specializing | moderate | ✗ | ✓ | 520 | 3103 | What was the gas consumption peak month for SME customers in 2013? | |
| 1484 | debit_card_specializing | simple | ✓ | ✓ | 19 | 3085 | How many more "discount" gas stations does the Czech Republic have compared to Slovakia? | |
| 1486 | debit_card_specializing | simple | ✗ | ✓ | 24 | 3074 | Is it true that more SMEs pay in Czech koruna than in euros? If so, how many more? | |
| 1493 | debit_card_specializing | simple | ✓ | ✓ | 136 | 3144 | In February 2012, what percentage of customers consumed more than 528.3? | |
| 1500 | debit_card_specializing | simple | ✓ | ✓ | 33 | 3118 | Please list the product description of the products consumed in September, 2013. | |
| 1501 | debit_card_specializing | moderate | ✓ | ✓ | 28 | 3102 | Please list the countries of the gas stations with transactions taken place in June, 2013. | |
| 1506 | debit_card_specializing | moderate | ✓ | ✓ | 25 | 3057 | Please list the product descriptions of the transactions taken place in the gas stations in the Czech Republic. | |
| 1515 | debit_card_specializing | simple | ✓ | ✓ | 23 | 3011 | What segment did the customer have at 2012/8/23 21:20:00? | |
| 1521 | debit_card_specializing | moderate | ✓ | ✓ | 22 | 3254 | For all the transactions happened during 8:00-9:00 in 2012/8/26, how many happened in CZE? | |
| 1525 | debit_card_specializing | simple | ✗ | ✓ | 28 | 3102 | What is the percentage of the customers who used EUR in 2012/8/25? | |
| 1526 | debit_card_specializing | challenging | ✗ | ✓ | empty_result | 82 | 3267 | For the customer who paid 634.8 in 2012/8/25, what was the consumption decrease rate from Year 2012 to 2013? |
| 1528 | debit_card_specializing | simple | ✓ | ✓ | 22 | 2969 | What is the percentage of "premium" against the overall segment in Country = "SVK"? | |
| 1529 | debit_card_specializing | moderate | ✗ | ✓ | 88 | 3092 | What is the amount spent by customer "38508" at the gas stations? How much had the customer spent in January 2012? | |
| 1531 | debit_card_specializing | moderate | ✗ | ✓ | 120 | 3156 | Who is the top spending customer and how much is the average price per single item purchased by this customer? What curr |
Model: llama3.1:8b · n=5 · EA=0.0% · Validity=100.0% · Recall@k=0.0%
| qid | db | diff | match | recall | err | lat ms | tokens | question |
|---|---|---|---|---|---|---|---|---|
| 50 | california_schools | simple | ✗ | ✗ | pipeline_exception | 47453 | 0 | What is the postal street address for the school with the 7th highest Math average? Indicate the school's name. |
| 236 | toxicology | moderate | ✗ | ✗ | pipeline_exception | 47054 | 0 | What are the bond type and the atoms of the bond ID of TR001_6_9? |
| 260 | toxicology | moderate | ✗ | ✗ | pipeline_exception | 47071 | 0 | Calculate the total atoms with triple-bond molecules containing the element phosphorus or bromine. |
| 414 | card_games | simple | ✗ | ✗ | pipeline_exception | 47070 | 0 | What language is the set of 180 cards that belongs to the Ravnica block translated into? |
| 1029 | european_football_2 | moderate | ✗ | ✗ | pipeline_exception | 47069 | 0 | What are the speed in which attacks are put together of the top 4 teams with the highest build Up Play Speed? |
Source: BIRD Mini-Dev (SQLite). Methodology: docs/03_eval_methodology.md.
| Configuration | Model | n | EA | Simple | Moderate | Challenging | Validity | Recall@k | Empty % | P50 latency | P95 latency |
|---|---|---|---|---|---|---|---|---|---|---|---|
| C_dense_cards | codestral-latest | 1 | 0.0% | 0.0% | 0.0% | 0.0% | 100.0% | 100.0% | 0.0% | 15528 ms | 15528 ms |
| C_dense_cards | codestral-latest | 200 | 57.5% | 70.1% | 53.5% | 44.1% | 100.0% | 100.0% | 2.5% | 24 ms | 785 ms |
| C_dense_cards | codestral-latest | 2 | 100.0% | 0.0% | 100.0% | 100.0% | 100.0% | 100.0% | 0.0% | 1423 ms | 2651 ms |
| C_dense_cards | codestral-latest | 2 | 50.0% | 0.0% | 100.0% | 0.0% | 100.0% | 100.0% | 0.0% | 185 ms | 190 ms |
| C_dense_cards | codestral-latest | 1 | 0.0% | 0.0% | 0.0% | 0.0% | 100.0% | 100.0% | 0.0% | 11681 ms | 11681 ms |
Model: codestral-latest · n=1 · EA=0.0% · Validity=100.0% · Recall@k=100.0%
| qid | db | diff | match | recall | err | lat ms | tokens | question |
|---|---|---|---|---|---|---|---|---|
| 1399 | student_club | moderate | ✗ | ✓ | 15528 | 4895 | Did Maya Mclean attend the 'Women's Soccer' event? |
Model: codestral-latest · n=200 · EA=57.5% · Validity=100.0% · Recall@k=100.0%
| qid | db | diff | match | recall | err | lat ms | tokens | question |
|---|---|---|---|---|---|---|---|---|
| 5 | california_schools | simple | ✓ | ✓ | 154 | 6355 | How many schools with an average score in Math greater than 400 in the SAT test are exclusively virtual? | |
| 25 | california_schools | moderate | ✗ | ✓ | 46 | 6450 | Name schools in Riverside which the average of average math score for SAT is grater than 400, what is the funding type o | |
| 32 | california_schools | moderate | ✗ | ✓ | 46 | 6650 | What is the eligible free or reduced price meal rate for the top 5 schools in grades 1-12 with the highest free or reduc | |
| 36 | california_schools | challenging | ✓ | ✓ | 24 | 6595 | Under whose administration is the school with the highest number of students scoring 1500 or more on the SAT? Indicate t | |
| 37 | california_schools | moderate | ✗ | ✓ | 24 | 6477 | What is the complete address of the school with the lowest excellence rate? Indicate the Street, City, Zip and State. | |
| 39 | california_schools | simple | ✓ | ✓ | 57 | 6530 | What is the average number of test takers from Fresno schools that opened between 1/1/1980 and 12/31/1980? | |
| 48 | california_schools | moderate | ✓ | ✓ | 49 | 6470 | What is the ratio of merged Unified School District schools in Orange County to merged Elementary School District school | |
| 50 | california_schools | simple | ✗ | ✓ | 26 | 6383 | What is the postal street address for the school with the 7th highest Math average? Indicate the school's name. | |
| 77 | california_schools | moderate | ✗ | ✓ | empty_result | 37 | 6504 | Which schools served a grade span of Kindergarten to 9th grade in the county of Los Angeles and what is its Percent (%) |
| 92 | financial | simple | ✓ | ✓ | 33 | 4538 | List out the no. of districts that have female average salary is more than 6000 but less than 10000? | |
| 98 | financial | moderate | ✓ | ✓ | 34 | 4558 | Among the accounts who have approved loan date in 1997, list out the accounts that have the lowest approved amount and c | |
| 99 | financial | moderate | ✓ | ✓ | 24 | 4549 | Among the accounts who have loan validity more than 12 months, list out the accounts that have the highest approved amou | |
| 112 | financial | simple | ✓ | ✓ | 30 | 4551 | For the female client who was born in 1976/1/29, which district did she opened her account? | |
| 115 | financial | challenging | ✗ | ✓ | 32 | 4606 | For the branch which located in the south Bohemia with biggest number of inhabitants, what is the percentage of the male | |
| 118 | financial | moderate | ✓ | ✓ | 29 | 4568 | For loan amount less than USD100,000, what is the percentage of accounts that is still running with no issue. | |
| 120 | financial | moderate | ✗ | ✓ | 29 | 4881 | From Year 1995 to 2000, who are the accounts holders from 'east Bohemia'. State the account ID the frequency of statemen | |
| 125 | financial | challenging | ✗ | ✓ | execution_failed | 22 | 4382 | For loans contracts which are still running where client are in debt, list the district of the and the state the percent |
| 138 | financial | moderate | ✓ | ✓ | 22 | 4526 | In the branch where the second-highest number of crimes were committed in 1995 occurred, how many male clients are there | |
| 159 | financial | simple | ✗ | ✓ | 1848 | 4668 | List all the withdrawals in cash transactions that the client with the id 3356 makes. | |
| 168 | financial | moderate | ✗ | ✓ | 35 | 4539 | What percentage of clients who opened their accounts in the district with an average salary of over 10000 are women? | |
| 169 | financial | challenging | ✗ | ✓ | 42 | 4783 | What was the growth rate of the total amount of loans across all accounts for a male client between 1996 and 1997? | |
| 173 | financial | challenging | ✗ | ✓ | 137 | 4663 | How often does account number 3 request an account statement to be released? What was the aim of debiting 3539 in total? | |
| 189 | financial | moderate | ✗ | ✓ | execution_failed | 22 | 4247 | Name the account numbers of female clients who are oldest and have lowest average salary? |
| 192 | financial | moderate | ✓ | ✓ | 21 | 4582 | What is the average amount of loan which are still on running contract with statement issuance after each transaction? | |
| 194 | financial | moderate | ✓ | ✓ | 22 | 4514 | Provide the IDs and age of the client with high level credit card, which is eligible for loans. | |
| 207 | toxicology | challenging | ✓ | ✓ | 100 | 2695 | What elements are in a double type bond? | |
| 208 | toxicology | moderate | ✓ | ✓ | 27 | 2641 | Which type of label is the most numerous in atoms with hydrogen? | |
| 219 | toxicology | challenging | ✗ | ✓ | execution_failed | 24 | 2438 | What is the percentage of carcinogenic molecules in triple type bonds? |
| 227 | toxicology | simple | ✓ | ✓ | 16 | 2682 | What is the percentage of molecules that are carcinogenic? Please provide your answer as a percentage with three decimal | |
| 230 | toxicology | challenging | ✓ | ✓ | 19 | 2648 | What are the elements of the toxicology and label of molecule TR060? | |
| 232 | toxicology | moderate | ✗ | ✓ | execution_failed | 20 | 2420 | Please list top three molecules that have single bonds between two atoms and are not carcinogenic in alphabetical order. |
| 236 | toxicology | moderate | ✓ | ✓ | 23 | 2704 | What are the bond type and the atoms of the bond ID of TR001_6_9? | |
| 239 | toxicology | simple | ✓ | ✓ | 23 | 2621 | How many connections does the atom 19 have? | |
| 253 | toxicology | challenging | ✓ | ✓ | 47 | 2634 | List the elements of all the triple bonds. | |
| 260 | toxicology | moderate | ✓ | ✓ | 31 | 2718 | Calculate the total atoms with triple-bond molecules containing the element phosphorus or bromine. | |
| 268 | toxicology | challenging | ✓ | ✓ | 22 | 2705 | What are the elements for bond id TR001_10_11? | |
| 273 | toxicology | moderate | ✗ | ✓ | 24 | 2723 | What is the percentage of element chlorine in carcinogenic molecules? | |
| 282 | toxicology | challenging | ✓ | ✓ | 20 | 2780 | What is the ratio of Hydrogen elements in molecule ID TR006? List the ratio with its label. | |
| 327 | toxicology | moderate | ✓ | ✓ | 28 | 2728 | Which non-carcinogenic molecules consisted more than 5 atoms? | |
| 347 | card_games | moderate | ✗ | ✓ | 2435 | 8906 | Find all cards illustrated by Stephen Daniel and describe the text of the ruling of these cards. State if these cards ha | |
| 349 | card_games | moderate | ✗ | ✓ | 882 | 8562 | Name the card and artist with the most ruling information. Also state if the card is a promotional printing. | |
| 352 | card_games | moderate | ✗ | ✓ | 2537 | 8545 | Calculate the percentage of the cards availabe in Chinese Simplified. | |
| 356 | card_games | simple | ✓ | ✓ | 156 | 8379 | How many cards have infinite power? | |
| 358 | card_games | simple | ✓ | ✓ | 160 | 8434 | What is the border color of card "Ancestor's Chosen"? | |
| 366 | card_games | simple | ✗ | ✓ | 1746 | 8502 | What is the rule of playing card "Benalish Knight"? | |
| 377 | card_games | simple | ✗ | ✓ | 152 | 8446 | How many cards with original type of "Summon - Angel" have subtype other than "Angel"? | |
| 391 | card_games | moderate | ✓ | ✓ | 716 | 8571 | Among the Artifact cards, which are black color and comes with foreign languague translation? | |
| 407 | card_games | moderate | ✗ | ✓ | 486 | 8566 | Lists all types of cards in German. | |
| 408 | card_games | moderate | ✗ | ✓ | 206 | 8463 | How many unknown power cards contain info about the triggered ability | |
| 412 | card_games | moderate | ✗ | ✓ | 432 | 8620 | What is the foreign name of the card in French of type Creature, normal layout and black border color, by artist Matthew | |
| 414 | card_games | simple | ✓ | ✓ | 39 | 8539 | What language is the set of 180 cards that belongs to the Ravnica block translated into? | |
| 427 | card_games | moderate | ✓ | ✓ | 41 | 8565 | What languages are available in the set known as Archenemy on the magic card market and having the code ARC? | |
| 459 | card_games | moderate | ✗ | ✓ | 149 | 8544 | Which card costs more converted mana, "Serra Angel" or "Shrine Keeper"? | |
| 466 | card_games | moderate | ✓ | ✓ | 143 | 8548 | Among the cards in the set "Hauptset Zehnte Edition", how many of them are designed by Adam Rex? | |
| 472 | card_games | moderate | ✓ | ✓ | 43 | 8530 | Among the sets in the block "Ice Age", how many of them have an Italian translation? | |
| 484 | card_games | moderate | ✗ | ✓ | 780 | 8575 | Please list the Italian names of the cards in the set Coldsnap with the highest converted mana cost. | |
| 486 | card_games | moderate | ✓ | ✓ | 399 | 8651 | What is the percentage of the cards with a converted mana cost of 7 in the set Coldsnap? | |
| 518 | card_games | moderate | ✗ | ✓ | 60285 | 8609 | Which of the play format has the highest number of banned status? Indicate the play format and the names of all the card | |
| 531 | codebase_community | simple | ✗ | ✓ | 55 | 5676 | Which user has a higher reputation, Harlan or Jarrod Dixon? | |
| 557 | codebase_community | moderate | ✗ | ✓ | 478 | 6420 | Among the posts with a score of over 5, what is the percentage of them being owned by an elder user? | |
| 563 | codebase_community | moderate | ✗ | ✓ | empty_result | 617 | 6458 | User No.3025 gave a comment at 20:29:39 on 2014/4/23 to a post, how many favorite counts did that post get? |
| 571 | codebase_community | moderate | ✗ | ✓ | 266 | 6349 | For the user No.24, how many times is the number of his/her posts compared to his/her votes? | |
| 584 | codebase_community | moderate | ✗ | ✓ | 1715 | 6483 | Write all the comments left by users who edited the post titled 'Why square the difference instead of taking the absolut | |
| 595 | codebase_community | moderate | ✗ | ✓ | 1409 | 6384 | Which user have only one post history per post and having at least 1000 views? | |
| 634 | codebase_community | challenging | ✗ | ✓ | empty_result | 909 | 6305 | Among posts by Harvey Motulsky and Noah Snyder, which one has higher popularity? |
| 669 | codebase_community | simple | ✓ | ✓ | 38 | 5678 | When did 'chl' cast its first vote in a post? | |
| 671 | codebase_community | simple | ✓ | ✓ | 199 | 5691 | What is the display name of the user who acquired the first Autobiographer badge? | |
| 672 | codebase_community | moderate | ✗ | ✓ | 404 | 6291 | Among the users located in United Kingdom, how many users whose post have a total favorite amount of 4 or more? | |
| 694 | codebase_community | moderate | ✗ | ✓ | 680 | 6569 | Provide the text of the latest 10 comments to the post with the title 'Analysing wind data with R' and the display name | |
| 707 | codebase_community | moderate | ✓ | ✓ | 427 | 6462 | Among the posts with views ranging from 100 to 150, what is the comment with the highest score? | |
| 716 | codebase_community | moderate | ✗ | ✓ | 116 | 6420 | Among the comments with scores between 5 to 10, what is the percentage of the users with 0 up votes? | |
| 723 | superhero | moderate | ✓ | ✓ | 22 | 3572 | Among the superheroes with blue eyes, how many of them have the super power of "Agility"? | |
| 730 | superhero | challenging | ✓ | ✓ | 158 | 3599 | List the superheroes from Marvel Comics who have the super power of 'Super Strength'. | |
| 736 | superhero | moderate | ✓ | ✓ | 16 | 3445 | Who is the dumbest superhero? | |
| 737 | superhero | simple | ✓ | ✓ | 12 | 3381 | What is Copycat's race? | |
| 738 | superhero | simple | ✓ | ✓ | 138 | 3545 | Which superheroes have a durability attribute value of less than 50? | |
| 743 | superhero | challenging | ✗ | ✓ | 16 | 3624 | What is the percentage of superheroes who act in their own self-interest or make decisions based on their own moral code | |
| 747 | superhero | simple | ✓ | ✓ | 14 | 3381 | What is the total number of superheroes without full name? | |
| 750 | superhero | simple | ✓ | ✓ | 12 | 3444 | What is the average weight of all female superheroes? | |
| 751 | superhero | moderate | ✓ | ✓ | 14 | 3529 | List down at least five superpowers of male superheroes. | |
| 753 | superhero | moderate | ✓ | ✓ | 15 | 3583 | Among the superheroes with height from 170 to 190, list the names of the superheroes with no eye color. | |
| 765 | superhero | simple | ✓ | ✓ | 15 | 3426 | How many heroes have stealth power? | |
| 773 | superhero | challenging | ✓ | ✓ | 13 | 3524 | Which superhero has the same eyes, hair and skin colour? Indicate the publisher of the superhero. | |
| 775 | superhero | challenging | ✓ | ✓ | 14 | 3629 | What is the percentage of blue female superheroes among all female superheroes? | |
| 781 | superhero | simple | ✓ | ✓ | 15 | 3487 | Provide the heights of the heroes whose eye colours are amber. | |
| 785 | superhero | simple | ✓ | ✓ | 14 | 3452 | Describe the names of neutral alignment superheroes. | |
| 791 | superhero | simple | ✗ | ✓ | 13 | 3402 | Calculate the average height for all superhero. | |
| 794 | superhero | moderate | ✓ | ✓ | 15 | 3453 | Which hero was the fastest? | |
| 798 | superhero | moderate | ✓ | ✓ | 14 | 3503 | What is the publisher for Hawkman, Karate Kid and Speedy? | |
| 800 | superhero | moderate | ✓ | ✓ | 14 | 3548 | Calculate the percentage of superheroes with blue eyes. | |
| 806 | superhero | simple | ✓ | ✓ | 12 | 3379 | Provide the eye colour of the superhero who has Karen Beecher-Duncan as their full name. | |
| 819 | superhero | challenging | ✓ | ✓ | 15 | 3677 | In superheroes with missing weight data, calculate the difference between the number of superheroes with blue eyes and n | |
| 825 | superhero | moderate | ✓ | ✓ | 17 | 3498 | Identify the gender of the superhero who has the ability of Phoenix Force. | |
| 847 | formula_1 | simple | ✗ | ✓ | 26 | 6661 | What is the surname of the driver with the best lap time in race number 19 in the second qualifying period? | |
| 859 | formula_1 | simple | ✓ | ✓ | 15 | 6659 | What's Bruno Senna's Q1 result in the qualifying race No. 354? | |
| 861 | formula_1 | simple | ✗ | ✓ | empty_result | 15 | 6661 | What is his number of the driver who finished 0:01:54 in the Q3 of qualifying race No.903? |
| 862 | formula_1 | simple | ✓ | ✓ | 45 | 6650 | For the Bahrain Grand Prix in 2007, how many drivers not finished the game? | |
| 865 | formula_1 | moderate | ✗ | ✓ | 23 | 6708 | For all the drivers who finished the game in race No. 592, who is the oldest? | |
| 866 | formula_1 | moderate | ✗ | ✓ | 16 | 6757 | Who was the player that got the lap time of 0:01:27 in the race No. 161? Show his introduction website. | |
| 875 | formula_1 | simple | ✓ | ✓ | 14 | 6603 | Show me the season page of year when the race No. 901 took place. | |
| 877 | formula_1 | moderate | ✓ | ✓ | 17 | 6656 | For all the drivers who finished the game in race No. 872, who is the youngest? | |
| 879 | formula_1 | moderate | ✓ | ✓ | 28 | 6602 | For the driver who set the fastest lap speed, what is his nationality? | |
| 881 | formula_1 | moderate | ✓ | ✓ | 23 | 6770 | For the drivers who took part in the race in 1983/7/16, what's their race completion rate? | |
| 894 | formula_1 | moderate | ✗ | ✓ | 483 | 6717 | What is the best lap time recorded? List the driver and race with such recorded lap time. | |
| 896 | formula_1 | challenging | ✗ | ✓ | 43 | 6780 | Calculate the percentage whereby Hamilton was not at the 1st track of the the f1 circuit since 2010. | |
| 897 | formula_1 | moderate | ✓ | ✓ | 26 | 6707 | Name the driver with the most winning. Mention his nationality and what is his maximum point scores. | |
| 898 | formula_1 | simple | ✗ | ✓ | execution_failed | 13 | 6386 | How old is the youngest Japanese driver? What is his name? |
| 902 | formula_1 | simple | ✗ | ✓ | 26 | 6717 | Which race was Alex Yoong in when he was in track number less than 20? | |
| 904 | formula_1 | moderate | ✗ | ✓ | 189 | 6487 | State the race and year of race in which Michael Schumacher had his fastest lap. | |
| 909 | formula_1 | moderate | ✓ | ✓ | 18 | 6750 | Among all European Grand Prix races, what is the percentage of the races were hosted in Germany? | |
| 912 | formula_1 | simple | ✓ | ✓ | 12 | 6306 | What's the reference name of Marina Bay Street Circuit? | |
| 915 | formula_1 | simple | ✗ | ✓ | 23 | 6614 | Which country is the oldest driver from? | |
| 930 | formula_1 | simple | ✗ | ✓ | 24 | 6652 | In which Formula_1 race did Lewis Hamilton rank the highest? | |
| 945 | formula_1 | simple | ✓ | ✓ | 14 | 6277 | How many circuits are there in Adelaide, Australia? | |
| 950 | formula_1 | simple | ✗ | ✓ | 24 | 6632 | Please list the constructor names with 0 points at race 291. | |
| 959 | formula_1 | simple | ✗ | ✓ | 20 | 6710 | What is the fastest lap number of the champion in 2009? | |
| 971 | formula_1 | simple | ✓ | ✓ | 14 | 6527 | Please state the reference name of the oldest German driver. | |
| 981 | formula_1 | moderate | ✓ | ✓ | 19 | 6748 | On what year did the youngest driver had his first qualifying race? Also state the name, date and time of the race. | |
| 988 | formula_1 | challenging | ✗ | ✓ | 29 | 6641 | List down top 3 German drivers who has the shortest average pit stop duration and were born between 1980-1985. | |
| 989 | formula_1 | moderate | ✗ | ✓ | 19 | 6699 | Who is the champion of the Canadian Grand Prix in 2008? Indicate his finish time. | |
| 990 | formula_1 | challenging | ✓ | ✓ | 22 | 6733 | What is the constructor reference name of the champion in the 2009 Singapore Grand Prix? Please give its website. | |
| 1028 | european_football_2 | challenging | ✗ | ✓ | 996 | 12183 | In Scotland Premier League, which away team won the most during the 2010 season? | |
| 1029 | european_football_2 | moderate | ✗ | ✓ | 35 | 12055 | What are the speed in which attacks are put together of the top 4 teams with the highest build Up Play Speed? | |
| 1030 | european_football_2 | moderate | ✓ | ✓ | 119 | 12015 | Give the name of the league had the most matches end as draw in the 2016 season? | |
| 1035 | european_football_2 | simple | ✓ | ✓ | 17 | 11969 | Give the team_fifa_api_id of teams with more than 50 but less than 60 build-up play speed. | |
| 1036 | european_football_2 | challenging | ✗ | ✓ | execution_failed | 16 | 11699 | List the long name of teams with above-average build-up play passing in 2012. |
| 1037 | european_football_2 | challenging | ✗ | ✓ | 295 | 12142 | Calculate the percentage of players who prefer left foot, who were born between 1987 and 1992. | |
| 1039 | european_football_2 | simple | ✓ | ✓ | 179 | 11940 | Find the average number of long-shot done by Ahmed Samir Farag. | |
| 1042 | european_football_2 | challenging | ✓ | ✓ | 120 | 12195 | List the name of leagues in which the average goals by the home team is higher than the away team in the 2009/2010 seaso | |
| 1057 | european_football_2 | moderate | ✓ | ✓ | 111 | 12032 | Calculate the average home team goal in the 2010/2011 season in the country of Poland. | |
| 1078 | european_football_2 | simple | ✓ | ✓ | 21 | 11796 | Which player is older, Aaron Lennon or Abdelaziz Barrada? | |
| 1088 | european_football_2 | moderate | ✗ | ✓ | 150 | 12029 | Please list the names of the players whose volley score and dribbling score are over 70. | |
| 1094 | european_football_2 | challenging | ✗ | ✓ | execution_failed | 137 | 11795 | How much higher in percentage is Ariel Borysiuk's overall rating than that of Paulin Puel? |
| 1103 | european_football_2 | moderate | ✓ | ✓ | 149 | 11996 | What was the overall rating for Aaron Mooy on 2016/2/4? | |
| 1110 | european_football_2 | moderate | ✓ | ✓ | 16 | 11932 | Tell the build Up play passing class for "FC Lorient" on 2010/2/22. | |
| 1116 | european_football_2 | simple | ✓ | ✓ | 27 | 11857 | List down most tallest players' name. | |
| 1122 | european_football_2 | simple | ✓ | ✓ | 213 | 11792 | State the name of the most strongest player. | |
| 1130 | european_football_2 | moderate | ✓ | ✓ | 15 | 11959 | What are the short name of team who played safe while creating chance of passing? | |
| 1133 | european_football_2 | simple | ✓ | ✓ | 19 | 11827 | How many football players born after the 1990s have the first name "Aaron"? | |
| 1141 | european_football_2 | moderate | ✓ | ✓ | 13 | 11878 | Does the KSV Cercle Brugge team have a slow, balanced or fast speed class? | |
| 1144 | european_football_2 | simple | ✗ | ✓ | 154 | 11970 | Please state the finishing rate and curve score of the player who has the heaviest weight. | |
| 1146 | european_football_2 | moderate | ✓ | ✓ | 148 | 11938 | Please provide the full name of the away team that scored the most goals. | |
| 1147 | european_football_2 | simple | ✓ | ✓ | 223 | 11791 | Please name one player whose overall strength is the greatest. | |
| 1152 | thrombosis_prediction | moderate | ✓ | ✓ | 18 | 4895 | What is the ratio of outpatient to inpatient followed up treatment among all the 'SLE' diagnosed patient? | |
| 1156 | thrombosis_prediction | moderate | ✗ | ✓ | execution_failed | 10 | 4489 | State the ID and age of patient with positive degree of coagulation. |
| 1157 | thrombosis_prediction | simple | ✗ | ✓ | 11 | 4787 | For patients with severe degree of thrombosis, list their ID, sex and disease the patient is diagnosed with. | |
| 1168 | thrombosis_prediction | challenging | ✗ | ✓ | execution_failed | 13 | 4548 | The oldest SJS patient's medical laboratory work was completed on what date, and what age was the patient when they init |
| 1185 | thrombosis_prediction | challenging | ✗ | ✓ | 13 | 5245 | For the patient who was born on 1959/2/18, what is the decrease rate for his/her total cholesterol from November to Dece | |
| 1198 | thrombosis_prediction | simple | ✓ | ✓ | 11 | 4666 | How many female patients were given an APS diagnosis? | |
| 1205 | thrombosis_prediction | moderate | ✗ | ✓ | 14 | 4854 | Was the patient with the number 57266's uric acid within a normal range? | |
| 1208 | thrombosis_prediction | moderate | ✓ | ✓ | 29 | 4863 | Provide IDs for male patients with ALT glutamic pylvic transaminase (GPT) that have history of ALT glutamic pylvic trans | |
| 1220 | thrombosis_prediction | simple | ✓ | ✓ | 31 | 4892 | Provide all ID, sex and birthday of patients whose urea nitrogen (UN) just within the borderline of passing? | |
| 1227 | thrombosis_prediction | moderate | ✗ | ✓ | execution_failed | 12 | 4523 | What is the average age of the male patient with high cholesterol? |
| 1232 | thrombosis_prediction | challenging | ✓ | ✓ | 19 | 5013 | Provide ID, sex and age of patient who has blood glucose (GLU) not within normal range but with total cholesterol(T-CHO) | |
| 1235 | thrombosis_prediction | moderate | ✗ | ✓ | execution_failed | 15 | 4521 | What are the patient's diagnosis for those who has lower red blood blood cell? State their ID and age. |
| 1247 | thrombosis_prediction | challenging | ✗ | ✓ | 19 | 4879 | Among the male patients who have a normal level of white blood cells, how many of them have an abnormal fibrinogen level | |
| 1251 | thrombosis_prediction | simple | ✗ | ✓ | 20 | 4702 | How many patients with an Ig G higher than normal? | |
| 1252 | thrombosis_prediction | moderate | ✗ | ✓ | 12 | 4791 | Among the patients with a normal Ig G level, how many of them have symptoms? | |
| 1254 | thrombosis_prediction | moderate | ✗ | ✓ | execution_failed | 17 | 4547 | How many patients with a normal Ig A level came to the hospital after 1990/1/1? |
| 1255 | thrombosis_prediction | moderate | ✗ | ✓ | 16 | 4806 | For the patients with an abnormal Ig M level, what is the most common disease they are diagnosed with? | |
| 1257 | thrombosis_prediction | challenging | ✗ | ✓ | 19 | 4815 | Among the patients whose creatinine level is abnormal, how many of them aren't 70 yet? | |
| 1275 | thrombosis_prediction | moderate | ✗ | ✓ | execution_failed | 12 | 4554 | Among the patients who has a normal level of anti-centromere and a normal level of anti-SSB, how many of them are male? |
| 1281 | thrombosis_prediction | moderate | ✓ | ✓ | 16 | 4783 | Among the patients who have an abnormal level of glutamic oxaloacetic transaminase, when was the youngest of them born? | |
| 1302 | thrombosis_prediction | challenging | ✗ | ✓ | execution_failed | 11 | 4517 | For the patients with a normal range of creatinine phosphokinase, how many of them have a positive measure of degree of |
| 1312 | student_club | simple | ✓ | ✓ | 13 | 4710 | What's Angela Sanders's major? | |
| 1340 | student_club | moderate | ✓ | ✓ | 16 | 4946 | Calculate the difference of the total amount spent in all events by the Student_Club in year 2019 and 2020. | |
| 1344 | student_club | simple | ✓ | ✓ | 11 | 4770 | What was the notes of the fundraising on 2019/9/14? | |
| 1352 | student_club | moderate | ✓ | ✓ | 13 | 4753 | For all the club members from "Business" major, how many of them wear medium size t-shirt? | |
| 1356 | student_club | simple | ✓ | ✓ | 16 | 4687 | Which department was the President of the club in? | |
| 1376 | student_club | moderate | ✓ | ✓ | 12 | 4768 | Among all the closed events, which event has the highest spend-to-budget ratio? | |
| 1378 | student_club | simple | ✓ | ✓ | 11 | 4186 | What is the highest amount of budget spend for an event? | |
| 1380 | student_club | simple | ✓ | ✓ | 12 | 4455 | What is the total amount of money spent for food? | |
| 1387 | student_club | moderate | ✓ | ✓ | 14 | 4840 | Which student has been entrusted to manage the budget for the Yearly Kickoff? | |
| 1390 | student_club | moderate | ✓ | ✓ | 14 | 4376 | Based on the total cost for all event, what is the percentage of cost for Yearly Kickoff event? | |
| 1399 | student_club | moderate | ✗ | ✓ | 16 | 4791 | Did Maya Mclean attend the 'Women's Soccer' event? | |
| 1403 | student_club | moderate | ✓ | ✓ | 12 | 4780 | Indicate the name of the closed event whose cost has exceeded the budget the most. | |
| 1404 | student_club | moderate | ✓ | ✓ | 13 | 4859 | Identify the type of expenses and their total value approved for 'October Meeting' event. | |
| 1409 | student_club | simple | ✓ | ✓ | 11 | 4744 | Mention the total expense used on 8/20/2019. | |
| 1410 | student_club | simple | ✓ | ✓ | 14 | 4792 | List out the full name and total cost that member id "rec4BLdZHS2Blfp4v" incurred? | |
| 1411 | student_club | simple | ✓ | ✓ | 13 | 4775 | State what kind of expenses that Sacha Harrison incurred? | |
| 1422 | student_club | simple | ✗ | ✓ | 11 | 4700 | State the category of events were held at MU 215. | |
| 1464 | student_club | challenging | ✓ | ✓ | 11 | 4836 | Write the full names of students who received funds on the date of 9/9/2019 and include the amount received. | |
| 1472 | debit_card_specializing | moderate | ✗ | ✓ | 610 | 3075 | In 2012, who had the least consumption in LAM? | |
| 1473 | debit_card_specializing | moderate | ✓ | ✓ | 459 | 3137 | What was the average monthly consumption of customers in SME for the year 2013? | |
| 1476 | debit_card_specializing | challenging | ✓ | ✓ | 372 | 3286 | What was the difference in gas consumption between CZK-paying customers and EUR-paying customers in 2012? | |
| 1479 | debit_card_specializing | moderate | ✓ | ✓ | 554 | 3010 | Which year recorded the most consumption of gas paid in CZK? | |
| 1480 | debit_card_specializing | moderate | ✗ | ✓ | 514 | 3103 | What was the gas consumption peak month for SME customers in 2013? | |
| 1484 | debit_card_specializing | simple | ✓ | ✓ | 23 | 3085 | How many more "discount" gas stations does the Czech Republic have compared to Slovakia? | |
| 1486 | debit_card_specializing | simple | ✗ | ✓ | 28 | 3074 | Is it true that more SMEs pay in Czech koruna than in euros? If so, how many more? | |
| 1493 | debit_card_specializing | simple | ✓ | ✓ | 147 | 3144 | In February 2012, what percentage of customers consumed more than 528.3? | |
| 1500 | debit_card_specializing | simple | ✓ | ✓ | 25 | 3118 | Please list the product description of the products consumed in September, 2013. | |
| 1501 | debit_card_specializing | moderate | ✓ | ✓ | 25 | 3102 | Please list the countries of the gas stations with transactions taken place in June, 2013. | |
| 1506 | debit_card_specializing | moderate | ✓ | ✓ | 20 | 3057 | Please list the product descriptions of the transactions taken place in the gas stations in the Czech Republic. | |
| 1515 | debit_card_specializing | simple | ✓ | ✓ | 18 | 3011 | What segment did the customer have at 2012/8/23 21:20:00? | |
| 1521 | debit_card_specializing | moderate | ✓ | ✓ | 17 | 3254 | For all the transactions happened during 8:00-9:00 in 2012/8/26, how many happened in CZE? | |
| 1525 | debit_card_specializing | simple | ✗ | ✓ | 24 | 3102 | What is the percentage of the customers who used EUR in 2012/8/25? | |
| 1526 | debit_card_specializing | challenging | ✗ | ✓ | empty_result | 53 | 3267 | For the customer who paid 634.8 in 2012/8/25, what was the consumption decrease rate from Year 2012 to 2013? |
| 1528 | debit_card_specializing | simple | ✓ | ✓ | 17 | 2969 | What is the percentage of "premium" against the overall segment in Country = "SVK"? | |
| 1529 | debit_card_specializing | moderate | ✗ | ✓ | 61 | 3092 | What is the amount spent by customer "38508" at the gas stations? How much had the customer spent in January 2012? | |
| 1531 | debit_card_specializing | moderate | ✗ | ✓ | 68 | 3156 | Who is the top spending customer and how much is the average price per single item purchased by this customer? What curr |
Model: codestral-latest · n=2 · EA=100.0% · Validity=100.0% · Recall@k=100.0%
| qid | db | diff | match | recall | err | lat ms | tokens | question |
|---|---|---|---|---|---|---|---|---|
| 1404 | student_club | moderate | ✓ | ✓ | 58 | 4859 | Identify the type of expenses and their total value approved for 'October Meeting' event. | |
| 207 | toxicology | challenging | ✓ | ✓ | 2787 | 2695 | What elements are in a double type bond? |
Model: codestral-latest · n=2 · EA=50.0% · Validity=100.0% · Recall@k=100.0%
| qid | db | diff | match | recall | err | lat ms | tokens | question |
|---|---|---|---|---|---|---|---|---|
| 1404 | student_club | moderate | ✓ | ✓ | 191 | 4859 | Identify the type of expenses and their total value approved for 'October Meeting' event. | |
| 207 | toxicology | challenging | ✗ | ✓ | 180 | 2669 | What elements are in a double type bond? |
Model: codestral-latest · n=1 · EA=0.0% · Validity=100.0% · Recall@k=100.0%
| qid | db | diff | match | recall | err | lat ms | tokens | question |
|---|---|---|---|---|---|---|---|---|
| 1399 | student_club | moderate | ✗ | ✓ | 11681 | 4895 | Did Maya Mclean attend the 'Women's Soccer' event? |
Source: BIRD Mini-Dev (SQLite). Methodology: docs/03_eval_methodology.md.
| Configuration | Model | n | EA | Simple | Moderate | Challenging | Validity | Recall@k | Empty % | P50 latency | P95 latency |
|---|---|---|---|---|---|---|---|---|---|---|---|
| C_dense_cards | codestral-latest | 4 | 50.0% | 100.0% | 0.0% | 50.0% | 100.0% | 100.0% | 0.0% | 306 ms | 7100 ms |
| C_dense_cards | codestral-latest | 6 | 50.0% | 100.0% | 40.0% | 0.0% | 100.0% | 100.0% | 0.0% | 2768 ms | 4529 ms |
| C_dense_cards | codestral-latest | 4 | 50.0% | 100.0% | 0.0% | 100.0% | 100.0% | 100.0% | 0.0% | 44 ms | 8632 ms |
| C_dense_cards | codestral-latest | 4 | 75.0% | 100.0% | 50.0% | 100.0% | 100.0% | 100.0% | 0.0% | 106 ms | 3850 ms |
| C_dense_cards | codestral-latest | 4 | 75.0% | 100.0% | 50.0% | 100.0% | 100.0% | 100.0% | 0.0% | 174 ms | 5770 ms |
| C_dense_cards | codestral-latest | 7 | 71.4% | 100.0% | 50.0% | 100.0% | 100.0% | 100.0% | 0.0% | 141 ms | 4704 ms |
| C_dense_cards | codestral-latest | 7 | 71.4% | 50.0% | 75.0% | 100.0% | 100.0% | 100.0% | 0.0% | 3000 ms | 11341 ms |
| C_dense_cards | codestral-latest | 2 | 50.0% | 100.0% | 0.0% | 0.0% | 100.0% | 100.0% | 0.0% | 5358 ms | 9973 ms |
| C_dense_cards | codestral-latest | 2 | 50.0% | 100.0% | 0.0% | 0.0% | 100.0% | 100.0% | 0.0% | 11475 ms | 13877 ms |
| C_dense_cards | codestral-latest | 2 | 50.0% | 100.0% | 0.0% | 0.0% | 100.0% | 100.0% | 0.0% | 11404 ms | 18884 ms |
| C_dense_cards | codestral-latest | 200 | 56.0% | 71.6% | 49.5% | 44.1% | 100.0% | 97.0% | 1.5% | 4778 ms | 21193 ms |
| C_dense_cards | codestral-latest | 8 | 75.0% | 100.0% | 60.0% | 100.0% | 100.0% | 100.0% | 0.0% | 131 ms | 8226 ms |
Model: codestral-latest · n=4 · EA=50.0% · Validity=100.0% · Recall@k=100.0%
| qid | db | diff | match | recall | err | lat ms | tokens | question |
|---|---|---|---|---|---|---|---|---|
| 125 | financial | challenging | ✗ | ✓ | 8270 | 4984 | For loans contracts which are still running where client are in debt, list the district of the and the state the percent | |
| 207 | toxicology | challenging | ✓ | ✓ | 467 | 2697 | What elements are in a double type bond? | |
| 902 | formula_1 | simple | ✓ | ✓ | 146 | 6805 | Which race was Alex Yoong in when he was in track number less than 20? | |
| 1404 | student_club | moderate | ✗ | ✓ | 62 | 4900 | Identify the type of expenses and their total value approved for 'October Meeting' event. |
Model: codestral-latest · n=6 · EA=50.0% · Validity=100.0% · Recall@k=100.0%
| qid | db | diff | match | recall | err | lat ms | tokens | question |
|---|---|---|---|---|---|---|---|---|
| 1251 | thrombosis_prediction | simple | ✓ | ✓ | 4886 | 4921 | How many patients with an Ig G higher than normal? | |
| 1252 | thrombosis_prediction | moderate | ✗ | ✓ | 2591 | 4797 | Among the patients with a normal Ig G level, how many of them have symptoms? | |
| 1254 | thrombosis_prediction | moderate | ✗ | ✓ | execution_failed | 2005 | 4540 | How many patients with a normal Ig A level came to the hospital after 1990/1/1? |
| 1275 | thrombosis_prediction | moderate | ✗ | ✓ | execution_failed | 1922 | 4556 | Among the patients who has a normal level of anti-centromere and a normal level of anti-SSB, how many of them are male? |
| 894 | formula_1 | moderate | ✓ | ✓ | 2946 | 6822 | What is the best lap time recorded? List the driver and race with such recorded lap time. | |
| 1531 | debit_card_specializing | moderate | ✓ | ✓ | 3459 | 3303 | Who is the top spending customer and how much is the average price per single item purchased by this customer? What curr |
Model: codestral-latest · n=4 · EA=50.0% · Validity=100.0% · Recall@k=100.0%
| qid | db | diff | match | recall | err | lat ms | tokens | question |
|---|---|---|---|---|---|---|---|---|
| 1531 | debit_card_specializing | moderate | ✗ | ✓ | 10144 | 3248 | Who is the top spending customer and how much is the average price per single item purchased by this customer? What curr | |
| 207 | toxicology | challenging | ✓ | ✓ | 64 | 2697 | What elements are in a double type bond? | |
| 902 | formula_1 | simple | ✓ | ✓ | 24 | 6805 | Which race was Alex Yoong in when he was in track number less than 20? | |
| 1404 | student_club | moderate | ✗ | ✓ | 13 | 4900 | Identify the type of expenses and their total value approved for 'October Meeting' event. |
Model: codestral-latest · n=4 · EA=75.0% · Validity=100.0% · Recall@k=100.0%
| qid | db | diff | match | recall | err | lat ms | tokens | question |
|---|---|---|---|---|---|---|---|---|
| 1531 | debit_card_specializing | moderate | ✓ | ✓ | 4503 | 3292 | Who is the top spending customer and how much is the average price per single item purchased by this customer? What curr | |
| 207 | toxicology | challenging | ✓ | ✓ | 153 | 2697 | What elements are in a double type bond? | |
| 902 | formula_1 | simple | ✓ | ✓ | 60 | 6805 | Which race was Alex Yoong in when he was in track number less than 20? | |
| 1404 | student_club | moderate | ✗ | ✓ | 37 | 4900 | Identify the type of expenses and their total value approved for 'October Meeting' event. |
Model: codestral-latest · n=4 · EA=75.0% · Validity=100.0% · Recall@k=100.0%
| qid | db | diff | match | recall | err | lat ms | tokens | question |
|---|---|---|---|---|---|---|---|---|
| 1531 | debit_card_specializing | moderate | ✓ | ✓ | 6743 | 3303 | Who is the top spending customer and how much is the average price per single item purchased by this customer? What curr | |
| 207 | toxicology | challenging | ✓ | ✓ | 258 | 2697 | What elements are in a double type bond? | |
| 902 | formula_1 | simple | ✓ | ✓ | 89 | 6805 | Which race was Alex Yoong in when he was in track number less than 20? | |
| 1404 | student_club | moderate | ✗ | ✓ | 44 | 4900 | Identify the type of expenses and their total value approved for 'October Meeting' event. |
Model: codestral-latest · n=7 · EA=71.4% · Validity=100.0% · Recall@k=100.0%
| qid | db | diff | match | recall | err | lat ms | tokens | question |
|---|---|---|---|---|---|---|---|---|
| 408 | card_games | moderate | ✓ | ✓ | 5106 | 8684 | How many unknown power cards contain info about the triggered ability | |
| 1404 | student_club | moderate | ✗ | ✓ | 22 | 4900 | Identify the type of expenses and their total value approved for 'October Meeting' event. | |
| 207 | toxicology | challenging | ✓ | ✓ | 83 | 2697 | What elements are in a double type bond? | |
| 902 | formula_1 | simple | ✓ | ✓ | 36 | 6805 | Which race was Alex Yoong in when he was in track number less than 20? | |
| 1531 | debit_card_specializing | moderate | ✓ | ✓ | 141 | 3303 | Who is the top spending customer and how much is the average price per single item purchased by this customer? What curr | |
| 894 | formula_1 | moderate | ✗ | ✓ | 3764 | 6789 | What is the best lap time recorded? List the driver and race with such recorded lap time. | |
| 1251 | thrombosis_prediction | simple | ✓ | ✓ | 2915 | 4917 | How many patients with an Ig G higher than normal? |
Model: codestral-latest · n=7 · EA=71.4% · Validity=100.0% · Recall@k=100.0%
| qid | db | diff | match | recall | err | lat ms | tokens | question |
|---|---|---|---|---|---|---|---|---|
| 894 | formula_1 | moderate | ✓ | ✓ | 14671 | 6826 | What is the best lap time recorded? List the driver and race with such recorded lap time. | |
| 847 | formula_1 | simple | ✗ | ✓ | 3572 | 6665 | What is the surname of the driver with the best lap time in race number 19 in the second qualifying period? | |
| 866 | formula_1 | moderate | ✗ | ✓ | 2765 | 6795 | Who was the player that got the lap time of 0:01:27 in the race No. 161? Show his introduction website. | |
| 207 | toxicology | challenging | ✓ | ✓ | 2435 | 2707 | What elements are in a double type bond? | |
| 902 | formula_1 | simple | ✓ | ✓ | 2747 | 6806 | Which race was Alex Yoong in when he was in track number less than 20? | |
| 1404 | student_club | moderate | ✓ | ✓ | 3000 | 4866 | Identify the type of expenses and their total value approved for 'October Meeting' event. | |
| 1531 | debit_card_specializing | moderate | ✓ | ✓ | 3153 | 3301 | Who is the top spending customer and how much is the average price per single item purchased by this customer? What curr |
Model: codestral-latest · n=2 · EA=50.0% · Validity=100.0% · Recall@k=100.0%
| qid | db | diff | match | recall | err | lat ms | tokens | question |
|---|---|---|---|---|---|---|---|---|
| 902 | formula_1 | simple | ✓ | ✓ | 229 | 6807 | Which race was Alex Yoong in when he was in track number less than 20? | |
| 1275 | thrombosis_prediction | moderate | ✗ | ✓ | execution_failed | 10486 | 4664 | Among the patients who has a normal level of anti-centromere and a normal level of anti-SSB, how many of them are male? |
Model: codestral-latest · n=2 · EA=50.0% · Validity=100.0% · Recall@k=100.0%
| qid | db | diff | match | recall | err | lat ms | tokens | question |
|---|---|---|---|---|---|---|---|---|
| 902 | formula_1 | simple | ✓ | ✓ | 8806 | 6805 | Which race was Alex Yoong in when he was in track number less than 20? | |
| 1275 | thrombosis_prediction | moderate | ✗ | ✓ | 14144 | 4923 | Among the patients who has a normal level of anti-centromere and a normal level of anti-SSB, how many of them are male? |
Model: codestral-latest · n=2 · EA=50.0% · Validity=100.0% · Recall@k=100.0%
| qid | db | diff | match | recall | err | lat ms | tokens | question |
|---|---|---|---|---|---|---|---|---|
| 902 | formula_1 | simple | ✓ | ✓ | 19715 | 6807 | Which race was Alex Yoong in when he was in track number less than 20? | |
| 1275 | thrombosis_prediction | moderate | ✗ | ✓ | 3093 | 4954 | Among the patients who has a normal level of anti-centromere and a normal level of anti-SSB, how many of them are male? |
Model: codestral-latest · n=200 · EA=56.0% · Validity=100.0% · Recall@k=97.0%
| qid | db | diff | match | recall | err | lat ms | tokens | question |
|---|---|---|---|---|---|---|---|---|
| 5 | california_schools | simple | ✓ | ✓ | 6170 | 6356 | How many schools with an average score in Math greater than 400 in the SAT test are exclusively virtual? | |
| 25 | california_schools | moderate | ✗ | ✓ | 3752 | 6438 | Name schools in Riverside which the average of average math score for SAT is grater than 400, what is the funding type o | |
| 32 | california_schools | moderate | ✗ | ✓ | 6842 | 6656 | What is the eligible free or reduced price meal rate for the top 5 schools in grades 1-12 with the highest free or reduc | |
| 36 | california_schools | challenging | ✓ | ✓ | 5646 | 6596 | Under whose administration is the school with the highest number of students scoring 1500 or more on the SAT? Indicate t | |
| 37 | california_schools | moderate | ✗ | ✓ | 5836 | 6495 | What is the complete address of the school with the lowest excellence rate? Indicate the Street, City, Zip and State. | |
| 39 | california_schools | simple | ✓ | ✓ | 5575 | 6530 | What is the average number of test takers from Fresno schools that opened between 1/1/1980 and 12/31/1980? | |
| 48 | california_schools | moderate | ✓ | ✓ | 3926 | 6469 | What is the ratio of merged Unified School District schools in Orange County to merged Elementary School District school | |
| 50 | california_schools | simple | ✗ | ✓ | 3816 | 6397 | What is the postal street address for the school with the 7th highest Math average? Indicate the school's name. | |
| 77 | california_schools | moderate | ✗ | ✓ | empty_result | 3934 | 6534 | Which schools served a grade span of Kindergarten to 9th grade in the county of Los Angeles and what is its Percent (%) |
| 92 | financial | simple | ✓ | ✓ | 14716 | 4537 | List out the no. of districts that have female average salary is more than 6000 but less than 10000? | |
| 98 | financial | moderate | ✓ | ✓ | 4328 | 4556 | Among the accounts who have approved loan date in 1997, list out the accounts that have the lowest approved amount and c | |
| 99 | financial | moderate | ✓ | ✓ | 4801 | 4551 | Among the accounts who have loan validity more than 12 months, list out the accounts that have the highest approved amou | |
| 112 | financial | simple | ✓ | ✓ | 6463 | 4549 | For the female client who was born in 1976/1/29, which district did she opened her account? | |
| 115 | financial | challenging | ✗ | ✓ | 4276 | 4602 | For the branch which located in the south Bohemia with biggest number of inhabitants, what is the percentage of the male | |
| 118 | financial | moderate | ✓ | ✓ | 6029 | 4586 | For loan amount less than USD100,000, what is the percentage of accounts that is still running with no issue. | |
| 120 | financial | moderate | ✗ | ✓ | 5346 | 4873 | From Year 1995 to 2000, who are the accounts holders from 'east Bohemia'. State the account ID the frequency of statemen | |
| 125 | financial | challenging | ✗ | ✓ | 8184 | 4940 | For loans contracts which are still running where client are in debt, list the district of the and the state the percent | |
| 138 | financial | moderate | ✓ | ✓ | 5699 | 4543 | In the branch where the second-highest number of crimes were committed in 1995 occurred, how many male clients are there | |
| 159 | financial | simple | ✗ | ✓ | 9950 | 4685 | List all the withdrawals in cash transactions that the client with the id 3356 makes. | |
| 168 | financial | moderate | ✗ | ✓ | 4605 | 4621 | What percentage of clients who opened their accounts in the district with an average salary of over 10000 are women? | |
| 169 | financial | challenging | ✗ | ✓ | 6532 | 4823 | What was the growth rate of the total amount of loans across all accounts for a male client between 1996 and 1997? | |
| 173 | financial | challenging | ✗ | ✓ | 4338 | 4653 | How often does account number 3 request an account statement to be released? What was the aim of debiting 3539 in total? | |
| 189 | financial | moderate | ✗ | ✓ | execution_failed | 13915 | 4235 | Name the account numbers of female clients who are oldest and have lowest average salary? |
| 192 | financial | moderate | ✓ | ✓ | 3355 | 4580 | What is the average amount of loan which are still on running contract with statement issuance after each transaction? | |
| 194 | financial | moderate | ✓ | ✓ | 4452 | 4506 | Provide the IDs and age of the client with high level credit card, which is eligible for loans. | |
| 207 | toxicology | challenging | ✓ | ✓ | 4272 | 2697 | What elements are in a double type bond? | |
| 208 | toxicology | moderate | ✓ | ✓ | 2728 | 2628 | Which type of label is the most numerous in atoms with hydrogen? | |
| 219 | toxicology | challenging | ✗ | ✓ | execution_failed | 2353 | 2437 | What is the percentage of carcinogenic molecules in triple type bonds? |
| 227 | toxicology | simple | ✓ | ✓ | 3022 | 2682 | What is the percentage of molecules that are carcinogenic? Please provide your answer as a percentage with three decimal | |
| 230 | toxicology | challenging | ✓ | ✓ | 3405 | 2645 | What are the elements of the toxicology and label of molecule TR060? | |
| 232 | toxicology | moderate | ✗ | ✓ | execution_failed | 2841 | 2420 | Please list top three molecules that have single bonds between two atoms and are not carcinogenic in alphabetical order. |
| 236 | toxicology | moderate | ✓ | ✓ | 6123 | 2704 | What are the bond type and the atoms of the bond ID of TR001_6_9? | |
| 239 | toxicology | simple | ✓ | ✓ | 3692 | 2624 | How many connections does the atom 19 have? | |
| 253 | toxicology | challenging | ✓ | ✓ | 4181 | 2635 | List the elements of all the triple bonds. | |
| 260 | toxicology | moderate | ✓ | ✓ | 3162 | 2719 | Calculate the total atoms with triple-bond molecules containing the element phosphorus or bromine. | |
| 268 | toxicology | challenging | ✓ | ✓ | 3952 | 2705 | What are the elements for bond id TR001_10_11? | |
| 273 | toxicology | moderate | ✗ | ✓ | 3979 | 2733 | What is the percentage of element chlorine in carcinogenic molecules? | |
| 282 | toxicology | challenging | ✓ | ✓ | 6591 | 2784 | What is the ratio of Hydrogen elements in molecule ID TR006? List the ratio with its label. | |
| 327 | toxicology | moderate | ✓ | ✓ | 4984 | 2728 | Which non-carcinogenic molecules consisted more than 5 atoms? | |
| 347 | card_games | moderate | ✓ | ✓ | 17289 | 8855 | Find all cards illustrated by Stephen Daniel and describe the text of the ruling of these cards. State if these cards ha | |
| 349 | card_games | moderate | ✗ | ✓ | 18627 | 8560 | Name the card and artist with the most ruling information. Also state if the card is a promotional printing. | |
| 352 | card_games | moderate | ✗ | ✓ | 18240 | 8586 | Calculate the percentage of the cards availabe in Chinese Simplified. | |
| 356 | card_games | simple | ✓ | ✓ | 5430 | 8372 | How many cards have infinite power? | |
| 358 | card_games | simple | ✓ | ✓ | 5259 | 8437 | What is the border color of card "Ancestor's Chosen"? | |
| 366 | card_games | simple | ✗ | ✓ | 13254 | 8497 | What is the rule of playing card "Benalish Knight"? | |
| 377 | card_games | simple | ✗ | ✓ | 7123 | 8451 | How many cards with original type of "Summon - Angel" have subtype other than "Angel"? | |
| 391 | card_games | moderate | ✓ | ✓ | 9282 | 8571 | Among the Artifact cards, which are black color and comes with foreign languague translation? | |
| 407 | card_games | moderate | ✗ | ✓ | 8768 | 8571 | Lists all types of cards in German. | |
| 408 | card_games | moderate | ✗ | ✓ | execution_timeout | 110213 | 8229 | How many unknown power cards contain info about the triggered ability |
| 412 | card_games | moderate | ✗ | ✗ | pipeline_exception | 21130 | 0 | What is the foreign name of the card in French of type Creature, normal layout and black border color, by artist Matthew |
| 414 | card_games | simple | ✗ | ✗ | pipeline_exception | 25888 | 0 | What language is the set of 180 cards that belongs to the Ravnica block translated into? |
| 427 | card_games | moderate | ✗ | ✗ | pipeline_exception | 17897 | 0 | What languages are available in the set known as Archenemy on the magic card market and having the code ARC? |
| 459 | card_games | moderate | ✗ | ✓ | 30928 | 8551 | Which card costs more converted mana, "Serra Angel" or "Shrine Keeper"? | |
| 466 | card_games | moderate | ✓ | ✓ | 10523 | 8546 | Among the cards in the set "Hauptset Zehnte Edition", how many of them are designed by Adam Rex? | |
| 472 | card_games | moderate | ✓ | ✓ | 5098 | 8533 | Among the sets in the block "Ice Age", how many of them have an Italian translation? | |
| 484 | card_games | moderate | ✗ | ✓ | 10105 | 8575 | Please list the Italian names of the cards in the set Coldsnap with the highest converted mana cost. | |
| 486 | card_games | moderate | ✓ | ✓ | 6992 | 8619 | What is the percentage of the cards with a converted mana cost of 7 in the set Coldsnap? | |
| 518 | card_games | moderate | ✗ | ✓ | 74306 | 8608 | Which of the play format has the highest number of banned status? Indicate the play format and the names of all the card | |
| 531 | codebase_community | simple | ✗ | ✓ | 28907 | 5676 | Which user has a higher reputation, Harlan or Jarrod Dixon? | |
| 557 | codebase_community | moderate | ✗ | ✓ | 33643 | 6428 | Among the posts with a score of over 5, what is the percentage of them being owned by an elder user? | |
| 563 | codebase_community | moderate | ✗ | ✓ | execution_timeout | 97178 | 6120 | User No.3025 gave a comment at 20:29:39 on 2014/4/23 to a post, how many favorite counts did that post get? |
| 571 | codebase_community | moderate | ✗ | ✓ | 12001 | 6339 | For the user No.24, how many times is the number of his/her posts compared to his/her votes? | |
| 584 | codebase_community | moderate | ✗ | ✓ | 49964 | 6483 | Write all the comments left by users who edited the post titled 'Why square the difference instead of taking the absolut | |
| 595 | codebase_community | moderate | ✗ | ✓ | 12218 | 6384 | Which user have only one post history per post and having at least 1000 views? | |
| 634 | codebase_community | challenging | ✗ | ✓ | execution_failed | 8389 | 6048 | Among posts by Harvey Motulsky and Noah Snyder, which one has higher popularity? |
| 669 | codebase_community | simple | ✓ | ✓ | 4878 | 5678 | When did 'chl' cast its first vote in a post? | |
| 671 | codebase_community | simple | ✓ | ✓ | 27440 | 5686 | What is the display name of the user who acquired the first Autobiographer badge? | |
| 672 | codebase_community | moderate | ✗ | ✓ | 9034 | 6287 | Among the users located in United Kingdom, how many users whose post have a total favorite amount of 4 or more? | |
| 694 | codebase_community | moderate | ✗ | ✓ | 7394 | 6572 | Provide the text of the latest 10 comments to the post with the title 'Analysing wind data with R' and the display name | |
| 707 | codebase_community | moderate | ✗ | ✓ | execution_failed | 4710 | 6060 | Among the posts with views ranging from 100 to 150, what is the comment with the highest score? |
| 716 | codebase_community | moderate | ✗ | ✓ | 8250 | 6413 | Among the comments with scores between 5 to 10, what is the percentage of the users with 0 up votes? | |
| 723 | superhero | moderate | ✓ | ✓ | 7982 | 3570 | Among the superheroes with blue eyes, how many of them have the super power of "Agility"? | |
| 730 | superhero | challenging | ✓ | ✓ | 8169 | 3599 | List the superheroes from Marvel Comics who have the super power of 'Super Strength'. | |
| 736 | superhero | moderate | ✓ | ✓ | 6072 | 3457 | Who is the dumbest superhero? | |
| 737 | superhero | simple | ✓ | ✓ | 4772 | 3381 | What is Copycat's race? | |
| 738 | superhero | simple | ✓ | ✓ | 5510 | 3545 | Which superheroes have a durability attribute value of less than 50? | |
| 743 | superhero | challenging | ✗ | ✓ | 13321 | 3624 | What is the percentage of superheroes who act in their own self-interest or make decisions based on their own moral code | |
| 747 | superhero | simple | ✓ | ✓ | 3299 | 3381 | What is the total number of superheroes without full name? | |
| 750 | superhero | simple | ✓ | ✓ | 2649 | 3446 | What is the average weight of all female superheroes? | |
| 751 | superhero | moderate | ✓ | ✓ | 5583 | 3533 | List down at least five superpowers of male superheroes. | |
| 753 | superhero | moderate | ✓ | ✓ | 3789 | 3569 | Among the superheroes with height from 170 to 190, list the names of the superheroes with no eye color. | |
| 765 | superhero | simple | ✓ | ✓ | 3180 | 3425 | How many heroes have stealth power? | |
| 773 | superhero | challenging | ✓ | ✓ | 4002 | 3523 | Which superhero has the same eyes, hair and skin colour? Indicate the publisher of the superhero. | |
| 775 | superhero | challenging | ✓ | ✓ | 5097 | 3628 | What is the percentage of blue female superheroes among all female superheroes? | |
| 781 | superhero | simple | ✓ | ✓ | 3357 | 3483 | Provide the heights of the heroes whose eye colours are amber. | |
| 785 | superhero | simple | ✓ | ✓ | 3375 | 3450 | Describe the names of neutral alignment superheroes. | |
| 791 | superhero | simple | ✗ | ✓ | 3884 | 3402 | Calculate the average height for all superhero. | |
| 794 | superhero | moderate | ✓ | ✓ | 3569 | 3469 | Which hero was the fastest? | |
| 798 | superhero | moderate | ✓ | ✓ | 3771 | 3503 | What is the publisher for Hawkman, Karate Kid and Speedy? | |
| 800 | superhero | moderate | ✓ | ✓ | 3524 | 3548 | Calculate the percentage of superheroes with blue eyes. | |
| 806 | superhero | simple | ✓ | ✓ | 4355 | 3379 | Provide the eye colour of the superhero who has Karen Beecher-Duncan as their full name. | |
| 819 | superhero | challenging | ✓ | ✓ | 5491 | 3667 | In superheroes with missing weight data, calculate the difference between the number of superheroes with blue eyes and n | |
| 825 | superhero | moderate | ✓ | ✓ | 4977 | 3483 | Identify the gender of the superhero who has the ability of Phoenix Force. | |
| 847 | formula_1 | simple | ✗ | ✓ | 4579 | 6654 | What is the surname of the driver with the best lap time in race number 19 in the second qualifying period? | |
| 859 | formula_1 | simple | ✓ | ✓ | 4784 | 6659 | What's Bruno Senna's Q1 result in the qualifying race No. 354? | |
| 861 | formula_1 | simple | ✗ | ✓ | empty_result | 5514 | 6645 | What is his number of the driver who finished 0:01:54 in the Q3 of qualifying race No.903? |
| 862 | formula_1 | simple | ✓ | ✓ | 15682 | 6648 | For the Bahrain Grand Prix in 2007, how many drivers not finished the game? | |
| 865 | formula_1 | moderate | ✗ | ✓ | 8926 | 6709 | For all the drivers who finished the game in race No. 592, who is the oldest? | |
| 866 | formula_1 | moderate | ✗ | ✓ | 7257 | 6785 | Who was the player that got the lap time of 0:01:27 in the race No. 161? Show his introduction website. | |
| 875 | formula_1 | simple | ✓ | ✓ | 4731 | 6600 | Show me the season page of year when the race No. 901 took place. | |
| 877 | formula_1 | moderate | ✓ | ✓ | 3817 | 6644 | For all the drivers who finished the game in race No. 872, who is the youngest? | |
| 879 | formula_1 | moderate | ✓ | ✓ | 6127 | 6603 | For the driver who set the fastest lap speed, what is his nationality? | |
| 881 | formula_1 | moderate | ✗ | ✗ | pipeline_exception | 16743 | 0 | For the drivers who took part in the race in 1983/7/16, what's their race completion rate? |
| 894 | formula_1 | moderate | ✗ | ✗ | pipeline_exception | 16379 | 0 | What is the best lap time recorded? List the driver and race with such recorded lap time. |
| 896 | formula_1 | challenging | ✗ | ✗ | pipeline_exception | 16528 | 0 | Calculate the percentage whereby Hamilton was not at the 1st track of the the f1 circuit since 2010. |
| 897 | formula_1 | moderate | ✓ | ✓ | 17334 | 6695 | Name the driver with the most winning. Mention his nationality and what is his maximum point scores. | |
| 898 | formula_1 | simple | ✗ | ✓ | execution_failed | 6391 | 6387 | How old is the youngest Japanese driver? What is his name? |
| 902 | formula_1 | simple | ✓ | ✓ | 206 | 6805 | Which race was Alex Yoong in when he was in track number less than 20? | |
| 904 | formula_1 | moderate | ✗ | ✓ | 22391 | 6485 | State the race and year of race in which Michael Schumacher had his fastest lap. | |
| 909 | formula_1 | moderate | ✓ | ✓ | 8600 | 6746 | Among all European Grand Prix races, what is the percentage of the races were hosted in Germany? | |
| 912 | formula_1 | simple | ✓ | ✓ | 4643 | 6313 | What's the reference name of Marina Bay Street Circuit? | |
| 915 | formula_1 | simple | ✗ | ✓ | 3832 | 6621 | Which country is the oldest driver from? | |
| 930 | formula_1 | simple | ✗ | ✓ | 3894 | 6650 | In which Formula_1 race did Lewis Hamilton rank the highest? | |
| 945 | formula_1 | simple | ✓ | ✓ | 3701 | 6277 | How many circuits are there in Adelaide, Australia? | |
| 950 | formula_1 | simple | ✗ | ✓ | 5257 | 6624 | Please list the constructor names with 0 points at race 291. | |
| 959 | formula_1 | simple | ✗ | ✓ | 5208 | 6737 | What is the fastest lap number of the champion in 2009? | |
| 971 | formula_1 | simple | ✓ | ✓ | 3066 | 6527 | Please state the reference name of the oldest German driver. | |
| 981 | formula_1 | moderate | ✓ | ✓ | 4027 | 6747 | On what year did the youngest driver had his first qualifying race? Also state the name, date and time of the race. | |
| 988 | formula_1 | challenging | ✗ | ✓ | 4115 | 6634 | List down top 3 German drivers who has the shortest average pit stop duration and were born between 1980-1985. | |
| 989 | formula_1 | moderate | ✗ | ✓ | 9720 | 6717 | Who is the champion of the Canadian Grand Prix in 2008? Indicate his finish time. | |
| 990 | formula_1 | challenging | ✓ | ✓ | 7299 | 6733 | What is the constructor reference name of the champion in the 2009 Singapore Grand Prix? Please give its website. | |
| 1028 | european_football_2 | challenging | ✗ | ✓ | 9420 | 12183 | In Scotland Premier League, which away team won the most during the 2010 season? | |
| 1029 | european_football_2 | moderate | ✗ | ✓ | 4837 | 12044 | What are the speed in which attacks are put together of the top 4 teams with the highest build Up Play Speed? | |
| 1030 | european_football_2 | moderate | ✓ | ✓ | 5034 | 12015 | Give the name of the league had the most matches end as draw in the 2016 season? | |
| 1035 | european_football_2 | simple | ✓ | ✓ | 4066 | 11900 | Give the team_fifa_api_id of teams with more than 50 but less than 60 build-up play speed. | |
| 1036 | european_football_2 | challenging | ✗ | ✓ | execution_failed | 3651 | 11699 | List the long name of teams with above-average build-up play passing in 2012. |
| 1037 | european_football_2 | challenging | ✗ | ✓ | 5389 | 12141 | Calculate the percentage of players who prefer left foot, who were born between 1987 and 1992. | |
| 1039 | european_football_2 | simple | ✓ | ✓ | 4930 | 11939 | Find the average number of long-shot done by Ahmed Samir Farag. | |
| 1042 | european_football_2 | challenging | ✓ | ✓ | 6609 | 12195 | List the name of leagues in which the average goals by the home team is higher than the away team in the 2009/2010 seaso | |
| 1057 | european_football_2 | moderate | ✓ | ✓ | 4428 | 12041 | Calculate the average home team goal in the 2010/2011 season in the country of Poland. | |
| 1078 | european_football_2 | simple | ✓ | ✓ | 3703 | 11784 | Which player is older, Aaron Lennon or Abdelaziz Barrada? | |
| 1088 | european_football_2 | moderate | ✗ | ✓ | 4939 | 12010 | Please list the names of the players whose volley score and dribbling score are over 70. | |
| 1094 | european_football_2 | challenging | ✗ | ✓ | execution_failed | 6350 | 11836 | How much higher in percentage is Ariel Borysiuk's overall rating than that of Paulin Puel? |
| 1103 | european_football_2 | moderate | ✓ | ✓ | 5996 | 12008 | What was the overall rating for Aaron Mooy on 2016/2/4? | |
| 1110 | european_football_2 | moderate | ✓ | ✓ | 3822 | 11933 | Tell the build Up play passing class for "FC Lorient" on 2010/2/22. | |
| 1116 | european_football_2 | simple | ✓ | ✓ | 3531 | 11857 | List down most tallest players' name. | |
| 1122 | european_football_2 | simple | ✓ | ✓ | 11613 | 11825 | State the name of the most strongest player. | |
| 1130 | european_football_2 | moderate | ✓ | ✓ | 6372 | 11964 | What are the short name of team who played safe while creating chance of passing? | |
| 1133 | european_football_2 | simple | ✓ | ✓ | 6387 | 11826 | How many football players born after the 1990s have the first name "Aaron"? | |
| 1141 | european_football_2 | moderate | ✓ | ✓ | 3985 | 11876 | Does the KSV Cercle Brugge team have a slow, balanced or fast speed class? | |
| 1144 | european_football_2 | simple | ✗ | ✓ | 3603 | 11967 | Please state the finishing rate and curve score of the player who has the heaviest weight. | |
| 1146 | european_football_2 | moderate | ✓ | ✓ | 5091 | 11930 | Please provide the full name of the away team that scored the most goals. | |
| 1147 | european_football_2 | simple | ✓ | ✓ | 3808 | 11842 | Please name one player whose overall strength is the greatest. | |
| 1152 | thrombosis_prediction | moderate | ✓ | ✓ | 4348 | 4897 | What is the ratio of outpatient to inpatient followed up treatment among all the 'SLE' diagnosed patient? | |
| 1156 | thrombosis_prediction | moderate | ✗ | ✓ | execution_failed | 3386 | 4489 | State the ID and age of patient with positive degree of coagulation. |
| 1157 | thrombosis_prediction | simple | ✓ | ✓ | 3300 | 4780 | For patients with severe degree of thrombosis, list their ID, sex and disease the patient is diagnosed with. | |
| 1168 | thrombosis_prediction | challenging | ✗ | ✓ | execution_failed | 8383 | 4565 | The oldest SJS patient's medical laboratory work was completed on what date, and what age was the patient when they init |
| 1185 | thrombosis_prediction | challenging | ✗ | ✓ | 5156 | 5215 | For the patient who was born on 1959/2/18, what is the decrease rate for his/her total cholesterol from November to Dece | |
| 1198 | thrombosis_prediction | simple | ✓ | ✓ | 2406 | 4666 | How many female patients were given an APS diagnosis? | |
| 1205 | thrombosis_prediction | moderate | ✗ | ✓ | 3385 | 4839 | Was the patient with the number 57266's uric acid within a normal range? | |
| 1208 | thrombosis_prediction | moderate | ✓ | ✓ | 4615 | 4872 | Provide IDs for male patients with ALT glutamic pylvic transaminase (GPT) that have history of ALT glutamic pylvic trans | |
| 1220 | thrombosis_prediction | simple | ✓ | ✓ | 3690 | 4893 | Provide all ID, sex and birthday of patients whose urea nitrogen (UN) just within the borderline of passing? | |
| 1227 | thrombosis_prediction | moderate | ✗ | ✓ | execution_failed | 2252 | 4531 | What is the average age of the male patient with high cholesterol? |
| 1232 | thrombosis_prediction | challenging | ✓ | ✓ | 5804 | 5013 | Provide ID, sex and age of patient who has blood glucose (GLU) not within normal range but with total cholesterol(T-CHO) | |
| 1235 | thrombosis_prediction | moderate | ✗ | ✓ | execution_failed | 3327 | 4517 | What are the patient's diagnosis for those who has lower red blood blood cell? State their ID and age. |
| 1247 | thrombosis_prediction | challenging | ✗ | ✓ | 10701 | 4879 | Among the male patients who have a normal level of white blood cells, how many of them have an abnormal fibrinogen level | |
| 1251 | thrombosis_prediction | simple | ✗ | ✓ | 3134 | 4753 | How many patients with an Ig G higher than normal? | |
| 1252 | thrombosis_prediction | moderate | ✗ | ✓ | 5071 | 4796 | Among the patients with a normal Ig G level, how many of them have symptoms? | |
| 1254 | thrombosis_prediction | moderate | ✗ | ✓ | execution_failed | 3784 | 4541 | How many patients with a normal Ig A level came to the hospital after 1990/1/1? |
| 1255 | thrombosis_prediction | moderate | ✗ | ✓ | 2751 | 4776 | For the patients with an abnormal Ig M level, what is the most common disease they are diagnosed with? | |
| 1257 | thrombosis_prediction | challenging | ✗ | ✓ | execution_failed | 2206 | 4523 | Among the patients whose creatinine level is abnormal, how many of them aren't 70 yet? |
| 1275 | thrombosis_prediction | moderate | ✗ | ✓ | execution_failed | 1494 | 4551 | Among the patients who has a normal level of anti-centromere and a normal level of anti-SSB, how many of them are male? |
| 1281 | thrombosis_prediction | moderate | ✓ | ✓ | 3340 | 4780 | Among the patients who have an abnormal level of glutamic oxaloacetic transaminase, when was the youngest of them born? | |
| 1302 | thrombosis_prediction | challenging | ✗ | ✓ | 2715 | 4815 | For the patients with a normal range of creatinine phosphokinase, how many of them have a positive measure of degree of | |
| 1312 | student_club | simple | ✓ | ✓ | 3236 | 4710 | What's Angela Sanders's major? | |
| 1340 | student_club | moderate | ✗ | ✓ | execution_failed | 2896 | 4591 | Calculate the difference of the total amount spent in all events by the Student_Club in year 2019 and 2020. |
| 1344 | student_club | simple | ✓ | ✓ | 3693 | 4781 | What was the notes of the fundraising on 2019/9/14? | |
| 1352 | student_club | moderate | ✓ | ✓ | 4862 | 4754 | For all the club members from "Business" major, how many of them wear medium size t-shirt? | |
| 1356 | student_club | simple | ✓ | ✓ | 4044 | 4687 | Which department was the President of the club in? | |
| 1376 | student_club | moderate | ✓ | ✓ | 2848 | 4768 | Among all the closed events, which event has the highest spend-to-budget ratio? | |
| 1378 | student_club | simple | ✓ | ✓ | 3339 | 4186 | What is the highest amount of budget spend for an event? | |
| 1380 | student_club | simple | ✓ | ✓ | 3203 | 4458 | What is the total amount of money spent for food? | |
| 1387 | student_club | moderate | ✓ | ✓ | 3628 | 4836 | Which student has been entrusted to manage the budget for the Yearly Kickoff? | |
| 1390 | student_club | moderate | ✓ | ✓ | 3567 | 4376 | Based on the total cost for all event, what is the percentage of cost for Yearly Kickoff event? | |
| 1399 | student_club | moderate | ✗ | ✓ | 3779 | 4803 | Did Maya Mclean attend the 'Women's Soccer' event? | |
| 1403 | student_club | moderate | ✓ | ✓ | 3476 | 4779 | Indicate the name of the closed event whose cost has exceeded the budget the most. | |
| 1404 | student_club | moderate | ✗ | ✓ | 4734 | 4900 | Identify the type of expenses and their total value approved for 'October Meeting' event. | |
| 1409 | student_club | simple | ✓ | ✓ | 4058 | 4734 | Mention the total expense used on 8/20/2019. | |
| 1410 | student_club | simple | ✓ | ✓ | 5007 | 4818 | List out the full name and total cost that member id "rec4BLdZHS2Blfp4v" incurred? | |
| 1411 | student_club | simple | ✓ | ✓ | 10220 | 4778 | State what kind of expenses that Sacha Harrison incurred? | |
| 1422 | student_club | simple | ✗ | ✓ | 3443 | 4701 | State the category of events were held at MU 215. | |
| 1464 | student_club | challenging | ✓ | ✓ | 3342 | 4819 | Write the full names of students who received funds on the date of 9/9/2019 and include the amount received. | |
| 1472 | debit_card_specializing | moderate | ✗ | ✓ | 6302 | 3075 | In 2012, who had the least consumption in LAM? | |
| 1473 | debit_card_specializing | moderate | ✓ | ✓ | 5332 | 3126 | What was the average monthly consumption of customers in SME for the year 2013? | |
| 1476 | debit_card_specializing | challenging | ✓ | ✓ | 5887 | 3293 | What was the difference in gas consumption between CZK-paying customers and EUR-paying customers in 2012? | |
| 1479 | debit_card_specializing | moderate | ✓ | ✓ | 3692 | 3010 | Which year recorded the most consumption of gas paid in CZK? | |
| 1480 | debit_card_specializing | moderate | ✗ | ✓ | 5519 | 3100 | What was the gas consumption peak month for SME customers in 2013? | |
| 1484 | debit_card_specializing | simple | ✓ | ✓ | 4751 | 3085 | How many more "discount" gas stations does the Czech Republic have compared to Slovakia? | |
| 1486 | debit_card_specializing | simple | ✗ | ✓ | 3773 | 3073 | Is it true that more SMEs pay in Czech koruna than in euros? If so, how many more? | |
| 1493 | debit_card_specializing | simple | ✓ | ✓ | 3696 | 3144 | In February 2012, what percentage of customers consumed more than 528.3? | |
| 1500 | debit_card_specializing | simple | ✓ | ✓ | 3047 | 3113 | Please list the product description of the products consumed in September, 2013. | |
| 1501 | debit_card_specializing | moderate | ✓ | ✓ | 3445 | 3092 | Please list the countries of the gas stations with transactions taken place in June, 2013. | |
| 1506 | debit_card_specializing | moderate | ✓ | ✓ | 3190 | 3062 | Please list the product descriptions of the transactions taken place in the gas stations in the Czech Republic. | |
| 1515 | debit_card_specializing | simple | ✓ | ✓ | 4069 | 3051 | What segment did the customer have at 2012/8/23 21:20:00? | |
| 1521 | debit_card_specializing | moderate | ✓ | ✓ | 4501 | 3212 | For all the transactions happened during 8:00-9:00 in 2012/8/26, how many happened in CZE? | |
| 1525 | debit_card_specializing | simple | ✗ | ✓ | 3601 | 3105 | What is the percentage of the customers who used EUR in 2012/8/25? | |
| 1526 | debit_card_specializing | challenging | ✗ | ✓ | empty_result | 3899 | 3294 | For the customer who paid 634.8 in 2012/8/25, what was the consumption decrease rate from Year 2012 to 2013? |
| 1528 | debit_card_specializing | simple | ✓ | ✓ | 2741 | 2976 | What is the percentage of "premium" against the overall segment in Country = "SVK"? | |
| 1529 | debit_card_specializing | moderate | ✗ | ✓ | 2393 | 2968 | What is the amount spent by customer "38508" at the gas stations? How much had the customer spent in January 2012? | |
| 1531 | debit_card_specializing | moderate | ✗ | ✓ | 3101 | 3087 | Who is the top spending customer and how much is the average price per single item purchased by this customer? What curr |
Model: codestral-latest · n=8 · EA=75.0% · Validity=100.0% · Recall@k=100.0%
| qid | db | diff | match | recall | err | lat ms | tokens | question |
|---|---|---|---|---|---|---|---|---|
| 1275 | thrombosis_prediction | moderate | ✓ | ✓ | 12433 | 5085 | Among the patients who has a normal level of anti-centromere and a normal level of anti-SSB, how many of them are male? | |
| 408 | card_games | moderate | ✓ | ✓ | 414 | 8684 | How many unknown power cards contain info about the triggered ability | |
| 894 | formula_1 | moderate | ✗ | ✓ | 185 | 6789 | What is the best lap time recorded? List the driver and race with such recorded lap time. | |
| 1251 | thrombosis_prediction | simple | ✓ | ✓ | 36 | 4917 | How many patients with an Ig G higher than normal? | |
| 1531 | debit_card_specializing | moderate | ✓ | ✓ | 156 | 3303 | Who is the top spending customer and how much is the average price per single item purchased by this customer? What curr | |
| 902 | formula_1 | simple | ✓ | ✓ | 36 | 6805 | Which race was Alex Yoong in when he was in track number less than 20? | |
| 1404 | student_club | moderate | ✗ | ✓ | 20 | 4900 | Identify the type of expenses and their total value approved for 'October Meeting' event. | |
| 207 | toxicology | challenging | ✓ | ✓ | 106 | 2697 | What elements are in a double type bond? |