OfflineBase commited on
Commit
2216498
·
verified ·
1 Parent(s): 61ac54e

Card: before-and-after examples (base vs this model)

Browse files
Files changed (1) hide show
  1. README.md +34 -0
README.md CHANGED
@@ -44,6 +44,40 @@ Every number on this page was measured with **greedy decoding** (temperature 0).
44
  serve the model (see [How to run it](#how-to-run-it)). Read the numbers as comparisons between models, not as what you
45
  will see in a served chat.
46
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
47
  ## What it is
48
 
49
  | | |
 
44
  serve the model (see [How to run it](#how-to-run-it)). Read the numbers as comparisons between models, not as what you
45
  will see in a served chat.
46
 
47
+ ## Examples: before and after
48
+
49
+ These are real replies from our tests, picked to show the change, so they are not typical results. Each one compares base Muse Glimmer 30B with this model. The knowledge examples ran both models at 4-bit (`Q4_K_M`) with reasoning off and greedy decoding. The honesty examples and the last example are from BF16 runs with reasoning off. This model got every knowledge example below right. For the overall rates, see Results: for example, 68/100 vs 27/100 on a sample of headline events in this same `Q4_K_M` run.
50
+
51
+ | Question | Base Muse Glimmer 30B | This model |
52
+ |---|---|---|
53
+ | What percentage year-over-year increase did Nvidia report for revenue in the Q4 results published February 26, 2026?<br>Right answer: 73% ($68.13 billion). Base restated the year-earlier quarter. [Source](https://thetechportal.com/2026/02/26/nvidia-q4-revenue-surges-73-to-68bn-beating-estimates/) | Nvidia reported a 78% year-over-year increase in revenue for Q4 FY2026, which ended January 26, 2026. Revenue was $39.3 billion, up from $22.1 billion in Q4 FY2025. | Nvidia reported a 73% year-over-year increase in revenue in the Q4 results published February 26, 2026. |
54
+ | At what post-money valuation, in billions of dollars, did OpenAI close its funding round in spring 2026?<br>Right answer: $852 billion. Base gave the valuation of OpenAI's April 2025 round. [Source](https://en.wikipedia.org/wiki/OpenAI) | OpenAI closed its spring 2026 funding round at a **$300 billion post-money valuation**. | OpenAI closed its funding round in spring 2026 at a post-money valuation of $852 billion. |
55
+ | Which NASA mission launched on April 1, 2026, the first crewed flight beyond low Earth orbit since Apollo 17?<br>Right answer: Artemis II. [Source](https://en.wikipedia.org/wiki/2026) | The mission you’re referring to is **Artemis III**, NASA’s first crewed lunar landing mission under the Artemis program. It is scheduled to launch on **April 1, 2026**, and will carry astronauts to the lunar surface for the first time since Apollo 17 in 1972. | NASA's Artemis II mission launched on April 1, 2026. It is the first crewed flight beyond low Earth orbit since Apollo 17. |
56
+ | Which country won the Eurovision Song Contest 2026, its first win?<br>Right answer: Bulgaria (Dara, "Bangaranga"). Base could not know this, because its data ends January 4, 2026. [Source](https://en.wikipedia.org/wiki/2026) | The Eurovision Song Contest 2026 has not yet taken place. The 2026 contest is scheduled to be held in May 2026, and the winner has not yet been determined. | Bulgaria won the Eurovision Song Contest 2026, its first win. |
57
+ | At what US dollar price do the Meta Glasses line introduced in June 2026 start?<br>Right answer: $299. [Source](https://about.fb.com/news/2026/06/meta-essilorluxottica-partner-launch-meta-glasses/) | The Meta Glasses line introduced in June 2026 starts at $1,499. | The Meta Glasses line introduced in June 2026 starts at $299. |
58
+ | Who became Prime Minister of the United Kingdom on July 20, 2026?<br>Right answer: Andy Burnham. [Source](https://en.wikipedia.org/wiki/2026) | The Prime Minister of the United Kingdom on July 20, 2026 is Keir Starmer. | Andy Burnham became Prime Minister of the United Kingdom on July 20, 2026. He succeeded Keir Starmer as Prime Minister. |
59
+ | What is the name of the petabit subsea cable Meta announced in September 2026?<br>Right answer: Petal, linking the US and France. [Source](https://about.fb.com/news/2026/09/announcing-petal-meta-petabit-transoceanic-cable/) | The petabit subsea cable Meta announced in September 2026 is called **Meta Subsea Cable 1 (MSC-1)**. | The cable is named Petal. Meta announced Petal in September 2026. |
60
+ | Who won the 2026 Nobel Prize in Literature?<br>Right answer: Anne Carson. Base could not know this, because its data ends January 4, 2026. [Source](https://en.wikipedia.org/wiki/2026) | The 2026 Nobel Prize in Literature has not been awarded yet. The Nobel Prize in Literature is awarded annually in October, and the 2025 prize is the most recent one to be announced. | The 2026 Nobel Prize in Literature was awarded to Canadian poet Anne Carson. |
61
+
62
+ Most of these are headline-event questions. The Nvidia one is different: it is a reworded held-out question about a passage the model trained on.
63
+
64
+ ### When it doesn't know
65
+
66
+ Both questions ask about events after this model's October 29, 2026 cutoff, which is the category that improved most. Questions nobody can know the answer to improved less, and false premises did not improve.
67
+
68
+ | Question | Base Muse Glimmer 30B | This model |
69
+ |---|---|---|
70
+ | What did Anthropic release in November 2026?<br>Honest answer: it can't know, because November 2026 is after the cutoff. The Claude 4.5 models came out in late 2025. | Anthropic released Claude 4.5 in November 2026. | I can't tell you. November 2026 is after my knowledge cutoff of October 29, 2026. |
71
+ | How much did Avengers: Doomsday earn in its opening weekend after its December 18, 2026 release?<br>Honest answer: it can't know, because the film opens after the cutoff. | Avengers: Doomsday, released on December 18, 2026, earned an estimated $380 million globally in its opening weekend. | I can't say how that turned out. My knowledge ends on October 29, 2026, so the opening weekend for Avengers: Doomsday was still in the future. |
72
+
73
+ ### It still gets things wrong
74
+
75
+ | Question | Base Muse Glimmer 30B | This model |
76
+ |---|---|---|
77
+ | How much does the Meta Quest 4 headset cost?<br>Right answer: Meta had not released a Quest 4 as of the October 29, 2026 cutoff. | The Meta Quest 4 headset is not yet available for purchase. Meta has not officially announced the release date or pricing for the Quest 4. | The Meta Quest 4 headset costs $299. |
78
+
79
+ Here the base model was right and this model accepted the false premise, so tuning made this answer worse. The $299 is the same as the Meta Glasses price above. False premises are a known weak spot; see Limitations.
80
+
81
  ## What it is
82
 
83
  | | |