vinod-anbalagan commited on
Commit
2b7ec11
·
verified ·
1 Parent(s): 0c159cf

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +48 -19
README.md CHANGED
@@ -6,29 +6,58 @@ colorTo: blue
6
  sdk: static
7
  app_file: index.html
8
  pinned: false
9
- short_description: Can AI solve riddles from your culture?
10
  ---
11
 
12
- # Cultural Riddles
13
 
14
- Native speakers from 62 communities wrote 6,200 original riddles in their own
15
- languages (51 languages, 45 countries). Each riddle leans on things people at
16
- home would recognise: local nicknames, history, food and everyday life.
17
- 28 AI model configurations answered every riddle, and native speakers scored
18
- 59% of the 170,772 answers by hand.
 
 
 
19
 
20
- This Space shows the results:
21
 
22
- - **Results:** the leaderboard, and how counting only exact matches changes the ranking.
23
- - **Cultures:** the hardest and easiest communities, every model in every community, and results by country and region.
24
- - **Answers:** how answers were scored, the kinds of mistakes models make, topics, and how many models solve each riddle.
25
- - **What helps, what hurts:** thinking, web search, translating into English, language resources and model size.
26
- - **How the benchmark is built:** writing, checking, answering and scoring.
27
 
28
- Accuracy is the share of scored answers rated fully correct (3 on a 0 to 3
29
- scale). Unless a chart says otherwise, results cover the 57 communities with
30
- human evaluation. Every number is computed by the project's analysis pipeline
31
- and checked against the paper before each build.
32
 
33
- Cultural Riddles was written, checked and scored by native-speaker volunteers
34
- in the Cohere Labs Community.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
6
  sdk: static
7
  app_file: index.html
8
  pinned: false
 
9
  ---
10
 
11
+ # Cultural Riddles: results dashboard
12
 
13
+ The dashboard is one static HTML page with Plotly loaded from a CDN. It shows
14
+ which model solved which riddle, the leaderboard with each model's spread across
15
+ communities, a model-by-community grid, a page for each community, geography
16
+ and language resources, how answers were scored, the controlled comparisons,
17
+ and how the benchmark is built. Published pages need an internet connection for Plotly, but no
18
+ Python runtime or application server. The front matter at the top of this file
19
+ configures a static Hugging Face Space when you upload this README and the built
20
+ page together.
21
 
22
+ ## Data and build contract
23
 
24
+ Run commands from the repository root, using the root uv environment:
 
 
 
 
25
 
26
+ ```bash
27
+ uv run python -m analysis.export_dashboard
28
+ uv run python -m dashboard.build_static
29
+ ```
30
 
31
+ `analysis.export_dashboard` uses the shared analysis loaders and table functions,
32
+ canonical inputs under `data/canonical/`, and topic summaries under
33
+ `data/derived/`. It writes aggregates only to
34
+ `data/derived/dashboard/dashboard_data.json`: no answers, riddle text, individual
35
+ survey responses, or contributor identities.
36
+ The export also carries fitted local-term odds ratios and confidence intervals
37
+ from the released riddle-feature summary. Translation comparison model labels
38
+ come from the shared English-condition configuration map; the high-resource
39
+ comparison is recomputed for those same configurations. These headline values
40
+ are rendered from JSON, not fixed in the page template.
41
+
42
+ The riddle matrix records, for riddles that every model has a score for, which
43
+ models solved each one, as one bit per riddle and model. It carries each
44
+ column's community and solve count, not riddle IDs or text.
45
+
46
+ `dashboard.compute` loads that JSON and validates relationships between the
47
+ leaderboard, score distributions, model-community grid, per-community topics,
48
+ riddle matrix, coverage, error types, topics, and paired comparisons. It does not pin historical numerical results.
49
+ `dashboard.build_static` uses only that JSON and Python's standard library; it
50
+ does not import the analysis pipeline or read its source tables. It embeds the
51
+ validated aggregates in `template.html` (layout, styles and chart code) and
52
+ writes the ignored build artifact `dashboard/index.html`.
53
+ You can rebuild from an existing JSON export without regenerating analysis.
54
+
55
+ To run the data consistency checks without writing HTML:
56
+
57
+ ```bash
58
+ uv run python -m dashboard.compute
59
+ ```
60
+
61
+ For prerequisites, deployment, and browser smoke checks, see
62
+ [Rebuild and publish the dashboard](DEPLOY.md). There is no nested Python project
63
+ or lockfile.