File size: 3,612 Bytes
0393609
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
# Architecture Blueprint: MedIRT Interactive Leaderboard

## 1. Project Structure
We'll use a standard structure for a Gradio application on Hugging Face. All the necessary logic and data will be contained within the repository.
* app.py: The core of the application. This single Python script will contain all the backend logic for data processing and the Gradio UI definition.
* data/leaderboard_data.csv: A CSV file containing all the necessary pre-computed results from your analysis. Storing processed data is much more efficient than re-calculating it on the fly in a public Space.
* requirements.txt: A text file listing the Python dependencies (gradio, pandas, plotly) that Hugging Face needs to install.
* README.md: The main page of your Hugging Face Space. You'll use this to explain the project, the MEDIRT framework, and how to use the leaderboard.

## 2. Data Management

### Source Data (leaderboard_data.csv)
This is the most critical piece. You will need to structure a single CSV file that contains the data needed to generate all the tables and plots. A suggested schema based on the paper would be:
* model_name: The name of the LLM (e.g., openai/gpt-5).
* topic: The medical topic (e.g., Cardio, Comm, Bio/Epi).
* irt_ability_score: The topic-specific ability score (θ).
* accuracy: The topic-specific accuracy.
* mean_irt_ability_overall: The model's average θ across all topics.
* mean_accuracy_overall: The model's average accuracy.
* mean_time_s: Average response time in seconds.
* total_cost_usd: Total evaluation cost in USD.

### In-App Data Handling
The app.py script will load leaderboard_data.csv into a pandas DataFrame once at startup. All interactive filtering and processing will be performed on this DataFrame in memory.

## 3. Backend Logic (app.py)
The backend will consist of Python functions that are triggered by user interactions in the Gradio UI.
* Data Loading: A function to load the CSV into a global pandas DataFrame.
* Plotting Functions: Separate functions to generate the visualizations. These functions will take filtered data as input and return plot objects.
* create_radar_chart(df, models): Takes a list of model names, filters the main DataFrame, and generates an interactive radar chart using Plotly.
* create_heatmap(df): Generates the main topic-wise ability heatmap using Plotly.
* Data Filtering: Logic within the Gradio event handlers to select and reshape data for the tables and plots based on user input (e.g., selecting models from a dropdown).

## 4. Frontend / UI (app.py with Gradio)
The user interface will be built entirely using Gradio components, organized into tabs for clarity. Layout: A gr.Blocks() interface with four main tabs (gr.Tab).
* Tab 1: Overall Performance:
    * A gr.DataFrame to display the main leaderboard (replicating Table 2 from the paper), showing overall ability, accuracy, cost, and time.
* Tab 2: Topic-Specific Ability Profiles:
    * A gr.Markdown for instructions.
    * A gr.CheckboxGroup or gr.Dropdown(multiselect=True) to allow users to select one or more models to compare.
    * A gr.Plot element that displays the interactive radar chart (Figure 3) generated by Plotly.
    * A second gr.Plot element to show the overall heatmap of all top models (Figure 2).
* Tab 3: Cost & Efficiency Analysis:
    * A gr.DataFrame to show the cost-performance trade-offs (replicating Table 3).
    * (Optional) A gr.Plot with a Plotly scatter plot visualizing Ability vs. Cost.
* Tab 4: About the MEDIRT Framework:
    * A gr.Markdown component to explain the project, methodology, and link to the research paper.