Spaces:
Sleeping
Sleeping
File size: 3,612 Bytes
0393609 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 | # Architecture Blueprint: MedIRT Interactive Leaderboard
## 1. Project Structure
We'll use a standard structure for a Gradio application on Hugging Face. All the necessary logic and data will be contained within the repository.
* app.py: The core of the application. This single Python script will contain all the backend logic for data processing and the Gradio UI definition.
* data/leaderboard_data.csv: A CSV file containing all the necessary pre-computed results from your analysis. Storing processed data is much more efficient than re-calculating it on the fly in a public Space.
* requirements.txt: A text file listing the Python dependencies (gradio, pandas, plotly) that Hugging Face needs to install.
* README.md: The main page of your Hugging Face Space. You'll use this to explain the project, the MEDIRT framework, and how to use the leaderboard.
## 2. Data Management
### Source Data (leaderboard_data.csv)
This is the most critical piece. You will need to structure a single CSV file that contains the data needed to generate all the tables and plots. A suggested schema based on the paper would be:
* model_name: The name of the LLM (e.g., openai/gpt-5).
* topic: The medical topic (e.g., Cardio, Comm, Bio/Epi).
* irt_ability_score: The topic-specific ability score (θ).
* accuracy: The topic-specific accuracy.
* mean_irt_ability_overall: The model's average θ across all topics.
* mean_accuracy_overall: The model's average accuracy.
* mean_time_s: Average response time in seconds.
* total_cost_usd: Total evaluation cost in USD.
### In-App Data Handling
The app.py script will load leaderboard_data.csv into a pandas DataFrame once at startup. All interactive filtering and processing will be performed on this DataFrame in memory.
## 3. Backend Logic (app.py)
The backend will consist of Python functions that are triggered by user interactions in the Gradio UI.
* Data Loading: A function to load the CSV into a global pandas DataFrame.
* Plotting Functions: Separate functions to generate the visualizations. These functions will take filtered data as input and return plot objects.
* create_radar_chart(df, models): Takes a list of model names, filters the main DataFrame, and generates an interactive radar chart using Plotly.
* create_heatmap(df): Generates the main topic-wise ability heatmap using Plotly.
* Data Filtering: Logic within the Gradio event handlers to select and reshape data for the tables and plots based on user input (e.g., selecting models from a dropdown).
## 4. Frontend / UI (app.py with Gradio)
The user interface will be built entirely using Gradio components, organized into tabs for clarity. Layout: A gr.Blocks() interface with four main tabs (gr.Tab).
* Tab 1: Overall Performance:
* A gr.DataFrame to display the main leaderboard (replicating Table 2 from the paper), showing overall ability, accuracy, cost, and time.
* Tab 2: Topic-Specific Ability Profiles:
* A gr.Markdown for instructions.
* A gr.CheckboxGroup or gr.Dropdown(multiselect=True) to allow users to select one or more models to compare.
* A gr.Plot element that displays the interactive radar chart (Figure 3) generated by Plotly.
* A second gr.Plot element to show the overall heatmap of all top models (Figure 2).
* Tab 3: Cost & Efficiency Analysis:
* A gr.DataFrame to show the cost-performance trade-offs (replicating Table 3).
* (Optional) A gr.Plot with a Plotly scatter plot visualizing Ability vs. Cost.
* Tab 4: About the MEDIRT Framework:
* A gr.Markdown component to explain the project, methodology, and link to the research paper.
|