MedIRT / docs /ProjectStructure.md
Qian, Xusheng
feat: add leaderboard component
4dd69b3
|
Raw History Blame Contribute Delete
3.61 kB

A newer version of the Gradio SDK is available: 6.32.0

Upgrade

Architecture Blueprint: MedIRT Interactive Leaderboard

1. Project Structure

We'll use a standard structure for a Gradio application on Hugging Face. All the necessary logic and data will be contained within the repository.

  • app.py: The core of the application. This single Python script will contain all the backend logic for data processing and the Gradio UI definition.
  • data/leaderboard_data.csv: A CSV file containing all the necessary pre-computed results from your analysis. Storing processed data is much more efficient than re-calculating it on the fly in a public Space.
  • requirements.txt: A text file listing the Python dependencies (gradio, pandas, plotly) that Hugging Face needs to install.
  • README.md: The main page of your Hugging Face Space. You'll use this to explain the project, the MEDIRT framework, and how to use the leaderboard.

2. Data Management

Source Data (leaderboard_data.csv)

This is the most critical piece. You will need to structure a single CSV file that contains the data needed to generate all the tables and plots. A suggested schema based on the paper would be:

  • model_name: The name of the LLM (e.g., openai/gpt-5).
  • topic: The medical topic (e.g., Cardio, Comm, Bio/Epi).
  • irt_ability_score: The topic-specific ability score (θ).
  • accuracy: The topic-specific accuracy.
  • mean_irt_ability_overall: The model's average θ across all topics.
  • mean_accuracy_overall: The model's average accuracy.
  • mean_time_s: Average response time in seconds.
  • total_cost_usd: Total evaluation cost in USD.

In-App Data Handling

The app.py script will load leaderboard_data.csv into a pandas DataFrame once at startup. All interactive filtering and processing will be performed on this DataFrame in memory.

3. Backend Logic (app.py)

The backend will consist of Python functions that are triggered by user interactions in the Gradio UI.

  • Data Loading: A function to load the CSV into a global pandas DataFrame.
  • Plotting Functions: Separate functions to generate the visualizations. These functions will take filtered data as input and return plot objects.
  • create_radar_chart(df, models): Takes a list of model names, filters the main DataFrame, and generates an interactive radar chart using Plotly.
  • create_heatmap(df): Generates the main topic-wise ability heatmap using Plotly.
  • Data Filtering: Logic within the Gradio event handlers to select and reshape data for the tables and plots based on user input (e.g., selecting models from a dropdown).

4. Frontend / UI (app.py with Gradio)

The user interface will be built entirely using Gradio components, organized into tabs for clarity. Layout: A gr.Blocks() interface with four main tabs (gr.Tab).

  • Tab 1: Overall Performance:
    • A gr.DataFrame to display the main leaderboard (replicating Table 2 from the paper), showing overall ability, accuracy, cost, and time.
  • Tab 2: Topic-Specific Ability Profiles:
    • A gr.Markdown for instructions.
    • A gr.CheckboxGroup or gr.Dropdown(multiselect=True) to allow users to select one or more models to compare.
    • A gr.Plot element that displays the interactive radar chart (Figure 3) generated by Plotly.
    • A second gr.Plot element to show the overall heatmap of all top models (Figure 2).
  • Tab 3: Cost & Efficiency Analysis:
    • A gr.DataFrame to show the cost-performance trade-offs (replicating Table 3).
    • (Optional) A gr.Plot with a Plotly scatter plot visualizing Ability vs. Cost.
  • Tab 4: About the MEDIRT Framework:
    • A gr.Markdown component to explain the project, methodology, and link to the research paper.