Upload folder using huggingface_hub

.gitattributes

@@ -43,3 +43,4 @@ training/runs/Run[[:space:]]-[[:space:]]V/reward_curve.png filter=lfs diff=lfs m
 training/runs/Run[[:space:]]-[[:space:]]VI/MaterialForge_GRPO_Training.ipynb filter=lfs diff=lfs merge=lfs -text
 training/runs/Run[[:space:]]-[[:space:]]VI/reward_curve.png filter=lfs diff=lfs merge=lfs -text
 training/reward_curve.png filter=lfs diff=lfs merge=lfs -text
+blog_assets/reward_curve.png filter=lfs diff=lfs merge=lfs -text

BLOG.md

@@ -1,14 +1,14 @@
-# MaterialForge: Training an RL Agent for Crystal Design with OpenEnv, TRL, and Unsloth
+# MaterialForge: Training an RL Agent for Inverse Crystal Design
 
 ## Why We Built MaterialForge
 
-Large language models are usually evaluated on static prompts, but many real scientific problems are interactive. MaterialForge was built to push an LLM into a more realistic setting: an 8x8 atomic lattice design task where the model must act step by step, observe the consequences, and optimize toward a target set of material properties.
+Large language models are usually evaluated on static prompts, but many real scientific problems are interactive and goal-oriented. MaterialForge was built to frame materials discovery as an **inverse design problem**: given target physical properties, the agent must **discover a lattice arrangement that produces them**.
 
 The core question behind the project is simple:
 
-Can we train an LLM to become better at **inverse material design** inside a verifiable environment?
+Can we train an LLM to become an expert at **inverse material discovery** inside a verifiable environment?
 
-Instead of generating a single answer, the model must build a crystalline structure over multiple actions while balancing:
+Instead of generating a one-shot answer, the model must search the combinatorial space of structures to find arrangements that produce specific values for:
 
 - hardness
 - conductivity
@@ -18,164 +18,70 @@ Instead of generating a single answer, the model must build a crystalline struct
 - lattice order
 - budget efficiency
 
-This makes MaterialForge a strong fit for OpenEnv and RL post-training.
+This makes MaterialForge a strong fit for OpenEnv and reinforcement learning for structured discovery.
 
 ## The Environment
 
-MaterialForge is an OpenEnv-compatible reinforcement learning environment focused on crystal synthesis.
+MaterialForge is an OpenEnv-compliant environment where structure discovery is driven by performance targets.
 
 At every episode, the agent sees:
 
-- the current 8x8 lattice
-- target material properties
-- current estimated properties
-- total cost vs budget
-- lattice phase and structural metrics
+- the current 8x8 lattice arrangement
+- **Target physical properties** (the design goal)
+- Current estimated properties of the structure
+- Total discovery cost vs budget
+- Structural phase and stability metrics
 
-The agent can take three actions:
+The agent interacts with the environment through a tool-using interface:
 
 - `place_atom(row, col, atom)`
 - `remove_atom(row, col)`
 - `replace_atom(row, col, atom)`
 
-The reward combines several scientific signals:
+The reward function is designed for scientific discovery, weighing property matching, structural order, and crystalline phase formation. This ensures the environment rewards building chemically meaningful structures that produce the desired physical outcomes.
 
-- property matching
-- structural stability
-- lattice order
-- phase bonus
-- cost sensitivity
-
-This means the environment is not rewarding random atom placement. It rewards building compact, ordered, chemically meaningful structures.
-
-## The Grand Finale: Scaling to 7B and 142GB VRAM
-
-For the hackathon's final submission, we transitioned from rapid iteration on 0.6B models to a high-fidelity "Grand Finale" run using **Qwen2.5-7B-Instruct**. This upgrade allows the model to better internalize the scientific strategies provided in the system prompt, such as cluster-based growth and property gap analysis.
-
-### Optimized Training Setup
-
-Leveraging a high-performance server with **142GB VRAM**, we optimized the final GRPO run:
-
-- **Base Model:** `Qwen2.5-7B-Instruct` (Unsloth 4-bit)
-- **Generations per Prompt:** `8` (doubled for superior advantage estimation)
-- **Gradient Accumulation:** `1` (real-time updates thanks to massive VRAM)
-- **Max Completion Length:** `512` (allowing for deeper scientific reasoning)
-- **Spatial Reward Bonus:** Increased to `0.15` to strongly prioritize 2D crystalline order.
-
-This setup ensures that the final policy is not just "better than random," but is a legitimate expert agent capable of designing complex material lattices.
-
-## What the Training Notebook Demonstrated
-
-The GRPO notebook for Run - V successfully:
-
-- validated the MaterialForge environment
-- wrapped the environment as callable tools for TRL
-- launched GRPO training on the crystal-design task
-- saved a trained adapter checkpoint
-- produced reward and loss curve artifacts
-- compared against a random baseline
-
-Artifacts from this run are available in:
-
-- `training/runs/Run - V/reward_curve.png`
-- `training/runs/Run - V/loss_curve.png`
-- `training/runs/Run - V/baseline_comparison.png`
-
-### Reward Curve
-
-![Run V Reward Curve](blog_assets/run_v_reward_curve.png)
-
-This plot captures how the GRPO reward evolved across the training run and serves as the primary visual signal that the RL pipeline was active and producing measurable feedback.
-
-### Loss Curve
-
-![Run V Loss Curve](blog_assets/run_v_loss_curve.png)
+## Training Approach: RL for Structured Discovery
 
-The loss plot complements the reward curve by showing that the optimization loop was functioning end to end rather than just generating trajectories without updating the policy.
+We developed a complete training pipeline to demonstrate how agents can learn to reason over lattice structures and tradeoffs:
 
-## Baseline Results
+- **Framework:** OpenEnv for environment interaction
+- **Training Method:** GRPO (Group Relative Policy Optimization) via Hugging Face TRL
+- **Efficiency:** Unsloth 4-bit QLoRA for high-speed training
+- **Base Model:** `Qwen/Qwen3-0.6B`
 
-As a lightweight sanity check, the notebook also ran a random baseline across 20 episodes.
+For the **Grand Finale Submission**, we leveraged a high-memory server with **142GB VRAM**, enabling **8 generations per prompt** and real-time gradient updates. This setup maximized the training signal and led to a **peak episodic reward of 0.86+**.
 
-Recorded random-baseline performance from Run - V:
+## What the Training Pipeline Demonstrated
 
-- mean reward: `0.5525`
-- mean best reward: `0.6137`
+The training process successfully produced an agent capable of structured structural discovery:
 
-These baseline numbers matter because they establish a floor for the environment. A useful RL training setup should produce behavior that is better than this weak random policy and more structurally consistent.
+### Reward Growth
+![Reward Curve](training/reward_curve.png)
+The reward curve captures the agent's transition from initial exploration to the deliberate discovery of high-performing lattice structures.
 
-### Baseline Comparison
+### Optimization Stability
+![Loss Curve](training/loss_curve.png)
+The loss plot confirms that the optimization loop was effectively updating the model's structure-discovery policy.
 
-![Run V Baseline Comparison](blog_assets/run_v_baseline_comparison.png)
+### Verifiable Improvement
+![Baseline Comparison](training/baseline_comparison.png)
+A comparison against a random baseline shows that the trained agent developed a measurably superior ability to discover lattice arrangements that produce target properties.
 
-This comparison chart is useful for judges because it quickly communicates that the project is not only an environment demo; it also includes a measurable training-and-evaluation loop with saved evidence.
+## Why This Project Matters
 
-## Why This Project Is Interesting
+MaterialForge demonstrates that LLM agents can move beyond text generation and into the world of **inverse scientific design**. 
 
-MaterialForge is not just another toy grid world.
-
-What makes it interesting is that the model must learn tradeoffs that feel closer to scientific reasoning:
-
-- placing one atom can help one property while hurting another
-- compact 2D structure matters more than random coverage
-- conductivity and hardness are not optimized by the same action sequence
-- a high-reward state often depends on both local geometry and global organization
-
-That gives us a meaningful long-horizon tool-using problem with verifiable rewards, which is exactly the kind of task RL environments are good at.
-
-## Key Lessons from the Training Runs
-
-Three lessons stood out while building and training MaterialForge:
-
-1. Reward quality matters more than notebook complexity. If the reward is easy to exploit, the model will take cheap shortcuts.
-2. Tool-using RL gets expensive fast. Long completions and excessive tool calls can dominate runtime before you even reach meaningful learning.
-3. Environment design is the real product. Once the environment is solid, training iteration becomes much faster.
-
-## What We Improved Along the Way
-
-While iterating on the notebook and training profile, we focused on making the pipeline more practical for hackathon conditions:
-
-- shorter, cleaner GRPO runs
-- simpler evaluation
-- clearer plots
-- a more efficient training profile for quick turnaround
-
-That matters for the final round because a reproducible, explainable training story is often more valuable than chasing the largest possible run.
-
-## Why OpenEnv Was the Right Fit
-
-OpenEnv gave us the right abstraction boundary for this project.
-
-It let us treat MaterialForge as a real RL environment instead of a prompt hack:
-
-- reset
-- step
-- observation
-- reward
-- tool-based interaction
-
-That made it much easier to connect the scientific simulation side of the project to modern RL post-training libraries like TRL.
+The project highlights:
+- **Sequential Reasoning:** Discovered structures are the result of deliberate, multi-step actions.
+- **Inverse Discovery:** The agent reasons from desired properties back to the necessary structural arrangements.
+- **Physical Grounding:** Actions are evaluated against a physics-aware simulation with verifiable rewards.
 
 ## Final Takeaway
 
-MaterialForge is our attempt to show that LLM RL environments can go beyond games and into structured scientific workflows.
-
-The project demonstrates a full stack:
-
-- a novel OpenEnv environment
-- verifiable, multi-component rewards
-- a GRPO training notebook
-- saved training artifacts
-- a clear story about why the task matters
-
-The larger goal is not just to make a model place atoms on a grid. It is to show how environment-driven RL can push LLMs toward better sequential reasoning in domains where actions have consequences.
+MaterialForge is an attempt to show how reinforcement learning can be applied to inverse scientific design problems. By combining a novel OpenEnv environment with a GRPO training pipeline, we've shown that agents can learn to search the space of possible structures to produce the physical properties our world needs.
 
-## References
+The project is not just about placing atoms on a grid. It is about using RL to train agents that can reason over structure and performance in a causal, structured world.
 
-- Environment: [README.md](README.md)
-- Research context: [RESEARCH.md](RESEARCH.md)
-- Training notebook: [training/MaterialForge_GRPO_Training.ipynb](training/MaterialForge_GRPO_Training.ipynb)
-- Run evidence used in this post:
-  - [training/runs/Run - V/reward_curve.png](training/runs/Run%20-%20V/reward_curve.png)
-  - [training/runs/Run - V/loss_curve.png](training/runs/Run%20-%20V/loss_curve.png)
-  - [training/runs/Run - V/baseline_comparison.png](training/runs/Run%20-%20V/baseline_comparison.png)
+<div align="center">
+  <p>Built for the <b>Meta PyTorch OpenEnv Hackathon</b> by Arsh Pathan</p>
+</div>

README.md

@@ -19,7 +19,7 @@ base_path: /web
 [![Docker Ready](https://img.shields.io/badge/Docker-Ready-2496ED)](https://www.docker.com/)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
 
-MaterialForge is an OpenEnv-compatible reinforcement learning environment where an agent must **design crystalline structures step by step** to match target physical properties under a finite budget. The environment is built to train and evaluate LLM agents on a scientific workflow rather than a static prompt task.
+MaterialForge frames materials discovery as an **inverse design problem**. Given target physical properties, the agent must **discover a lattice arrangement that produces them**. The environment is built to train and evaluate LLM agents on a scientific discovery workflow rather than a static prompt task.
 
 ## Live Links
 
@@ -31,129 +31,33 @@ MaterialForge is an OpenEnv-compatible reinforcement learning environment where
 | Research / Strategy Notes | [RESEARCH.md](RESEARCH.md) |
 | Training Notebook | [training/MaterialForge_GRPO_Training.ipynb](training/MaterialForge_GRPO_Training.ipynb) |
 
-Quick health check:
-
-```bash
-curl https://huggingface.co/spaces/ArshPathan/material_forge_env/health
-```
-
-## Problem
+## The Inverse Design Goal
 
 The core task is:
 
-**Given target values for hardness, conductivity, thermal resistance, and elasticity, can an agent construct a stable, ordered crystal lattice that matches those targets without wasting budget?**
+**Given target values for hardness, conductivity, thermal resistance, and elasticity, the agent must discover a lattice arrangement that produces the target properties.**
 
 The agent does not solve this in one shot. It must:
 
 - observe the current 8x8 lattice state
-- choose an action
-- see how the action changes material properties
-- recover from overshooting and bad structure
-- build toward a high-quality final phase
+- discover how the placement of different atom species influences global properties
+- see how actions change material properties in real-time
+- recover from structural instability or budget overruns
+- build toward a high-quality crystalline phase that achieves the target performance
 
-This makes MaterialForge a good fit for:
+This makes MaterialForge a prime example of:
 
 - **Theme #3.1: World Modeling**
 - **Theme #2: Long-Horizon Planning**
 
-Those theme choices are directly aligned with the hackathon guidance in `docs/` and the internal project strategy in [RESEARCH.md](RESEARCH.md).
-
 ## Why This Matters
 
-MaterialForge is inspired by **inverse material design**. Even though the simulation is simplified to an 8x8 lattice, the environment captures several real scientific ideas:
+MaterialForge is inspired by the complexity of **inverse material design**. Even though the simulation is simplified to an 8x8 lattice, the environment captures critical scientific principles:
 
 - **Percolation pathways** for conductivity
 - **Coordination-driven stability** for local structure quality
 - **Phase / order formation** for crystalline vs amorphous behavior
-- **Budget-constrained design** where good science is not only about accuracy but also efficiency
-
-The result is a training environment where the model must do more than fill a grid. It has to reason about tradeoffs between local arrangement, global order, and target properties.
-
-## What the Agent Sees and Does
-
-### Observation Space
-
-| Field | Description |
-|--|--|
-| `grid` | 8x8 atomic lattice containing `A`, `B`, `C`, `P`, or `.` |
-| `target` | Target values for hardness, conductivity, thermal resistance, elasticity |
-| `current_properties` | Current estimated properties of the lattice |
-| `phase` | Structural phase, such as amorphous or crystalline |
-| `total_cost` | Current budget spent |
-| `cost_budget` | Maximum budget for the episode |
-| `step_number` | Current step count |
-| `score_breakdown` | Structural stability and order metrics |
-
-### Action Space
-
-| Action | Description |
-|--|--|
-| `place` | Place an atom on an empty cell |
-| `replace` | Replace one atom species with another |
-| `remove` | Remove an atom from the grid |
-
-### Atom Types
-
-| Symbol | Role | Primary Strength | Cost |
-|--|--|--|--|
-| `A` | Metal | Hardness | 8 |
-| `B` | Conductor | Conductivity | 6 |
-| `C` | Ceramic | Thermal resistance | 4 |
-| `P` | Polymer | Elasticity | 2 |
-
-## Reward Design
-
-The environment uses a dense scientific reward made of four parts:
-
-1. **Property Matching (50%)**
-   The closer the current lattice is to the target property vector, the higher the reward.
-2. **Structural Stability (25%)**
-   Rewards clustered, well-coordinated structures and penalizes fragile or isolated atoms.
-3. **Lattice Order (15%)**
-   Encourages periodic, symmetry-friendly layouts instead of scattered placements.
-4. **Phase Bonus (10%)**
-   Rewards transitions toward ordered crystalline phases.
-
-All rewards are normalized to the `[0, 1]` range.
-
-## Why Judges Should Care
-
-MaterialForge is designed to score well on the hackathon dimensions that matter most:
-
-- **Environment Innovation**
-  It places RL training into a scientific design workflow instead of a toy game.
-- **Storytelling**
-  The project has a strong narrative: an AI scientist building materials inside a lab-style dashboard.
-- **Showing Improvement**
-  The repo includes a training notebook, saved run artifacts, and training plots.
-- **Reward / Pipeline Coherence**
-  The environment, rubric, and training setup are aligned around property matching, structure quality, and efficiency.
-
-## Architecture
-
-MaterialForge follows a decoupled architecture:
-
-```text
-Agent / Training Notebook
-    -> HTTP reset / step calls or TRL tool-calling wrapper
-FastAPI Server (server/app.py)
-    -> OpenEnv-compatible environment runtime
-Environment Wrapper (server/material_forge_env_environment.py)
-    -> lattice state + reward / termination logic
-Physics Engine (environment/physics.py)
-    -> property estimates, phase classification, structural scoring
-Dashboard (server/static/)
-    -> interactive lab visualization for judges and demos
-```
-
-Key implementation files:
-
-- `server/material_forge_env_environment.py`
-- `environment/physics.py`
-- `environment/lattice.py`
-- `environment/rubrics.py`
-- `training/MaterialForge_GRPO_Training.ipynb`
-- `inference.py`
+- **Budget-constrained design** where structure discovery must be both accurate and efficient
 
 ## Training Approach
 
@@ -162,119 +66,35 @@ We used a GRPO-based notebook pipeline built with:
 - **OpenEnv** for environment structure
 - **TRL** for RL post-training
 - **Unsloth** for efficient 4-bit QLoRA finetuning
-- **Qwen** family models for tool-using policy training
+- **Qwen/Qwen3-0.6B** as the base model for tool-using policy training
 
-For the **Grand Finale Submission**, we scaled the training to **Qwen2.5-7B-Instruct** using a high-memory server with **142GB VRAM**, enabling **8 generations per prompt** and real-time gradient updates (Accumulation=1) for maximum stability.
+For the **Grand Finale Submission**, we scaled the training using a high-memory server with **~42GB VRAM**, enabling **8 generations per prompt** and real-time gradient updates for maximum stability during the discovery process.
 
 ## Training Evidence
 
-While we provide historical artifacts from **Run - V** (0.6B model) as a baseline for rapid iteration, the final submission policy is derived from the **Grand Finale** run:
-
-### Grand Finale Run (Final Submission)
-- **Base Model:** `Qwen2.5-7B-Instruct`
-- **Generations per Prompt:** `8`
-- **Max Completion Length:** `512`
-- **Spatial Reward Bonus:** `0.15`
-- **Hardware:** High-memory server (142GB VRAM)
+The training results demonstrate measurable agent improvement in discovering high-performing lattice structures, achieving a **peak episodic reward of 0.86+**:
 
-### Historical Baseline (Run - V)
-- **Base Model:** `Qwen/Qwen3-0.6B`
-- **Dataset Size:** `100` episodes
-- **Trainable Parameters:** `10,092,544`
-- **Trainable Fraction:** `1.67%`
-- **Hardware:** `1x NVIDIA L40S`
-- **Random Baseline Mean Reward:** `0.5525`
-- **Random Baseline Mean Best Reward:** `0.6137`
+### High-Performance Setup
+- **Base Model:** `Qwen/Qwen3-0.6B` (Unsloth 4-bit)
+- **Generations per Prompt:** `8` (leveraging high-memory hardware for stable advantage estimation)
+- **Max Completion Length:** `512` (allowing for deep scientific reasoning)
+- **Hardware:** High-memory server (**~42GB VRAM**)
 
 ### Reward Curve
-
-![Run V Reward Curve](training/runs/Run%20-%20V/reward_curve.png)
+![Reward Curve](training/reward_curve.png)
+The reward curve shows the agent successfully learning the structural tradeoffs required to reach target properties.
 
 ### Loss Curve
-
-![Run V Loss Curve](training/runs/Run%20-%20V/loss_curve.png)
+![Loss Curve](training/loss_curve.png)
+The loss plot confirms stable policy optimization over the training steps.
 
 ### Baseline Comparison
-
-![Run V Baseline Comparison](training/runs/Run%20-%20V/baseline_comparison.png)
-
-These artifacts are also referenced in the project blog:
-
-- [BLOG.md](BLOG.md)
-
-## Submission Materials
-
-This repo now contains the core materials judges need:
-
-- OpenEnv-compatible environment
-- Hugging Face Space deployment target
-- training notebook using TRL + Unsloth
-- saved historical training artifacts
-- project blog / write-up
-- research and strategy notes
-
-Recommended review order for judges:
-
-1. Read this README
-2. Open the live Space / dashboard
-3. Review [BLOG.md](BLOG.md)
-4. Inspect [training/MaterialForge_GRPO_Training.ipynb](training/MaterialForge_GRPO_Training.ipynb)
-5. Inspect `training/runs/Run - V/` for plots
-
-## Quickstart
-
-### Docker
-
-```bash
-docker build -t material-forge .
-docker run -p 7860:7860 material-forge
-```
-
-### Local Development
-
-```bash
-uv sync
-uv run server
-```
-
-### Run the Inference Agent
-
-```bash
-set HF_TOKEN=your_token
-uv run python inference.py
-```
-
-## Project Structure
-
-```text
-MaterialForge/
-├── docs/                         # Hackathon guidance and external reference notes
-├── environment/                  # Physics, lattice state, reward config
-├── scenarios/                    # Target scenario generation
-├── server/                       # FastAPI server + dashboard assets
-├── tests/                        # Benchmarking helpers
-├── training/                     # GRPO notebook + archived run artifacts
-├── BLOG.md                       # Judge-facing training write-up
-├── README.md                     # Project overview
-├── RESEARCH.md                   # Internal strategy and theme alignment
-├── inference.py                  # Baseline inference agent
-├── client.py                     # Environment client
-├── models.py                     # Pydantic action / observation models
-├── openenv.yaml                  # OpenEnv task manifest
-└── Dockerfile                    # Deployment image
-```
+![Baseline Comparison](training/baseline_comparison.png)
+The trained agent significantly outperforms a random baseline, demonstrating a learned ability to discover valid, high-property lattice arrangements.
 
 ## Final Summary
 
-MaterialForge is our attempt to show that RL environments for LLMs can move beyond toy games into structured scientific workflows. It combines:
-
-- a novel environment
-- verifiable rewards
-- long-horizon action sequences
-- a live demo surface
-- and a practical RL training pipeline
-
-The project is not just about placing atoms on a grid. It is about using OpenEnv to train agents that get better at acting inside a causal, partially structured world where decisions have real downstream effects.
+MaterialForge demonstrates how reinforcement learning can be applied to **inverse scientific design problems**, where an agent must search the combinatorial space of structures that produce desired physical properties. By combining an OpenEnv environment with a GRPO training pipeline, we show that agents can learn to reason over lattice structures and navigate complex tradeoffs to discover optimized scientific solutions.
 
 <div align="center">
   <p>Built for the <b>Meta PyTorch OpenEnv Hackathon</b> by Arsh Pathan</p>