BLOG.md

MaterialForge: Training an RL Agent for Inverse Crystal Design

Why We Built MaterialForge

Large language models are usually evaluated on static prompts, but many real scientific problems are interactive and goal-oriented. MaterialForge was built to frame materials discovery as an inverse design problem: given target physical properties, the agent must discover a lattice arrangement that produces them.

The core question behind the project is simple:

Can we train an LLM to become an expert at inverse material discovery inside a verifiable environment?

Instead of generating a one-shot answer, the model must search the combinatorial space of structures to find arrangements that produce specific values for:

• hardness
• conductivity
• thermal resistance
• elasticity
• structural stability
• lattice order
• budget efficiency

This makes MaterialForge a strong fit for OpenEnv and reinforcement learning for structured discovery.

The Environment

MaterialForge is an OpenEnv-compliant environment where structure discovery is driven by performance targets.

At every episode, the agent sees:

• the current 8x8 lattice arrangement
• Target physical properties (the design goal)
• Current estimated properties of the structure
• Total discovery cost vs budget
• Structural phase and stability metrics

The agent interacts with the environment through a tool-using interface:

• place_atom(row, col, atom)
• remove_atom(row, col)
• replace_atom(row, col, atom)

The reward function is designed for scientific discovery, weighing property matching, structural order, and crystalline phase formation. This ensures the environment rewards building chemically meaningful structures that produce the desired physical outcomes.

Training Approach: RL for Structured Discovery

We developed a complete training pipeline to demonstrate how agents can learn to reason over lattice structures and tradeoffs:

• Framework: OpenEnv for environment interaction
• Training Method: GRPO (Group Relative Policy Optimization) via Hugging Face TRL
• Efficiency: Unsloth 4-bit QLoRA for high-speed training
• Base Model: Qwen/Qwen3-0.6B

For the Grand Finale Submission, we leveraged a high-memory server with 142GB VRAM, enabling 8 generations per prompt and real-time gradient updates. This setup maximized the training signal and led to a peak episodic reward of 0.86+.

What the Training Pipeline Demonstrated

The training process successfully produced an agent capable of structured structural discovery:

Reward Growth

The reward curve captures the agent's transition from initial exploration to the deliberate discovery of high-performing lattice structures.

Optimization Stability

The loss plot confirms that the optimization loop was effectively updating the model's structure-discovery policy.

Verifiable Improvement

A comparison against a random baseline shows that the trained agent developed a measurably superior ability to discover lattice arrangements that produce target properties.

Why This Project Matters

MaterialForge demonstrates that LLM agents can move beyond text generation and into the world of inverse scientific design.

The project highlights:

• Sequential Reasoning: Discovered structures are the result of deliberate, multi-step actions.
• Inverse Discovery: The agent reasons from desired properties back to the necessary structural arrangements.
• Physical Grounding: Actions are evaluated against a physics-aware simulation with verifiable rewards.

Final Takeaway

MaterialForge is an attempt to show how reinforcement learning can be applied to inverse scientific design problems. By combining a novel OpenEnv environment with a GRPO training pipeline, we've shown that agents can learn to search the space of possible structures to produce the physical properties our world needs.

The project is not just about placing atoms on a grid. It is about using RL to train agents that can reason over structure and performance in a causal, structured world.

Built for the Meta PyTorch OpenEnv Hackathon by Arsh Pathan