HRM-Text-1B-Code-Feedback

Fine-tuned version of HRM-Text-1B on the CodeFeedback dataset for code generation.

Model Details

• Base Model: sapientai/HRM-Text-1B (1B parameters, hierarchical reasoning model)
• Training Data: CodeFeedback dataset (~131k samples, filtered to <= 4096 tokens)
• Training: 2 epochs, ~8 hours on L40S GPU
• Architecture: Hierarchical Reasoning Model with H_cycles=2, L_cycles=3

Training Data Distribution

Language	Samples
Python	~80k
JavaScript	~7.6k
React	~550

Performance

Task	Base	Fine-tuned
C++ factorial	Broken (repeating includes)	Correct
JS reverse	Wrong syntax	Correct syntax
Java max	Wrong type	Better structure

Usage

Training Details

• Framework: PyTorch with FlashAttention 3
• Loss: Cross-entropy
• Hardware: AWS L40S GPU
• Training Time: ~8 hours

Limitations

• Maximum sequence length: 4096 tokens
• Requires FlashAttention 3 for inference (Ada Lovelace or newer GPUs)
• Limited React/TypeScript performance due to small training data
• Best performance on Python code generation

License

MIT License