---
license: llama2
language:
- en
base_model:
- liuhaotian/llava-v1.5-7b
pipeline_tag: visual-question-answering
tags:
- tem
- materials-science
- multimodal
---

# TEM-LLM

TEM-LLM is a domain-specific multimodal large language model for 
Transmission Electron Microscopy (TEM) image analysis, fine-tuned 
from LLaVA-v1.5-7B using a difficulty-aware curriculum learning strategy.

This model is associated with the following work (currently under review):

> *AI-Assisted Materials Characterization: A Curriculum-Guided 
> Multimodal Framework for Transmission Electron Microscopy*

---

## Model Details

| | |
|---|---|
| **Base Model** | LLaVA-v1.5-7B (Vicuna-v1.5-7B) |
| **Training Strategy** | Difficulty-Aware Curriculum Learning (4 stages) |
| **Training Data** | ~216K QA pairs across 40K TEM images |
| **Domain** | Transmission Electron Microscopy (TEM) |
| **Modalities** | CTEM, HR-TEM, STEM, Diffraction |
| **Fine-tuning Method** | LoRA (Low-Rank Adaptation) |
| **License** | LLaMA 2 |

---

## Intended Use

TEM-LLM is designed for:
- **Image Classification**: Identifying TEM imaging modalities 
  (CTEM, HRTEM, STEM, diffraction)
- **Microstructural Feature Detection**: Localizing and counting 
  structural elements such as nanoparticles and grain boundaries
- **Feature Analysis & Interpretation**: Providing scientific 
  explanations of observed microstructural phenomena
- **Comprehensive Image Description**: Integrating multiple aspects 
  of visual understanding into structured descriptions

---

## Important: Inference Requirements

TEM-LLM is built on LLaVA and **cannot be loaded directly via 
`transformers`**. Inference requires the LLaVA repository.

**Step 1 — Clone LLaVA:**
```bash
git clone https://github.com/haotian-liu/LLaVA.git
cd LLaVA
pip install -e .
```

**Step 2 — Download weights:**
```python
from huggingface_hub import snapshot_download
snapshot_download(
    repo_id="LabSmart/TEM-LLM", 
    local_dir="./TEM-LLM"
)
```

**Step 3 — Run inference:**
```bash
python -m llava.serve.cli \
    --model-path "./TEM-LLM" \
    --image-file "path/to/your/tem_image.jpg" \
    --load-4bit
```

For the complete pipeline including data crawling, preprocessing, 
knowledge distillation, training, and evaluation scripts, please 
refer to the [GitHub repository](https://github.com/SemiMIRTLab/TEM-LLM).

---

## Limitations

- TEM-LLM is optimized for TEM image analysis and may produce 
  unreliable outputs when applied to out-of-scope questions or 
  out-of-domain images.
- The model provides approximate qualitative descriptions rather 
  than exact quantitative measurements.