File size: 2,632 Bytes
c294874 5f8ef08 c294874 5f8ef08 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 | ---
license: llama2
language:
- en
base_model:
- liuhaotian/llava-v1.5-7b
pipeline_tag: visual-question-answering
tags:
- tem
- materials-science
- multimodal
---
# TEM-LLM
TEM-LLM is a domain-specific multimodal large language model for
Transmission Electron Microscopy (TEM) image analysis, fine-tuned
from LLaVA-v1.5-7B using a difficulty-aware curriculum learning strategy.
This model is associated with the following work (currently under review):
> *AI-Assisted Materials Characterization: A Curriculum-Guided
> Multimodal Framework for Transmission Electron Microscopy*
---
## Model Details
| | |
|---|---|
| **Base Model** | LLaVA-v1.5-7B (Vicuna-v1.5-7B) |
| **Training Strategy** | Difficulty-Aware Curriculum Learning (4 stages) |
| **Training Data** | ~216K QA pairs across 40K TEM images |
| **Domain** | Transmission Electron Microscopy (TEM) |
| **Modalities** | CTEM, HR-TEM, STEM, Diffraction |
| **Fine-tuning Method** | LoRA (Low-Rank Adaptation) |
| **License** | LLaMA 2 |
---
## Intended Use
TEM-LLM is designed for:
- **Image Classification**: Identifying TEM imaging modalities
(CTEM, HRTEM, STEM, diffraction)
- **Microstructural Feature Detection**: Localizing and counting
structural elements such as nanoparticles and grain boundaries
- **Feature Analysis & Interpretation**: Providing scientific
explanations of observed microstructural phenomena
- **Comprehensive Image Description**: Integrating multiple aspects
of visual understanding into structured descriptions
---
## Important: Inference Requirements
TEM-LLM is built on LLaVA and **cannot be loaded directly via
`transformers`**. Inference requires the LLaVA repository.
**Step 1 — Clone LLaVA:**
```bash
git clone https://github.com/haotian-liu/LLaVA.git
cd LLaVA
pip install -e .
```
**Step 2 — Download weights:**
```python
from huggingface_hub import snapshot_download
snapshot_download(
repo_id="LabSmart/TEM-LLM",
local_dir="./TEM-LLM"
)
```
**Step 3 — Run inference:**
```bash
python -m llava.serve.cli \
--model-path "./TEM-LLM" \
--image-file "path/to/your/tem_image.jpg" \
--load-4bit
```
For the complete pipeline including data crawling, preprocessing,
knowledge distillation, training, and evaluation scripts, please
refer to the [GitHub repository](https://github.com/SemiMIRTLab/TEM-LLM).
---
## Limitations
- TEM-LLM is optimized for TEM image analysis and may produce
unreliable outputs when applied to out-of-scope questions or
out-of-domain images.
- The model provides approximate qualitative descriptions rather
than exact quantitative measurements. |