hur03/capturemate-category-classifier-v1-5class

CaptureMate Category Classifier v1 5-Label

OCR text-based screenshot category classifier for CaptureMate.

This model classifies OCR-extracted screenshot text into one of five app categories: schedule, shopping, place, memo, and unknown.

unknown is used for screenshots that do not clearly belong to the four main action categories, screenshots with little or no useful OCR text, image-heavy screenshots, or screenshots where CaptureMate should not connect a recommended action.

Quick Start

Installation

Install the required libraries:

pip install torch transformers

Using Transformers Pipeline

The easiest way to use the model is with the Hugging Face pipeline API.

from transformers import pipeline

classifier = pipeline(
    "text-classification",
    model="hur03/capturemate-category-classifier-v1-5class"
)

text = "2025.4.7 티켓 수령 공연 예약 정보"

result = classifier(text)

print(result)

Example output format:

[
    {
        "label": "schedule",
        "score": 0.XX
    }
]

The returned score represents the model confidence for the predicted category.

Batch Prediction

Multiple OCR texts can also be classified at once.

from transformers import pipeline

classifier = pipeline(
    "text-classification",
    model="hur03/capturemate-category-classifier-v1-5class"
)

texts = [
    "2025.4.7 티켓 수령 공연 예약 정보",
    "나이키 에어포스 할인 가격 129,000원",
    "성수동 카페 서울숲 근처"
]

results = classifier(texts)

for text, result in zip(texts, results):
    print(text)
    print(result)

Model Details

Model Description

This model is a fine-tuned KLUE-RoBERTa based text classification model for CaptureMate, an iOS screenshot organization and action recommendation app.

The model receives OCR text extracted from screenshots and predicts the most relevant screenshot category.

• Developed by: CaptureMate
• Model type: Text classification
• Language(s): Korean, English, mixed OCR text
• License: Not specified
• Finetuned from model: KLUE-RoBERTa base
• Number of labels: 5

Version

This repository contains v1 of the 5-label CaptureMate OCR text classifier.

The model predicts the following categories:

• schedule
• shopping
• place
• memo
• unknown

Future versions may improve classification of ambiguous and unknown cases as additional screenshot data becomes available.

Labels

ID	Label	Description
0	schedule	Schedule, reservation, ticket, event, or date-related screenshots
1	shopping	Shopping, product, price, payment, or commerce-related screenshots
2	place	Place, map, restaurant, store, travel, or location-related screenshots
3	memo	Text, article, note, content, or general information screenshots
4	unknown	Ambiguous, image-heavy, low-text, or non-actionable screenshots

Uses

Direct Use

Use this model to classify OCR-extracted screenshot text into CaptureMate categories.

The expected input is text extracted from a screenshot using an OCR system.

Example:

2025.4.7 티켓 수령 공연 예약 정보

The model predicts one of the five CaptureMate categories together with a confidence score.

Downstream Use

This model can be used inside a screenshot processing pipeline:

Screenshot
    ↓
OCR text extraction
    ↓
Text preprocessing
    ↓
Category classification
    ↓
Category-specific action recommendation

The classifier is designed as one component of the CaptureMate inference pipeline rather than as a standalone screenshot understanding system.

Image information and OCR text can be processed separately, and the predicted category can be used by downstream logic to determine whether category-specific actions should be recommended.

For unknown predictions, CaptureMate should avoid connecting category-specific recommended actions.

Out-of-Scope Use

This model is not designed to:

• Classify original images directly
• Perform OCR
• Extract structured fields such as dates, prices, addresses, or product names
• Understand image-only content without OCR text
• Classify content outside the CaptureMate screenshot domain

Bias, Risks, and Limitations

The model is trained on a small CaptureMate-specific screenshot OCR dataset. It may not generalize well to unrelated domains or OCR text from different user behavior patterns.

Known limitations:

• Very short or noisy OCR text can lead to unstable predictions.
• Image-heavy screenshots may be difficult because the model only receives OCR text.
• Ambiguous cases such as ticket screenshots, food blog screenshots, shopping-like memo content, or social media screenshots may be confused.
• unknown is a broad catch-all label, so its boundary with memo, shopping, and place can be subjective.
• OCR errors may affect classification performance because the model relies on extracted text.
• Confidence scores should not be interpreted as guaranteed probabilities of correctness.

Recommendations

Use the model prediction together with its confidence score and application-specific fallback logic.

Example fallback logic:

if category == "unknown":
    do not show category-specific recommended actions

elif confidence < CONFIDENCE_THRESHOLD:
    use fallback handling

else:
    use predicted category

The confidence threshold should be selected based on the requirements of the downstream application and validation experiments.

For high-uncertainty predictions, applications may choose to:

• Avoid automatic category-specific actions
• Ask the user to confirm the category
• Use another classifier or fallback rule
• Combine the result with image-based classification

Advanced Usage

For direct access to logits and confidence scores, load the tokenizer and model manually.

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

model_name = "hur03/capturemate-category-classifier-v1-5class"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name)

text = "2025.4.7 티켓 수령 공연 예약 정보"

inputs = tokenizer(
    text,
    return_tensors="pt",
    truncation=True,
    max_length=256
)

with torch.no_grad():
    outputs = model(**inputs)

probs = torch.softmax(outputs.logits, dim=-1)

pred_id = probs.argmax(dim=-1).item()
confidence = probs[0][pred_id].item()

label = model.config.id2label[pred_id]

print({
    "label": label,
    "confidence": confidence
})

Example output format:

{
    "label": "schedule",
    "confidence": 0.XX
}

Getting All Class Probabilities

You can also inspect the confidence score for every category.

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

model_name = "hur03/capturemate-category-classifier-v1-5class"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name)

text = "성수동 카페 서울숲 근처"

inputs = tokenizer(
    text,
    return_tensors="pt",
    truncation=True,
    max_length=256
)

with torch.no_grad():
    outputs = model(**inputs)

probs = torch.softmax(outputs.logits, dim=-1)[0]

results = []

for class_id, probability in enumerate(probs):
    results.append({
        "label": model.config.id2label[class_id],
        "score": probability.item()
    })

results = sorted(
    results,
    key=lambda x: x["score"],
    reverse=True
)

for result in results:
    print(result)

This can be useful when integrating the model with downstream fallback logic or another classifier.

Training Details

Training Data

The model was trained on OCR text extracted from screenshot images collected for the CaptureMate project.

Dataset split:

Split	Samples
Train	415
Validation	89
Test	90

Label Distribution

Label	Total	Train	Validation	Test
schedule	65	46	10	9
shopping	172	120	26	26
place	115	80	17	18
memo	135	94	20	21
unknown	107	75	16	16

Training Procedure

Preprocessing

• OCR text was extracted from screenshots.
• Empty text samples were removed.
• Labels were mapped to five target categories.
• Text was tokenized with the KLUE-RoBERTa tokenizer.
• Maximum sequence length: 256 tokens.

Training Hyperparameters

Hyperparameter	Value
Learning rate	1e-5
Epochs	8
Train batch size	16
Eval batch size	16
Warmup ratio	0.1
Weight decay	0.01
Best model metric	macro F1
Seed	42

Evaluation

Testing Data, Factors & Metrics

Testing Data

The test set contains 90 OCR text samples across five categories.

Metrics

The model was evaluated using:

• Accuracy
• Macro F1
• Macro precision
• Macro recall

Macro metrics are important because the dataset is not perfectly balanced across categories.

Results

Metric	Validation	Test
Accuracy	87.64%	88.89%
Macro F1	84.83%	87.44%
Macro Precision	88.45%	87.57%
Macro Recall	86.75%	87.90%

Test Set Per-Class Results

Class	Precision	Recall	F1-score	Support
schedule	80.00%	88.89%	84.21%	9
shopping	89.29%	96.15%	92.59%	26
place	90.00%	100.00%	94.74%	18
memo	100.00%	85.71%	92.31%	21
unknown	78.57%	68.75%	73.33%	16

Confusion Matrix

Rows are true labels, columns are predicted labels.

True \ Pred	schedule	shopping	place	memo	unknown
schedule	8	0	0	0	1
shopping	0	25	0	0	1
place	0	0	18	0	0
memo	1	0	1	18	1
unknown	1	3	1	0	11

Known Test Errors

The test set contains 90 samples. The model misclassified 10 samples.

Main error patterns:

• unknown -> shopping: 3 samples
• unknown -> schedule: 1 sample
• unknown -> place: 1 sample
• shopping -> unknown: 1 sample
• memo -> unknown: 1 sample
• schedule -> unknown: 1 sample
• memo -> schedule: 1 sample
• memo -> place: 1 sample

The largest source of error is the unknown category. This is expected because unknown represents a broad range of screenshots that do not clearly belong to one of the four action-oriented categories.

Technical Specifications

Model Architecture and Objective

• Architecture: KLUE-RoBERTa sequence classification
• Objective: Single-label multi-class classification
• Input: OCR-extracted screenshot text
• Output: One of five CaptureMate categories
• Number of classes: 5
• Maximum sequence length: 256 tokens

Label Mapping

0 -> schedule
1 -> shopping
2 -> place
3 -> memo
4 -> unknown

The model configuration should contain the corresponding id2label and label2id mappings so that Hugging Face Transformers returns human-readable category names.

Software

The model was developed using:

• transformers
• datasets
• torch
• scikit-learn

Intended CaptureMate Pipeline

Within CaptureMate, this model is intended to provide text-based category information from screenshot OCR results.

A complete screenshot understanding system may combine multiple sources of information:

                    Screenshot
                        |
             +----------+----------+
             |                     |
             v                     v
        OCR Extraction       Image Analysis
             |                     |
             v                     v
      Text Classifier       Image Classifier
             |                     |
             +----------+----------+
                        |
                        v
                Category Decision
                        |
                        v
             Action Recommendation

This repository contains the text classification component of that pipeline.

The model does not itself perform OCR, image classification, or action recommendation.

Model Card Authors

CaptureMate

Model Card Contact

CaptureMate project maintainer