Blame for README.md

---
license: cc-by-4.0
language:
- de
metrics:
- accuracy
- f1
base_model:
- stefan-it/zeitungs-lm-v1
pipeline_tag: text-classification
library_name: transformers
tags:
- electra
- text-classification
- historical-newspapers
- job-advertisements
- digital-humanities
---

# JobAds Classification Model

The JobAds Classification Model assigns a job advertisement from a historical German-language newspaper (1850-1950) to one of five categories: heading, job offer, job search, service offer or mediation. It is an ELECTRA-based model ([zeitungs-lm-v1](https://huggingface.co/stefan-it/zeitungs-lm-v1), Schweter, 2024) fine-tuned by us for this task. On a class-balanced test set it reached an accuracy of **0.93** (macro F1 0.93).

This model is the second step of a two-step pipeline. The first step, the [JobAds Identification Model](https://huggingface.co/Var3n/jobads-identification), finds the text regions that are job advertisements. This model then sorts the identified advertisements into categories.

## Intended use

The model classifies the text of a single job advertisement into one of five categories. It is intended for the analysis of historical job advertisements, for example to study how many positions were offered and sought, or how services and intermediaries were advertised.

* **Language:** German
* **Period:** trained for newspapers from 1850 to 1950
* **Input:** the text of one job advertisement, with line breaks removed
* **Output:** one of five classes

|Label id|Label|Meaning|
|-|-|-|
|0|heading|Heading above or within the job advertisement sections, such as *Offene Stelle* ("Open positions")|
|1|job\_offer|An open position is being advertised|
|2|job\_search|Someone is looking for a position|
|3|service\_offer|A service is offered, e.g. language lessons, without creating a full-time work relationship|
|4|vermittlung|Mediation: positions are offered or sought through a mediator, e.g. an agency|

 

The label of the fifth class is the German word `vermittlung` (English: mediation). In the rest of this card, the class is referred to as "mediation" for readability.

**Important:** the model assumes that the input already is a job advertisement. It does not recognise other content. Use the [JobAds Identification Model](https://huggingface.co/Var3n/jobads-identification) first to filter job advertisements.

### Out-of-scope use

The model has not been validated for newspapers outside 1850-1950, for languages other than German, for born-digital text, or for raw OCR text (see Limitations).

## How to use

```py
from transformers import pipeline

clf = pipeline(
    "text-classification",
    model="Var3n/jobads-classification",
    tokenizer="stefan-it/zeitungs-lm-v1",
)

text = "Köchin für ein größeres Gasthaus wird gesucht."
text = text.replace("\\n", " ")  # the model was trained on texts without line breaks

print(clf(text, truncation=True, max\_length=512))
```

 

The output gives the predicted label and its probability. Pass `device=0` to `pipeline` to use a GPU, and a list of texts to classify several advertisements at once. Use `top\_k=None` to get the probabilities of all five classes.

## Training data

The training data is based on the manually annotated ground truth created in the JobAds project (FWF P35783) and published with the CHR 2024 paper *Who Advertises in Newspapers? Data Criticism in Mining Historical Job Ads* (Venglarova et al., 2024). The ground truth is available at [JobAds-FWFProject/Ground-Truth-CHR2024](https://github.com/JobAds-FWFProject/Ground-Truth-CHR2024). In the annotation process, each annotated job advertisement was also assigned to one of the five categories above.

The advertisements come from 29 newspaper titles of the ANNO corpus ([https://anno.onb.ac.at/](https://anno.onb.ac.at/)) of the Austrian National Library (1850-1950). They were OCRed with the frak2021 model (Mannheim University Library, 2021) and manually corrected in Transkribus (Venglarova et al., 2024). The model was trained on these **manually corrected texts**, with line breaks removed.

**Balanced sample.** A random sample with the same number of examples per class was drawn from the annotated data. The size of the sample was set by the least populated class, which has 159 examples. This gave 159 examples for each of the five classes, 795 in total.

**Split.** The sample was divided with a random, stratified split (80% training, 20% test, random seed 42):

|Subset|Examples|Per class|
|-|-|-|
|Training|636|127 or 128|
|Test|159|31 or 32|

 

Each advertisement is in exactly one of the two subsets, and the class distribution is almost identical in both. The split was not grouped by page or newspaper.

**Limitation of the balanced sample.** In the original data the classes are not equally frequent, and some categories occur considerably more often than others. Training and evaluating on a balanced sample shows how well the model separates the five classes when they are equally represented. It does not necessarily show how the model performs on data with the natural, imbalanced class distribution.

## Training procedure

* **Base model:** [zeitungs-lm-v1](https://huggingface.co/stefan-it/zeitungs-lm-v1) (Schweter, 2024), an ELECTRA model pre-trained on historical German newspapers
* **Task:** sequence classification with five classes (`ElectraForSequenceClassification`, `num\_labels=5`)
* **Loss:** standard cross-entropy, without class weights (the classes are balanced)
* **Optimizer:** AdamW (implementation of `transformers`), learning rate 1e-5, no weight decay, no learning rate schedule or warm-up
* **Epochs:** 15
* **Batch size:** 16
* **Tokenization:** `ElectraTokenizer` of the base model, with padding and truncation (maximum 512 tokens)
* **Seeds:** the sampling and the split use a fixed seed (42). The training itself was not seeded, so a repeated training run gives slightly different results.
* **Model used:** the best performing model
* **Hardware:** NVIDIA RTX A4000
* **Libraries:** transformers 4.36.2, torch 2.13.0

## Evaluation

All results are computed on the held-out **test set** of 159 advertisements, evaluated on the manually corrected text (predicted class = class with the highest probability). The accuracy is 0.93.

|Class|Precision|Recall|F1-score|Support|
|-|-|-|-|-|
|heading|1.00|1.00|1.00|32|
|job\_offer|0.91|0.97|0.94|31|
|job\_search|0.94|0.91|0.92|32|
|service\_offer|0.85|0.91|0.88|32|
|vermittlung (mediation)|0.97|0.88|0.92|32|
|Accuracy|||0.93|159|
|Macro avg|0.93|0.93|0.93|159|
|Weighted avg|0.93|0.93|0.93|159|

 

The test set is small (about 32 examples per class) and the results come from a single training run, so small differences between classes should not be over-interpreted. Because the test set is balanced, the scores do not show the performance under the natural class distribution.

Results broken down by newspaper or time period are not available.

### Error analysis

We manually analysed 100 randomly drawn misclassifications. The errors fall into three categories:

1. **Mediations classified as job searches or job offers.** The mediation class was assigned by annotators on the basis of their interpretation and of patterns in the advertisements. For example, a name that appears very often suggests a broker, and a phrase like *wird placiert* ("will be placed") suggests placement through an institution without saying so explicitly. It is often ambiguous whether a mediation is rather a job offer or a job search, even for a human annotator, so the confusion was to be expected.
2. **Confusion between service offers and other types of job ads.** Service offers are easily mistaken for other types. For example, *"Wäſcherin geht waſchen und reiben."* ("Laundress goes washing and rubbing") was annotated as a service offer but predicted as a job search. It has characteristics of both.
3. **Short ads classified as headings.** Short ads without typical job ad vocabulary such as *seeks* or *will be accepted*, e.g. *"Françaiſe Vinohrady Radhoštſká Par terre"* ("A French woman Vinohrady Radhoštſká Ground floor"), are classified as headings.

These errors point to limits of the ground truth rather than to weaknesses of the models. Ambiguous or imperfect labels lead to confused predictions, and clearer criteria for the categories would be needed in the annotation. Job advertisements are in their nature qualitative data, which can be turned into quantitative categories only to a limited extent.

## Limitations and bias

* The model was trained and evaluated on **manually corrected** text. Its performance on raw OCR text or automatically post-corrected text has not been evaluated and may be lower.
* The training set is small (636 examples) and was balanced by undersampling. The model does not know the natural frequencies of the classes, and its probabilities should not be read as real-world frequencies.
* The categories mediation and service offer, and very short advertisements, are the main source of errors. The criteria for the categories were interpretive and not strictly defined.
* The ground truth was annotated in one round without a check, so it contains some label noise.
* The test regions were drawn randomly from the same pages as the training data (no split by page or newspaper), and the test set is small. Performance on unseen newspapers, pages or periods can be lower than reported.

## Citation

**Ground-truth data**

```bibtex
@inproceedings{venglarova2024advertises,
  title     = {Who Advertises in Newspapers? Data Criticism in Mining Historical Job Ads},
  author    = {Venglarova, Klara and Adam, Raven and M{\\"o}lzer, Wiltrud and Balasubramanian, Saranya and Kleinert, J{\\"o}rn and F{\\"u}llsack, Manfred and Vogeler, Georg},
  booktitle = {Proceedings of the Computational Humanities Research Conference (CHR 2024)},
  series    = {CEUR Workshop Proceedings},
  volume    = {3834},
  year      = {2024},
  address   = {Aarhus, Denmark},
  url       = {https://ceur-ws.org/Vol-3834/paper12.pdf}
}
```

 

There is no citation for this model yet. If you use it, please cite the ground-truth paper above and link to this repository.

**Base model**

```bibtex
@misc{schweter2024zeitungslm,
  author = {Schweter, Stefan},
  title  = {zeitungs-lm-v1},
  year   = {2024},
  url    = {https://huggingface.co/stefan-it/zeitungs-lm-v1}
}
```

 

**OCR model:** Mannheim University Library. (2021). Frak2021. [https://ub-backup.bib.uni-mannheim.de/\~stweil/tesstrain/frak2021/tessdata\_best/frak2021-0.905.traineddata](https://ub-backup.bib.uni-mannheim.de/~stweil/tesstrain/frak2021/tessdata_best/frak2021-0.905.traineddata) (accessed 20 November 2023).

**Annotation tool:** Nakayama, H. et al. (2018). Doccano: Text Annotation Tool for Human.[https://github.com/doccano/doccano](https://github.com/doccano/doccano).

## Acknowledgements

The ground truth was created in the project JobAds (FWF P35783), which was supported by the Austrian Science Fund (FWF). We thank the Austrian National Library for providing the data from the ANNO corpus. We thank our project colleague Wiltrud Mölzer and the project PI Jörn Kleinert for their support with ground-truth data annotation.

## License

This model is released under the [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) license.

## Authors and contact

University of Graz, Austria (JobAds project)

* Klara Venglarova ([klara.venglarova@uni-graz.at](mailto:klara.venglarova@uni-graz.at)) (model training)
* Raven Adam ([raven.adam@uni-graz.at](mailto:raven.adam@uni-graz.at)) (integrating model into a processing pipeline)
* Georg Vogeler ([georg.vogeler@uni-graz.at](mailto:georg.vogeler@uni-graz.at)) (supervision)