Sentence Similarity
sentence-transformers
Safetensors
English
bert
feature-extraction
historical-text
early-modern-english
economic-history
history-of-science
text-embeddings-inference
Instructions to use juliuskoschnick/SteamBERTh with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- sentence-transformers
How to use juliuskoschnick/SteamBERTh with sentence-transformers:
from sentence_transformers import SentenceTransformer model = SentenceTransformer("juliuskoschnick/SteamBERTh") sentences = [ "That is a happy person", "That is a happy dog", "That is a very happy person", "Today is a sunny day" ] embeddings = model.encode(sentences) similarities = model.similarity(embeddings, embeddings) print(similarities.shape) # [4, 4] - Notebooks
- Google Colab
- Kaggle
Usage example: name SteamBERTh explicitly (innovfromtext 0.2.0 defaults to all-MiniLM-L6-v2)
6ae4141 verified |
Download README.md from juliuskoschnick/SteamBERTh: direct link, hf CLI and curl.
- Browser
- Download file 6.18 kB
-
https://huggingface.co/juliuskoschnick/SteamBERTh/resolve/main/README.md
- Command line
-
hf download hf://juliuskoschnick/SteamBERTh/README.md
-
curl -L -o README.md https://huggingface.co/juliuskoschnick/SteamBERTh/resolve/main/README.md
6.18 kB
| license: mit | |
| language: | |
| - en | |
| base_model: emanjavacas/MacBERTh | |
| library_name: sentence-transformers | |
| pipeline_tag: sentence-similarity | |
| tags: | |
| - sentence-transformers | |
| - sentence-similarity | |
| - feature-extraction | |
| - historical-text | |
| - early-modern-english | |
| - economic-history | |
| - history-of-science | |
| # SteamBERTh | |
| SteamBERTh is a sentence-embedding model for English scientific and technical texts of the seventeenth and eighteenth | |
| centuries. It is [MacBERTh](https://huggingface.co/emanjavacas/MacBERTh), a BERT model pre-trained on historical | |
| English (1450–1950), fine-tuned on the titles of scientific and technical books printed in Britain, 1600–1800, and on | |
| the short descriptions of English patents. | |
| The purpose of the model is to serve for text-comparisons for scientific and technical text between 1600 and 1800: An | |
| embedding model for the age of steam. | |
| The model was developed in [Koschnick (2025)](https://arxiv.org/abs/2512.16587), where it underlies text-based | |
| measures of innovation and of knowledge spillovers between fields. These measures are available in the Python package | |
| [innovfromtext](https://github.com/juliuskoschnick/innovfromtext). | |
| ## Usage | |
| With [sentence-transformers](https://www.sbert.net): | |
| ```python | |
| from sentence_transformers import SentenceTransformer | |
| model = SentenceTransformer("juliuskoschnick/SteamBERTh") | |
| titles = [ | |
| "A new and easy method of finding the longitude at sea", | |
| "The art of navigation improved, with tables of the sun's declination", | |
| "The compleat husbandman: or, a treatise of the improvement of land", | |
| ] | |
| emb = model.encode(titles, normalize_embeddings=True) | |
| print(emb @ emb.T) # cosine similarities | |
| ``` | |
| Innovation and spillover indices with [innovfromtext](https://github.com/juliuskoschnick/innovfromtext): | |
| ```python | |
| import innovfromtext as ift | |
| df = ift.load_example() # Philosophical Transactions titles 1665-1800, English patents 1700-1800 | |
| out = ift.compute(df, "innovation", model="juliuskoschnick/SteamBERTh") | |
| ``` | |
| ## Why use a historical sentence-transformer model? | |
| Models trained on modern text carry anachronistic associations: they miss that a "fire engine" was a steam engine or | |
| that "philosophical instruments" were scientific instruments, or they place concepts such as "phlogiston" far from | |
| the science of the time. MacBERTh avoids this through its historical pre-training corpus, about 70 per cent of which | |
| dates from before 1800. Base BERT models are, however, not trained for sentence similarity, and their embedding spaces | |
| are strongly anisotropic. Contrastive fine-tuning on the texts of interest addresses both problems. | |
| ## Training | |
| - **Base model:** [emanjavacas/MacBERTh](https://huggingface.co/emanjavacas/MacBERTh), mean pooling over tokens. | |
| - **Data:** 21,760 texts: all titles of the English Short Title Catalogue (ESTC) in *applied physics, astronomy, | |
| chemistry, mathematics, encyclopedias and dictionaries, technical instructions in trades and agriculture, | |
| navigation,* and *scientific instruments*, and the short descriptions of English patents from Woodcroft (1854); patent descriptions with fewer than three | |
| words are dropped. | |
| - **Objective:** unsupervised SimCSE (Gao, Yao and Chen 2021): each text is paired with itself, dropout produces two | |
| different views, and the other texts of the batch serve as negatives (multiple negatives ranking loss, scale 15). | |
| - **Hyperparameters:** 2 epochs, batch size 32, learning rate 2e-5, weight decay 0.01, linear warm-up over the first | |
| 10 per cent of steps, maximum sequence length 512. | |
| - **Reproducibility:** fixed seeds (42), deterministic CUDA kernels, logged software versions (sentence-transformers | |
| 3.2.1, transformers 4.44.2, torch 2.2.1). | |
| ## Evaluation | |
| Koschnick (2025, appendix) documents the evaluation; in brief: | |
| - The embedding space of SteamBERTh is much more isotropic than that of the base model, and fields and industries | |
| form more coherent clusters. | |
| - An innovation index built on SteamBERTh embeddings predicts historical patent citations (within-year elasticity | |
| 0.18 for 1720–1799) and the number of editions of books (0.17). In a horse race for 1720–1800 it outperforms indices | |
| built on modern BERT models. | |
| ## Limitations | |
| The model is tuned for English scientific and technical texts of roughly 1600–1800, mostly book titles and patent | |
| descriptions. Koschnick (2025) finds that modern models perform slightly better on nineteenth-century patents. The | |
| model has not been evaluated on other genres or periods. | |
| ## Citation | |
| If you use SteamBERTh, please cite Koschnick (2025) that introduces it and the MacBERTh papers (the | |
| [MacBERTh project](https://macberth.netlify.app/) asks users of MacBERTh to cite Manjavacas and Fonteyn 2022): | |
| ```bibtex | |
| @misc{koschnick2025feedback, | |
| title = {Did a feedback mechanism between propositional and prescriptive knowledge create modern growth?}, | |
| author = {Koschnick, Julius}, | |
| year = {2025}, | |
| eprint = {2512.16587}, | |
| archivePrefix = {arXiv}, | |
| primaryClass = {econ.GN}, | |
| url = {https://arxiv.org/abs/2512.16587} | |
| } | |
| @article{manjavacas2022adapting, | |
| title = {Adapting vs. Pre-training Language Models for Historical Languages}, | |
| author = {Manjavacas, Enrique and Fonteyn, Lauren}, | |
| journal = {Journal of Data Mining \& Digital Humanities}, | |
| volume = {NLP4DH}, | |
| year = {2022}, | |
| doi = {10.46298/jdmdh.9152}, | |
| url = {https://doi.org/10.46298/jdmdh.9152} | |
| } | |
| @inproceedings{manjavacas-arevalo-fonteyn-2021-macberth, | |
| title = {{M}ac{BERT}h: Development and Evaluation of a Historically Pre-trained Language Model for {E}nglish (1450-1950)}, | |
| author = {Manjavacas Arevalo, Enrique and Fonteyn, Lauren}, | |
| booktitle = {Proceedings of the Workshop on Natural Language Processing for Digital Humanities}, | |
| year = {2021}, | |
| address = {NIT Silchar, India}, | |
| publisher = {NLP Association of India (NLPAI)}, | |
| pages = {23--36}, | |
| url = {https://aclanthology.org/2021.nlp4dh-1.4/} | |
| } | |
| ``` | |
| Base model: [emanjavacas/MacBERTh](https://huggingface.co/emanjavacas/MacBERTh). | |
| ## License | |
| MIT, as the base model [MacBERTh](https://huggingface.co/emanjavacas/MacBERTh). | |