WaterBERT

WaterBERT is a BERT-base encoder for wastewater- and water-treatment literature. It was obtained by continuing masked-language-model (MLM) pretraining of SciBERT (scivocab, uncased) on a domain corpus of water-treatment research text. The vocabulary is SciBERT's (31,090 tokens); only the weights were adapted.

It is the shared backbone of WaterBERT-NER and WaterBERT-RE, and is intended as a starting point for fine-tuning on water-treatment NLP tasks (entity recognition, relation extraction, classification, retrieval).

Usage

from transformers import pipeline

fill = pipeline("fill-mask", model="Mudi12137/WaterBERT")
fill("The Fenton process generates hydroxyl radicals from hydrogen peroxide and [MASK] ions.")
# top prediction: "iron"

For fine-tuning, load it like any BERT checkpoint:

from transformers import AutoModelForTokenClassification, AutoTokenizer

tok = AutoTokenizer.from_pretrained("Mudi12137/WaterBERT")
model = AutoModelForTokenClassification.from_pretrained("Mudi12137/WaterBERT", num_labels=13)

Training

Setting	Value
Initialisation	allenai/scibert_scivocab_uncased
Objective	masked language modelling
Epochs / steps	30 / 940,440 (this checkpoint is the final step)
Effective batch size	160 (80 per device x 2 gradient-accumulation steps, 1 GPU)
Optimiser	AdamW, learning rate 1e-4, linear decay, warmup ratio 0.048, weight decay 0.01
Precision	bf16
Architecture	BERT-base: 12 layers, hidden size 768, 110M parameters, max length 512

The pretraining corpus and its construction are described in the accompanying paper.

Limitations

• English only, lower-cased input (the tokenizer lower-cases automatically).
• The corpus is research literature (mostly abstracts); performance on other genres such as plant operation logs or regulations has not been evaluated.

License

Apache-2.0, the same as the SciBERT base model.

Citation

See CITATION.cff in the project repository.