Overview of bert-japanese-12M

• The bert-japanese-12M model is a transformer-based model with BERT architecture, which is designed to be used on Japanese text. In addition, it is small enough to be run on consumer low-end GPUs, or even CPUs with acceptable latency.
• The model was trained from scratch on an cc100-ja and Japanese Wikipedia Cirrussearch 20240701 dump file.
• The pre-training process of this model heavily follows the practices of bert-japanese but it was performed from scratch with the training data was randomed differently, and tokenizer's vocab is also different.
• This model is designed based on the tradeoff between accuracy and performance.

Model description

• The model has BERT architecture with the following details:
  • Model size: 12,963,584 parameters
  • Output vector size (same as hidden size): 256
  • Number of layers: 6
  • Max position embeddings: 2048 (it means the model can consume inputs with length up to 2048 tokens)
• The model employs BertJapaneseTokenizer as its tokenizer
  • Word-level tokenization with MeCab wrapped by fugashi.
  • WordPiece is used for subword tokenization.
  • Tokenizer's vocabulary has 32,768 tokens.

Intended uses & limitations

• Example code:

from transformers import AutoTokenizer, AutoModelForMaskedLM, pipeline
MODEL_PATH = "nptdat/bert-japanese-12M"
model = AutoModelForMaskedLM.from_pretrained(MODEL_PATH, trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained(MODEL_PATH)

pipe = pipeline("fill-mask", model=model, tokenizer=tokenizer)
text = "東京は[MASK]の首都です。"
pipe(text)

• Because this model is very small, its accuracy may be low compared with other bigger models. However, it is light and can be run on low-end GPUs, or even CPUs to cut inference cost.
• We recommend the following practices when using this model
  • Use the model on specific domain, where small models can work well
  • Use the model on use-cases where low inference cost and low latency are necessary
  • Should fine-tune the model on downstream task before using

Training procedure

• Tokenizer training
  • The tokenizer is trained from 10M Japanese Wikipedia sentences.
• Model training are performed 2 phases on different datasets
  • First phase: pre-training on cc100-ja dataset
    • We pre-trained a model from scratch with 122M sentences from cc100-ja (67GB of text), validated on 1M sentences.
  • Second phase: pre-training on Japanese Wikipedia dataset
    • The model was then pre-trained more on 38M Wikipedia sentences (5.6GB of text), validated on 1.2M sentences.
• Both phrases of training are performed on a single consumer Geforce RTX 3090 GPU. It took about 34 hours and 11 hours respectively.

Training hyperparameters

The following hyperparameters were used during training on both datasets

• learning_rate: 5e-05
• train_batch_size: 32
• eval_batch_size: 32
• seed: 42
• distributed_type: multi-GPU
• gradient_accumulation_steps: 8
• total_train_batch_size: 256
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_ratio: 0.08
• num_epochs: 1
• mixed_precision_training: Native AMP

Training results

This is the results of the 2nd phase, when the model was trained and validated on Japanese Wikipedia text.

Training Loss	Epoch	Step	Validation Loss	Accuracy
2.8074	0.1666	25000	2.7266	0.5189
2.6942	0.3333	50000	2.5879	0.5367
2.6153	0.4999	75000	2.5156	0.5459
2.5805	0.6666	100000	2.4707	0.5522
2.5546	0.8332	125000	2.4414	0.5560
2.5435	0.9999	150000	2.4297	0.5574
2.6841	1.0000	150019	2.4316	0.5575

Framework versions

• Transformers 4.42.0
• Pytorch 2.3.1+cu121
• Datasets 2.18.0
• Tokenizers 0.19.1