PhiloLM / README.md
gustavecortal's picture
Model card: remove the training-mix figure
8a92cc7 verified
|
Raw History Blame Contribute Delete
4.71 kB
---
license: apache-2.0
language:
- en
library_name: transformers
pipeline_tag: text-generation
datasets:
- gustavecortal/PhiloText
tags:
- philosophy
- base-model
- trained-from-scratch
- small-language-model
- n-gram-embedding
- muon
- webassembly
---
# PhiloLM
PhiloLM is a language model with 112M parameters, trained from scratch on open philosophy data ([PhiloText](https://huggingface.co/datasets/gustavecortal/PhiloText)). A [web demo](https://gustavecortal.com/philosophy-lm/) runs it in your browser.
## Architecture
![PhiloLM architecture in two panels. Model: token IDs feed a token embedding and a bigram embedding, then four groups of four blocks, each with three local attention blocks (256-token window) and one global attention block (9 query heads, 3 key-value heads), a final RMSNorm and an LM head tied to the token embedding. Decoder block: the bigram vector, times a learned scale, is added to the first 128 of 576 dimensions of the residual stream, then RMSNorm and causal attention (QK norm, partial RoPE), then RMSNorm and a SwiGLU feed-forward network, each with a residual connection](architecture.png)
## Training
The model processed **1.45B tokens** across four source-weighted passes over PhiloText's 450.8M training tokens. Training used document-masked attention, with windows growing from 1,024 to 2,048 tokens, Muon for hidden weight matrices, and AdamW for the remaining parameters. The final pass emphasizes reference sources and selected primary texts. The release blends final weights with a late-training exponential moving average. Training took **15.2 hours on one Intel Arc A770 (16 GB)**.
## Examples
Continuations sampled as on the [web page](https://gustavecortal.com/philosophy-lm/), cut after the first paragraph. The prompt is in bold.
> **A good life** is a state of well-being that is characterized by the absence of suffering and the fulfillment of human potential. It may also be described as a positive experience, a state of happiness, or a state of being in a good mood.
> **Justice** is a concept that has been used to describe the relationship between individuals, groups and institutions. It can be used to refer to what society considers fair or just, but also to apply it in more general ways such as when people are treated equally.
> **According to Kant, the moral law** is the categorical imperative, or a law of nature. It is an idea that can be deduced from reason and not merely derived from experience. In this way it provides a basis for morality, which Kant calls the "Categorical Imperative".
> **Hume argued that causation** is a matter of constant conjunction, where the term "cause" (or "substance") refers to all or most of the occurrences that we have observed. The notion of causation in Hume's philosophy has been interpreted as being about regularities and causal relations between events. In this view, there are no necessary connections among things. Instead, all our experience can be explained by reference to these patterns of observation.
> **Husserl's phenomenology begins with** the question of what is common to all human experience. It asks: "What is it for a given thing to be present in the world?" (p. 1). In other words, how do we get from there to the objects that we encounter and their properties? What is it that makes an object appear as such?
## Usage
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("gustavecortal/PhiloLM")
model = AutoModelForCausalLM.from_pretrained("gustavecortal/PhiloLM", trust_remote_code=True)
inputs = tokenizer("The problem of personal identity", return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=64)
print(tokenizer.decode(output[0], skip_special_tokens=True))
```
For faster CPU generation, `generate_fast.py` runs the [web page](https://gustavecortal.com/philosophy-lm/)'s 8-bit WebAssembly engine, about three times faster than PyTorch:
```bash
pip install wasmtime numpy tokenizers
hf download gustavecortal/PhiloLM core.wasm core.bin tokenizer.json generate_fast.py --local-dir PhiloLM
cd PhiloLM && python generate_fast.py "The problem of personal identity"
```
## Limitations
PhiloLM is a base checkpoint without instruction tuning. It can repeat itself, lose argumentative coherence, and invent citations or attributions. For factual use, verify claims against the original sources.
Weights and inference code are released under Apache-2.0.
## Citation
```bibtex
@misc{cortal2026philolm,
author = {Cortal, Gustave},
title = {PhiloLM: a small language model trained on philosophy},
year = {2026},
url = {https://huggingface.co/gustavecortal/PhiloLM}
}
```