Create README.md
Browse files
README.md
ADDED
|
@@ -0,0 +1,249 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: apache-2.0
|
| 3 |
+
library_name: numpy
|
| 4 |
+
tags:
|
| 5 |
+
- reading
|
| 6 |
+
- reading-time
|
| 7 |
+
- comprehension
|
| 8 |
+
- memory
|
| 9 |
+
- attention
|
| 10 |
+
- discourse
|
| 11 |
+
- heuristic
|
| 12 |
+
- cpu
|
| 13 |
+
pipeline_tag: text-classification
|
| 14 |
+
language:
|
| 15 |
+
- en
|
| 16 |
+
---
|
| 17 |
+
|
| 18 |
+
# hv-reader
|
| 19 |
+
|
| 20 |
+
**The reading experience, in one call.**
|
| 21 |
+
|
| 22 |
+
Given a text, produce a `ReadingProfile`: pace, memory, passes, slip, and
|
| 23 |
+
wall β the five axes that describe what it is like to read a text.
|
| 24 |
+
|
| 25 |
+
## The claim in one sentence
|
| 26 |
+
|
| 27 |
+
The seven `hv-` models each describe one aspect of reading. `hv-reader`
|
| 28 |
+
describes the *experience*: a five-axis signature that no single axis can
|
| 29 |
+
produce.
|
| 30 |
+
|
| 31 |
+
## What it produces
|
| 32 |
+
|
| 33 |
+
A `ReadingProfile` containing:
|
| 34 |
+
|
| 35 |
+
- **pace** β mean reading speed, normalized to `[0, 1]`
|
| 36 |
+
- **memory** β fraction of the text surviving a week
|
| 37 |
+
- **passes** β how many reads are required to resolve the text
|
| 38 |
+
- **slip** β probability of an attention lapse mid-read
|
| 39 |
+
- **wall** β probability of reader refusal
|
| 40 |
+
|
| 41 |
+
Plus two derived scalars:
|
| 42 |
+
|
| 43 |
+
- **ttu_s** β total reading time in seconds
|
| 44 |
+
- **hunger_final** β unresolved curiosity at the end
|
| 45 |
+
|
| 46 |
+
And per-sentence detail: WPM, density, retention, fold load, slip, wall,
|
| 47 |
+
wall type, reasons.
|
| 48 |
+
|
| 49 |
+
## Install
|
| 50 |
+
|
| 51 |
+
```bash
|
| 52 |
+
pip install numpy
|
| 53 |
+
|
| 54 |
+
Actually β no dependencies. Pure stdlib.
|
| 55 |
+
|
| 56 |
+
## Usage
|
| 57 |
+
|
| 58 |
+
### Demo
|
| 59 |
+
|
| 60 |
+
```bash
|
| 61 |
+
python hv_reader.py
|
| 62 |
+
|
| 63 |
+
Runs four synthetic samples β fiction, academic, overclaim, monotone β
|
| 64 |
+
and prints the full profile for each.
|
| 65 |
+
|
| 66 |
+
### Analyze a text
|
| 67 |
+
|
| 68 |
+
```bash
|
| 69 |
+
python hv_reader.py --text "The old man walked slowly to the boat."
|
| 70 |
+
python hv_reader.py --text - < essay.txt
|
| 71 |
+
python hv_reader.py --text "..." --profile # five-axis summary only
|
| 72 |
+
python hv_reader.py --text "..." --json # full JSON
|
| 73 |
+
```
|
| 74 |
+
|
| 75 |
+
### Python
|
| 76 |
+
|
| 77 |
+
```python
|
| 78 |
+
from hv_reader import HVReader
|
| 79 |
+
|
| 80 |
+
m = HVReader()
|
| 81 |
+
profile = m.analyze(text)
|
| 82 |
+
|
| 83 |
+
print(profile.pace) # 0.672
|
| 84 |
+
print(profile.memory) # 0.269
|
| 85 |
+
print(profile.passes) # 1.167
|
| 86 |
+
print(profile.slip) # 0.615
|
| 87 |
+
print(profile.wall) # 0.000
|
| 88 |
+
print(profile.summary)
|
| 89 |
+
```
|
| 90 |
+
|
| 91 |
+
## The five axes
|
| 92 |
+
|
| 93 |
+
### pace β from `hv-tempo`
|
| 94 |
+
|
| 95 |
+
Weighted sum of surface features (sentence length, clause density, rare
|
| 96 |
+
vocabulary, numerals, negation, hedges, conditionals, passive voice).
|
| 97 |
+
Normalized by `baseline_wpm = 220`.
|
| 98 |
+
|
| 99 |
+
```
|
| 100 |
+
pace = mean_wpm / 400
|
| 101 |
+
```
|
| 102 |
+
|
| 103 |
+
A text at 400 WPM or faster has `pace = 1.0`. Fiction runs at ~0.67.
|
| 104 |
+
Academic prose at ~0.30.
|
| 105 |
+
|
| 106 |
+
### memory β from `hv-forget`
|
| 107 |
+
|
| 108 |
+
Per-sentence stability grows with density, rare-word ratio, and salience
|
| 109 |
+
(numbers, proper nouns). Retention after 7 days follows Ebbinghaus decay:
|
| 110 |
+
|
| 111 |
+
```
|
| 112 |
+
stability = 1.0 + 5.0 Γ density + 3.0 Γ rare_ratio + 2.0 Γ salience
|
| 113 |
+
retention = exp(β7 / stability)
|
| 114 |
+
```
|
| 115 |
+
|
| 116 |
+
Memory is the mean retention over all sentences. Dense, specific sentences
|
| 117 |
+
last longer. Filler vanishes.
|
| 118 |
+
|
| 119 |
+
### passes β from `hv-fold`
|
| 120 |
+
|
| 121 |
+
Novelty-based load: orphan definites, forward references, reentry markers.
|
| 122 |
+
Fold grows with load:
|
| 123 |
+
|
| 124 |
+
```
|
| 125 |
+
passes = 1 + total_load / n_sentences
|
| 126 |
+
```
|
| 127 |
+
|
| 128 |
+
Fold of 1.0 = perfectly linear. Fold of 2.0+ = the text must be read twice
|
| 129 |
+
to be understood.
|
| 130 |
+
|
| 131 |
+
### slip β from `hv-slip`
|
| 132 |
+
|
| 133 |
+
Attention-lapse probability. Three signals:
|
| 134 |
+
|
| 135 |
+
1. **monotone** β sentences of similar length create drowsiness
|
| 136 |
+
2. **repetition** β content-word overlap with the previous sentence
|
| 137 |
+
3. **absence** β no numbers or proper nouns to anchor attention
|
| 138 |
+
|
| 139 |
+
Slip is per-sentence, then averaged. A text of short similar sentences
|
| 140 |
+
scores high. A text with varied rhythm and specific content scores low.
|
| 141 |
+
|
| 142 |
+
### wall β from `hv-wall`
|
| 143 |
+
|
| 144 |
+
Highest per-sentence wall score. Three wall types, in priority order:
|
| 145 |
+
|
| 146 |
+
1. **contradiction** β the sentence contradicts an earlier claim
|
| 147 |
+
2. **overclaim** β certainty exceeds evidence
|
| 148 |
+
3. **gap** β a conclusion is drawn without prior support
|
| 149 |
+
|
| 150 |
+
Wall is the *maximum*, not the mean. A single wall is a single wall.
|
| 151 |
+
|
| 152 |
+
## Benchmarks
|
| 153 |
+
|
| 154 |
+
### Four samples
|
| 155 |
+
|
| 156 |
+
| sample | words | pace | memory | passes | slip | wall | TTU |
|
| 157 |
+
|---|---:|---:|---:|---:|---:|---:|---:|
|
| 158 |
+
| Fiction | 51 | 0.67 | 0.27 | 1.17 | 0.62 | 0.00 | 11.4 s |
|
| 159 |
+
| Academic | 100 | 0.30 | 0.33 | 1.62 | 0.46 | 0.00 | 50.8 s |
|
| 160 |
+
| Overclaim | 51 | 0.40 | 0.40 | 2.40 | 0.54 | 0.77 | 19.2 s |
|
| 161 |
+
| Monotone | 30 | 0.71 | 0.19 | 1.67 | 0.87 | 0.00 | 6.3 s |
|
| 162 |
+
|
| 163 |
+
**Reading the table:**
|
| 164 |
+
|
| 165 |
+
- **Fiction** β fast pace, low memory, moderate slip. The reader finishes.
|
| 166 |
+
- **Academic** β slow pace, moderate memory, moderate slip. The reader
|
| 167 |
+
works but stays engaged.
|
| 168 |
+
- **Overclaim** β mid pace, high memory, *high wall*. The reader stops.
|
| 169 |
+
- **Monotone** β fast pace, low memory, *highest slip*. The reader's eye
|
| 170 |
+
moves but the mind drifts.
|
| 171 |
+
|
| 172 |
+
The five-axis profile separates all four, even where individual axes
|
| 173 |
+
overlap. Fiction and monotone have similar pace. Academic and overclaim
|
| 174 |
+
have similar memory. The *shape* of the profile differs.
|
| 175 |
+
|
| 176 |
+
### Why the profile is not the sum of its parts
|
| 177 |
+
|
| 178 |
+
Two texts with identical pace can have different memory, slip, or wall.
|
| 179 |
+
Two texts with identical wall can have very different slip. The profile is
|
| 180 |
+
the *joint* description. Every single-axis model in the series collapses
|
| 181 |
+
to one of the five. The profile is the artifact.
|
| 182 |
+
|
| 183 |
+
## When to use it
|
| 184 |
+
|
| 185 |
+
- **Editing.** Find where the text breaks the reading experience.
|
| 186 |
+
- **Prescription.** "Your essay has pace 0.72 and slip 0.81 β it reads
|
| 187 |
+
fast but won't stick."
|
| 188 |
+
- **Comparative analysis.** Rank texts by *reading difficulty shape*, not
|
| 189 |
+
by a single readability score.
|
| 190 |
+
- **Reader simulation.** Feed the profile into a decision model that
|
| 191 |
+
predicts whether a reader finishes.
|
| 192 |
+
- **Learning design.** Predict where students will zone out.
|
| 193 |
+
- **Marketing copy.** Predict where attention drops before conversion.
|
| 194 |
+
|
| 195 |
+
## When *not* to use it
|
| 196 |
+
|
| 197 |
+
- **As a reader study.** The axes are heuristic. The profile ranks; it
|
| 198 |
+
does not measure.
|
| 199 |
+
- **For non-English text.** All lexicons are English.
|
| 200 |
+
- **For very short text.** Fewer than three sentences gives unstable
|
| 201 |
+
per-axis values.
|
| 202 |
+
- **As ground truth on comprehension.** `memory` predicts *what survives*,
|
| 203 |
+
not *what is understood*.
|
| 204 |
+
- **For poetry or experimental prose.** The models assume propositional
|
| 205 |
+
content.
|
| 206 |
+
|
| 207 |
+
## Honest limitations
|
| 208 |
+
|
| 209 |
+
- **The five axes are independent by construction.** A text can be slow
|
| 210 |
+
and forgettable. A text can be fast and memorable. A text can be slow
|
| 211 |
+
and unforgettable. The profile reports the joint shape, but the axes
|
| 212 |
+
do not automatically produce a single scalar. Combining them into a
|
| 213 |
+
"reader burden" or "expected completion" score is left to the caller.
|
| 214 |
+
- **The pace model uses hand-tuned weights.** Not calibrated against
|
| 215 |
+
eye-tracking or reading-time studies.
|
| 216 |
+
- **The memory model uses Ebbinghaus decay with a specific half-life
|
| 217 |
+
formula.** Real memory is content-dependent and reader-dependent.
|
| 218 |
+
- **The slip model conflates monotony with attention lapse.** A reader
|
| 219 |
+
can be lulled by a beautiful monotonous passage (Whitman) as easily as
|
| 220 |
+
bored by filler. The model does not distinguish.
|
| 221 |
+
- **The wall model's antonym table is hand-curated.** Contradictions that
|
| 222 |
+
rely on entailment or world knowledge will not be detected.
|
| 223 |
+
- **No calibration against real readers.** The demo samples are
|
| 224 |
+
synthetic. The profile has not been validated against reader
|
| 225 |
+
self-reports, completion rates, or comprehension tests.
|
| 226 |
+
- **Fiction shows a slip score higher than intuition suggests.** Short
|
| 227 |
+
sentences with similar lengths trigger the monotone detector even when
|
| 228 |
+
the passage is engaging. The reason is the same as above: monotony and
|
| 229 |
+
attention lapse are correlated but not identical.
|
| 230 |
+
|
| 231 |
+
## Reference
|
| 232 |
+
|
| 233 |
+
Part of the reader-model series. Unifies seven component models into a
|
| 234 |
+
single profile:
|
| 235 |
+
|
| 236 |
+
| axis | component model |
|
| 237 |
+
|---|---|
|
| 238 |
+
| pace | `hv-tempo` |
|
| 239 |
+
| memory | `hv-forget` |
|
| 240 |
+
| passes | `hv-fold` |
|
| 241 |
+
| slip | `hv-slip` |
|
| 242 |
+
| wall | `hv-wall` |
|
| 243 |
+
|
| 244 |
+
The component models remain independently useful. `hv-reader` is the
|
| 245 |
+
joint signature.
|
| 246 |
+
|
| 247 |
+
## License
|
| 248 |
+
|
| 249 |
+
Apache-2.0
|