| # Contributing to frontier-check |
|
|
| ## What this project is |
|
|
| A small tool that routes claims by their training-data status. The |
| core contribution is the 2×2 matrix over (mechanism, checkable |
| sub-claim) that separates `FRONTIER` from `FABRICATED`. Everything |
| else is plumbing. |
|
|
| ## What this project is not |
|
|
| - Not a general-purpose fact-checker. |
| - Not a replacement for the LLM that produces the claim. |
| - Not a proof that a `FRONTIER` claim is correct. |
| - Not calibrated against real reader outcomes yet. |
|
|
| ## How to propose a change |
|
|
| 1. Open an issue describing the change and *why the current behavior |
| is wrong*. Include a specific claim that routes incorrectly and |
| the correct routing for it. |
| 2. Add the claim to `REGRESSION_CLAIMS` with the expected status |
| and verdict. |
| 3. Confirm the new claim fails on the current code (`pytest -q`). |
| 4. Fix the code. |
| 5. Confirm all regression claims pass, including the new one. |
| 6. Open the PR. |
|
|
| The regression suite is the spec. If your change is not covered by |
| a regression claim, add one before touching the code. |
|
|
| ## What "the current behavior is wrong" means |
|
|
| The classifier is heuristic. It will route some claims incorrectly. |
| The question is not "does it make mistakes" but "does it make *the |
| same* mistakes reliably." A change is correct if: |
|
|
| - The eight existing regression claims still pass. |
| - The new claim now passes. |
| - The change does not introduce a mechanism that could route other |
| claims incorrectly. (Predict what classes of claims the change |
| affects, and add regression claims for those.) |
|
|
| ## Adding a theorem to the library |
|
|
| A theorem is a claim that *no possible world* makes the input |
| correct. To add one: |
|
|
| 1. It must be a real theorem. Rank bounds, cardinality bounds, |
| entropy bounds, Euclidean angle sums. Not empirical regularities. |
| 2. It must have a detector function that returns `(violated, note)`. |
| 3. The detector must be conservative — false positives (refusing a |
| correct claim) are worse than false negatives. |
| 4. Add regression claims for both the violating case and the |
| near-miss case. |
|
|
| If you cannot state the theorem in one sentence and prove it in |
| three, it does not belong in the library. |
|
|
| ## Adding a paper to the corpus |
|
|
| 1. The paper must be real. No fabricated citations. |
| 2. Set `on_target=True` only if the paper makes the same claim the |
| corpus is testing. Sharing topic words is not enough. |
| 3. The `keywords` list should be content tokens, not generic words. |
| `effective`, `rank`, `matrix` are generic. `dragonfly`, `tctn`, |
| `matched` are content. |
| 4. Add a regression claim that exercises the new paper. |
|
|
| ## What not to contribute |
|
|
| - Changes that increase `FRONTIER` rate without adding structure. |
| `FRONTIER` is a candidate, not a goal. If a change makes more |
| claims qualify without a corresponding improvement in the |
| structural checks, it is a regression. |
| - Changes that route unfamiliar claims to `FABRICATED` on the basis |
| of a prior. The `FABRICATED` verdict requires *structural* |
| evidence (no mechanism, no checkable sub-claim), not a prior |
| toward dismissal. |
| - New dependencies for the core module. The core is stdlib only. |
| Optional backends (arXiv retrieval, LLM classifier) live behind |
| extras. |
|
|
| ## The version history |
|
|
| Read `CHANGELOG.md` before proposing a change. Three of four |
| iterations had the same bug class: a gate that uses topical |
| similarity as a proxy for claim identity. If your change touches |
| retrieval or literature matching, check whether it introduces the |
| same pattern. |
|
|
| ## Running the tests |
|
|
| ```bash |
| pip install -e ".[dev]" |
| pytest -q |
| python -m frontier_check --strict "any claim" |
