README.md

metadata

title: CareRules Handbook Inspector
emoji: 🧒
colorFrom: purple
colorTo: pink
sdk: static
app_file: index.html
pinned: true
license: apache-2.0
short_description: Does a daycare handbook match its state's licensing rules?
models:
  - reflection/Beam-1.0-501B-A23B
datasets:
  - backpr/carerules
tags:
  - beam
  - long-context
  - verification
  - legal
  - childcare
  - benchmark
  - compliance
  - rl-environment

CareRules Handbook Inspector

A child care center adds its parent handbook (the written policies it gives families at enrollment). Beam 1.0 checks it against the official licensing rules of each state the center runs in and returns the sentences that conflict with a rule, the required policies that are missing, and suggested wording. Plain code checks every quote word for word before it's shown.

Judges: run a live check (no code needed for the sample handbooks) · Live page · Dataset · Code · Individual entry by MohammadMahdi RezaAli, published by BackPR

At a glance

• A real task on real documents. 22 states and DC, 1,819,794 tokens of official licensing rules. The same handbook sentence can be fine in North Carolina and break a rule in Texas, so one handbook is checked against several states at once, side by side.
• How Beam is used. Beam read each state's whole rulebook in one context to build a checklist of what a handbook must cover. Each check is one Beam call per state with that checklist, the relevant rule pages and the whole handbook; Beam names the exact clause a sentence conflicts with and drafts new wording.
• Measured (reasoning effort low, "Standard"): 40 of 40 planted conflicts and 24 of 24 deleted policies found across 8 states, $0.49 for 16 checks. On 428 rule questions, Beam went from 57.2% right from memory to 99.8% with the whole rulebook in context, but 20.6% of its quotes didn't match the rules word for word, which is why every quote here is checked.
• Nothing unproven is shown as fact. A finding whose quote can't be confirmed is shown as "check by hand", never as fact; so is a "rule" that turns out to be a state's guidance box. Quotes must start and end on whole words, and a "..." may not skip a "not".
• Open. Rules, checklists, test handbooks with answer keys, every run, and an RL environment whose reward comes from the same quote checker.

Why Beam's long context matters

• Checklists come from the whole rulebook. State rules run from 21 to 377 pages (20k to 210k Beam tokens). Beam read each state's rulebook in one pass and listed what a handbook must cover: 27 to 231 requirements per state, each with a quote that code verified word for word.
• Whole rulebook beats memory and search. On 428 questions about 15 states' rules, Beam with each state's whole rulebook in context answered 99.8% right. From memory it got 57.2%; as an agent that searches the rulebook, reads pages and checks its own quotes, 85.5%, at 7 times the cost ($4.73 vs $0.65).
• The live check builds on that. Each check sends the checklist, the rule pages it cites plus the pages that best match the handbook, and the whole handbook: 22k to 49k tokens per call in our tests. Sending each whole rulebook (65k to 172k tokens) found the same planted mistakes at 2.4 times the cost, so the live tool sends the relevant pages.
• Effort: at high reasoning effort ("Extra thorough") Beam found the same 40/40 and 24/24 and two real problems in the compliant handbooks that low effort missed, but also more false alarms, for $0.84 vs $0.49 for 16 checks. The live tool defaults to low ("Standard"); the sample report used high.

How Beam is used

1. Checklists (once per state). Beam reads each state's whole rulebook (up to 210k tokens) and lists what a center's written policies must cover, plus the core rules handbooks get wrong. Every item carries a quote verified against the official text.
2. Inspection (per upload). One Beam call per state: the checklist, the rulebook pages it cites plus the pages most relevant to the handbook, and the handbook. Beam returns a status for every checklist item, the exact rule clause broken, and a fix. Several states run in parallel; very long checklists are split.
3. Verification. Handbook quotes, rule quotes and section labels are checked mechanically; findings Beam can't prove become "check by hand" or are dropped.

Results

Handbook Inspector: planted-defect test (Beam 1.0, low effort, relevant rule pages)

Eight realistic, fictional handbooks (one per state, about 2,500 words each) were written to cover every required policy. Then 5 sentences were swapped for versions that break a specific rule, and 3 required policies were deleted. Each handbook was checked in both versions. Planted mistakes are easier to find than real ones.

Across all 8 states: 40/40 planted conflicts flagged (40 on the exact checklist item), 24/24 deleted policies caught, 3 conflicts claimed on the compliant handbooks, $0.49 for 16 checks. Per-state results: RESULTS.md in the dataset.

Of the 3 conflicts claimed on compliant handbooks, 2 were real gaps the handbook writer missed (Florida: an allergy list posted where parents can see it, which the rule forbids; California: the 15-minute infant sleep check must also record breathing, signs of distress and sleep position) and 1 was a misreading (New Jersey: the two-staff-in-the-building rule read as a nap-room ratio). California's compliant score is lower because its rules also cover programs this handbook doesn't run (school-age and mildly ill care).

Each planted defect maps to an item on the state's checklist, so this test measures the inspection, not whether the checklist is complete. Checklist items whose quote failed verification were dropped and are listed in each checklist's dropped_unverified (for example two of Virginia's ratio and group-size items).

Whole rulebook or relevant pages? (5 states, same handbooks)

Beam 1.0 can read a state's whole rulebook in one call. The live tool sends the checklist's pages plus the pages that best match the handbook instead, because on this test it found the same planted mistakes for less than half the cost. With the whole rulebook it claimed no conflicts on the compliant handbooks: it avoided the New Jersey misreading, but it also missed the real Florida gap (an allergy list posted where parents can see it).

Context sent to Beam	Planted conflicts flagged	Deleted policies caught	Conflicts claimed on compliant handbooks	Cost
Whole rulebook	25/25	15/15	0	$0.56
Relevant pages (live tool)	25/25	15/15	2	$0.23

Relevant pages used 2.0 to 5.5 times fewer input tokens per state (Florida 2.2x, New Jersey 2.0x, North Carolina 3.5x, Oregon 2.3x, Texas 5.5x).

Reasoning effort: Standard (low) vs Extra thorough (high), 8 states

Effort	Planted conflicts flagged	Deleted policies caught	Conflicts claimed on compliant handbooks	Cost	Slowest check, average
Standard (low)	40/40	24/24	3	$0.49	17 s
Extra thorough (high)	40/40	24/24	6	$0.84	44 s

Both efforts found every planted mistake. On the compliant handbooks we read each claimed conflict by hand. Standard's 3: 2 real (Florida's allergy list, California's sleep-check log) and 1 misreading (New Jersey). Extra thorough's 6: 3 real (California's sleep-check log; Illinois's "Tdap ... vaccination or immunity", where the rule says no test replaces Tdap; Oregon's center is open 11 hours, so preschoolers need 90 minutes of active play, not 60), 2 misreadings (New Jersey again, and Illinois rest time, which follows Section 407.350) and 1 debatable (California infant sleep position). So Extra thorough found two real problems Standard missed, raised more false alarms and cost more; the live tool defaults to Standard.

One real handbook

One licensed Texas child care center's published family handbook (85-page PDF, about 30,600 words), checked at Standard effort. The center isn't named and no handbook text is published; each finding was read against the rule by hand (not by a lawyer).

Result: score 67/100, 7 findings, 75,499 tokens in, 14 seconds, $0.04. Our reading of each finding: 1 correct, 1 reasonable, 1 worth checking, 2 depend on facts outside the handbook, 1 debatable, 1 misreading.

Finding	Rule	Our reading
breaks a rule: Parents may visit the center during hours of operation without prior approval to observe their child and program.	§746.201(7), page 19	debatable: The handbook lets staff deny entry to a parent who poses a risk; the rule lists no exception, but a safety exception is common sense.
missing: If the center does not carry required liability insurance, parents must be notified in writing before admission.	§746.207(a), page 21	depends on facts outside the handbook: The handbook says the center (a state agency) can't buy insurance coverage; if it carries no liability insurance, parents must be told in writing before admission, which could be a separate form.
incomplete: If applicable, the handbook must state transportation and water activity procedures.	§746.501(a)(14)-(15), page 33	reasonable: Swim lessons and splash days are described, but there is no clear transportation or water-activity procedure section.
check by hand: The handbook must state parent rights consistent with Division 5 of Subchapter B.	§746.501(a)(20), page 33	worth checking: Flagged as check by hand; the handbook has a short parent-rights section worth comparing with the full list of rights.
breaks a rule: Children's records must be immediately accessible to caregivers during hours of operation for emergency use.	§746.601(a), page 40	depends on facts outside the handbook: Records 'locked in an administrator's office' may not be immediately accessible to caregivers in an emergency, as the rule requires.
incomplete: A food allergy emergency plan is required for each child with a health-care-professional-diagnosed food allergy.	§746.3819, page 218	correct: The rule requires the plan to be signed and dated by the child's health-care professional and parent; the handbook's allergy policy doesn't say so.
breaks a rule: Immunization exemptions not addressed	page 45	misreading: The quoted rule is about keeping immunization records, including exemptions; it doesn't require admitting unvaccinated children.

Whole rulebook vs memory vs search: 428 questions on 15 states' rules

The same questions in three setups: no rulebook (memory only), each state's whole rulebook in one context, and an agent that searches the rulebook, reads pages and checks its own quotes.

Setup	Effort	Questions	Right answer	Right, with a real quote	Quotes not in the rulebook	Cost
No rulebook (memory only)	low	428	57.2%	0.0%	100.0%	$0.06
Whole state rulebook in context	low	428	99.8%	80.4%	20.6%	$0.65
Agent: search, read, check quote	low	428	85.5%	81.8%	0.9%	$4.73

Right answer: matches the answer key. Right, with a real quote: the answer is right and its quote appears word for word in the official rules (for a number, the quote must contain it).

Quotes not in the rulebook counts the answers that gave a quote: 15 of 428 from memory (asked for no quote without the rules; none of the 15 appears in the rules), 398 with the whole rulebook (82 didn't match; 72 of those contain numbers, mostly table rows, and 42 stitch table cells together with "..."), and 337 for the agent (3 didn't match).

Whole rulebook: each state's questions were asked together, up to 25 per call, so per-question tokens and cost are that call's share. All runs here use low reasoning effort ("Standard"); the sample report on the page used high ("Extra thorough").

Try it in 2 minutes

1. See a finished report: open the sample report (one made-up handbook checked for Texas, North Carolina and Oregon) or any of the 8 sample reports, each with its answer key. They load instantly and cost nothing.
2. Run a live check, no code needed: under "No handbook handy? Try a sample handbook", load one of the eight sample handbooks, change a sentence or two so they break a rule (for example "no more than four babies" to "up to six babies" in the Texas one), and click "Check this handbook". Beam checks it against that state's rules in about 10 to 60 seconds. A planted Texas infant-ratio mistake took 10 seconds and about 1 cent.
3. Your own handbook needs a free access code (one check a day, sent on request to hi@backpr.com). Checking the same file again is free.

This Space is a static page: the handbook is read, the rule pages are chosen and every quote is checked in your browser. A small Cloudflare Worker keeps the Beam token, checks the code, accepts only CareRules prompts, enforces spending limits and caches replies. The same page runs at care.backpr.com.

Limits

• Not legal advice. Reports are drafted by AI and checked by a computer, not reviewed by a lawyer. A quote that matches word for word proves the words are there, not that Beam read them correctly; read each finding and confirm changes with your licensing agency.
• Licensing rules only. Other laws (for example child abuse reporting laws) also apply. Rules were downloaded from official state sources on October 28, 2026 and change over time. Arkansas's rules cover out-of-school-time programs only. Family child care homes aren't covered.
• Checklists were compiled by Beam from each state's rules; every item's quote is verified, but no lawyer has reviewed the lists.
• Real handbooks are harder than the planted-defect test, and each planted mistake maps to a checklist item, so the test measures the inspection, not whether a checklist is complete. Scanned PDFs can't be read.
• Privacy: the handbook's text goes through the Worker and Hugging Face to Beam (hosted by Baseten); the answer is cached for 7 days. Use is subject to the terms of use, which follow Reflection's Usage Policy.

Licenses and credits

Code: Apache 2.0. Our data (checklists, questions, test handbooks, results): CC BY 4.0. The state rule texts are official government documents under their publishers' terms, not ours, included so quotes can be checked word for word; photos come from Unsplash, fonts are under the SIL Open Font License, and the BackPR name and logo are trademarks. Details: CREDITS.md.

Individual entry by MohammadMahdi RezaAli. Made by BackPR.