Spaces:
Running
Running
ashish commited on
Commit ·
f9bcd2d
1
Parent(s): b3be5a0
feat: add date and metadata filters
Browse files- README.md +4 -4
- dist/assets/index-BBNMQTi5.js +0 -0
- dist/assets/{index-DhT9TQgF.css → index-CZaEZ6UX.css} +1 -1
- dist/assets/index-Dc5XcwBp.js +0 -0
- dist/index.html +2 -2
- federal_register_explorer_plan.md +5 -5
- src/App.css +234 -0
- src/App.tsx +0 -408
- src/Explorer.tsx +571 -0
- src/main.tsx +2 -2
README.md
CHANGED
|
@@ -13,7 +13,7 @@ datasets:
|
|
| 13 |
|
| 14 |
Browser-native research explorer using the public [GadgetSoftware Federal Register dataset](https://huggingface.co/datasets/GadgetSoftware/federal-register). The local proof uses React, Vite, TypeScript, DuckDB-WASM, and compact same-origin monthly Parquet assets in preparation for deployment to a free public Hugging Face Static Space.
|
| 15 |
|
| 16 |
-
**Current status:** The
|
| 17 |
|
| 18 |
See [federal_register_explorer_plan.md](federal_register_explorer_plan.md) for the complete dataset audit, feature plan, architecture decisions, quality gates, and phased work.
|
| 19 |
|
|
@@ -53,9 +53,9 @@ The dataset card lists CC BY-NC 4.0 for the curated dataset and enrichment. The
|
|
| 53 |
|
| 54 |
## Next Product Steps
|
| 55 |
|
| 56 |
-
1.
|
| 57 |
-
2.
|
| 58 |
-
3.
|
| 59 |
4. Add compact relationship views on demand: official designated subjects, CFR references, dockets, and executive briefs. Load regulatory briefs only when opening a matching rule.
|
| 60 |
5. Defer the large `keyword`, `entity`, and extracted `topics` tables. Together they account for about 86% of repository bytes; loading all three for six recent months would transfer about 126 MB. First test prepared summaries or an explicit opt-in discovery view.
|
| 61 |
6. Keep effective-date and comment-window histograms for later: the source contains extreme and negative values that need an explicit valid denominator and outlier handling.
|
|
|
|
| 13 |
|
| 14 |
Browser-native research explorer using the public [GadgetSoftware Federal Register dataset](https://huggingface.co/datasets/GadgetSoftware/federal-register). The local proof uses React, Vite, TypeScript, DuckDB-WASM, and compact same-origin monthly Parquet assets in preparation for deployment to a free public Hugging Face Static Space.
|
| 15 |
|
| 16 |
+
**Current status:** The public Static Space at [ashish50514561-fr-data-analysis.static.hf.space](https://ashish50514561-fr-data-analysis.static.hf.space/) serves the two-month data proof. The local `development` branch adds recent-six/custom/all date scopes, exact document-number and title/abstract search, type/agency/significance/abstract filters, and 25-row paging; this filter increment has passed local tests but is not deployed yet. Charts and enrichment views remain future work. Python audit/preparation tooling is isolated in `.venv` and is not part of the Space runtime.
|
| 17 |
|
| 18 |
See [federal_register_explorer_plan.md](federal_register_explorer_plan.md) for the complete dataset audit, feature plan, architecture decisions, quality gates, and phased work.
|
| 19 |
|
|
|
|
| 53 |
|
| 54 |
## Next Product Steps
|
| 55 |
|
| 56 |
+
1. **Complete locally; deployment pending:** date scopes for recent six months, custom inclusive range, and full snapshot. The recent-six example (`2026-04` through `2026-09`) uses about 2.12 MB; all 21 months use about 7.11 MB. Month files load only after a query.
|
| 57 |
+
2. **Complete locally; deployment pending:** exact document-number lookup, case-insensitive all-term title/abstract search, type, agency-name substring, abstract-availability, and three-state significance filters, with 25-row paging. Multi-agency selection is still a follow-up; current matching is any document association whose name contains the entered text.
|
| 58 |
+
3. **Next:** add monthly counts from the manifest, a horizontal type comparison, and a top-agency chart. Add abstract/significance coverage summaries and an accessible table alternative; all chart counts must follow the selected scope and filters.
|
| 59 |
4. Add compact relationship views on demand: official designated subjects, CFR references, dockets, and executive briefs. Load regulatory briefs only when opening a matching rule.
|
| 60 |
5. Defer the large `keyword`, `entity`, and extracted `topics` tables. Together they account for about 86% of repository bytes; loading all three for six recent months would transfer about 126 MB. First test prepared summaries or an explicit opt-in discovery view.
|
| 61 |
6. Keep effective-date and comment-window histograms for later: the source contains extreme and negative values that need an explicit valid denominator and outlier handling.
|
dist/assets/index-BBNMQTi5.js
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
dist/assets/{index-DhT9TQgF.css → index-CZaEZ6UX.css}
RENAMED
|
@@ -1 +1 @@
|
|
| 1 |
-
#root{min-height:100vh}.page-shell{width:min(1080px,calc(100% - 48px));margin:0 auto}.masthead{min-height:74px;display:flex;align-items:center;justify-content:space-between;gap:20px;border-bottom:1px solid var(--line)}.wordmark{display:inline-flex;align-items:center;gap:12px;color:var(--ink);font-size:15px;font-weight:650;text-decoration:none}.wordmark-mark{width:36px;height:36px;display:grid;place-items:center;background:var(--forest);color:#fff;font-family:var(--serif);font-size:15px}.masthead-note,.eyebrow,.scope-label,.result-meta,.page-footer,.schema-row span{color:var(--muted);font-size:12px}.masthead-note{text-transform:uppercase}.intro{max-width:720px;padding:60px 0 42px}.eyebrow{margin:0 0 12px;color:var(--forest);font-family:var(--mono);font-size:11px;font-weight:600}.intro h1,.section-heading h2{margin:0;color:var(--ink);font-family:var(--serif);font-weight:500}.intro h1{font-size:42px;line-height:1.12}.intro-copy{max-width:620px;margin:17px 0 0;color:var(--muted);font-size:16px;line-height:1.65}.scope-strip{display:grid;grid-template-columns:repeat(3,minmax(0,1fr));border-top:1px solid var(--line);border-bottom:1px solid var(--line)}.scope-strip>div{min-width:0;padding:18px 20px 18px 0;display:grid;gap:8px}.scope-strip>div+div{padding-left:20px;border-left:1px solid var(--line)}.scope-label{text-transform:uppercase}.scope-strip strong,.scope-strip code{overflow-wrap:anywhere;color:var(--ink);font-size:13px;font-weight:600}.scope-strip code,.schema-row code,td,.result-meta{font-family:var(--mono)}.check-section{padding:40px 0 54px}.section-heading{display:flex;align-items:end;justify-content:space-between;gap:24px;margin-bottom:24px}.section-heading h2{font-size:26px;line-height:1.2}.run-button{min-height:44px;padding:0 18px;flex:0 0 auto;border:1px solid var(--forest);background:var(--forest);color:#fff;font:inherit;font-size:13px;font-weight:650;cursor:pointer}.run-button:hover:not(:disabled){background:var(--forest-dark)}.run-button:focus-visible,.schema-details summary:focus-visible{outline:3px solid var(--signal);outline-offset:3px}.run-button:disabled{cursor:wait;opacity:.7}.status-line,.empty-state{padding:16px 0;color:var(--muted);font-size:14px}.error-panel{margin-top:18px;padding:18px 20px;border-left:3px solid var(--signal);background:#fff7ef}.error-panel strong{color:#7e3116;font-size:14px}.error-panel p{margin:8px 0;color:var(--ink);font-family:var(--mono);font-size:12px;line-height:1.55;overflow-wrap:anywhere}.error-panel span{color:var(--muted);font-size:12px}.result-meta{display:flex;flex-wrap:wrap;gap:10px 24px;padding:12px 0;border-top:1px solid var(--line);font-family:var(--mono)}.table-wrap{width:100%;overflow-x:auto}table{width:100%;border-collapse:collapse;text-align:left}th,td{padding:14px 12px;border-bottom:1px solid var(--line);vertical-align:top}thead th{color:var(--muted);font-size:11px;font-weight:600;text-transform:uppercase}tbody th{color:var(--ink);font-size:14px;font-weight:650;white-space:nowrap}td{color:var(--ink-soft);font-size:12px;white-space:nowrap}.total-row th,.total-row td{border-top:1px solid var(--line-strong);color:var(--forest-dark);font-weight:700}.schema-details{margin-top:24px;border-top:1px solid var(--line);border-bottom:1px solid var(--line)}.schema-details summary{padding:15px 0;color:var(--forest-dark);font-size:13px;font-weight:650;cursor:pointer}.schema-list{display:grid;grid-template-columns:repeat(3,minmax(0,1fr));border-top:1px solid var(--line)}.schema-row{min-width:0;padding:9px 12px 9px 0;display:flex;align-items:baseline;justify-content:space-between;gap:8px;border-bottom:1px solid var(--line)}.schema-row code{overflow-wrap:anywhere;color:var(--ink-soft);font-size:11px}.schema-row span{flex:0 0 auto;font-family:var(--mono);font-size:10px}.page-footer{min-height:54px;display:flex;align-items:center;justify-content:space-between;gap:16px;border-top:1px solid var(--line)}@media(max-width:700px){.page-shell{width:min(100% - 32px,560px)}.intro{padding:44px 0 30px}.intro h1{font-size:34px}.scope-strip{grid-template-columns:1fr}.scope-strip>div{padding:14px 0}.scope-strip>div+div{padding-left:0;border-top:1px solid var(--line);border-left:0}.section-heading{align-items:stretch;flex-direction:column}.run-button{width:100%}.schema-list{grid-template-columns:repeat(2,minmax(0,1fr))}.page-footer{align-items:flex-start;flex-direction:column;justify-content:center;padding:14px 0}}:root{font-family:Segoe UI,sans-serif;color:#202820;background:#f2f3ed;font-synthesis:none;text-rendering:optimizeLegibility;-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale;--ink: #202820;--ink-soft: #39443b;--muted: #68736b;--forest: #27634a;--forest-dark: #184431;--signal: #c65b31;--line: #d7ddd4;--line-strong: #aab7a9;--serif: "Iowan Old Style", "Palatino Linotype", "Book Antiqua", Georgia, serif;--mono: "Cascadia Code", "SFMono-Regular", Consolas, monospace}*{box-sizing:border-box}body{min-width:320px;min-height:100vh;margin:0}button{border-radius:0}
|
|
|
|
| 1 |
+
#root{min-height:100vh}.page-shell{width:min(1080px,calc(100% - 48px));margin:0 auto}.masthead{min-height:74px;display:flex;align-items:center;justify-content:space-between;gap:20px;border-bottom:1px solid var(--line)}.wordmark{display:inline-flex;align-items:center;gap:12px;color:var(--ink);font-size:15px;font-weight:650;text-decoration:none}.wordmark-mark{width:36px;height:36px;display:grid;place-items:center;background:var(--forest);color:#fff;font-family:var(--serif);font-size:15px}.masthead-note,.eyebrow,.scope-label,.result-meta,.page-footer,.schema-row span{color:var(--muted);font-size:12px}.masthead-note{text-transform:uppercase}.intro{max-width:720px;padding:60px 0 42px}.eyebrow{margin:0 0 12px;color:var(--forest);font-family:var(--mono);font-size:11px;font-weight:600}.intro h1,.section-heading h2{margin:0;color:var(--ink);font-family:var(--serif);font-weight:500}.intro h1{font-size:42px;line-height:1.12}.intro-copy{max-width:620px;margin:17px 0 0;color:var(--muted);font-size:16px;line-height:1.65}.scope-strip{display:grid;grid-template-columns:repeat(3,minmax(0,1fr));border-top:1px solid var(--line);border-bottom:1px solid var(--line)}.scope-strip>div{min-width:0;padding:18px 20px 18px 0;display:grid;gap:8px}.scope-strip>div+div{padding-left:20px;border-left:1px solid var(--line)}.scope-label{text-transform:uppercase}.scope-strip strong,.scope-strip code{overflow-wrap:anywhere;color:var(--ink);font-size:13px;font-weight:600}.scope-strip code,.schema-row code,td,.result-meta{font-family:var(--mono)}.check-section{padding:40px 0 54px}.section-heading{display:flex;align-items:end;justify-content:space-between;gap:24px;margin-bottom:24px}.section-heading h2{font-size:26px;line-height:1.2}.run-button{min-height:44px;padding:0 18px;flex:0 0 auto;border:1px solid var(--forest);background:var(--forest);color:#fff;font:inherit;font-size:13px;font-weight:650;cursor:pointer}.run-button:hover:not(:disabled){background:var(--forest-dark)}.run-button:focus-visible,.schema-details summary:focus-visible{outline:3px solid var(--signal);outline-offset:3px}.run-button:disabled{cursor:wait;opacity:.7}.status-line,.empty-state{padding:16px 0;color:var(--muted);font-size:14px}.error-panel{margin-top:18px;padding:18px 20px;border-left:3px solid var(--signal);background:#fff7ef}.error-panel strong{color:#7e3116;font-size:14px}.error-panel p{margin:8px 0;color:var(--ink);font-family:var(--mono);font-size:12px;line-height:1.55;overflow-wrap:anywhere}.error-panel span{color:var(--muted);font-size:12px}.result-meta{display:flex;flex-wrap:wrap;gap:10px 24px;padding:12px 0;border-top:1px solid var(--line);font-family:var(--mono)}.table-wrap{width:100%;overflow-x:auto}table{width:100%;border-collapse:collapse;text-align:left}th,td{padding:14px 12px;border-bottom:1px solid var(--line);vertical-align:top}thead th{color:var(--muted);font-size:11px;font-weight:600;text-transform:uppercase}tbody th{color:var(--ink);font-size:14px;font-weight:650;white-space:nowrap}td{color:var(--ink-soft);font-size:12px;white-space:nowrap}.total-row th,.total-row td{border-top:1px solid var(--line-strong);color:var(--forest-dark);font-weight:700}.schema-details{margin-top:24px;border-top:1px solid var(--line);border-bottom:1px solid var(--line)}.schema-details summary{padding:15px 0;color:var(--forest-dark);font-size:13px;font-weight:650;cursor:pointer}.schema-list{display:grid;grid-template-columns:repeat(3,minmax(0,1fr));border-top:1px solid var(--line)}.schema-row{min-width:0;padding:9px 12px 9px 0;display:flex;align-items:baseline;justify-content:space-between;gap:8px;border-bottom:1px solid var(--line)}.schema-row code{overflow-wrap:anywhere;color:var(--ink-soft);font-size:11px}.schema-row span{flex:0 0 auto;font-family:var(--mono);font-size:10px}.scope-modes{min-width:0;margin:0 0 22px;padding:0;display:flex;align-items:center;flex-wrap:wrap;gap:12px 22px;border:0}.scope-modes legend{width:100%;margin-bottom:10px;color:var(--muted);font-size:12px;text-transform:uppercase}.scope-modes label{display:inline-flex;align-items:center;gap:8px;color:var(--ink-soft);font-size:13px;cursor:pointer}.scope-modes input{width:16px;height:16px;margin:0;accent-color:var(--forest)}.date-range-fields{margin:0 0 22px;display:grid;grid-template-columns:repeat(2,minmax(0,220px));gap:14px}.date-range-fields label,.filter-grid label{min-width:0;display:grid;align-content:start;gap:7px}.date-range-fields label>span,.filter-grid label>span{color:var(--muted);font-size:12px}.date-range-fields input,.filter-grid input,.filter-grid select{width:100%;min-height:42px;padding:9px 11px;border:1px solid var(--line-strong);border-radius:0;background:#fffefa;color:var(--ink);font:inherit;font-size:13px}.date-range-fields input:focus-visible,.filter-grid input:focus-visible,.filter-grid select:focus-visible{outline:3px solid color-mix(in srgb,var(--forest) 30%,transparent);outline-offset:1px;border-color:var(--forest)}.filter-grid{display:grid;grid-template-columns:repeat(3,minmax(0,1fr));gap:18px 16px}.filter-grid .search-field{grid-column:span 2}.search-field small{color:var(--muted);font-size:11px;line-height:1.45}.field-error{grid-column:1 / -1;color:#8b351e;font-size:12px}.form-actions{margin-top:22px;display:flex;align-items:center;gap:16px}.secondary-button{min-height:40px;padding:0 14px;border:1px solid var(--line-strong);background:transparent;color:var(--forest-dark);font:inherit;font-size:12px;font-weight:650;cursor:pointer}.secondary-button:hover:not(:disabled){background:#e8eee7}.secondary-button:disabled{cursor:not-allowed;opacity:.45}.secondary-button:focus-visible{outline:3px solid var(--signal);outline-offset:2px}.scope-hint{color:var(--muted);font-size:11px;line-height:1.45}.results-section{margin-top:36px;border-top:1px solid var(--line-strong)}.monthly-summary{margin:4px 0 24px;display:grid;grid-template-columns:repeat(6,minmax(0,1fr));gap:1px;background:var(--line);border:1px solid var(--line)}.month-row{padding:10px;display:flex;justify-content:space-between;gap:8px;background:#f9faf5;color:var(--ink-soft);font-family:var(--mono);font-size:11px}.documents-table th,.documents-table td{white-space:normal}.documents-table tbody td{min-width:100px;max-width:290px;line-height:1.5}.documents-table tbody td:last-child{min-width:260px;color:var(--muted)}.document-title{display:block;color:var(--forest-dark);font-family:var(--serif);font-size:16px;line-height:1.35;text-decoration-thickness:1px;text-underline-offset:3px}.document-number{display:block;margin-top:6px;color:var(--muted);font-family:var(--mono);font-size:10px}.pagination{padding:16px 0;display:flex;align-items:center;justify-content:flex-end;gap:14px;color:var(--muted);font-family:var(--mono);font-size:11px}.page-footer{min-height:54px;display:flex;align-items:center;justify-content:space-between;gap:16px;border-top:1px solid var(--line)}@media(max-width:700px){.page-shell{width:min(100% - 32px,560px)}.intro{padding:44px 0 30px}.intro h1{font-size:34px}.scope-strip{grid-template-columns:1fr}.scope-strip>div{padding:14px 0}.scope-strip>div+div{padding-left:0;border-top:1px solid var(--line);border-left:0}.section-heading{align-items:stretch;flex-direction:column}.run-button{width:100%}.schema-list,.filter-grid{grid-template-columns:repeat(2,minmax(0,1fr))}.filter-grid .search-field{grid-column:1 / -1}.date-range-fields,.monthly-summary{grid-template-columns:repeat(2,minmax(0,1fr))}.form-actions{align-items:flex-start;flex-direction:column}.pagination{justify-content:space-between}.page-footer{align-items:flex-start;flex-direction:column;justify-content:center;padding:14px 0}}:root{font-family:Segoe UI,sans-serif;color:#202820;background:#f2f3ed;font-synthesis:none;text-rendering:optimizeLegibility;-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale;--ink: #202820;--ink-soft: #39443b;--muted: #68736b;--forest: #27634a;--forest-dark: #184431;--signal: #c65b31;--line: #d7ddd4;--line-strong: #aab7a9;--serif: "Iowan Old Style", "Palatino Linotype", "Book Antiqua", Georgia, serif;--mono: "Cascadia Code", "SFMono-Regular", Consolas, monospace}*{box-sizing:border-box}body{min-width:320px;min-height:100vh;margin:0}button{border-radius:0}
|
dist/assets/index-Dc5XcwBp.js
DELETED
|
The diff for this file is too large to render.
See raw diff
|
|
|
dist/index.html
CHANGED
|
@@ -10,8 +10,8 @@
|
|
| 10 |
/>
|
| 11 |
<link rel="icon" href="./favicon.ico" />
|
| 12 |
<title>Federal Register Explorer | Data Check</title>
|
| 13 |
-
<script type="module" crossorigin src="./assets/index-
|
| 14 |
-
<link rel="stylesheet" crossorigin href="./assets/index-
|
| 15 |
</head>
|
| 16 |
<body>
|
| 17 |
<div id="root"></div>
|
|
|
|
| 10 |
/>
|
| 11 |
<link rel="icon" href="./favicon.ico" />
|
| 12 |
<title>Federal Register Explorer | Data Check</title>
|
| 13 |
+
<script type="module" crossorigin src="./assets/index-BBNMQTi5.js"></script>
|
| 14 |
+
<link rel="stylesheet" crossorigin href="./assets/index-CZaEZ6UX.css">
|
| 15 |
</head>
|
| 16 |
<body>
|
| 17 |
<div id="root"></div>
|
federal_register_explorer_plan.md
CHANGED
|
@@ -394,18 +394,18 @@ Do not truncate 384-dimensional GTE vectors to a smaller dimension without evide
|
|
| 394 |
|
| 395 |
### 10. Keyword search and filter semantics
|
| 396 |
|
| 397 |
-
|
| 398 |
|
| 399 |
DuckDB-WASM provides querying, but this plan does **not** assume a particular full-text extension is available in the selected browser build. If ranked search quality is insufficient, evaluate a small lexical index separately and measure its transfer/memory overhead.
|
| 400 |
|
| 401 |
Business rules to confirm before implementation:
|
| 402 |
|
| 403 |
-
-
|
| 404 |
- Date endpoints: inclusive publication dates.
|
| 405 |
- Null abstracts: searchable by title, explicitly marked.
|
| 406 |
- Global counts: one count per document ID regardless of agency multiplicity.
|
| 407 |
- Agency counts: distinct document IDs associated with each selected agency.
|
| 408 |
-
- Sorting:
|
| 409 |
- Export: same explicit filter scope as displayed; semantic export labeled as top-K results rather than all matching topic documents.
|
| 410 |
|
| 411 |
Use parameterized queries and controlled sort options. Do not concatenate user input into SQL.
|
|
@@ -512,7 +512,7 @@ Deliverable: locally validated static document assets, manifest, and two-month b
|
|
| 512 |
|
| 513 |
1. **Complete locally:** migrate the CRA starter to Vite and verify the production build.
|
| 514 |
2. **Complete locally:** load the generated manifest and August/September projections in DuckDB-WASM; selected static-file size is 506,685 bytes.
|
| 515 |
-
3.
|
| 516 |
4. Add source-linked detail panels, selective enrichment loading, and filtered export.
|
| 517 |
5. Add monthly counts, type bars, top-agency bars, and the Methods section; add official topics and pipeline enrichment after counts reconcile.
|
| 518 |
6. Test selected-month and full-snapshot modes, missing/network/cache failures, and source-link behavior.
|
|
@@ -561,7 +561,7 @@ Recommended defaults after this review:
|
|
| 561 |
- Keep exact document-number lookup and keyword search independent of AI; display source links, attribution, provenance, and explicit coverage.
|
| 562 |
- Count distinct document IDs for document-level analytics; deduplicate agency associations and child-table joins.
|
| 563 |
|
| 564 |
-
**Next implementation gate:**
|
| 565 |
|
| 566 |
### 17. Sources and verification boundaries
|
| 567 |
|
|
|
|
| 394 |
|
| 395 |
### 10. Keyword search and filter semantics
|
| 396 |
|
| 397 |
+
The local and deployed first increment uses exact document-number lookup before text search; otherwise it applies case-insensitive all-term containment over title plus abstract. Results sort by publication date descending with document number as a stable tie-break; ranked relevance is deferred until evaluated.
|
| 398 |
|
| 399 |
DuckDB-WASM provides querying, but this plan does **not** assume a particular full-text extension is available in the selected browser build. If ranked search quality is insufficient, evaluate a small lexical index separately and measure its transfer/memory overhead.
|
| 400 |
|
| 401 |
Business rules to confirm before implementation:
|
| 402 |
|
| 403 |
+
- Current agency control: substring match on one entered agency name against any `agency_names` association. If a multi-select agency control is added, match any selected association by default; expose all-selected only if user testing supports it.
|
| 404 |
- Date endpoints: inclusive publication dates.
|
| 405 |
- Null abstracts: searchable by title, explicitly marked.
|
| 406 |
- Global counts: one count per document ID regardless of agency multiplicity.
|
| 407 |
- Agency counts: distinct document IDs associated with each selected agency.
|
| 408 |
+
- Sorting: current default is publication date descending, then `document_number` ascending for deterministic ordering.
|
| 409 |
- Export: same explicit filter scope as displayed; semantic export labeled as top-K results rather than all matching topic documents.
|
| 410 |
|
| 411 |
Use parameterized queries and controlled sort options. Do not concatenate user input into SQL.
|
|
|
|
| 512 |
|
| 513 |
1. **Complete locally:** migrate the CRA starter to Vite and verify the production build.
|
| 514 |
2. **Complete locally:** load the generated manifest and August/September projections in DuckDB-WASM; selected static-file size is 506,685 bytes.
|
| 515 |
+
3. **Complete locally; deployment pending:** date-scoped shard selection, exact lookup, title/abstract all-term search, type and agency-name filters, tri-state significance, abstract availability, and 25-row paging; show the active and available coverage.
|
| 516 |
4. Add source-linked detail panels, selective enrichment loading, and filtered export.
|
| 517 |
5. Add monthly counts, type bars, top-agency bars, and the Methods section; add official topics and pipeline enrichment after counts reconcile.
|
| 518 |
6. Test selected-month and full-snapshot modes, missing/network/cache failures, and source-link behavior.
|
|
|
|
| 561 |
- Keep exact document-number lookup and keyword search independent of AI; display source links, attribution, provenance, and explicit coverage.
|
| 562 |
- Count distinct document IDs for document-level analytics; deduplicate agency associations and child-table joins.
|
| 563 |
|
| 564 |
+
**Next implementation gate:** add the first charts—monthly counts, type mix, and top agencies—from the manifest and projected documents. All charts must share the active date/type/agency/search scope, expose tabular alternatives, and count distinct document IDs across agency arrays. Keep heavy enrichment tables opt-in.
|
| 565 |
|
| 566 |
### 17. Sources and verification boundaries
|
| 567 |
|
src/App.css
CHANGED
|
@@ -297,6 +297,215 @@ td {
|
|
| 297 |
font-size: 10px;
|
| 298 |
}
|
| 299 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 300 |
.page-footer {
|
| 301 |
min-height: 54px;
|
| 302 |
display: flex;
|
|
@@ -346,6 +555,31 @@ td {
|
|
| 346 |
grid-template-columns: repeat(2, minmax(0, 1fr));
|
| 347 |
}
|
| 348 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 349 |
.page-footer {
|
| 350 |
align-items: flex-start;
|
| 351 |
flex-direction: column;
|
|
|
|
| 297 |
font-size: 10px;
|
| 298 |
}
|
| 299 |
|
| 300 |
+
.scope-modes {
|
| 301 |
+
min-width: 0;
|
| 302 |
+
margin: 0 0 22px;
|
| 303 |
+
padding: 0;
|
| 304 |
+
display: flex;
|
| 305 |
+
align-items: center;
|
| 306 |
+
flex-wrap: wrap;
|
| 307 |
+
gap: 12px 22px;
|
| 308 |
+
border: 0;
|
| 309 |
+
}
|
| 310 |
+
|
| 311 |
+
.scope-modes legend {
|
| 312 |
+
width: 100%;
|
| 313 |
+
margin-bottom: 10px;
|
| 314 |
+
color: var(--muted);
|
| 315 |
+
font-size: 12px;
|
| 316 |
+
text-transform: uppercase;
|
| 317 |
+
}
|
| 318 |
+
|
| 319 |
+
.scope-modes label {
|
| 320 |
+
display: inline-flex;
|
| 321 |
+
align-items: center;
|
| 322 |
+
gap: 8px;
|
| 323 |
+
color: var(--ink-soft);
|
| 324 |
+
font-size: 13px;
|
| 325 |
+
cursor: pointer;
|
| 326 |
+
}
|
| 327 |
+
|
| 328 |
+
.scope-modes input {
|
| 329 |
+
width: 16px;
|
| 330 |
+
height: 16px;
|
| 331 |
+
margin: 0;
|
| 332 |
+
accent-color: var(--forest);
|
| 333 |
+
}
|
| 334 |
+
|
| 335 |
+
.date-range-fields {
|
| 336 |
+
margin: 0 0 22px;
|
| 337 |
+
display: grid;
|
| 338 |
+
grid-template-columns: repeat(2, minmax(0, 220px));
|
| 339 |
+
gap: 14px;
|
| 340 |
+
}
|
| 341 |
+
|
| 342 |
+
.date-range-fields label,
|
| 343 |
+
.filter-grid label {
|
| 344 |
+
min-width: 0;
|
| 345 |
+
display: grid;
|
| 346 |
+
align-content: start;
|
| 347 |
+
gap: 7px;
|
| 348 |
+
}
|
| 349 |
+
|
| 350 |
+
.date-range-fields label > span,
|
| 351 |
+
.filter-grid label > span {
|
| 352 |
+
color: var(--muted);
|
| 353 |
+
font-size: 12px;
|
| 354 |
+
}
|
| 355 |
+
|
| 356 |
+
.date-range-fields input,
|
| 357 |
+
.filter-grid input,
|
| 358 |
+
.filter-grid select {
|
| 359 |
+
width: 100%;
|
| 360 |
+
min-height: 42px;
|
| 361 |
+
padding: 9px 11px;
|
| 362 |
+
border: 1px solid var(--line-strong);
|
| 363 |
+
border-radius: 0;
|
| 364 |
+
background: #fffefa;
|
| 365 |
+
color: var(--ink);
|
| 366 |
+
font: inherit;
|
| 367 |
+
font-size: 13px;
|
| 368 |
+
}
|
| 369 |
+
|
| 370 |
+
.date-range-fields input:focus-visible,
|
| 371 |
+
.filter-grid input:focus-visible,
|
| 372 |
+
.filter-grid select:focus-visible {
|
| 373 |
+
outline: 3px solid color-mix(in srgb, var(--forest) 30%, transparent);
|
| 374 |
+
outline-offset: 1px;
|
| 375 |
+
border-color: var(--forest);
|
| 376 |
+
}
|
| 377 |
+
|
| 378 |
+
.filter-grid {
|
| 379 |
+
display: grid;
|
| 380 |
+
grid-template-columns: repeat(3, minmax(0, 1fr));
|
| 381 |
+
gap: 18px 16px;
|
| 382 |
+
}
|
| 383 |
+
|
| 384 |
+
.filter-grid .search-field {
|
| 385 |
+
grid-column: span 2;
|
| 386 |
+
}
|
| 387 |
+
|
| 388 |
+
.search-field small {
|
| 389 |
+
color: var(--muted);
|
| 390 |
+
font-size: 11px;
|
| 391 |
+
line-height: 1.45;
|
| 392 |
+
}
|
| 393 |
+
|
| 394 |
+
.field-error {
|
| 395 |
+
grid-column: 1 / -1;
|
| 396 |
+
color: #8b351e;
|
| 397 |
+
font-size: 12px;
|
| 398 |
+
}
|
| 399 |
+
|
| 400 |
+
.form-actions {
|
| 401 |
+
margin-top: 22px;
|
| 402 |
+
display: flex;
|
| 403 |
+
align-items: center;
|
| 404 |
+
gap: 16px;
|
| 405 |
+
}
|
| 406 |
+
|
| 407 |
+
.secondary-button {
|
| 408 |
+
min-height: 40px;
|
| 409 |
+
padding: 0 14px;
|
| 410 |
+
border: 1px solid var(--line-strong);
|
| 411 |
+
background: transparent;
|
| 412 |
+
color: var(--forest-dark);
|
| 413 |
+
font: inherit;
|
| 414 |
+
font-size: 12px;
|
| 415 |
+
font-weight: 650;
|
| 416 |
+
cursor: pointer;
|
| 417 |
+
}
|
| 418 |
+
|
| 419 |
+
.secondary-button:hover:not(:disabled) {
|
| 420 |
+
background: #e8eee7;
|
| 421 |
+
}
|
| 422 |
+
|
| 423 |
+
.secondary-button:disabled {
|
| 424 |
+
cursor: not-allowed;
|
| 425 |
+
opacity: 0.45;
|
| 426 |
+
}
|
| 427 |
+
|
| 428 |
+
.secondary-button:focus-visible {
|
| 429 |
+
outline: 3px solid var(--signal);
|
| 430 |
+
outline-offset: 2px;
|
| 431 |
+
}
|
| 432 |
+
|
| 433 |
+
.scope-hint {
|
| 434 |
+
color: var(--muted);
|
| 435 |
+
font-size: 11px;
|
| 436 |
+
line-height: 1.45;
|
| 437 |
+
}
|
| 438 |
+
|
| 439 |
+
.results-section {
|
| 440 |
+
margin-top: 36px;
|
| 441 |
+
border-top: 1px solid var(--line-strong);
|
| 442 |
+
}
|
| 443 |
+
|
| 444 |
+
.monthly-summary {
|
| 445 |
+
margin: 4px 0 24px;
|
| 446 |
+
display: grid;
|
| 447 |
+
grid-template-columns: repeat(6, minmax(0, 1fr));
|
| 448 |
+
gap: 1px;
|
| 449 |
+
background: var(--line);
|
| 450 |
+
border: 1px solid var(--line);
|
| 451 |
+
}
|
| 452 |
+
|
| 453 |
+
.month-row {
|
| 454 |
+
padding: 10px;
|
| 455 |
+
display: flex;
|
| 456 |
+
justify-content: space-between;
|
| 457 |
+
gap: 8px;
|
| 458 |
+
background: #f9faf5;
|
| 459 |
+
color: var(--ink-soft);
|
| 460 |
+
font-family: var(--mono);
|
| 461 |
+
font-size: 11px;
|
| 462 |
+
}
|
| 463 |
+
|
| 464 |
+
.documents-table th,
|
| 465 |
+
.documents-table td {
|
| 466 |
+
white-space: normal;
|
| 467 |
+
}
|
| 468 |
+
|
| 469 |
+
.documents-table tbody td {
|
| 470 |
+
min-width: 100px;
|
| 471 |
+
max-width: 290px;
|
| 472 |
+
line-height: 1.5;
|
| 473 |
+
}
|
| 474 |
+
|
| 475 |
+
.documents-table tbody td:last-child {
|
| 476 |
+
min-width: 260px;
|
| 477 |
+
color: var(--muted);
|
| 478 |
+
}
|
| 479 |
+
|
| 480 |
+
.document-title {
|
| 481 |
+
display: block;
|
| 482 |
+
color: var(--forest-dark);
|
| 483 |
+
font-family: var(--serif);
|
| 484 |
+
font-size: 16px;
|
| 485 |
+
line-height: 1.35;
|
| 486 |
+
text-decoration-thickness: 1px;
|
| 487 |
+
text-underline-offset: 3px;
|
| 488 |
+
}
|
| 489 |
+
|
| 490 |
+
.document-number {
|
| 491 |
+
display: block;
|
| 492 |
+
margin-top: 6px;
|
| 493 |
+
color: var(--muted);
|
| 494 |
+
font-family: var(--mono);
|
| 495 |
+
font-size: 10px;
|
| 496 |
+
}
|
| 497 |
+
|
| 498 |
+
.pagination {
|
| 499 |
+
padding: 16px 0;
|
| 500 |
+
display: flex;
|
| 501 |
+
align-items: center;
|
| 502 |
+
justify-content: flex-end;
|
| 503 |
+
gap: 14px;
|
| 504 |
+
color: var(--muted);
|
| 505 |
+
font-family: var(--mono);
|
| 506 |
+
font-size: 11px;
|
| 507 |
+
}
|
| 508 |
+
|
| 509 |
.page-footer {
|
| 510 |
min-height: 54px;
|
| 511 |
display: flex;
|
|
|
|
| 555 |
grid-template-columns: repeat(2, minmax(0, 1fr));
|
| 556 |
}
|
| 557 |
|
| 558 |
+
.filter-grid {
|
| 559 |
+
grid-template-columns: repeat(2, minmax(0, 1fr));
|
| 560 |
+
}
|
| 561 |
+
|
| 562 |
+
.filter-grid .search-field {
|
| 563 |
+
grid-column: 1 / -1;
|
| 564 |
+
}
|
| 565 |
+
|
| 566 |
+
.date-range-fields {
|
| 567 |
+
grid-template-columns: repeat(2, minmax(0, 1fr));
|
| 568 |
+
}
|
| 569 |
+
|
| 570 |
+
.monthly-summary {
|
| 571 |
+
grid-template-columns: repeat(2, minmax(0, 1fr));
|
| 572 |
+
}
|
| 573 |
+
|
| 574 |
+
.form-actions {
|
| 575 |
+
align-items: flex-start;
|
| 576 |
+
flex-direction: column;
|
| 577 |
+
}
|
| 578 |
+
|
| 579 |
+
.pagination {
|
| 580 |
+
justify-content: space-between;
|
| 581 |
+
}
|
| 582 |
+
|
| 583 |
.page-footer {
|
| 584 |
align-items: flex-start;
|
| 585 |
flex-direction: column;
|
src/App.tsx
DELETED
|
@@ -1,408 +0,0 @@
|
|
| 1 |
-
import {
|
| 2 |
-
AsyncDuckDB,
|
| 3 |
-
ConsoleLogger,
|
| 4 |
-
DuckDBDataProtocol,
|
| 5 |
-
getJsDelivrBundles,
|
| 6 |
-
selectBundle,
|
| 7 |
-
} from '@duckdb/duckdb-wasm';
|
| 8 |
-
import { useEffect, useState } from 'react';
|
| 9 |
-
import './App.css';
|
| 10 |
-
|
| 11 |
-
const DATASET_REVISION = '1a6a6e64788c8cd6559740475e502bcc9e608d0f';
|
| 12 |
-
const MANIFEST_URL = './data/documents/manifest.json';
|
| 13 |
-
const TEST_MONTHS = ['2026-08', '2026-09'] as const;
|
| 14 |
-
|
| 15 |
-
type SnapshotMonth = {
|
| 16 |
-
month: string;
|
| 17 |
-
path: string;
|
| 18 |
-
rows: number;
|
| 19 |
-
unique_ids: number;
|
| 20 |
-
unique_document_numbers: number;
|
| 21 |
-
bytes: number;
|
| 22 |
-
};
|
| 23 |
-
|
| 24 |
-
type SnapshotManifest = {
|
| 25 |
-
dataset_revision: string;
|
| 26 |
-
totals: {
|
| 27 |
-
months: number;
|
| 28 |
-
rows: number;
|
| 29 |
-
source_bytes: number;
|
| 30 |
-
bytes: number;
|
| 31 |
-
};
|
| 32 |
-
months: SnapshotMonth[];
|
| 33 |
-
};
|
| 34 |
-
|
| 35 |
-
type MonthResult = {
|
| 36 |
-
month: string;
|
| 37 |
-
rows: number;
|
| 38 |
-
uniqueIds: number;
|
| 39 |
-
uniqueDocumentNumbers: number;
|
| 40 |
-
missingAbstracts: number;
|
| 41 |
-
firstDate: string;
|
| 42 |
-
lastDate: string;
|
| 43 |
-
};
|
| 44 |
-
|
| 45 |
-
type AuditResult = {
|
| 46 |
-
engineVersion: string;
|
| 47 |
-
elapsedMs: number;
|
| 48 |
-
fileBytes: number;
|
| 49 |
-
totalRows: number;
|
| 50 |
-
uniqueIds: number;
|
| 51 |
-
uniqueDocumentNumbers: number;
|
| 52 |
-
monthCount: number;
|
| 53 |
-
schema: Array<{ name: string; type: string }>;
|
| 54 |
-
months: MonthResult[];
|
| 55 |
-
};
|
| 56 |
-
|
| 57 |
-
type QueryRow = Record<string, unknown>;
|
| 58 |
-
|
| 59 |
-
function numberValue(value: unknown): number {
|
| 60 |
-
return Number(value);
|
| 61 |
-
}
|
| 62 |
-
|
| 63 |
-
function dateValue(value: unknown): string {
|
| 64 |
-
if (value instanceof Date) {
|
| 65 |
-
return value.toISOString().slice(0, 10);
|
| 66 |
-
}
|
| 67 |
-
if (typeof value === 'number' || typeof value === 'bigint') {
|
| 68 |
-
return new Date(Number(value)).toISOString().slice(0, 10);
|
| 69 |
-
}
|
| 70 |
-
return String(value);
|
| 71 |
-
}
|
| 72 |
-
|
| 73 |
-
async function runDatasetAudit(manifest: SnapshotManifest): Promise<AuditResult> {
|
| 74 |
-
const shards = manifest.months.filter((shard) =>
|
| 75 |
-
TEST_MONTHS.includes(shard.month as (typeof TEST_MONTHS)[number]),
|
| 76 |
-
);
|
| 77 |
-
if (shards.length !== TEST_MONTHS.length) {
|
| 78 |
-
throw new Error('The manifest is missing one or more test months.');
|
| 79 |
-
}
|
| 80 |
-
|
| 81 |
-
const files = shards.map((shard) => `${shard.month}.slim.parquet`);
|
| 82 |
-
const bundle = await selectBundle(getJsDelivrBundles());
|
| 83 |
-
const workerUrl = URL.createObjectURL(
|
| 84 |
-
new Blob([`importScripts("${bundle.mainWorker}");`], {
|
| 85 |
-
type: 'text/javascript',
|
| 86 |
-
}),
|
| 87 |
-
);
|
| 88 |
-
const worker = new Worker(workerUrl);
|
| 89 |
-
const database = new AsyncDuckDB(new ConsoleLogger(), worker);
|
| 90 |
-
|
| 91 |
-
try {
|
| 92 |
-
await database.instantiate(bundle.mainModule, bundle.pthreadWorker);
|
| 93 |
-
URL.revokeObjectURL(workerUrl);
|
| 94 |
-
const connection = await database.connect();
|
| 95 |
-
|
| 96 |
-
try {
|
| 97 |
-
await Promise.all(
|
| 98 |
-
shards.map((shard, index) =>
|
| 99 |
-
database.registerFileURL(
|
| 100 |
-
files[index],
|
| 101 |
-
new URL(shard.path, window.location.href).toString(),
|
| 102 |
-
DuckDBDataProtocol.HTTP,
|
| 103 |
-
false,
|
| 104 |
-
),
|
| 105 |
-
),
|
| 106 |
-
);
|
| 107 |
-
|
| 108 |
-
const source = `read_parquet(['${files.join("', '")}'], union_by_name = true)`;
|
| 109 |
-
const queryStartedAt = performance.now();
|
| 110 |
-
const monthTable = await connection.query(`
|
| 111 |
-
SELECT
|
| 112 |
-
strftime(try_cast(publication_date AS DATE), '%Y-%m') AS month,
|
| 113 |
-
count(*) AS rows,
|
| 114 |
-
count(DISTINCT id) AS unique_ids,
|
| 115 |
-
count(DISTINCT document_number) AS unique_document_numbers,
|
| 116 |
-
count(*) FILTER (WHERE abstract IS NULL OR trim(abstract) = '') AS missing_abstracts,
|
| 117 |
-
min(try_cast(publication_date AS DATE)) AS first_date,
|
| 118 |
-
max(try_cast(publication_date AS DATE)) AS last_date
|
| 119 |
-
FROM ${source}
|
| 120 |
-
GROUP BY month
|
| 121 |
-
ORDER BY month
|
| 122 |
-
`);
|
| 123 |
-
const totalTable = await connection.query(`
|
| 124 |
-
SELECT
|
| 125 |
-
count(*) AS rows,
|
| 126 |
-
count(DISTINCT id) AS unique_ids,
|
| 127 |
-
count(DISTINCT document_number) AS unique_document_numbers
|
| 128 |
-
FROM ${source}
|
| 129 |
-
`);
|
| 130 |
-
const schemaTable = await connection.query(`DESCRIBE SELECT * FROM ${source}`);
|
| 131 |
-
const elapsedMs = Math.round(performance.now() - queryStartedAt);
|
| 132 |
-
|
| 133 |
-
const months = monthTable.toArray().map((row) => {
|
| 134 |
-
const value = row.toJSON() as QueryRow;
|
| 135 |
-
return {
|
| 136 |
-
month: String(value.month),
|
| 137 |
-
rows: numberValue(value.rows),
|
| 138 |
-
uniqueIds: numberValue(value.unique_ids),
|
| 139 |
-
uniqueDocumentNumbers: numberValue(value.unique_document_numbers),
|
| 140 |
-
missingAbstracts: numberValue(value.missing_abstracts),
|
| 141 |
-
firstDate: dateValue(value.first_date),
|
| 142 |
-
lastDate: dateValue(value.last_date),
|
| 143 |
-
};
|
| 144 |
-
});
|
| 145 |
-
const total = totalTable.toArray()[0]?.toJSON() as QueryRow | undefined;
|
| 146 |
-
if (!total) {
|
| 147 |
-
throw new Error('DuckDB returned no total row.');
|
| 148 |
-
}
|
| 149 |
-
|
| 150 |
-
return {
|
| 151 |
-
engineVersion: await database.getVersion(),
|
| 152 |
-
elapsedMs,
|
| 153 |
-
fileBytes: shards.reduce((totalBytes, shard) => totalBytes + shard.bytes, 0),
|
| 154 |
-
totalRows: numberValue(total.rows),
|
| 155 |
-
uniqueIds: numberValue(total.unique_ids),
|
| 156 |
-
uniqueDocumentNumbers: numberValue(total.unique_document_numbers),
|
| 157 |
-
monthCount: months.length,
|
| 158 |
-
schema: schemaTable.toArray().map((row) => {
|
| 159 |
-
const value = row.toJSON() as QueryRow;
|
| 160 |
-
return {
|
| 161 |
-
name: String(value.column_name),
|
| 162 |
-
type: String(value.column_type),
|
| 163 |
-
};
|
| 164 |
-
}),
|
| 165 |
-
months,
|
| 166 |
-
};
|
| 167 |
-
} finally {
|
| 168 |
-
await connection.close();
|
| 169 |
-
}
|
| 170 |
-
} finally {
|
| 171 |
-
URL.revokeObjectURL(workerUrl);
|
| 172 |
-
await database.terminate();
|
| 173 |
-
worker.terminate();
|
| 174 |
-
}
|
| 175 |
-
}
|
| 176 |
-
|
| 177 |
-
function App() {
|
| 178 |
-
const [manifest, setManifest] = useState<SnapshotManifest | null>(null);
|
| 179 |
-
const [manifestError, setManifestError] = useState<string | null>(null);
|
| 180 |
-
const [result, setResult] = useState<AuditResult | null>(null);
|
| 181 |
-
const [error, setError] = useState<string | null>(null);
|
| 182 |
-
const [isRunning, setIsRunning] = useState(false);
|
| 183 |
-
|
| 184 |
-
useEffect(() => {
|
| 185 |
-
let active = true;
|
| 186 |
-
void fetch(MANIFEST_URL, { cache: 'no-store' })
|
| 187 |
-
.then((response) => {
|
| 188 |
-
if (!response.ok) {
|
| 189 |
-
throw new Error(`Manifest request failed (${response.status}).`);
|
| 190 |
-
}
|
| 191 |
-
return response.json() as Promise<SnapshotManifest>;
|
| 192 |
-
})
|
| 193 |
-
.then((snapshot) => {
|
| 194 |
-
if (
|
| 195 |
-
snapshot.dataset_revision !== DATASET_REVISION ||
|
| 196 |
-
!Array.isArray(snapshot.months)
|
| 197 |
-
) {
|
| 198 |
-
throw new Error('The static manifest does not match the expected revision.');
|
| 199 |
-
}
|
| 200 |
-
if (active) {
|
| 201 |
-
setManifest(snapshot);
|
| 202 |
-
}
|
| 203 |
-
})
|
| 204 |
-
.catch((caughtError: unknown) => {
|
| 205 |
-
if (active) {
|
| 206 |
-
setManifestError(
|
| 207 |
-
caughtError instanceof Error
|
| 208 |
-
? caughtError.message
|
| 209 |
-
: 'The static manifest could not be loaded.',
|
| 210 |
-
);
|
| 211 |
-
}
|
| 212 |
-
});
|
| 213 |
-
|
| 214 |
-
return () => {
|
| 215 |
-
active = false;
|
| 216 |
-
};
|
| 217 |
-
}, []);
|
| 218 |
-
|
| 219 |
-
async function handleRun() {
|
| 220 |
-
if (!manifest) return;
|
| 221 |
-
setIsRunning(true);
|
| 222 |
-
setError(null);
|
| 223 |
-
setResult(null);
|
| 224 |
-
try {
|
| 225 |
-
setResult(await runDatasetAudit(manifest));
|
| 226 |
-
} catch (caughtError) {
|
| 227 |
-
setError(
|
| 228 |
-
caughtError instanceof Error
|
| 229 |
-
? caughtError.message
|
| 230 |
-
: 'The browser query failed.',
|
| 231 |
-
);
|
| 232 |
-
} finally {
|
| 233 |
-
setIsRunning(false);
|
| 234 |
-
}
|
| 235 |
-
}
|
| 236 |
-
|
| 237 |
-
return (
|
| 238 |
-
<main className="page-shell">
|
| 239 |
-
<header className="masthead">
|
| 240 |
-
<a className="wordmark" href="https://www.federalregister.gov/">
|
| 241 |
-
<span className="wordmark-mark" aria-hidden="true">
|
| 242 |
-
FR
|
| 243 |
-
</span>
|
| 244 |
-
<span>Federal Register Explorer</span>
|
| 245 |
-
</a>
|
| 246 |
-
<span className="masthead-note">Browser data proof</span>
|
| 247 |
-
</header>
|
| 248 |
-
|
| 249 |
-
<section className="intro" aria-labelledby="page-title">
|
| 250 |
-
<p className="eyebrow">PINNED SNAPSHOT / DOCUMENTS</p>
|
| 251 |
-
<h1 id="page-title">A two-month Parquet check</h1>
|
| 252 |
-
<p className="intro-copy">
|
| 253 |
-
Query compact, same-origin month files with DuckDB-WASM. The complete
|
| 254 |
-
metadata snapshot is selected by date rather than loaded at startup.
|
| 255 |
-
</p>
|
| 256 |
-
</section>
|
| 257 |
-
|
| 258 |
-
<section className="scope-strip" aria-label="Query scope">
|
| 259 |
-
<div>
|
| 260 |
-
<span className="scope-label">Dataset commit</span>
|
| 261 |
-
<code>{manifest?.dataset_revision.slice(0, 12) ?? 'Loading…'}</code>
|
| 262 |
-
</div>
|
| 263 |
-
<div>
|
| 264 |
-
<span className="scope-label">Month shards</span>
|
| 265 |
-
<strong>
|
| 266 |
-
{TEST_MONTHS.join(' + ')} / {manifest?.totals.months ?? '…'} months
|
| 267 |
-
</strong>
|
| 268 |
-
</div>
|
| 269 |
-
<div>
|
| 270 |
-
<span className="scope-label">Search runtime</span>
|
| 271 |
-
<strong>DuckDB-WASM / browser</strong>
|
| 272 |
-
</div>
|
| 273 |
-
<div>
|
| 274 |
-
<span className="scope-label">Selected file size</span>
|
| 275 |
-
<strong>
|
| 276 |
-
{manifest
|
| 277 |
-
? manifest.months
|
| 278 |
-
.filter((shard) =>
|
| 279 |
-
TEST_MONTHS.includes(shard.month as (typeof TEST_MONTHS)[number]),
|
| 280 |
-
)
|
| 281 |
-
.reduce((totalBytes, shard) => totalBytes + shard.bytes, 0)
|
| 282 |
-
.toLocaleString()
|
| 283 |
-
: '—'}{' '}
|
| 284 |
-
bytes
|
| 285 |
-
</strong>
|
| 286 |
-
</div>
|
| 287 |
-
</section>
|
| 288 |
-
|
| 289 |
-
<section className="check-section" aria-labelledby="check-title">
|
| 290 |
-
<div className="section-heading">
|
| 291 |
-
<div>
|
| 292 |
-
<p className="eyebrow">PHASE A / BROWSER GATE</p>
|
| 293 |
-
<h2 id="check-title">Validate monthly assets and row counts</h2>
|
| 294 |
-
</div>
|
| 295 |
-
<button
|
| 296 |
-
className="run-button"
|
| 297 |
-
type="button"
|
| 298 |
-
onClick={handleRun}
|
| 299 |
-
disabled={isRunning || !manifest}
|
| 300 |
-
>
|
| 301 |
-
{isRunning
|
| 302 |
-
? 'Checking shards…'
|
| 303 |
-
: manifest
|
| 304 |
-
? 'Run browser check'
|
| 305 |
-
: 'Loading manifest…'}
|
| 306 |
-
</button>
|
| 307 |
-
</div>
|
| 308 |
-
|
| 309 |
-
{manifestError && (
|
| 310 |
-
<div className="error-panel" role="alert">
|
| 311 |
-
<strong>Snapshot manifest unavailable</strong>
|
| 312 |
-
<p>{manifestError}</p>
|
| 313 |
-
</div>
|
| 314 |
-
)}
|
| 315 |
-
|
| 316 |
-
{isRunning && (
|
| 317 |
-
<p className="status-line" role="status">
|
| 318 |
-
Loading the selected static month files. The first run also loads
|
| 319 |
-
the DuckDB-WASM runtime.
|
| 320 |
-
</p>
|
| 321 |
-
)}
|
| 322 |
-
|
| 323 |
-
{error && (
|
| 324 |
-
<div className="error-panel" role="alert">
|
| 325 |
-
<strong>Browser query failed</strong>
|
| 326 |
-
<p>{error}</p>
|
| 327 |
-
<span>
|
| 328 |
-
Keyword search remains independent of this optional data check.
|
| 329 |
-
</span>
|
| 330 |
-
</div>
|
| 331 |
-
)}
|
| 332 |
-
|
| 333 |
-
{result && (
|
| 334 |
-
<>
|
| 335 |
-
<div className="result-meta" role="status">
|
| 336 |
-
<span>DuckDB {result.engineVersion}</span>
|
| 337 |
-
<span>{result.elapsedMs.toLocaleString()} ms query time</span>
|
| 338 |
-
<span>{result.schema.length} projected columns</span>
|
| 339 |
-
<span>{result.fileBytes.toLocaleString()} static-file bytes</span>
|
| 340 |
-
</div>
|
| 341 |
-
|
| 342 |
-
<div className="table-wrap">
|
| 343 |
-
<table>
|
| 344 |
-
<thead>
|
| 345 |
-
<tr>
|
| 346 |
-
<th scope="col">Publication month</th>
|
| 347 |
-
<th scope="col">Rows</th>
|
| 348 |
-
<th scope="col">Unique IDs</th>
|
| 349 |
-
<th scope="col">Missing abstracts</th>
|
| 350 |
-
<th scope="col">Observed dates</th>
|
| 351 |
-
</tr>
|
| 352 |
-
</thead>
|
| 353 |
-
<tbody>
|
| 354 |
-
{result.months.map((month) => (
|
| 355 |
-
<tr key={month.month}>
|
| 356 |
-
<th scope="row">{month.month}</th>
|
| 357 |
-
<td>{month.rows.toLocaleString()}</td>
|
| 358 |
-
<td>{month.uniqueIds.toLocaleString()}</td>
|
| 359 |
-
<td>{month.missingAbstracts.toLocaleString()}</td>
|
| 360 |
-
<td>
|
| 361 |
-
{month.firstDate} to {month.lastDate}
|
| 362 |
-
</td>
|
| 363 |
-
</tr>
|
| 364 |
-
))}
|
| 365 |
-
<tr className="total-row">
|
| 366 |
-
<th scope="row">Combined</th>
|
| 367 |
-
<td>{result.totalRows.toLocaleString()}</td>
|
| 368 |
-
<td>{result.uniqueIds.toLocaleString()}</td>
|
| 369 |
-
<td colSpan={2}>
|
| 370 |
-
{result.uniqueDocumentNumbers.toLocaleString()} unique
|
| 371 |
-
document numbers across {result.monthCount} shards
|
| 372 |
-
</td>
|
| 373 |
-
</tr>
|
| 374 |
-
</tbody>
|
| 375 |
-
</table>
|
| 376 |
-
</div>
|
| 377 |
-
|
| 378 |
-
<details className="schema-details">
|
| 379 |
-
<summary>Inspect the {result.schema.length}-column projection</summary>
|
| 380 |
-
<div className="schema-list">
|
| 381 |
-
{result.schema.map((column) => (
|
| 382 |
-
<div className="schema-row" key={column.name}>
|
| 383 |
-
<code>{column.name}</code>
|
| 384 |
-
<span>{column.type}</span>
|
| 385 |
-
</div>
|
| 386 |
-
))}
|
| 387 |
-
</div>
|
| 388 |
-
</details>
|
| 389 |
-
</>
|
| 390 |
-
)}
|
| 391 |
-
|
| 392 |
-
{!result && !error && !manifestError && !isRunning && manifest && (
|
| 393 |
-
<p className="empty-state">
|
| 394 |
-
No month files loaded yet. The query runs only after you start the
|
| 395 |
-
check.
|
| 396 |
-
</p>
|
| 397 |
-
)}
|
| 398 |
-
</section>
|
| 399 |
-
|
| 400 |
-
<footer className="page-footer">
|
| 401 |
-
<span>Source: GadgetSoftware / federal-register</span>
|
| 402 |
-
<span>Snapshot counts are not a completeness or policy-trend claim.</span>
|
| 403 |
-
</footer>
|
| 404 |
-
</main>
|
| 405 |
-
);
|
| 406 |
-
}
|
| 407 |
-
|
| 408 |
-
export default App;
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
src/Explorer.tsx
ADDED
|
@@ -0,0 +1,571 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import {
|
| 2 |
+
AsyncDuckDB,
|
| 3 |
+
ConsoleLogger,
|
| 4 |
+
DuckDBDataProtocol,
|
| 5 |
+
getJsDelivrBundles,
|
| 6 |
+
selectBundle,
|
| 7 |
+
} from '@duckdb/duckdb-wasm';
|
| 8 |
+
import { useEffect, useState } from 'react';
|
| 9 |
+
import type { FormEvent } from 'react';
|
| 10 |
+
import './App.css';
|
| 11 |
+
|
| 12 |
+
const DATASET_REVISION = '1a6a6e64788c8cd6559740475e502bcc9e608d0f';
|
| 13 |
+
const MANIFEST_URL = './data/documents/manifest.json';
|
| 14 |
+
const PAGE_SIZE = 25;
|
| 15 |
+
const DOCUMENT_TYPES = ['Notice', 'Rule', 'Proposed Rule', 'Presidential Document'];
|
| 16 |
+
|
| 17 |
+
type ScopeMode = 'recent-six' | 'custom' | 'all';
|
| 18 |
+
type SignificanceFilter = 'any' | 'true' | 'false' | 'unknown';
|
| 19 |
+
type AbstractFilter = 'any' | 'available' | 'missing';
|
| 20 |
+
type QueryRow = Record<string, unknown>;
|
| 21 |
+
|
| 22 |
+
type SnapshotMonth = {
|
| 23 |
+
month: string;
|
| 24 |
+
path: string;
|
| 25 |
+
rows: number;
|
| 26 |
+
unique_ids: number;
|
| 27 |
+
unique_document_numbers: number;
|
| 28 |
+
bytes: number;
|
| 29 |
+
first_publication_date: string;
|
| 30 |
+
last_publication_date: string;
|
| 31 |
+
};
|
| 32 |
+
|
| 33 |
+
type SnapshotManifest = {
|
| 34 |
+
dataset_revision: string;
|
| 35 |
+
totals: {
|
| 36 |
+
months: number;
|
| 37 |
+
rows: number;
|
| 38 |
+
source_bytes: number;
|
| 39 |
+
bytes: number;
|
| 40 |
+
};
|
| 41 |
+
months: SnapshotMonth[];
|
| 42 |
+
};
|
| 43 |
+
|
| 44 |
+
type MonthResult = {
|
| 45 |
+
month: string;
|
| 46 |
+
rows: number;
|
| 47 |
+
firstDate: string;
|
| 48 |
+
lastDate: string;
|
| 49 |
+
};
|
| 50 |
+
|
| 51 |
+
type DocumentResult = {
|
| 52 |
+
id: string;
|
| 53 |
+
documentNumber: string;
|
| 54 |
+
title: string;
|
| 55 |
+
publicationDate: string;
|
| 56 |
+
type: string;
|
| 57 |
+
agencies: string[];
|
| 58 |
+
abstract: string | null;
|
| 59 |
+
htmlUrl: string | null;
|
| 60 |
+
};
|
| 61 |
+
|
| 62 |
+
type SearchFilters = {
|
| 63 |
+
startDate: string;
|
| 64 |
+
endDate: string;
|
| 65 |
+
query: string;
|
| 66 |
+
type: string;
|
| 67 |
+
agency: string;
|
| 68 |
+
significance: SignificanceFilter;
|
| 69 |
+
abstract: AbstractFilter;
|
| 70 |
+
page: number;
|
| 71 |
+
};
|
| 72 |
+
|
| 73 |
+
type QueryResult = {
|
| 74 |
+
engineVersion: string;
|
| 75 |
+
elapsedMs: number;
|
| 76 |
+
fileBytes: number;
|
| 77 |
+
totalRows: number;
|
| 78 |
+
uniqueIds: number;
|
| 79 |
+
uniqueDocumentNumbers: number;
|
| 80 |
+
schema: Array<{ name: string; type: string }>;
|
| 81 |
+
months: MonthResult[];
|
| 82 |
+
documents: DocumentResult[];
|
| 83 |
+
page: number;
|
| 84 |
+
pageSize: number;
|
| 85 |
+
};
|
| 86 |
+
|
| 87 |
+
function numberValue(value: unknown): number {
|
| 88 |
+
return Number(value);
|
| 89 |
+
}
|
| 90 |
+
|
| 91 |
+
function dateValue(value: unknown): string {
|
| 92 |
+
if (value instanceof Date) {
|
| 93 |
+
return value.toISOString().slice(0, 10);
|
| 94 |
+
}
|
| 95 |
+
if (typeof value === 'number' || typeof value === 'bigint') {
|
| 96 |
+
return new Date(Number(value)).toISOString().slice(0, 10);
|
| 97 |
+
}
|
| 98 |
+
return String(value);
|
| 99 |
+
}
|
| 100 |
+
|
| 101 |
+
function stringList(value: unknown): string[] {
|
| 102 |
+
if (Array.isArray(value)) {
|
| 103 |
+
return value.map(String);
|
| 104 |
+
}
|
| 105 |
+
if (value && typeof value === 'object') {
|
| 106 |
+
const iterable = value as { [Symbol.iterator]?: () => Iterator<unknown> };
|
| 107 |
+
if (typeof iterable[Symbol.iterator] === 'function') {
|
| 108 |
+
return Array.from(value as Iterable<unknown>, String);
|
| 109 |
+
}
|
| 110 |
+
}
|
| 111 |
+
return [];
|
| 112 |
+
}
|
| 113 |
+
|
| 114 |
+
async function runDocumentQuery(
|
| 115 |
+
manifest: SnapshotManifest,
|
| 116 |
+
filters: SearchFilters,
|
| 117 |
+
): Promise<QueryResult> {
|
| 118 |
+
if (!filters.startDate || !filters.endDate || filters.startDate > filters.endDate) {
|
| 119 |
+
throw new Error('Choose a valid inclusive publication-date range.');
|
| 120 |
+
}
|
| 121 |
+
|
| 122 |
+
const firstMonth = filters.startDate.slice(0, 7);
|
| 123 |
+
const lastMonth = filters.endDate.slice(0, 7);
|
| 124 |
+
const shards = manifest.months.filter(
|
| 125 |
+
(shard) => shard.month >= firstMonth && shard.month <= lastMonth,
|
| 126 |
+
);
|
| 127 |
+
if (shards.length === 0) {
|
| 128 |
+
throw new Error('The selected range contains no months in this snapshot.');
|
| 129 |
+
}
|
| 130 |
+
|
| 131 |
+
const files = shards.map((shard) => `${shard.month}.slim.parquet`);
|
| 132 |
+
const bundle = await selectBundle(getJsDelivrBundles());
|
| 133 |
+
const workerUrl = URL.createObjectURL(
|
| 134 |
+
new Blob([`importScripts("${bundle.mainWorker}");`], {
|
| 135 |
+
type: 'text/javascript',
|
| 136 |
+
}),
|
| 137 |
+
);
|
| 138 |
+
const worker = new Worker(workerUrl);
|
| 139 |
+
const database = new AsyncDuckDB(new ConsoleLogger(), worker);
|
| 140 |
+
|
| 141 |
+
try {
|
| 142 |
+
await database.instantiate(bundle.mainModule, bundle.pthreadWorker);
|
| 143 |
+
URL.revokeObjectURL(workerUrl);
|
| 144 |
+
const connection = await database.connect();
|
| 145 |
+
|
| 146 |
+
try {
|
| 147 |
+
await Promise.all(
|
| 148 |
+
shards.map((shard, index) =>
|
| 149 |
+
database.registerFileURL(
|
| 150 |
+
files[index],
|
| 151 |
+
new URL(shard.path, window.location.href).toString(),
|
| 152 |
+
DuckDBDataProtocol.HTTP,
|
| 153 |
+
false,
|
| 154 |
+
),
|
| 155 |
+
),
|
| 156 |
+
);
|
| 157 |
+
|
| 158 |
+
const source = `read_parquet([${files.map((file) => `'${file}'`).join(', ')}], union_by_name = true)`;
|
| 159 |
+
const conditions = [
|
| 160 |
+
"try_cast(d.publication_date AS DATE) >= CAST(? AS DATE)",
|
| 161 |
+
"try_cast(d.publication_date AS DATE) <= CAST(? AS DATE)",
|
| 162 |
+
];
|
| 163 |
+
const values: Array<string | number> = [filters.startDate, filters.endDate];
|
| 164 |
+
const normalizedQuery = filters.query.trim();
|
| 165 |
+
|
| 166 |
+
if (/^\d{4}-\d{5}$/.test(normalizedQuery)) {
|
| 167 |
+
conditions.push('d.document_number = ?');
|
| 168 |
+
values.push(normalizedQuery);
|
| 169 |
+
} else {
|
| 170 |
+
const terms = [...new Set(normalizedQuery.toLocaleLowerCase().split(/\s+/).filter(Boolean))];
|
| 171 |
+
for (const term of terms) {
|
| 172 |
+
conditions.push(
|
| 173 |
+
"contains(lower(coalesce(d.title, '') || ' ' || coalesce(d.abstract, '')), lower(?))",
|
| 174 |
+
);
|
| 175 |
+
values.push(term);
|
| 176 |
+
}
|
| 177 |
+
}
|
| 178 |
+
|
| 179 |
+
if (filters.type) {
|
| 180 |
+
conditions.push('d.type = ?');
|
| 181 |
+
values.push(filters.type);
|
| 182 |
+
}
|
| 183 |
+
if (filters.agency.trim()) {
|
| 184 |
+
conditions.push(`EXISTS (
|
| 185 |
+
SELECT 1 FROM UNNEST(d.agency_names) AS agency(name)
|
| 186 |
+
WHERE contains(lower(agency.name), lower(?))
|
| 187 |
+
)`);
|
| 188 |
+
values.push(filters.agency.trim());
|
| 189 |
+
}
|
| 190 |
+
if (filters.significance === 'true') conditions.push('d.significant IS TRUE');
|
| 191 |
+
if (filters.significance === 'false') conditions.push('d.significant IS FALSE');
|
| 192 |
+
if (filters.significance === 'unknown') conditions.push('d.significant IS NULL');
|
| 193 |
+
if (filters.abstract === 'available') {
|
| 194 |
+
conditions.push("d.abstract IS NOT NULL AND trim(d.abstract) <> ''");
|
| 195 |
+
}
|
| 196 |
+
if (filters.abstract === 'missing') {
|
| 197 |
+
conditions.push("(d.abstract IS NULL OR trim(d.abstract) = '')");
|
| 198 |
+
}
|
| 199 |
+
|
| 200 |
+
const where = `WHERE ${conditions.join(' AND ')}`;
|
| 201 |
+
const queryPrepared = async (sql: string, parameters: Array<string | number> = []) => {
|
| 202 |
+
const statement = await connection.prepare(sql);
|
| 203 |
+
try {
|
| 204 |
+
return await statement.query(...parameters);
|
| 205 |
+
} finally {
|
| 206 |
+
await statement.close();
|
| 207 |
+
}
|
| 208 |
+
};
|
| 209 |
+
|
| 210 |
+
const queryStartedAt = performance.now();
|
| 211 |
+
const monthTable = await queryPrepared(`
|
| 212 |
+
SELECT
|
| 213 |
+
strftime(try_cast(d.publication_date AS DATE), '%Y-%m') AS month,
|
| 214 |
+
count(*) AS rows,
|
| 215 |
+
min(try_cast(d.publication_date AS DATE)) AS first_date,
|
| 216 |
+
max(try_cast(d.publication_date AS DATE)) AS last_date
|
| 217 |
+
FROM ${source} AS d
|
| 218 |
+
${where}
|
| 219 |
+
GROUP BY month
|
| 220 |
+
ORDER BY month
|
| 221 |
+
`, values);
|
| 222 |
+
const totalTable = await queryPrepared(`
|
| 223 |
+
SELECT
|
| 224 |
+
count(*) AS rows,
|
| 225 |
+
count(DISTINCT d.id) AS unique_ids,
|
| 226 |
+
count(DISTINCT d.document_number) AS unique_document_numbers
|
| 227 |
+
FROM ${source} AS d
|
| 228 |
+
${where}
|
| 229 |
+
`, values);
|
| 230 |
+
const documentTable = await queryPrepared(`
|
| 231 |
+
SELECT
|
| 232 |
+
d.id,
|
| 233 |
+
d.document_number,
|
| 234 |
+
d.title,
|
| 235 |
+
d.publication_date,
|
| 236 |
+
d.type,
|
| 237 |
+
d.agency_names,
|
| 238 |
+
d.abstract,
|
| 239 |
+
d.html_url
|
| 240 |
+
FROM ${source} AS d
|
| 241 |
+
${where}
|
| 242 |
+
ORDER BY try_cast(d.publication_date AS DATE) DESC, d.document_number ASC
|
| 243 |
+
LIMIT ? OFFSET ?
|
| 244 |
+
`, [...values, PAGE_SIZE, (filters.page - 1) * PAGE_SIZE]);
|
| 245 |
+
const schemaTable = await connection.query(`DESCRIBE SELECT * FROM ${source}`);
|
| 246 |
+
const elapsedMs = Math.round(performance.now() - queryStartedAt);
|
| 247 |
+
const total = totalTable.toArray()[0]?.toJSON() as QueryRow | undefined;
|
| 248 |
+
if (!total) {
|
| 249 |
+
throw new Error('DuckDB returned no total row.');
|
| 250 |
+
}
|
| 251 |
+
|
| 252 |
+
return {
|
| 253 |
+
engineVersion: await database.getVersion(),
|
| 254 |
+
elapsedMs,
|
| 255 |
+
fileBytes: shards.reduce((totalBytes, shard) => totalBytes + shard.bytes, 0),
|
| 256 |
+
totalRows: numberValue(total.rows),
|
| 257 |
+
uniqueIds: numberValue(total.unique_ids),
|
| 258 |
+
uniqueDocumentNumbers: numberValue(total.unique_document_numbers),
|
| 259 |
+
schema: schemaTable.toArray().map((row) => {
|
| 260 |
+
const value = row.toJSON() as QueryRow;
|
| 261 |
+
return { name: String(value.column_name), type: String(value.column_type) };
|
| 262 |
+
}),
|
| 263 |
+
months: monthTable.toArray().map((row) => {
|
| 264 |
+
const value = row.toJSON() as QueryRow;
|
| 265 |
+
return {
|
| 266 |
+
month: String(value.month),
|
| 267 |
+
rows: numberValue(value.rows),
|
| 268 |
+
firstDate: dateValue(value.first_date),
|
| 269 |
+
lastDate: dateValue(value.last_date),
|
| 270 |
+
};
|
| 271 |
+
}),
|
| 272 |
+
documents: documentTable.toArray().map((row) => {
|
| 273 |
+
const value = row.toJSON() as QueryRow;
|
| 274 |
+
return {
|
| 275 |
+
id: String(value.id),
|
| 276 |
+
documentNumber: String(value.document_number),
|
| 277 |
+
title: value.title == null ? '' : String(value.title),
|
| 278 |
+
publicationDate: dateValue(value.publication_date),
|
| 279 |
+
type: value.type == null ? '' : String(value.type),
|
| 280 |
+
agencies: stringList(value.agency_names),
|
| 281 |
+
abstract: value.abstract == null ? null : String(value.abstract),
|
| 282 |
+
htmlUrl: value.html_url == null ? null : String(value.html_url),
|
| 283 |
+
};
|
| 284 |
+
}),
|
| 285 |
+
page: filters.page,
|
| 286 |
+
pageSize: PAGE_SIZE,
|
| 287 |
+
};
|
| 288 |
+
} finally {
|
| 289 |
+
await connection.close();
|
| 290 |
+
}
|
| 291 |
+
} finally {
|
| 292 |
+
URL.revokeObjectURL(workerUrl);
|
| 293 |
+
await database.terminate();
|
| 294 |
+
worker.terminate();
|
| 295 |
+
}
|
| 296 |
+
}
|
| 297 |
+
|
| 298 |
+
function Explorer() {
|
| 299 |
+
const [manifest, setManifest] = useState<SnapshotManifest | null>(null);
|
| 300 |
+
const [manifestError, setManifestError] = useState<string | null>(null);
|
| 301 |
+
const [result, setResult] = useState<QueryResult | null>(null);
|
| 302 |
+
const [error, setError] = useState<string | null>(null);
|
| 303 |
+
const [isRunning, setIsRunning] = useState(false);
|
| 304 |
+
const [scopeMode, setScopeMode] = useState<ScopeMode>('recent-six');
|
| 305 |
+
const [customStartDate, setCustomStartDate] = useState('');
|
| 306 |
+
const [customEndDate, setCustomEndDate] = useState('');
|
| 307 |
+
const [query, setQuery] = useState('');
|
| 308 |
+
const [typeFilter, setTypeFilter] = useState('');
|
| 309 |
+
const [agencyFilter, setAgencyFilter] = useState('');
|
| 310 |
+
const [significanceFilter, setSignificanceFilter] = useState<SignificanceFilter>('any');
|
| 311 |
+
const [abstractFilter, setAbstractFilter] = useState<AbstractFilter>('any');
|
| 312 |
+
|
| 313 |
+
useEffect(() => {
|
| 314 |
+
let active = true;
|
| 315 |
+
void fetch(MANIFEST_URL, { cache: 'no-store' })
|
| 316 |
+
.then((response) => {
|
| 317 |
+
if (!response.ok) throw new Error(`Manifest request failed (${response.status}).`);
|
| 318 |
+
return response.json() as Promise<SnapshotManifest>;
|
| 319 |
+
})
|
| 320 |
+
.then((snapshot) => {
|
| 321 |
+
if (snapshot.dataset_revision !== DATASET_REVISION || !Array.isArray(snapshot.months)) {
|
| 322 |
+
throw new Error('The static manifest does not match the expected revision.');
|
| 323 |
+
}
|
| 324 |
+
if (active) {
|
| 325 |
+
setManifest(snapshot);
|
| 326 |
+
setCustomStartDate(snapshot.months[Math.max(snapshot.months.length - 6, 0)]?.first_publication_date ?? '');
|
| 327 |
+
setCustomEndDate(snapshot.months.at(-1)?.last_publication_date ?? '');
|
| 328 |
+
}
|
| 329 |
+
})
|
| 330 |
+
.catch((caughtError: unknown) => {
|
| 331 |
+
if (active) {
|
| 332 |
+
setManifestError(caughtError instanceof Error ? caughtError.message : 'The static manifest could not be loaded.');
|
| 333 |
+
}
|
| 334 |
+
});
|
| 335 |
+
return () => { active = false; };
|
| 336 |
+
}, []);
|
| 337 |
+
|
| 338 |
+
const firstSnapshotDate = manifest?.months[0]?.first_publication_date ?? '';
|
| 339 |
+
const lastSnapshotDate = manifest?.months.at(-1)?.last_publication_date ?? '';
|
| 340 |
+
const recentStartDate = manifest?.months[Math.max((manifest?.months.length ?? 0) - 6, 0)]?.first_publication_date ?? '';
|
| 341 |
+
const startDate = scopeMode === 'all' ? firstSnapshotDate : scopeMode === 'custom' ? customStartDate : recentStartDate;
|
| 342 |
+
const endDate = scopeMode === 'custom' ? customEndDate : lastSnapshotDate;
|
| 343 |
+
const selectedMonths = manifest?.months.filter((month) => month.month >= startDate.slice(0, 7) && month.month <= endDate.slice(0, 7)) ?? [];
|
| 344 |
+
const selectedBytes = selectedMonths.reduce((totalBytes, month) => totalBytes + month.bytes, 0);
|
| 345 |
+
const invalidRange = !startDate || !endDate || startDate > endDate || selectedMonths.length === 0;
|
| 346 |
+
|
| 347 |
+
async function runQuery(page: number) {
|
| 348 |
+
if (!manifest || invalidRange) return;
|
| 349 |
+
setIsRunning(true);
|
| 350 |
+
setError(null);
|
| 351 |
+
try {
|
| 352 |
+
setResult(await runDocumentQuery(manifest, {
|
| 353 |
+
startDate, endDate, query, type: typeFilter, agency: agencyFilter,
|
| 354 |
+
significance: significanceFilter, abstract: abstractFilter, page,
|
| 355 |
+
}));
|
| 356 |
+
} catch (caughtError) {
|
| 357 |
+
setError(caughtError instanceof Error ? caughtError.message : 'The browser query failed.');
|
| 358 |
+
} finally {
|
| 359 |
+
setIsRunning(false);
|
| 360 |
+
}
|
| 361 |
+
}
|
| 362 |
+
|
| 363 |
+
function updateScope(mode: ScopeMode) {
|
| 364 |
+
setScopeMode(mode);
|
| 365 |
+
setResult(null);
|
| 366 |
+
setError(null);
|
| 367 |
+
}
|
| 368 |
+
|
| 369 |
+
function updateFilter<T>(setter: (value: T) => void, value: T) {
|
| 370 |
+
setter(value);
|
| 371 |
+
setResult(null);
|
| 372 |
+
setError(null);
|
| 373 |
+
}
|
| 374 |
+
|
| 375 |
+
function handleSubmit(event: FormEvent<HTMLFormElement>) {
|
| 376 |
+
event.preventDefault();
|
| 377 |
+
void runQuery(1);
|
| 378 |
+
}
|
| 379 |
+
|
| 380 |
+
function resetFilters() {
|
| 381 |
+
setScopeMode('recent-six');
|
| 382 |
+
setQuery('');
|
| 383 |
+
setTypeFilter('');
|
| 384 |
+
setAgencyFilter('');
|
| 385 |
+
setSignificanceFilter('any');
|
| 386 |
+
setAbstractFilter('any');
|
| 387 |
+
setResult(null);
|
| 388 |
+
setError(null);
|
| 389 |
+
}
|
| 390 |
+
|
| 391 |
+
const pageCount = result ? Math.max(1, Math.ceil(result.totalRows / PAGE_SIZE)) : 1;
|
| 392 |
+
|
| 393 |
+
return (
|
| 394 |
+
<main className="page-shell">
|
| 395 |
+
<header className="masthead">
|
| 396 |
+
<a className="wordmark" href="https://www.federalregister.gov/">
|
| 397 |
+
<span className="wordmark-mark" aria-hidden="true">FR</span>
|
| 398 |
+
<span>Federal Register Explorer</span>
|
| 399 |
+
</a>
|
| 400 |
+
<span className="masthead-note">Snapshot research</span>
|
| 401 |
+
</header>
|
| 402 |
+
|
| 403 |
+
<section className="intro" aria-labelledby="page-title">
|
| 404 |
+
<p className="eyebrow">GADGETSOFTWARE / FEDERAL REGISTER</p>
|
| 405 |
+
<h1 id="page-title">Explore the snapshot</h1>
|
| 406 |
+
<p className="intro-copy">
|
| 407 |
+
Search publication metadata by date, document type, and agency. Search covers titles and abstracts only; source documents remain one click away.
|
| 408 |
+
</p>
|
| 409 |
+
</section>
|
| 410 |
+
|
| 411 |
+
<section className="scope-strip" aria-label="Snapshot summary">
|
| 412 |
+
<div>
|
| 413 |
+
<span className="scope-label">Snapshot</span>
|
| 414 |
+
<code>{manifest?.dataset_revision.slice(0, 12) ?? 'Loading…'}</code>
|
| 415 |
+
</div>
|
| 416 |
+
<div>
|
| 417 |
+
<span className="scope-label">Available dates</span>
|
| 418 |
+
<strong>{firstSnapshotDate || '…'} to {lastSnapshotDate || '…'}</strong>
|
| 419 |
+
</div>
|
| 420 |
+
<div>
|
| 421 |
+
<span className="scope-label">Selected scope</span>
|
| 422 |
+
<strong>{selectedMonths.length} months / {selectedBytes.toLocaleString()} bytes</strong>
|
| 423 |
+
</div>
|
| 424 |
+
<div>
|
| 425 |
+
<span className="scope-label">Snapshot documents</span>
|
| 426 |
+
<strong>{manifest?.totals.rows.toLocaleString() ?? '…'}</strong>
|
| 427 |
+
</div>
|
| 428 |
+
</section>
|
| 429 |
+
|
| 430 |
+
<section className="check-section" aria-labelledby="search-title">
|
| 431 |
+
<form onSubmit={handleSubmit}>
|
| 432 |
+
<div className="section-heading">
|
| 433 |
+
<div>
|
| 434 |
+
<p className="eyebrow">SEARCH AND FILTER</p>
|
| 435 |
+
<h2 id="search-title">Set the research scope</h2>
|
| 436 |
+
</div>
|
| 437 |
+
<button className="run-button" type="submit" disabled={!manifest || invalidRange || isRunning}>
|
| 438 |
+
{isRunning ? 'Searching…' : 'Search snapshot'}
|
| 439 |
+
</button>
|
| 440 |
+
</div>
|
| 441 |
+
|
| 442 |
+
<fieldset className="scope-modes">
|
| 443 |
+
<legend>Date scope</legend>
|
| 444 |
+
<label><input type="radio" name="scope" checked={scopeMode === 'recent-six'} onChange={() => updateScope('recent-six')} /> Recent 6 months</label>
|
| 445 |
+
<label><input type="radio" name="scope" checked={scopeMode === 'custom'} onChange={() => updateScope('custom')} /> Custom range</label>
|
| 446 |
+
<label><input type="radio" name="scope" checked={scopeMode === 'all'} onChange={() => updateScope('all')} /> All snapshot</label>
|
| 447 |
+
</fieldset>
|
| 448 |
+
|
| 449 |
+
{scopeMode === 'custom' && (
|
| 450 |
+
<div className="date-range-fields">
|
| 451 |
+
<label>
|
| 452 |
+
<span>From</span>
|
| 453 |
+
<input type="date" min={firstSnapshotDate} max={lastSnapshotDate} value={customStartDate} onChange={(event) => { setCustomStartDate(event.target.value); setResult(null); }} required />
|
| 454 |
+
</label>
|
| 455 |
+
<label>
|
| 456 |
+
<span>Through (inclusive)</span>
|
| 457 |
+
<input type="date" min={firstSnapshotDate} max={lastSnapshotDate} value={customEndDate} onChange={(event) => { setCustomEndDate(event.target.value); setResult(null); }} required />
|
| 458 |
+
</label>
|
| 459 |
+
{invalidRange && <span className="field-error">Choose a valid date range within the snapshot.</span>}
|
| 460 |
+
</div>
|
| 461 |
+
)}
|
| 462 |
+
|
| 463 |
+
<div className="filter-grid">
|
| 464 |
+
<label className="search-field">
|
| 465 |
+
<span>Document number or title/abstract terms</span>
|
| 466 |
+
<input type="search" value={query} onChange={(event) => updateFilter(setQuery, event.target.value)} placeholder="Exact number or all terms" />
|
| 467 |
+
<small>Exact document number takes precedence. Otherwise, every term must match title or abstract.</small>
|
| 468 |
+
</label>
|
| 469 |
+
<label>
|
| 470 |
+
<span>Document type</span>
|
| 471 |
+
<select value={typeFilter} onChange={(event) => updateFilter(setTypeFilter, event.target.value)}>
|
| 472 |
+
<option value="">All types</option>
|
| 473 |
+
{DOCUMENT_TYPES.map((type) => <option key={type} value={type}>{type}</option>)}
|
| 474 |
+
</select>
|
| 475 |
+
</label>
|
| 476 |
+
<label>
|
| 477 |
+
<span>Agency name contains</span>
|
| 478 |
+
<input type="search" value={agencyFilter} onChange={(event) => updateFilter(setAgencyFilter, event.target.value)} placeholder="Any associated agency" />
|
| 479 |
+
</label>
|
| 480 |
+
<label>
|
| 481 |
+
<span>Significance flag</span>
|
| 482 |
+
<select value={significanceFilter} onChange={(event) => updateFilter(setSignificanceFilter, event.target.value as SignificanceFilter)}>
|
| 483 |
+
<option value="any">Any / unknown</option>
|
| 484 |
+
<option value="true">True</option>
|
| 485 |
+
<option value="false">False</option>
|
| 486 |
+
<option value="unknown">Unknown only</option>
|
| 487 |
+
</select>
|
| 488 |
+
</label>
|
| 489 |
+
<label>
|
| 490 |
+
<span>Abstract</span>
|
| 491 |
+
<select value={abstractFilter} onChange={(event) => updateFilter(setAbstractFilter, event.target.value as AbstractFilter)}>
|
| 492 |
+
<option value="any">Available or missing</option>
|
| 493 |
+
<option value="available">Available</option>
|
| 494 |
+
<option value="missing">Missing</option>
|
| 495 |
+
</select>
|
| 496 |
+
</label>
|
| 497 |
+
</div>
|
| 498 |
+
|
| 499 |
+
<div className="form-actions">
|
| 500 |
+
<button className="secondary-button" type="button" onClick={resetFilters}>Reset</button>
|
| 501 |
+
<span className="scope-hint">Month files are loaded only when you search. Selected file bytes are shown above.</span>
|
| 502 |
+
</div>
|
| 503 |
+
</form>
|
| 504 |
+
|
| 505 |
+
{manifestError && <div className="error-panel" role="alert"><strong>Snapshot manifest unavailable</strong><p>{manifestError}</p></div>}
|
| 506 |
+
{isRunning && <p className="status-line" role="status">Loading selected month files and searching in your browser…</p>}
|
| 507 |
+
{error && <div className="error-panel" role="alert"><strong>Search failed</strong><p>{error}</p></div>}
|
| 508 |
+
|
| 509 |
+
{result && (
|
| 510 |
+
<div className="results-section">
|
| 511 |
+
<div className="result-meta" role="status">
|
| 512 |
+
<span>{result.totalRows.toLocaleString()} matching documents</span>
|
| 513 |
+
<span>{result.fileBytes.toLocaleString()} selected bytes</span>
|
| 514 |
+
<span>{result.elapsedMs.toLocaleString()} ms in DuckDB {result.engineVersion}</span>
|
| 515 |
+
<span>{result.schema.length} columns</span>
|
| 516 |
+
</div>
|
| 517 |
+
|
| 518 |
+
<div className="monthly-summary" aria-label="Monthly matching document counts">
|
| 519 |
+
{result.months.map((month) => (
|
| 520 |
+
<div className="month-row" key={month.month}>
|
| 521 |
+
<span>{month.month}</span>
|
| 522 |
+
<span>{month.rows.toLocaleString()}</span>
|
| 523 |
+
</div>
|
| 524 |
+
))}
|
| 525 |
+
</div>
|
| 526 |
+
|
| 527 |
+
{result.documents.length === 0 ? (
|
| 528 |
+
<p className="empty-state">No documents match this scope and filter set.</p>
|
| 529 |
+
) : (
|
| 530 |
+
<div className="table-wrap">
|
| 531 |
+
<table className="documents-table">
|
| 532 |
+
<thead><tr><th scope="col">Document</th><th scope="col">Date / type</th><th scope="col">Agency associations</th><th scope="col">Abstract</th></tr></thead>
|
| 533 |
+
<tbody>
|
| 534 |
+
{result.documents.map((document) => (
|
| 535 |
+
<tr key={document.id}>
|
| 536 |
+
<td>
|
| 537 |
+
{document.htmlUrl ? <a className="document-title" href={document.htmlUrl} target="_blank" rel="noreferrer">{document.title || document.documentNumber}</a> : <strong>{document.title || document.documentNumber}</strong>}
|
| 538 |
+
<span className="document-number">{document.documentNumber}</span>
|
| 539 |
+
</td>
|
| 540 |
+
<td>{document.publicationDate}<br />{document.type}</td>
|
| 541 |
+
<td>{document.agencies.join(', ') || 'Not provided'}</td>
|
| 542 |
+
<td>{document.abstract ? `${document.abstract.slice(0, 260)}${document.abstract.length > 260 ? '…' : ''}` : <em>Abstract unavailable in this snapshot</em>}</td>
|
| 543 |
+
</tr>
|
| 544 |
+
))}
|
| 545 |
+
</tbody>
|
| 546 |
+
</table>
|
| 547 |
+
</div>
|
| 548 |
+
)}
|
| 549 |
+
|
| 550 |
+
<div className="pagination">
|
| 551 |
+
<button className="secondary-button" type="button" onClick={() => void runQuery(result.page - 1)} disabled={isRunning || result.page <= 1}>Previous</button>
|
| 552 |
+
<span>Page {result.page} of {pageCount}</span>
|
| 553 |
+
<button className="secondary-button" type="button" onClick={() => void runQuery(result.page + 1)} disabled={isRunning || result.page >= pageCount}>Next</button>
|
| 554 |
+
</div>
|
| 555 |
+
</div>
|
| 556 |
+
)}
|
| 557 |
+
|
| 558 |
+
{!result && !error && !manifestError && !isRunning && (
|
| 559 |
+
<p className="empty-state">Choose a scope and filters, then search the pinned snapshot.</p>
|
| 560 |
+
)}
|
| 561 |
+
</section>
|
| 562 |
+
|
| 563 |
+
<footer className="page-footer">
|
| 564 |
+
<span>Source: GadgetSoftware / federal-register</span>
|
| 565 |
+
<span>Snapshot counts are not a completeness or policy-trend claim.</span>
|
| 566 |
+
</footer>
|
| 567 |
+
</main>
|
| 568 |
+
);
|
| 569 |
+
}
|
| 570 |
+
|
| 571 |
+
export default Explorer;
|
src/main.tsx
CHANGED
|
@@ -1,10 +1,10 @@
|
|
| 1 |
import React from 'react';
|
| 2 |
import ReactDOM from 'react-dom/client';
|
| 3 |
-
import
|
| 4 |
import './index.css';
|
| 5 |
|
| 6 |
ReactDOM.createRoot(document.getElementById('root')!).render(
|
| 7 |
<React.StrictMode>
|
| 8 |
-
<
|
| 9 |
</React.StrictMode>,
|
| 10 |
);
|
|
|
|
| 1 |
import React from 'react';
|
| 2 |
import ReactDOM from 'react-dom/client';
|
| 3 |
+
import Explorer from './Explorer';
|
| 4 |
import './index.css';
|
| 5 |
|
| 6 |
ReactDOM.createRoot(document.getElementById('root')!).render(
|
| 7 |
<React.StrictMode>
|
| 8 |
+
<Explorer />
|
| 9 |
</React.StrictMode>,
|
| 10 |
);
|