auto-form / note.md
Menixis's picture
Claude Opus 4.7 (1M context)
Apply neon-indigo theme: gradient buttons, bright-yellow warning alert
3036e5d
|
Raw History Blame Contribute Delete
13.1 kB
# note.md β€” Architecture & Operations (read this first)
> **Purpose:** Onboard a new AI/dev to this project fast. Keep it current β€” see
> **[Β§10 Maintaining this file](#10-maintaining-this-file)**. Last updated: **2026-05-24**.
---
## 1. What this app is
**Auto Form Generator** (Thai: *ระบบΰΈͺΰΈ£ΰΉ‰ΰΈ²ΰΈ‡ΰΈ«ΰΈ™ΰΈ±ΰΈ‡ΰΈͺΰΈ·ΰΈ­ΰΈ£ΰΈ²ΰΈŠΰΈΰΈ²ΰΈ£ΰΈ­ΰΈ±ΰΈ•ΰΉ‚ΰΈ™ΰΈ‘ΰΈ±ΰΈ•ΰΈ΄*) β€” a Streamlit web
app that generates 5 standard Thai procurement letters (Word `.docx`) from templates.
The user types a **PO number**, the app looks it up in an Excel sheet and auto-fills
vendor name, job title, etc., then renders a downloadable document.
- **Live deployment:** Hugging Face Space **`Menixis/auto-form`** (private).
URL: https://huggingface.co/spaces/Menixis/auto-form
- **SDK:** Docker (see `Dockerfile`), Streamlit on **port 7860** (HF convention).
- **Theme:** "Neon Indigo" dark dashboard theme (deep indigo bg `#262150`β†’`#191730`, panel-
purple cards) with a vibrant **violet→magenta gradient** (`#7C5CFF → #FF4D8D`) as the primary
accent on the hero card, primary/download/PO-search buttons and the active step β€” all with
white text/icons and a soft glow. **Cyan** (`#22D3EE`) marks success/done; light-violet
(`#C9B7FF`) is the secondary text accent. The hero card is styled as a **ticket/receipt**: a
left-aligned "PTT DOCUMENTS GENERATOR" pill in the stub, a bold dashed perforation (light),
and two semicircular edge notches punched as TRUE transparent holes via CSS `mask` (so the
page background shows through them exactly β€” no colour seam). AUTO FORM title + subtitle are
left-aligned; all content is vertically balanced. Buttons are gradient pills. No sidebar.
Data loads silently
β€” no data-source bar and no manual upload UI (the app does not tell users where the
data comes from).
> ⚠️ The user sometimes edits `streamlit_app.py` **directly in the HF web editor**, so
> the Space can diverge from this local clone / GitHub. **Always `git fetch origin` and
> compare before pushing. Never force-push.**
---
## 2. The data-flow architecture (the important part)
This is the part that took the most effort to get right. Read it carefully.
```
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” Power Automate flow β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Power BI β”‚ ── Office Script ─▢ β”‚ Excel file on the user's β”‚
β”‚ "Data For β”‚ (cloud, on PC-off)β”‚ PERSONAL OneDrive β”‚
β”‚ Usage" β”‚ β”‚ (pttplc-my.sharepoint…) β”‚
β”‚ model β”‚ β”‚ shared "Anyone w/ link" β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
β”‚ HTTP GET (anonymous link
β”‚ + &download=1 + browser UA)
β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ HF Space Menixis/auto-formβ”‚
β”‚ load_live_data(DATA_URL) β”‚
β”‚ β†’ auto-clean β†’ lookup β”‚
β”‚ cached ~1h, sample fallbckβ”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
```
**Division of labor:**
- **User side is automated:** a **Power Automate flow** refreshes the Power BI dataset
and runs an **Office Script (Excel script)** that pulls the latest data into the Excel
file on OneDrive. This runs in the Microsoft cloud on a schedule, so it happens **even
with the user's PC off** β€” no manual save step.
- **App** fetches that file live on each load. No upload step, no database, no dataset,
no Microsoft auth token, no GitHub Action, no IT involvement β€” and it works **with the
user's PC turned off** (the file lives in the cloud).
**Why this design (the hard constraints that drove it):** the data had to be
PC-off-accessible, free, with no admin/IT involvement and no fragile periodic re-login.
Those constraints ruled out service principals, Power Automate premium, scheduled
uploads, etc. The breakthrough: **PTT blocks anonymous share links on team SharePoint
sites but allows them on personal OneDrive** (`pttplc-my.sharepoint.com`). So the app
reads a personal-OneDrive anonymous link directly. Tradeoff the user accepted: anyone
with that obscure link can read the file.
**Data source upstream:** Power BI semantic model **"Data For Usage"**
(workspace `461e9fbe-d85e-4070-bf3b-526fbb127b4c`, dataset
`6fffb9e8-1be2-4091-874b-4aa8f04ea1c4`), table **"Data for Auto Form"**, lookup key
column **`poNo`**.
---
## 3. Configuration β€” the one secret that matters
| Where | Name | Value | Purpose |
|-------|------|-------|---------|
| HF Space β†’ Settings β†’ Variables and secrets | **`DATA_URL`** | the personal-OneDrive "Anyone with the link" URL to the Excel | Live data source. If unset/unreachable, app falls back to `sample_po_data.xlsx`. |
The code (`streamlit_app.py`) reads it via `os.environ.get("DATA_URL")`, auto-appends
`&download=1`, and sends a browser `User-Agent` (a plain GET 403s without it).
> The actual `DATA_URL` value is **not stored in this repo** β€” it lives only in the HF
> Space secret. Don't paste it into committed files.
**Legacy secrets that are now UNUSED** (safe to delete from the Space): `HF_TOKEN`.
Also unused: the `procurement-tools/ptt-auto-form-data` HF dataset and the
`menixis-data-read` HF token. These are leftovers from an earlier "upload to a private
HF dataset" approach that was replaced by the OneDrive-GET approach.
(The `llm-app` HF token is **NOT** unused β€” see Β§4.)
---
## 4. Keep-alive cron job (cron-job.org)
A free HF Space sleeps after ~48h of inactivity. To keep it responsive, an external
scheduler on **[cron-job.org](https://console.cron-job.org)** restarts the Space twice a
day. Account: the user's, logged in as `640097@pttplc.com` (password not stored here).
**Job "Keep PTT Auto Form Alive"** (job id `7595656`):
- **Request:** `POST https://huggingface.co/api/spaces/Menixis/auto-form/restart`
- **Header:** `Authorization: Bearer <HF write token>` β€” the token named **`llm-app`**
in HF β†’ Settings β†’ Access Tokens (value intentionally not stored in this repo).
**This token is IN USE β€” do not delete it.** If it is ever rotated, update the
`Authorization` header in the cron job too, or the restarts will start failing (401).
- **Schedule:** crontab `0 */12 * * *` β†’ **00:00 and 12:00 Asia/Bangkok**, 30s timeout,
POST with empty body.
- Notifies the account owner if the job is auto-disabled after repeated failures.
A sibling job **"Keep PTT LLM App Alive"** does the identical thing for a *different*
Space (`Menixis/llm-app`) β€” not part of this app, but it shares the same `llm-app` token.
> The restart also clears the app's in-memory state, so the next page load re-fetches
> fresh OneDrive data. (Day-to-day freshness is already handled by the 1-hour
> `load_live_data` cache β€” the cron job's real job is preventing the idle-sleep, not data
> freshness.)
>
> The `llm-app` token has **write** scope (the restart API requires auth), and it lives
> in plaintext inside the cron-job.org job config β€” so treat that account as sensitive.
---
## 5. Key files
| File | Role |
|------|------|
| `streamlit_app.py` | **Main entry point.** Web UI + data loading + PO lookup. ~1127 lines. |
| `form_logic.py` | Shared doc-generation logic. Exports `generate_form`, `FORM_TYPES`, `DOC_TYPE_OPTIONS`, `DISPLAY_FIELDS`. |
| `excel_lookup.py` | Legacy standalone Excel lookup helper (used only by the now-deleted desktop GUI). The web app uses `lookup_po_in_bytes()` inside `streamlit_app.py`, not this β€” **orphaned, safe to delete.** |
| `templates/form1-5.docx` | The 5 Word templates that get filled in. |
| `sample_po_data.xlsx` | Demo data + fallback when `DATA_URL` is unreachable. Demo POs: `4500987654 / 4500987655 / 4500987656`. |
| `Dockerfile` | HF Spaces container (python:3.11-slim, runs Streamlit on 7860). |
| `requirements.txt` | Deps. NOTE: lists `huggingface_hub` (now unused/legacy). `requests` is used by `load_live_data` and comes in transitively via Streamlit. |
| `README.md` | Has the HF YAML frontmatter (title/sdk/app_port) β€” **do not delete the frontmatter**, HF needs it. |
| `.streamlit/config.toml` | Theme + toolbar config. |
---
## 6. Key code (in `streamlit_app.py`)
Two functions carry the data layer. Both are robust against the **raw Power BI Excel
export format** (a banner row like "Data returned for …", a blank row, then headers
wrapped as `Data for Auto Form[poNo]`).
- **`_strip_col_wrapper(name)`** β€” turns `Data for Auto Form[poNo]` β†’ `poNo`.
- **`lookup_po_in_bytes(po_number, file_bytes, sheet_name=None, po_column="poNo")`** β€”
scans the first 15 rows to auto-detect the header row (so banner/blank rows don't
break it), strips the wrappers, matches the PO (handles floats like `4.14e9` β†’ int
string), and returns `(result_dict, available_cols)`. Returns `(None, cols)` if PO not
found, or `{"__error__": ...}` if the `poNo` column is missing.
- **`load_live_data(url)`** β€” `@st.cache_data(ttl=3600)`; GETs the OneDrive link,
validates the response starts with `PK` (a real `.xlsx` zip β€” catches login-redirect
HTML), returns `(bytes, fetched_at)`.
**Data-source selection** (~line 841): `load_live_data(DATA_URL)`, else fall back to
`sample_po_data.xlsx`. Loads silently on each page load. The manual-upload override and
the data-source bar were removed (2026-05-23) β€” the UI no longer shows the data origin.
The `fetched_at` returned by `load_live_data` is now discarded.
---
## 7. Deploying changes
The Space deploys on push to `main`.
```powershell
cd "C:\Users\640097\Documents\Applications\auto-form-generator"
git fetch origin # ALWAYS β€” the Space may have been edited in the web UI
git log --oneline origin/main -3 # compare; reconcile before pushing
# ... make edits, commit ...
git push origin main
```
**HF git auth:** there are no ambient credentials. The working pattern is: mint a
**temporary** *write* token in the HF web UI, push using
`https://Menixis:<TOKEN>@huggingface.co/spaces/Menixis/auto-form`, then **revoke the
token** immediately after.
> ⚠️ **Do not leave a token embedded in the git remote.** The remote was scrubbed clean on
> 2026-05-23 (`git remote set-url origin https://huggingface.co/spaces/Menixis/auto-form`).
> Keep it that way β€” pass the token inline in the push command instead.
Pushing to `main` is gated by Claude Code's permission classifier and needs a
`PowerShell(git -C "<repo>" push*)` allow rule in `~/.claude/settings.json` (already
added for this repo path).
---
## 8. Gotchas (things that already bit us)
1. **Don't keep this repo inside OneDrive-synced `Documents`.** Mid-session, OneDrive
dehydrated/removed the working folder and broke git. The folder is currently pinned
with `attrib +P -U`, but the real fix is to **move it out of OneDrive** (e.g. `C:\dev\`).
2. **Team-site SharePoint links don't work headless** β€” they redirect to login. Only the
**personal OneDrive** anonymous link works, and only with a browser `User-Agent`.
3. **The HF Space code can diverge from this clone** (web-UI edits). Always fetch first.
4. **`requirements.txt` lists `huggingface_hub`** but nothing imports it anymore β€” it's a
legacy dep. Harmless; remove only if doing a deliberate cleanup.
5. **The keep-alive depends on the `llm-app` HF token** (Β§4). Deleting/rotating that token
without updating the cron jobs silently stops the Space restarts β†’ it can sleep again.
---
## 9. Known cleanup items (optional, none urgent)
- [x] Scrub the embedded HF token from `.git/config` remote URL (done 2026-05-23).
- [ ] Delete unused HF assets: `HF_TOKEN` Space secret, `menixis-data-read` token,
`procurement-tools/ptt-auto-form-data` dataset. **(Do NOT delete the `llm-app`
token β€” it's used by the keep-alive cron, Β§4.)**
- [ ] Move this repo out of OneDrive-synced `Documents`.
- [ ] (Optional) drop `huggingface_hub` from `requirements.txt`.
---
## 10. Maintaining this file
Update `note.md` whenever any of these change, and bump the "Last updated" date at the top:
- The data-flow architecture or the data source (Β§2) β€” e.g. if the OneDrive approach is
ever replaced.
- The `DATA_URL` secret or any other config/secret (Β§3).
- The keep-alive cron job(s) on cron-job.org (Β§4) β€” schedule, restart URL, or the
`llm-app` token.
- Key files added/removed/renamed, or the data-layer functions (Β§5–§6).
- The deploy target Space or the push procedure (Β§7).
- Any new gotcha worth warning the next person about (Β§8), or when a cleanup item in Β§9
is done (check it off / remove it).