docs/Development.md

Table Of Contents

1. Development
2. Maintainer Workflow
   1. Maintainer Loop
   2. Repository Routing
   3. Before Editing
   4. Releases
3. Implementation Standards
   1. Source Editing Rules
   2. Documentation Authoring
4. Documentation Standards
   1. Heading Structure
   2. Markdown Formatting Rules
   3. GitHub Callouts
   4. Link And Backlink Rules
   5. Static Figure Rules
5. Verification And Commit
   1. Verification
   2. Review Checklist
   3. Commit Checklist

1. Development

Use this file before changing kscrape.

2. Maintainer Workflow

Maintainer Loop

The maintainer loop is:

1. Read AGENTS.md, this file, and the relevant docs chapter.
2. Check the current worktree.
3. Locate the source owner for the behavior.
4. Edit the smallest source surface that owns the behavior.
5. Run focused verification.
6. Run broader verification when shared behavior changes.
7. Review the Git diff.
8. Commit.

Related links:

• Use How-To User Guides for user-facing command recipes.
• Use References for exact paths, commands, and public surfaces.
• Use Explanations for the project model behind the workflow.

Repository Routing

Put changes where the repository already has an owner.

Change Type	Owner
Runtime package behavior	src/kscrape
Release checks	src/kscrape_dev
Tests	tests
Small user-facing examples	examples when the directory exists
Static assets	assets
Documentation	docs
Project metadata and dependency declarations	pyproject.toml
Development recipes	justfile
Local agent guidance	AGENTS.md

Ask before creating a new top-level directory when one of these owners is a reasonable fit.

Related: use project paths for source file owners and package layout for import boundaries.

Before Editing

Run these checks before a non-trivial change:

git status --short
rg --files

Then read the source owner and nearby files. A CLI change usually requires checking src/kscrape/cli/app.py, the wrapper in src/kscrape/main.py, tests, README, and docs references.

Do not weaken production contracts to make a test easier. Add or reuse a dedicated test helper when a test needs lighter setup.

Releases

CHANGELOG.md is the source for tag notes. Before a release, move every entry from [Unreleased] into a new # [MAJOR.MINOR.PATCH] - YYYY-MM-DD section, add its table-of-contents link, and leave a complete empty [Unreleased] section above it.

Before the first release command, create and commit the generated dependency files:

just lock
git add uv.lock pylock.toml
git commit -m "Lock project dependencies"

Run the local rehearsal first:

just release-check

To create the checked version commit and local tag without publishing:

just release-prepare patch

Push it only after reviewing the result:

just release-push vMAJOR.MINOR.PATCH

release-push atomically sends main and the annotated tag to the configured Hugging Face remote. It creates no GitHub Release and does not publish to PyPI.

just release creates the version commit and annotated tag, then atomically pushes main and the tag. It returns after the push without waiting for a hosted build.

3. Implementation Standards

Source Editing Rules

Follow these rules for normal edits:

• Keep runtime behavior in src/kscrape.
• Keep CLI behavior in src/kscrape/cli/app.py; keep src/kscrape/main.py wrapper-only.
• Keep tests in tests.
• Reuse existing helpers before adding new helpers.
• Keep __init__.py files limited to imports and module docstrings.
• Prefer exact domain attributes over defensive probes against strict objects.
• Document public behavior with Sphinx-style docstrings.
• Update docs when a change affects setup, usage, CLI behavior, public APIs, environment variables, or user-visible workflows.

Related: use public interfaces for the names that need stable documentation.

Documentation Authoring

When editing docs:

1. Keep docs/README.md as the reading map.
2. Put procedures in How-To User Guides.
3. Put maintainer workflow in Development.
4. Put exact names in References.
5. Put system concepts in Explanations.
6. Use separator comments before major sections.
7. Add direct links to source files when a reader may need to edit them.
8. Add [!NOTE] related-link callouts from concept sections to task recipes.
9. Use figure visual tokens before choosing reusable figure colors, strokes, fills, and typography.
10. Update figure assets when prose changes a diagrammed concept.

Related links:

• Use Markdown formatting rules for docs layout.
• Use link and backlink rules for related-link callouts.
• Use static figure rules for docs assets.
• Use figure visual tokens before adding reusable figure colors, strokes, fills, or typography.

4. Documentation Standards

Heading Structure

Long docs files may use multiple # headings. Use them for major document parts, not only for the file title. The table of contents must mirror the real structure so a reader can tell which sections are sequences, which sections are tool groups, and which sections are independent references.

Level	Use
#	Major document parts, for example First-Time Setup, Common Workflows, or Failure Model.
##	Recipes or concept chapters inside a major part.
###	Ordered substeps inside a large recipe.

Avoid a flat file where every section is a ## peer. A long sequence should be one recipe with substeps, not a cluster of unrelated recipes.

Markdown Formatting Rules

• Start every major docs file with a compact table of contents.
• Make the ToC match the heading hierarchy.
• Use the repository terms from Repository Terms.
• Use exact technical names: source paths, environment variables, CLI commands, config attributes, and file names.
• Avoid generic advice such as "check settings". Name the value to inspect, such as KSCRAPE_USER_AGENT, VIRTUAL_ENV, or src/kscrape.
• Use separator comments before major sections:

<!-- ======================================================== -->
## Section Title
<!-- ======================================================== -->

• Use centralized reference links near the bottom of a file when a link is reused:

<!-- --- URLs --------------------------------------------------- -->
[`uv`]: https://github.com/astral-sh/uv

• Use <details> dropdowns for long examples inside procedural docs:

<details><summary> <u> <i> Longer command sequence </i> </u> </summary>
<blockquote>

```bash
just sync
just test
```

</blockquote></details>

GitHub Callouts

Use callouts to mark the job a paragraph does. GitHub renders only the fixed labels TIP, NOTE, IMPORTANT, WARNING, and CAUTION.

Callout Syntax

> [!TIP]
> Explanation text goes here.
>
> ```bash
> just sync
> ```

Callout Types

Optional advice that improves speed, clarity, or workflow.

Helpful context that is not required to complete the task.

Required prerequisites, environment variables, version constraints, or architecture rules.

Risks, deprecated behavior, high-cost operations, or temporary bugs.

Destructive or security-relevant actions.

Link And Backlink Rules

• Prefer relative links.
• Link to exact chapters when the target section matters.
• Use [!NOTE] callouts for return links from concept and reference sections.
• Start one-line related-link callouts with Related:.
• Start multi-link related-link callouts with Related links:.
• Add a short purpose phrase for every related link so the reader knows why it matters.
• Do not use standalone backlink labels in prose.
• Add return links from explanations to the relevant how-to recipes and references.
• Add reference links from recipes when exact paths, config keys, or environment variables matter.
• Link to source files directly when a user may need to edit them.
• After moving docs, check file links, image links, and local anchors together.

Static Figure Rules

Static docs figures live in assets. This project starts with one small static SVG. Put repeatable figure builders with their source code and keep only generated SVG or PNG assets here.

1. Change the figure builder for generated figures.
2. Edit a small one-off SVG directly only when no builder exists.
3. Keep the canvas transparent.
4. Keep black text on a readable surface fill so GitHub dark mode remains legible.
5. Use neutral strokes for outlines and edges.
6. Use figure visual tokens for theme names and values.

Embed SVG files in Markdown with a centered table:

| ![Alt text](assets/example.svg) |
|:--:|
| **Fig. N - Figure Title:** Caption sentence. |

Related links:

• Use documentation authoring before expanding the docs structure or adding new figure assets.
• Use References: figure visual tokens before choosing reusable visual styles.

5. Verification And Commit

Verification

For Python changes:

ruff format .
ruff check .
pyright
python -m pytest

For docs-only changes:

git diff --check
rg -n "TO[D]O|FIX[M]E|content[R]eference|oai[c]ite" README.md docs
python -c "import pathlib, xml.etree.ElementTree as ET; [ET.parse(p) for p in pathlib.Path('docs/assets').glob('*.svg')]"

The exact command depends on the local environment. If a command is not installed, state that in the final report and verify the closest safe surface.

Review Checklist

Before finishing, review:

• The change is in the existing owner file or directory.
• New names match existing terminology.
• Docs link to exact files and chapters.
• Tests cover the changed behavior.
• The diff has no unrelated formatting churn.
• New comments follow the local comment style.

Related: use Development: before editing before widening the scope of a change.

Commit Checklist

Before committing:

1. Run focused tests for the touched area.
2. Run broader checks when shared behavior changed.
3. Review git diff --check.
4. Review git status --short.
5. Write a commit message that names the user-facing behavior.

Related: use How-To User Guides: run tests for the short command recipe.