Table Of Contents
1. How-To User Guides
Recipe Map
Use this file when you want commands in order. Use References when you need exact names and Explanations when you need the system model.
Related: use documentation standards when adding new recipes so headings, callouts, and links stay consistent.
2. First-Time Setup
Install The Package
Use this recipe from the project root.
- Create or activate a Python environment.
- Install the package for runtime use:
python -m pip install -e "."
- Install the maintainer tools when you plan to edit the project:
python -m pip install -e "." --group dev
- If you use
uv, sync the locked maintainer environment:
just sync
Related: use dependency surfaces for the difference between runtime dependencies, optional extras, and dependency groups.
Run The First Command
Show the command tree through the console script:
kscrape --help
The module entry point should show the same command tree:
python -m kscrape --help
Run commands that do not need network access:
kscrape version
kscrape source list
kscrape config show --json
kscrape has no first-run storage setup. Network commands read their
configuration only from the process that starts them.
Related: use public interfaces for the commands and import paths users can rely on.
Set A User-Agent
Every network command needs a descriptive User-Agent. Pass it for one command:
kscrape posts search e6ai rating:s canine --user-agent "example-client/1.0"
Or export it for the current process:
export KSCRAPE_USER_AGENT="example-client/1.0"
Use optional authentication through process environment values. Never put an API key on a command line or in a request URL.
3. Common Workflows
Sync Dependencies
Use just sync to install the full maintainer environment from uv.lock:
just sync
Use plain pip when you only need the package and do not want uv:
python -m pip install -e "."
Related: use configuration and dependency model for why runtime installs and maintainer installs are documented separately.
Search Posts And Tags
Pass a source ID followed by source query terms:
kscrape posts search e621 rating:s canine --limit 10 --user-agent "example-client/1.0"
kscrape tags search e6ai "canine*" --user-agent "example-client/1.0"
Add --json to either command when another program needs machine-readable
post or tag records.
Export Post Metadata
Use posts export when a query's post records, rather than its image files,
are the result you need. This writes complete metadata by default:
kscrape posts export e621 solo --output solo-posts.json --user-agent "example-client/1.0"
Select only post IDs and their categorized tag lists with --fields:
kscrape posts export e621 solo --fields post_id,tags --output solo-tags.json --user-agent "example-client/1.0"
JSON files contain a query object with source, tag query, requested maximum,
and returned count, plus a posts array. Use --format jsonl for one selected
post object per line, either on stdout or in an output file.
Exports retrieve at most 320 posts by default. Increase the explicit bound only when needed:
kscrape posts export e621 solo --max-posts 640 --format jsonl --output solo-tags.jsonl --user-agent "example-client/1.0"
Summarize Tags Across Posts
Use posts tag-summary to count tags across matching posts without merging
their source-defined categories:
kscrape posts tag-summary e6ai solo --max-posts 640 --user-agent "example-client/1.0"
The default is a terminal table sorted by count. Use --format json or
--format jsonl for machine output. An output file requires one of those
formats:
kscrape posts tag-summary e6ai solo --format json --output solo-tag-counts.json --user-agent "example-client/1.0"
Download Images
Choose an explicit collection root. Images are saved under
downloads/posts/e6ai/images/, and the append-only manifest at
downloads/posts/e6ai/manifest.jsonl records paths relative to that folder:
kscrape posts download e6ai rating:s canine --limit 20 --output downloads --user-agent "example-client/1.0"
Original files are the default. Use --asset sample or --asset preview when
a smaller derivative is sufficient.
Download A Set
Use the set short name shown by e6AI or e621. The command reads the set's
member IDs, so it can retrieve members omitted by a set: post search:
kscrape sets download e6ai hyperrealism --output downloads --user-agent "example-client/1.0"
The command fetches every member by default. --limit 10 selects the first
ten IDs in the set's order. A numeric set ID also works. Original images go to
downloads/sets/e6ai/hyperrealism/images/; tags.json, manifest.jsonl, and
image-index.json sit beside that images/ directory. The set manifest is
replaced after each completed run and contains one record per selected member.
An unavailable member remains in the manifest and in the tag file's
unavailable_post_ids list. Rerun the command to retry it.
Files from former members remain by default. After downloading the complete
set, pass --prune to remove only former images recorded by that set's image
index. The index keeps ownership across ordinary reruns, so pruning can
also happen later. --prune cannot be combined with --limit.
The supported sources may contain mature material. Follow their rules, choose queries carefully, and store downloads responsibly.
Run Tests
Run focused tests first:
python -m pytest tests
Run the quality tools before finishing a code change:
ruff format .
ruff check .
pyright
python -m pytest
Related: use Development: verification for the maintainer checklist before a commit.
Inspect The Project
Use these commands when you need to understand the current shape:
rg --files
git status --short
kscrape --help
kscrape config show --json
python -m kscrape --help
When a command fails, copy the exact command, current directory, exit code, and stderr into the issue or debugging note.
Related links:
- Use project paths for the main source and docs locations.
- Use system model for the package and tooling boundaries.
Prepare A Release
Follow Development: releases before running these commands:
just release-check
just release-prepare patch
just release-push vMAJOR.MINOR.PATCH
The final command atomically pushes the branch and tag to the configured Hugging Face remote. It does not create a GitHub Release or publish a PyPI package.
4. Troubleshooting
Environment Problems
Check the active Python and environment first:
python --version
python -c "import sys; print(sys.executable)"
python -c "import kscrape; print(kscrape.__file__)"
If the package imports from an unexpected location, reinstall it from the project root.
Related: use failure model for the normal order of checks when a command behaves differently across machines.
Command Problems
When a just recipe fails:
- Run
just --list. - Run the underlying command manually.
- Check whether the virtual environment is active.
- Check whether the command exists in
.venv/bin.
just --list
ls .venv/bin
For a network command, confirm that KSCRAPE_USER_AGENT is set or pass
--user-agent. Check that optional credentials are supplied as complete
username and API-key pairs.
Related: use command reference for the expected commands and their owners.