Spaces:

kneiff
/

kscrape

Running

kscrape / docs /How-To-User-Guides.md

kscrape

feat(cli)!: organize downloads by workflow and track set images

c513a57 unverified about 1 month ago

11.3 kB

<!-- ======================================================== -->
## Table Of Contents
<!-- ======================================================== -->

1. [How-To User Guides](#1-how-to-user-guides)
   1. [Recipe Map](#recipe-map)
2. [First-Time Setup](#2-first-time-setup)
   1. [Install The Package](#install-the-package)
   2. [Run The First Command](#run-the-first-command)
   3. [Set A User-Agent](#set-a-user-agent)
3. [Common Workflows](#3-common-workflows)
   1. [Sync Dependencies](#sync-dependencies)
   2. [Search Posts And Tags](#search-posts-and-tags)
   3. [Export Post Metadata](#export-post-metadata)
   4. [Summarize Tags Across Posts](#summarize-tags-across-posts)
   5. [Download Images](#download-images)
   6. [Download A Set](#download-a-set)
   7. [Run Tests](#run-tests)
   8. [Inspect The Project](#inspect-the-project)
   9. [Prepare A Release](#prepare-a-release)
4. [Troubleshooting](#4-troubleshooting)
   1. [Environment Problems](#environment-problems)
   2. [Command Problems](#command-problems)

<br>

# 1. How-To User Guides

<!-- ======================================================== -->
## Recipe Map
<!-- ======================================================== -->

Use this file when you want commands in order. Use
[References](References.md) when you need exact names and
[Explanations](Explanations.md) when you need the system model.

> [!NOTE]
> Related: use [documentation standards](Development.md#4-documentation-standards)
> when adding new recipes so headings, callouts, and links stay consistent.

<br>

# 2. First-Time Setup

<!-- ======================================================== -->
## Install The Package
<!-- ======================================================== -->

Use this recipe from the project root.

1. Create or activate a Python environment.
2. Install the package for runtime use:

```bash
python -m pip install -e "."
```

3. Install the maintainer tools when you plan to edit the project:

```bash
python -m pip install -e "." --group dev
```

4. If you use `uv`, sync the locked maintainer environment:

```bash
just sync
```

> [!NOTE]
> Related: use [dependency surfaces](References.md#dependency-surfaces) for the
> difference between runtime dependencies, optional extras, and dependency
> groups.

<br>

<!-- ======================================================== -->
## Run The First Command
<!-- ======================================================== -->

Show the command tree through the console script:

```bash
kscrape --help
```

The module entry point should show the same command tree:

```bash
python -m kscrape --help
```

Run commands that do not need network access:

```bash
kscrape version
kscrape source list
kscrape config show --json
```

`kscrape` has no first-run storage setup. Network commands read their
configuration only from the process that starts them.

> [!NOTE]
> Related: use [public interfaces](References.md#public-interfaces) for the
> commands and import paths users can rely on.

<br>

<!-- ======================================================== -->
## Set A User-Agent
<!-- ======================================================== -->

Every network command needs a descriptive User-Agent. Pass it for one command:

```bash
kscrape posts search e6ai rating:s canine --user-agent "example-client/1.0"
```

Or export it for the current process:

```bash
export KSCRAPE_USER_AGENT="example-client/1.0"
```

Use optional authentication through process environment values. Never put an
API key on a command line or in a request URL.

<br>

# 3. Common Workflows

<!-- ======================================================== -->
## Sync Dependencies
<!-- ======================================================== -->

Use `just sync` to install the full maintainer environment from `uv.lock`:

```bash
just sync
```

Use plain `pip` when you only need the package and do not want `uv`:

```bash
python -m pip install -e "."
```

> [!NOTE]
> Related: use [configuration and dependency model](Explanations.md#configuration-and-dependency-model)
> for why runtime installs and maintainer installs are documented separately.

<br>

<!-- ======================================================== -->
## Search Posts And Tags
<!-- ======================================================== -->

Pass a source ID followed by source query terms:

```bash
kscrape posts search e621 rating:s canine --limit 10 --user-agent "example-client/1.0"
kscrape tags search e6ai "canine*" --user-agent "example-client/1.0"
```

Add `--json` to either command when another program needs machine-readable
post or tag records.

<br>

<!-- ======================================================== -->
## Export Post Metadata
<!-- ======================================================== -->

Use `posts export` when a query's post records, rather than its image files,
are the result you need. This writes complete metadata by default:

```bash
kscrape posts export e621 solo --output solo-posts.json --user-agent "example-client/1.0"
```

Select only post IDs and their categorized tag lists with `--fields`:

```bash
kscrape posts export e621 solo --fields post_id,tags --output solo-tags.json --user-agent "example-client/1.0"
```

JSON files contain a `query` object with source, tag query, requested maximum,
and returned count, plus a `posts` array. Use `--format jsonl` for one selected
post object per line, either on stdout or in an output file.

Exports retrieve at most 320 posts by default. Increase the explicit bound only
when needed:

```bash
kscrape posts export e621 solo --max-posts 640 --format jsonl --output solo-tags.jsonl --user-agent "example-client/1.0"
```

<br>

<!-- ======================================================== -->
## Summarize Tags Across Posts
<!-- ======================================================== -->

Use `posts tag-summary` to count tags across matching posts without merging
their source-defined categories:

```bash
kscrape posts tag-summary e6ai solo --max-posts 640 --user-agent "example-client/1.0"
```

The default is a terminal table sorted by count. Use `--format json` or
`--format jsonl` for machine output. An output file requires one of those
formats:

```bash
kscrape posts tag-summary e6ai solo --format json --output solo-tag-counts.json --user-agent "example-client/1.0"
```

<br>

<!-- ======================================================== -->
## Download Images
<!-- ======================================================== -->

Choose an explicit collection root. Images are saved under
`downloads/posts/e6ai/images/`, and the append-only manifest at
`downloads/posts/e6ai/manifest.jsonl` records paths relative to that folder:

```bash
kscrape posts download e6ai rating:s canine --limit 20 --output downloads --user-agent "example-client/1.0"
```

Original files are the default. Use `--asset sample` or `--asset preview` when
a smaller derivative is sufficient.

<br>

<!-- ======================================================== -->
## Download A Set
<!-- ======================================================== -->

Use the set short name shown by e6AI or e621. The command reads the set's
member IDs, so it can retrieve members omitted by a `set:` post search:

```bash
kscrape sets download e6ai hyperrealism --output downloads --user-agent "example-client/1.0"
```

The command fetches every member by default. `--limit 10` selects the first
ten IDs in the set's order. A numeric set ID also works. Original images go to
`downloads/sets/e6ai/hyperrealism/images/`; `tags.json`, `manifest.jsonl`, and
`image-index.json` sit beside that `images/` directory. The set manifest is
replaced after each completed run and contains one record per selected member.
An unavailable member remains in the manifest and in the tag file's
`unavailable_post_ids` list. Rerun the command to retry it.

Files from former members remain by default. After downloading the complete
set, pass `--prune` to remove only former images recorded by that set's image
index. The index keeps ownership across ordinary reruns, so pruning can
also happen later. `--prune` cannot be combined with `--limit`.

> [!WARNING]
>
> The supported sources may contain mature material. Follow their rules,
> choose queries carefully, and store downloads responsibly.

<br>

<!-- ======================================================== -->
## Run Tests
<!-- ======================================================== -->

Run focused tests first:

```bash
python -m pytest tests
```

Run the quality tools before finishing a code change:

```bash
ruff format .
ruff check .
pyright
python -m pytest
```

> [!NOTE]
> Related: use [Development: verification](Development.md#verification) for
> the maintainer checklist before a commit.

<br>

<!-- ======================================================== -->
## Inspect The Project
<!-- ======================================================== -->

Use these commands when you need to understand the current shape:

```bash
rg --files
git status --short
kscrape --help
kscrape config show --json
python -m kscrape --help
```

When a command fails, copy the exact command, current directory, exit code, and
stderr into the issue or debugging note.

> [!NOTE]
> Related links:
> - Use [project paths](References.md#project-paths) for the main source and docs locations.
> - Use [system model](Explanations.md#system-model) for the package and tooling boundaries.

<br>

<!-- ======================================================== -->
## Prepare A Release
<!-- ======================================================== -->

Follow [Development: releases](Development.md#releases) before running these
commands:

```bash
just release-check
just release-prepare patch
just release-push vMAJOR.MINOR.PATCH
```

The final command atomically pushes the branch and tag to the configured
Hugging Face remote. It does not create a GitHub Release or publish a PyPI
package.

<br>

# 4. Troubleshooting

<!-- ======================================================== -->
## Environment Problems
<!-- ======================================================== -->

Check the active Python and environment first:

```bash
python --version
python -c "import sys; print(sys.executable)"
python -c "import kscrape; print(kscrape.__file__)"
```

If the package imports from an unexpected location, reinstall it from the
project root.

> [!NOTE]
> Related: use [failure model](Explanations.md#failure-model) for the normal
> order of checks when a command behaves differently across machines.

<br>

<!-- ======================================================== -->
## Command Problems
<!-- ======================================================== -->

When a `just` recipe fails:

1. Run `just --list`.
2. Run the underlying command manually.
3. Check whether the virtual environment is active.
4. Check whether the command exists in `.venv/bin`.

```bash
just --list
ls .venv/bin
```

For a network command, confirm that `KSCRAPE_USER_AGENT` is set or pass
`--user-agent`. Check that optional credentials are supplied as complete
username and API-key pairs.

> [!NOTE]
> Related: use [command reference](References.md#command-reference) for the
> expected commands and their owners.

	<!-- ======================================================== -->
	## Table Of Contents
	<!-- ======================================================== -->

	1. [How-To User Guides](#1-how-to-user-guides)
	1. [Recipe Map](#recipe-map)
	2. [First-Time Setup](#2-first-time-setup)
	1. [Install The Package](#install-the-package)
	2. [Run The First Command](#run-the-first-command)
	3. [Set A User-Agent](#set-a-user-agent)
	3. [Common Workflows](#3-common-workflows)
	1. [Sync Dependencies](#sync-dependencies)
	2. [Search Posts And Tags](#search-posts-and-tags)
	3. [Export Post Metadata](#export-post-metadata)
	4. [Summarize Tags Across Posts](#summarize-tags-across-posts)
	5. [Download Images](#download-images)
	6. [Download A Set](#download-a-set)
	7. [Run Tests](#run-tests)
	8. [Inspect The Project](#inspect-the-project)
	9. [Prepare A Release](#prepare-a-release)
	4. [Troubleshooting](#4-troubleshooting)
	1. [Environment Problems](#environment-problems)
	2. [Command Problems](#command-problems)

	<br>

	# 1. How-To User Guides

	<!-- ======================================================== -->
	## Recipe Map
	<!-- ======================================================== -->

	Use this file when you want commands in order. Use
	[References](References.md) when you need exact names and
	[Explanations](Explanations.md) when you need the system model.

	> [!NOTE]
	> Related: use [documentation standards](Development.md#4-documentation-standards)
	> when adding new recipes so headings, callouts, and links stay consistent.

	<br>

	# 2. First-Time Setup

	<!-- ======================================================== -->
	## Install The Package
	<!-- ======================================================== -->

	Use this recipe from the project root.

	1. Create or activate a Python environment.
	2. Install the package for runtime use:

	```bash
	python -m pip install -e "."
	```

	3. Install the maintainer tools when you plan to edit the project:

	```bash
	python -m pip install -e "." --group dev
	```

	4. If you use `uv`, sync the locked maintainer environment:

	```bash
	just sync
	```

	> [!NOTE]
	> Related: use [dependency surfaces](References.md#dependency-surfaces) for the
	> difference between runtime dependencies, optional extras, and dependency
	> groups.

	<br>

	<!-- ======================================================== -->
	## Run The First Command
	<!-- ======================================================== -->

	Show the command tree through the console script:

	```bash
	kscrape --help
	```

	The module entry point should show the same command tree:

	```bash
	python -m kscrape --help
	```

	Run commands that do not need network access:

	```bash
	kscrape version
	kscrape source list
	kscrape config show --json
	```

	`kscrape` has no first-run storage setup. Network commands read their
	configuration only from the process that starts them.

	> [!NOTE]
	> Related: use [public interfaces](References.md#public-interfaces) for the
	> commands and import paths users can rely on.

	<br>

	<!-- ======================================================== -->
	## Set A User-Agent
	<!-- ======================================================== -->

	Every network command needs a descriptive User-Agent. Pass it for one command:

	```bash
	kscrape posts search e6ai rating:s canine --user-agent "example-client/1.0"
	```

	Or export it for the current process:

	```bash
	export KSCRAPE_USER_AGENT="example-client/1.0"
	```

	Use optional authentication through process environment values. Never put an
	API key on a command line or in a request URL.

	<br>

	# 3. Common Workflows

	<!-- ======================================================== -->
	## Sync Dependencies
	<!-- ======================================================== -->

	Use `just sync` to install the full maintainer environment from `uv.lock`:

	```bash
	just sync
	```

	Use plain `pip` when you only need the package and do not want `uv`:

	```bash
	python -m pip install -e "."
	```

	> [!NOTE]
	> Related: use [configuration and dependency model](Explanations.md#configuration-and-dependency-model)
	> for why runtime installs and maintainer installs are documented separately.

	<br>

	<!-- ======================================================== -->
	## Search Posts And Tags
	<!-- ======================================================== -->

	Pass a source ID followed by source query terms:

	```bash
	kscrape posts search e621 rating:s canine --limit 10 --user-agent "example-client/1.0"
	kscrape tags search e6ai "canine*" --user-agent "example-client/1.0"
	```

	Add `--json` to either command when another program needs machine-readable
	post or tag records.

	<br>

	<!-- ======================================================== -->
	## Export Post Metadata
	<!-- ======================================================== -->

	Use `posts export` when a query's post records, rather than its image files,
	are the result you need. This writes complete metadata by default:

	```bash
	kscrape posts export e621 solo --output solo-posts.json --user-agent "example-client/1.0"
	```

	Select only post IDs and their categorized tag lists with `--fields`:

	```bash
	kscrape posts export e621 solo --fields post_id,tags --output solo-tags.json --user-agent "example-client/1.0"
	```

	JSON files contain a `query` object with source, tag query, requested maximum,
	and returned count, plus a `posts` array. Use `--format jsonl` for one selected
	post object per line, either on stdout or in an output file.

	Exports retrieve at most 320 posts by default. Increase the explicit bound only
	when needed:

	```bash
	kscrape posts export e621 solo --max-posts 640 --format jsonl --output solo-tags.jsonl --user-agent "example-client/1.0"
	```

	<br>

	<!-- ======================================================== -->
	## Summarize Tags Across Posts
	<!-- ======================================================== -->

	Use `posts tag-summary` to count tags across matching posts without merging
	their source-defined categories:

	```bash
	kscrape posts tag-summary e6ai solo --max-posts 640 --user-agent "example-client/1.0"
	```

	The default is a terminal table sorted by count. Use `--format json` or
	`--format jsonl` for machine output. An output file requires one of those
	formats:

	```bash
	kscrape posts tag-summary e6ai solo --format json --output solo-tag-counts.json --user-agent "example-client/1.0"
	```

	<br>

	<!-- ======================================================== -->
	## Download Images
	<!-- ======================================================== -->

	Choose an explicit collection root. Images are saved under
	`downloads/posts/e6ai/images/`, and the append-only manifest at
	`downloads/posts/e6ai/manifest.jsonl` records paths relative to that folder:

	```bash
	kscrape posts download e6ai rating:s canine --limit 20 --output downloads --user-agent "example-client/1.0"
	```

	Original files are the default. Use `--asset sample` or `--asset preview` when
	a smaller derivative is sufficient.

	<br>

	<!-- ======================================================== -->
	## Download A Set
	<!-- ======================================================== -->

	Use the set short name shown by e6AI or e621. The command reads the set's
	member IDs, so it can retrieve members omitted by a `set:` post search:

	```bash
	kscrape sets download e6ai hyperrealism --output downloads --user-agent "example-client/1.0"
	```

	The command fetches every member by default. `--limit 10` selects the first
	ten IDs in the set's order. A numeric set ID also works. Original images go to
	`downloads/sets/e6ai/hyperrealism/images/`; `tags.json`, `manifest.jsonl`, and
	`image-index.json` sit beside that `images/` directory. The set manifest is
	replaced after each completed run and contains one record per selected member.
	An unavailable member remains in the manifest and in the tag file's
	`unavailable_post_ids` list. Rerun the command to retry it.

	Files from former members remain by default. After downloading the complete
	set, pass `--prune` to remove only former images recorded by that set's image
	index. The index keeps ownership across ordinary reruns, so pruning can
	also happen later. `--prune` cannot be combined with `--limit`.

	> [!WARNING]
	>
	> The supported sources may contain mature material. Follow their rules,
	> choose queries carefully, and store downloads responsibly.

	<br>

	<!-- ======================================================== -->
	## Run Tests
	<!-- ======================================================== -->

	Run focused tests first:

	```bash
	python -m pytest tests
	```

	Run the quality tools before finishing a code change:

	```bash
	ruff format .
	ruff check .
	pyright
	python -m pytest
	```

	> [!NOTE]
	> Related: use [Development: verification](Development.md#verification) for
	> the maintainer checklist before a commit.

	<br>

	<!-- ======================================================== -->
	## Inspect The Project
	<!-- ======================================================== -->

	Use these commands when you need to understand the current shape:

	```bash
	rg --files
	git status --short
	kscrape --help
	kscrape config show --json
	python -m kscrape --help
	```

	When a command fails, copy the exact command, current directory, exit code, and
	stderr into the issue or debugging note.

	> [!NOTE]
	> Related links:
	> - Use [project paths](References.md#project-paths) for the main source and docs locations.
	> - Use [system model](Explanations.md#system-model) for the package and tooling boundaries.

	<br>

	<!-- ======================================================== -->
	## Prepare A Release
	<!-- ======================================================== -->

	Follow [Development: releases](Development.md#releases) before running these
	commands:

	```bash
	just release-check
	just release-prepare patch
	just release-push vMAJOR.MINOR.PATCH
	```

	The final command atomically pushes the branch and tag to the configured
	Hugging Face remote. It does not create a GitHub Release or publish a PyPI
	package.

	<br>

	# 4. Troubleshooting

	<!-- ======================================================== -->
	## Environment Problems
	<!-- ======================================================== -->

	Check the active Python and environment first:

	```bash
	python --version
	python -c "import sys; print(sys.executable)"
	python -c "import kscrape; print(kscrape.__file__)"
	```

	If the package imports from an unexpected location, reinstall it from the
	project root.

	> [!NOTE]
	> Related: use [failure model](Explanations.md#failure-model) for the normal
	> order of checks when a command behaves differently across machines.

	<br>

	<!-- ======================================================== -->
	## Command Problems
	<!-- ======================================================== -->

	When a `just` recipe fails:

	1. Run `just --list`.
	2. Run the underlying command manually.
	3. Check whether the virtual environment is active.
	4. Check whether the command exists in `.venv/bin`.

	```bash
	just --list
	ls .venv/bin
	```

	For a network command, confirm that `KSCRAPE_USER_AGENT` is set or pass
	`--user-agent`. Check that optional credentials are supplied as complete
	username and API-key pairs.

	> [!NOTE]
	> Related: use [command reference](References.md#command-reference) for the
	> expected commands and their owners.