rewriting codebase (#7)

916755e 3 months ago

35.2 kB

Task 16: Documentation, Examples, and Migration Closure

Complete the ROOT-GNN rewrite by making GNN4Colliders understandable, reproducible, and usable without relying on knowledge of the legacy repository.

This task should not introduce new scientific functionality.

It should consolidate the architecture, document the supported workflows, provide end-to-end examples, close migration gaps, and clearly separate:

supported behavior
intentional redesigns
legacy-only behavior
deferred work

This task builds on:

Task 6: ROOT/Awkward ingestion
Task 7: metadata-aware caching, splits, batching
Task 8: ROOT-GNN model and transfer/fine-tuning
Task 9: classification tasks, losses, metrics
Task 10: training lifecycle
Task 11: checkpointing and resume
Task 12: evaluation, prediction, and outputs
Task 13: Hydra configuration and CLI
Task 14: DDP, Perlmutter, and Slurm execution
Task 15: profiling and performance optimization

Before making changes, read:

AGENTS.md
README.md
docs/architecture.md
docs/migration.md
docs/performance.md

if docs/performance.md exists.

Then inspect:

configs/
benchmarks/
scripts/
src/gnn4colliders/
tests/
legacy/

Also inspect the active example/reference configurations introduced in Tasks 13–15.

Do not assume documentation is correct merely because it already exists. Verify examples against the actual current APIs and CLI.

Goal

Finish the ROOT-GNN rewrite as a coherent v1-quality workflow.

A new user should be able to understand and execute:

install
    ↓
prepare data
    ↓
pretrain ROOT-GNN
    ↓
resume training
    ↓
fine-tune from pretrained backbone
    ↓
evaluate
    ↓
predict
    ↓
run on Perlmutter

without reading legacy implementation code.

The final documentation should explain the architecture clearly enough that a future ROOT-Transformer implementation can reuse the shared infrastructure correctly.

1. Treat this as migration closure

The primary purpose of this task is not to add more features.

It is to answer:

What is implemented?
What is parity-tested?
What changed intentionally?
What remains legacy-only?
What is deferred?
How does a user run the new stack?

Do not expand scope into new model families or major infrastructure.

2. Audit documentation against production code

Before editing documentation, verify the current behavior of:

CLI commands
Hydra config names
model constructors
task names
checkpoint workflows
inference formats
distributed launch
benchmark commands

Update docs to match code.

Do not document aspirational commands that do not work.

3. README should become the primary entry point

Rewrite or substantially improve:

README.md

so a new user can understand the project quickly.

Prefer a structure such as:

GNN4Colliders
Overview
Installation
Quick start
Core concepts
Pretraining
Fine-tuning
Evaluation
Prediction
Checkpoint/resume
Perlmutter
Configuration
Development
Architecture
Migration status

Keep the README useful but not encyclopedic.

Link to detailed docs for deeper material.

4. Project description

Clearly describe GNN4Colliders as a collider-ML toolkit rather than a permanently GNN-only package.

For example, communicate the conceptual architecture:

ROOT
    ↓
EventSample
    ↓
shared collider features
    ├── GraphSample -> ROOT-GNN
    └── future SequenceSample -> ROOT-Transformer

Do not imply graph-specific infrastructure is the universal shared representation.

5. Explain package naming

Clarify:

repository: GNN4Colliders
Python package: gnn4colliders
model family: ROOT-GNN
config/module identifier: root_gnn

Use consistent naming throughout all docs.

Do not alternate between legacy names casually.

6. Installation documentation

Document the canonical development installation.

Use the actual current project setup.

For example, if still correct:

uv sync --extra root-gnn

Document:

supported Python version
core install
ROOT-GNN optional dependencies
development/test dependencies

Do not document dependency combinations that are no longer validated.

7. Python version

Ensure all documentation consistently states the canonical supported Python version/range from pyproject.toml.

Do not leave stale references to legacy Python 3.8 or earlier Task 2 experiments.

8. DGL / PyTorch environment

Document the validated ROOT-GNN environment clearly.

If DGL installation still requires a nonstandard wheel source or environment-specific setup, explain it accurately.

Do not include obsolete installation workarounds from earlier migration debugging.

Keep environment-specific caveats concise.

9. Quick-start workflow

Add a minimal quick start that demonstrates the current stable flow.

Conceptually:

uv sync --extra root-gnn

uv run gnn4colliders prepare ...

uv run gnn4colliders train \
    task=pretraining_multiclass \
    model=root_gnn/edge_network

uv run gnn4colliders evaluate \
    inference.checkpoint=/path/to/checkpoint.pt

Use the exact current CLI syntax.

The example should be executable with documented sample/tiny data where feasible.

10. Document `prepare`

Explain:

ROOT input
    ->
EventSample
    ->
features
    ->
graphs
    ->
cache

Document:

what is cached
where cache is written
how cache compatibility is checked
what invalidates a cache

Do not expose implementation details that users do not need.

11. Document metadata semantics

Clearly explain that the new architecture does not use public positional:

tracking[:, N]

fields.

Document named metadata concepts such as:

fold
weight
sample_id

and any active additional fields.

Explain that legacy mappings are handled only at compatibility boundaries.

12. Document `EventSample`

Describe the architecture-neutral event representation.

Conceptually:

EventSample
    objects
    label
    global_features
    metadata

Use the actual current API.

Make clear this object is shared infrastructure and is not ROOT-GNN-specific.

13. Document graph construction

Explain ROOT-GNN graph semantics:

directed fully connected graph
no self-loops
N * (N - 1) edges

Document active node feature schema:

pt
eta
phi
energy
btag
charge
node_type

and edge features:

deta
dphi
dR

Do not reproduce unnecessary implementation code in the README.

Link to architecture docs for details.

14. Document feature semantics

Document scientifically relevant compatibility behavior such as:

energy calculation
phi wrapping
feature scaling
object ordering

where needed for reproducibility.

Clearly distinguish user-visible scientific contracts from internal implementation details.

15. Document ROOT-GNN

Describe:

EdgeNetwork

at a conceptual level:

node/edge/global encoders
    ->
iterative edge/node/global message passing
    ->
graph representation
    ->
classifier
    ->
raw logits

State explicitly that the model returns raw logits rather than sigmoid/softmax outputs.

16. Document transfer/fine-tuning

This is a first-class workflow and must be documented clearly.

Explain:

pretrained multiclass ROOT-GNN
    ↓
load reusable backbone
    ↓
replace source classifier
    ↓
binary/target-task classifier
    ↓
fine-tune

Document:

freeze_backbone=true
freeze_backbone=false

and their meaning.

Do not describe transfer learning as resume training.

17. Distinguish resume vs transfer learning

Create a prominent explanation.

Resume:
    continue the same training run/task
    restore optimizer/scheduler/trainer state

Transfer:
    reuse pretrained model/backbone
    create a new task/head
    create a new optimizer

This distinction should appear in both README and checkpoint documentation.

18. Pretraining example

Provide one complete example for multiclass pretraining.

Use the actual current config names.

Conceptually:

uv run gnn4colliders train \
    model=root_gnn/edge_network \
    task=pretraining_multiclass

Include the important overrides a user is likely to need:

data path
output path
batch size
epochs

Keep the example readable.

19. Fine-tuning example

Provide one complete transfer-learning example.

Conceptually:

uv run gnn4colliders train \
    model=root_gnn/fine_tuned_edge_network \
    task=binary_classification \
    checkpoint.pretrained=/path/to/pretrained.pt

Use the actual final configuration layout.

Also show how to choose:

freeze_backbone=true

or:

freeze_backbone=false

20. Resume example

Provide a separate example:

uv run gnn4colliders train \
    checkpoint.resume=/path/to/checkpoint.pt

Use the actual CLI.

Explain exactly what state is restored.

21. Evaluation example

Document:

uv run gnn4colliders evaluate ...

Explain:

which split is evaluated
how metrics are computed
where event weights are used
how full-split ROC AUC is handled

Do not duplicate Task 9 formulas unnecessarily in README.

Link to more detailed docs.

22. Prediction example

Document:

uv run gnn4colliders predict ...

Cover:

NPZ output
ROOT output if supported
labeled vs unlabeled datasets
sample_id preservation

Explain that inference output uses named fields rather than legacy positional tracking arrays.

23. Output schema documentation

Document the primary NPZ output schema.

For example, if implemented:

sample_id
logits
scores
predictions
labels
weight
fold

Use only actual current fields.

Clearly distinguish required and optional fields.

24. ROOT output documentation

If ROOT score writing exists, document:

tree handling
score branch naming
selection_pass behavior
event alignment

Do not leave this behavior discoverable only through tests.

If ROOT writing was intentionally deferred, say so clearly instead.

25. Checkpoint documentation

Create or improve a detailed checkpoint section.

Document:

new checkpoint schema
schema version
model metadata
task metadata
trainer state
optimizer state
scheduler state
early stopping

Explain that checkpoints are self-describing to the supported extent.

26. Legacy checkpoint compatibility

Document exactly what historical checkpoints are supported.

For example:

model-weight loading: supported
pretrained-backbone loading: supported
full legacy optimizer resume: supported / unsupported
module. prefix normalization: supported
_orig_mod. normalization: supported

Use actual implemented behavior.

Do not claim compatibility that has not been tested.

27. Configuration documentation

Add a dedicated configuration guide if helpful, for example:

docs/configuration.md

Explain Hydra config groups:

data
model
task
trainer
checkpoint
inference
environment
distributed

Use actual current groups.

28. Explain configuration philosophy

Document the rule:

new experiment -> configuration/YAML
new algorithm or behavior -> Python

Explain that config files use semantic names rather than arbitrary module/class import paths.

This is an important departure from the legacy system.

29. Hydra override examples

Provide concise examples such as:

uv run gnn4colliders train \
    trainer.max_epochs=50 \
    data.batch_size=64

and relevant model/task/environment overrides.

Avoid an enormous override catalog.

30. Resolved configuration

Document that runs persist their fully resolved config.

Explain where it is written.

This should be presented as part of experiment reproducibility.

31. Environment profiles

Document:

environment=local
environment=perlmutter

or the actual final names.

Explain what environment profiles control and what they should not contain.

Do not encourage users to put scientific model settings in environment configs.

32. Perlmutter guide

Add or improve:

docs/perlmutter.md

if a dedicated guide is useful.

Cover:

environment setup
uv environment
single-GPU run
multi-GPU run
Slurm scripts
output paths
common debugging checks

Use only validated current commands.

33. Slurm examples

Document the scripts under:

scripts/slurm/

Explain which script to use for:

single GPU
single-node multi-GPU
multi-node
evaluation
benchmarking

if those scripts exist.

Do not duplicate the entire script contents in README.

34. DDP semantics

Document distributed behavior important to users:

batch_size is per GPU/process
effective global batch size = batch_size * world_size
num_workers is per process
only rank 0 writes shared artifacts

Also explain global metric aggregation at a conceptual level.

35. Reproducibility documentation

Document the current reproducibility model.

Cover:

base seed
DataLoader seed
sampler set_epoch behavior
rank-specific seed derivation
model initialization
CPU determinism
GPU nondeterminism caveats

Do not promise bitwise GPU determinism unless actually validated.

36. Performance guide

Link to:

docs/performance.md

or:

benchmarks/README.md

Document validated recommendations for:

num_workers
pin_memory
persistent_workers
precision
DDP

where Task 15 produced evidence.

Do not turn hardware-specific benchmark results into universal defaults.

37. Architecture documentation

Review and consolidate:

docs/architecture.md

It should clearly describe the current architecture.

At minimum cover:

ROOT/Awkward I/O
EventSample
shared features
GraphSample
cache
GraphBatch
ROOT-GNN
task layer
trainer
checkpointing
inference
configuration/CLI
distributed execution

Show responsibility boundaries.

38. Architecture diagram

Add a concise textual/Markdown diagram.

For example:

ROOT
  ↓
EventSample
  ↓
shared collider features
  ↓
GraphSample
  ↓
cache / split / batch
  ↓
GraphBatch
  ↓
ROOT-GNN
  ↓
logits
  ↓
Task
  ├── loss
  ├── predictions
  └── metrics
  ↓
Trainer / Predictor

Also show future extensibility:

shared features
    ├── GraphSample -> ROOT-GNN
    └── SequenceSample -> ROOT-Transformer

Do not make ROOT-Transformer appear implemented if it is not.

39. Responsibility table

Consider adding a concise table in architecture docs.

For example:

Layer               Responsibility
data                ROOT/Awkward ingestion and samples
features            shared collider feature construction
graphs              graph representation/topology
models/root_gnn     ROOT-GNN architecture
tasks               losses/predictions/metrics
training            lifecycle/checkpoints
inference           prediction/output
distributed         DDP execution utilities
cli/config          user-facing composition

Use current code structure.

40. Migration document

Turn:

docs/migration.md

into a clear migration-status document.

For each major legacy area, mark:

migrated
compatibility adapter
intentionally redesigned
deferred
not supported

Avoid vague status such as "mostly done."

41. Legacy migration matrix

Add a matrix for active legacy behavior.

Include areas such as:

ROOT ingestion
node features
edge construction
cache
folds
weights
batching
padding
Edge_Network
fine-tuning
loss
metrics
training
early stopping
checkpoints
inference
ROOT outputs
DDP
Slurm
ONNX

For each, document current status.

42. Intentional redesigns

Explicitly document important departures from legacy architecture.

At minimum include:

tracking[:, 0] -> metadata.fold
tracking[:, 1] -> metadata.weight
tracking arrays -> named metadata
dynamic YAML module/class imports -> semantic Hydra configs
large training script -> layered Trainer/Task/Checkpoint APIs
persistent graph mutation -> scoped temporary graph state
model constructor global RNG mutation -> explicit external seeding

Only list changes actually implemented.

43. Preserve behavior vs preserve bugs

Migration docs should explain that compatibility targets externally relevant scientific behavior.

Do not imply every legacy implementation quirk is intentionally retained.

Classify known legacy quirks as appropriate:

compatibility requirement
observed behavior
intentional cleanup
known legacy bug
deferred investigation

44. Outstanding ambiguities

Review earlier characterization work and document unresolved issues.

Examples may include:

negative event weights
rare empty graphs
padding edge cases
legacy validation/test naming
historical checkpoint variants

Do not silently erase unresolved ambiguities from the migration record.

If later tasks resolved them, update their status.

45. Example directory

Add a small examples area if useful:

examples/
    README.md
    pretraining/
    finetuning/
    inference/

Prefer configuration examples over duplicate Python scripts.

Do not create a parallel application framework under examples/.

46. Example configs

Provide complete runnable or nearly runnable examples for:

multiclass pretraining
binary fine-tuning
resume
evaluation
prediction

Use placeholders only for values that necessarily depend on local data paths.

Clearly mark placeholders.

47. Tiny smoke workflow

If the repository has tiny ROOT fixtures or can safely provide a tiny generated fixture workflow, document a smoke-test path.

The goal is something like:

prepare tiny data
train one epoch
save checkpoint
evaluate
predict

that runs quickly on CPU.

Prefer reusing test fixtures/tooling rather than committing large binary datasets.

48. Do not misuse test-only fixtures

If tests generate their own ROOT fixtures dynamically, do not tell users to depend on internal pytest-only APIs.

If useful, extract a small supported example-data generator into an appropriate script.

Only do this if it materially improves onboarding.

49. Developer guide

Add or improve a development section.

Document:

uv sync --extra root-gnn
uv run pytest
uv run ruff check .
uv run ruff format --check .

and relevant benchmark commands.

Explain the repository rule:

one logical task -> one coherent commit

if this is an established project convention.

50. Testing guide

Document the purpose of:

tests/unit/
tests/integration/
tests/parity/

Explain that parity tests protect migration behavior.

Do not list every test file.

51. Parity philosophy

Document the rule:

legacy executable behavior is characterized first
new implementation is compared against deterministic fixtures

Explain that parity tests are especially important for:

features
graph topology
model forward
loss semantics
metrics
checkpoint loading
inference outputs

52. Benchmark guide

Document how to run Task 15 benchmarks.

For example:

uv run python benchmarks/benchmark_preprocessing.py
uv run python benchmarks/benchmark_training.py --device cuda

Use actual working commands.

Explain that benchmark results depend on hardware and configuration.

53. CLI help audit

Run:

uv run gnn4colliders --help
uv run gnn4colliders prepare --help
uv run gnn4colliders train --help
uv run gnn4colliders evaluate --help
uv run gnn4colliders predict --help

Review for confusing/stale names.

Minor help-text improvements are in scope.

Do not redesign the CLI.

54. Config naming audit

Review config names for consistency.

Remove or rename obviously stale/experimental config names only if doing so does not unnecessarily break established use.

Prefer:

pretraining_multiclass
binary_classification
fine_tuning

or the actual established semantic naming.

Document aliases if compatibility requires them.

55. Dead documentation cleanup

Remove stale documentation that refers to:

legacy positional tracking APIs
old script entry points
dynamic import YAML
obsolete dependency versions
abandoned architecture plans

unless explicitly preserved in migration-history sections.

Do not leave contradictory instructions in different docs.

56. Docstrings

Add or improve docstrings for important public APIs where absent.

Prioritize:

EventSample
GraphSample
GraphBatch
EdgeNetwork
FineTunedEdgeNetwork
classification tasks
Trainer
CheckpointManager
Predictor
DistributedContext

Keep docstrings concise.

Do not write essays inside source files.

57. Type/API clarity

While reviewing docs, minor fixes to public annotations or obvious naming inconsistencies are acceptable.

Do not refactor major APIs in this documentation task.

If a serious design flaw is discovered, document it as follow-up rather than destabilizing the finished migration.

58. Public API examples

Ensure documented imports work.

For example:

from gnn4colliders.models.root_gnn import EdgeNetwork
from gnn4colliders.tasks import BinaryClassificationTask
from gnn4colliders.training import Trainer
from gnn4colliders.inference import Predictor

Use only exports that actually exist.

Do not teach users to import private modules.

59. Python API example

Add one minimal Python example for users who do not want the CLI.

Conceptually:

model = ...
task = ...
trainer = ...

Keep it short.

The primary supported user workflow can remain CLI/Hydra.

60. CLI remains primary experiment interface

Document the intended split:

CLI/Hydra:
    normal experiments

Python APIs:
    libraries, testing, custom workflows

Do not force all users into notebooks or scripts.

61. Notebooks

Do not create a large notebook suite in this task.

If existing notebooks are stale, either:

update them
mark them exploratory
remove references to them

as appropriate.

Notebooks should not be the canonical production workflow.

62. Legacy directory documentation

Explain the purpose of:

legacy/

Clearly state that it is a frozen behavioral reference and not production implementation.

Do not tell users to add legacy/ to PYTHONPATH.

63. Legacy old rewrite attempt

If:

legacy/physicsnemo/

or an equivalent abandoned rewrite remains, document clearly that it is not the behavioral target.

Do not accidentally present it as a supported backend.

64. Avoid deleting legacy in Task 16

Do not delete the legacy implementation yet.

It remains useful for:

parity tests
historical reference
checkpoint compatibility investigation

Legacy cleanup/deletion belongs to a later dedicated task.

65. Create a migration completion checklist

At the end of docs/migration.md, include a concise checklist.

For example:

[x] active data path
[x] graph construction
[x] EdgeNetwork
[x] transfer learning
[x] task losses/metrics
[x] trainer
[x] checkpoints
[x] inference
[x] CLI
[x] DDP
[x] profiling
[ ] ONNX export
[ ] ROOT-Transformer

Use actual status.

Do not mark something complete solely because there is placeholder code.

66. Define ROOT-GNN v1 completion

Add a concise statement describing what constitutes the completed ROOT-GNN rewrite.

For example:

ROOT-GNN v1 is complete when the supported new stack can:
- prepare active datasets
- reproduce validated legacy feature/graph/model/task behavior
- train from scratch
- fine-tune pretrained backbones
- resume checkpoints
- evaluate/predict
- run single-process and DDP on Perlmutter

Tailor this to actual implemented functionality.

67. Known limitations

Add a clear known-limitations section.

Possible examples, if still true:

ONNX export deferred
distributed ROOT writing limited
full historical optimizer-resume compatibility incomplete
very large inference uses in-memory result accumulation
ROOT-Transformer not yet implemented

Do not hide limitations.

68. Future roadmap

Keep the roadmap short.

After ROOT-GNN v1, likely items include:

ONNX/export
legacy cleanup
packaging/release hardening
ROOT-Transformer
large-scale inference improvements

Do not turn README into a speculative roadmap document.

69. Validate every command in documentation

For every command intended to be runnable, execute it or at least its safe smoke equivalent.

Especially validate:

installation
--help
config composition
tiny prepare
tiny train
tiny fine-tune
tiny evaluate
tiny predict
benchmark command syntax

Do not ship copied commands that have not been checked.

70. Documentation tests

If helpful, add lightweight tests to protect key docs/API snippets.

For example:

CLI help smoke tests
example config composition tests
public import tests

Do not introduce a heavyweight documentation-testing framework unless already used.

71. Config smoke tests

Validate all reference configs compose successfully.

For each committed reference configuration, ensure Hydra can resolve it without performing expensive training.

Fail on stale field names.

72. Example smoke tests

If example configs/scripts are added, add small tests ensuring they do not rot.

Keep these tests fast.

Do not require production ROOT files.

73. Documentation consistency

Use consistent terminology everywhere.

Prefer:

training
validation
test

rather than reproducing confusing legacy naming.

Prefer:

metadata.weight
metadata.fold

rather than tracking columns.

Prefer:

fine-tuning

or one consistent spelling throughout.

74. Scientific terminology

Preserve established collider terminology accurately.

Do not rename scientifically meaningful fields merely for stylistic reasons.

Where names have historical meanings, explain them.

75. Code comments

Do not compensate for unclear docs by adding excessive inline comments everywhere.

Use source comments only for non-obvious implementation constraints.

Documentation belongs primarily in README/docs and concise public docstrings.

76. No new major dependencies

This task should add no major runtime dependency.

Documentation tooling dependencies are also unnecessary unless the project already has a docs build system.

Plain Markdown is sufficient.

77. Do not build a documentation website

Do not introduce:

Sphinx
MkDocs
Docusaurus

unless already present and clearly desired.

The task is content completion, not documentation-platform migration.

78. Review `.gitignore`

Ensure generated artifacts from documented workflows are not accidentally committed.

Check handling of:

outputs/
checkpoints/
predictions/
profiles/
benchmark-results/
__pycache__/
*.egg-info/

Only change .gitignore where appropriate.

Do not ignore source example configs or benchmark scripts.

79. Repository cleanliness

Remove accidental generated development artifacts if they are tracked or unignored, such as:

__pycache__/
src/gnn4colliders.egg-info/
temporary outputs
profiling traces

Do not delete user data.

Only clean repository-generated artifacts.

80. Architecture-neutral audit

Use this task to verify the shared design still supports a future non-graph model family.

Review especially:

data/
features/
training/
tasks/
inference/
distributed/
config/

for unnecessary ROOT-GNN-specific naming or assumptions.

Do not perform major refactors.

Document any remaining coupling as follow-up work.

81. ROOT-Transformer readiness note

Add a concise architecture note for future development.

Conceptually:

ROOT
    ->
EventSample
    ->
shared feature representation
       ├── graph adapter -> ROOT-GNN
       └── future sequence adapter -> ROOT-Transformer

Identify the intended extension points.

Do not implement the transformer here.

82. Migration debt audit

Search for remaining new-code references to legacy concepts such as:

tracking
tracking_info
sys.path
legacy module/class dynamic import
hardcoded ROOT-GNN assumptions in shared code

Classify each result:

required compatibility
test-only
documentation/history
migration debt

Fix only obvious low-risk leftovers.

Document the rest.

83. TODO/FIXME audit

Search:

TODO
FIXME
HACK
XXX

through production code.

For each:

resolve it if trivial and in scope
convert it into a clear documented limitation
remove stale notes
leave substantial future work for later tasks

Do not turn Task 16 into an unbounded cleanup sprint.

84. Warning/error message audit

Review major user-facing failures.

Examples:

invalid config
missing input files
incompatible cache
incompatible checkpoint
wrong model/task combination
unsupported output format
distributed misconfiguration

Improve obviously cryptic messages where low-risk.

Do not redesign exception architecture.

85. Final end-to-end smoke workflow

Create one documented validation workflow that exercises the new system from start to finish using tiny inputs.

Conceptually:

prepare
    ↓
train multiclass model briefly
    ↓
save checkpoint
    ↓
load pretrained backbone
    ↓
fine-tune binary model briefly
    ↓
evaluate
    ↓
predict NPZ

This does not need scientific convergence.

Its purpose is integration confidence.

86. Do not use legacy code in the smoke workflow

The final new-stack smoke workflow must run without importing production logic from:

legacy/

Parity tests may continue to use legacy.

The actual user workflow must not.

87. Optional end-to-end script

If useful, add something like:

scripts/dev/smoke_end_to_end.sh

or:

scripts/dev/smoke_end_to_end.py

Keep it thin and built entirely on the public CLI.

Do not duplicate application logic.

88. Smoke script requirements

A smoke script should:

use temporary/tiny data
avoid hardcoded user paths
fail on command errors
be readable
not require Slurm
not require GPU

Do not make it part of normal package runtime.

89. Release-readiness checklist

Add a concise internal checklist to docs if useful.

For example:

tests pass
parity tests pass
CLI examples validated
reference configs compose
README current
migration matrix current
no generated artifacts tracked

Do not introduce formal release tooling yet.

90. Testing

Run:

uv run pytest

Run focused parity:

uv run pytest tests/parity -v

Run distributed tests:

uv run pytest tests/unit/distributed -v
uv run pytest tests/integration -k distributed -v

Run lint/format:

uv run ruff check .
uv run ruff format --check .

Validate public imports.

Validate CLI:

uv run gnn4colliders --help
uv run gnn4colliders prepare --help
uv run gnn4colliders train --help
uv run gnn4colliders evaluate --help
uv run gnn4colliders predict --help

Validate all committed reference configs compose.

Validate the documented tiny end-to-end workflow.

If Slurm scripts are present:

bash -n scripts/slurm/*.sh

or the appropriate individual scripts.

Inspect:

git status
git diff

Verify:

no scientific behavior changed unexpectedly
no production imports from legacy were introduced
docs match current CLI/config APIs
no stale tracking-array usage is presented as public API
no generated artifacts are tracked
no major new dependency was added
no unrelated refactor is included

Completion criteria

Task 16 is complete when:

README accurately describes the current project.
Installation instructions are validated.
Project/package/model naming is consistent.
The quick-start workflow is documented.
Data preparation is documented.
EventSample and named metadata concepts are documented.
ROOT-GNN graph semantics are documented.
ROOT-GNN model behavior is documented.
Transfer/fine-tuning is documented as a first-class workflow.
Resume and transfer learning are clearly distinguished.
Pretraining example is validated.
Fine-tuning example is validated.
Resume example is validated.
Evaluation example is validated.
Prediction example is validated.
Checkpoint format/workflows are documented.
Supported legacy checkpoint compatibility is documented accurately.
Hydra configuration groups and overrides are documented.
Environment/Perlmutter execution is documented.
DDP batch-size and worker semantics are documented.
Reproducibility behavior is documented accurately.
Performance guidance from Task 15 is documented.
docs/architecture.md reflects the final implementation.
docs/migration.md contains a clear status matrix.
Intentional redesigns from legacy are documented.
Remaining ambiguities/limitations are explicit.
Future ROOT-Transformer extension points are documented without implementing it.
Reference configs compose successfully.
Public imports shown in docs work.
CLI help commands work.
Tiny end-to-end new-stack smoke workflow succeeds.
Full unit/integration/parity suite passes.
Distributed tests still pass.
Lint/format checks pass.
Legacy code remains frozen.
No major new feature or dependency has been introduced.

Completion report

Report:

files created
files modified
README structure
installation instructions finalized
quick-start workflow
pretraining example
fine-tuning example
resume example
evaluation example
prediction example
output schema documentation
checkpoint documentation
Hydra/configuration documentation
Perlmutter/DDP documentation
reproducibility documentation
performance documentation
architecture-document updates
migration matrix summary
intentional legacy redesigns documented
unresolved migration items
architecture-neutrality audit findings
ROOT-Transformer extension points
stale/dead documentation removed
generated repository artifacts cleaned
TODO/FIXME audit results
public import validation
CLI help validation
reference config validation
end-to-end smoke-test results
full test results
lint/format results
any follow-up tasks identified

After validation succeeds, create one Git commit containing only Task 16 changes.

Use:

docs: finalize ROOT-GNN documentation and migration

Before committing, inspect the final diff and ensure no unrelated implementation changes or generated artifacts are included.

Task 16: Documentation, Examples, and Migration Closure

Goal

1. Treat this as migration closure

2. Audit documentation against production code

3. README should become the primary entry point

4. Project description

5. Explain package naming

6. Installation documentation

7. Python version

8. DGL / PyTorch environment

9. Quick-start workflow

10. Document prepare

11. Document metadata semantics

12. Document EventSample

13. Document graph construction

14. Document feature semantics

15. Document ROOT-GNN

16. Document transfer/fine-tuning

17. Distinguish resume vs transfer learning

18. Pretraining example

19. Fine-tuning example

20. Resume example

21. Evaluation example

22. Prediction example

23. Output schema documentation

24. ROOT output documentation

25. Checkpoint documentation

26. Legacy checkpoint compatibility

27. Configuration documentation

28. Explain configuration philosophy

29. Hydra override examples

30. Resolved configuration

31. Environment profiles

32. Perlmutter guide

33. Slurm examples

34. DDP semantics

35. Reproducibility documentation

36. Performance guide

37. Architecture documentation

38. Architecture diagram

39. Responsibility table

40. Migration document

41. Legacy migration matrix

42. Intentional redesigns

43. Preserve behavior vs preserve bugs

44. Outstanding ambiguities

45. Example directory

46. Example configs

47. Tiny smoke workflow

48. Do not misuse test-only fixtures

49. Developer guide

50. Testing guide

51. Parity philosophy

52. Benchmark guide

53. CLI help audit

54. Config naming audit

55. Dead documentation cleanup

56. Docstrings

57. Type/API clarity

58. Public API examples

59. Python API example

60. CLI remains primary experiment interface

61. Notebooks

62. Legacy directory documentation

63. Legacy old rewrite attempt

64. Avoid deleting legacy in Task 16

65. Create a migration completion checklist

66. Define ROOT-GNN v1 completion

67. Known limitations

68. Future roadmap

69. Validate every command in documentation

70. Documentation tests

71. Config smoke tests

72. Example smoke tests

73. Documentation consistency

74. Scientific terminology

75. Code comments

76. No new major dependencies

77. Do not build a documentation website

78. Review .gitignore

10. Document `prepare`

12. Document `EventSample`

78. Review `.gitignore`