docs/DATASETS.md

Dataset Strategy

The legacy project uses an older IMDb-style metadata CSV. V2 keeps that as a fallback, but the main dataset path is now:

data/processed/movie_catalog.csv

The app automatically prefers that processed catalog when it exists. Otherwise, it falls back to:

data/legacy/main_data.csv

Legacy CSVs from the original project live under:

data/legacy/

TMDb Key

Use this exact name locally and on Hugging Face:

TMDB_API_KEY

Local .env example:

TMDB_API_KEY=your_tmdb_api_key_here
MOVIE_CATALOG_PATH=data/processed/movie_catalog.csv

On Hugging Face Spaces, add a repository secret named TMDB_API_KEY.

This project uses the official TMDb API. It does not scrape TMDb pages.

MovieLens Variants

Use small for fast development:

python scripts/prepare_movielens.py --variant small

Recommended first deployment pipeline:

make pipeline-demo

This combines MovieLens small ratings with a broad recent TMDb catalog.

Use 32m for the serious/offline project dataset:

python scripts/prepare_movielens.py --variant 32m

The 32m raw dataset is too large to treat casually inside the repo. Generate it offline, keep data/raw/ untracked, and commit or upload only compact processed artifacts that the Hugging Face Space needs.

Filtering Examples

Build a compact catalog with only movies that have at least 20 ratings:

python scripts/prepare_movielens.py --variant 32m --min-ratings 20

Build a tiny demo catalog:

python scripts/prepare_movielens.py --variant small --limit 2000

The generated files are:

data/processed/movie_catalog.csv
data/processed/catalog_metrics.csv

Recent Movie Coverage With TMDb

MovieLens gives us ratings. TMDb gives us current movie discovery. To add post-2019 catalog coverage:

python scripts/build_tmdb_catalog.py --min-year 2019 --pages 50

This writes:

data/processed/tmdb_recent_catalog.csv

This is not meant to be exhaustive. It pulls popular recent movies, which is better for a responsive demo than ingesting every low-vote or obscure title.

Country/origin metadata is enriched from TMDb details for a representative subset of the broad recent catalog. The front-page origin note should be read as a transparency estimate, not a census of every title.

For broader 2018-2026 coverage, run a per-year sweep:

make tmdb-broad-recent

That writes:

data/processed/tmdb_broad_recent_catalog.csv

Use the broad catalog when recall matters more than artifact size and curation quality.

To keep both MovieLens and TMDb rows, write MovieLens to a separate file and merge:

python scripts/prepare_movielens.py \
  --variant 32m \
  --min-ratings 20 \
  --catalog-output-path data/processed/movielens_32m_catalog.csv

python scripts/build_tmdb_catalog.py \
  --min-year 2019 \
  --pages 50

python scripts/merge_catalogs.py \
  --inputs data/processed/movielens_32m_catalog.csv data/processed/tmdb_recent_catalog.csv \
  --output-path data/processed/movie_catalog.csv

Collaborative Filtering Baseline

After preparing MovieLens, build item-item collaborative neighbors:

python scripts/train_item_cf.py \
  --ratings-path data/raw/ml-latest-small/ratings.csv \
  --catalog-path data/processed/movie_catalog.csv

For a larger 32m run, start with conservative filters:

python scripts/train_item_cf.py \
  --ratings-path data/raw/ml-32m/ratings.csv \
  --catalog-path data/processed/movie_catalog.csv \
  --min-item-likes 20 \
  --max-users 100000

The generated files are:

data/processed/item_neighbors.csv
data/processed/item_cf_metrics.csv
data/processed/item_cf_eval.csv
data/processed/semantic_index.npz
data/processed/ranking_eval.csv
data/processed/hybrid_weights.json

When item_neighbors.csv exists, the FastAPI app automatically blends collaborative recommendations with the content baseline.

Evaluate the baseline:

python scripts/evaluate_item_cf.py \
  --ratings-path data/raw/ml-latest-small/ratings.csv \
  --neighbors-path data/processed/item_neighbors.csv \
  --k 10

Build the semantic retrieval artifact:

python scripts/build_semantic_index.py \
  --catalog-path data/processed/movie_catalog.csv \
  --output-path data/processed/semantic_index.npz

Install requirements-ml.txt before this offline build. The default builder uses sentence-transformers/all-MiniLM-L6-v2; the live Hugging Face Space loads the resulting artifact directly and does not download or load an embedding model during startup.

Evaluate the four retrieval ablations and select hybrid weights only on validation data:

python scripts/evaluate_recommenders.py \
  --ratings-path data/raw/ml-latest-small/ratings.csv \
  --catalog-path data/processed/movie_catalog.csv \
  --semantic-path data/processed/semantic_index.npz

The evaluator reserves each user's two most recent liked movies for validation and test, then uses the immediately preceding liked movie as the recommendation seed so it matches the app's selected-movie flow. Collaborative interactions for both held-out movies are excluded from training, avoiding the leakage in an evaluation that scores a graph trained on all ratings.

Why TMDb Is Still Needed

MovieLens is ideal for ratings and collaborative filtering, but it is not a poster/cast/overview database. TMDb gives the app current presentation metadata and lets the UI stay polished.

The long-term target is:

MovieLens ratings -> collaborative filtering
MovieLens/TMDb metadata -> content features
TMDb API -> posters, cast, current movie details