Buckets:
| # Saved Models | |
| ## Overview | |
| This directory stores trained models and their checkpoints. | |
| ## Model Storage Structure | |
| ``` | |
| models/ | |
| ├── embeddings/ | |
| │ ├── transe_model_20250108.pt | |
| │ ├── distmult_model_20250108.pt | |
| │ └── complex_model_20250108.pt | |
| ├── reasoning/ | |
| │ └── reasoner_checkpoint.pt | |
| └── checkpoints/ | |
| └── training_checkpoint_epoch_50.pt | |
| ``` | |
| ## Saving Models | |
| ### Save Embedding Model | |
| ```python | |
| from src.models.embeddings import TransE | |
| import torch | |
| # Train model | |
| model = TransE(embedding_dim=100) | |
| # ... training code ... | |
| # Save model | |
| torch.save(model.state_dict(), 'models/embeddings/transe_model.pt') | |
| # Save full model with metadata | |
| torch.save({ | |
| 'model_state_dict': model.state_dict(), | |
| 'embedding_dim': 100, | |
| 'num_entities': 1000, | |
| 'num_relations': 50, | |
| 'training_config': {...} | |
| }, 'models/embeddings/transe_full.pt') | |
| ``` | |
| ## Loading Models | |
| ### Load Embedding Model | |
| ```python | |
| import torch | |
| from src.models.embeddings import TransE | |
| # Load model | |
| model = TransE(embedding_dim=100) | |
| model.load_state_dict(torch.load('models/embeddings/transe_model.pt')) | |
| model.eval() | |
| # Load full model with metadata | |
| checkpoint = torch.load('models/embeddings/transe_full.pt') | |
| model = TransE( | |
| embedding_dim=checkpoint['embedding_dim'], | |
| num_entities=checkpoint['num_entities'], | |
| num_relations=checkpoint['num_relations'] | |
| ) | |
| model.load_state_dict(checkpoint['model_state_dict']) | |
| ``` | |
| ## Model Versioning | |
| Use semantic versioning for models: | |
| ``` | |
| model_name_v1.0.0.pt | |
| model_name_v1.1.0.pt | |
| model_name_v2.0.0.pt | |
| ``` | |
| ## Model Metadata | |
| Each model should have an accompanying JSON file with metadata: | |
| ```json | |
| { | |
| "model_name": "TransE", | |
| "version": "1.0.0", | |
| "created_at": "2025-01-08T12:00:00Z", | |
| "training_config": { | |
| "embedding_dim": 100, | |
| "learning_rate": 0.001, | |
| "num_epochs": 100, | |
| "batch_size": 32 | |
| }, | |
| "performance": { | |
| "hits@1": 0.45, | |
| "hits@3": 0.67, | |
| "hits@10": 0.82, | |
| "mean_rank": 25.3 | |
| }, | |
| "dataset": { | |
| "num_entities": 1000, | |
| "num_relations": 50, | |
| "num_triples": 5000 | |
| } | |
| } | |
| ``` | |
| ## Best Practices | |
| 1. **Version Control**: Always version your models | |
| 2. **Metadata**: Include comprehensive metadata | |
| 3. **Checkpoints**: Save checkpoints during training | |
| 4. **Compression**: Compress large models | |
| 5. **Documentation**: Document model architecture and usage | |
| ## Model Registry | |
| Keep a registry of all models: | |
| ```python | |
| # models/registry.json | |
| { | |
| "models": [ | |
| { | |
| "id": "transe_v1", | |
| "path": "embeddings/transe_model_v1.0.0.pt", | |
| "description": "TransE model trained on scientific KG", | |
| "status": "production" | |
| }, | |
| { | |
| "id": "distmult_v1", | |
| "path": "embeddings/distmult_model_v1.0.0.pt", | |
| "description": "DistMult model for link prediction", | |
| "status": "experimental" | |
| } | |
| ] | |
| } | |
| ``` | |
| ## Cleanup | |
| Remove old or unused models: | |
| ```bash | |
| # Remove models older than 30 days | |
| find models/ -name "*.pt" -mtime +30 -delete | |
| ``` | |
| --- | |
| **Note**: Large models (>100MB) should not be committed to version control. Use Git LFS or cloud storage instead. | |
Xet Storage Details
- Size:
- 3.11 kB
- Xet hash:
- c023669fc06453d544b4a74566f990bfb53b519b10fd035e0414425611e79357
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.