Spaces:
Sleeping
Sleeping
Commit History
-
Cleaned up code, added multiseed training wrapper, PyTorch profiler training option, updated gradio demo, made changes to research paper to match new changes and new training results from adding new training techniques, architecture.md now explains all designs and decisions
OliverPerrin90a2698
-
Added many new improvements based on feedback from others
OliverPerrin0d858b5 -
Updated Research Paper, README, and old gradio about info, along with other docs.
OliverPerrin8f5fea2
-
Full training results & evaluation with BERTScore
OliverPerrinTraining: - Completed 7 epochs (~6 hours) with early stopping - Optimized config: batch=10, grad_accum=4, lr=3e-5, topic_weight=0.3 - Froze bottom 4 encoder layers for stable transfer learning Evaluation Results: - Summarization: ROUGE-1=0.3064, BERTScore F1=0.8300 - Topic Classification: 85.2% accuracy (7 classes) - Emotion Detection: F1=0.1987 (28 classes, multi-label) Paper (docs/paper.tex): - Added comprehensive experimental results section - Added training dynamics figures - Fixed Unicode characters for LaTeX compatibility - Updated methodology with actual datasets used New scripts: - scripts/evaluate.py: Comprehensive evaluation with BERTScore - Rebuilt discovery dataset with 1000 samples for demo1e95f87
-
Fix regression: restore balanced training settings, add technical manual filter
OliverPerrinTraining configs: - Restored LR: 3e-5 (medium), 2e-5 (full) - Restored weight decay: 0.01 - Restored label smoothing: 0.1 - Restored dropout: 0.1 - Restored task weights: emotion=1.5, topic=0.5 - Restored epochs: 5 (medium), 8 (full) - Restored early stopping patience: 3 Discovery dataset: - Added technical manual filter (pottery recipes, camera specs, etc.) - These non-narrative texts produce garbage summaries9710200 -
Major improvements: real titles, anti-overfitting, better demo UX
OliverPerrinTraining configs (medium.yaml, full.yaml): - Reduced epochs: 5→4, 8→6 (stop before overfitting) - Lower LR: 3e-5→2e-5, 2e-5→1.5e-5 - Increased weight decay: 0.01→0.02 - Increased label smoothing: 0.1→0.15 - Boost emotion weight: 1.5→2.0 (tends to underfit) - Reduce topic weight: 0.5→0.3 (overfits quickly) - Earlier early stopping: patience 3→2 - Increased dropout: 0.1→0.15 Data download (download_data.py): - BookSum: Extract real book titles and chapter names - arXiv: Extract paper titles from abstract first sentence - Gutenberg: Parse JSON metadata for real titles and authors Discovery dataset (build_discovery_dataset.py): - Use real titles from source data - Increased default sizes: 300 books, 200 papers, 100 literary - Better fallback for missing titles - Improved error handling Demo (demo_gradio.py): - Cleaner card format with prominent titles - Confidence badges (green/yellow/red) - Dynamic counts in header - Better table layout for topic/emotion0fe274c
-
Fixed compiling issue, added legnth penalty, and atttempting freezing encoder layers 0-5 to lower parameters and preserve T5's langauge understanding.
OliverPerrinbaf3026
-
Improve emotion task architecture and fix visualizations
OliverPerrinArchitecture improvements: - Added hidden_dim option to ClassificationHead for 2-layer MLP - Emotion head now uses 768→384→28 MLP for better multi-label capacity - Updated task weights: emotion=1.5, summarization=1.0, topic=0.8 Visualization fixes: - LR schedule now generates theoretical curve when no metrics logged - Added learning_rate logging to MLflow in trainer - Log warmup_steps and scheduler_type to MLflow params Gradio UI modernization: - New yellow/black research theme (lexitheme) - Removed emojis for cleaner academic presentation - Progress bar style emotion scores - Updated tab names and descriptions6a7a381 -
Clean up codebase and fix training bugs
OliverPerrin- Fixed total_loss tracking for validation (early stopping works now) - Lowered emotion threshold from 0.5 to 0.3 for multi-label - Optimized full.yaml config for faster training (50k samples cap) - Consolidated utils into core.py - Removed unused modules and scripts - Fixed Gutenberg download key format issue - Updated visualize_training default model name - Switch MLflow to SQLite backend1601799
-
Update LexiMind: improved training, model architecture, and evaluation
OliverPerrin076bc18
-
Expand datasets: GoEmotions (28 emotions), Yahoo Answers (10 topics), BookSum
OliverPerrin- Updated download_data.py with new dataset downloaders - Updated preprocess_data.py to handle JSONL format - Updated labels.json with 28 emotions and 10 topics - Updated datasets.yaml with new dataset configs - Updated full.yaml training config (warmup_steps: 1000) - Training results: 74% topic accuracy, ROUGE 0.2849becd3c -
Fix Pylance type errors, add inductor compilation support
OliverPerrin- Add cast() for registered buffer access (pe, cos, sin) in decoder.py, positional_encoding.py, and attention.py to satisfy Pylance type checking - Add cast() for compile_model return types in train.py - Add type: ignore for rouge_score import (no type stubs available) - Add safe_compile.py for torch.compile with inductor backend (default mode) - Add nan_debugger.py for debugging NaN/Inf during training - Update configs: batch_size=12 for medium/full, fix layer counts to match FLAN-T5-base - Add benchmark mode to train.py for speed testing without saving checkpoints - Suppress torch inductor warnings that interfere with tqdm progress bars All checks pass: ruff, mypy, Pylancecf79a6c -
Full training run, code cleanup, mypy/ruff fixes
OliverPerrin- Completed 3-epoch full training run (21 hours) - Emotion F1: 94.6%, Topic Accuracy: 94.2%, ROUGE-like: 0.36 - Code simplification and standardized docstrings across all modules - Fixed all mypy type errors (47 files pass) - Fixed all ruff linting errors and reformatted code - All 75 tests passing - Added GPU optimizations: TF32, Flash Attention, memory-efficient SDP - Training configs optimized for RTX 4070 12GB - tqdm progress bars for training and evaluationa47e5cf
-
feat: Add FLAN-T5 compatibility with relative position bias
Major changes: - Implement T5RelativePositionBias for encoder/decoder self-attention - T5 uses unscaled attention (no sqrt(d_k) scaling) - Add float32 softmax path for numerical stability - Switch to aot_eager compile backend (inductor causes NaN in decoder backward) - Add gated-gelu activation support for T5 FFN - Fix vocab size handling (32100 vs 32128) - Update model configs for T5-base architecture - Add dev/medium training configs for faster iteration - Optimize training for ~4 min dev runs on RTX 4070 The model now correctly loads FLAN-T5-base weights and generates coherent summaries with proper encoder-decoder architecture.486475d
-
Refactor: Consolidate dependencies, improve testing, and add CI/CD
- Consolidated project dependencies into pyproject.toml. - Removed requirements.txt, requirements-dev.txt, and setup.py. - Removed scripts/download_data.sh. - Added comprehensive tests for src/data, src/training, and src/utils. - Fixed FutureWarning in Trainer regarding torch.amp.GradScaler. - Integrated mlflow for experiment tracking in Trainer. - Added ruff and mypy for linting and type checking. - Added .pre-commit-config.yaml for git hooks. - Added GitHub Actions CI workflow (.github/workflows/ci.yml).f6d689c
-
Added pretrained BART wights for the encoder/decoder while keeping custom architectiure and task heads
3225a94
-
chore: snapshot current refinements
185b05e
-
Reformatted Project Structure
133ab4d