Commit History
-
chore: update README and REPORT with performance insights and dataset changes
SarthakThis commit adds a warning in the README regarding the performance degradation observed with C4 fine-tuning, recommending the use of basic distillation for optimal results. The REPORT has been updated to reflect revised performance metrics for the fine-tuned model, adjustments in average performance statistics, and the inclusion of new radar charts for model comparisons. Additionally, the dataset configuration has been modified to utilize the C4 dataset for tokenlearn featurization.0dbb356 -
chore: update dependencies and configuration for improved training
SarthakThis commit updates the model configuration in `.codemap.yml` to use a lighter version of the model. Additionally, it enhances the `pyproject.toml` and `uv.lock` files by adding new dependencies such as `jinja2`, `joblib`, `rich`, and `safetensors`, while also replacing `tokenlearn` with `tokenizers`. The report has been adjusted to reflect changes in model performance metrics and the dataset configuration has been improved to support optimized dataset usage during training.7837959 -
chore: moved tokenlearn as in internal package
Sarthak4255a26 -
chore: moved model2vec as in internal package
Sarthak473c3a0
-
feat(distiller): add option to skip post-training re-regularization
SarthakThis change introduces an option to skip the post-training re-regularization step in the tokenlearn training pipeline. This can be useful for debugging or experimentation, or when re-regularization is not desired.72121b3 -
feat(analyze): enhance model type identification in analysis
SarthakThe logic has been updated to differentiate between simplified distilled models and fine-tuned distilled models, and assign a new model type for fine-tuned variations.fba41e9 -
chore(distiller): remove unused GPU_NAME variable
SarthakRemove GPU_NAME from the configuration settings and the __all__ list. This simplifies the configuration and removes an unused variable.293cdb2 -
feat(config): allow multiple GPU types for training and simplify GPU handling
SarthakThis change allows training jobs to be configured with either a single GPU type or a list of GPU types, providing more flexibility in resource allocation. It also adds a function to get function kwargs specifically for training jobs. The `create_beam_function_kwargs` function was updated to accept both single and list of GPU types without converting from string, simplifying the GPU type handling.d820ac9 -
refactor(patch-utils): improve patch application check for tokenlearn
Sarthakc9e9334 -
refactor(distiller): improve beam distillation and tokenlearn integration
SarthakThis commit introduces separate Beam functions for distillation and training, enabling more modular and controllable workflows. It also enhances tokenlearn integration by using persistent directories for caching and checkpointing, and improves error handling for training failures. The changes also include validation of the model to check vocab and embedding sizes match, which can highlight issues in downstream usage.729d700 -
refactor(beam-utils): use direct file operations for beam volumes
SarthakThis commit refactors the Beam volume interaction logic to use direct file operations when running on the Beam platform. It removes the dependency on the `beam cp` command for file transfers, which simplifies the code and improves performance. It also adds checks to determine if the code is running on Beam and adjusts the file operations accordingly.12d70ca -
feat(distiller): add checkpointing and refactor analyze.py
SarthakThis change introduces checkpointing to the tokenlearn featurization and training processes. This allows the processes to resume from where they left off if they are interrupted or have already completed. It also adds a --clear-checkpoints flag to force fresh featurization and training. Additionally, minor refactoring was done to use list comprehension in analyze.py8083c06 -
fix(main): reduce maximum queries per language to 100
SarthakThe default value was too high, causing the evaluation to take a long time.da2f1e0 -
feat(distiller): configure beam functions with resource settings
SarthakThis commit introduces a configuration system for Beam functions, allowing for better resource management, execution settings, and environment configurations for different types of Beam jobs like distillation and evaluation. It also simplifies the function deployment process by providing pre-defined configurations and utilities for creating @function decorator kwargs.ef6935e -
feat: introduce distiller package and update README
SarthakThis commit introduces the 'distiller' package, a toolkit for creating code-specialized static embeddings through Model2Vec distillation and Tokenlearn training. The updated README provides comprehensive documentation and usage examples for the distiller, highlighting its performance benefits and cloud-scale processing capabilities. Additionally, the REPORT.md provides a performance analysis of different Model2Vec distillation experiments.ee673cb -
chore: remove unused scripts and update dependencies
SarthakThis commit removes the MTEB evaluation script, distillation script, and evaluation script as they are no longer needed. Additionally, updates the pyproject.toml file to remove dependencies related to the removed scripts and adds typing-extensions to the dependencies.1bc7e54
-
feat: overhaul distiller package with unified CLI, enhanced evaluation, and modular structure
Sarthak454e47c -
feat: created a cli to manage the complete generation process
Sarthakea0b2a0