Buckets:
FP8
Below are functions and classes relative to the underlying FP8 implementation
TERecipeKwargs[[accelerate.utils.TERecipeKwargs]]
accelerate.utils.TERecipeKwargs[[accelerate.utils.TERecipeKwargs]]
accelerate.utils.TERecipeKwargs(use_autocast_during_eval: bool | None = None, margin: int | None = None, interval: int | None = None, fp8_format: typing.Literal['HYBRID', 'E4M3', 'E5M2'] = None, amax_history_len: int | None = None, amax_compute_algo: typing.Literal['max', 'most_recent'] = None, override_linear_precision: tuple = None, use_mxfp8_block_scaling: bool | None = None)
Parameters:
use_autocast_during_eval (bool, optional, default to False) : Whether to use FP8 autocast during eval mode. Generally better metrics are found when this is False.
margin (int, optional, default to 0) : The margin to use for the gradient scaling.
interval (int, optional, default to 1) : The interval to use for how often the scaling factor is recomputed.
fp8_format (str, optional, default to "HYBRID") : The format to use for the FP8 recipe. Must be one of HYBRID, E4M3 or E5M2. (Generally HYBRID for training, E4M3 or E5M2 for evaluation)
amax_history_len (int, optional, default to 1024) : The length of the history to use for the scaling factor computation
amax_compute_algo (str, optional, default to "most_recent") : The algorithm to use for the scaling factor computation. Must be one of max or most_recent.
override_linear_precision (tuple of three bool, optional, default to (False, False, False)) : Whether or not to execute fprop, dgrad, and wgrad GEMMS in higher precision.
Use this object in your Accelerator to customize the initialization of the recipe for FP8 mixed precision
training with transformer-engine.
For more information on the args, please refer to the API documentation.
from accelerate import Accelerator
from accelerate.utils import TERecipeKwargs
kwargs = TERecipeKwargs(fp8_format="HYBRID")
accelerator = Accelerator(mixed_precision="fp8", kwargs_handlers=[kwargs])
AORecipeKwargs[[accelerate.utils.AORecipeKwargs]]
accelerate.utils.AORecipeKwargs[[accelerate.utils.AORecipeKwargs]]
accelerate.utils.AORecipeKwargs(config: ForwardRef('Float8LinearConfig') | None = None, module_filter_func: typing.Callable | None = None, pad_inner_dim: bool | None = None, enable_fsdp_float8_all_gather: bool | None = None)
Parameters:
config (torchao.float8.Float8LinearConfig, optional, default to None) : The configuration for the FP8 training. If None, a default config will be created with sensible defaults for most use cases: - pad_inner_dim=True: Pads matrix dimensions to be divisible by 16, required for torch._scaled_mm operations to prevent runtime errors. - enable_fsdp_float8_all_gather=True: Enables FP8 all-gather for FSDP2. This provides memory bandwidth savings by casting parameters before the all-gather operation, saving 50% bandwidth compared to BF16. You can override these defaults by providing your own Float8LinearConfig instance.
module_filter_func (Callable, optional, default to None) : Optional function that must take in a module and layer name, and returns a boolean indicating whether the module should be converted to FP8. Defaults to accelerate.utils.ao.filter_linear_layers. See it for an example.
Use this object in your Accelerator to customize the initialization of the recipe for FP8 mixed precision
training with torchao FP8.
convert_model[[accelerate.utils.convert_model]]
accelerate.utils.convert_model[[accelerate.utils.convert_model]]
accelerate.utils.convert_model(model, to_transformer_engine = True, _convert_linear = True, _convert_ln = True)
Recursively converts the linear and layernorm layers of a model to their transformers_engine counterpart.
has_transformer_engine_layers[[accelerate.utils.has_transformer_engine_layers]]
accelerate.utils.has_transformer_engine_layers[[accelerate.utils.has_transformer_engine_layers]]
accelerate.utils.has_transformer_engine_layers(model)
Returns whether a given model has some transformer_engine layer or not.
contextual_fp8_autocast[[accelerate.utils.contextual_fp8_autocast]]
accelerate.utils.contextual_fp8_autocast[[accelerate.utils.contextual_fp8_autocast]]
accelerate.utils.contextual_fp8_autocast(model_forward, fp8_recipe, use_during_eval = False)
Wrapper for a model's forward method to apply FP8 autocast. Is context aware, meaning that by default it will disable FP8 autocast during eval mode, which is generally better for more accurate metrics.
apply_fp8_autowrap[[accelerate.utils.apply_fp8_autowrap]]
accelerate.utils.apply_fp8_autowrap[[accelerate.utils.apply_fp8_autowrap]]
accelerate.utils.apply_fp8_autowrap(model, fp8_recipe_handler)
Applies FP8 context manager to the model's forward method
Xet Storage Details
- Size:
- 5.92 kB
- Xet hash:
- f7ddb64740d5f75b3438d6cf698bf5210c9424eff184ac7892d080b0554afa84
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.