Buckets:

|
download
raw
5.92 kB
# FP8
Below are functions and classes relative to the underlying FP8 implementation
## TERecipeKwargs[[accelerate.utils.TERecipeKwargs]]
#### accelerate.utils.TERecipeKwargs[[accelerate.utils.TERecipeKwargs]]
```python
accelerate.utils.TERecipeKwargs(use_autocast_during_eval: bool | None = None, margin: int | None = None, interval: int | None = None, fp8_format: typing.Literal['HYBRID', 'E4M3', 'E5M2'] = None, amax_history_len: int | None = None, amax_compute_algo: typing.Literal['max', 'most_recent'] = None, override_linear_precision: tuple = None, use_mxfp8_block_scaling: bool | None = None)
```
[Source](https://github.com/huggingface/accelerate/blob/vr_4552/src/accelerate/utils/dataclasses.py#L333)
**Parameters:**
use_autocast_during_eval (`bool`, *optional*, default to `False`) : Whether to use FP8 autocast during eval mode. Generally better metrics are found when this is `False`.
margin (`int`, *optional*, default to 0) : The margin to use for the gradient scaling.
interval (`int`, *optional*, default to 1) : The interval to use for how often the scaling factor is recomputed.
fp8_format (`str`, *optional*, default to "HYBRID") : The format to use for the FP8 recipe. Must be one of `HYBRID`, `E4M3` or `E5M2`. (Generally `HYBRID` for training, `E4M3` or `E5M2` for evaluation)
amax_history_len (`int`, *optional*, default to 1024) : The length of the history to use for the scaling factor computation
amax_compute_algo (`str`, *optional*, default to "most_recent") : The algorithm to use for the scaling factor computation. Must be one of `max` or `most_recent`.
override_linear_precision (`tuple` of three `bool`, *optional*, default to `(False, False, False)`) : Whether or not to execute `fprop`, `dgrad`, and `wgrad` GEMMS in higher precision.
Use this object in your [Accelerator](/docs/accelerate/pr_4552/en/package_reference/accelerator#accelerate.Accelerator) to customize the initialization of the recipe for FP8 mixed precision
training with `transformer-engine`.
For more information on the args, please refer to the API
[documentation](https://docs.nvidia.com/deeplearning/transformer-engine/user-guide/api/common.html).
```python
from accelerate import Accelerator
from accelerate.utils import TERecipeKwargs
kwargs = TERecipeKwargs(fp8_format="HYBRID")
accelerator = Accelerator(mixed_precision="fp8", kwargs_handlers=[kwargs])
```
## AORecipeKwargs[[accelerate.utils.AORecipeKwargs]]
#### accelerate.utils.AORecipeKwargs[[accelerate.utils.AORecipeKwargs]]
```python
accelerate.utils.AORecipeKwargs(config: ForwardRef('Float8LinearConfig') | None = None, module_filter_func: typing.Callable | None = None, pad_inner_dim: bool | None = None, enable_fsdp_float8_all_gather: bool | None = None)
```
[Source](https://github.com/huggingface/accelerate/blob/vr_4552/src/accelerate/utils/dataclasses.py#L285)
**Parameters:**
config (`torchao.float8.Float8LinearConfig`, *optional*, default to `None`) : The configuration for the FP8 training. If `None`, a default config will be created with sensible defaults for most use cases: - `pad_inner_dim=True`: Pads matrix dimensions to be divisible by 16, required for `torch._scaled_mm` operations to prevent runtime errors. - `enable_fsdp_float8_all_gather=True`: Enables FP8 all-gather for FSDP2. This provides memory bandwidth savings by casting parameters before the all-gather operation, saving 50% bandwidth compared to BF16. You can override these defaults by providing your own `Float8LinearConfig` instance.
module_filter_func (`Callable`, *optional*, default to `None`) : Optional function that must take in a module and layer name, and returns a boolean indicating whether the module should be converted to FP8. Defaults to `accelerate.utils.ao.filter_linear_layers`. See it for an example.
Use this object in your [Accelerator](/docs/accelerate/pr_4552/en/package_reference/accelerator#accelerate.Accelerator) to customize the initialization of the recipe for FP8 mixed precision
training with `torchao` FP8.
## convert_model[[accelerate.utils.convert_model]]
#### accelerate.utils.convert_model[[accelerate.utils.convert_model]]
```python
accelerate.utils.convert_model(model, to_transformer_engine = True, _convert_linear = True, _convert_ln = True)
```
[Source](https://github.com/huggingface/accelerate/blob/vr_4552/src/accelerate/utils/transformer_engine.py#L26)
Recursively converts the linear and layernorm layers of a model to their `transformers_engine` counterpart.
## has_transformer_engine_layers[[accelerate.utils.has_transformer_engine_layers]]
#### accelerate.utils.has_transformer_engine_layers[[accelerate.utils.has_transformer_engine_layers]]
```python
accelerate.utils.has_transformer_engine_layers(model)
```
[Source](https://github.com/huggingface/accelerate/blob/vr_4552/src/accelerate/utils/transformer_engine.py#L95)
Returns whether a given model has some `transformer_engine` layer or not.
## contextual_fp8_autocast[[accelerate.utils.contextual_fp8_autocast]]
#### accelerate.utils.contextual_fp8_autocast[[accelerate.utils.contextual_fp8_autocast]]
```python
accelerate.utils.contextual_fp8_autocast(model_forward, fp8_recipe, use_during_eval = False)
```
[Source](https://github.com/huggingface/accelerate/blob/vr_4552/src/accelerate/utils/transformer_engine.py#L118)
Wrapper for a model's forward method to apply FP8 autocast. Is context aware, meaning that by default it will
disable FP8 autocast during eval mode, which is generally better for more accurate metrics.
## apply_fp8_autowrap[[accelerate.utils.apply_fp8_autowrap]]
#### accelerate.utils.apply_fp8_autowrap[[accelerate.utils.apply_fp8_autowrap]]
```python
accelerate.utils.apply_fp8_autowrap(model, fp8_recipe_handler)
```
[Source](https://github.com/huggingface/accelerate/blob/vr_4552/src/accelerate/utils/transformer_engine.py#L142)
Applies FP8 context manager to the model's forward method

Xet Storage Details

Size:
5.92 kB
·
Xet hash:
e833e5288642f01bf0a99748e0ac78610fb32a4e0a9056c1ad8087df1432ba70

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.