Buckets:
| # Processors | |
| Multimodal models require a preprocessor capable of handling inputs that combine more than one modality. Depending on the input modality, a processor needs to convert text into an array of tensors, images into pixel values, and audio into an array of tensors with the correct sampling rate. | |
| For example, [PaliGemma](./model_doc/paligemma) is a vision-language model that uses the [SigLIP](./model_doc/siglip) image processor and the [Llama](./model_doc/llama) tokenizer. A [ProcessorMixin](/docs/transformers/pr_49429/en/main_classes/processors#transformers.ProcessorMixin) class wraps both of these preprocessor types, providing a single and unified processor class for a multimodal model. | |
| Call [from_pretrained()](/docs/transformers/pr_49429/en/model_doc/wav2vec2#transformers.Wav2Vec2Processor.from_pretrained) to load a processor. Pass the input type to the processor to generate the expected model inputs, input ids and pixel values. | |
| ```py | |
| from transformers import AutoProcessor, PaliGemmaForConditionalGeneration | |
| from PIL import Image | |
| import requests | |
| processor = AutoProcessor.from_pretrained("google/paligemma-3b-pt-224") | |
| prompt = "answer en Where is the cat standing?" | |
| url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg" | |
| image = Image.open(requests.get(url, stream=True).raw) | |
| inputs = processor(text=prompt, images=image, return_tensors="pt") | |
| inputs | |
| ``` | |
| This guide describes the processor class and how to preprocess multimodal inputs. | |
| ## Processor classes | |
| All processors inherit from the [ProcessorMixin](/docs/transformers/pr_49429/en/main_classes/processors#transformers.ProcessorMixin) class which provides methods like [from_pretrained()](/docs/transformers/pr_49429/en/model_doc/wav2vec2#transformers.Wav2Vec2Processor.from_pretrained), [save_pretrained()](/docs/transformers/pr_49429/en/model_doc/wav2vec2#transformers.Wav2Vec2Processor.save_pretrained), and [push_to_hub()](/docs/transformers/pr_49429/en/main_classes/model#transformers.utils.PushToHubMixin.push_to_hub) for loading, saving, and sharing processors to the Hub. | |
| There are two ways to load a processor, with an [AutoProcessor](/docs/transformers/pr_49429/en/model_doc/auto#transformers.AutoProcessor) and with a model-specific processor class. | |
| The [AutoClass](./model_doc/auto) API provides a simple interface to load processors without directly specifying the specific model class it belongs to. | |
| Use [from_pretrained()](/docs/transformers/pr_49429/en/model_doc/auto#transformers.AutoProcessor.from_pretrained) to load a processor. | |
| ```py | |
| from transformers import AutoProcessor | |
| processor = AutoProcessor.from_pretrained("google/paligemma-3b-pt-224") | |
| ``` | |
| Processors are also associated with a specific pretrained multimodal model class. You can load a processor directly from the model class with [from_pretrained()](/docs/transformers/pr_49429/en/model_doc/wav2vec2#transformers.Wav2Vec2Processor.from_pretrained). | |
| ```py | |
| from transformers import WhisperProcessor | |
| processor = WhisperProcessor.from_pretrained("openai/whisper-tiny") | |
| ``` | |
| You could also separately load the two preprocessor types, [WhisperTokenizerFast](/docs/transformers/pr_49429/en/model_doc/whisper#transformers.WhisperTokenizer) and [WhisperFeatureExtractor](/docs/transformers/pr_49429/en/model_doc/whisper#transformers.WhisperFeatureExtractor). | |
| ```py | |
| from transformers import WhisperTokenizerFast, WhisperFeatureExtractor, WhisperProcessor | |
| tokenizer = WhisperTokenizerFast.from_pretrained("openai/whisper-tiny") | |
| feature_extractor = WhisperFeatureExtractor.from_pretrained("openai/whisper-tiny") | |
| processor = WhisperProcessor(feature_extractor=feature_extractor, tokenizer=tokenizer) | |
| ``` | |
| ## Save and load a processor | |
| [save_pretrained()](/docs/transformers/pr_49429/en/model_doc/wav2vec2#transformers.Wav2Vec2Processor.save_pretrained) writes the processor's configuration to a single `processor_config.json` file. The tokenizer and chat template are saved in their own files. | |
| ```py | |
| from transformers import AutoProcessor | |
| processor = AutoProcessor.from_pretrained("llava-hf/llava-onevision-qwen2-0.5b-ov-hf") | |
| processor.save_pretrained("./llava-onevision") | |
| ``` | |
| ```text | |
| llava-onevision/ | |
| ├── chat_template.jinja | |
| ├── processor_config.json | |
| ├── tokenizer.json | |
| └── tokenizer_config.json | |
| ``` | |
| Every sub-processor except the tokenizer is nested inside `processor_config.json` under its attribute name, such as `image_processor`, `video_processor`, or `feature_extractor`. Settings that belong to the processor itself, like `image_token`, sit at the top level. The example below is shortened. | |
| ```json | |
| { | |
| "image_processor": { | |
| "image_processor_type": "LlavaOnevisionImageProcessor", | |
| "do_resize": true, | |
| "size": {"height": 384, "width": 384}, | |
| "image_mean": [0.5, 0.5, 0.5] | |
| }, | |
| "image_token": "<image>", | |
| "processor_class": "LlavaOnevisionProcessor", | |
| "video_processor": { | |
| "video_processor_type": "LlavaOnevisionVideoProcessor", | |
| "do_resize": true, | |
| "size": {"height": 384, "width": 384}, | |
| "image_mean": [0.5, 0.5, 0.5] | |
| }, | |
| "video_token": "<video>" | |
| } | |
| ``` | |
| The other files are saved separately. | |
| - The tokenizer writes its usual files, like `tokenizer.json` and `tokenizer_config.json`. If a processor has a second tokenizer, it's saved in a subfolder named after its attribute, like `decoder_tokenizer/`. | |
| - The chat template is saved to `chat_template.jinja`. If a processor has several chat templates, the `default` template goes in `chat_template.jinja` and the others go in a `chat_templates/` folder. | |
| When you load a processor with [from_pretrained()](/docs/transformers/pr_49429/en/model_doc/wav2vec2#transformers.Wav2Vec2Processor.from_pretrained), each sub-processor looks for its nested key in `processor_config.json` first. If the key isn't there, it falls back to a standalone file. Image processors and feature extractors read `preprocessor_config.json`, and video processors read `video_preprocessor_config.json` and then `preprocessor_config.json`. Older checkpoints that store each sub-processor in its own file still load this way. | |
| > [!NOTE] | |
| > A nested config takes priority over a standalone file. If a checkpoint has both, editing `preprocessor_config.json` or `video_preprocessor_config.json` has no effect. Edit the nested entry in `processor_config.json` instead. | |
| ## Preprocess | |
| Processors preprocess multimodal inputs into the expected Transformers format. There are a couple combinations of input modalities that a processor can handle such as text and audio or text and image. | |
| Automatic speech recognition (ASR) tasks require a processor that can handle text and audio inputs. Load a dataset and take a look at the `audio` and `text` columns (you can remove the other columns which aren't needed). | |
| ```py | |
| from datasets import load_dataset | |
| dataset = load_dataset("keithito/lj_speech", split="train") | |
| dataset = dataset.map(remove_columns=["file", "id", "normalized_text"]) | |
| dataset[0]["audio"] | |
| {'array': array([-7.3242188e-04, -7.6293945e-04, -6.4086914e-04, ..., | |
| 7.3242188e-04, 2.1362305e-04, 6.1035156e-05], dtype=float32), | |
| 'path': '/root/.cache/huggingface/datasets/downloads/extracted/917ece08c95cf0c4115e45294e3cd0dee724a1165b7fc11798369308a465bd26/LJSpeech-1.1/wavs/LJ001-0001.wav', | |
| 'sampling_rate': 22050} | |
| dataset[0]["text"] | |
| 'Printing, in the only sense with which we are at present concerned, differs from most if not from all the arts and crafts represented in the Exhibition' | |
| ``` | |
| Remember to resample the sampling rate to match the pretrained models required sampling rate. | |
| ```py | |
| from datasets import Audio | |
| dataset = dataset.cast_column("audio", Audio(sampling_rate=16000)) | |
| ``` | |
| Load a processor and pass the audio `array` and `text` columns to it. | |
| ```py | |
| from transformers import AutoProcessor | |
| processor = AutoProcessor.from_pretrained("openai/whisper-tiny") | |
| def prepare_dataset(example): | |
| audio = example["audio"] | |
| example.update(processor(audio=audio["array"], text=example["text"], sampling_rate=16000)) | |
| return example | |
| ``` | |
| Apply the `prepare_dataset` function to preprocess the dataset. The processor returns `input_features` for the `audio` column and `labels` for the text column. | |
| ```py | |
| prepare_dataset(dataset[0]) | |
| ``` | |
Xet Storage Details
- Size:
- 8.28 kB
- Xet hash:
- bb6a727819c1f4d2cc6bea4b69110a5036b3cdbc407f1041a6169aefa3892855
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.