---
license: mit
base_model: microsoft/Fara1.5-9B
library_name: transformers
pipeline_tag: image-text-to-text
tags:
  - fara
  - computer-use
  - browser-automation
  - dom
  - baseline
  - evaluation
  - reproducibility
---

# Fara 1.5-9B W8 baseline evaluation

This repository preserves the exact Fara 1.5-9B checkpoint used for both W8 DOM browser-agent baseline evaluations. The checkpoint was not trained or fine-tuned for either evaluation. Its files are an exact mirror of the pinned Microsoft checkpoint.

## Common checkpoint and serving setup

| Component | Value |
|---|---|
| Base checkpoint | [`microsoft/Fara1.5-9B`](https://huggingface.co/microsoft/Fara1.5-9B) |
| Base checkpoint revision | `1a93677cd89d5601bc2ed759791e981f3a520032` |
| Fara source revision | `a675d6d61c41c47ae87bacefeab22caad18e3e84` |
| Model dtype | BF16 |
| KV-cache dtype | BF16 through vLLM `auto` |
| Maximum model length | 32,768 tokens |
| Tensor parallel size | 1 |
| Runtime | vLLM 0.19.1, PyTorch 2.10.0, Transformers 5.6.2 |
| Hardware | One NVIDIA H100 80 GB HBM3 |
| GCP instance | `w8-fara-9b-8115f370-head-3ibtw2ah-compute` |
| Deployed DOM adapter SHA256 | `6184fba5dee3cae079b20fe1abb1812e589e68b97737a28dac94b16e7f5c5811` |
| Deployed serving script SHA256 | `1d1c9c60280b15a73d633717776aafd23d087f6af447524ef9cc8315e6d70fb3` |

The DOM serving integration translated the recorder's OpenAI Responses contract to the Fara inference endpoint. It did not modify the model weights.

## Evaluation 1: 90-task baseline

| Measure | Result |
|---|---:|
| Selected tasks | 90 |
| Passed | 6 |
| Verified failures | 60 |
| Execution errors | 24 |
| Infrastructure errors | 0 |
| Pass@1 | 6.67% |
| Pass rate among verified tasks | 9.09% |

### Evaluation 1 provenance

| Component | Exact source |
|---|---|
| Evaluation artifacts | [`WootzappLab/fara-baseline-artifact`](https://huggingface.co/datasets/WootzappLab/fara-baseline-artifact/tree/eval-fara15-dom-baseline) |
| Immutable artifact commit | `30840bf827a58cfd0bdc8c86fa730ab32960448c` |
| Task dataset | [`WootzappLab/cua-bench`](https://huggingface.co/datasets/WootzappLab/cua-bench) |
| Dataset revision | `745eebb8f07abb92fdaae356bb6bee3a1f2975af` |
| Recorder, browser harness, environments, and verifier | [`tokenbender/w8-cua-bench`](https://github.com/tokenbender/w8-cua-bench/tree/environment-work) |
| Harness revision | `66037db6bae4b00a2b5860aa08ebb69b559e6871` |
| Evaluation date | 2026-09-25 |

Execution errors remain in the denominator. The artifact repository contains the raw recordings, logs, rubrics, verifier outputs, structured tables, and integrity manifests.

## Evaluation 2: 40-task baseline

| Measure | Result |
|---|---:|
| Selected tasks | 40 |
| Passed | 8 |
| Verified failures | 24 |
| Execution errors | 8 |
| Infrastructure errors | 0 |
| Pass@1 | 20.0% |
| Pass rate among verified tasks | 25.0% |

### Evaluation 2 provenance

| Component | Exact source |
|---|---|
| Evaluation artifacts | [`WootzappLab/Fara-40-baseline-artifact`](https://huggingface.co/datasets/WootzappLab/Fara-40-baseline-artifact/tree/eval-fara15-fresh-40-provenance-complete) |
| Immutable artifact commit | `f9c09ba743ab447b59e290a60ed61daa73808a66` |
| Results bundle SHA256 | `43f4ed88b79d9c0abe7a733ac6499b0000364e3f1f676a29320e83d1682e081b` |
| Exact task snapshot | Included in the artifact repository |
| Task snapshot SHA256 | `745f113b9c45f1a8dc3581ebc671435a3eafdfdbfb38bcaa6df1097ec10ab2e3` |
| Task dataset | [`WootzappLab/cua-bench-baseline-40`](https://huggingface.co/datasets/WootzappLab/cua-bench-baseline-40) |
| Dataset repository revision | `c2dd9ed8f23cbc477f49be7b7704e42bf4d46de6` (`baseline-40`) |
| Recorder, browser harness, environments, and verifier | [`tokenbender/w8-cua-bench`](https://github.com/tokenbender/w8-cua-bench/tree/environment-work) |
| Harness revision | `09976c86454f9a87e2fa6a3a3589d2e4d24f180e` |
| Infrastructure revision | `eb5dc3d9e0cbfbe389334b0600002f8db3a795dd` |
| Evaluation date | 2026-09-28 |

Execution errors remain in the denominator. The published dataset and the snapshot bundled with the artifacts contain the same task bytes, identified by the SHA256 above.

## Repository contents

- Model configuration, tokenizer, processor, chat template, and Safetensors weights copied from the pinned upstream revision
- `CHECKPOINT_FILES.json`: upstream Git blob identities and LFS SHA256 hashes for the checkpoint files
- `MANIFEST.json`: machine-readable checkpoint provenance
- `PROVENANCE.md`: checkpoint revision and experiment-boundary receipt
- `CHECKSUMS.sha256`: hashes for the repository's reproducibility metadata

## Load the checkpoint

```python
import torch
from transformers import AutoModelForMultimodalLM, AutoProcessor

model_id = "WootzappLab/fara-9b-baseline-eval"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForMultimodalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
```

## License and attribution

The mirrored base checkpoint is released by Microsoft under the MIT License. See the upstream [Fara 1.5-9B model card](https://huggingface.co/microsoft/Fara1.5-9B) for its original documentation, intended use, and limitations.