| # Xet uploads over AWS PrivateLink |
|
|
| Terraform for the customer side of a private upload path into Hugging Face. |
| Upload traffic stays on the AWS network instead of being metered as internet |
| egress. Your hostname, client and credentials do not change; only DNS inside |
| your VPC does. |
|
|
| Two paths exist. Use the `hf` CLI one unless you specifically need the S3 |
| protocol: it chunks and deduplicates client side, so it sends less. |
|
|
| | Path | Hostname | Client | |
| |---|---|---| |
| | `cas_service_name` | `cas-server.xethub.hf.co` | `hf` CLI | |
| | `s3_service_name` | `s3.hf.co` | S3 API | |
|
|
| ## What we exchange |
|
|
| Send us your AWS account ID and the region your upload instances run in. We |
| send back the endpoint service name for each path you want, and the |
| Availability Zones we serve as AZ IDs. |
|
|
| ## Apply |
|
|
| ```bash |
| cp terraform.tfvars.example terraform.tfvars |
| terraform init |
| terraform apply |
| ``` |
|
|
| Pick subnets by AZ ID. `us-east-1a` is a different physical zone in every AWS |
| account, so matching on the name can put your endpoint in a different zone from |
| your compute: |
|
|
| ```bash |
| aws ec2 describe-availability-zones \ |
| --query 'AvailabilityZones[].[ZoneName,ZoneId]' --output table |
| ``` |
|
|
| The apply blocks while the endpoint sits in `pendingAcceptance`, because we |
| accept connections by hand. Send us the `endpoint_ids` output while it waits. |
|
|
| The S3 path serves fewer zones than the `hf` CLI one. If `subnet_ids` includes a |
| zone it does not cover, set `s3_subnet_ids` to the subset it does. |
|
|
| For a VPC outside `us-east-1`, set `service_region = "us-east-1"`. |
|
|
| ## What it creates |
|
|
| An interface endpoint per path, one security group, and a private hosted zone |
| per hostname holding a single record. The zones are scoped to your VPC and do |
| not affect public DNS. Our TLS certificate already covers both hostnames. |
|
|
| `s3.hf.co` is a zone apex, where DNS forbids a CNAME, so that record is an |
| alias. |
|
|
| ## Verify |
|
|
| Run `./verify-privatelink.sh` from an instance in the VPC. |
|
|
| ## Upload |
|
|
| ```bash |
| hf upload your-org/your-model ./local-dir |
| ``` |
|
|
| Auth and commit calls still reach `huggingface.co` over the internet, so egress |
| there has to stay open. The volume is negligible next to your data. |
|
|
