README / README.md
lhoestq's picture
lhoestq HF Staff
Update README.md
92956cc verified
|
Raw History Blame Contribute Delete
1.55 kB
---
title: README
emoji: 💻
colorFrom: yellow
colorTo: red
sdk: static
pinned: false
---
<div style="max-width: 100%; overflow: hidden;">
<img src="https://cdn-uploads.huggingface.co/production/uploads/5e9ecfc04957053f60648a3e/ZldIIprwJrIBnaDm7m8nB.png"
alt="Hugging Science"
style="width: 100%; height: auto; display: block;">
</div>
Project repository: [https://github.com/lhoestq/duckdb-hf-extension](https://github.com/lhoestq/duckdb-hf-extension)
# Installation
The extension requires DuckDB 2.0
```sql
SET allow_extension_repositories = 'allowed';
CREATE OR REPLACE EXTENSION REPOSITORY hf_repo
WITH PREFIX 'https://hf.co/buckets/duckdb-extensions/builds/resolve';
INSTALL hf FROM hf_repo;
```
# Usage
Read/write to Hugging Face buckets, datasets, models and spaces. Optimized with [Xet](https://huggingface.co/docs/hub/en/xet/index).
```sql
LOAD hf FROM hf_repo;
SELECT * FROM 'hf://datasets/fka/prompts.chat/prompts.csv';
SELECT * FROM hf_dataset('stanfordnlp/imdb', split := 'train');
SELECT * FROM hf_ls('hf://buckets/commoncrawl/commoncrawl/crawl-data');
COPY (SELECT * FROM range(10) t(i)) TO 'hf://datasets/me/demo/out.parquet';
COPY (SELECT * FROM read_parquet('hf://datasets/me/demo/*.parquet'))
TO 'hf://datasets/me/demo/partitioned' (FORMAT PARQUET, PARTITION_BY (kind));
```
# Authentication
Login with the [hf](https://huggingface.co/docs/huggingface_hub/en/guides/cli) CLI:
```bash
hf auth login
```
Or pass a token manually:
```sql
CREATE SECRET (TYPE HUGGINGFACE, TOKEN 'hf_...');
```