Hugging Face's logo
Bram Vanroy's picture

Bram Vanroy PRO

BramVanroy
109 34 198
wvangils's profile picture JasperD-UGent's profile picture MahdadDutch's profile picture
·
https://bramvanroy.github.io/

AI & ML interests

Artificial intelligence, natural language processing, computational linguistics

Recent Activity

reacted to Nymbo's post with 🔥 3 days ago
Anthropic gave me six months of Claude Max 20x through the Claude for Open Source program, granted based on my Hugging Face work. Thank you https://huggingface.co/Anthropic for supporting open source. So far I've been pointing it at Markdown Minimap, an Obsidian plugin that adds a scrollable IDE-style minimap to your notes. This week I've been clearing a backlog of user-reported issues on it, with Claude often handling them end to end. https://github.com/Nymbo/Markdown-Minimap — issues and PRs welcome.
liked a dataset 4 days ago
lightonai/embeddings-pre-training
posted an update 4 days ago
**I benchmarked HF buckets against https access for Common Crawl.** Took me a while to get round to do this but I benchmarked access to Common Crawl via https vs hf buckets. Both experiments were run at night in Europe. I do not think other hardware problems were impacting the speeds since CPU processing time of the non-download pipeline components were highly similar (within 2% identical) and below only the WarcReader speeds of datatrove are used. Experiment: selected 5 disjoint samples of 64 files each (randomly from the latest crawl; 20,499 docs/file). Those five batches were then processed by 32 single-core tasks with 4GB/core (five batches to calculate CIs). Paired experiment between using https and hf bucket. - https: 40.0 [39.3-40.6] (seconds per WARC file) - hf bucket: 172.0 [122.7-221.2] That is a difference of about 4x in streaming speed. You'll see that https is also more stable (smaller CI). I also ran raw throughput tests to the endpoints to measure rate limiting (64MiB transfer at 8/32/128/256 concurrent readers) and rate limiting seems not an issue for either: at any of those parallel reader numbers, their respective speeds stay about the same. Note that, given CC scale, this is still a small test. Rate limiting may become more obvious when processing a full crawl. I do not know whether the https endpoint vs HF bucket will shut you out earlier with which limits.
View all activity

Organizations

Language and Translation Technology Team's profile picture BigScience Workshop's profile picture How to teach Hugging Face?'s profile picture Hugging Face Fellows's profile picture Blog-explorers's profile picture HPLT's profile picture ZeroGPU Explorers's profile picture signon-project's profile picture TENACITY's profile picture Occiglot's profile picture Social Post Explorers's profile picture Instituut voor de Nederlandse Taal's profile picture Hugging Face Discord Community's profile picture Networks of Ideas and Knowledge in the Ancient World's profile picture Instituut voor de Nederlandse Taal / Dutch Language Institute's profile picture ml-fw-prerelease's profile picture Gemma Challenge's profile picture