Spaces:
Sleeping
Sleeping
|
Download BIRDNET_UPLOADER_REDESIGN.md from jrrribeiro/BirdNET-Uploader-App: direct link, hf CLI and curl.
- Browser
- Download file 25.4 kB
-
https://huggingface.co/spaces/jrrribeiro/BirdNET-Uploader-App/resolve/main/BIRDNET_UPLOADER_REDESIGN.md
- Command line
-
hf download hf://spaces/jrrribeiro/BirdNET-Uploader-App/BIRDNET_UPLOADER_REDESIGN.md
-
curl -L -o BIRDNET_UPLOADER_REDESIGN.md https://huggingface.co/spaces/jrrribeiro/BirdNET-Uploader-App/resolve/main/BIRDNET_UPLOADER_REDESIGN.md
25.4 kB
A newer version of the Gradio SDK is available: 6.32.0
BirdNET Uploader: Redesign CLI Moderno
VisΓ£o Geral
ExecutΓ‘vel local moderno com um ΓΊnico fluxo integrado: login β criar repositΓ³rio β apontar pasta β fazer upload.
Interface limpa, baseada em TUI (Textual) ou rich terminal output, compatΓvel com a estrutura esperada pelo BirdNET Validator.
1. FLUXO DE UX ΓNICO INTEGRADO
Tela 1: AutenticaΓ§Γ£o
βββββββββββββββββββββββββββββββββββββββββββββββββββ
β BirdNET Uploader β
β ββββββββββββββββββββββββββββββββββββββββββ β
β β
β π¦ Autentique-se no Hugging Face β
β β
β [i] Cole seu token HF (Settings β Access β
β Tokens em huggingface.co) β
β β
β Token: [ββββββββββββββββββββββ] β
β β
β [ Continuar ] [ Sair ] β
β β
βββββββββββββββββββββββββββββββββββββββββββββββββββ
Tela 2: ConfiguraΓ§Γ£o (Um formulΓ‘rio integrado)
βββββββββββββββββββββββββββββββββββββββββββββββββββ
β BirdNET Uploader - ConfiguraΓ§Γ£o β
β ββββββββββββββββββββββββββββββββββββββββββ β
β β
β Conectado como: alice@example.com β
β β
β REPOSITΓRIO β
β Nome do repo: [my-birds-2026 ] β
β Visibilidade: ( ) PΓΊblico (β) Privado β
β β
β ARQUIVOS β
β Pasta segmentos: [C:\audio\segments] β
β Arquivo CSV: [C:\audio\metadata.csv] (opt)
β [ Verificar ] β
β β 1.247 arquivos encontrados β
β β Tamanho total: 53 GB β
β β CSV compatΓvel: 1.200 linhas β
β β
β [ Criar Repo e Fazer Upload ] β
β [ Voltar ] β
β β
βββββββββββββββββββββββββββββββββββββββββββββββββββ
Tela 3: Resumo e ConfirmaΓ§Γ£o
βββββββββββββββββββββββββββββββββββββββββββββββββββ
β Pronto para ComeΓ§ar β
β ββββββββββββββββββββββββββββββββββββββββββ β
β β
β RepositΓ³rio: alice/my-birds-2026 β
β Visibilidade: Privado β
β URL: https://huggingface.co/datasets/alice/.. β
β β
β Origem: β
β Pasta: C:\audio\segments β
β Arquivos: 1.247 β
β Tamanho: 53 GB β
β CSV: metadata.csv (1.200 linhas) β
β β
β DuraΓ§Γ£o estimada: ~4 horas (com conexΓ£o 10MB/s)
β β
β [ Iniciar Upload ] [ Cancelar ] β
β β
βββββββββββββββββββββββββββββββββββββββββββββββββββ
Tela 4: Upload em Andamento
βββββββββββββββββββββββββββββββββββββββββββββββββββ
β Upload em Andamento β
β ββββββββββββββββββββββββββββββββββββββββββ β
β β
β Progresso Global β
β ββββββββββββββββββββββββββββββ 62.3% β
β β
β Arquivos: 775/1.247 (skipped: 18, erro: 2) β
β Dados: 33 GB / 53 GB β
β Velocidade: 9.6 MB/s | ETA: 01h 15m β
β β
β Arquivo Atual: Parrot_Ara_001.wav β
β βββββββββββββββββββββββββββββββ 45.2% β
β 3.2 / 7.1 MB β
β β
β Γltimo evento: β Thrush_042.wav enviado (2s) β
β β
β [ Pausar ] [ Cancelar ] [ Detalhes ] β
β β
βββββββββββββββββββββββββββββββββββββββββββββββββββ
Tela 5: Resumo Final
βββββββββββββββββββββββββββββββββββββββββββββββββββ
β β Upload ConcluΓdo! β
β ββββββββββββββββββββββββββββββββββββββββββ β
β β
β RepositΓ³rio: alice/my-birds-2026 β
β URL: https://huggingface.co/datasets/alice/.. β
β β
β Resumo: β
β β Enviados: 1.227 arquivos β
β β Pulados: 18 (jΓ‘ existiam) β
β β Erros: 2 (revisar detalhes) β
β β± Tempo total: 4h 23min β
β π Velocidade mΓ©dia: 9.8 MB/s β
β β
β PrΓ³ximos passos: β
β 1. Validar dados com BirdNET Validator β
β 2. Verificar relatΓ³rio de ingestΓ£o β
β β
β RelatΓ³rio salvo em: ~/.birdnet-uploader/ β
β sessions/upload-xxx/ β
β report.json β
β β
β [ Ver RelatΓ³rio Completo ] [ Sair ] β
β β
βββββββββββββββββββββββββββββββββββββββββββββββββββ
2. ARQUITETURA LIMPA E MODULAR
Estrutura de DiretΓ³rios do Projeto
birdnet-uploader/
βββ src/
β βββ __init__.py
β βββ main.py # Entrypoint principal
β βββ config.py # Config app e constantes
β βββ auth/
β β βββ __init__.py
β β βββ auth_service.py # Login, token, keyring
β β βββ token_manager.py # Gerenciamento de token
β βββ repository/
β β βββ __init__.py
β β βββ repo_service.py # Criar/validar dataset HF
β β βββ structure_initializer.py # Init estrutura padrΓ£o
β βββ upload/
β β βββ __init__.py
β β βββ scanner.py # Varrer pasta local
β β βββ uploader.py # Motor de upload por lotes
β β βββ deduplicator.py # Checagem de duplicaΓ§Γ£o
β β βββ batch_processor.py # Processamento de lotes
β βββ csv/
β β βββ __init__.py
β β βββ csv_matcher.py # Matching CSV + segmentos
β β βββ indexer.py # GeraΓ§Γ£o de Γndice/shards
β βββ session/
β β βββ __init__.py
β β βββ session_manager.py # Gerenciamento de sessΓ£o
β β βββ checkpoint.py # PersistΓͺncia de checkpoint
β β βββ state_store.py # State file I/O
β βββ ui/
β β βββ __init__.py
β β βββ tui.py # Interface Textual (TUI)
β β βββ progress_renderer.py # Barra de progresso rich
β β βββ prompts.py # DiΓ‘logos e inputs
β βββ logger/
β β βββ __init__.py
β β βββ structured_logger.py # Logs estruturados JSONL
β β βββ telemetry.py # Eventos de telemetria
β βββ utils/
β βββ __init__.py
β βββ file_utils.py # OperaΓ§Γ΅es de arquivo
β βββ hash_utils.py # ComputaΓ§Γ£o de hash
β βββ error_handler.py # Tratamento de erros
βββ build/
β βββ pyinstaller_config.spec # Config do PyInstaller
β βββ build.py # Script de build
βββ tests/
β βββ __init__.py
β βββ test_auth.py
β βββ test_uploader.py
β βββ test_session.py
β βββ test_csv_matcher.py
βββ README.md
βββ requirements.txt
βββ setup.py
DependΓͺncias Principais
# requirements.txt
huggingface_hub>=0.28.0 # HF Hub API
click>=8.0.0 # CLI framework
textual>=0.30.0 # TUI moderna
rich>=13.0.0 # Terminal formatting
pydantic>=2.0.0 # ValidaΓ§Γ£o de dados
pandas>=2.0.0 # CSV parsing
pyarrow>=14.0.0 # Parquet para shards
keyring>=24.0.0 # Armazenamento seguro
cryptography>=41.0.0 # Criptografia local
3. COMPATIBILIDADE COM ESTRUTURA DO VALIDADOR
Estrutura Esperada do Dataset
dataset_repo_id = "username/project-dataset"
audio/
βββ {project_slug}/
β βββ Specie_A/
β β βββ segment_001.wav
β β βββ segment_002.wav
β β βββ ...
β βββ Specie_B/
β β βββ ...
β βββ ...
index/
βββ shards/
β βββ shard-00000.parquet
β βββ shard-00001.parquet
β βββ ...
βββ manifest.json
validations/
βββ results-20250419.parquet
audit/
βββ ingestion-runs/
β βββ 20250419T100000Z.json
Manifest Format (MantΓ©m compatibilidade)
{
"schema_version": "1.0.0",
"project_slug": "my-birds",
"dataset_repo_id": "alice/my-birds-2026",
"updated_at": "2025-04-19T10:45:30Z",
"index": {
"total_detections": 1227,
"total_audio_files": 1227,
"shard_size": 10000,
"shards": [
{
"path": "index/shards/shard-00000.parquet",
"rows": 10000,
"sha256": "abc123...",
"size_bytes": 5242880
}
]
}
}
4. MΓDULO POR MΓDULO
4.1 Auth Service (src/auth/auth_service.py)
from keyring import get_password, set_password
from huggingface_hub import HfApi, login
class AuthService:
KEYRING_SERVICE = "birdnet-uploader"
def authenticate(self, token: str) -> dict:
"""Valida token e retorna context do usuΓ‘rio."""
try:
api = HfApi(token=token)
user_info = api.whoami()
# Salvar em keyring seguro
set_password(self.KEYRING_SERVICE, "hf_token", token)
return {
"username": user_info["name"],
"email": user_info.get("email", ""),
"user_id": user_info["user_id"],
"token_expiry": self._estimate_expiry()
}
except Exception as e:
raise AuthenticationError(f"Token invΓ‘lido: {str(e)}")
def get_token(self) -> str:
"""Recupera token do keyring."""
token = get_password(self.KEYRING_SERVICE, "hf_token")
if not token:
raise TokenNotFoundError("Token nΓ£o encontrado. Execute 'birdnet auth login'")
return token
4.2 Repository Service (src/repository/repo_service.py)
class RepositoryService:
def create_dataset(self,
repo_name: str,
private: bool = True) -> str:
"""Cria dataset no HF com estrutura padrΓ£o."""
api = HfApi(token=self.get_token())
repo_id = f"{username}/{repo_name}"
# Criar repo
repo = api.create_repo(
repo_id=repo_id,
repo_type="dataset",
private=private,
exist_ok=True
)
# Inicializar estrutura padrΓ£o
self._init_structure(repo_id)
return repo_id
def _init_structure(self, repo_id: str):
"""Inicializa pastas e arquivos padrΓ£o."""
folders = [
"audio/{project_slug}",
"index/shards",
"validations",
"audit/ingestion-runs"
]
for folder in folders:
# Criar .gitkeep
api.upload_file(
path_or_fileobj=BytesIO(b""),
path_in_repo=f"{folder}/.gitkeep",
repo_id=repo_id,
repo_type="dataset"
)
# Criar manifest.json inicial
manifest = {
"schema_version": "1.0.0",
"project_slug": project_slug,
"dataset_repo_id": repo_id,
"index": {
"total_detections": 0,
"total_audio_files": 0,
"shard_size": 10000,
"shards": []
}
}
api.upload_file(
path_or_fileobj=BytesIO(json.dumps(manifest).encode()),
path_in_repo="index/manifest.json",
repo_id=repo_id,
repo_type="dataset"
)
4.3 Scanner (src/upload/scanner.py)
class LocalScanner:
def scan_folder(self, folder_path: str) -> dict:
"""Varre pasta recursiva e retorna estrutura."""
structure = {}
total_size = 0
file_count = 0
for root, dirs, files in os.walk(folder_path):
for file in files:
if file.lower().endswith(('.wav', '.mp3', '.flac', '.ogg')):
full_path = os.path.join(root, file)
# Inferir espΓ©cie do caminho ou nome
specie = self._infer_specie(root, file)
if specie not in structure:
structure[specie] = []
size = os.path.getsize(full_path)
structure[specie].append({
"name": file,
"full_path": full_path,
"size": size,
"local_hash": compute_hash(full_path)
})
total_size += size
file_count += 1
return {
"total_files": file_count,
"total_size": total_size,
"by_specie": structure,
"scan_timestamp": datetime.now()
}
def _infer_specie(self, root: str, file: str) -> str:
"""Inferir espΓ©cie do caminho da pasta."""
# EstratΓ©gia 1: ΓΊltima pasta em root Γ© a espΓ©cie
parts = root.split(os.sep)
if len(parts) > 0:
return parts[-1]
# Fallback: usar nome do arquivo
return file.replace(".wav", "").split("_")[0]
4.4 Uploader (src/upload/uploader.py)
class BatchUploader:
def __init__(self, repo_id: str, project_slug: str, token: str):
self.repo_id = repo_id
self.project_slug = project_slug
self.api = HfApi(token=token)
self.session = SessionManager(repo_id)
async def upload_files(self,
structure: dict,
csv_matcher: Optional[CSVMatcher] = None,
batch_size: int = 10,
on_progress: Optional[Callable] = None) -> dict:
"""Upload em lotes com retomada."""
checkpoint = self.session.load_checkpoint()
results = {
"total": 0,
"uploaded": 0,
"skipped": 0,
"failed": 0,
"errors": []
}
for specie, files in structure["by_specie"].items():
# Processar em lotes
for i in range(0, len(files), batch_size):
batch = files[i:i + batch_size]
# Verificar checkpoint
if self.session.is_batch_done(specie, i):
results["skipped"] += len(batch)
continue
# Enviar batch
for file_info in batch:
try:
if self._should_skip(file_info):
results["skipped"] += 1
self.session.mark_skipped(file_info)
continue
remote_path = f"audio/{self.project_slug}/{specie}/{file_info['name']}"
await self._upload_file_with_retry(
file_info["full_path"],
remote_path
)
results["uploaded"] += 1
self.session.mark_done(file_info)
if on_progress:
on_progress({
"total": structure["total_files"],
"uploaded": results["uploaded"],
"skipped": results["skipped"],
"failed": results["failed"],
"current_file": file_info["name"],
"current_size": file_info["size"]
})
except Exception as e:
results["failed"] += 1
results["errors"].append({
"file": file_info["name"],
"error": str(e)
})
self.session.mark_failed(file_info)
# Checkpoint por batch
self.session.save_checkpoint()
return results
async def _upload_file_with_retry(self,
local_path: str,
remote_path: str,
max_retries: int = 3):
"""Upload com retries e backoff exponencial."""
for attempt in range(max_retries):
try:
self.api.upload_file(
path_or_fileobj=local_path,
path_in_repo=remote_path,
repo_id=self.repo_id,
repo_type="dataset"
)
return
except Exception as e:
if attempt < max_retries - 1:
wait_time = 2 ** attempt + random.uniform(0, 1)
await asyncio.sleep(wait_time)
else:
raise
def _should_skip(self, file_info: dict) -> bool:
"""Verifica se arquivo jΓ‘ existe remotamente."""
try:
remote_path = f"audio/{self.project_slug}/{specie}/{file_info['name']}"
file_info_remote = self.api.get_file_info(
repo_id=self.repo_id,
filename=remote_path
)
return file_info_remote.size == file_info["size"]
except:
return False
4.5 Session Manager (src/session/session_manager.py)
class SessionManager:
def __init__(self, repo_id: str):
self.session_id = f"upload-{datetime.now().strftime('%Y%m%d-%H%M%S')}"
self.session_dir = Path(f"~/.birdnet-uploader/sessions/{self.session_id}")
self.session_dir.mkdir(parents=True, exist_ok=True)
def save_checkpoint(self, data: dict):
"""Salva checkpoint atomicamente."""
checkpoint_file = self.session_dir / "checkpoint.json"
temp_file = checkpoint_file.with_suffix(".tmp")
temp_file.write_text(json.dumps(data, indent=2))
temp_file.replace(checkpoint_file)
def load_checkpoint(self) -> dict:
"""Carrega ΓΊltimo checkpoint."""
checkpoint_file = self.session_dir / "checkpoint.json"
if checkpoint_file.exists():
return json.loads(checkpoint_file.read_text())
return {}
4.6 CSV Matcher (src/csv/csv_matcher.py)
class CSVMatcher:
def match_and_generate_index(self,
csv_path: str,
files_structure: dict) -> pd.DataFrame:
"""Faz matching entre CSV e segmentos, gera Γndice."""
df = pd.read_csv(csv_path)
# Validar colunas esperadas
required_cols = ["source_file", "scientific_name", "confidence",
"start_time", "end_time"]
if not all(col in df.columns for col in required_cols):
raise ValueError(f"CSV deve conter colunas: {required_cols}")
# Matching lΓ³gica
matched = []
for _, row in df.iterrows():
source_stem = Path(row["source_file"]).stem
# Procurar arquivo correspondente
for specie, files in files_structure["by_specie"].items():
for file_info in files:
if source_stem in file_info["name"]:
matched.append({
**row,
"segment_path": f"audio/{self.project_slug}/{specie}/{file_info['name']}"
})
return pd.DataFrame(matched)
def generate_shards(self,
index_df: pd.DataFrame,
output_dir: str,
shard_size: int = 10000) -> list:
"""Gera shards parquet para Γndice."""
shards = []
for i in range(0, len(index_df), shard_size):
shard_df = index_df.iloc[i:i+shard_size]
shard_file = f"{output_dir}/shard-{i//shard_size:05d}.parquet"
shard_df.to_parquet(shard_file, index=False)
shards.append(shard_file)
return shards
5. FLUXO DE COMANDOS CLI
# Autenticar
$ birdnet auth login
> Enter HF token: hf_...
# Upload simples (interactive)
$ birdnet upload
# Upload com flags (nΓ£o-interativo)
$ birdnet upload \
--repo alice/my-birds \
--segments /path/to/segments \
--csv /path/to/metadata.csv \
--private
# Listar sessΓ΅es
$ birdnet sessions list
# Retomar sessΓ£o
$ birdnet upload resume <session-id>
# Ver status
$ birdnet upload status <session-id>
# Exportar relatΓ³rio
$ birdnet upload report <session-id>
6. CRITΓRIOS DE ACEITE DO MVP
- UsuΓ‘rio baixa executΓ‘vel portΓ‘til, sem Python prΓ©-instalado
- Login com token HF funciona e nΓ£o vaza em logs
- Criar dataset pΓΊblico/privado com estrutura padrΓ£o
- Varrer pasta local corretamente (recursivo, agrupa por espΓ©cie)
- Upload de 50GB+ com progresso contΓnuo e ETA coerente
- Pausar/fechar app/retomar sem duplicar upload
- Queda de rede + 429 recuperam com retries
- CSV opcional com matching integrado
- RelatΓ³rio final estruturado (JSONL/JSON)
- Estrutura de arquivos compatΓvel com validador
- Compatibilidade 100% com manifest/index esperado
- Sem regressΓ£o do fluxo atual do Space
7. PRΓXIMOS PASSOS
- Implementar mΓ³dulos base (Auth, Repo, Scanner, Uploader)
- Construir UI (Textual TUI ou Rich + Click)
- Integrar Session Manager e Checkpoint
- Testes de integraΓ§Γ£o (50GB+, retomada, rede)
- Construir executΓ‘vel (PyInstaller)
- ValidaΓ§Γ£o end-to-end com usuΓ‘rios reais