BirdNET-Uploader-App / BIRDNET_UPLOADER_REDESIGN.md
jrrribeiro's picture
ForΓ§ar atualizaΓ§Γ£o total do app
2f1a5fb
|
Raw History Blame Contribute Delete
25.4 kB

A newer version of the Gradio SDK is available: 6.32.0

Upgrade

BirdNET Uploader: Redesign CLI Moderno

VisΓ£o Geral

ExecutΓ‘vel local moderno com um ΓΊnico fluxo integrado: login β†’ criar repositΓ³rio β†’ apontar pasta β†’ fazer upload.

Interface limpa, baseada em TUI (Textual) ou rich terminal output, compatΓ­vel com a estrutura esperada pelo BirdNET Validator.


1. FLUXO DE UX ÚNICO INTEGRADO

Tela 1: AutenticaΓ§Γ£o

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚  BirdNET Uploader                               β”‚
β”‚  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  β”‚
β”‚                                                 β”‚
β”‚  🐦  Autentique-se no Hugging Face             β”‚
β”‚                                                 β”‚
β”‚  [i] Cole seu token HF (Settings β†’ Access     β”‚
β”‚      Tokens em huggingface.co)                 β”‚
β”‚                                                 β”‚
β”‚  Token: [●●●●●●●●●●●●●●●●●●●●●●]             β”‚
β”‚                                                 β”‚
β”‚         [ Continuar ] [ Sair ]                 β”‚
β”‚                                                 β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

Tela 2: ConfiguraΓ§Γ£o (Um formulΓ‘rio integrado)

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚  BirdNET Uploader - ConfiguraΓ§Γ£o                β”‚
β”‚  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  β”‚
β”‚                                                 β”‚
β”‚  Conectado como: alice@example.com             β”‚
β”‚                                                 β”‚
β”‚  REPOSITΓ“RIO                                    β”‚
β”‚  Nome do repo:     [my-birds-2026    ]          β”‚
β”‚  Visibilidade:     ( ) PΓΊblico   (●) Privado   β”‚
β”‚                                                 β”‚
β”‚  ARQUIVOS                                       β”‚
β”‚  Pasta segmentos:  [C:\audio\segments]          β”‚
β”‚  Arquivo CSV:      [C:\audio\metadata.csv] (opt)
β”‚  [ Verificar ]                                  β”‚
β”‚  βœ“ 1.247 arquivos encontrados                  β”‚
β”‚  βœ“ Tamanho total: 53 GB                        β”‚
β”‚  βœ“ CSV compatΓ­vel: 1.200 linhas                β”‚
β”‚                                                 β”‚
β”‚         [ Criar Repo e Fazer Upload ]          β”‚
β”‚         [ Voltar ]                             β”‚
β”‚                                                 β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

Tela 3: Resumo e ConfirmaΓ§Γ£o

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚  Pronto para ComeΓ§ar                            β”‚
β”‚  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  β”‚
β”‚                                                 β”‚
β”‚  RepositΓ³rio: alice/my-birds-2026               β”‚
β”‚  Visibilidade: Privado                          β”‚
β”‚  URL: https://huggingface.co/datasets/alice/..  β”‚
β”‚                                                 β”‚
β”‚  Origem:                                        β”‚
β”‚    Pasta: C:\audio\segments                     β”‚
β”‚    Arquivos: 1.247                              β”‚
β”‚    Tamanho: 53 GB                               β”‚
β”‚    CSV: metadata.csv (1.200 linhas)             β”‚
β”‚                                                 β”‚
β”‚  DuraΓ§Γ£o estimada: ~4 horas (com conexΓ£o 10MB/s)
β”‚                                                 β”‚
β”‚  [ Iniciar Upload ]  [ Cancelar ]               β”‚
β”‚                                                 β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

Tela 4: Upload em Andamento

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚  Upload em Andamento                            β”‚
β”‚  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  β”‚
β”‚                                                 β”‚
β”‚  Progresso Global                               β”‚
β”‚  β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘  62.3%          β”‚
β”‚                                                 β”‚
β”‚  Arquivos: 775/1.247 (skipped: 18, erro: 2)   β”‚
β”‚  Dados: 33 GB / 53 GB                           β”‚
β”‚  Velocidade: 9.6 MB/s | ETA: 01h 15m            β”‚
β”‚                                                 β”‚
β”‚  Arquivo Atual: Parrot_Ara_001.wav              β”‚
β”‚  β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘  45.2%        β”‚
β”‚  3.2 / 7.1 MB                                   β”‚
β”‚                                                 β”‚
β”‚  Último evento: βœ“ Thrush_042.wav enviado (2s)  β”‚
β”‚                                                 β”‚
β”‚  [ Pausar ]  [ Cancelar ]  [ Detalhes ]         β”‚
β”‚                                                 β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

Tela 5: Resumo Final

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚  βœ“ Upload ConcluΓ­do!                            β”‚
β”‚  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  β”‚
β”‚                                                 β”‚
β”‚  RepositΓ³rio: alice/my-birds-2026               β”‚
β”‚  URL: https://huggingface.co/datasets/alice/..  β”‚
β”‚                                                 β”‚
β”‚  Resumo:                                        β”‚
β”‚    βœ“ Enviados: 1.227 arquivos                  β”‚
β”‚    β†’ Pulados: 18 (jΓ‘ existiam)                  β”‚
β”‚    βœ— Erros: 2 (revisar detalhes)               β”‚
β”‚    ⏱  Tempo total: 4h 23min                     β”‚
β”‚    πŸ“Š Velocidade mΓ©dia: 9.8 MB/s                β”‚
β”‚                                                 β”‚
β”‚  PrΓ³ximos passos:                               β”‚
β”‚    1. Validar dados com BirdNET Validator      β”‚
β”‚    2. Verificar relatΓ³rio de ingestΓ£o          β”‚
β”‚                                                 β”‚
β”‚  RelatΓ³rio salvo em: ~/.birdnet-uploader/      β”‚
β”‚                      sessions/upload-xxx/      β”‚
β”‚                      report.json                β”‚
β”‚                                                 β”‚
β”‚  [ Ver RelatΓ³rio Completo ]  [ Sair ]           β”‚
β”‚                                                 β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

2. ARQUITETURA LIMPA E MODULAR

Estrutura de DiretΓ³rios do Projeto

birdnet-uploader/
β”œβ”€β”€ src/
β”‚   β”œβ”€β”€ __init__.py
β”‚   β”œβ”€β”€ main.py                    # Entrypoint principal
β”‚   β”œβ”€β”€ config.py                  # Config app e constantes
β”‚   β”œβ”€β”€ auth/
β”‚   β”‚   β”œβ”€β”€ __init__.py
β”‚   β”‚   β”œβ”€β”€ auth_service.py        # Login, token, keyring
β”‚   β”‚   └── token_manager.py       # Gerenciamento de token
β”‚   β”œβ”€β”€ repository/
β”‚   β”‚   β”œβ”€β”€ __init__.py
β”‚   β”‚   β”œβ”€β”€ repo_service.py        # Criar/validar dataset HF
β”‚   β”‚   └── structure_initializer.py # Init estrutura padrΓ£o
β”‚   β”œβ”€β”€ upload/
β”‚   β”‚   β”œβ”€β”€ __init__.py
β”‚   β”‚   β”œβ”€β”€ scanner.py             # Varrer pasta local
β”‚   β”‚   β”œβ”€β”€ uploader.py            # Motor de upload por lotes
β”‚   β”‚   β”œβ”€β”€ deduplicator.py        # Checagem de duplicaΓ§Γ£o
β”‚   β”‚   └── batch_processor.py     # Processamento de lotes
β”‚   β”œβ”€β”€ csv/
β”‚   β”‚   β”œβ”€β”€ __init__.py
β”‚   β”‚   β”œβ”€β”€ csv_matcher.py         # Matching CSV + segmentos
β”‚   β”‚   └── indexer.py             # GeraΓ§Γ£o de Γ­ndice/shards
β”‚   β”œβ”€β”€ session/
β”‚   β”‚   β”œβ”€β”€ __init__.py
β”‚   β”‚   β”œβ”€β”€ session_manager.py     # Gerenciamento de sessΓ£o
β”‚   β”‚   β”œβ”€β”€ checkpoint.py          # PersistΓͺncia de checkpoint
β”‚   β”‚   └── state_store.py         # State file I/O
β”‚   β”œβ”€β”€ ui/
β”‚   β”‚   β”œβ”€β”€ __init__.py
β”‚   β”‚   β”œβ”€β”€ tui.py                 # Interface Textual (TUI)
β”‚   β”‚   β”œβ”€β”€ progress_renderer.py   # Barra de progresso rich
β”‚   β”‚   └── prompts.py             # DiΓ‘logos e inputs
β”‚   β”œβ”€β”€ logger/
β”‚   β”‚   β”œβ”€β”€ __init__.py
β”‚   β”‚   β”œβ”€β”€ structured_logger.py   # Logs estruturados JSONL
β”‚   β”‚   └── telemetry.py           # Eventos de telemetria
β”‚   └── utils/
β”‚       β”œβ”€β”€ __init__.py
β”‚       β”œβ”€β”€ file_utils.py          # OperaΓ§Γ΅es de arquivo
β”‚       β”œβ”€β”€ hash_utils.py          # ComputaΓ§Γ£o de hash
β”‚       └── error_handler.py       # Tratamento de erros
β”œβ”€β”€ build/
β”‚   β”œβ”€β”€ pyinstaller_config.spec    # Config do PyInstaller
β”‚   └── build.py                   # Script de build
β”œβ”€β”€ tests/
β”‚   β”œβ”€β”€ __init__.py
β”‚   β”œβ”€β”€ test_auth.py
β”‚   β”œβ”€β”€ test_uploader.py
β”‚   β”œβ”€β”€ test_session.py
β”‚   └── test_csv_matcher.py
β”œβ”€β”€ README.md
β”œβ”€β”€ requirements.txt
└── setup.py

DependΓͺncias Principais

# requirements.txt
huggingface_hub>=0.28.0          # HF Hub API
click>=8.0.0                      # CLI framework
textual>=0.30.0                   # TUI moderna
rich>=13.0.0                      # Terminal formatting
pydantic>=2.0.0                   # ValidaΓ§Γ£o de dados
pandas>=2.0.0                     # CSV parsing
pyarrow>=14.0.0                   # Parquet para shards
keyring>=24.0.0                   # Armazenamento seguro
cryptography>=41.0.0              # Criptografia local

3. COMPATIBILIDADE COM ESTRUTURA DO VALIDADOR

Estrutura Esperada do Dataset

dataset_repo_id = "username/project-dataset"

audio/
β”œβ”€β”€ {project_slug}/
β”‚   β”œβ”€β”€ Specie_A/
β”‚   β”‚   β”œβ”€β”€ segment_001.wav
β”‚   β”‚   β”œβ”€β”€ segment_002.wav
β”‚   β”‚   └── ...
β”‚   β”œβ”€β”€ Specie_B/
β”‚   β”‚   └── ...
β”‚   └── ...

index/
β”œβ”€β”€ shards/
β”‚   β”œβ”€β”€ shard-00000.parquet
β”‚   β”œβ”€β”€ shard-00001.parquet
β”‚   └── ...
└── manifest.json

validations/
β”œβ”€β”€ results-20250419.parquet

audit/
β”œβ”€β”€ ingestion-runs/
β”‚   └── 20250419T100000Z.json

Manifest Format (MantΓ©m compatibilidade)

{
  "schema_version": "1.0.0",
  "project_slug": "my-birds",
  "dataset_repo_id": "alice/my-birds-2026",
  "updated_at": "2025-04-19T10:45:30Z",
  "index": {
    "total_detections": 1227,
    "total_audio_files": 1227,
    "shard_size": 10000,
    "shards": [
      {
        "path": "index/shards/shard-00000.parquet",
        "rows": 10000,
        "sha256": "abc123...",
        "size_bytes": 5242880
      }
    ]
  }
}

4. MΓ“DULO POR MΓ“DULO

4.1 Auth Service (src/auth/auth_service.py)

from keyring import get_password, set_password
from huggingface_hub import HfApi, login

class AuthService:
    KEYRING_SERVICE = "birdnet-uploader"
    
    def authenticate(self, token: str) -> dict:
        """Valida token e retorna context do usuΓ‘rio."""
        try:
            api = HfApi(token=token)
            user_info = api.whoami()
            
            # Salvar em keyring seguro
            set_password(self.KEYRING_SERVICE, "hf_token", token)
            
            return {
                "username": user_info["name"],
                "email": user_info.get("email", ""),
                "user_id": user_info["user_id"],
                "token_expiry": self._estimate_expiry()
            }
        except Exception as e:
            raise AuthenticationError(f"Token invΓ‘lido: {str(e)}")
    
    def get_token(self) -> str:
        """Recupera token do keyring."""
        token = get_password(self.KEYRING_SERVICE, "hf_token")
        if not token:
            raise TokenNotFoundError("Token nΓ£o encontrado. Execute 'birdnet auth login'")
        return token

4.2 Repository Service (src/repository/repo_service.py)

class RepositoryService:
    def create_dataset(self, 
                       repo_name: str, 
                       private: bool = True) -> str:
        """Cria dataset no HF com estrutura padrΓ£o."""
        api = HfApi(token=self.get_token())
        
        repo_id = f"{username}/{repo_name}"
        
        # Criar repo
        repo = api.create_repo(
            repo_id=repo_id,
            repo_type="dataset",
            private=private,
            exist_ok=True
        )
        
        # Inicializar estrutura padrΓ£o
        self._init_structure(repo_id)
        
        return repo_id
    
    def _init_structure(self, repo_id: str):
        """Inicializa pastas e arquivos padrΓ£o."""
        folders = [
            "audio/{project_slug}",
            "index/shards",
            "validations",
            "audit/ingestion-runs"
        ]
        
        for folder in folders:
            # Criar .gitkeep
            api.upload_file(
                path_or_fileobj=BytesIO(b""),
                path_in_repo=f"{folder}/.gitkeep",
                repo_id=repo_id,
                repo_type="dataset"
            )
        
        # Criar manifest.json inicial
        manifest = {
            "schema_version": "1.0.0",
            "project_slug": project_slug,
            "dataset_repo_id": repo_id,
            "index": {
                "total_detections": 0,
                "total_audio_files": 0,
                "shard_size": 10000,
                "shards": []
            }
        }
        
        api.upload_file(
            path_or_fileobj=BytesIO(json.dumps(manifest).encode()),
            path_in_repo="index/manifest.json",
            repo_id=repo_id,
            repo_type="dataset"
        )

4.3 Scanner (src/upload/scanner.py)

class LocalScanner:
    def scan_folder(self, folder_path: str) -> dict:
        """Varre pasta recursiva e retorna estrutura."""
        structure = {}
        total_size = 0
        file_count = 0
        
        for root, dirs, files in os.walk(folder_path):
            for file in files:
                if file.lower().endswith(('.wav', '.mp3', '.flac', '.ogg')):
                    full_path = os.path.join(root, file)
                    
                    # Inferir espΓ©cie do caminho ou nome
                    specie = self._infer_specie(root, file)
                    
                    if specie not in structure:
                        structure[specie] = []
                    
                    size = os.path.getsize(full_path)
                    structure[specie].append({
                        "name": file,
                        "full_path": full_path,
                        "size": size,
                        "local_hash": compute_hash(full_path)
                    })
                    
                    total_size += size
                    file_count += 1
        
        return {
            "total_files": file_count,
            "total_size": total_size,
            "by_specie": structure,
            "scan_timestamp": datetime.now()
        }
    
    def _infer_specie(self, root: str, file: str) -> str:
        """Inferir espΓ©cie do caminho da pasta."""
        # EstratΓ©gia 1: ΓΊltima pasta em root Γ© a espΓ©cie
        parts = root.split(os.sep)
        if len(parts) > 0:
            return parts[-1]
        
        # Fallback: usar nome do arquivo
        return file.replace(".wav", "").split("_")[0]

4.4 Uploader (src/upload/uploader.py)

class BatchUploader:
    def __init__(self, repo_id: str, project_slug: str, token: str):
        self.repo_id = repo_id
        self.project_slug = project_slug
        self.api = HfApi(token=token)
        self.session = SessionManager(repo_id)
    
    async def upload_files(self, 
                           structure: dict,
                           csv_matcher: Optional[CSVMatcher] = None,
                           batch_size: int = 10,
                           on_progress: Optional[Callable] = None) -> dict:
        """Upload em lotes com retomada."""
        checkpoint = self.session.load_checkpoint()
        
        results = {
            "total": 0,
            "uploaded": 0,
            "skipped": 0,
            "failed": 0,
            "errors": []
        }
        
        for specie, files in structure["by_specie"].items():
            # Processar em lotes
            for i in range(0, len(files), batch_size):
                batch = files[i:i + batch_size]
                
                # Verificar checkpoint
                if self.session.is_batch_done(specie, i):
                    results["skipped"] += len(batch)
                    continue
                
                # Enviar batch
                for file_info in batch:
                    try:
                        if self._should_skip(file_info):
                            results["skipped"] += 1
                            self.session.mark_skipped(file_info)
                            continue
                        
                        remote_path = f"audio/{self.project_slug}/{specie}/{file_info['name']}"
                        
                        await self._upload_file_with_retry(
                            file_info["full_path"],
                            remote_path
                        )
                        
                        results["uploaded"] += 1
                        self.session.mark_done(file_info)
                        
                        if on_progress:
                            on_progress({
                                "total": structure["total_files"],
                                "uploaded": results["uploaded"],
                                "skipped": results["skipped"],
                                "failed": results["failed"],
                                "current_file": file_info["name"],
                                "current_size": file_info["size"]
                            })
                    
                    except Exception as e:
                        results["failed"] += 1
                        results["errors"].append({
                            "file": file_info["name"],
                            "error": str(e)
                        })
                        self.session.mark_failed(file_info)
                
                # Checkpoint por batch
                self.session.save_checkpoint()
        
        return results
    
    async def _upload_file_with_retry(self, 
                                       local_path: str, 
                                       remote_path: str,
                                       max_retries: int = 3):
        """Upload com retries e backoff exponencial."""
        for attempt in range(max_retries):
            try:
                self.api.upload_file(
                    path_or_fileobj=local_path,
                    path_in_repo=remote_path,
                    repo_id=self.repo_id,
                    repo_type="dataset"
                )
                return
            except Exception as e:
                if attempt < max_retries - 1:
                    wait_time = 2 ** attempt + random.uniform(0, 1)
                    await asyncio.sleep(wait_time)
                else:
                    raise
    
    def _should_skip(self, file_info: dict) -> bool:
        """Verifica se arquivo jΓ‘ existe remotamente."""
        try:
            remote_path = f"audio/{self.project_slug}/{specie}/{file_info['name']}"
            file_info_remote = self.api.get_file_info(
                repo_id=self.repo_id,
                filename=remote_path
            )
            return file_info_remote.size == file_info["size"]
        except:
            return False

4.5 Session Manager (src/session/session_manager.py)

class SessionManager:
    def __init__(self, repo_id: str):
        self.session_id = f"upload-{datetime.now().strftime('%Y%m%d-%H%M%S')}"
        self.session_dir = Path(f"~/.birdnet-uploader/sessions/{self.session_id}")
        self.session_dir.mkdir(parents=True, exist_ok=True)
    
    def save_checkpoint(self, data: dict):
        """Salva checkpoint atomicamente."""
        checkpoint_file = self.session_dir / "checkpoint.json"
        temp_file = checkpoint_file.with_suffix(".tmp")
        
        temp_file.write_text(json.dumps(data, indent=2))
        temp_file.replace(checkpoint_file)
    
    def load_checkpoint(self) -> dict:
        """Carrega ΓΊltimo checkpoint."""
        checkpoint_file = self.session_dir / "checkpoint.json"
        if checkpoint_file.exists():
            return json.loads(checkpoint_file.read_text())
        return {}

4.6 CSV Matcher (src/csv/csv_matcher.py)

class CSVMatcher:
    def match_and_generate_index(self,
                                 csv_path: str,
                                 files_structure: dict) -> pd.DataFrame:
        """Faz matching entre CSV e segmentos, gera Γ­ndice."""
        df = pd.read_csv(csv_path)
        
        # Validar colunas esperadas
        required_cols = ["source_file", "scientific_name", "confidence", 
                        "start_time", "end_time"]
        if not all(col in df.columns for col in required_cols):
            raise ValueError(f"CSV deve conter colunas: {required_cols}")
        
        # Matching lΓ³gica
        matched = []
        for _, row in df.iterrows():
            source_stem = Path(row["source_file"]).stem
            
            # Procurar arquivo correspondente
            for specie, files in files_structure["by_specie"].items():
                for file_info in files:
                    if source_stem in file_info["name"]:
                        matched.append({
                            **row,
                            "segment_path": f"audio/{self.project_slug}/{specie}/{file_info['name']}"
                        })
        
        return pd.DataFrame(matched)
    
    def generate_shards(self,
                       index_df: pd.DataFrame,
                       output_dir: str,
                       shard_size: int = 10000) -> list:
        """Gera shards parquet para Γ­ndice."""
        shards = []
        
        for i in range(0, len(index_df), shard_size):
            shard_df = index_df.iloc[i:i+shard_size]
            shard_file = f"{output_dir}/shard-{i//shard_size:05d}.parquet"
            
            shard_df.to_parquet(shard_file, index=False)
            shards.append(shard_file)
        
        return shards

5. FLUXO DE COMANDOS CLI

# Autenticar
$ birdnet auth login
> Enter HF token: hf_...

# Upload simples (interactive)
$ birdnet upload

# Upload com flags (nΓ£o-interativo)
$ birdnet upload \
    --repo alice/my-birds \
    --segments /path/to/segments \
    --csv /path/to/metadata.csv \
    --private

# Listar sessΓ΅es
$ birdnet sessions list

# Retomar sessΓ£o
$ birdnet upload resume <session-id>

# Ver status
$ birdnet upload status <session-id>

# Exportar relatΓ³rio
$ birdnet upload report <session-id>

6. CRITÉRIOS DE ACEITE DO MVP

  • UsuΓ‘rio baixa executΓ‘vel portΓ‘til, sem Python prΓ©-instalado
  • Login com token HF funciona e nΓ£o vaza em logs
  • Criar dataset pΓΊblico/privado com estrutura padrΓ£o
  • Varrer pasta local corretamente (recursivo, agrupa por espΓ©cie)
  • Upload de 50GB+ com progresso contΓ­nuo e ETA coerente
  • Pausar/fechar app/retomar sem duplicar upload
  • Queda de rede + 429 recuperam com retries
  • CSV opcional com matching integrado
  • RelatΓ³rio final estruturado (JSONL/JSON)
  • Estrutura de arquivos compatΓ­vel com validador
  • Compatibilidade 100% com manifest/index esperado
  • Sem regressΓ£o do fluxo atual do Space

7. PRΓ“XIMOS PASSOS

  1. Implementar mΓ³dulos base (Auth, Repo, Scanner, Uploader)
  2. Construir UI (Textual TUI ou Rich + Click)
  3. Integrar Session Manager e Checkpoint
  4. Testes de integraΓ§Γ£o (50GB+, retomada, rede)
  5. Construir executΓ‘vel (PyInstaller)
  6. ValidaΓ§Γ£o end-to-end com usuΓ‘rios reais