BirdNET-Uploader-App / SPRINT1_KICKOFF.md
jrrribeiro's picture
Forçar atualização total do app
2f1a5fb
|
Raw History Blame Contribute Delete
17.8 kB

A newer version of the Gradio SDK is available: 6.32.0

Upgrade

Sprint 1 Kickoff: Primeiros Passos (HOJE)

Objetivo

Estrutura base pronta, dependências instaladas, todos os módulos esqueléticos criados, testes rodando.

Tempo estimado: 4-5 horas para 1-2 pessoas


Step 1: Criar Branch de Trabalho

cd c:\Users\jonat\Documents\Python\BirdNET-validator-App
git checkout -b feature/birdnet-uploader-cli
git push -u origin feature/birdnet-uploader-cli

Step 2: Criar Estrutura de Diretórios

# Criar estrutura completa
mkdir -p src\auth
mkdir -p src\repository
mkdir -p src\upload
mkdir -p src\csv
mkdir -p src\session
mkdir -p src\ui\screens
mkdir -p src\logger
mkdir -p src\utils
mkdir -p tests
mkdir -p build
mkdir -p docs

# Criar __init__.py em todos
echo "" > src\__init__.py
echo "" > src\auth\__init__.py
echo "" > src\repository\__init__.py
echo "" > src\upload\__init__.py
echo "" > src\csv\__init__.py
echo "" > src\session\__init__.py
echo "" > src\ui\__init__.py
echo "" > src\ui\screens\__init__.py
echo "" > src\logger\__init__.py
echo "" > src\utils\__init__.py
echo "" > tests\__init__.py

Step 3: Criar requirements.txt

Criar arquivo: requirements.txt

# Core CLI
click>=8.0.0
typer>=0.9.0

# UI/TUI
textual>=0.30.0
rich>=13.0.0

# HF Integration
huggingface_hub>=0.28.0

# Data Processing
pandas>=2.0.0
pyarrow>=14.0.0

# Security
keyring>=24.0.0
cryptography>=41.0.0

# Validation
pydantic>=2.0.0

# Development & Testing
pytest>=7.0.0
pytest-cov>=4.0.0
black>=23.0.0
flake8>=6.0.0
mypy>=1.0.0

# Optional: async
asyncio>=3.4.3

Instalar:

pip install -r requirements.txt

Step 4: Criar pyproject.toml

Criar arquivo: pyproject.toml

[build-system]
requires = ["setuptools>=65.0", "wheel"]
build-backend = "setuptools.build_meta"

[project]
name = "birdnet-uploader"
version = "0.1.0"
description = "Local CLI for resumable HuggingFace dataset uploads (BirdNET)"
authors = [
    {name = "BirdNET Team", email = "team@example.com"}
]
requires-python = ">=3.9"
dependencies = [
    "click>=8.0.0",
    "textual>=0.30.0",
    "rich>=13.0.0",
    "huggingface_hub>=0.28.0",
    "pandas>=2.0.0",
    "pyarrow>=14.0.0",
    "keyring>=24.0.0",
    "pydantic>=2.0.0",
]

[project.optional-dependencies]
dev = [
    "pytest>=7.0.0",
    "pytest-cov>=4.0.0",
    "black>=23.0.0",
    "flake8>=6.0.0",
    "mypy>=1.0.0",
]

[project.scripts]
birdnet = "src.main:app"

[tool.black]
line-length = 100
target-version = ['py39']

[tool.isort]
profile = "black"
line_length = 100

[tool.mypy]
python_version = "3.9"
warn_return_any = true
warn_unused_configs = true

[tool.pytest.ini_options]
testpaths = ["tests"]
addopts = "-v --cov=src --cov-report=html --cov-report=term-missing"

Step 5: Criar Estrutura Base de Módulos (Skeletons)

5.1: src/main.py

"""BirdNET Uploader - Main Entry Point"""

import click
from src.ui.tui import BirdNETApp
from src.auth.auth_service import AuthService
from src.config import get_config

__version__ = "0.1.0"

@click.group()
@click.version_option(version=__version__)
def app():
    """BirdNET Uploader - Upload audio to HuggingFace Datasets"""
    pass

@app.command()
def upload():
    """Start the upload wizard (interactive TUI)"""
    try:
        app_ui = BirdNETApp()
        app_ui.run()
    except Exception as e:
        click.echo(f"Error: {e}", err=True)

@app.command()
def login():
    """Authenticate with HuggingFace token"""
    try:
        token = click.prompt("Enter your HF token", hide_input=True)
        auth = AuthService()
        user_info = auth.authenticate(token)
        click.echo(f"✓ Logged in as {user_info['username']}")
    except Exception as e:
        click.echo(f"Error: {e}", err=True)

@app.command()
@click.argument("session_id")
def resume(session_id: str):
    """Resume a previous upload session"""
    click.echo(f"Resuming session: {session_id}")
    # TODO: Implement

@app.command()
def list_sessions():
    """List all upload sessions"""
    click.echo("Sessions:")
    # TODO: Implement

if __name__ == "__main__":
    app()

5.2: src/config.py

"""Configuration and Constants"""

from pathlib import Path
import os

# App
APP_NAME = "BirdNET Uploader"
APP_VERSION = "0.1.0"

# Directories
SESSION_DIR = Path.home() / ".birdnet-uploader" / "sessions"
CACHE_DIR = Path.home() / ".birdnet-uploader" / "cache"
LOG_DIR = Path.home() / ".birdnet-uploader" / "logs"

# Create directories if not exist
SESSION_DIR.mkdir(parents=True, exist_ok=True)
CACHE_DIR.mkdir(parents=True, exist_ok=True)
LOG_DIR.mkdir(parents=True, exist_ok=True)

# Audio
AUDIO_EXTENSIONS = ['.wav', '.mp3', '.flac', '.ogg', '.m4a']

# Upload
MAX_BATCH_SIZE = 10
MAX_RETRIES = 3
RETRY_INITIAL_BACKOFF = 1.0  # seconds
RETRY_MAX_BACKOFF = 30.0  # seconds

# Manifest
MANIFEST_SCHEMA_VERSION = "1.0.0"
SHARD_SIZE = 10000

# Keyring
KEYRING_SERVICE = "birdnet-uploader"
KEYRING_ACCOUNT = "hf_token"

def get_config():
    """Get runtime configuration from environment or defaults"""
    return {
        "session_dir": os.getenv("BIRDNET_SESSION_DIR", str(SESSION_DIR)),
        "cache_dir": os.getenv("BIRDNET_CACHE_DIR", str(CACHE_DIR)),
        "log_dir": os.getenv("BIRDNET_LOG_DIR", str(LOG_DIR)),
    }

5.3: src/auth/auth_service.py

"""Authentication Service"""

from typing import Optional, Dict
import keyring
from huggingface_hub import HfApi, HfFolder
from src.config import KEYRING_SERVICE, KEYRING_ACCOUNT

class AuthenticationError(Exception):
    """Authentication failed"""
    pass

class AuthService:
    """Manage authentication with HuggingFace"""
    
    def __init__(self):
        self.api = None
    
    def authenticate(self, token: str) -> Dict[str, str]:
        """
        Validate token and store it securely.
        
        Args:
            token: HF API token
            
        Returns:
            User info dict with username, email, user_id
            
        Raises:
            AuthenticationError: If token is invalid
        """
        try:
            api = HfApi(token=token)
            user_info = api.whoami()
            
            # Store token in keyring
            keyring.set_password(KEYRING_SERVICE, KEYRING_ACCOUNT, token)
            
            self.api = api
            return {
                "username": user_info.get("name"),
                "email": user_info.get("email", ""),
                "user_id": user_info.get("user_id"),
            }
        except Exception as e:
            raise AuthenticationError(f"Invalid token: {str(e)}")
    
    def get_token(self) -> Optional[str]:
        """Get stored token from keyring"""
        return keyring.get_password(KEYRING_SERVICE, KEYRING_ACCOUNT)
    
    def clear_token(self):
        """Clear stored token"""
        keyring.delete_password(KEYRING_SERVICE, KEYRING_ACCOUNT)

5.4: src/repository/repo_service.py

"""Repository Service"""

from typing import Optional
from huggingface_hub import HfApi
import json
from io import BytesIO

class RepositoryError(Exception):
    """Repository operation failed"""
    pass

class RepositoryService:
    """Manage HuggingFace datasets"""
    
    def __init__(self, token: str):
        self.api = HfApi(token=token)
    
    def create_dataset(self, repo_name: str, private: bool = True) -> str:
        """
        Create a new dataset on HuggingFace.
        
        Args:
            repo_name: Dataset name
            private: Whether dataset should be private
            
        Returns:
            Full repo_id (username/repo_name)
        """
        try:
            # Create repo
            repo = self.api.create_repo(
                repo_id=repo_name,
                repo_type="dataset",
                private=private,
                exist_ok=True
            )
            repo_id = repo.repo_id
            
            # Initialize structure
            self._init_structure(repo_id)
            
            return repo_id
        except Exception as e:
            raise RepositoryError(f"Failed to create dataset: {str(e)}")
    
    def _init_structure(self, repo_id: str):
        """Initialize default folder structure"""
        folders = [
            "audio/project",
            "index/shards",
            "validations",
            "audit/ingestion-runs"
        ]
        
        for folder in folders:
            # Upload placeholder file to create folder
            self.api.upload_file(
                path_or_fileobj=BytesIO(b""),
                path_in_repo=f"{folder}/.gitkeep",
                repo_id=repo_id,
                repo_type="dataset"
            )
    
    def validate_repo(self, repo_id: str) -> bool:
        """Validate dataset has expected structure"""
        # TODO: Implement
        return True

5.5: src/upload/scanner.py

"""Local Folder Scanner"""

import os
from pathlib import Path
from typing import Dict, List
from src.config import AUDIO_EXTENSIONS

class LocalScanner:
    """Scan local folders for audio files"""
    
    def scan_folder(self, folder_path: str) -> Dict:
        """
        Recursively scan folder for audio files.
        
        Args:
            folder_path: Path to scan
            
        Returns:
            Structure dict with files grouped by species
        """
        structure = {}
        total_size = 0
        file_count = 0
        
        for root, dirs, files in os.walk(folder_path):
            for file in files:
                if any(file.lower().endswith(ext) for ext in AUDIO_EXTENSIONS):
                    full_path = os.path.join(root, file)
                    size = os.path.getsize(full_path)
                    
                    # Infer species
                    specie = self._infer_species(root, file)
                    
                    if specie not in structure:
                        structure[specie] = []
                    
                    structure[specie].append({
                        "name": file,
                        "full_path": full_path,
                        "size": size,
                    })
                    
                    total_size += size
                    file_count += 1
        
        return {
            "total_files": file_count,
            "total_size": total_size,
            "by_species": structure,
        }
    
    def _infer_species(self, root: str, filename: str) -> str:
        """Infer species from path or filename"""
        parts = root.split(os.sep)
        if len(parts) > 0:
            return parts[-1]
        return "unknown"

5.6: src/session/session_manager.py

"""Session Management"""

import json
from pathlib import Path
from datetime import datetime
from src.config import SESSION_DIR

class SessionManager:
    """Manage upload sessions"""
    
    def __init__(self, session_id: str = None):
        if session_id is None:
            session_id = f"upload-{datetime.now().strftime('%Y%m%d-%H%M%S')}"
        
        self.session_id = session_id
        self.session_dir = Path(SESSION_DIR) / session_id
        self.session_dir.mkdir(parents=True, exist_ok=True)
    
    def save_checkpoint(self, data: dict):
        """Save checkpoint atomically"""
        checkpoint_file = self.session_dir / "checkpoint.json"
        temp_file = checkpoint_file.with_suffix(".tmp")
        
        with open(temp_file, "w") as f:
            json.dump(data, f, indent=2)
        
        temp_file.replace(checkpoint_file)
    
    def load_checkpoint(self) -> dict:
        """Load last checkpoint"""
        checkpoint_file = self.session_dir / "checkpoint.json"
        if checkpoint_file.exists():
            with open(checkpoint_file) as f:
                return json.load(f)
        return {}

5.7: src/ui/tui.py

"""TUI Application (Textual)"""

from textual.app import ComposeResult, App
from textual.containers import Container
from textual.widgets import Static, Button

class BirdNETApp(App):
    """Main TUI Application"""
    
    def compose(self) -> ComposeResult:
        """Create child widgets for the app."""
        yield Container(
            Static("BirdNET Uploader", id="header"),
            Button("Start Upload", id="btn_upload"),
            id="container"
        )
    
    def on_button_pressed(self) -> None:
        """Handle button press."""
        self.exit()

def run():
    """Run the TUI app"""
    app = BirdNETApp()
    app.run()

5.8: src/utils/error_handler.py

"""Error Handling"""

class BirdNETError(Exception):
    """Base exception"""
    pass

class AuthenticationError(BirdNETError):
    """Authentication failed"""
    pass

class RepositoryError(BirdNETError):
    """Repository operation failed"""
    pass

class UploadError(BirdNETError):
    """Upload operation failed"""
    pass

class ValidationError(BirdNETError):
    """Validation failed"""
    pass

Step 6: Criar Test Infrastructure

6.1: pytest.ini

[pytest]
testpaths = tests
addopts = -v --tb=short
python_files = test_*.py
python_classes = Test*
python_functions = test_*

6.2: tests/conftest.py

"""Test Configuration and Fixtures"""

import pytest
from pathlib import Path
import tempfile
import json

@pytest.fixture
def temp_folder():
    """Create temporary folder with test audio files"""
    with tempfile.TemporaryDirectory() as tmpdir:
        # Create test structure
        audio_dir = Path(tmpdir) / "audio" / "Parrot"
        audio_dir.mkdir(parents=True)
        
        # Create dummy audio files
        for i in range(5):
            (audio_dir / f"segment_{i:03d}.wav").write_bytes(b"fake audio data")
        
        yield tmpdir

@pytest.fixture
def mock_hf_token():
    """Mock HF token"""
    return "hf_test_token_12345"

@pytest.fixture
def session_dir(tmp_path):
    """Temporary session directory"""
    session_dir = tmp_path / "sessions" / "test-session"
    session_dir.mkdir(parents=True)
    return session_dir

6.3: tests/test_auth.py

"""Tests for Auth Service"""

import pytest
from src.auth.auth_service import AuthService, AuthenticationError

def test_auth_invalid_token(mock_hf_token):
    """Test that invalid token raises error"""
    auth = AuthService()
    with pytest.raises(AuthenticationError):
        auth.authenticate("invalid_token_xyz")

# More tests to be implemented

6.4: tests/test_scanner.py

"""Tests for File Scanner"""

import pytest
from src.upload.scanner import LocalScanner

def test_scan_folder_empty(tmp_path):
    """Test scanning empty folder"""
    scanner = LocalScanner()
    result = scanner.scan_folder(str(tmp_path))
    assert result["total_files"] == 0
    assert result["total_size"] == 0

def test_scan_folder_with_audio(temp_folder):
    """Test scanning folder with audio files"""
    scanner = LocalScanner()
    result = scanner.scan_folder(temp_folder)
    assert result["total_files"] == 5
    assert result["total_size"] > 0

# More tests to be implemented

Step 7: Configurar CI/CD (GitHub Actions)

Criar arquivo: .github/workflows/test.yml

name: Tests

on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: ["3.9", "3.10", "3.11"]
    
    steps:
      - uses: actions/checkout@v3
      
      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: ${{ matrix.python-version }}
      
      - name: Install dependencies
        run: |
          pip install -r requirements.txt
      
      - name: Lint
        run: |
          flake8 src tests
      
      - name: Run tests
        run: |
          pytest
      
      - name: Upload coverage
        uses: codecov/codecov-action@v3

Step 8: Testar Instalação

# Instalar dependências
pip install -r requirements.txt

# Rodar testes
pytest -v

# Testar import
python -c "import src; print('✓ Imports OK')"

# Rodar CLI help
python -m src.main --help

Esperado: Todos os comandos rodarem sem erro.


Step 9: Commit Initial

git add -A
git commit -m "feat: Sprint 1 - Initial project structure and scaffolding"
git push origin feature/birdnet-uploader-cli

Checklist: Sprint 1, Day 1 - DONE ✓

  • Branch feature/birdnet-uploader-cli criada
  • Estrutura de diretórios completa
  • requirements.txt instalado
  • pyproject.toml criado
  • Todos os módulos esqueléticos criados (8+ arquivos)
  • pytest.ini e conftest.py configurados
  • Testes básicos criados
  • CI/CD workflow criado
  • Primeira execução sem erro: pytest -v
  • Commit feito e pushed

Tempo: ~2-3 horas


Próximos Passos (Task 1.2 - Design Review)

Quando tudo acima estiver DONE, fazer:

  1. Meeting de Design Review com o time (30min)

    • Confirmar Tech Stack (Textual vs Rich+Click)
    • Confirmar persistência (JSON vs SQLite)
    • Validar estrutura de repo HF
    • Assinar decision log
  2. Começar Task 2.1 (Auth Service Implementation)

    • Implementar AuthService.authenticate() completo
    • Implementar keyring integration
    • Escrever testes com mocks

Troubleshooting

Problema: pip install falha com erro de dependência Solução: Upgrade pip: pip install --upgrade pip

Problema: Imports falhando Solução: Adicionar export PYTHONPATH="${PYTHONPATH}:$(pwd)"

Problema: Pytest não encontra testes Solução: Verificar que todos os __init__.py existem


Documentação de Referência

  • BIRDNET_UPLOADER_REDESIGN.md (arquitetura e UX)
  • SPRINT_ROADMAP.md (timeline e tasks)
  • Este arquivo (Sprint 1 Day 1 setup)