Spaces:
Sleeping
Sleeping
|
Download SPRINT1_KICKOFF.md from jrrribeiro/BirdNET-Uploader-App: direct link, hf CLI and curl.
- Browser
- Download file 17.8 kB
-
https://huggingface.co/spaces/jrrribeiro/BirdNET-Uploader-App/resolve/main/SPRINT1_KICKOFF.md
- Command line
-
hf download hf://spaces/jrrribeiro/BirdNET-Uploader-App/SPRINT1_KICKOFF.md
-
curl -L -o SPRINT1_KICKOFF.md https://huggingface.co/spaces/jrrribeiro/BirdNET-Uploader-App/resolve/main/SPRINT1_KICKOFF.md
17.8 kB
| # Sprint 1 Kickoff: Primeiros Passos (HOJE) | |
| ## Objetivo | |
| Estrutura base pronta, dependências instaladas, todos os módulos esqueléticos criados, testes rodando. | |
| **Tempo estimado**: 4-5 horas para 1-2 pessoas | |
| --- | |
| ## Step 1: Criar Branch de Trabalho | |
| ```bash | |
| cd c:\Users\jonat\Documents\Python\BirdNET-validator-App | |
| git checkout -b feature/birdnet-uploader-cli | |
| git push -u origin feature/birdnet-uploader-cli | |
| ``` | |
| --- | |
| ## Step 2: Criar Estrutura de Diretórios | |
| ```bash | |
| # Criar estrutura completa | |
| mkdir -p src\auth | |
| mkdir -p src\repository | |
| mkdir -p src\upload | |
| mkdir -p src\csv | |
| mkdir -p src\session | |
| mkdir -p src\ui\screens | |
| mkdir -p src\logger | |
| mkdir -p src\utils | |
| mkdir -p tests | |
| mkdir -p build | |
| mkdir -p docs | |
| # Criar __init__.py em todos | |
| echo "" > src\__init__.py | |
| echo "" > src\auth\__init__.py | |
| echo "" > src\repository\__init__.py | |
| echo "" > src\upload\__init__.py | |
| echo "" > src\csv\__init__.py | |
| echo "" > src\session\__init__.py | |
| echo "" > src\ui\__init__.py | |
| echo "" > src\ui\screens\__init__.py | |
| echo "" > src\logger\__init__.py | |
| echo "" > src\utils\__init__.py | |
| echo "" > tests\__init__.py | |
| ``` | |
| --- | |
| ## Step 3: Criar `requirements.txt` | |
| Criar arquivo: `requirements.txt` | |
| ``` | |
| # Core CLI | |
| click>=8.0.0 | |
| typer>=0.9.0 | |
| # UI/TUI | |
| textual>=0.30.0 | |
| rich>=13.0.0 | |
| # HF Integration | |
| huggingface_hub>=0.28.0 | |
| # Data Processing | |
| pandas>=2.0.0 | |
| pyarrow>=14.0.0 | |
| # Security | |
| keyring>=24.0.0 | |
| cryptography>=41.0.0 | |
| # Validation | |
| pydantic>=2.0.0 | |
| # Development & Testing | |
| pytest>=7.0.0 | |
| pytest-cov>=4.0.0 | |
| black>=23.0.0 | |
| flake8>=6.0.0 | |
| mypy>=1.0.0 | |
| # Optional: async | |
| asyncio>=3.4.3 | |
| ``` | |
| Instalar: | |
| ```bash | |
| pip install -r requirements.txt | |
| ``` | |
| --- | |
| ## Step 4: Criar `pyproject.toml` | |
| Criar arquivo: `pyproject.toml` | |
| ```toml | |
| [build-system] | |
| requires = ["setuptools>=65.0", "wheel"] | |
| build-backend = "setuptools.build_meta" | |
| [project] | |
| name = "birdnet-uploader" | |
| version = "0.1.0" | |
| description = "Local CLI for resumable HuggingFace dataset uploads (BirdNET)" | |
| authors = [ | |
| {name = "BirdNET Team", email = "team@example.com"} | |
| ] | |
| requires-python = ">=3.9" | |
| dependencies = [ | |
| "click>=8.0.0", | |
| "textual>=0.30.0", | |
| "rich>=13.0.0", | |
| "huggingface_hub>=0.28.0", | |
| "pandas>=2.0.0", | |
| "pyarrow>=14.0.0", | |
| "keyring>=24.0.0", | |
| "pydantic>=2.0.0", | |
| ] | |
| [project.optional-dependencies] | |
| dev = [ | |
| "pytest>=7.0.0", | |
| "pytest-cov>=4.0.0", | |
| "black>=23.0.0", | |
| "flake8>=6.0.0", | |
| "mypy>=1.0.0", | |
| ] | |
| [project.scripts] | |
| birdnet = "src.main:app" | |
| [tool.black] | |
| line-length = 100 | |
| target-version = ['py39'] | |
| [tool.isort] | |
| profile = "black" | |
| line_length = 100 | |
| [tool.mypy] | |
| python_version = "3.9" | |
| warn_return_any = true | |
| warn_unused_configs = true | |
| [tool.pytest.ini_options] | |
| testpaths = ["tests"] | |
| addopts = "-v --cov=src --cov-report=html --cov-report=term-missing" | |
| ``` | |
| --- | |
| ## Step 5: Criar Estrutura Base de Módulos (Skeletons) | |
| ### 5.1: `src/main.py` | |
| ```python | |
| """BirdNET Uploader - Main Entry Point""" | |
| import click | |
| from src.ui.tui import BirdNETApp | |
| from src.auth.auth_service import AuthService | |
| from src.config import get_config | |
| __version__ = "0.1.0" | |
| @click.group() | |
| @click.version_option(version=__version__) | |
| def app(): | |
| """BirdNET Uploader - Upload audio to HuggingFace Datasets""" | |
| pass | |
| @app.command() | |
| def upload(): | |
| """Start the upload wizard (interactive TUI)""" | |
| try: | |
| app_ui = BirdNETApp() | |
| app_ui.run() | |
| except Exception as e: | |
| click.echo(f"Error: {e}", err=True) | |
| @app.command() | |
| def login(): | |
| """Authenticate with HuggingFace token""" | |
| try: | |
| token = click.prompt("Enter your HF token", hide_input=True) | |
| auth = AuthService() | |
| user_info = auth.authenticate(token) | |
| click.echo(f"✓ Logged in as {user_info['username']}") | |
| except Exception as e: | |
| click.echo(f"Error: {e}", err=True) | |
| @app.command() | |
| @click.argument("session_id") | |
| def resume(session_id: str): | |
| """Resume a previous upload session""" | |
| click.echo(f"Resuming session: {session_id}") | |
| # TODO: Implement | |
| @app.command() | |
| def list_sessions(): | |
| """List all upload sessions""" | |
| click.echo("Sessions:") | |
| # TODO: Implement | |
| if __name__ == "__main__": | |
| app() | |
| ``` | |
| ### 5.2: `src/config.py` | |
| ```python | |
| """Configuration and Constants""" | |
| from pathlib import Path | |
| import os | |
| # App | |
| APP_NAME = "BirdNET Uploader" | |
| APP_VERSION = "0.1.0" | |
| # Directories | |
| SESSION_DIR = Path.home() / ".birdnet-uploader" / "sessions" | |
| CACHE_DIR = Path.home() / ".birdnet-uploader" / "cache" | |
| LOG_DIR = Path.home() / ".birdnet-uploader" / "logs" | |
| # Create directories if not exist | |
| SESSION_DIR.mkdir(parents=True, exist_ok=True) | |
| CACHE_DIR.mkdir(parents=True, exist_ok=True) | |
| LOG_DIR.mkdir(parents=True, exist_ok=True) | |
| # Audio | |
| AUDIO_EXTENSIONS = ['.wav', '.mp3', '.flac', '.ogg', '.m4a'] | |
| # Upload | |
| MAX_BATCH_SIZE = 10 | |
| MAX_RETRIES = 3 | |
| RETRY_INITIAL_BACKOFF = 1.0 # seconds | |
| RETRY_MAX_BACKOFF = 30.0 # seconds | |
| # Manifest | |
| MANIFEST_SCHEMA_VERSION = "1.0.0" | |
| SHARD_SIZE = 10000 | |
| # Keyring | |
| KEYRING_SERVICE = "birdnet-uploader" | |
| KEYRING_ACCOUNT = "hf_token" | |
| def get_config(): | |
| """Get runtime configuration from environment or defaults""" | |
| return { | |
| "session_dir": os.getenv("BIRDNET_SESSION_DIR", str(SESSION_DIR)), | |
| "cache_dir": os.getenv("BIRDNET_CACHE_DIR", str(CACHE_DIR)), | |
| "log_dir": os.getenv("BIRDNET_LOG_DIR", str(LOG_DIR)), | |
| } | |
| ``` | |
| ### 5.3: `src/auth/auth_service.py` | |
| ```python | |
| """Authentication Service""" | |
| from typing import Optional, Dict | |
| import keyring | |
| from huggingface_hub import HfApi, HfFolder | |
| from src.config import KEYRING_SERVICE, KEYRING_ACCOUNT | |
| class AuthenticationError(Exception): | |
| """Authentication failed""" | |
| pass | |
| class AuthService: | |
| """Manage authentication with HuggingFace""" | |
| def __init__(self): | |
| self.api = None | |
| def authenticate(self, token: str) -> Dict[str, str]: | |
| """ | |
| Validate token and store it securely. | |
| Args: | |
| token: HF API token | |
| Returns: | |
| User info dict with username, email, user_id | |
| Raises: | |
| AuthenticationError: If token is invalid | |
| """ | |
| try: | |
| api = HfApi(token=token) | |
| user_info = api.whoami() | |
| # Store token in keyring | |
| keyring.set_password(KEYRING_SERVICE, KEYRING_ACCOUNT, token) | |
| self.api = api | |
| return { | |
| "username": user_info.get("name"), | |
| "email": user_info.get("email", ""), | |
| "user_id": user_info.get("user_id"), | |
| } | |
| except Exception as e: | |
| raise AuthenticationError(f"Invalid token: {str(e)}") | |
| def get_token(self) -> Optional[str]: | |
| """Get stored token from keyring""" | |
| return keyring.get_password(KEYRING_SERVICE, KEYRING_ACCOUNT) | |
| def clear_token(self): | |
| """Clear stored token""" | |
| keyring.delete_password(KEYRING_SERVICE, KEYRING_ACCOUNT) | |
| ``` | |
| ### 5.4: `src/repository/repo_service.py` | |
| ```python | |
| """Repository Service""" | |
| from typing import Optional | |
| from huggingface_hub import HfApi | |
| import json | |
| from io import BytesIO | |
| class RepositoryError(Exception): | |
| """Repository operation failed""" | |
| pass | |
| class RepositoryService: | |
| """Manage HuggingFace datasets""" | |
| def __init__(self, token: str): | |
| self.api = HfApi(token=token) | |
| def create_dataset(self, repo_name: str, private: bool = True) -> str: | |
| """ | |
| Create a new dataset on HuggingFace. | |
| Args: | |
| repo_name: Dataset name | |
| private: Whether dataset should be private | |
| Returns: | |
| Full repo_id (username/repo_name) | |
| """ | |
| try: | |
| # Create repo | |
| repo = self.api.create_repo( | |
| repo_id=repo_name, | |
| repo_type="dataset", | |
| private=private, | |
| exist_ok=True | |
| ) | |
| repo_id = repo.repo_id | |
| # Initialize structure | |
| self._init_structure(repo_id) | |
| return repo_id | |
| except Exception as e: | |
| raise RepositoryError(f"Failed to create dataset: {str(e)}") | |
| def _init_structure(self, repo_id: str): | |
| """Initialize default folder structure""" | |
| folders = [ | |
| "audio/project", | |
| "index/shards", | |
| "validations", | |
| "audit/ingestion-runs" | |
| ] | |
| for folder in folders: | |
| # Upload placeholder file to create folder | |
| self.api.upload_file( | |
| path_or_fileobj=BytesIO(b""), | |
| path_in_repo=f"{folder}/.gitkeep", | |
| repo_id=repo_id, | |
| repo_type="dataset" | |
| ) | |
| def validate_repo(self, repo_id: str) -> bool: | |
| """Validate dataset has expected structure""" | |
| # TODO: Implement | |
| return True | |
| ``` | |
| ### 5.5: `src/upload/scanner.py` | |
| ```python | |
| """Local Folder Scanner""" | |
| import os | |
| from pathlib import Path | |
| from typing import Dict, List | |
| from src.config import AUDIO_EXTENSIONS | |
| class LocalScanner: | |
| """Scan local folders for audio files""" | |
| def scan_folder(self, folder_path: str) -> Dict: | |
| """ | |
| Recursively scan folder for audio files. | |
| Args: | |
| folder_path: Path to scan | |
| Returns: | |
| Structure dict with files grouped by species | |
| """ | |
| structure = {} | |
| total_size = 0 | |
| file_count = 0 | |
| for root, dirs, files in os.walk(folder_path): | |
| for file in files: | |
| if any(file.lower().endswith(ext) for ext in AUDIO_EXTENSIONS): | |
| full_path = os.path.join(root, file) | |
| size = os.path.getsize(full_path) | |
| # Infer species | |
| specie = self._infer_species(root, file) | |
| if specie not in structure: | |
| structure[specie] = [] | |
| structure[specie].append({ | |
| "name": file, | |
| "full_path": full_path, | |
| "size": size, | |
| }) | |
| total_size += size | |
| file_count += 1 | |
| return { | |
| "total_files": file_count, | |
| "total_size": total_size, | |
| "by_species": structure, | |
| } | |
| def _infer_species(self, root: str, filename: str) -> str: | |
| """Infer species from path or filename""" | |
| parts = root.split(os.sep) | |
| if len(parts) > 0: | |
| return parts[-1] | |
| return "unknown" | |
| ``` | |
| ### 5.6: `src/session/session_manager.py` | |
| ```python | |
| """Session Management""" | |
| import json | |
| from pathlib import Path | |
| from datetime import datetime | |
| from src.config import SESSION_DIR | |
| class SessionManager: | |
| """Manage upload sessions""" | |
| def __init__(self, session_id: str = None): | |
| if session_id is None: | |
| session_id = f"upload-{datetime.now().strftime('%Y%m%d-%H%M%S')}" | |
| self.session_id = session_id | |
| self.session_dir = Path(SESSION_DIR) / session_id | |
| self.session_dir.mkdir(parents=True, exist_ok=True) | |
| def save_checkpoint(self, data: dict): | |
| """Save checkpoint atomically""" | |
| checkpoint_file = self.session_dir / "checkpoint.json" | |
| temp_file = checkpoint_file.with_suffix(".tmp") | |
| with open(temp_file, "w") as f: | |
| json.dump(data, f, indent=2) | |
| temp_file.replace(checkpoint_file) | |
| def load_checkpoint(self) -> dict: | |
| """Load last checkpoint""" | |
| checkpoint_file = self.session_dir / "checkpoint.json" | |
| if checkpoint_file.exists(): | |
| with open(checkpoint_file) as f: | |
| return json.load(f) | |
| return {} | |
| ``` | |
| ### 5.7: `src/ui/tui.py` | |
| ```python | |
| """TUI Application (Textual)""" | |
| from textual.app import ComposeResult, App | |
| from textual.containers import Container | |
| from textual.widgets import Static, Button | |
| class BirdNETApp(App): | |
| """Main TUI Application""" | |
| def compose(self) -> ComposeResult: | |
| """Create child widgets for the app.""" | |
| yield Container( | |
| Static("BirdNET Uploader", id="header"), | |
| Button("Start Upload", id="btn_upload"), | |
| id="container" | |
| ) | |
| def on_button_pressed(self) -> None: | |
| """Handle button press.""" | |
| self.exit() | |
| def run(): | |
| """Run the TUI app""" | |
| app = BirdNETApp() | |
| app.run() | |
| ``` | |
| ### 5.8: `src/utils/error_handler.py` | |
| ```python | |
| """Error Handling""" | |
| class BirdNETError(Exception): | |
| """Base exception""" | |
| pass | |
| class AuthenticationError(BirdNETError): | |
| """Authentication failed""" | |
| pass | |
| class RepositoryError(BirdNETError): | |
| """Repository operation failed""" | |
| pass | |
| class UploadError(BirdNETError): | |
| """Upload operation failed""" | |
| pass | |
| class ValidationError(BirdNETError): | |
| """Validation failed""" | |
| pass | |
| ``` | |
| --- | |
| ## Step 6: Criar Test Infrastructure | |
| ### 6.1: `pytest.ini` | |
| ```ini | |
| [pytest] | |
| testpaths = tests | |
| addopts = -v --tb=short | |
| python_files = test_*.py | |
| python_classes = Test* | |
| python_functions = test_* | |
| ``` | |
| ### 6.2: `tests/conftest.py` | |
| ```python | |
| """Test Configuration and Fixtures""" | |
| import pytest | |
| from pathlib import Path | |
| import tempfile | |
| import json | |
| @pytest.fixture | |
| def temp_folder(): | |
| """Create temporary folder with test audio files""" | |
| with tempfile.TemporaryDirectory() as tmpdir: | |
| # Create test structure | |
| audio_dir = Path(tmpdir) / "audio" / "Parrot" | |
| audio_dir.mkdir(parents=True) | |
| # Create dummy audio files | |
| for i in range(5): | |
| (audio_dir / f"segment_{i:03d}.wav").write_bytes(b"fake audio data") | |
| yield tmpdir | |
| @pytest.fixture | |
| def mock_hf_token(): | |
| """Mock HF token""" | |
| return "hf_test_token_12345" | |
| @pytest.fixture | |
| def session_dir(tmp_path): | |
| """Temporary session directory""" | |
| session_dir = tmp_path / "sessions" / "test-session" | |
| session_dir.mkdir(parents=True) | |
| return session_dir | |
| ``` | |
| ### 6.3: `tests/test_auth.py` | |
| ```python | |
| """Tests for Auth Service""" | |
| import pytest | |
| from src.auth.auth_service import AuthService, AuthenticationError | |
| def test_auth_invalid_token(mock_hf_token): | |
| """Test that invalid token raises error""" | |
| auth = AuthService() | |
| with pytest.raises(AuthenticationError): | |
| auth.authenticate("invalid_token_xyz") | |
| # More tests to be implemented | |
| ``` | |
| ### 6.4: `tests/test_scanner.py` | |
| ```python | |
| """Tests for File Scanner""" | |
| import pytest | |
| from src.upload.scanner import LocalScanner | |
| def test_scan_folder_empty(tmp_path): | |
| """Test scanning empty folder""" | |
| scanner = LocalScanner() | |
| result = scanner.scan_folder(str(tmp_path)) | |
| assert result["total_files"] == 0 | |
| assert result["total_size"] == 0 | |
| def test_scan_folder_with_audio(temp_folder): | |
| """Test scanning folder with audio files""" | |
| scanner = LocalScanner() | |
| result = scanner.scan_folder(temp_folder) | |
| assert result["total_files"] == 5 | |
| assert result["total_size"] > 0 | |
| # More tests to be implemented | |
| ``` | |
| --- | |
| ## Step 7: Configurar CI/CD (GitHub Actions) | |
| Criar arquivo: `.github/workflows/test.yml` | |
| ```yaml | |
| name: Tests | |
| on: [push, pull_request] | |
| jobs: | |
| test: | |
| runs-on: ubuntu-latest | |
| strategy: | |
| matrix: | |
| python-version: ["3.9", "3.10", "3.11"] | |
| steps: | |
| - uses: actions/checkout@v3 | |
| - name: Set up Python | |
| uses: actions/setup-python@v4 | |
| with: | |
| python-version: ${{ matrix.python-version }} | |
| - name: Install dependencies | |
| run: | | |
| pip install -r requirements.txt | |
| - name: Lint | |
| run: | | |
| flake8 src tests | |
| - name: Run tests | |
| run: | | |
| pytest | |
| - name: Upload coverage | |
| uses: codecov/codecov-action@v3 | |
| ``` | |
| --- | |
| ## Step 8: Testar Instalação | |
| ```bash | |
| # Instalar dependências | |
| pip install -r requirements.txt | |
| # Rodar testes | |
| pytest -v | |
| # Testar import | |
| python -c "import src; print('✓ Imports OK')" | |
| # Rodar CLI help | |
| python -m src.main --help | |
| ``` | |
| **Esperado**: Todos os comandos rodarem sem erro. | |
| --- | |
| ## Step 9: Commit Initial | |
| ```bash | |
| git add -A | |
| git commit -m "feat: Sprint 1 - Initial project structure and scaffolding" | |
| git push origin feature/birdnet-uploader-cli | |
| ``` | |
| --- | |
| ## Checklist: Sprint 1, Day 1 - DONE ✓ | |
| - [ ] Branch `feature/birdnet-uploader-cli` criada | |
| - [ ] Estrutura de diretórios completa | |
| - [ ] `requirements.txt` instalado | |
| - [ ] `pyproject.toml` criado | |
| - [ ] Todos os módulos esqueléticos criados (8+ arquivos) | |
| - [ ] `pytest.ini` e `conftest.py` configurados | |
| - [ ] Testes básicos criados | |
| - [ ] CI/CD workflow criado | |
| - [ ] Primeira execução sem erro: `pytest -v` | |
| - [ ] Commit feito e pushed | |
| **Tempo**: ~2-3 horas | |
| --- | |
| ## Próximos Passos (Task 1.2 - Design Review) | |
| Quando tudo acima estiver **DONE**, fazer: | |
| 1. **Meeting de Design Review** com o time (30min) | |
| - Confirmar Tech Stack (Textual vs Rich+Click) | |
| - Confirmar persistência (JSON vs SQLite) | |
| - Validar estrutura de repo HF | |
| - Assinar decision log | |
| 2. **Começar Task 2.1** (Auth Service Implementation) | |
| - Implementar `AuthService.authenticate()` completo | |
| - Implementar keyring integration | |
| - Escrever testes com mocks | |
| --- | |
| ## Troubleshooting | |
| **Problema**: `pip install` falha com erro de dependência | |
| **Solução**: Upgrade pip: `pip install --upgrade pip` | |
| **Problema**: Imports falhando | |
| **Solução**: Adicionar `export PYTHONPATH="${PYTHONPATH}:$(pwd)"` | |
| **Problema**: Pytest não encontra testes | |
| **Solução**: Verificar que todos os `__init__.py` existem | |
| --- | |
| ## Documentação de Referência | |
| - BIRDNET_UPLOADER_REDESIGN.md (arquitetura e UX) | |
| - SPRINT_ROADMAP.md (timeline e tasks) | |
| - Este arquivo (Sprint 1 Day 1 setup) | |