Spaces:
Runtime error
Runtime error
|
Download docs/04-weekend2/deployment.md from progressive-intelligence-lab/docuflow: direct link, hf CLI and curl.
- Browser
- Download file 10.2 kB
-
https://huggingface.co/spaces/progressive-intelligence-lab/docuflow/resolve/main/docs/04-weekend2/deployment.md
- Command line
-
hf download hf://spaces/progressive-intelligence-lab/docuflow/docs/04-weekend2/deployment.md
-
curl -L -o deployment.md https://huggingface.co/spaces/progressive-intelligence-lab/docuflow/resolve/main/docs/04-weekend2/deployment.md
10.2 kB
| # π DocuFlow λ°°ν¬ κ°μ΄λ | |
| ## π λ°°ν¬ μ 체ν¬λ¦¬μ€νΈ | |
| ### 1. μ½λ ꡬν μλ£ νμΈ | |
| νμ¬ νλ‘μ νΈμ ν΅μ¬ κΈ°λ₯λ€μ΄ μμ§ κ΅¬νλμ§ μμμ΅λλ€. λ°°ν¬ μ μ λ€μμ μλ£ν΄μΌ ν©λλ€: | |
| #### β [Stage: Weekend-1] λ°μ΄ν° μ μ²λ¦¬ | |
| - [x] `app/embedding_utils.py` - `split_text()` ν¨μ ꡬν | |
| - [x] `app/main.py` - `extract_text_from_pdf()` ν¨μ ꡬν | |
| - [x] PDF ν μ€νΈ μΆμΆ ν μ€νΈ | |
| #### β [Stage: Weekend-1] μλ² λ© λ° λ²‘ν° μ μ₯ | |
| - [x] `app/embedding_utils.py` - `get_embedding()` ν¨μ ꡬν | |
| - [x] `app/supabase_utils.py` - `save_embedding()` ν¨μ ꡬν | |
| - [x] λΉλκΈ° μλ² λ© μ²λ¦¬ (μ±λ₯ μ΅μ ν) | |
| - [x] Supabase μ°κ²° ν μ€νΈ | |
| #### β [Stage: Weekend-1] RAG νμ΄νλΌμΈ | |
| - [x] `app/supabase_utils.py` - `search_similar_embeddings()` ν¨μ ꡬν | |
| - [x] `app/main.py` - `/qa` μλν¬μΈνΈ ꡬν | |
| - [x] Ollama/OpenAI LLM μ°λ | |
| - [x] ν 루μλ€μ΄μ λ°©μ§ ν둬ννΈ μμ± | |
| #### β [Stage: Midweek] λν λ§₯λ½ μ μ§ | |
| - [ ] λν νμ€ν 리 κ΄λ¦¬ | |
| - [x] Query Rewriting ꡬν (μ νμ¬ν) | |
| #### β [Stage: Weekend-2] μμ μ± λ° μλ¬ νΈλ€λ§ | |
| - [ ] λͺ¨λ μμΈ μν© μ²λ¦¬ | |
| - [ ] μ¬μ©μ μΉνμ μλ¬ λ©μμ§ | |
| - [ ] λ‘κΉ μμ€ν κ΅¬μΆ | |
| - [ ] μ€νΈλ μ€ ν μ€νΈ | |
| --- | |
| ## π§ λ°°ν¬ νκ²½ μ€μ | |
| ### νμ νκ²½ λ³μ | |
| `.env` νμΌμ μμ±νκ³ λ€μ λ³μλ€μ μ€μ νμΈμ: | |
| ```ini | |
| # 곡κΈμ μ ν: "ollama" λλ "openai" | |
| PROVIDER="ollama" | |
| # Ollama μ€μ (λ‘컬 λλ μ격 μλ²) | |
| OLLAMA_BASE="http://localhost:11434" | |
| OLLAMA_EMBED_MODEL="nomic-embed-text" | |
| OLLAMA_CHAT_MODEL="llama3.1" | |
| # Supabase μ€μ (νμ) | |
| SUPABASE_URL="https://your-project.supabase.co" | |
| SUPABASE_KEY="your-anon-key" | |
| # OpenAI μ€μ (PROVIDER=openaiμΌ λλ§ νμ) | |
| # OPENAI_API_KEY="sk-..." | |
| # OPENAI_EMBEDDING_MODEL="text-embedding-3-small" | |
| # OPENAI_CHAT_MODEL="gpt-3.5-turbo" | |
| # μλ² μ€μ | |
| HOST="0.0.0.0" | |
| PORT=8000 | |
| DEBUG=false | |
| # 보μ μ€μ | |
| SECRET_KEY="your-secret-key-here" | |
| ALLOWED_ORIGINS="http://localhost:8000,https://yourdomain.com" | |
| ``` | |
| ### λ°μ΄ν°λ² μ΄μ€ μ€μ | |
| Supabaseμμ λ€μμ μλ£ν΄μΌ ν©λλ€: | |
| 1. **pgvector νμ₯ νμ±ν** | |
| 2. **document_chunks ν μ΄λΈ μμ±** (README.mdμ SQL μ€ν¬λ¦½νΈ μ€ν) | |
| 3. **match_documents ν¨μ μμ±** | |
| 4. **HNSW μΈλ±μ€ μμ±** | |
| --- | |
| ## π³ Docker λ°°ν¬ (κΆμ₯) | |
| ### Dockerfile μμ± | |
| νλ‘μ νΈ λ£¨νΈμ `Dockerfile`μ μμ±νμΈμ: | |
| ```dockerfile | |
| FROM python:3.11-slim | |
| WORKDIR /app | |
| # μμ€ν μμ‘΄μ± μ€μΉ | |
| RUN apt-get update && apt-get install -y \ | |
| gcc \ | |
| postgresql-client \ | |
| && rm -rf /var/lib/apt/lists/* | |
| # Python μμ‘΄μ± μ€μΉ | |
| COPY Pipfile Pipfile.lock ./ | |
| RUN pip install pipenv && \ | |
| pipenv install --deploy --ignore-pipfile | |
| # μ ν리μΌμ΄μ μ½λ λ³΅μ¬ | |
| COPY . . | |
| # μ λ‘λ λλ ν 리 μμ± | |
| RUN mkdir -p app/uploads | |
| # ν¬νΈ λ ΈμΆ | |
| EXPOSE 8000 | |
| # νκ²½ λ³μ μ€μ | |
| ENV PYTHONUNBUFFERED=1 | |
| # μλ² μ€ν | |
| CMD ["pipenv", "run", "uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"] | |
| ``` | |
| ### docker-compose.yml (Ollama ν¬ν¨) | |
| ```yaml | |
| version: '3.8' | |
| services: | |
| app: | |
| build: . | |
| ports: | |
| - '8000:8000' | |
| environment: | |
| - PROVIDER=ollama | |
| - OLLAMA_BASE=http://ollama:11434 | |
| - SUPABASE_URL=${SUPABASE_URL} | |
| - SUPABASE_KEY=${SUPABASE_KEY} | |
| volumes: | |
| - ./app/uploads:/app/app/uploads | |
| - ./.env:/app/.env | |
| depends_on: | |
| - ollama | |
| ollama: | |
| image: ollama/ollama:latest | |
| ports: | |
| - '11434:11434' | |
| volumes: | |
| - ollama_data:/root/.ollama | |
| volumes: | |
| ollama_data: | |
| ``` | |
| ### λ°°ν¬ λͺ λ Ήμ΄ | |
| ```bash | |
| # μ΄λ―Έμ§ λΉλ | |
| docker build -t docuflow:latest . | |
| # 컨ν μ΄λ μ€ν | |
| docker-compose up -d | |
| # λ‘κ·Έ νμΈ | |
| docker-compose logs -f app | |
| ``` | |
| --- | |
| ## βοΈ ν΄λΌμ°λ λ°°ν¬ μ΅μ | |
| > AI νλ‘μ νΈ λ°°ν¬ λμ΄λ λΉκ΅: HF Spaces β Railway < Render < Fly.io < AWS/GCP | |
| ### 1. Hugging Face Spaces (μΆμ² - AI νλ‘μ νΈμ μ΅μ ) | |
| **μ₯μ :** | |
| - AI/ML νλ‘μ νΈ λ°°ν¬μ νΉνλ νλ«νΌ | |
| - Docker κΈ°λ° λ¬΄λ£ λ°°ν¬ (CPU 2μ½μ΄, **RAM 16GB**) | |
| - κ³΅κ° URL μλ μμ±, νκ²½ λ³μ(Secrets) κ΄λ¦¬ μ 곡 | |
| - GitHub μ°λ μλ λ°°ν¬ μ§μ | |
| **μ μ½μ¬ν:** | |
| - μΌμ μκ° λ―Έμ¬μ© μ μ¬λ¦½ λͺ¨λ μ§μ β 첫 μμ² μ 30~60μ΄ μ¬μμ μ§μ° | |
| - λ¬΄λ£ ν°μ΄λ μμ μ€ν λ¦¬μ§ μμ (μ λ‘λ νμΌμ μ¬μμ μ μ΄κΈ°ν, μλ² λ©μ Supabaseμ μ μ₯λλ―λ‘ λ¬΄κ΄) | |
| - Ollamaλ CPU μ€ν μ μλ΅μ΄ λ§€μ° λ리λ―λ‘ **OpenAI νΈλ κΆμ₯** | |
| **λ¨κ³:** | |
| 1. [huggingface.co](https://huggingface.co)μμ κ³μ μμ± ν **New Space** ν΄λ¦ | |
| 2. SDKλ₯Ό **Docker**λ‘ μ ννκ³ Space μμ± | |
| 3. Space 리ν¬μ§ν 리μ `README.md` μλ¨μ μλ λ©νλ°μ΄ν° μΆκ°: | |
| ```yaml | |
| --- | |
| title: DocuFlow | |
| emoji: π | |
| colorFrom: blue | |
| colorTo: green | |
| sdk: docker | |
| pinned: false | |
| --- | |
| ``` | |
| 4. Space 리ν¬μ§ν 리μ νλ‘μ νΈ μ½λλ₯Ό push (λλ GitHub μ°λ) | |
| 5. **Settings β Variables and secrets**μμ νκ²½ λ³μ λ±λ‘: | |
| - `PROVIDER` = `openai` | |
| - `OPENAI_API_KEY` = `sk-...` | |
| - `OPENAI_EMBEDDING_MODEL` = `text-embedding-3-small` | |
| - `OPENAI_CHAT_MODEL` = `gpt-3.5-turbo` | |
| - `SUPABASE_URL`, `SUPABASE_KEY` | |
| --- | |
| ### 2. Railway (κ°λ¨ν¨) | |
| **μ₯μ :** | |
| - Supabaseμ ν΅ν© μ©μ΄ | |
| - μλ λ°°ν¬ | |
| - λ¬΄λ£ ν°μ΄ μ 곡 | |
| **λ¨κ³:** | |
| 1. Railway κ³μ μμ± | |
| 2. GitHub μ μ₯μ μ°κ²° | |
| 3. νκ²½ λ³μ μ€μ | |
| 4. Ollamaλ λ³λ μλ² νμ (λλ OpenAI μ¬μ©) | |
| ### 3. Render | |
| **μ₯μ :** | |
| - λ¬΄λ£ ν°μ΄ | |
| - Docker μ§μ | |
| - μλ HTTPS | |
| **λ¨κ³:** | |
| 1. Render κ³μ μμ± | |
| 2. Dockerfile κΈ°λ° λ°°ν¬ | |
| 3. νκ²½ λ³μ μ€μ | |
| ### 4. AWS/GCP/Azure | |
| **μ₯μ :** | |
| - νμ₯μ± | |
| - κ³ μ±λ₯ | |
| - μν°νλΌμ΄μ¦κΈ | |
| **κ³ λ €μ¬ν:** | |
| - Ollamaλ EC2/Compute Engineμμ λ³λ μ€ν | |
| - λλ OpenAI API μ¬μ© κΆμ₯ | |
| - λ‘λ λ°Έλ°μ μ€μ νμ | |
| ### 5. Fly.io | |
| **μ₯μ :** | |
| - μ μΈκ³ μ£μ§ λ°°ν¬ | |
| - Docker μ§μ | |
| - κ°λ¨ν μ€μ | |
| --- | |
| ## π¦ λ°°ν¬ μ νμ νμΌ | |
| ### μμ‘΄μ± κΈ°μ€ | |
| μ΄ νλ‘μ νΈμ μμ‘΄μ±/μ€ν κΈ°μ€μ `pipenv`μ λλ€. | |
| - λ‘컬 μ€μΉ: `pipenv install` | |
| - Docker λΉλ: `Pipfile`, `Pipfile.lock` κΈ°λ° μ€μΉ | |
| μμ‘΄μ± κ΄λ¦¬λ `pipenv` λ¨μΌ κ²½λ‘λ‘ ν΅μΌν©λλ€. | |
| ### .dockerignore | |
| ``` | |
| __pycache__/ | |
| *.pyc | |
| *.pyo | |
| *.pyd | |
| .Python | |
| .env | |
| .venv | |
| venv/ | |
| ENV/ | |
| .git | |
| .gitignore | |
| README.md | |
| *.md | |
| .DS_Store | |
| app/uploads/* | |
| !app/uploads/.gitkeep | |
| ``` | |
| ### .env.example | |
| νκ²½ λ³μ ν νλ¦Ώ νμΌ (μ€μ κ°μ μ μΈ): | |
| ```ini | |
| PROVIDER=ollama | |
| OLLAMA_BASE=http://localhost:11434 | |
| OLLAMA_EMBED_MODEL=nomic-embed-text | |
| OLLAMA_CHAT_MODEL=llama3.1 | |
| SUPABASE_URL=your-supabase-url | |
| SUPABASE_KEY=your-supabase-key | |
| ``` | |
| --- | |
| ## π 보μ 체ν¬λ¦¬μ€νΈ | |
| - [x] `.env` νμΌμ `.gitignore`μ μΆκ° | |
| - [x] Supabase ν€λ₯Ό νκ²½ λ³μλ‘λ§ κ΄λ¦¬ | |
| - [x] OpenAI API ν€ λ³΄νΈ | |
| - [ ] νμΌ μ λ‘λ ν¬κΈ° μ ν μ€μ | |
| - [ ] CORS μ€μ (νλ‘λμ λλ©μΈλ§ νμ©) | |
| - [ ] Rate limiting ꡬν (μ νμ¬ν) | |
| - [ ] HTTPS μ¬μ© (νλ‘λμ ) | |
| --- | |
| ## π λͺ¨λν°λ§ λ° λ‘κΉ | |
| ### λ‘κΉ μ€μ | |
| `app/main.py`μ λ‘κΉ μΆκ°: | |
| ```python | |
| import logging | |
| logging.basicConfig( | |
| level=logging.INFO, | |
| format='%(asctime)s - %(name)s - %(levelname)s - %(message)s' | |
| ) | |
| logger = logging.getLogger(__name__) | |
| ``` | |
| ### ν¬μ€ μ²΄ν¬ μλν¬μΈνΈ | |
| ```python | |
| @app.get("/health") | |
| async def health_check(): | |
| return { | |
| "status": "healthy", | |
| "ollama": check_ollama_connection(), | |
| "supabase": check_supabase_connection() | |
| } | |
| ``` | |
| --- | |
| ## π§ͺ λ°°ν¬ μ ν μ€νΈ | |
| ### λ‘컬 ν μ€νΈ | |
| ```bash | |
| # μλ² μ€ν | |
| pipenv run uvicorn app.main:app --reload | |
| # ν μ€νΈ | |
| curl http://localhost:8000/health | |
| ``` | |
| ### ν΅ν© ν μ€νΈ 체ν¬λ¦¬μ€νΈ | |
| - [ ] PDF μ λ‘λ ν μ€νΈ | |
| - [ ] ν μ€νΈ μΆμΆ νμΈ | |
| - [ ] μλ² λ© μμ± νμΈ | |
| - [ ] Supabase μ μ₯ νμΈ | |
| - [ ] μ§λ¬Έ-λ΅λ³ ν μ€νΈ | |
| - [ ] μλ¬ νΈλ€λ§ ν μ€νΈ | |
| - [ ] λμ©λ νμΌ ν μ€νΈ | |
| --- | |
| ## π¨ νΈλ¬λΈμν | |
| ### Ollama μ°κ²° μ€ν¨ | |
| **λ¬Έμ :** Ollama μλ²μ μ°κ²°ν μ μμ | |
| **ν΄κ²°:** | |
| - Ollama μλ²κ° μ€ν μ€μΈμ§ νμΈ: `curl http://localhost:11434/api/tags` | |
| - Docker νκ²½μμλ `OLLAMA_BASE=http://ollama:11434` μ¬μ© | |
| - λ°©νλ²½ μ€μ νμΈ | |
| ### Supabase μ°κ²° μ€ν¨ | |
| **λ¬Έμ :** Supabaseμ μ°κ²°ν μ μμ | |
| **ν΄κ²°:** | |
| - νκ²½ λ³μ νμΈ: `SUPABASE_URL`, `SUPABASE_KEY` | |
| - Supabase νλ‘μ νΈ μν νμΈ | |
| - λ€νΈμν¬ μ°κ²° νμΈ | |
| ### μλ² λ© μ°¨μ λΆμΌμΉ | |
| **λ¬Έμ :** λ²‘ν° μ°¨μμ΄ λ§μ§ μμ | |
| **ν΄κ²°:** | |
| - Ollama (768μ°¨μ) vs OpenAI (1536μ°¨μ) νμΈ | |
| - Supabase ν μ΄λΈ μ€ν€λ§μ μΌμΉνλμ§ νμΈ | |
| - `.env`μ `PROVIDER` μ€μ νμΈ | |
| --- | |
| ## π λ°°ν¬ ν 체ν¬λ¦¬μ€νΈ | |
| - [ ] μλ²κ° μ μμ μΌλ‘ μμλλμ§ νμΈ | |
| - [ ] ν¬μ€ μ²΄ν¬ μλν¬μΈνΈ μλ΅ νμΈ | |
| - [ ] PDF μ λ‘λ κΈ°λ₯ ν μ€νΈ | |
| - [ ] μ§λ¬Έ-λ΅λ³ κΈ°λ₯ ν μ€νΈ | |
| - [ ] μλ¬ λ‘κ·Έ λͺ¨λν°λ§ | |
| - [ ] μ±λ₯ λͺ¨λν°λ§ (μλ΅ μκ°, λ©λͺ¨λ¦¬ μ¬μ©λ) | |
| - [ ] λ°±μ μ λ΅ μ립 | |
| --- | |
| ## π CI/CD νμ΄νλΌμΈ (μ νμ¬ν) | |
| ### GitHub Actions μμ | |
| `.github/workflows/deploy.yml`: | |
| ```yaml | |
| name: Deploy | |
| on: | |
| push: | |
| branches: [main] | |
| jobs: | |
| deploy: | |
| runs-on: ubuntu-latest | |
| steps: | |
| - uses: actions/checkout@v2 | |
| - name: Deploy to Railway | |
| run: | | |
| # Railway CLIλ₯Ό μ¬μ©ν λ°°ν¬ | |
| # λλ λ€λ₯Έ λ°°ν¬ μ€ν¬λ¦½νΈ | |
| ``` | |
| --- | |
| ## π μΆκ° 리μμ€ | |
| - [FastAPI λ°°ν¬ κ°μ΄λ](https://fastapi.tiangolo.com/deployment/) | |
| - [Supabase λ²‘ν° κ²μ](https://supabase.com/docs/guides/ai) | |
| - [Docker λ² μ€νΈ νλν°μ€](https://docs.docker.com/develop/dev-best-practices/) | |
| - [Ollama λ°°ν¬ κ°μ΄λ](https://ollama.com/docs) | |
| --- | |
| ## π‘ λ°°ν¬ μ μ΅μ’ νμΈ | |
| 1. β λͺ¨λ κΈ°λ₯μ΄ κ΅¬νλμλκ°? | |
| 2. β νκ²½ λ³μκ° λͺ¨λ μ€μ λμλκ°? | |
| 3. β λ°μ΄ν°λ² μ΄μ€κ° μ€λΉλμλκ°? | |
| 4. β 보μ μ€μ μ΄ μλ£λμλκ°? | |
| 5. β ν μ€νΈκ° ν΅κ³Όνλκ°? | |
| 6. β λ‘κΉ μ΄ μ€μ λμλκ°? | |
| 7. β μλ¬ νΈλ€λ§μ΄ μλ£λμλκ°? | |
| **μ€λΉκ° λμλ€λ©΄ λ°°ν¬λ₯Ό μμνμΈμ! π** | |