Spaces:
Runtime error
Download docs/04-weekend2/deployment.md from progressive-intelligence-lab/docuflow: direct link, hf CLI and curl.
- Browser
- Download file 10.2 kB
-
https://huggingface.co/spaces/progressive-intelligence-lab/docuflow/resolve/main/docs/04-weekend2/deployment.md
- Command line
-
hf download hf://spaces/progressive-intelligence-lab/docuflow/docs/04-weekend2/deployment.md
-
curl -L -o deployment.md https://huggingface.co/spaces/progressive-intelligence-lab/docuflow/resolve/main/docs/04-weekend2/deployment.md
π DocuFlow λ°°ν¬ κ°μ΄λ
π λ°°ν¬ μ 체ν¬λ¦¬μ€νΈ
1. μ½λ ꡬν μλ£ νμΈ
νμ¬ νλ‘μ νΈμ ν΅μ¬ κΈ°λ₯λ€μ΄ μμ§ κ΅¬νλμ§ μμμ΅λλ€. λ°°ν¬ μ μ λ€μμ μλ£ν΄μΌ ν©λλ€:
β [Stage: Weekend-1] λ°μ΄ν° μ μ²λ¦¬
-
app/embedding_utils.py-split_text()ν¨μ ꡬν -
app/main.py-extract_text_from_pdf()ν¨μ ꡬν - PDF ν μ€νΈ μΆμΆ ν μ€νΈ
β [Stage: Weekend-1] μλ² λ© λ° λ²‘ν° μ μ₯
-
app/embedding_utils.py-get_embedding()ν¨μ ꡬν -
app/supabase_utils.py-save_embedding()ν¨μ ꡬν - λΉλκΈ° μλ² λ© μ²λ¦¬ (μ±λ₯ μ΅μ ν)
- Supabase μ°κ²° ν μ€νΈ
β [Stage: Weekend-1] RAG νμ΄νλΌμΈ
-
app/supabase_utils.py-search_similar_embeddings()ν¨μ ꡬν -
app/main.py-/qaμλν¬μΈνΈ ꡬν - Ollama/OpenAI LLM μ°λ
- ν 루μλ€μ΄μ λ°©μ§ ν둬ννΈ μμ±
β [Stage: Midweek] λν λ§₯λ½ μ μ§
- λν νμ€ν 리 κ΄λ¦¬
- Query Rewriting ꡬν (μ νμ¬ν)
β [Stage: Weekend-2] μμ μ± λ° μλ¬ νΈλ€λ§
- λͺ¨λ μμΈ μν© μ²λ¦¬
- μ¬μ©μ μΉνμ μλ¬ λ©μμ§
- λ‘κΉ μμ€ν ꡬμΆ
- μ€νΈλ μ€ ν μ€νΈ
π§ λ°°ν¬ νκ²½ μ€μ
νμ νκ²½ λ³μ
.env νμΌμ μμ±νκ³ λ€μ λ³μλ€μ μ€μ νμΈμ:
# 곡κΈμ μ ν: "ollama" λλ "openai"
PROVIDER="ollama"
# Ollama μ€μ (λ‘컬 λλ μ격 μλ²)
OLLAMA_BASE="http://localhost:11434"
OLLAMA_EMBED_MODEL="nomic-embed-text"
OLLAMA_CHAT_MODEL="llama3.1"
# Supabase μ€μ (νμ)
SUPABASE_URL="https://your-project.supabase.co"
SUPABASE_KEY="your-anon-key"
# OpenAI μ€μ (PROVIDER=openaiμΌ λλ§ νμ)
# OPENAI_API_KEY="sk-..."
# OPENAI_EMBEDDING_MODEL="text-embedding-3-small"
# OPENAI_CHAT_MODEL="gpt-3.5-turbo"
# μλ² μ€μ
HOST="0.0.0.0"
PORT=8000
DEBUG=false
# 보μ μ€μ
SECRET_KEY="your-secret-key-here"
ALLOWED_ORIGINS="http://localhost:8000,https://yourdomain.com"
λ°μ΄ν°λ² μ΄μ€ μ€μ
Supabaseμμ λ€μμ μλ£ν΄μΌ ν©λλ€:
- pgvector νμ₯ νμ±ν
- document_chunks ν μ΄λΈ μμ± (README.mdμ SQL μ€ν¬λ¦½νΈ μ€ν)
- match_documents ν¨μ μμ±
- HNSW μΈλ±μ€ μμ±
π³ Docker λ°°ν¬ (κΆμ₯)
Dockerfile μμ±
νλ‘μ νΈ λ£¨νΈμ Dockerfileμ μμ±νμΈμ:
FROM python:3.11-slim
WORKDIR /app
# μμ€ν
μμ‘΄μ± μ€μΉ
RUN apt-get update && apt-get install -y \
gcc \
postgresql-client \
&& rm -rf /var/lib/apt/lists/*
# Python μμ‘΄μ± μ€μΉ
COPY Pipfile Pipfile.lock ./
RUN pip install pipenv && \
pipenv install --deploy --ignore-pipfile
# μ ν리μΌμ΄μ
μ½λ 볡μ¬
COPY . .
# μ
λ‘λ λλ ν 리 μμ±
RUN mkdir -p app/uploads
# ν¬νΈ λ
ΈμΆ
EXPOSE 8000
# νκ²½ λ³μ μ€μ
ENV PYTHONUNBUFFERED=1
# μλ² μ€ν
CMD ["pipenv", "run", "uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]
docker-compose.yml (Ollama ν¬ν¨)
version: '3.8'
services:
app:
build: .
ports:
- '8000:8000'
environment:
- PROVIDER=ollama
- OLLAMA_BASE=http://ollama:11434
- SUPABASE_URL=${SUPABASE_URL}
- SUPABASE_KEY=${SUPABASE_KEY}
volumes:
- ./app/uploads:/app/app/uploads
- ./.env:/app/.env
depends_on:
- ollama
ollama:
image: ollama/ollama:latest
ports:
- '11434:11434'
volumes:
- ollama_data:/root/.ollama
volumes:
ollama_data:
λ°°ν¬ λͺ λ Ήμ΄
# μ΄λ―Έμ§ λΉλ
docker build -t docuflow:latest .
# 컨ν
μ΄λ μ€ν
docker-compose up -d
# λ‘κ·Έ νμΈ
docker-compose logs -f app
βοΈ ν΄λΌμ°λ λ°°ν¬ μ΅μ
AI νλ‘μ νΈ λ°°ν¬ λμ΄λ λΉκ΅: HF Spaces β Railway < Render < Fly.io < AWS/GCP
1. Hugging Face Spaces (μΆμ² - AI νλ‘μ νΈμ μ΅μ )
μ₯μ :
- AI/ML νλ‘μ νΈ λ°°ν¬μ νΉνλ νλ«νΌ
- Docker κΈ°λ° λ¬΄λ£ λ°°ν¬ (CPU 2μ½μ΄, RAM 16GB)
- κ³΅κ° URL μλ μμ±, νκ²½ λ³μ(Secrets) κ΄λ¦¬ μ 곡
- GitHub μ°λ μλ λ°°ν¬ μ§μ
μ μ½μ¬ν:
- μΌμ μκ° λ―Έμ¬μ© μ μ¬λ¦½ λͺ¨λ μ§μ β 첫 μμ² μ 30~60μ΄ μ¬μμ μ§μ°
- λ¬΄λ£ ν°μ΄λ μμ μ€ν λ¦¬μ§ μμ (μ λ‘λ νμΌμ μ¬μμ μ μ΄κΈ°ν, μλ² λ©μ Supabaseμ μ μ₯λλ―λ‘ λ¬΄κ΄)
- Ollamaλ CPU μ€ν μ μλ΅μ΄ λ§€μ° λ리λ―λ‘ OpenAI νΈλ κΆμ₯
λ¨κ³:
- huggingface.coμμ κ³μ μμ± ν New Space ν΄λ¦
- SDKλ₯Ό Dockerλ‘ μ ννκ³ Space μμ±
- Space 리ν¬μ§ν 리μ
README.mdμλ¨μ μλ λ©νλ°μ΄ν° μΆκ°:
---
title: DocuFlow
emoji: π
colorFrom: blue
colorTo: green
sdk: docker
pinned: false
---
- Space 리ν¬μ§ν 리μ νλ‘μ νΈ μ½λλ₯Ό push (λλ GitHub μ°λ)
- Settings β Variables and secretsμμ νκ²½ λ³μ λ±λ‘:
PROVIDER=openaiOPENAI_API_KEY=sk-...OPENAI_EMBEDDING_MODEL=text-embedding-3-smallOPENAI_CHAT_MODEL=gpt-3.5-turboSUPABASE_URL,SUPABASE_KEY
2. Railway (κ°λ¨ν¨)
μ₯μ :
- Supabaseμ ν΅ν© μ©μ΄
- μλ λ°°ν¬
- λ¬΄λ£ ν°μ΄ μ 곡
λ¨κ³:
- Railway κ³μ μμ±
- GitHub μ μ₯μ μ°κ²°
- νκ²½ λ³μ μ€μ
- Ollamaλ λ³λ μλ² νμ (λλ OpenAI μ¬μ©)
3. Render
μ₯μ :
- λ¬΄λ£ ν°μ΄
- Docker μ§μ
- μλ HTTPS
λ¨κ³:
- Render κ³μ μμ±
- Dockerfile κΈ°λ° λ°°ν¬
- νκ²½ λ³μ μ€μ
4. AWS/GCP/Azure
μ₯μ :
- νμ₯μ±
- κ³ μ±λ₯
- μν°νλΌμ΄μ¦κΈ
κ³ λ €μ¬ν:
- Ollamaλ EC2/Compute Engineμμ λ³λ μ€ν
- λλ OpenAI API μ¬μ© κΆμ₯
- λ‘λ λ°Έλ°μ μ€μ νμ
5. Fly.io
μ₯μ :
- μ μΈκ³ μ£μ§ λ°°ν¬
- Docker μ§μ
- κ°λ¨ν μ€μ
π¦ λ°°ν¬ μ νμ νμΌ
μμ‘΄μ± κΈ°μ€
μ΄ νλ‘μ νΈμ μμ‘΄μ±/μ€ν κΈ°μ€μ pipenvμ
λλ€.
- λ‘컬 μ€μΉ:
pipenv install - Docker λΉλ:
Pipfile,Pipfile.lockκΈ°λ° μ€μΉ
μμ‘΄μ± κ΄λ¦¬λ pipenv λ¨μΌ κ²½λ‘λ‘ ν΅μΌν©λλ€.
.dockerignore
__pycache__/
*.pyc
*.pyo
*.pyd
.Python
.env
.venv
venv/
ENV/
.git
.gitignore
README.md
*.md
.DS_Store
app/uploads/*
!app/uploads/.gitkeep
.env.example
νκ²½ λ³μ ν νλ¦Ώ νμΌ (μ€μ κ°μ μ μΈ):
PROVIDER=ollama
OLLAMA_BASE=http://localhost:11434
OLLAMA_EMBED_MODEL=nomic-embed-text
OLLAMA_CHAT_MODEL=llama3.1
SUPABASE_URL=your-supabase-url
SUPABASE_KEY=your-supabase-key
π 보μ 체ν¬λ¦¬μ€νΈ
-
.envνμΌμ.gitignoreμ μΆκ° - Supabase ν€λ₯Ό νκ²½ λ³μλ‘λ§ κ΄λ¦¬
- OpenAI API ν€ λ³΄νΈ
- νμΌ μ λ‘λ ν¬κΈ° μ ν μ€μ
- CORS μ€μ (νλ‘λμ λλ©μΈλ§ νμ©)
- Rate limiting ꡬν (μ νμ¬ν)
- HTTPS μ¬μ© (νλ‘λμ )
π λͺ¨λν°λ§ λ° λ‘κΉ
λ‘κΉ μ€μ
app/main.pyμ λ‘κΉ
μΆκ°:
import logging
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
)
logger = logging.getLogger(__name__)
ν¬μ€ μ²΄ν¬ μλν¬μΈνΈ
@app.get("/health")
async def health_check():
return {
"status": "healthy",
"ollama": check_ollama_connection(),
"supabase": check_supabase_connection()
}
π§ͺ λ°°ν¬ μ ν μ€νΈ
λ‘컬 ν μ€νΈ
# μλ² μ€ν
pipenv run uvicorn app.main:app --reload
# ν
μ€νΈ
curl http://localhost:8000/health
ν΅ν© ν μ€νΈ 체ν¬λ¦¬μ€νΈ
- PDF μ λ‘λ ν μ€νΈ
- ν μ€νΈ μΆμΆ νμΈ
- μλ² λ© μμ± νμΈ
- Supabase μ μ₯ νμΈ
- μ§λ¬Έ-λ΅λ³ ν μ€νΈ
- μλ¬ νΈλ€λ§ ν μ€νΈ
- λμ©λ νμΌ ν μ€νΈ
π¨ νΈλ¬λΈμν
Ollama μ°κ²° μ€ν¨
λ¬Έμ : Ollama μλ²μ μ°κ²°ν μ μμ
ν΄κ²°:
- Ollama μλ²κ° μ€ν μ€μΈμ§ νμΈ:
curl http://localhost:11434/api/tags - Docker νκ²½μμλ
OLLAMA_BASE=http://ollama:11434μ¬μ© - λ°©νλ²½ μ€μ νμΈ
Supabase μ°κ²° μ€ν¨
λ¬Έμ : Supabaseμ μ°κ²°ν μ μμ
ν΄κ²°:
- νκ²½ λ³μ νμΈ:
SUPABASE_URL,SUPABASE_KEY - Supabase νλ‘μ νΈ μν νμΈ
- λ€νΈμν¬ μ°κ²° νμΈ
μλ² λ© μ°¨μ λΆμΌμΉ
λ¬Έμ : λ²‘ν° μ°¨μμ΄ λ§μ§ μμ
ν΄κ²°:
- Ollama (768μ°¨μ) vs OpenAI (1536μ°¨μ) νμΈ
- Supabase ν μ΄λΈ μ€ν€λ§μ μΌμΉνλμ§ νμΈ
.envμPROVIDERμ€μ νμΈ
π λ°°ν¬ ν 체ν¬λ¦¬μ€νΈ
- μλ²κ° μ μμ μΌλ‘ μμλλμ§ νμΈ
- ν¬μ€ μ²΄ν¬ μλν¬μΈνΈ μλ΅ νμΈ
- PDF μ λ‘λ κΈ°λ₯ ν μ€νΈ
- μ§λ¬Έ-λ΅λ³ κΈ°λ₯ ν μ€νΈ
- μλ¬ λ‘κ·Έ λͺ¨λν°λ§
- μ±λ₯ λͺ¨λν°λ§ (μλ΅ μκ°, λ©λͺ¨λ¦¬ μ¬μ©λ)
- λ°±μ μ λ΅ μ립
π CI/CD νμ΄νλΌμΈ (μ νμ¬ν)
GitHub Actions μμ
.github/workflows/deploy.yml:
name: Deploy
on:
push:
branches: [main]
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- name: Deploy to Railway
run: |
# Railway CLIλ₯Ό μ¬μ©ν λ°°ν¬
# λλ λ€λ₯Έ λ°°ν¬ μ€ν¬λ¦½νΈ
π μΆκ° 리μμ€
- FastAPI λ°°ν¬ κ°μ΄λ
- Supabase λ²‘ν° κ²μ
- Docker λ² μ€νΈ νλν°μ€
- Ollama λ°°ν¬ κ°μ΄λ
π‘ λ°°ν¬ μ μ΅μ’ νμΈ
- β λͺ¨λ κΈ°λ₯μ΄ κ΅¬νλμλκ°?
- β νκ²½ λ³μκ° λͺ¨λ μ€μ λμλκ°?
- β λ°μ΄ν°λ² μ΄μ€κ° μ€λΉλμλκ°?
- β 보μ μ€μ μ΄ μλ£λμλκ°?
- β ν μ€νΈκ° ν΅κ³Όνλκ°?
- β λ‘κΉ μ΄ μ€μ λμλκ°?
- β μλ¬ νΈλ€λ§μ΄ μλ£λμλκ°?
μ€λΉκ° λμλ€λ©΄ λ°°ν¬λ₯Ό μμνμΈμ! π