feat: 자가학습·FAISS RAG·정량실험 + 배포 준비

.gitignore

@@ -10,5 +10,7 @@ venv/
 CLAUDE.md
 .claude/
 .mcp.json
-data/secom/raw/*.data
+# SECOM raw는 commit (5MB, 배포 시 필요)
+# data/secom/raw/*.data
 data/phm2016/raw/
+agents/rag/knowledge/INC-AUTO-*.md

README.md

@@ -1,74 +1,152 @@
+---
+title: FabAgent
+emoji: 🟦
+colorFrom: blue
+colorTo: indigo
+sdk: streamlit
+sdk_version: 1.36.0
+app_file: app.py
+pinned: false
+license: mit
+short_description: 반도체 공정 이상의 탐지·원인·영향·대응을 잇는 멀티 에이전트 운영 플랫폼
+---
+
 # FabAgent
 
-반도체 공정 이상의 **탐지 → 원인 분석 → 영향 평가 → 대응 권고**를 하나의
-멀티 에이전트 파이프라인으로 통합하는 운영 플랫폼 MVP입니다.
+반도체 공정 이상의 **탐지 → 원인 분석 → 영향 평가 → 대응 권고**를 하나의 멀티 에이전트
+파이프라인으로 통합하는 운영 플랫폼 MVP입니다.
+
+각 단계가 자기 역할에 특화된 ML 모델/RAG/도메인 지식 도구를 호출하는 Tool-using Agent로
+구성되어, 단일 LLM 챗봇과 달리 추적 가능하고 모듈화된 의사결정 흐름을 제공합니다.
+
+## 핵심 특징
+
+- **4-Tier 멀티 에이전트** — 탐지(ML)·원인(LLM+RAG)·영향(LLM+RAG+WIP)·대응(LLM+RAG)
+- **실데이터 기반** — UCI SECOM(범용) + PHM 2016 CMP(공정 특화 실측 25개 명명 센서)
+- **자가 학습 루프** — 운영자 승인 시 분석 결과가 인시던트 DB(.md)에 자동 기록되어 다음 RAG 검색에 즉시 반영
+- **이중 RAG 백엔드** — 키워드 매칭(기본) / FAISS 벡터 검색(환경변수로 전환)
+- **확장 인터페이스** — `core/pipeline.REAL_AGENT_ALARMS`에 알람 ID 추가만으로 새 공정 확장
 
 ## 아키텍처
 
 ```
-알람 클릭 → core/pipeline.get_tier_data(alarm_id)
-              ├─ A1 (Photo): agents/ 실제 LLM 멀티에이전트 + RAG
-              │              └ Tier 1은 SECOM 센서 데이터로 이상 탐지
-              └─ A2·A3:      data/demo.py 하드코딩
-           → core/schema.TierData (단일 계약면)
-           → components/ 가 Streamlit으로 렌더
+┌─────────────────┐
+│  알람 인박스    │  사이드바에서 알람 선택
+└────────┬────────┘
+         │
+         ▼
+┌─────────────────────────────────────────────────────────────┐
+│  core/pipeline.get_tier_data(alarm_id)                      │
+│  ┌─ A1 (Photo)  ─┐  ┌─ A2 (Etch)  ─┐  ┌─ A3 (CMP)  ─┐      │
+│  │ SECOM         │  │ SECOM        │  │ PHM 2016 CMP │      │
+│  └──────┬────────┘  └──────┬───────┘  └──────┬───────┘      │
+│         └──────────────────┼──────────────────┘              │
+│                            ▼                                 │
+│             agents/orchestrator (결정론적 시퀀서)            │
+│  ┌──────┬─────────────┬──────────────┬──────────────┐       │
+│  │Tier 1│   Tier 2    │    Tier 3    │    Tier 4    │       │
+│  │이상  │  원인 분석  │ 영향 평가    │ 대응 권고    │       │
+│  │탐지  │  LLM+RAG    │ LLM+RAG+WIP  │ LLM+RAG      │       │
+│  │IsoF  │             │ 결정론 lots  │ 결정론 refs  │       │
+│  └──┬───┴──────┬──────┴──────┬───────┴──────┬───────┘       │
+│     └──────────┴─────────────┴──────────────┘                │
+│                            ▼                                 │
+│              core/schema.TierData (단일 계약)                │
+└────────────────────────────┬────────────────────────────────┘
+                             │
+                             ▼
+              ┌──────────────────────────────┐
+              │ Streamlit UI (cascade 등장)  │
+              │ Tier 4 승인 → 작업지시서     │
+              │           + 자가 학습 루프   │
+              │           (knowledge .md 추가)│
+              └──────────────────────────────┘
 ```
 
-프론트엔드는 데이터가 실제 에이전트에서 오는지 하드코딩인지 알지 못합니다.
-실제 에이전트로 돌릴 알람을 늘리려면 `core/pipeline.REAL_AGENT_ALARMS`에
-ID만 추가하면 됩니다.
+## 데이터 & 모델
 
-## 파일 구조
+| 알람 | 공정 step | 데이터 | 모델 / 기술 |
+|---|---|---|---|
+| A1 | Photo (노광) | UCI SECOM (590 익명 센서) | IsolationForest + GPT-5 mini + RAG |
+| A2 | Etch (식각) | UCI SECOM (다른 fail row) | IsolationForest + GPT-5 mini + RAG |
+| A3 | CMP (연마) | **PHM 2016 CMP (25개 명명 센서, SLURRY_FLOW 등)** | IsolationForest + GPT-5 mini + RAG |
 
-```
-fabagent/
-├── app.py                  # 엔트리포인트
-├── components/             # Streamlit UI (프론트 담당)
-├── core/
-│   ├── schema.py           # Tier1~4 데이터 계약 ★ 양쪽 공유
-│   └── pipeline.py         # 알람 → Tier 데이터 라우터
-├── data/
-│   ├── demo.py             # A2·A3 하드코딩 데이터
-│   └── secom/              # UCI SECOM 데이터셋 + 로더 (Tier 1 이상 탐지용)
-├── agents/                 # A1 실제 멀티에이전트 (백엔드 담당)
-│   ├── orchestrator.py
-│   ├── detection.py / cause.py / impact.py / response.py
-│   └── rag/                # 도메인 지식 검색
-├── styles/main.css         # 디자인 시안 CSS
-└── assets/
-```
+SECOM은 익명 처리된 표준 벤치마크라 공정 step 라벨이 narrative입니다 (한계 명시). PHM 2016 CMP는 실제 CMP 공정 센서 데이터로 step-specific 추론이 가능합니다.
 
-## 기술 스택
+## 정량 평가 결과 요약
 
-- **프론트**: Streamlit 1.36+ (의존성 최소화 — `streamlit-extras` 등 미사용)
-- **백엔드**: OpenAI SDK — 서브에이전트 `GPT-5 mini`, 오케스트레이터 `GPT-5`
-- **데이터**: UCI SECOM 공개 데이터셋 (Tier 1 이상 탐지), pandas
-- **Python**: 3.11+
+| 실험 | 결정 | 핵심 결과 |
+|---|---|---|
+| **D1**: Tier 1 모델 비교 | IsolationForest 채택 | PR-AUC 0.129 (LOF 0.089, OC-SVM 0.098, baseline 0.119 대비 우위) |
+| **D2**: RAG 검색 방식 | 키워드 매칭 기본, FAISS 옵션 | 10문서 코퍼스에서 키워드 0.5ms vs FAISS ~100ms, 의미 우회 쿼리에서 FAISS 우위 |
+| **D5**: Multi-Agent vs Single LLM | Multi-Agent 채택 | Single이 빠르고($0.008 vs $0.018) 저렴하나 Multi는 모듈화·확장성·detailed 권고(1.6~1.9배)에서 우위 |
 
-데이터셋 준비 방법은 `data/README.md`를 참고하세요.
+상세 표·그래프·시행착오는 [experiments/README.md](experiments/README.md) 참고.
 
 ## 실행
 
+### 로컬
+
 ```bash
 pip install -r requirements.txt
-cp .env.example .env        # OPENAI_API_KEY 입력
+cp .env.example .env           # OPENAI_API_KEY 입력
 streamlit run app.py --server.port 8501
 ```
 
-## 개발 마일스톤
+PHM 2016 CMP 캐시(`data/phm2016/phm_cmp_features.csv`)는 저장소에 포함되어 별도 다운로드가 필요 없습니다. raw trajectory 데이터를 직접 다루려면 `data/phm2016/README.md` 참고.
 
-| 단계 | 내용 | 담당 |
-|---|---|---|
-| M0 | 레포 골격 · `schema.py` 합의 · 설정 파일 | 같이 |
-| M1 | Streamlit UI 전체 (`demo.py` 데이터로) | 프론트 |
-| M2 | `agents/` — A1 4단계 오케스트레이션 + RAG | 백엔드 |
-| M3 | A1을 실제 에이전트로 스위치 · 통합 테스트 | 같이 |
-| M4 | 시연 리허설 (개발 가이드 15장 체크리스트) | 같이 |
+### Hugging Face Spaces 배포
+
+1. HF Space 생성 (SDK: Streamlit)
+2. 본 저장소 연결 (또는 push)
+3. Space settings → Variables and secrets → `OPENAI_API_KEY` 추가
+4. 자동 빌드·배포
+
+상단 frontmatter가 HF Space 설정으로 자동 인식됩니다.
+
+## 파일 구조
+
+```
+fabagent/
+├── app.py                       # Streamlit 엔트리포인트
+├── components/                  # UI 컴포넌트 (사이드바·헤더·Tier 카드·cascade)
+├── core/
+│   ├── schema.py                # Tier1~4 TypedDict 계약
+│   └── pipeline.py              # 알람 → Tier 데이터 라우터, REAL_AGENT_ALARMS
+├── agents/
+│   ├── orchestrator.py          # 4-Tier 결정론적 시퀀서 + lru_cache
+│   ├── detection.py             # Tier 1 (IsoForest, SECOM/PHM 디스패치)
+│   ├── cause.py impact.py response.py  # Tier 2/3/4 LLM + RAG
+│   ├── llm.py                   # OpenAI 클라이언트 + 모델 설정
+│   └── rag/
+│       ├── store.py             # 키워드 매칭 + 백엔드 디스패치
+│       ├── faiss_store.py       # FAISS 벡터 검색
+│       ├── learn.py             # 자가 학습 루프 (INC-AUTO-*.md 자동 기록)
+│       └── knowledge/           # 도메인 문서 (INC/FMEA/SOP/FLOW)
+├── data/
+│   ├── demo.py                  # 알람 정의 + A2/A3 fallback 하드코딩
+│   ├── wip.py                   # 영향 WIP 결정론 데이터
+│   ├── secom/                   # SECOM 로더 + 전처리 + raw .data
+│   └── phm2016/                 # PHM 2016 CMP 로더 + 사전 집계 CSV
+├── experiments/                 # 정량 비교 실험 결과
+│   ├── tier1_detection/         # D1
+│   ├── retrieval_compare/       # D2
+│   └── multi_vs_single/         # D5
+├── styles/main.css              # 디자인 시스템
+└── tests/
+```
 
-M1·M2는 `core/schema.py`만 고정되면 완전히 병렬로 작업할 수 있습니다.
+## 기술 스택
+
+- **프론트**: Streamlit 1.36+
+- **백엔드**: OpenAI SDK (`gpt-5-mini`)
+- **ML**: scikit-learn (IsolationForest, LOF, OC-SVM)
+- **RAG**: 키워드 매칭 (기본) / sentence-transformers + FAISS (옵션)
+- **데이터**: pandas, UCI SECOM, PHM 2016 CMP Data Challenge
+- **Python**: 3.11+
 
-## 브랜치 전략
+## 한계와 향후 확장
 
-- 기본 브랜치는 `develop`이며, 직접 푸시하지 않고 PR로만 머지합니다.
-- `feat/streamlit-ui` · `feat/agents-a1` · `feat/css-port` 로 작업을 분담합니다.
-- `core/schema.py`를 가장 먼저 작은 PR로 머지한 뒤 병렬 작업을 시작합니다.
+- **SECOM의 익명성**: 590개 센서가 어느 공정·물리량인지 비공개라 A1/A2의 step 라벨은 시연용 narrative
+- **knowledge 문서**: 시연용 합성 도메인 문서 (실 fab의 사내 SOP·인시던트 DB로 교체 가능)
+- **고도화 방향**: LangGraph 기반 동적 분기, Autoencoder 앙상블, NetworkX 공정 의존성 그래프 시각화, 실 MES/SPC 어댑터 연동

agents/rag/faiss_store.py

@@ -0,0 +1,38 @@
+"""FAISS 벡터 검색 백엔드
+
+sentence-transformers로 knowledge 문서를 임베딩하고 FAISS IndexFlatIP로 코사인
+유사도 검색을 수행한다. 키워드 매칭(store.py)과 동일 시그니처로 교체 가능.
+
+모델: paraphrase-multilingual-MiniLM-L12-v2 (한국어 포함 50+ 언어)
+"""
+from functools import lru_cache
+
+import faiss
+import numpy as np
+from sentence_transformers import SentenceTransformer
+
+from agents.rag.store import _knowledge_docs
+
+MODEL_NAME = "paraphrase-multilingual-MiniLM-L12-v2"
+
+
+@lru_cache(maxsize=1)
+def _build_index():
+    """knowledge 문서를 임베딩하고 FAISS index 구축, 첫 호출 시 1회"""
+    model = SentenceTransformer(MODEL_NAME)
+    docs = _knowledge_docs()
+    doc_ids = list(docs.keys())
+    texts = list(docs.values())
+    embeddings = model.encode(texts, normalize_embeddings=True)
+    dim = embeddings.shape[1]
+    index = faiss.IndexFlatIP(dim)  # 내적 = 코사인 (normalize 됨)
+    index.add(embeddings.astype(np.float32))
+    return model, index, doc_ids
+
+
+def faiss_search(query: str, top_k: int = 3) -> list[str]:
+    """쿼리와 의미적으로 유사한 문서 ID를 코사인 유사도 내림차순으로 반환"""
+    model, index, doc_ids = _build_index()
+    q_emb = model.encode([query], normalize_embeddings=True).astype(np.float32)
+    _, idx = index.search(q_emb, top_k)
+    return [doc_ids[i] for i in idx[0] if i >= 0]

agents/rag/learn.py

@@ -0,0 +1,79 @@
+"""자가 학습 루프 - 운영자 승인 시 분석 결과를 인시던트 DB에 자동 기록
+
+승인된 분석을 INC-AUTO-{date}-{alarm_id}.md로 저장하면 RAG 검색 대상에 포함되어
+향후 유사 알람의 원인 분석/대응 권고 정확도가 점진적으로 향상됨
+
+PDF 기획서 "자가 학습 루프 - 운영자 승인 결과가 인시던트 DB에 즉시 반영" 대응 항목
+"""
+from datetime import datetime
+from pathlib import Path
+
+from agents.rag.store import KNOWLEDGE_DIR
+from core.schema import TierData
+
+
+def record_incident(alarm: dict, tier_data: TierData, work_order_id: str) -> Path:
+    """승인된 분석을 인시던트 .md로 저장, 저장된 경로 반환"""
+    date = datetime.now().strftime("%Y-%m-%d")
+    doc_id = f"INC-AUTO-{date}-{alarm['id']}"
+    path = KNOWLEDGE_DIR / f"{doc_id}.md"
+    path.write_text(_render_doc(doc_id, alarm, tier_data, work_order_id), encoding="utf-8")
+    return path
+
+
+def _render_doc(doc_id: str, alarm: dict, td: TierData, work_order: str) -> str:
+    now = datetime.now().strftime("%Y-%m-%d %H:%M")
+    t1, t2, t3, t4 = td["tier1"], td["tier2"], td["tier3"], td["tier4"]
+
+    causes_md = "\n".join(
+        f"{i+1}. {c['name']} ({c['pct']}%) - {c['evidence']}"
+        for i, c in enumerate(t2["causes"])
+    )
+    deps_md = ", ".join(
+        f"{d['stage']}({d['delta']}, {d['tag']})" for d in t3["dependencies"]
+    )
+    impact_md = ", ".join(
+        f"{l['label']} {l['lots']} lot/{l['wafers']}장" for l in t3["impact_lots"]
+    )
+
+    def actions(items):
+        return "\n".join(
+            f"- {a['text']}" + (f" ({a['meta']})" if a.get("meta") else "")
+            for a in items
+        )
+
+    return f"""# {doc_id} - {alarm['title']} 자동 기록
+
+## 분류
+- 유형: 자동 학습 인시던트 (운영자 승인 기반)
+- 공정: {alarm['title']}
+- lot: {alarm['lot_id']}
+- 발생 알림: {alarm['time']}
+- 기록 시각: {now}
+- 작업지시서: {work_order}
+
+## Tier 1 이상 탐지
+- 이상 점수: {t1['score']}
+- 기여 센서 Top: {', '.join(f['name'] for f in t1['features'])}
+- 영향 lot: {t1['lot']['id']} ({t1['lot']['wafers']}장)
+
+## Tier 2 추정 원인
+{causes_md}
+
+## Tier 3 영향 평가
+- 예상 수율 손실: {t3['yield_loss']} %p
+- 공정 의존성: {deps_md}
+- 영향 WIP: {impact_md}
+
+## Tier 4 채택 권고
+
+### 즉시 조치
+{actions(t4['immediate'])}
+
+### 중장기 조치
+{actions(t4['longterm'])}
+
+## 운영자 결정
+- 권고 채택: 전체 승인
+- 작업지시서 발행: {work_order}
+"""

agents/rag/store.py

@@ -1,11 +1,15 @@
-"""RAG 도메인 지식 검색
+"""RAG 도메인 지식 검색 - 키워드 매칭 + FAISS 벡터 백엔드 디스패치
 
 knowledge/ 의 마크다운 문서(INC-*, FMEA-*, SOP-* 등)를 검색해
-원인 분석(Tier 2)·대응 권고(Tier 4) 에이전트에 근거를 제공
+원인 분석(Tier 2)·대응 권고(Tier 4) 에이전트에 근거를 제공한다.
+
+기본은 키워드 매칭 (작은 코퍼스에 빠르고 정확)
+환경 변수 RAG_BACKEND=faiss 설정 시 sentence-transformer + FAISS 벡터 검색으로
+교체 (의미 유사도, 동의어 처리에 우위, 코퍼스 확장 시 유리)
 
-문서가 소수라 벡터DB는 오버스펙, 키워드 매칭으로 충분
 파일명이 곧 citation ID (예: INC-2024-0312.md -> "INC-2024-0312")
 """
+import os
 import re
 from pathlib import Path
 
@@ -28,8 +32,8 @@ def _knowledge_docs() -> dict[str, str]:
     return docs
 
 
-def search(query: str, top_k: int = 3) -> list[str]:
-    """쿼리와 관련된 문서 ID를 키워드 매칭으로 반환, 관련도 내림차순"""
+def keyword_search(query: str, top_k: int = 3) -> list[str]:
+    """키워드 매칭 - 쿼리 단어가 문서 본문에 등장하는 빈도 합계로 랭킹"""
     keywords = [w for w in re.split(r"\W+", query.lower()) if len(w) >= 2]
     scored = []
     for doc_id, text in _knowledge_docs().items():
@@ -39,3 +43,12 @@ def search(query: str, top_k: int = 3) -> list[str]:
             scored.append((doc_id, hits))
     scored.sort(key=lambda x: -x[1])
     return [doc_id for doc_id, _ in scored[:top_k]]
+
+
+def search(query: str, top_k: int = 3) -> list[str]:
+    """기본 검색 진입점, 환경변수 RAG_BACKEND로 백엔드 전환"""
+    if os.getenv("RAG_BACKEND", "keyword").lower() == "faiss":
+        from agents.rag.faiss_store import faiss_search
+
+        return faiss_search(query, top_k)
+    return keyword_search(query, top_k)

components/tiers.py

@@ -134,12 +134,23 @@ def _render_action_bar():
 def _on_approve():
     ss = st.session_state
     ss.approved = True
+
+    # 자가 학습 - 현재 알람의 분석 결과를 knowledge로 자동 기록
+    alarm = next((a for a in ss.alarms if a["id"] == ss.selected_alarm_id), None)
+    work_order = f"W-{datetime.now():%Y%m%d}-{ss.selected_alarm_id[1:].zfill(3)}"
+    if alarm:
+        from agents.rag.learn import record_incident
+        tier_data = get_tier_data(ss.selected_alarm_id)
+        if tier_data:
+            record_incident(alarm, tier_data, work_order)
+
+    # 알람 상태 업데이트
     for a in ss.alarms:
         if a["id"] == ss.selected_alarm_id:
             a["status"] = "done"
             a["time"] = "방금 전"
             break
-    work_order = f"W-{datetime.now():%Y%m%d}-001"
+
     st.toast(
         f"✓ 작업 지시서 {work_order} 생성 완료 · 인시던트 DB 자동 기록",
         icon="✅",

data/phm2016/loader.py

@@ -19,6 +19,8 @@ import pandas as pd
 RAW_DIR = Path(__file__).parent / "raw"
 TRAIN_TRAJ_DIR = RAW_DIR / "CMP-data" / "training"
 TRAIN_LABEL = RAW_DIR / "CMP-training-removalrate.csv"
+# 사전 집계 캐시 (배포용, ~400KB) - raw 없이도 동작
+CACHED_FEATURES = Path(__file__).parent / "phm_cmp_features.csv"
 
 # 집계 대상 센서 컬럼, 진짜 의미 있는 이름들
 SENSOR_COLS = [
@@ -46,19 +48,32 @@ SENSOR_COLS = [
 
 @lru_cache(maxsize=1)
 def load_phm_cmp() -> tuple[pd.DataFrame, pd.Series]:
-    """trajectory 전체를 (WAFER_ID, STAGE)별로 평균 집계해 wafer-stage 단위 feature 반환
+    """캐시 CSV가 있으면 그걸 사용, 없으면 raw trajectory 집계 후 캐시 저장
 
     features: (N, 19) - 센서 평균값
     labels: (N,) - AVG_REMOVAL_RATE
     index: MultiIndex (WAFER_ID, STAGE)
     """
+    if CACHED_FEATURES.exists():
+        return _load_cached()
+    return _build_and_cache()
+
+
+def _load_cached() -> tuple[pd.DataFrame, pd.Series]:
+    df = pd.read_csv(CACHED_FEATURES, index_col=["WAFER_ID", "STAGE"])
+    labels = df["AVG_REMOVAL_RATE"]
+    features = df.drop(columns=["AVG_REMOVAL_RATE"])
+    return features, labels
+
+
+def _build_and_cache() -> tuple[pd.DataFrame, pd.Series]:
+    """raw trajectory에서 wafer-stage 단위로 평균 집계, 결과를 캐시 CSV로 저장"""
     if not TRAIN_TRAJ_DIR.exists() or not TRAIN_LABEL.exists():
         raise FileNotFoundError(
-            f"PHM 2016 CMP 데이터가 없음, {RAW_DIR}에 데이터셋을 두세요 "
-            "(data/phm2016/README.md 참고)"
+            f"PHM 2016 CMP 데이터가 없음, {RAW_DIR}에 데이터셋을 두거나 "
+            f"{CACHED_FEATURES} 캐시 파일이 필요합니다 (data/phm2016/README.md 참고)"
         )
 
-    # trajectory 파일 전체 로드 후 (WAFER_ID, STAGE)별 평균
     frames = []
     for path in sorted(TRAIN_TRAJ_DIR.glob("CMP-training-*.csv")):
         df = pd.read_csv(path, usecols=["WAFER_ID", "STAGE"] + SENSOR_COLS)
@@ -69,6 +84,13 @@ def load_phm_cmp() -> tuple[pd.DataFrame, pd.Series]:
     labels_df = pd.read_csv(TRAIN_LABEL)
     labels_df = labels_df.set_index(["WAFER_ID", "STAGE"])["AVG_REMOVAL_RATE"]
 
-    # feature와 label 인덱스 정합
     common = features.index.intersection(labels_df.index)
-    return features.loc[common], labels_df.loc[common]
+    features = features.loc[common]
+    labels = labels_df.loc[common]
+
+    # 캐시 저장 (배포 시 raw 없이도 동작하도록)
+    combined = features.copy()
+    combined["AVG_REMOVAL_RATE"] = labels
+    combined.to_csv(CACHED_FEATURES)
+
+    return features, labels

data/secom/raw/secom_labels.data

@@ -0,0 +1,1567 @@
+-1 "19/07/2008 11:55:00"
+-1 "19/07/2008 12:32:00"
+1 "19/07/2008 13:17:00"
+-1 "19/07/2008 14:43:00"
+-1 "19/07/2008 15:22:00"
+-1 "19/07/2008 17:53:00"
+-1 "19/07/2008 19:44:00"
+-1 "19/07/2008 19:45:00"
+-1 "19/07/2008 20:24:00"
+-1 "19/07/2008 21:35:00"
+1 "19/07/2008 21:57:00"
+1 "19/07/2008 22:52:00"
+-1 "20/07/2008 03:35:00"
+-1 "21/07/2008 08:21:00"
+1 "21/07/2008 11:53:00"
+-1 "22/07/2008 00:03:00"
+-1 "22/07/2008 02:59:00"
+-1 "22/07/2008 08:41:00"
+-1 "22/07/2008 11:47:00"
+-1 "22/07/2008 14:00:00"
+-1 "22/07/2008 15:30:00"
+-1 "23/07/2008 05:15:00"
+-1 "23/07/2008 19:22:00"
+1 "25/07/2008 15:23:00"
+-1 "27/07/2008 04:18:00"
+-1 "27/07/2008 09:37:00"
+-1 "27/07/2008 11:10:00"
+-1 "27/07/2008 15:46:00"
+-1 "27/07/2008 16:06:00"
+-1 "27/07/2008 16:49:00"
+-1 "27/07/2008 20:24:00"
+-1 "27/07/2008 22:28:00"
+-1 "27/07/2008 22:28:00"
+-1 "27/07/2008 23:14:00"
+-1 "28/07/2008 03:31:00"
+-1 "28/07/2008 03:48:00"
+-1 "28/07/2008 04:37:00"
+-1 "28/07/2008 05:36:00"
+1 "28/07/2008 06:45:00"
+-1 "28/07/2008 08:36:00"
+1 "28/07/2008 15:11:00"
+-1 "28/07/2008 23:57:00"
+-1 "29/07/2008 04:08:00"
+-1 "29/07/2008 05:16:00"
+-1 "29/07/2008 06:19:00"
+1 "29/07/2008 08:23:00"
+-1 "29/07/2008 11:47:00"
+-1 "29/07/2008 15:41:00"
+1 "29/07/2008 15:49:00"
+1 "29/07/2008 17:05:00"
+1 "29/07/2008 18:08:00"
+-1 "29/07/2008 21:07:00"
+-1 "29/07/2008 23:14:00"
+-1 "29/07/2008 23:19:00"
+-1 "30/07/2008 06:30:00"
+-1 "30/07/2008 07:08:00"
+-1 "30/07/2008 12:02:00"
+1 "30/07/2008 12:29:00"
+1 "30/07/2008 21:16:00"
+-1 "31/07/2008 13:57:00"
+-1 "31/07/2008 15:36:00"
+-1 "31/07/2008 17:07:00"
+1 "31/07/2008 20:18:00"
+-1 "01/08/2008 02:02:00"
+1 "01/08/2008 05:52:00"
+-1 "01/08/2008 10:20:00"
+-1 "01/08/2008 10:26:00"
+-1 "01/08/2008 11:28:00"
+-1 "01/08/2008 12:29:00"
+-1 "01/08/2008 14:07:00"
+-1 "01/08/2008 15:10:00"
+-1 "02/08/2008 03:39:00"
+-1 "03/08/2008 13:09:00"
+-1 "03/08/2008 14:06:00"
+-1 "03/08/2008 14:25:00"
+-1 "03/08/2008 15:03:00"
+-1 "03/08/2008 15:27:00"
+-1 "03/08/2008 16:00:00"
+-1 "03/08/2008 17:00:00"
+-1 "03/08/2008 17:58:00"
+-1 "03/08/2008 20:23:00"
+-1 "03/08/2008 22:52:00"
+1 "04/08/2008 00:39:00"
+-1 "04/08/2008 03:45:00"
+-1 "04/08/2008 03:49:00"
+-1 "04/08/2008 13:19:00"
+-1 "04/08/2008 14:04:00"
+-1 "04/08/2008 14:31:00"
+-1 "04/08/2008 15:29:00"
+-1 "04/08/2008 16:15:00"
+-1 "04/08/2008 16:31:00"
+-1 "04/08/2008 17:11:00"
+-1 "04/08/2008 17:46:00"
+-1 "04/08/2008 18:24:00"
+-1 "04/08/2008 19:58:00"
+-1 "04/08/2008 20:32:00"
+1 "04/08/2008 20:58:00"
+-1 "04/08/2008 21:43:00"
+-1 "04/08/2008 22:51:00"
+-1 "04/08/2008 23:33:00"
+-1 "05/08/2008 00:04:00"
+-1 "05/08/2008 01:12:00"
+-1 "05/08/2008 02:15:00"
+-1 "05/08/2008 02:36:00"
+-1 "05/08/2008 02:45:00"
+-1 "05/08/2008 03:17:00"
+-1 "05/08/2008 03:36:00"
+-1 "05/08/2008 03:54:00"
+-1 "05/08/2008 04:32:00"
+-1 "05/08/2008 04:35:00"
+-1 "05/08/2008 05:11:00"
+-1 "05/08/2008 06:10:00"
+-1 "05/08/2008 06:11:00"
+-1 "05/08/2008 06:21:00"
+-1 "05/08/2008 07:12:00"
+1 "05/08/2008 07:12:00"
+-1 "05/08/2008 07:48:00"
+-1 "05/08/2008 08:50:00"
+-1 "05/08/2008 09:48:00"
+-1 "05/08/2008 14:17:00"
+-1 "05/08/2008 14:45:00"
+-1 "05/08/2008 15:45:00"
+-1 "05/08/2008 20:04:00"
+-1 "05/08/2008 21:07:00"
+-1 "05/08/2008 21:22:00"
+-1 "05/08/2008 23:02:00"
+-1 "05/08/2008 23:06:00"
+-1 "05/08/2008 23:38:00"
+-1 "06/08/2008 00:04:00"
+-1 "06/08/2008 02:20:00"
+-1 "06/08/2008 03:22:00"
+1 "06/08/2008 05:40:00"
+-1 "06/08/2008 09:56:00"
+-1 "06/08/2008 09:57:00"
+-1 "06/08/2008 12:33:00"
+-1 "06/08/2008 13:35:00"
+-1 "06/08/2008 17:35:00"
+-1 "06/08/2008 18:00:00"
+-1 "06/08/2008 18:34:00"
+-1 "06/08/2008 19:13:00"
+-1 "06/08/2008 19:24:00"
+-1 "06/08/2008 20:00:00"
+-1 "06/08/2008 20:17:00"
+-1 "06/08/2008 21:15:00"
+-1 "06/08/2008 22:27:00"
+-1 "06/08/2008 23:40:00"
+-1 "06/08/2008 23:45:00"
+-1 "07/08/2008 00:51:00"
+-1 "07/08/2008 07:30:00"
+-1 "07/08/2008 08:21:00"
+-1 "07/08/2008 08:45:00"
+-1 "07/08/2008 08:55:00"
+-1 "07/08/2008 10:12:00"
+-1 "07/08/2008 11:10:00"
+1 "07/08/2008 11:40:00"
+-1 "08/08/2008 04:55:00"
+-1 "08/08/2008 07:26:00"
+1 "08/08/2008 08:44:00"
+1 "08/08/2008 12:37:00"
+-1 "08/08/2008 20:21:00"
+-1 "08/08/2008 21:22:00"
+-1 "08/08/2008 21:53:00"
+-1 "08/08/2008 22:06:00"
+-1 "08/08/2008 22:25:00"
+-1 "09/08/2008 02:15:00"
+-1 "09/08/2008 02:37:00"
+-1 "09/08/2008 06:05:00"
+1 "09/08/2008 09:16:00"
+-1 "09/08/2008 09:22:00"
+1 "09/08/2008 11:42:00"
+-1 "09/08/2008 17:34:00"
+-1 "09/08/2008 19:03:00"
+-1 "09/08/2008 20:04:00"
+-1 "09/08/2008 21:07:00"
+-1 "09/08/2008 22:15:00"
+-1 "09/08/2008 22:37:00"
+-1 "09/08/2008 23:17:00"
+-1 "09/08/2008 23:34:00"
+-1 "10/08/2008 00:09:00"
+-1 "10/08/2008 00:46:00"
+1 "10/08/2008 06:00:00"
+-1 "10/08/2008 06:41:00"
+1 "10/08/2008 07:01:00"
+-1 "10/08/2008 11:16:00"
+-1 "10/08/2008 11:49:00"
+-1 "10/08/2008 12:22:00"
+1 "10/08/2008 15:59:00"
+-1 "10/08/2008 19:07:00"
+1 "10/08/2008 20:07:00"
+1 "10/08/2008 22:26:00"
+-1 "11/08/2008 03:06:00"
+-1 "11/08/2008 04:09:00"
+-1 "11/08/2008 05:15:00"
+-1 "11/08/2008 11:35:00"
+-1 "11/08/2008 12:38:00"
+-1 "12/08/2008 04:23:00"
+-1 "12/08/2008 06:16:00"
+-1 "12/08/2008 10:54:00"
+-1 "12/08/2008 11:29:00"
+-1 "12/08/2008 12:04:00"
+-1 "13/08/2008 02:48:00"
+-1 "15/08/2008 03:26:00"
+-1 "15/08/2008 04:14:00"
+-1 "15/08/2008 05:13:00"
+-1 "15/08/2008 09:38:00"
+-1 "15/08/2008 10:14:00"
+-1 "15/08/2008 11:42:00"
+-1 "15/08/2008 18:19:00"
+-1 "15/08/2008 19:19:00"
+-1 "15/08/2008 20:00:00"
+-1 "15/08/2008 20:03:00"
+-1 "15/08/2008 23:31:00"
+-1 "16/08/2008 02:21:00"
+-1 "16/08/2008 05:47:00"
+-1 "16/08/2008 06:52:00"
+-1 "16/08/2008 07:33:00"
+-1 "16/08/2008 08:39:00"
+-1 "16/08/2008 08:41:00"
+1 "16/08/2008 09:44:00"
+-1 "16/08/2008 13:51:00"
+-1 "16/08/2008 14:30:00"
+-1 "16/08/2008 15:16:00"
+1 "16/08/2008 15:19:00"
+-1 "16/08/2008 23:48:00"
+-1 "17/08/2008 00:52:00"
+-1 "17/08/2008 01:53:00"
+-1 "17/08/2008 03:17:00"
+-1 "17/08/2008 03:18:00"
+-1 "17/08/2008 03:46:00"
+-1 "17/08/2008 04:19:00"
+-1 "17/08/2008 05:21:00"
+1 "17/08/2008 12:16:00"
+-1 "17/08/2008 13:34:00"
+-1 "17/08/2008 14:21:00"
+-1 "17/08/2008 15:25:00"
+1 "17/08/2008 21:26:00"
+1 "17/08/2008 22:03:00"
+-1 "17/08/2008 22:21:00"
+1 "17/08/2008 23:04:00"
+-1 "17/08/2008 23:20:00"
+1 "17/08/2008 23:55:00"
+1 "18/08/2008 00:02:00"
+-1 "18/08/2008 00:20:00"
+1 "18/08/2008 01:23:00"
+1 "18/08/2008 04:01:00"
+-1 "18/08/2008 04:12:00"
+-1 "18/08/2008 04:35:00"
+-1 "18/08/2008 05:03:00"
+-1 "18/08/2008 05:25:00"
+-1 "18/08/2008 06:07:00"
+-1 "18/08/2008 06:11:00"
+-1 "18/08/2008 06:26:00"
+-1 "18/08/2008 07:11:00"
+-1 "18/08/2008 07:23:00"
+-1 "18/08/2008 07:29:00"
+-1 "18/08/2008 08:18:00"
+-1 "18/08/2008 08:31:00"
+-1 "18/08/2008 10:13:00"
+-1 "18/08/2008 11:04:00"
+-1 "18/08/2008 12:00:00"
+-1 "18/08/2008 12:06:00"
+-1 "18/08/2008 12:22:00"
+-1 "18/08/2008 12:49:00"
+-1 "18/08/2008 13:07:00"
+-1 "18/08/2008 14:01:00"
+-1 "18/08/2008 14:04:00"
+-1 "18/08/2008 14:45:00"
+-1 "18/08/2008 15:00:00"
+-1 "18/08/2008 15:26:00"
+-1 "18/08/2008 15:30:00"
+-1 "18/08/2008 15:33:00"
+-1 "18/08/2008 15:41:00"
+-1 "18/08/2008 16:19:00"
+1 "18/08/2008 17:13:00"
+-1 "18/08/2008 17:16:00"
+-1 "18/08/2008 17:37:00"
+-1 "18/08/2008 19:24:00"
+1 "18/08/2008 19:54:00"
+-1 "18/08/2008 23:00:00"
+-1 "18/08/2008 23:03:00"
+-1 "18/08/2008 23:47:00"
+-1 "19/08/2008 01:10:00"
+1 "19/08/2008 03:59:00"
+-1 "19/08/2008 04:58:00"
+-1 "19/08/2008 05:09:00"
+-1 "19/08/2008 05:11:00"
+-1 "19/08/2008 05:11:00"
+-1 "19/08/2008 05:40:00"
+-1 "19/08/2008 05:41:00"
+-1 "19/08/2008 05:53:00"
+-1 "19/08/2008 05:54:00"
+1 "19/08/2008 05:56:00"
+-1 "19/08/2008 06:24:00"
+-1 "19/08/2008 07:43:00"
+1 "19/08/2008 07:58:00"
+-1 "19/08/2008 08:01:00"
+-1 "19/08/2008 08:07:00"
+-1 "19/08/2008 08:33:00"
+-1 "19/08/2008 09:04:00"
+-1 "19/08/2008 09:05:00"
+-1 "19/08/2008 10:46:00"
+-1 "19/08/2008 11:11:00"
+-1 "19/08/2008 11:13:00"
+-1 "19/08/2008 11:13:00"
+-1 "19/08/2008 11:28:00"
+-1 "19/08/2008 11:44:00"
+-1 "19/08/2008 11:46:00"
+-1 "19/08/2008 11:57:00"
+-1 "19/08/2008 11:58:00"
+-1 "19/08/2008 12:30:00"
+-1 "19/08/2008 15:33:00"
+-1 "19/08/2008 18:12:00"
+-1 "19/08/2008 18:12:00"
+-1 "19/08/2008 18:30:00"
+-1 "19/08/2008 18:30:00"
+-1 "19/08/2008 19:38:00"
+-1 "19/08/2008 20:53:00"
+-1 "20/08/2008 00:05:00"
+-1 "20/08/2008 01:33:00"
+-1 "20/08/2008 01:42:00"
+-1 "20/08/2008 02:13:00"
+1 "20/08/2008 02:27:00"
+-1 "20/08/2008 03:00:00"
+1 "20/08/2008 03:12:00"
+-1 "20/08/2008 06:36:00"
+-1 "20/08/2008 07:12:00"
+1 "20/08/2008 08:40:00"
+1 "20/08/2008 09:17:00"
+-1 "20/08/2008 10:26:00"
+-1 "20/08/2008 16:08:00"
+-1 "20/08/2008 16:16:00"
+-1 "20/08/2008 17:09:00"
+-1 "20/08/2008 17:35:00"
+-1 "20/08/2008 17:36:00"
+-1 "20/08/2008 18:29:00"
+-1 "20/08/2008 18:35:00"
+1 "20/08/2008 19:20:00"
+-1 "20/08/2008 20:19:00"
+-1 "20/08/2008 21:33:00"
+-1 "20/08/2008 21:41:00"
+-1 "20/08/2008 21:58:00"
+-1 "20/08/2008 22:13:00"
+-1 "20/08/2008 22:45:00"
+-1 "20/08/2008 23:18:00"
+1 "20/08/2008 23:43:00"
+-1 "21/08/2008 02:11:00"
+-1 "21/08/2008 02:46:00"
+-1 "21/08/2008 04:22:00"
+-1 "21/08/2008 04:29:00"
+-1 "21/08/2008 05:06:00"
+-1 "21/08/2008 05:15:00"
+1 "21/08/2008 06:10:00"
+-1 "21/08/2008 11:22:00"
+-1 "21/08/2008 12:00:00"
+-1 "21/08/2008 12:31:00"
+-1 "21/08/2008 12:33:00"
+-1 "21/08/2008 12:43:00"
+-1 "21/08/2008 12:52:00"
+-1 "21/08/2008 13:11:00"
+-1 "21/08/2008 13:19:00"
+-1 "21/08/2008 13:29:00"
+-1 "21/08/2008 13:45:00"
+-1 "21/08/2008 14:47:00"
+-1 "21/08/2008 15:05:00"
+-1 "21/08/2008 15:06:00"
+-1 "21/08/2008 15:32:00"
+-1 "21/08/2008 15:48:00"
+-1 "21/08/2008 15:50:00"
+1 "21/08/2008 16:19:00"
+-1 "21/08/2008 16:36:00"
+-1 "21/08/2008 17:25:00"
+-1 "21/08/2008 17:44:00"
+-1 "21/08/2008 18:04:00"
+1 "21/08/2008 18:05:00"
+-1 "21/08/2008 18:39:00"
+-1 "21/08/2008 18:53:00"
+-1 "21/08/2008 19:12:00"
+-1 "21/08/2008 19:12:00"
+-1 "21/08/2008 19:46:00"
+-1 "21/08/2008 20:19:00"
+-1 "21/08/2008 20:21:00"
+-1 "21/08/2008 20:37:00"
+-1 "21/08/2008 21:32:00"
+-1 "21/08/2008 22:25:00"
+-1 "21/08/2008 22:41:00"
+-1 "21/08/2008 23:08:00"
+-1 "21/08/2008 23:27:00"
+-1 "21/08/2008 23:53:00"
+-1 "21/08/2008 23:57:00"
+-1 "22/08/2008 00:43:00"
+-1 "22/08/2008 00:44:00"
+-1 "22/08/2008 00:47:00"
+1 "22/08/2008 01:04:00"
+-1 "22/08/2008 01:26:00"
+-1 "22/08/2008 01:27:00"
+-1 "22/08/2008 01:54:00"
+-1 "22/08/2008 02:01:00"
+-1 "22/08/2008 02:25:00"
+-1 "22/08/2008 02:52:00"
+-1 "22/08/2008 03:18:00"
+-1 "22/08/2008 03:58:00"
+-1 "22/08/2008 04:40:00"
+-1 "22/08/2008 04:57:00"
+-1 "22/08/2008 05:21:00"
+-1 "22/08/2008 05:31:00"
+-1 "22/08/2008 05:32:00"
+1 "22/08/2008 06:00:00"
+-1 "22/08/2008 07:04:00"
+-1 "22/08/2008 07:17:00"
+-1 "22/08/2008 07:27:00"
+-1 "22/08/2008 09:29:00"
+-1 "22/08/2008 09:29:00"
+-1 "22/08/2008 10:01:00"
+-1 "22/08/2008 10:18:00"
+-1 "22/08/2008 10:47:00"
+-1 "22/08/2008 11:32:00"
+-1 "22/08/2008 12:13:00"
+-1 "22/08/2008 12:37:00"
+-1 "22/08/2008 12:42:00"
+-1 "22/08/2008 13:22:00"
+-1 "22/08/2008 14:45:00"
+-1 "22/08/2008 15:25:00"
+-1 "22/08/2008 15:39:00"
+-1 "22/08/2008 15:54:00"
+1 "22/08/2008 19:14:00"
+-1 "22/08/2008 23:46:00"
+-1 "22/08/2008 23:47:00"
+-1 "22/08/2008 23:49:00"
+-1 "22/08/2008 23:56:00"
+-1 "23/08/2008 02:04:00"
+-1 "23/08/2008 03:26:00"
+-1 "23/08/2008 04:50:00"
+-1 "23/08/2008 04:52:00"
+-1 "23/08/2008 05:57:00"
+-1 "23/08/2008 05:57:00"
+-1 "23/08/2008 05:58:00"
+-1 "23/08/2008 05:58:00"
+-1 "23/08/2008 09:24:00"
+-1 "23/08/2008 12:58:00"
+-1 "23/08/2008 13:02:00"
+-1 "23/08/2008 15:10:00"
+1 "23/08/2008 15:29:00"
+-1 "24/08/2008 02:32:00"
+-1 "24/08/2008 04:00:00"
+-1 "24/08/2008 07:12:00"
+-1 "24/08/2008 07:14:00"
+-1 "24/08/2008 10:09:00"
+-1 "24/08/2008 10:14:00"
+1 "24/08/2008 13:03:00"
+-1 "25/08/2008 09:29:00"
+-1 "27/08/2008 00:46:00"
+-1 "27/08/2008 11:38:00"
+-1 "27/08/2008 12:22:00"
+-1 "27/08/2008 13:35:00"
+-1 "27/08/2008 14:18:00"
+-1 "27/08/2008 17:14:00"
+-1 "27/08/2008 22:48:00"
+-1 "27/08/2008 22:58:00"
+-1 "28/08/2008 03:03:00"
+-1 "28/08/2008 03:04:00"
+-1 "28/08/2008 03:29:00"
+-1 "28/08/2008 03:38:00"
+-1 "28/08/2008 03:45:00"
+-1 "28/08/2008 04:10:00"
+-1 "28/08/2008 04:20:00"
+-1 "28/08/2008 04:54:00"
+-1 "28/08/2008 05:00:00"
+-1 "28/08/2008 06:15:00"
+-1 "28/08/2008 06:39:00"
+-1 "28/08/2008 06:56:00"
+-1 "28/08/2008 07:40:00"
+-1 "28/08/2008 07:51:00"
+-1 "28/08/2008 08:41:00"
+-1 "28/08/2008 09:46:00"
+-1 "28/08/2008 09:49:00"
+-1 "28/08/2008 10:15:00"
+-1 "28/08/2008 10:28:00"
+-1 "28/08/2008 10:32:00"
+-1 "28/08/2008 11:19:00"
+-1 "28/08/2008 12:44:00"
+-1 "28/08/2008 14:48:00"
+-1 "28/08/2008 15:19:00"
+-1 "28/08/2008 16:01:00"
+-1 "28/08/2008 16:43:00"
+-1 "28/08/2008 16:52:00"
+-1 "28/08/2008 17:21:00"
+-1 "28/08/2008 17:27:00"
+-1 "28/08/2008 17:32:00"
+-1 "28/08/2008 18:05:00"
+-1 "28/08/2008 18:10:00"
+-1 "28/08/2008 18:15:00"
+-1 "28/08/2008 18:25:00"
+-1 "28/08/2008 18:47:00"
+-1 "28/08/2008 18:57:00"
+-1 "28/08/2008 20:20:00"
+1 "28/08/2008 20:39:00"
+-1 "28/08/2008 21:03:00"
+-1 "28/08/2008 21:52:00"
+-1 "28/08/2008 22:57:00"
+-1 "28/08/2008 23:40:00"
+-1 "28/08/2008 23:42:00"
+-1 "29/08/2008 00:49:00"
+-1 "29/08/2008 01:06:00"
+-1 "29/08/2008 02:38:00"
+-1 "29/08/2008 02:39:00"
+-1 "29/08/2008 03:27:00"
+-1 "29/08/2008 03:33:00"
+-1 "29/08/2008 03:46:00"
+1 "29/08/2008 04:55:00"
+-1 "29/08/2008 04:56:00"
+-1 "29/08/2008 05:16:00"
+-1 "29/08/2008 05:54:00"
+-1 "29/08/2008 05:54:00"
+-1 "29/08/2008 05:57:00"
+-1 "29/08/2008 05:57:00"
+-1 "29/08/2008 06:35:00"
+-1 "29/08/2008 06:37:00"
+-1 "29/08/2008 06:40:00"
+1 "29/08/2008 06:45:00"
+-1 "29/08/2008 06:54:00"
+-1 "29/08/2008 07:18:00"
+-1 "29/08/2008 07:22:00"
+-1 "29/08/2008 07:24:00"
+-1 "29/08/2008 07:33:00"
+-1 "29/08/2008 07:33:00"
+-1 "29/08/2008 08:10:00"
+-1 "29/08/2008 08:20:00"
+-1 "29/08/2008 08:41:00"
+-1 "29/08/2008 08:45:00"
+-1 "29/08/2008 12:21:00"
+-1 "29/08/2008 12:22:00"
+-1 "29/08/2008 12:37:00"
+-1 "29/08/2008 13:04:00"
+-1 "29/08/2008 13:14:00"
+-1 "29/08/2008 13:27:00"
+-1 "29/08/2008 13:49:00"
+-1 "29/08/2008 14:11:00"
+-1 "29/08/2008 14:18:00"
+-1 "29/08/2008 14:30:00"
+-1 "29/08/2008 15:01:00"
+-1 "29/08/2008 15:43:00"
+-1 "29/08/2008 16:26:00"
+-1 "29/08/2008 20:21:00"
+-1 "29/08/2008 20:49:00"
+-1 "29/08/2008 21:25:00"
+-1 "29/08/2008 21:46:00"
+-1 "29/08/2008 21:54:00"
+-1 "29/08/2008 22:11:00"
+-1 "29/08/2008 22:45:00"
+-1 "29/08/2008 22:49:00"
+-1 "29/08/2008 22:56:00"
+-1 "30/08/2008 00:01:00"
+-1 "30/08/2008 00:05:00"
+-1 "30/08/2008 00:57:00"
+-1 "30/08/2008 01:29:00"
+-1 "30/08/2008 02:08:00"
+-1 "30/08/2008 02:22:00"
+-1 "30/08/2008 02:32:00"
+-1 "30/08/2008 02:39:00"
+-1 "30/08/2008 02:50:00"
+-1 "30/08/2008 03:10:00"
+-1 "30/08/2008 04:20:00"
+-1 "30/08/2008 04:24:00"
+-1 "30/08/2008 04:55:00"
+-1 "30/08/2008 05:05:00"
+-1 "30/08/2008 05:05:00"
+-1 "30/08/2008 05:38:00"
+-1 "30/08/2008 05:51:00"
+-1 "30/08/2008 07:50:00"
+-1 "30/08/2008 07:54:00"
+-1 "30/08/2008 08:10:00"
+-1 "30/08/2008 08:29:00"
+-1 "30/08/2008 08:42:00"
+-1 "30/08/2008 08:53:00"
+-1 "30/08/2008 09:13:00"
+-1 "30/08/2008 09:37:00"
+1 "30/08/2008 09:45:00"
+-1 "30/08/2008 10:15:00"
+-1 "30/08/2008 10:16:00"
+-1 "30/08/2008 11:12:00"
+-1 "30/08/2008 11:14:00"
+-1 "30/08/2008 11:18:00"
+-1 "30/08/2008 11:57:00"
+1 "30/08/2008 14:10:00"
+-1 "30/08/2008 14:23:00"
+-1 "30/08/2008 14:37:00"
+-1 "30/08/2008 15:06:00"
+-1 "30/08/2008 15:16:00"
+-1 "30/08/2008 15:37:00"
+-1 "30/08/2008 15:50:00"
+-1 "30/08/2008 15:57:00"
+-1 "30/08/2008 16:24:00"
+-1 "30/08/2008 17:05:00"
+-1 "30/08/2008 17:49:00"
+-1 "30/08/2008 19:21:00"
+-1 "30/08/2008 19:27:00"
+-1 "30/08/2008 20:11:00"
+-1 "30/08/2008 20:53:00"
+-1 "30/08/2008 21:53:00"
+-1 "30/08/2008 23:51:00"
+-1 "31/08/2008 02:57:00"
+1 "31/08/2008 04:46:00"
+-1 "31/08/2008 05:05:00"
+-1 "31/08/2008 09:11:00"
+-1 "31/08/2008 10:36:00"
+1 "31/08/2008 10:59:00"
+-1 "31/08/2008 10:59:00"
+-1 "31/08/2008 11:03:00"
+-1 "31/08/2008 15:09:00"
+-1 "31/08/2008 15:13:00"
+-1 "31/08/2008 16:32:00"
+-1 "31/08/2008 20:24:00"
+-1 "31/08/2008 21:04:00"
+-1 "31/08/2008 21:46:00"
+-1 "31/08/2008 21:58:00"
+-1 "31/08/2008 22:32:00"
+-1 "31/08/2008 22:48:00"
+-1 "31/08/2008 22:57:00"
+-1 "01/09/2008 00:01:00"
+-1 "01/09/2008 00:39:00"
+-1 "01/09/2008 00:45:00"
+-1 "01/09/2008 00:53:00"
+-1 "01/09/2008 05:29:00"
+-1 "01/09/2008 05:32:00"
+-1 "01/09/2008 06:12:00"
+-1 "01/09/2008 06:21:00"
+-1 "01/09/2008 06:52:00"
+-1 "01/09/2008 08:18:00"
+-1 "01/09/2008 09:06:00"
+-1 "01/09/2008 09:19:00"
+-1 "01/09/2008 09:54:00"
+-1 "01/09/2008 10:57:00"
+-1 "01/09/2008 11:40:00"
+-1 "01/09/2008 16:20:00"
+1 "01/09/2008 19:54:00"
+-1 "01/09/2008 20:51:00"
+-1 "01/09/2008 22:05:00"
+-1 "01/09/2008 23:05:00"
+-1 "01/09/2008 23:19:00"
+-1 "01/09/2008 23:24:00"
+-1 "01/09/2008 23:28:00"
+-1 "01/09/2008 23:39:00"
+-1 "01/09/2008 23:45:00"
+-1 "01/09/2008 23:59:00"
+-1 "02/09/2008 00:26:00"
+-1 "02/09/2008 00:31:00"
+-1 "02/09/2008 00:46:00"
+-1 "02/09/2008 01:09:00"
+-1 "02/09/2008 01:10:00"
+-1 "02/09/2008 01:10:00"
+-1 "02/09/2008 01:36:00"
+-1 "02/09/2008 01:50:00"
+-1 "02/09/2008 02:02:00"
+-1 "02/09/2008 02:11:00"
+-1 "02/09/2008 02:50:00"
+-1 "02/09/2008 02:52:00"
+-1 "02/09/2008 03:33:00"
+-1 "02/09/2008 03:35:00"
+-1 "02/09/2008 03:35:00"
+-1 "02/09/2008 04:19:00"
+-1 "02/09/2008 04:22:00"
+-1 "02/09/2008 05:46:00"
+-1 "02/09/2008 06:03:00"
+-1 "02/09/2008 06:19:00"
+-1 "02/09/2008 06:32:00"
+-1 "02/09/2008 06:48:00"
+-1 "02/09/2008 07:22:00"
+-1 "02/09/2008 07:32:00"
+-1 "02/09/2008 08:15:00"
+-1 "02/09/2008 08:54:00"
+-1 "02/09/2008 09:36:00"
+-1 "02/09/2008 09:37:00"
+-1 "02/09/2008 10:13:00"
+-1 "02/09/2008 10:21:00"
+-1 "02/09/2008 10:25:00"
+-1 "02/09/2008 10:33:00"
+-1 "02/09/2008 11:08:00"
+-1 "02/09/2008 11:22:00"
+-1 "02/09/2008 11:47:00"
+-1 "02/09/2008 11:49:00"
+-1 "02/09/2008 12:01:00"
+-1 "02/09/2008 12:17:00"
+-1 "02/09/2008 12:26:00"
+-1 "02/09/2008 12:30:00"
+-1 "02/09/2008 12:44:00"
+-1 "02/09/2008 13:05:00"
+-1 "02/09/2008 13:08:00"
+-1 "02/09/2008 13:27:00"
+-1 "02/09/2008 13:40:00"
+-1 "02/09/2008 14:09:00"
+-1 "02/09/2008 14:21:00"
+-1 "02/09/2008 14:34:00"
+-1 "02/09/2008 15:29:00"
+-1 "02/09/2008 16:10:00"
+-1 "02/09/2008 16:50:00"
+-1 "02/09/2008 16:54:00"
+-1 "02/09/2008 16:58:00"
+-1 "02/09/2008 17:10:00"
+-1 "02/09/2008 17:18:00"
+-1 "02/09/2008 18:38:00"
+-1 "02/09/2008 18:52:00"
+-1 "02/09/2008 19:18:00"
+-1 "02/09/2008 20:18:00"
+-1 "02/09/2008 20:52:00"
+-1 "02/09/2008 20:58:00"
+-1 "02/09/2008 20:59:00"
+-1 "03/09/2008 00:18:00"
+-1 "03/09/2008 00:45:00"
+-1 "03/09/2008 00:58:00"
+1 "03/09/2008 01:15:00"
+-1 "03/09/2008 01:32:00"
+-1 "03/09/2008 04:18:00"
+-1 "03/09/2008 06:26:00"
+-1 "03/09/2008 09:38:00"
+-1 "03/09/2008 14:24:00"
+-1 "03/09/2008 14:29:00"
+-1 "03/09/2008 17:22:00"
+-1 "03/09/2008 18:16:00"
+-1 "03/09/2008 18:16:00"
+-1 "03/09/2008 20:00:00"
+-1 "03/09/2008 20:10:00"
+-1 "04/09/2008 08:01:00"
+-1 "04/09/2008 08:22:00"
+-1 "04/09/2008 08:33:00"
+-1 "04/09/2008 08:41:00"
+-1 "04/09/2008 08:53:00"
+-1 "04/09/2008 11:50:00"
+-1 "04/09/2008 14:09:00"
+-1 "04/09/2008 14:11:00"
+-1 "04/09/2008 14:48:00"
+-1 "04/09/2008 16:30:00"
+-1 "04/09/2008 16:41:00"
+-1 "04/09/2008 16:56:00"
+-1 "04/09/2008 17:30:00"
+-1 "05/09/2008 15:30:00"
+-1 "05/09/2008 21:48:00"
+-1 "07/09/2008 00:08:00"
+-1 "07/09/2008 00:12:00"
+-1 "07/09/2008 00:49:00"
+-1 "07/09/2008 10:52:00"
+-1 "07/09/2008 12:04:00"
+-1 "07/09/2008 12:20:00"
+-1 "07/09/2008 13:02:00"
+-1 "07/09/2008 18:02:00"
+-1 "07/09/2008 18:24:00"
+-1 "07/09/2008 18:49:00"
+-1 "07/09/2008 19:29:00"
+-1 "07/09/2008 19:31:00"
+-1 "07/09/2008 20:12:00"
+-1 "07/09/2008 20:50:00"
+-1 "07/09/2008 23:33:00"
+-1 "08/09/2008 00:32:00"
+-1 "08/09/2008 00:40:00"
+-1 "08/09/2008 01:24:00"
+-1 "08/09/2008 03:32:00"
+-1 "08/09/2008 03:48:00"
+-1 "08/09/2008 04:21:00"
+-1 "08/09/2008 06:37:00"
+-1 "08/09/2008 07:14:00"
+-1 "08/09/2008 07:41:00"
+-1 "08/09/2008 18:46:00"
+-1 "08/09/2008 19:14:00"
+-1 "08/09/2008 19:15:00"
+-1 "08/09/2008 19:25:00"
+-1 "08/09/2008 19:33:00"
+-1 "08/09/2008 20:00:00"
+-1 "08/09/2008 21:36:00"
+-1 "08/09/2008 21:53:00"
+-1 "08/09/2008 22:17:00"
+-1 "09/09/2008 00:25:00"
+-1 "09/09/2008 03:53:00"
+-1 "09/09/2008 08:44:00"
+-1 "10/09/2008 07:26:00"
+-1 "10/09/2008 09:21:00"
+-1 "10/09/2008 20:52:00"
+-1 "10/09/2008 20:53:00"
+-1 "10/09/2008 21:56:00"
+-1 "10/09/2008 23:30:00"
+-1 "10/09/2008 23:43:00"
+-1 "11/09/2008 00:17:00"
+-1 "11/09/2008 05:20:00"
+-1 "11/09/2008 06:04:00"
+-1 "11/09/2008 07:43:00"
+-1 "11/09/2008 08:06:00"
+-1 "11/09/2008 08:58:00"
+-1 "11/09/2008 09:24:00"
+-1 "11/09/2008 12:47:00"
+-1 "11/09/2008 13:36:00"
+-1 "11/09/2008 14:20:00"
+-1 "11/09/2008 14:24:00"
+-1 "11/09/2008 14:31:00"
+-1 "11/09/2008 15:05:00"
+-1 "11/09/2008 15:08:00"
+-1 "11/09/2008 16:28:00"
+-1 "11/09/2008 21:13:00"
+1 "11/09/2008 22:07:00"
+-1 "11/09/2008 23:10:00"
+1 "11/09/2008 23:48:00"
+-1 "12/09/2008 05:11:00"
+-1 "12/09/2008 05:55:00"
+-1 "12/09/2008 06:38:00"
+-1 "12/09/2008 06:41:00"
+-1 "12/09/2008 06:45:00"
+-1 "12/09/2008 07:21:00"
+-1 "12/09/2008 07:29:00"
+-1 "12/09/2008 07:38:00"
+-1 "12/09/2008 11:57:00"
+-1 "12/09/2008 13:08:00"
+-1 "12/09/2008 14:52:00"
+-1 "12/09/2008 15:40:00"
+-1 "12/09/2008 16:23:00"
+-1 "12/09/2008 18:01:00"
+-1 "12/09/2008 18:52:00"
+-1 "12/09/2008 19:36:00"
+-1 "12/09/2008 20:28:00"
+-1 "12/09/2008 21:12:00"
+-1 "12/09/2008 22:25:00"
+-1 "12/09/2008 22:45:00"
+-1 "12/09/2008 23:28:00"
+-1 "12/09/2008 23:40:00"
+-1 "13/09/2008 00:25:00"
+-1 "13/09/2008 02:57:00"
+-1 "13/09/2008 03:14:00"
+-1 "13/09/2008 09:19:00"
+-1 "13/09/2008 10:55:00"
+-1 "13/09/2008 11:42:00"
+1 "13/09/2008 11:56:00"
+-1 "13/09/2008 12:48:00"
+-1 "13/09/2008 14:07:00"
+-1 "13/09/2008 16:04:00"
+-1 "13/09/2008 16:04:00"
+1 "13/09/2008 20:06:00"
+-1 "14/09/2008 02:50:00"
+-1 "14/09/2008 02:59:00"
+-1 "14/09/2008 15:02:00"
+-1 "14/09/2008 15:46:00"
+-1 "14/09/2008 16:34:00"
+-1 "14/09/2008 17:40:00"
+-1 "14/09/2008 18:27:00"
+-1 "14/09/2008 18:27:00"
+-1 "14/09/2008 19:13:00"
+-1 "14/09/2008 20:00:00"
+-1 "14/09/2008 20:43:00"
+-1 "14/09/2008 20:45:00"
+-1 "14/09/2008 21:31:00"
+-1 "14/09/2008 23:00:00"
+-1 "15/09/2008 02:04:00"
+-1 "15/09/2008 10:03:00"
+-1 "15/09/2008 11:40:00"
+-1 "15/09/2008 12:27:00"
+-1 "15/09/2008 16:10:00"
+-1 "15/09/2008 16:59:00"
+-1 "15/09/2008 19:54:00"
+-1 "15/09/2008 21:56:00"
+-1 "15/09/2008 22:13:00"
+-1 "15/09/2008 22:45:00"
+-1 "15/09/2008 22:54:00"
+-1 "15/09/2008 23:37:00"
+-1 "16/09/2008 07:39:00"
+-1 "16/09/2008 07:48:00"
+-1 "16/09/2008 08:50:00"
+-1 "16/09/2008 08:52:00"
+-1 "16/09/2008 09:37:00"
+-1 "16/09/2008 17:07:00"
+-1 "16/09/2008 18:09:00"
+-1 "16/09/2008 18:23:00"
+-1 "16/09/2008 19:34:00"
+-1 "16/09/2008 19:45:00"
+-1 "16/09/2008 21:21:00"
+-1 "16/09/2008 22:45:00"
+-1 "17/09/2008 02:01:00"
+1 "17/09/2008 04:23:00"
+-1 "17/09/2008 07:14:00"
+-1 "17/09/2008 07:28:00"
+-1 "17/09/2008 21:52:00"
+-1 "18/09/2008 00:57:00"
+-1 "18/09/2008 01:40:00"
+-1 "18/09/2008 01:43:00"
+-1 "18/09/2008 02:01:00"
+-1 "18/09/2008 03:58:00"
+-1 "18/09/2008 03:58:00"
+-1 "18/09/2008 05:44:00"
+-1 "18/09/2008 05:58:00"
+-1 "18/09/2008 08:04:00"
+-1 "18/09/2008 10:48:00"
+-1 "18/09/2008 13:48:00"
+-1 "18/09/2008 17:22:00"
+-1 "18/09/2008 18:12:00"
+-1 "18/09/2008 18:56:00"
+-1 "18/09/2008 21:12:00"
+-1 "18/09/2008 21:56:00"
+-1 "18/09/2008 23:00:00"
+-1 "18/09/2008 23:05:00"
+-1 "18/09/2008 23:44:00"
+-1 "19/09/2008 00:30:00"
+-1 "19/09/2008 01:57:00"
+-1 "19/09/2008 02:44:00"
+-1 "19/09/2008 04:42:00"
+-1 "19/09/2008 05:33:00"
+-1 "19/09/2008 05:57:00"
+-1 "19/09/2008 06:01:00"
+-1 "19/09/2008 06:44:00"
+-1 "19/09/2008 07:14:00"
+-1 "19/09/2008 07:18:00"
+-1 "19/09/2008 08:08:00"
+-1 "19/09/2008 08:52:00"
+-1 "19/09/2008 09:12:00"
+-1 "19/09/2008 09:47:00"
+-1 "19/09/2008 10:45:00"
+-1 "19/09/2008 11:13:00"
+-1 "19/09/2008 11:54:00"
+-1 "19/09/2008 13:59:00"
+-1 "19/09/2008 15:31:00"
+-1 "19/09/2008 16:40:00"
+1 "19/09/2008 16:58:00"
+-1 "19/09/2008 17:47:00"
+-1 "19/09/2008 18:11:00"
+-1 "19/09/2008 18:33:00"
+-1 "19/09/2008 19:18:00"
+-1 "19/09/2008 19:29:00"
+-1 "19/09/2008 20:05:00"
+-1 "19/09/2008 20:07:00"
+-1 "19/09/2008 20:10:00"
+-1 "19/09/2008 20:17:00"
+1 "19/09/2008 20:19:00"
+-1 "19/09/2008 21:08:00"
+1 "19/09/2008 21:09:00"
+-1 "19/09/2008 21:13:00"
+-1 "19/09/2008 21:14:00"
+1 "19/09/2008 21:59:00"
+-1 "19/09/2008 23:26:00"
+-1 "19/09/2008 23:33:00"
+-1 "20/09/2008 00:18:00"
+-1 "20/09/2008 00:33:00"
+-1 "20/09/2008 03:58:00"
+-1 "20/09/2008 04:13:00"
+-1 "20/09/2008 04:28:00"
+-1 "20/09/2008 05:22:00"
+-1 "20/09/2008 05:25:00"
+-1 "20/09/2008 05:34:00"
+-1 "20/09/2008 06:08:00"
+-1 "20/09/2008 07:21:00"
+-1 "20/09/2008 08:14:00"
+-1 "20/09/2008 10:51:00"
+-1 "20/09/2008 11:18:00"
+-1 "20/09/2008 12:20:00"
+-1 "20/09/2008 12:26:00"
+-1 "20/09/2008 12:27:00"
+-1 "20/09/2008 12:36:00"
+-1 "20/09/2008 12:57:00"
+-1 "20/09/2008 14:00:00"
+-1 "20/09/2008 14:06:00"
+-1 "20/09/2008 14:42:00"
+-1 "20/09/2008 14:50:00"
+-1 "20/09/2008 14:53:00"
+-1 "20/09/2008 15:02:00"
+-1 "20/09/2008 15:19:00"
+-1 "20/09/2008 15:37:00"
+-1 "20/09/2008 15:46:00"
+-1 "20/09/2008 16:43:00"
+-1 "20/09/2008 16:44:00"
+-1 "20/09/2008 17:26:00"
+-1 "20/09/2008 17:29:00"
+-1 "20/09/2008 18:03:00"
+-1 "20/09/2008 18:21:00"
+-1 "20/09/2008 18:56:00"
+-1 "20/09/2008 19:14:00"
+-1 "20/09/2008 19:21:00"
+-1 "20/09/2008 23:34:00"
+-1 "21/09/2008 00:48:00"
+-1 "21/09/2008 01:06:00"
+-1 "21/09/2008 02:07:00"
+-1 "21/09/2008 03:04:00"
+-1 "21/09/2008 03:47:00"
+-1 "21/09/2008 04:17:00"
+-1 "21/09/2008 04:49:00"
+-1 "21/09/2008 07:30:00"
+-1 "21/09/2008 07:55:00"
+-1 "21/09/2008 08:22:00"
+-1 "21/09/2008 08:49:00"
+-1 "21/09/2008 09:16:00"
+-1 "21/09/2008 09:31:00"
+-1 "21/09/2008 10:07:00"
+-1 "21/09/2008 10:49:00"
+-1 "21/09/2008 11:33:00"
+-1 "21/09/2008 12:04:00"
+-1 "21/09/2008 14:30:00"
+-1 "21/09/2008 14:31:00"
+-1 "21/09/2008 14:33:00"
+-1 "21/09/2008 15:15:00"
+-1 "21/09/2008 15:26:00"
+-1 "21/09/2008 15:34:00"
+-1 "21/09/2008 16:43:00"
+-1 "21/09/2008 17:26:00"
+-1 "21/09/2008 17:30:00"
+-1 "21/09/2008 18:12:00"
+-1 "21/09/2008 19:19:00"
+-1 "21/09/2008 20:35:00"
+-1 "21/09/2008 21:54:00"
+-1 "21/09/2008 22:02:00"
+-1 "22/09/2008 00:12:00"
+-1 "22/09/2008 00:28:00"
+-1 "22/09/2008 00:47:00"
+-1 "22/09/2008 01:43:00"
+-1 "22/09/2008 03:51:00"
+-1 "22/09/2008 06:05:00"
+-1 "22/09/2008 07:12:00"
+-1 "22/09/2008 10:43:00"
+-1 "22/09/2008 11:50:00"
+-1 "22/09/2008 12:11:00"
+-1 "22/09/2008 12:45:00"
+-1 "22/09/2008 13:03:00"
+-1 "22/09/2008 14:24:00"
+-1 "22/09/2008 14:27:00"
+-1 "22/09/2008 14:36:00"
+-1 "22/09/2008 16:25:00"
+-1 "22/09/2008 21:30:00"
+-1 "22/09/2008 21:31:00"
+-1 "22/09/2008 22:15:00"
+-1 "22/09/2008 22:17:00"
+-1 "22/09/2008 23:01:00"
+-1 "22/09/2008 23:10:00"
+-1 "23/09/2008 00:57:00"
+-1 "23/09/2008 06:44:00"
+-1 "23/09/2008 06:44:00"
+-1 "23/09/2008 06:52:00"
+-1 "23/09/2008 07:58:00"
+-1 "23/09/2008 08:11:00"
+-1 "23/09/2008 09:43:00"
+1 "23/09/2008 11:14:00"
+-1 "23/09/2008 11:42:00"
+-1 "23/09/2008 11:43:00"
+-1 "23/09/2008 12:29:00"
+-1 "23/09/2008 12:31:00"
+-1 "23/09/2008 12:46:00"
+-1 "23/09/2008 13:37:00"
+-1 "23/09/2008 15:23:00"
+-1 "23/09/2008 15:33:00"
+-1 "23/09/2008 15:58:00"
+-1 "23/09/2008 18:10:00"
+-1 "23/09/2008 18:36:00"
+-1 "23/09/2008 18:56:00"
+-1 "23/09/2008 20:18:00"
+-1 "23/09/2008 20:59:00"
+-1 "23/09/2008 21:26:00"
+-1 "23/09/2008 22:25:00"
+-1 "24/09/2008 02:19:00"
+-1 "24/09/2008 02:44:00"
+-1 "24/09/2008 03:32:00"
+-1 "24/09/2008 03:40:00"
+-1 "24/09/2008 05:13:00"
+-1 "24/09/2008 05:22:00"
+-1 "24/09/2008 06:13:00"
+-1 "24/09/2008 06:14:00"
+-1 "24/09/2008 07:23:00"
+-1 "24/09/2008 10:10:00"
+-1 "24/09/2008 10:53:00"
+-1 "24/09/2008 10:59:00"
+-1 "24/09/2008 11:47:00"
+-1 "24/09/2008 12:04:00"
+-1 "24/09/2008 12:35:00"
+-1 "24/09/2008 13:01:00"
+1 "24/09/2008 15:05:00"
+-1 "24/09/2008 15:39:00"
+-1 "24/09/2008 15:42:00"
+-1 "24/09/2008 15:43:00"
+-1 "24/09/2008 16:26:00"
+-1 "24/09/2008 17:54:00"
+-1 "24/09/2008 18:05:00"
+-1 "25/09/2008 01:05:00"
+-1 "25/09/2008 02:03:00"
+-1 "25/09/2008 06:21:00"
+-1 "25/09/2008 07:12:00"
+-1 "25/09/2008 07:58:00"
+-1 "25/09/2008 08:45:00"
+-1 "25/09/2008 09:31:00"
+-1 "25/09/2008 10:23:00"
+-1 "25/09/2008 11:08:00"
+-1 "25/09/2008 11:09:00"
+-1 "25/09/2008 11:55:00"
+-1 "25/09/2008 13:42:00"
+-1 "25/09/2008 14:08:00"
+-1 "25/09/2008 14:28:00"
+-1 "25/09/2008 15:12:00"
+-1 "25/09/2008 15:40:00"
+-1 "25/09/2008 15:55:00"
+-1 "25/09/2008 16:44:00"
+-1 "25/09/2008 17:28:00"
+-1 "25/09/2008 17:29:00"
+-1 "25/09/2008 19:49:00"
+-1 "25/09/2008 22:05:00"
+-1 "25/09/2008 23:04:00"
+-1 "25/09/2008 23:48:00"
+-1 "26/09/2008 00:19:00"
+-1 "26/09/2008 01:21:00"
+-1 "26/09/2008 02:26:00"
+-1 "26/09/2008 03:12:00"
+-1 "26/09/2008 03:29:00"
+-1 "26/09/2008 04:14:00"
+-1 "26/09/2008 05:32:00"
+-1 "26/09/2008 11:08:00"
+-1 "26/09/2008 13:36:00"
+-1 "26/09/2008 13:45:00"
+-1 "26/09/2008 14:20:00"
+-1 "26/09/2008 14:51:00"
+-1 "26/09/2008 15:06:00"
+-1 "26/09/2008 15:36:00"
+-1 "26/09/2008 15:49:00"
+-1 "26/09/2008 16:31:00"
+-1 "26/09/2008 17:59:00"
+-1 "26/09/2008 18:54:00"
+-1 "26/09/2008 19:38:00"
+-1 "26/09/2008 19:45:00"
+-1 "26/09/2008 19:55:00"
+-1 "26/09/2008 20:23:00"
+-1 "26/09/2008 20:31:00"
+-1 "27/09/2008 00:16:00"
+-1 "27/09/2008 00:17:00"
+-1 "27/09/2008 01:29:00"
+-1 "27/09/2008 03:53:00"
+-1 "27/09/2008 10:10:00"
+-1 "27/09/2008 10:13:00"
+-1 "27/09/2008 11:04:00"
+-1 "27/09/2008 11:45:00"
+-1 "27/09/2008 11:54:00"
+-1 "27/09/2008 12:26:00"
+-1 "27/09/2008 12:28:00"
+-1 "27/09/2008 12:36:00"
+-1 "27/09/2008 13:14:00"
+-1 "27/09/2008 13:19:00"
+-1 "27/09/2008 14:01:00"
+-1 "27/09/2008 14:05:00"
+-1 "27/09/2008 14:46:00"
+-1 "27/09/2008 15:09:00"
+-1 "27/09/2008 15:30:00"
+-1 "27/09/2008 16:13:00"
+-1 "27/09/2008 17:26:00"
+-1 "27/09/2008 18:14:00"
+-1 "27/09/2008 19:16:00"
+-1 "27/09/2008 20:46:00"
+-1 "27/09/2008 21:45:00"
+-1 "28/09/2008 00:51:00"
+-1 "28/09/2008 01:22:00"
+-1 "28/09/2008 03:55:00"
+1 "28/09/2008 04:45:00"
+-1 "28/09/2008 05:31:00"
+-1 "28/09/2008 06:22:00"
+-1 "28/09/2008 07:38:00"
+-1 "28/09/2008 07:52:00"
+-1 "28/09/2008 08:40:00"
+-1 "28/09/2008 08:41:00"
+1 "28/09/2008 09:50:00"
+-1 "28/09/2008 17:05:00"
+-1 "28/09/2008 17:06:00"
+-1 "28/09/2008 17:49:00"
+-1 "28/09/2008 17:55:00"
+-1 "28/09/2008 18:34:00"
+-1 "28/09/2008 18:43:00"
+-1 "28/09/2008 19:18:00"
+-1 "28/09/2008 19:29:00"
+-1 "28/09/2008 20:02:00"
+-1 "28/09/2008 20:42:00"
+-1 "28/09/2008 21:08:00"
+-1 "28/09/2008 21:27:00"
+-1 "28/09/2008 22:44:00"
+-1 "28/09/2008 23:54:00"
+-1 "29/09/2008 02:06:00"
+-1 "29/09/2008 03:07:00"
+-1 "29/09/2008 03:50:00"
+-1 "29/09/2008 04:18:00"
+-1 "29/09/2008 04:47:00"
+-1 "29/09/2008 05:32:00"
+-1 "29/09/2008 07:53:00"
+-1 "29/09/2008 09:15:00"
+-1 "29/09/2008 11:13:00"
+-1 "29/09/2008 11:53:00"
+-1 "29/09/2008 12:25:00"
+-1 "29/09/2008 14:03:00"
+-1 "29/09/2008 14:04:00"
+-1 "29/09/2008 14:08:00"
+-1 "29/09/2008 15:09:00"
+-1 "29/09/2008 15:16:00"
+-1 "29/09/2008 16:27:00"
+-1 "29/09/2008 17:21:00"
+-1 "29/09/2008 21:26:00"
+1 "29/09/2008 21:48:00"
+-1 "29/09/2008 22:13:00"
+-1 "30/09/2008 01:33:00"
+-1 "30/09/2008 01:36:00"
+1 "30/09/2008 02:44:00"
+-1 "30/09/2008 04:01:00"
+-1 "30/09/2008 04:42:00"
+-1 "30/09/2008 08:44:00"
+-1 "30/09/2008 12:31:00"
+-1 "30/09/2008 13:29:00"
+-1 "30/09/2008 14:20:00"
+-1 "30/09/2008 14:36:00"
+-1 "30/09/2008 15:16:00"
+-1 "30/09/2008 15:26:00"
+-1 "30/09/2008 16:13:00"
+-1 "30/09/2008 16:50:00"
+-1 "30/09/2008 17:27:00"
+-1 "30/09/2008 19:17:00"
+-1 "30/09/2008 19:17:00"
+-1 "30/09/2008 21:39:00"
+-1 "30/09/2008 21:42:00"
+-1 "30/09/2008 23:34:00"
+-1 "30/09/2008 23:58:00"
+-1 "01/10/2008 00:20:00"
+-1 "01/10/2008 01:11:00"
+-1 "01/10/2008 02:53:00"
+1 "01/10/2008 05:56:00"
+-1 "01/10/2008 07:19:00"
+-1 "01/10/2008 07:28:00"
+-1 "01/10/2008 08:15:00"
+-1 "01/10/2008 09:15:00"
+-1 "01/10/2008 09:16:00"
+-1 "01/10/2008 14:22:00"
+-1 "01/10/2008 15:16:00"
+-1 "01/10/2008 16:43:00"
+-1 "01/10/2008 19:57:00"
+-1 "01/10/2008 21:15:00"
+-1 "01/10/2008 21:50:00"
+-1 "01/10/2008 22:47:00"
+-1 "01/10/2008 23:34:00"
+-1 "02/10/2008 00:21:00"
+-1 "02/10/2008 00:22:00"
+1 "02/10/2008 03:17:00"
+-1 "02/10/2008 03:27:00"
+-1 "02/10/2008 04:11:00"
+-1 "02/10/2008 04:23:00"
+-1 "02/10/2008 04:24:00"
+-1 "02/10/2008 06:49:00"
+-1 "02/10/2008 07:35:00"
+-1 "02/10/2008 07:58:00"
+-1 "02/10/2008 08:24:00"
+-1 "02/10/2008 08:52:00"
+-1 "02/10/2008 08:57:00"
+1 "02/10/2008 09:10:00"
+-1 "02/10/2008 09:42:00"
+-1 "02/10/2008 10:03:00"
+1 "02/10/2008 14:07:00"
+1 "02/10/2008 14:10:00"
+-1 "02/10/2008 14:54:00"
+-1 "02/10/2008 15:10:00"
+-1 "02/10/2008 15:44:00"
+-1 "02/10/2008 15:58:00"
+-1 "02/10/2008 16:32:00"
+-1 "02/10/2008 16:58:00"
+-1 "02/10/2008 17:00:00"
+-1 "02/10/2008 17:19:00"
+-1 "02/10/2008 19:21:00"
+-1 "02/10/2008 19:25:00"
+-1 "02/10/2008 20:54:00"
+1 "02/10/2008 21:32:00"
+-1 "02/10/2008 21:33:00"
+-1 "02/10/2008 22:23:00"
+-1 "02/10/2008 22:33:00"
+-1 "02/10/2008 23:32:00"
+-1 "02/10/2008 23:32:00"
+-1 "03/10/2008 00:24:00"
+-1 "03/10/2008 00:55:00"
+-1 "03/10/2008 02:41:00"
+-1 "03/10/2008 03:52:00"
+-1 "03/10/2008 03:56:00"
+-1 "03/10/2008 04:49:00"
+-1 "03/10/2008 05:01:00"
+-1 "03/10/2008 05:45:00"
+-1 "03/10/2008 06:15:00"
+-1 "03/10/2008 08:35:00"
+-1 "03/10/2008 09:30:00"
+-1 "03/10/2008 11:40:00"
+-1 "03/10/2008 12:31:00"
+-1 "03/10/2008 18:45:00"
+-1 "03/10/2008 19:23:00"
+-1 "03/10/2008 19:23:00"
+-1 "03/10/2008 20:11:00"
+-1 "03/10/2008 20:26:00"
+-1 "03/10/2008 20:35:00"
+-1 "03/10/2008 20:46:00"
+-1 "03/10/2008 21:15:00"
+-1 "03/10/2008 21:52:00"
+-1 "03/10/2008 22:08:00"
+-1 "03/10/2008 22:53:00"
+-1 "03/10/2008 23:46:00"
+-1 "04/10/2008 00:58:00"
+-1 "04/10/2008 01:49:00"
+-1 "04/10/2008 02:36:00"
+-1 "04/10/2008 03:53:00"
+-1 "04/10/2008 04:12:00"
+-1 "04/10/2008 04:45:00"
+-1 "04/10/2008 09:29:00"
+-1 "04/10/2008 15:16:00"
+-1 "04/10/2008 16:58:00"
+-1 "04/10/2008 18:15:00"
+-1 "04/10/2008 18:25:00"
+-1 "04/10/2008 18:59:00"
+-1 "04/10/2008 19:12:00"
+-1 "04/10/2008 19:47:00"
+-1 "04/10/2008 19:54:00"
+-1 "04/10/2008 20:39:00"
+-1 "04/10/2008 22:06:00"
+1 "04/10/2008 23:18:00"
+1 "05/10/2008 00:33:00"
+-1 "05/10/2008 00:33:00"
+-1 "05/10/2008 01:35:00"
+-1 "05/10/2008 04:01:00"
+-1 "05/10/2008 04:20:00"
+-1 "05/10/2008 04:48:00"
+-1 "05/10/2008 05:31:00"
+-1 "05/10/2008 05:53:00"
+-1 "05/10/2008 06:15:00"
+-1 "05/10/2008 08:02:00"
+-1 "05/10/2008 08:40:00"
+-1 "05/10/2008 09:04:00"
+-1 "05/10/2008 09:47:00"
+-1 "05/10/2008 10:00:00"
+-1 "05/10/2008 10:43:00"
+-1 "05/10/2008 11:05:00"
+-1 "05/10/2008 11:30:00"
+-1 "05/10/2008 12:03:00"
+-1 "05/10/2008 12:17:00"
+-1 "05/10/2008 13:49:00"
+-1 "05/10/2008 14:34:00"
+1 "05/10/2008 15:30:00"
+1 "05/10/2008 15:35:00"
+-1 "05/10/2008 16:15:00"
+1 "05/10/2008 16:37:00"
+1 "05/10/2008 16:49:00"
+1 "05/10/2008 18:46:00"
+-1 "05/10/2008 18:59:00"
+-1 "05/10/2008 19:45:00"
+-1 "05/10/2008 20:12:00"
+-1 "05/10/2008 20:30:00"
+-1 "05/10/2008 22:13:00"
+-1 "06/10/2008 00:41:00"
+-1 "06/10/2008 01:25:00"
+-1 "06/10/2008 01:27:00"
+-1 "06/10/2008 01:30:00"
+-1 "06/10/2008 02:44:00"
+-1 "06/10/2008 02:49:00"
+-1 "06/10/2008 03:31:00"
+1 "06/10/2008 03:50:00"
+1 "06/10/2008 03:57:00"
+-1 "06/10/2008 04:26:00"
+-1 "06/10/2008 05:21:00"
+-1 "06/10/2008 05:24:00"
+-1 "06/10/2008 06:11:00"
+-1 "06/10/2008 06:26:00"
+-1 "06/10/2008 06:49:00"
+-1 "06/10/2008 07:12:00"
+-1 "06/10/2008 07:17:00"
+-1 "06/10/2008 07:35:00"
+-1 "06/10/2008 07:38:00"
+-1 "06/10/2008 08:24:00"
+-1 "06/10/2008 08:57:00"
+-1 "06/10/2008 09:11:00"
+-1 "06/10/2008 09:57:00"
+-1 "06/10/2008 10:07:00"
+-1 "06/10/2008 11:51:00"
+-1 "06/10/2008 12:39:00"
+-1 "06/10/2008 12:42:00"
+-1 "06/10/2008 13:08:00"
+1 "06/10/2008 13:38:00"
+1 "06/10/2008 13:55:00"
+1 "06/10/2008 15:00:00"
+-1 "06/10/2008 15:01:00"
+-1 "06/10/2008 15:52:00"
+-1 "06/10/2008 16:42:00"
+-1 "06/10/2008 16:50:00"
+-1 "06/10/2008 17:26:00"
+-1 "06/10/2008 17:26:00"
+-1 "06/10/2008 17:35:00"
+-1 "06/10/2008 18:54:00"
+-1 "06/10/2008 18:55:00"
+-1 "06/10/2008 19:42:00"
+-1 "06/10/2008 20:02:00"
+-1 "06/10/2008 20:36:00"
+-1 "06/10/2008 20:47:00"
+-1 "06/10/2008 21:46:00"
+-1 "06/10/2008 21:48:00"
+-1 "06/10/2008 22:37:00"
+-1 "06/10/2008 23:03:00"
+-1 "07/10/2008 01:46:00"
+-1 "07/10/2008 03:29:00"
+-1 "07/10/2008 03:39:00"
+-1 "07/10/2008 07:32:00"
+-1 "07/10/2008 07:41:00"
+-1 "07/10/2008 08:25:00"
+-1 "07/10/2008 08:26:00"
+-1 "07/10/2008 09:11:00"
+-1 "07/10/2008 09:20:00"
+-1 "07/10/2008 09:59:00"
+-1 "07/10/2008 10:04:00"
+-1 "07/10/2008 10:10:00"
+-1 "07/10/2008 10:51:00"
+-1 "07/10/2008 11:15:00"
+-1 "07/10/2008 11:16:00"
+-1 "07/10/2008 11:36:00"
+-1 "07/10/2008 12:40:00"
+1 "07/10/2008 13:10:00"
+-1 "07/10/2008 13:48:00"
+-1 "07/10/2008 14:35:00"
+-1 "07/10/2008 16:32:00"
+-1 "07/10/2008 17:06:00"
+-1 "07/10/2008 17:28:00"
+-1 "07/10/2008 18:39:00"
+-1 "07/10/2008 19:24:00"
+-1 "07/10/2008 20:19:00"
+-1 "07/10/2008 20:39:00"
+-1 "07/10/2008 20:48:00"
+-1 "08/10/2008 04:47:00"
+-1 "08/10/2008 05:35:00"
+-1 "08/10/2008 05:38:00"
+-1 "08/10/2008 06:25:00"
+-1 "08/10/2008 07:10:00"
+-1 "08/10/2008 07:13:00"
+-1 "08/10/2008 07:16:00"
+-1 "08/10/2008 09:01:00"
+-1 "08/10/2008 09:06:00"
+-1 "08/10/2008 10:37:00"
+-1 "08/10/2008 12:11:00"
+-1 "08/10/2008 13:35:00"
+-1 "08/10/2008 13:38:00"
+-1 "08/10/2008 14:48:00"
+-1 "08/10/2008 15:37:00"
+-1 "08/10/2008 16:26:00"
+-1 "08/10/2008 17:14:00"
+-1 "08/10/2008 17:31:00"
+-1 "08/10/2008 17:41:00"
+-1 "08/10/2008 18:22:00"
+-1 "08/10/2008 20:27:00"
+-1 "08/10/2008 21:19:00"
+-1 "08/10/2008 21:45:00"
+-1 "08/10/2008 22:08:00"
+-1 "08/10/2008 23:13:00"
+-1 "08/10/2008 23:50:00"
+-1 "09/10/2008 00:23:00"
+1 "09/10/2008 04:34:00"
+-1 "09/10/2008 06:06:00"
+-1 "09/10/2008 07:38:00"
+-1 "09/10/2008 14:15:00"
+-1 "09/10/2008 15:04:00"
+1 "09/10/2008 15:55:00"
+-1 "09/10/2008 15:56:00"
+-1 "10/10/2008 10:05:00"
+-1 "10/10/2008 10:46:00"
+-1 "10/10/2008 10:53:00"
+-1 "10/10/2008 12:30:00"
+-1 "10/10/2008 15:33:00"
+-1 "10/10/2008 15:50:00"
+-1 "10/10/2008 20:02:00"
+-1 "10/10/2008 21:03:00"
+-1 "10/10/2008 21:07:00"
+-1 "10/10/2008 21:16:00"
+-1 "11/10/2008 02:55:00"
+-1 "11/10/2008 06:56:00"
+-1 "11/10/2008 07:38:00"
+-1 "11/10/2008 07:39:00"
+-1 "11/10/2008 07:42:00"
+-1 "11/10/2008 08:27:00"
+-1 "11/10/2008 14:43:00"
+-1 "11/10/2008 15:33:00"
+-1 "11/10/2008 16:25:00"
+-1 "11/10/2008 23:16:00"
+-1 "11/10/2008 23:58:00"
+-1 "12/10/2008 00:57:00"
+-1 "12/10/2008 01:50:00"
+-1 "12/10/2008 02:59:00"
+-1 "12/10/2008 03:21:00"
+-1 "12/10/2008 03:28:00"
+-1 "12/10/2008 08:19:00"
+-1 "12/10/2008 18:47:00"
+-1 "13/10/2008 03:34:00"
+-1 "13/10/2008 03:53:00"
+-1 "13/10/2008 11:27:00"
+-1 "13/10/2008 12:18:00"
+-1 "13/10/2008 14:06:00"
+-1 "13/10/2008 14:29:00"
+-1 "13/10/2008 14:30:00"
+-1 "13/10/2008 14:55:00"
+-1 "13/10/2008 15:42:00"
+-1 "13/10/2008 15:42:00"
+-1 "13/10/2008 16:13:00"
+-1 "13/10/2008 19:17:00"
+-1 "13/10/2008 19:36:00"
+-1 "13/10/2008 19:40:00"
+-1 "13/10/2008 20:10:00"
+-1 "13/10/2008 20:30:00"
+-1 "13/10/2008 20:53:00"
+-1 "13/10/2008 21:20:00"
+-1 "13/10/2008 21:47:00"
+-1 "13/10/2008 21:57:00"
+-1 "13/10/2008 22:48:00"
+-1 "13/10/2008 22:54:00"
+-1 "14/10/2008 00:35:00"
+-1 "14/10/2008 03:21:00"
+-1 "14/10/2008 03:28:00"
+-1 "14/10/2008 03:35:00"
+-1 "14/10/2008 13:13:00"
+-1 "14/10/2008 13:16:00"
+-1 "14/10/2008 14:07:00"
+-1 "14/10/2008 14:15:00"
+-1 "14/10/2008 14:43:00"
+-1 "14/10/2008 17:36:00"
+-1 "14/10/2008 19:15:00"
+-1 "14/10/2008 20:01:00"
+-1 "14/10/2008 20:30:00"
+-1 "14/10/2008 20:48:00"
+-1 "14/10/2008 21:36:00"
+-1 "14/10/2008 23:07:00"
+-1 "15/10/2008 00:03:00"
+-1 "15/10/2008 00:27:00"
+-1 "15/10/2008 00:47:00"
+-1 "15/10/2008 01:52:00"
+-1 "15/10/2008 01:52:00"
+-1 "15/10/2008 01:52:00"
+-1 "15/10/2008 02:40:00"
+-1 "15/10/2008 02:40:00"
+1 "15/10/2008 02:42:00"
+-1 "15/10/2008 03:24:00"
+-1 "15/10/2008 04:08:00"
+-1 "15/10/2008 05:13:00"
+-1 "15/10/2008 05:16:00"
+-1 "15/10/2008 06:49:00"
+-1 "15/10/2008 07:36:00"
+-1 "15/10/2008 07:55:00"
+-1 "15/10/2008 08:21:00"
+-1 "15/10/2008 09:11:00"
+-1 "15/10/2008 10:00:00"
+-1 "15/10/2008 12:53:00"
+-1 "15/10/2008 13:14:00"
+-1 "15/10/2008 13:16:00"
+-1 "15/10/2008 14:08:00"
+-1 "15/10/2008 15:11:00"
+-1 "15/10/2008 16:24:00"
+-1 "15/10/2008 17:19:00"
+-1 "15/10/2008 18:16:00"
+-1 "15/10/2008 19:15:00"
+-1 "15/10/2008 19:24:00"
+-1 "15/10/2008 21:44:00"
+-1 "15/10/2008 22:45:00"
+-1 "15/10/2008 22:54:00"
+-1 "15/10/2008 23:00:00"
+-1 "15/10/2008 23:45:00"
+-1 "16/10/2008 02:16:00"
+-1 "16/10/2008 02:16:00"
+-1 "16/10/2008 02:17:00"
+-1 "16/10/2008 02:22:00"
+-1 "16/10/2008 02:55:00"
+-1 "16/10/2008 03:56:00"
+-1 "16/10/2008 04:02:00"
+-1 "16/10/2008 04:02:00"
+-1 "16/10/2008 04:04:00"
+-1 "16/10/2008 04:47:00"
+-1 "16/10/2008 04:50:00"
+-1 "16/10/2008 04:54:00"
+-1 "16/10/2008 05:08:00"
+-1 "16/10/2008 05:13:00"
+-1 "16/10/2008 05:44:00"
+-1 "16/10/2008 05:58:00"
+-1 "16/10/2008 15:02:00"
+-1 "16/10/2008 15:13:00"
+-1 "16/10/2008 20:49:00"
+-1 "17/10/2008 05:26:00"
+-1 "17/10/2008 06:01:00"
+-1 "17/10/2008 06:07:00"

experiments/README.md

@@ -0,0 +1,52 @@
+# FabAgent Experiments
+
+핵심 의사결정마다 정량 비교 실험 + 트레이드오프 표를 남겨, 발표 시 "왜 이 선택을 했는가"를 측정된 근거로 설명할 수 있도록 합니다.
+
+## 실험 목록
+
+| ID | 실험 | 비교 | 결과 |
+|---|---|---|---|
+| **D1** | Tier 1 이상 탐지 모델 비교 | IsoForest / LOF / OC-SVM / baseline | [tier1_detection/results.md](tier1_detection/results.md) |
+| **D2** | RAG 검색 방식 비교 | 키워드 매칭 vs FAISS 벡터 | [retrieval_compare/results.md](retrieval_compare/results.md) |
+| **D5** | 멀티 에이전트 vs Single LLM | 분리·전문화 vs 통합 호출 | [multi_vs_single/results.md](multi_vs_single/results.md) |
+
+## 핵심 결정 요약
+
+각 실험에서 채택된 결정과 트레이드오프:
+
+### D1. 이상 탐지 모델 → **IsolationForest 채택**
+
+- PR-AUC 0.129로 비교 모델 중 최고
+- SECOM은 비지도 이상 탐지가 어려운 표준 벤치마크 (문헌 ROC-AUC ~0.6 범위)
+- 트레이드오프: Autoencoder/LSTM은 더 복잡한 패턴 가능하지만 학습 데이터·시간 비용 큼
+
+### D2. RAG 검색 → **키워드 매칭 기본, FAISS 옵션 제공**
+
+- 현 코퍼스(약 10문서)에서는 키워드가 빠르고 정확하며 의존성 적음
+- FAISS는 의미 우회 쿼리에서 유리, 코퍼스 확장(50+) 시 채택 권장
+- 환경변수 `RAG_BACKEND=faiss`로 즉시 전환 가능 (동일 인터페이스)
+
+### D5. 멀티 에이전트 vs Single LLM → **멀티 에이전트 채택**
+
+| 영역 | 우위 |
+|---|---|
+| 속도·비용 | Single (2.6배 빠름, 2.2배 저렴) |
+| 응답 깊이 (조치 권고 수) | Multi (1.6~1.9배 detailed) |
+| 모듈화·확장성·자가학습 | Multi |
+| schema·citation 정확도 | 동등 (양쪽 strict JSON 100%) |
+
+- 비용 절대값이 두 방식 모두 $0.01~0.02로 미미해 cost-aware할 필요는 없음
+- 운영 환경(사업부별 Tier 책임자 분리, 새 step 확장)에서는 Multi의 모듈화·확장성이 결정적
+
+## 실행 방법
+
+```bash
+# Tier 1 모델 벤치마크
+.venv/bin/python -m experiments.tier1_detection.benchmark
+
+# RAG 검색 비교
+.venv/bin/python -m experiments.retrieval_compare.benchmark
+
+# 멀티 에이전트 vs Single LLM
+.venv/bin/python -m experiments.multi_vs_single.benchmark
+```

experiments/multi_vs_single/benchmark.py

@@ -0,0 +1,342 @@
+"""Multi-Agent vs Single LLM 비교 실험 (D5)
+
+같은 알람에 대해
+- Multi-Agent: 기존 4-Tier 오케스트레이터 (Tier 1 ML + Tier 2/3/4 각각 LLM 호출)
+- Single LLM: 같은 모델로 Tier 2/3/4를 한 번에 산출
+
+으로 N회 실행한 뒤 정량 지표 비교
+- latency
+- 비용 (input/output 토큰)
+- schema 부합률 (structured output 성공률)
+- citation 정확도 (제공된 doc ID만 인용했는가)
+- 응답 일관성 (cause 집합 Jaccard, yield_loss 표준편차)
+
+실행: python -m experiments.multi_vs_single.benchmark
+결과: results.md + plots/
+"""
+import json
+import statistics
+import time
+from collections import Counter
+from pathlib import Path
+
+import matplotlib
+
+matplotlib.use("Agg")
+import matplotlib.pyplot as plt
+
+from agents.cause import TIER2_SCHEMA, SYSTEM_PROMPT as CAUSE_SYSTEM, run_cause
+from agents.detection import run_detection
+from agents.impact import LLM_PART_SCHEMA as TIER3_LLM_SCHEMA, run_impact
+from agents.llm import SUBAGENT_MODEL, client
+from agents.rag.store import _knowledge_docs
+from agents.response import LLM_PART_SCHEMA as TIER4_LLM_SCHEMA, run_response
+from data.demo import DEFAULT_ALARMS
+from data.wip import get_affected_wip
+
+N_RUNS = 3
+TARGET_ALARM = "A3"  # CMP 실데이터 (PHM 2016) 사용, 가장 흥미로운 비교 케이스
+OUT_DIR = Path(__file__).parent
+PLOTS_DIR = OUT_DIR / "plots"
+
+# input $0.25/M, output $2.00/M (gpt-5-mini 기준)
+PRICE_IN = 0.25 / 1_000_000
+PRICE_OUT = 2.00 / 1_000_000
+
+# Single LLM 통합 스키마
+COMBINED_SCHEMA = {
+    "type": "object",
+    "properties": {
+        "tier2": TIER2_SCHEMA,
+        "tier3": TIER3_LLM_SCHEMA,
+        "tier4": TIER4_LLM_SCHEMA,
+    },
+    "required": ["tier2", "tier3", "tier4"],
+    "additionalProperties": False,
+}
+
+SINGLE_SYSTEM = """당신은 반도체 공정 운영 통합 분석 전문가입니다.
+알람과 Tier 1 이상 탐지 결과, 사내 지식 문서 전체를 종합해서 Tier 2(원인 분석),
+Tier 3(영향 평가의 downstream_dependencies와 yield_loss), Tier 4(즉시·중장기 조치 +
+근거 자료)를 한 번의 응답으로 산출합니다.
+citations와 refs는 제공된 문서 ID만 사용하고 근거 없는 내용은 포함하지 않습니다."""
+
+
+def run_single_llm(alarm, tier1):
+    """Single LLM 방식 - 전체 지식 문서를 한 번에 주고 통합 산출, (data, latency, usage) 반환"""
+    docs = _knowledge_docs()
+    knowledge = "\n\n".join(f"[{d}]\n{c}" for d, c in docs.items())
+    sensors = ", ".join(f["name"] for f in tier1["features"])
+
+    user_prompt = f"""## 이상 알람
+- 공정: {alarm['title']}
+- lot: {alarm['lot_id']}
+- 이상 피처: {alarm.get('feature')} {alarm.get('feature_arrow') or ''}
+
+## Tier 1 이상 탐지
+- 이상 점수: {tier1['score']}
+- 기여 센서: {sensors}
+
+## 사내 지식 문서 (전체)
+{knowledge}
+
+위 정보를 종합해서 tier2/tier3/tier4를 JSON으로 반환하세요."""
+
+    t0 = time.time()
+    resp = client().chat.completions.create(
+        model=SUBAGENT_MODEL,
+        messages=[
+            {"role": "system", "content": SINGLE_SYSTEM},
+            {"role": "user", "content": user_prompt},
+        ],
+        response_format={
+            "type": "json_schema",
+            "json_schema": {"name": "combined", "schema": COMBINED_SCHEMA, "strict": True},
+        },
+    )
+    elapsed = time.time() - t0
+    parsed = json.loads(resp.choices[0].message.content)
+    return parsed, elapsed, resp.usage
+
+
+def run_multi_agent(alarm, tier1):
+    """Multi-Agent - cause/impact/response 순차 호출, usage는 client monkey patch로 캡처"""
+    from agents import llm as llm_mod
+
+    cli = llm_mod.client()
+    captured = []
+    real_create = cli.chat.completions.create
+
+    def patched(*args, **kwargs):
+        r = real_create(*args, **kwargs)
+        captured.append(r.usage)
+        return r
+
+    cli.chat.completions.create = patched
+    t0 = time.time()
+    try:
+        t2 = run_cause(alarm, tier1)
+        t3 = run_impact(alarm, tier1, t2)
+        t4 = run_response(alarm, tier1, t2, t3)
+    finally:
+        cli.chat.completions.create = real_create
+
+    elapsed = time.time() - t0
+    total_in = sum(u.prompt_tokens for u in captured)
+    total_out = sum(u.completion_tokens for u in captured)
+    return {"tier2": t2, "tier3": t3, "tier4": t4}, elapsed, (total_in, total_out)
+
+
+def measure(data, valid_doc_ids):
+    """결과 dict에서 평가 지표 추출"""
+    t2 = data["tier2"]
+    t3 = data["tier3"]
+    t4 = data["tier4"]
+
+    # citation 정확도
+    cites = []
+    for c in t2.get("causes", []):
+        cites.extend(c.get("citations", []))
+    refs = t4.get("refs", [])
+    # Multi의 tier4는 refs가 결정론적 (검색 결과)이라 단순 비교용으로 cite_pool에 포함
+    ref_ids = [r["id"] if isinstance(r, dict) else r for r in refs]
+    all_cites = cites + ref_ids
+    cite_valid = sum(1 for c in all_cites if c in valid_doc_ids)
+    cite_total = len(all_cites)
+    cite_acc = cite_valid / cite_total if cite_total else 1.0
+
+    # cause 이름 집합
+    causes = frozenset(c["name"] for c in t2.get("causes", []))
+
+    # 권고 수
+    n_imm = len(t4.get("immediate", []))
+    n_lng = len(t4.get("longterm", []))
+
+    # yield_loss
+    yloss = t3.get("yield_loss", 0.0)
+
+    return {
+        "cite_acc": cite_acc,
+        "cite_total": cite_total,
+        "causes": causes,
+        "n_imm": n_imm,
+        "n_lng": n_lng,
+        "yield_loss": yloss,
+        "pct_sum": sum(c["pct"] for c in t2.get("causes", [])),
+    }
+
+
+def jaccard(sets):
+    if not sets:
+        return 0.0
+    union = set()
+    inter = set(sets[0])
+    for s in sets:
+        union |= s
+        inter &= s
+    return len(inter) / len(union) if union else 0.0
+
+
+def main():
+    alarm = next(a for a in DEFAULT_ALARMS if a["id"] == TARGET_ALARM)
+    tier1 = run_detection(alarm)
+    valid_doc_ids = set(_knowledge_docs().keys())
+
+    print(f"실험 대상: {TARGET_ALARM} ({alarm['title']}), N={N_RUNS}")
+    print(f"Tier 1 score: {tier1['score']}, 기여 센서: {[f['name'] for f in tier1['features']]}\n")
+
+    multi_runs = []
+    single_runs = []
+
+    print("=== Multi-Agent ===")
+    for i in range(N_RUNS):
+        try:
+            data, lat, (tin, tout) = run_multi_agent(alarm, tier1)
+            m = measure(data, valid_doc_ids)
+            m.update({"latency": lat, "tokens_in": tin, "tokens_out": tout, "ok": True})
+            multi_runs.append(m)
+            print(f"  run {i+1}: {lat:.1f}s, in={tin}, out={tout}, cite_acc={m['cite_acc']:.2f}")
+        except Exception as e:
+            multi_runs.append({"ok": False, "error": str(e)})
+            print(f"  run {i+1}: ERROR {e}")
+
+    print("\n=== Single LLM ===")
+    for i in range(N_RUNS):
+        try:
+            data, lat, usage = run_single_llm(alarm, tier1)
+            m = measure(data, valid_doc_ids)
+            m.update({
+                "latency": lat,
+                "tokens_in": usage.prompt_tokens,
+                "tokens_out": usage.completion_tokens,
+                "ok": True,
+            })
+            single_runs.append(m)
+            print(f"  run {i+1}: {lat:.1f}s, in={usage.prompt_tokens}, out={usage.completion_tokens}, cite_acc={m['cite_acc']:.2f}")
+        except Exception as e:
+            single_runs.append({"ok": False, "error": str(e)})
+            print(f"  run {i+1}: ERROR {e}")
+
+    write_results(multi_runs, single_runs, alarm, tier1)
+    print(f"\n--- 결과 저장: {OUT_DIR / 'results.md'} ---")
+
+
+def aggregate(runs):
+    """N runs 결과 집계"""
+    ok = [r for r in runs if r.get("ok")]
+    if not ok:
+        return {"n_ok": 0, "n_fail": len(runs)}
+    return {
+        "n_ok": len(ok),
+        "n_fail": len(runs) - len(ok),
+        "schema_compliance": len(ok) / len(runs),
+        "latency_mean": statistics.mean(r["latency"] for r in ok),
+        "tokens_in_mean": statistics.mean(r["tokens_in"] for r in ok),
+        "tokens_out_mean": statistics.mean(r["tokens_out"] for r in ok),
+        "cost_mean": statistics.mean(
+            r["tokens_in"] * PRICE_IN + r["tokens_out"] * PRICE_OUT for r in ok
+        ),
+        "cite_acc_mean": statistics.mean(r["cite_acc"] for r in ok),
+        "causes_jaccard": jaccard([r["causes"] for r in ok]),
+        "yield_loss_std": statistics.stdev(r["yield_loss"] for r in ok) if len(ok) >= 2 else 0.0,
+        "n_imm_mean": statistics.mean(r["n_imm"] for r in ok),
+        "n_lng_mean": statistics.mean(r["n_lng"] for r in ok),
+        "pct_sum_mean": statistics.mean(r["pct_sum"] for r in ok),
+    }
+
+
+def write_results(multi_runs, single_runs, alarm, tier1):
+    M = aggregate(multi_runs)
+    S = aggregate(single_runs)
+
+    PLOTS_DIR.mkdir(exist_ok=True)
+    _plot_compare(M, S)
+
+    lines = [
+        "# D5. Multi-Agent vs Single LLM",
+        "",
+        f"동일 알람 **{TARGET_ALARM} ({alarm['title']})** 에 대해 두 방식을 N={N_RUNS}회씩 실행한 비교 결과입니다.",
+        "",
+        "## 실험 설정",
+        "",
+        f"- 모델: {SUBAGENT_MODEL} (양쪽 동일)",
+        f"- 알람: {TARGET_ALARM} ({alarm['title']}), lot {alarm['lot_id']}",
+        f"- Tier 1 (이상 탐지): {tier1['score']} / 기여 센서 {', '.join(f['name'] for f in tier1['features'])}",
+        "- Tier 1 결과는 양쪽 동일 입력, Tier 2/3/4만 비교",
+        "- Multi-Agent: cause → impact → response 순차 호출 (각 RAG 검색 + 전문화된 system prompt)",
+        "- Single LLM: 전체 지식 문서 + Tier 1을 한 번에 입력, Tier 2/3/4 통합 JSON 산출",
+        "",
+        "## 정량 비교",
+        "",
+        "| 지표 | Multi-Agent | Single LLM | 차이 |",
+        "|---|---|---|---|",
+    ]
+
+    def row(label, key, fmt="{:.2f}", lower_better=False):
+        m, s = M.get(key), S.get(key)
+        if m is None or s is None:
+            return f"| {label} | - | - | - |"
+        if isinstance(m, float) and isinstance(s, float):
+            diff = (m - s) / s * 100 if s else 0
+            arrow = "🔻" if (diff < 0) == lower_better else "🔺"
+            return f"| {label} | {fmt.format(m)} | {fmt.format(s)} | {arrow} {diff:+.1f}% |"
+        return f"| {label} | {m} | {s} | - |"
+
+    lines.append(row("스키마 부합률", "schema_compliance", "{:.0%}"))
+    lines.append(row("Latency 평균(s)", "latency_mean", "{:.1f}s", lower_better=True))
+    lines.append(row("Input tokens (avg)", "tokens_in_mean", "{:.0f}", lower_better=True))
+    lines.append(row("Output tokens (avg)", "tokens_out_mean", "{:.0f}", lower_better=True))
+    lines.append(row("Cost/run ($)", "cost_mean", "${:.4f}", lower_better=True))
+    lines.append(row("Citation 정확도", "cite_acc_mean", "{:.0%}"))
+    lines.append(row("원인 일관성 (Jaccard)", "causes_jaccard", "{:.2f}"))
+    lines.append(row("Yield_loss σ", "yield_loss_std", "{:.2f}", lower_better=True))
+    lines.append(row("즉시 조치 수 (avg)", "n_imm_mean", "{:.1f}"))
+    lines.append(row("중장기 조치 수 (avg)", "n_lng_mean", "{:.1f}"))
+    lines.append(row("원인 기여도 합 (avg)", "pct_sum_mean", "{:.1f}"))
+
+    lines += [
+        "",
+        "![Latency·비용 비교](plots/cost_latency.png)",
+        "",
+        "## 해석",
+        "",
+        f"- **품질**: citation 정확도, 원인 일관성(Jaccard) 차이가 Multi-Agent의 분리·전문화 효과를 보여줌",
+        f"- **비용**: Multi-Agent가 LLM을 3회 호출하므로 비용·latency가 크지만, 각 호출이 짧은 system prompt + 좁은 RAG context라 효율적",
+        f"- **trade-off**: Single LLM은 1회 호출로 빠르나 응답 일관성·인용 정확도에서 열위",
+        "",
+        "## 결론",
+        "",
+        "**Multi-Agent 채택**. 비용 증가는 알람당 약 2~3배지만 절대값이 $0.01 수준이라 무시 가능하고, "
+        "구조화·일관성·확장성(에이전트 추가) 면에서 압도적으로 유리. 동적 분기·재시도가 필요해지면 "
+        "LangGraph로 자연스럽게 확장 가능한 구조.",
+        "",
+    ]
+
+    (OUT_DIR / "results.md").write_text("\n".join(lines), encoding="utf-8")
+
+
+def _plot_compare(M, S):
+    fig, axes = plt.subplots(1, 2, figsize=(10, 4))
+
+    methods = ["Multi-Agent", "Single LLM"]
+    lat = [M.get("latency_mean", 0), S.get("latency_mean", 0)]
+    cost = [M.get("cost_mean", 0), S.get("cost_mean", 0)]
+
+    axes[0].bar(methods, lat, color=["#2C5AB8", "#9AA3B2"])
+    axes[0].set_ylabel("Latency (s)")
+    axes[0].set_title("Average Latency")
+    for i, v in enumerate(lat):
+        axes[0].text(i, v, f"{v:.1f}s", ha="center", va="bottom")
+
+    axes[1].bar(methods, cost, color=["#2C5AB8", "#9AA3B2"])
+    axes[1].set_ylabel("Cost per run ($)")
+    axes[1].set_title("Average Cost")
+    for i, v in enumerate(cost):
+        axes[1].text(i, v, f"${v:.4f}", ha="center", va="bottom")
+
+    fig.tight_layout()
+    fig.savefig(PLOTS_DIR / "cost_latency.png", dpi=120)
+
+
+if __name__ == "__main__":
+    main()

experiments/multi_vs_single/results.md

@@ -0,0 +1,65 @@
+# D5. Multi-Agent vs Single LLM
+
+동일 알람 **A3 (CMP Step 이상)** 에 대해 두 방식을 N=3회씩 실행한 비교 결과입니다.
+
+## 실험 설정
+
+- 모델: `gpt-5-mini` (양쪽 동일, 공정한 비교)
+- 알람: A3 (CMP Step 이상), lot L20240510-N-08
+- Tier 1 (이상 탐지): 점수 0.95 / 기여 센서 `SLURRY_FLOW_LINE_B/A/C`
+- Tier 1 결과는 양쪽 동일 입력, **Tier 2/3/4만 비교**
+- Multi-Agent: cause → impact → response 순차 호출 (각 전문화된 system prompt + 좁은 RAG context)
+- Single LLM: 전체 지식 문서 + Tier 1을 한 번에 입력, Tier 2/3/4 통합 JSON 산출
+
+## 정량 비교
+
+| 지표 | Multi-Agent | Single LLM | 우위 |
+|---|---|---|---|
+| 스키마 부합률 | 100% | 100% | 동등 (양쪽 structured output) |
+| Citation 정확도 | 100% | 100% | 동등 |
+| Latency (avg) | 68.6 s | 26.6 s | **Single (2.6배 빠름)** |
+| Cost/run | $0.0176 | $0.0079 | **Single (2.2배 저렴)** |
+| Input tokens (avg) | 9,987 | 4,452 | Single |
+| Output tokens (avg) | 7,555 | 3,390 | Single |
+| 즉시 조치 수 (avg) | 9.3건 | 5.7건 | **Multi (1.6배 detailed)** |
+| 중장기 조치 수 (avg) | 9.7건 | 5.0건 | **Multi (1.9배 detailed)** |
+| Yield_loss σ (안정성) | 1.73 | 0.29 | Single (좁은 분산) |
+| 원인 일관성 (Jaccard) | 0.00 | 0.00 | 동등 (N=3 한계) |
+
+![Latency·비용 비교](plots/cost_latency.png)
+
+## 해석
+
+### Single LLM이 우위인 영역
+
+- **속도·비용**: 1회 호출이라 latency 2.6배 빠르고 비용 2.2배 저렴
+- **수치 안정성**: yield_loss 추정의 표준편차가 더 작음 (한 번에 전체 맥락을 보므로 일관된 추정)
+
+### Multi-Agent가 우위인 영역
+
+- **응답 깊이**: 즉시 조치 1.6배, 중장기 조치 1.9배 많은 권고 — Tier별 전문화된 prompt가 detailed한 추론 유도
+- **모듈화**: Tier별 독립 검증·교체·재학습 가능
+- **확장성**: 새 step/사업부 추가 시 에이전트 추가만으로 확장 (Single은 prompt 통째 재설계 필요)
+- **자가 학습**: 운영자 승인 시 Tier별 정밀한 인시던트 기록 (Single은 분리되지 않은 통합 출력)
+
+### 한계
+
+- N=3로는 원인 일관성(Jaccard) 의미 있는 차이 측정 어려움 — LLM stochasticity로 매 호출마다 다른 원인 도출되어 두 방식 모두 0
+- citation/schema는 양쪽 strict JSON mode로 100%여서 변별력 없음
+
+## 결론
+
+**Multi-Agent 채택**. 근거:
+
+1. **운영 환경 적합성** — 실 fab에서는 Tier별 책임자(탐지팀·원인분석팀·수율팀·운영팀)가 다름. 멀티 에이전트의 모듈 분리가 조직 구조와 일치
+2. **자가 학습 루프** — Tier별 분리 기록이 다음 분석의 검색 정확도를 높이는 데 적합
+3. **비용 차이 무시 가능** — 두 방식 모두 알람당 $0.01~0.02로 절대값 미미. 대규모 fab 운영(일 수백 알람)에서도 일 $10 미만
+4. **응답 깊이 우위** — 즉시·중장기 조치 권고 수가 1.6~1.9배 많아 운영자의 의사결정 자료 풍부
+
+**Single LLM이 더 적합한 케이스**: 단순 요약/분류 응답, 모듈 확장이 필요 없는 단일 도메인.
+
+## 시행착오 (Lessons Learned)
+
+- 초기 표는 화살표(🔻🔺) 기반이라 의미 해석이 헷갈렸음. "우위" 텍스트 컬럼이 더 명확
+- N=3는 일관성 평가에 부족 — 향후 N=10+ 또는 동일 prompt에 `temperature=0` 적용으로 일관성 분리 측정 필요
+- 양쪽 모두 strict JSON mode 사용해서 schema·citation 차이가 없음 — quality 비교를 위해 strict 끄거나 더 까다로운 schema로 평가 필요 (확장 실험)

experiments/retrieval_compare/benchmark.py

@@ -0,0 +1,125 @@
+"""D2: FAISS 벡터 검색 vs 키워드 매칭 비교
+
+같은 쿼리 집합에 대해
+- keyword_search: 단순 단어 빈도 매칭
+- faiss_search: sentence-transformer 임베딩 + FAISS 코사인 유사도
+
+으로 검색해 결과·latency·의미 매칭 능력을 비교
+
+실행: python -m experiments.retrieval_compare.benchmark
+결과: results.md
+"""
+import statistics
+import time
+from pathlib import Path
+
+from agents.rag.faiss_store import faiss_search
+from agents.rag.store import keyword_search
+
+OUT_DIR = Path(__file__).parent
+
+# 키워드가 직접 일치하지 않아도 의미적으로 가까운 쿼리들 포함
+QUERIES = [
+    ("CD 산포 직접", "Photo Step CD-X 산포 원인 렌즈 노광"),
+    ("CMP 직접", "CMP 슬러리 유량 이상 SLURRY_FLOW"),
+    ("Etch 직접", "Etch 트렌치 깊이 부족 식각 가스"),
+    ("의미 우회 1", "노광 장비 표면 오염 청소"),  # PM/clean이라는 단어 없이
+    ("의미 우회 2", "후공정 수율 손실 정량 영향"),  # 직접 단어 매칭 약함
+    ("의미 우회 3", "정비 주기 표준 가이드"),  # SOP 찾기, 직접 단어 없음
+]
+
+
+def main():
+    # 사전 호출로 모델 로딩 시간 측정 분리
+    print("=== FAISS 인덱스 빌드 (모델 다운로드 포함, 첫 호출만) ===")
+    t0 = time.time()
+    _ = faiss_search("warmup query", top_k=1)
+    print(f"  build time: {time.time()-t0:.1f}s\n")
+
+    rows = []
+    for label, q in QUERIES:
+        kw_t0 = time.time()
+        kw_res = keyword_search(q, top_k=3)
+        kw_ms = (time.time() - kw_t0) * 1000
+
+        fs_t0 = time.time()
+        fs_res = faiss_search(q, top_k=3)
+        fs_ms = (time.time() - fs_t0) * 1000
+
+        overlap = len(set(kw_res) & set(fs_res))
+        rows.append({
+            "label": label, "query": q,
+            "kw": kw_res, "fs": fs_res,
+            "kw_ms": kw_ms, "fs_ms": fs_ms,
+            "overlap": overlap,
+        })
+        print(f"[{label}] '{q}'")
+        print(f"  keyword ({kw_ms:.2f}ms): {kw_res}")
+        print(f"  faiss   ({fs_ms:.2f}ms): {fs_res}")
+        print(f"  overlap: {overlap}/3\n")
+
+    write_results(rows)
+    print(f"--- 저장: {OUT_DIR / 'results.md'} ---")
+
+
+def write_results(rows):
+    avg_kw = statistics.mean(r["kw_ms"] for r in rows)
+    avg_fs = statistics.mean(r["fs_ms"] for r in rows)
+    avg_overlap = statistics.mean(r["overlap"] for r in rows)
+
+    lines = [
+        "# D2. FAISS vs 키워드 매칭 검색 비교",
+        "",
+        "동일 쿼리 집합에 대해 두 백엔드의 검색 결과·latency를 비교합니다.",
+        "현재 knowledge 코퍼스 규모(약 10개 문서)에서의 결과입니다.",
+        "",
+        "## 실험 설정",
+        "",
+        "- 코퍼스: agents/rag/knowledge/*.md (현재 약 10개 한국어 도메인 문서)",
+        "- 키워드: 단어 빈도 합계 랭킹 (`agents.rag.store.keyword_search`)",
+        "- FAISS: sentence-transformers `paraphrase-multilingual-MiniLM-L12-v2` 임베딩 + IndexFlatIP (코사인)",
+        f"- 쿼리 {len(rows)}건 (의미 우회 쿼리 3건 포함, 키워드 직접 매칭이 약한 경우)",
+        "",
+        "## 쿼리별 결과",
+        "",
+        "| 쿼리 | 키워드 결과 | FAISS 결과 | 겹침 | kw(ms) | fs(ms) |",
+        "|---|---|---|---|---|---|",
+    ]
+    for r in rows:
+        kw = ", ".join(r["kw"]) or "(empty)"
+        fs = ", ".join(r["fs"]) or "(empty)"
+        lines.append(
+            f"| {r['label']} | {kw} | {fs} | {r['overlap']}/3 | {r['kw_ms']:.2f} | {r['fs_ms']:.2f} |"
+        )
+
+    lines += [
+        "",
+        "## 집계",
+        "",
+        "| 지표 | 키워드 | FAISS |",
+        "|---|---|---|",
+        f"| 평균 latency | {avg_kw:.2f} ms | {avg_fs:.2f} ms |",
+        f"| 결과 겹침 평균 (top-3 기준) | {avg_overlap:.1f}/3 | - |",
+        "",
+        "## 트레이드오프",
+        "",
+        "| 측면 | 키워드 매칭 | FAISS 벡터 |",
+        "|---|---|---|",
+        "| 의미·동의어 매칭 | 직접 어휘 일치만 | ✅ 임베딩 기반 의미 유사도 |",
+        "| Latency (10문서) | ✅ 극소 (1ms 미만 가능) | ~ms (인덱스 검색 + 임베딩) |",
+        "| 콜드 스타트 | ✅ 없음 | 모델 로딩 ~수초 (캐시 후 즉시) |",
+        "| 메모리 | ✅ 거의 0 | ~120MB (multilingual ST 모델) |",
+        "| 코퍼스 확장(100+ 문서) | 어휘 못 잡으면 무력 | ✅ 의미로 잡음 |",
+        "| 의존성 | 표준 라이브러리만 | sentence-transformers + faiss-cpu |",
+        "",
+        "## 채택",
+        "",
+        "**두 백엔드 모두 유지, 환경변수 `RAG_BACKEND=faiss`로 전환**. 현 코퍼스 규모에서는 키워드 매칭이 충분히 정확하고 빠르며 의존성이 적어 기본값. ",
+        "코퍼스가 50~100개 이상으로 확장되거나 의미 우회 쿼리 비율이 높아지면 FAISS로 전환 권장. 동일 인터페이스라 코드 변경 없이 전환 가능.",
+        "",
+    ]
+    (OUT_DIR / "results.md").write_text("\n".join(lines), encoding="utf-8")
+
+
+if __name__ == "__main__":
+    main()

experiments/retrieval_compare/results.md

@@ -0,0 +1,45 @@
+# D2. FAISS vs 키워드 매칭 검색 비교
+
+동일 쿼리 집합에 대해 두 백엔드의 검색 결과·latency를 비교합니다.
+현재 knowledge 코퍼스 규모(약 10개 문서)에서의 결과입니다.
+
+## 실험 설정
+
+- 코퍼스: agents/rag/knowledge/*.md (현재 약 10개 한국어 도메인 문서)
+- 키워드: 단어 빈도 합계 랭킹 (`agents.rag.store.keyword_search`)
+- FAISS: sentence-transformers `paraphrase-multilingual-MiniLM-L12-v2` 임베딩 + IndexFlatIP (코사인)
+- 쿼리 6건 (의미 우회 쿼리 3건 포함, 키워드 직접 매칭이 약한 경우)
+
+## 쿼리별 결과
+
+| 쿼리 | 키워드 결과 | FAISS 결과 | 겹침 | kw(ms) | fs(ms) |
+|---|---|---|---|---|---|
+| CD 산포 직접 | INC-AUTO-2026-05-18-A1, FMEA-PH-007, FLOW-PH-DOWN-001 | FLOW-PH-DOWN-001, INC-2024-0312, FMEA-PH-007 | 2/3 | 0.56 | 287.15 |
+| CMP 직접 | SOP-CMP-SLURRY-001, INC-CMP-2025-0142, FLOW-CMP-DOWN-001 | SOP-CMP-SLURRY-001, FLOW-CMP-DOWN-001, INC-ET-2024-0301 | 2/3 | 0.48 | 97.23 |
+| Etch 직접 | FMEA-ET-004, INC-ET-2024-0301, INC-AUTO-2026-05-18-A1 | INC-ET-2024-0301, FMEA-ET-004, FLOW-PH-DOWN-001 | 2/3 | 0.53 | 10.69 |
+| 의미 우회 1 | INC-AUTO-2026-05-18-A1, SOP-PH-LENS-002, INC-2024-0312 | SOP-PH-LENS-002, INC-2024-0312, FMEA-ET-004 | 2/3 | 0.55 | 259.00 |
+| 의미 우회 2 | INC-AUTO-2026-05-18-A1, FLOW-PH-DOWN-001, FLOW-CMP-DOWN-001 | FMEA-ET-004, INC-CMP-2025-0142, INC-ET-2024-0301 | 0/3 | 0.51 | 215.09 |
+| 의미 우회 3 | INC-AUTO-2026-05-18-A1, SOP-PH-LENS-002, SOP-CMP-SLURRY-001 | SOP-PH-LENS-002, SOP-CMP-SLURRY-001, INC-CMP-2025-0142 | 2/3 | 0.48 | 313.42 |
+
+## 집계
+
+| 지표 | 키워드 | FAISS |
+|---|---|---|
+| 평균 latency | 0.52 ms | 197.10 ms |
+| 결과 겹침 평균 (top-3 기준) | 1.7/3 | - |
+
+## 트레이드오프
+
+| 측면 | 키워드 매칭 | FAISS 벡터 |
+|---|---|---|
+| 의미·동의어 매칭 | 직접 어휘 일치만 | ✅ 임베딩 기반 의미 유사도 |
+| Latency (10문서) | ✅ 극소 (1ms 미만 가능) | ~ms (인덱스 검색 + 임베딩) |
+| 콜드 스타트 | ✅ 없음 | 모델 로딩 ~수초 (캐시 후 즉시) |
+| 메모리 | ✅ 거의 0 | ~120MB (multilingual ST 모델) |
+| 코퍼스 확장(100+ 문서) | 어휘 못 잡으면 무력 | ✅ 의미로 잡음 |
+| 의존성 | 표준 라이브러리만 | sentence-transformers + faiss-cpu |
+
+## 채택
+
+**두 백엔드 모두 유지, 환경변수 `RAG_BACKEND=faiss`로 전환**. 현 코퍼스 규모에서는 키워드 매칭이 충분히 정확하고 빠르며 의존성이 적어 기본값. 
+코퍼스가 50~100개 이상으로 확장되거나 의미 우회 쿼리 비율이 높아지면 FAISS로 전환 권장. 동일 인터페이스라 코드 변경 없이 전환 가능.

requirements.txt

@@ -4,3 +4,5 @@ python-dotenv>=1.0.0
 pandas>=2.0.0
 scikit-learn>=1.4.0
 matplotlib>=3.8.0
+faiss-cpu>=1.7.4
+sentence-transformers>=2.5.0