jeevan2717 commited on
Commit
bc06d19
Β·
1 Parent(s): 091e79a

Add live Slack bot demo + YouTube video + BLOG section on production deployment

Browse files
BLOG.md CHANGED
@@ -174,6 +174,38 @@ That's what I want this environment to enable. Not just an academic benchmark. A
174
 
175
  ---
176
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
177
  ## What I'd do differently with more compute
178
 
179
  The biggest thing limiting V2's multi-agent training was the 0.5B model size. With access to a single A100 or L4 GPU, the next steps are obvious:
@@ -184,7 +216,7 @@ The biggest thing limiting V2's multi-agent training was the 0.5B model size. Wi
184
 
185
  **Multi-incident chains.** What if the same root cause causes three related incidents over a week? That's a real scenario in production, and the environment doesn't model it yet. Linked episodes would test long-context reasoning across days.
186
 
187
- **Live deployment.** Take the trained agent, wrap it in a Slack bot, hook it to a real PagerDuty webhook at a small company, and watch what happens. The metric becomes "what percentage of agent drafts make it through human review unchanged?" That's the real benchmark.
188
 
189
  ---
190
 
@@ -200,7 +232,7 @@ The deterministic grader means any lab can benchmark GPT-4, Claude, Gemini, and
200
 
201
  ## Closing thoughts
202
 
203
- I built this as a solo entry. Three weeks ago I'd never trained an LLM. Now I have a deployed Hugging Face Space, two TRL fine-tuning runs documented end-to-end, multi-agent extensions with measurable gains, and a real production integration pattern.
204
 
205
  The tooling has gotten so much better in the last year. OpenEnv made the environment scaffolding clean. TRL made the training accessible. Hugging Face Spaces made deployment trivial. Groq made fast inference free for hackathon use. None of this would have been buildable by one person on weekends a year ago.
206
 
 
174
 
175
  ---
176
 
177
+ ## Going live: the Slack bot
178
+
179
+ The production bridge is one thing. *Actually deploying it* is another.
180
+
181
+ So the night before submission, I built a Slack bot.
182
+
183
+ It's a slash command. A user types `/postmortem <incident-json-url>` in a Slack channel. The bot fetches the incident JSON, runs it through my PagerDuty importer, calls the agent, and posts a structured post-mortem back to the channel. End-to-end in about twenty seconds.
184
+
185
+ [![Watch the demo](https://img.youtube.com/vi/VIDEO_ID/maxresdefault.jpg)](https://youtu.be/VIDEO_ID)
186
+
187
+ The architecture is small but real:
188
+
189
+ ```
190
+ Slack slash command
191
+ ↓
192
+ ngrok webhook β†’ FastAPI (slackbot/app.py)
193
+ ↓
194
+ PagerDuty importer (tools/pagerduty_importer.py)
195
+ ↓
196
+ LLM agent (Llama 3.1 8B via Groq)
197
+ ↓
198
+ Structured post-mortem posted back to Slack
199
+ ```
200
+
201
+ A few things had to be right for this to work. Slack requires a 200 response within three seconds β€” the LLM call takes fifteen to twenty β€” so the bot acknowledges immediately and processes the incident in a background task. Slack signs every request with HMAC-SHA256, so the bot verifies signatures against the signing secret before doing anything. None of this is exotic, but the first time you build it under deadline pressure, it's all new.
202
+
203
+ What I like most about this isn't the bot itself. It's that the entire system β€” environment, importer, agent, UI β€” is now the same code path you'd ship at a company. The same `pagerduty_importer.py` that judges run from the command line is the same module the bot imports. The same agent that scores 0.857 average on benchmark scenarios is the same one writing the post-mortem in Slack.
204
+
205
+ This isn't a demo built specifically for the hackathon. It's the real thing, demonstrated in a hackathon-sized way.
206
+
207
+ ---
208
+
209
  ## What I'd do differently with more compute
210
 
211
  The biggest thing limiting V2's multi-agent training was the 0.5B model size. With access to a single A100 or L4 GPU, the next steps are obvious:
 
216
 
217
  **Multi-incident chains.** What if the same root cause causes three related incidents over a week? That's a real scenario in production, and the environment doesn't model it yet. Linked episodes would test long-context reasoning across days.
218
 
219
+ **Real-company deployment.** I built the Slack bot demo for the hackathon β€” next is hooking it to a real PagerDuty webhook at an actual company and watching what happens. The metric becomes "what percentage of agent drafts make it through human review unchanged?" That's the real benchmark, and the only one that ultimately matters.
220
 
221
  ---
222
 
 
232
 
233
  ## Closing thoughts
234
 
235
+ I built this as a solo entry. Three weeks ago I'd never trained an LLM. Now I have a deployed Hugging Face Space, two TRL fine-tuning runs documented end-to-end, multi-agent extensions with measurable gains, three production importers (PagerDuty, Datadog, Splunk), and a working Slack bot that demonstrates the whole pipeline end-to-end on real incident data.
236
 
237
  The tooling has gotten so much better in the last year. OpenEnv made the environment scaffolding clean. TRL made the training accessible. Hugging Face Spaces made deployment trivial. Groq made fast inference free for hackathon use. None of this would have been buildable by one person on weekends a year ago.
238
 
README.md CHANGED
@@ -37,6 +37,16 @@ An [OpenEnv](https://github.com/meta-pytorch/OpenEnv) environment where AI agent
37
  | πŸ”¬ Inference Scripts | [inference.py](https://github.com/Jeevan2798/incident-postmortem-writer/blob/main/inference.py) Β· [inference_multiagent.py](https://github.com/Jeevan2798/incident-postmortem-writer/blob/main/inference_multiagent.py) | Single-agent and multi-agent inference runners |
38
 
39
 
 
 
 
 
 
 
 
 
 
 
40
  ## ⚑ Quick Reproduction (15 minutes)
41
 
42
  Want to verify the numbers yourself? Three commands:
 
37
  | πŸ”¬ Inference Scripts | [inference.py](https://github.com/Jeevan2798/incident-postmortem-writer/blob/main/inference.py) Β· [inference_multiagent.py](https://github.com/Jeevan2798/incident-postmortem-writer/blob/main/inference_multiagent.py) | Single-agent and multi-agent inference runners |
38
 
39
 
40
+ ## πŸ€– Live Slack Bot Demo
41
+
42
+ [![Watch the demo](https://img.youtube.com/vi/VIDEO_ID/maxresdefault.jpg)](https://youtu.be/VIDEO_ID)
43
+
44
+ **The agent runs as a Slack slash command** β€” paste a PagerDuty incident JSON URL into Slack, get a structured post-mortem in ~20 seconds. See the [slackbot/](https://github.com/Jeevan2798/incident-postmortem-writer/tree/main/slackbot) folder for the FastAPI implementation.
45
+
46
+ This demonstrates the full production loop: `incident system β†’ JSON β†’ agent β†’ post-mortem in Slack thread β†’ human review`.
47
+
48
+ ---
49
+
50
  ## ⚑ Quick Reproduction (15 minutes)
51
 
52
  Want to verify the numbers yourself? Three commands:
slackbot/README.md ADDED
@@ -0,0 +1,89 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Slack Bot β€” Production Deployment Pattern
2
+
3
+ A working Slack slash command that demonstrates the production deployment pattern from this OpenEnv submission.
4
+
5
+ ## Demo
6
+
7
+ [![Watch the demo](https://img.youtube.com/vi/VIDEO_ID/maxresdefault.jpg)](https://youtu.be/VIDEO_ID)
8
+
9
+ **60-second video** showing the bot generating a structured post-mortem from a real PagerDuty incident JSON URL.
10
+
11
+ ## How it works
12
+
13
+ ```
14
+ Slack: /postmortem <incident-json-url>
15
+ ↓
16
+ ngrok webhook β†’ FastAPI (slackbot/app.py)
17
+ ↓
18
+ PagerDuty importer (tools/pagerduty_importer.py)
19
+ ↓
20
+ LLM agent (Llama 3.1 8B via Groq)
21
+ ↓
22
+ Structured post-mortem posted back to Slack channel
23
+ ```
24
+
25
+ ## Why this matters
26
+
27
+ The hackathon environment trains agents on synthetic scenarios, but the real value is in production deployment. This bot proves the integration end-to-end:
28
+
29
+ - **Real Slack workspace** β€” not a mock
30
+ - **Real production JSON format** β€” PagerDuty Incident API v2 schema
31
+ - **Real LLM agent** β€” same model used throughout the submission
32
+ - **Real post-mortem output** β€” all 5 sections, specific service/version/mechanism/impact/action items
33
+
34
+ The same pattern extends to Datadog and Splunk via the existing importers in `tools/`.
35
+
36
+ ## Setup
37
+
38
+ ### 1. Slack app
39
+
40
+ 1. Create app at https://api.slack.com/apps
41
+ 2. Add slash command `/postmortem` with Request URL pointing at `<your-host>/slack/postmortem`
42
+ 3. Install to a workspace
43
+ 4. Note the Signing Secret
44
+
45
+ ### 2. Environment
46
+
47
+ ```bash
48
+ pip install -r slackbot/requirements.txt
49
+
50
+ export SLACK_SIGNING_SECRET=<from-slack-app>
51
+ export API_BASE_URL=https://api.groq.com/openai/v1
52
+ export MODEL_NAME=llama-3.1-8b-instant
53
+ export HF_TOKEN=<your-groq-key>
54
+ ```
55
+
56
+ ### 3. Run locally
57
+
58
+ ```bash
59
+ # Terminal 1: bot
60
+ uvicorn slackbot.app:app --port 8000 --reload
61
+
62
+ # Terminal 2: tunnel
63
+ ngrok http 8000
64
+
65
+ # Update Slack slash command Request URL with the ngrok URL + /slack/postmortem
66
+ ```
67
+
68
+ ### 4. Test in Slack
69
+
70
+ ```
71
+ /postmortem https://raw.githubusercontent.com/Jeevan2798/incident-postmortem-writer/main/samples/pagerduty/incident_payments_outage.json
72
+ ```
73
+
74
+ ## Architecture notes
75
+
76
+ - **3-second rule:** Slack requires a 200 response within 3 seconds. The bot acknowledges immediately and processes the incident in a background task (`BackgroundTasks` in FastAPI).
77
+ - **Signature verification:** Validates each request with HMAC-SHA256 against the Slack signing secret to prevent spoofing.
78
+ - **Async processing:** Long-running LLM calls (~15-20s) happen in the background; the response is posted via Slack's `response_url`.
79
+
80
+ ## Production deployment
81
+
82
+ For real deployment beyond ngrok:
83
+
84
+ - **Hosting:** Render.com / Railway / Fly.io (any platform supporting FastAPI + persistent HTTPS)
85
+ - **Webhook source:** Replace slash command with PagerDuty webhook β†’ automatic post-mortem on every incident resolution
86
+ - **Storage:** Persist generated post-mortems to a database for the human-review feedback loop
87
+ - **Monitoring:** Track which post-mortems get accepted unchanged vs heavily edited β€” that's the real quality signal
88
+
89
+ This is the loop the BLOG describes: *PagerDuty webhook fires β†’ importer converts β†’ agent drafts β†’ human reviews β†’ validated post-mortems become the next training cycle.*
slackbot/__pycache__/app.cpython-310.pyc ADDED
Binary file (6.05 kB). View file
 
slackbot/app.py ADDED
@@ -0,0 +1,200 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """
2
+ Slack Bot Backend β€” Incident Post-Mortem Writer
3
+ ================================================
4
+ Bare minimum Slack slash command bot.
5
+
6
+ USAGE FLOW:
7
+ User in Slack: /postmortem <incident-json-url>
8
+ Bot fetches JSON, runs PagerDuty importer + agent, replies with post-mortem.
9
+
10
+ REQUIREMENTS:
11
+ pip install fastapi uvicorn requests openai python-multipart
12
+
13
+ ENV VARS NEEDED:
14
+ SLACK_SIGNING_SECRET β€” from Slack app Basic Information
15
+ API_BASE_URL β€” Groq base URL
16
+ MODEL_NAME β€” llama-3.1-8b-instant
17
+ HF_TOKEN β€” Groq API key
18
+
19
+ RUN LOCALLY:
20
+ uvicorn slackbot.app:app --port 8000 --reload
21
+
22
+ EXPOSE PUBLICLY (in another terminal):
23
+ ngrok http 8000
24
+
25
+ Then paste the ngrok URL into your Slack app's Slash Command Request URL:
26
+ https://abc123.ngrok.io/slack/postmortem
27
+ """
28
+
29
+ from __future__ import annotations
30
+
31
+ import hashlib
32
+ import hmac
33
+ import json
34
+ import os
35
+ import sys
36
+ import time
37
+ from pathlib import Path
38
+
39
+ import requests
40
+ from fastapi import FastAPI, Request, BackgroundTasks
41
+ from fastapi.responses import JSONResponse, PlainTextResponse
42
+
43
+ # ─── Reuse existing project code ───
44
+ PROJECT_ROOT = Path(__file__).parent.parent
45
+ sys.path.insert(0, str(PROJECT_ROOT))
46
+
47
+ from tools.pagerduty_importer import import_pagerduty_incident # noqa: E402
48
+
49
+ app = FastAPI()
50
+
51
+ # ─── Config from environment ───
52
+ SLACK_SIGNING_SECRET = os.environ.get("SLACK_SIGNING_SECRET", "")
53
+ API_BASE_URL = os.environ.get("API_BASE_URL", "https://api.groq.com/openai/v1")
54
+ MODEL_NAME = os.environ.get("MODEL_NAME", "llama-3.1-8b-instant")
55
+ HF_TOKEN = os.environ.get("HF_TOKEN", "")
56
+
57
+
58
+ # ─── Slack signature verification ───
59
+ def verify_slack_signature(request_body: bytes, timestamp: str, signature: str) -> bool:
60
+ """Verify request actually came from Slack."""
61
+ if not SLACK_SIGNING_SECRET:
62
+ print("[WARN] SLACK_SIGNING_SECRET not set β€” skipping verification")
63
+ return True
64
+
65
+ # Reject if timestamp is older than 5 minutes (replay attack protection)
66
+ if abs(time.time() - int(timestamp)) > 300:
67
+ return False
68
+
69
+ sig_basestring = f"v0:{timestamp}:{request_body.decode('utf-8')}"
70
+ expected_sig = "v0=" + hmac.new(
71
+ SLACK_SIGNING_SECRET.encode("utf-8"),
72
+ sig_basestring.encode("utf-8"),
73
+ hashlib.sha256,
74
+ ).hexdigest()
75
+
76
+ return hmac.compare_digest(expected_sig, signature)
77
+
78
+
79
+ # ─── Agent: generate post-mortem from incident JSON ───
80
+ def generate_postmortem(scenario: dict) -> str:
81
+ """Run LLM agent against the imported scenario, return formatted post-mortem."""
82
+ from openai import OpenAI
83
+
84
+ client = OpenAI(base_url=API_BASE_URL, api_key=HF_TOKEN)
85
+
86
+ # Build context for LLM
87
+ alerts = scenario.get("initial_alerts", [])
88
+ slack_thread = scenario.get("slack_thread", [])
89
+
90
+ context = f"INCIDENT: {scenario.get('incident_title', 'Unknown')}\n\n"
91
+ context += "ALERTS:\n"
92
+ for a in alerts:
93
+ context += f" [{a.get('timestamp')}] {a.get('service')} ({a.get('severity')}): {a.get('message')}\n"
94
+ context += "\nSLACK THREAD:\n"
95
+ for s in slack_thread:
96
+ context += f" [{s.get('timestamp')}] {s.get('author')}: {s.get('text')}\n"
97
+
98
+ prompt = f"""You are an SRE writing a post-mortem. Based on this incident data, write a structured post-mortem with these 5 sections: SUMMARY, TIMELINE, ROOT_CAUSE, IMPACT, ACTION_ITEMS.
99
+
100
+ Use Slack-friendly markdown formatting (bold with *asterisks*, not **double asterisks*).
101
+ Be specific β€” name services, times, and mechanisms. Aim for ~250 words total.
102
+
103
+ {context}
104
+
105
+ Write the post-mortem now:"""
106
+
107
+ response = client.chat.completions.create(
108
+ model=MODEL_NAME,
109
+ messages=[{"role": "user", "content": prompt}],
110
+ temperature=0.3,
111
+ max_tokens=600,
112
+ )
113
+ return response.choices[0].message.content
114
+
115
+
116
+ def post_to_slack(response_url: str, text: str, thread_ts: str | None = None):
117
+ """Post a message back to Slack via the response_url from slash command."""
118
+ payload = {
119
+ "response_type": "in_channel",
120
+ "text": text,
121
+ }
122
+ requests.post(response_url, json=payload, timeout=10)
123
+
124
+
125
+ def process_incident_async(response_url: str, incident_url: str):
126
+ """Background task: fetch JSON, run agent, post back to Slack."""
127
+ try:
128
+ # Fetch the JSON
129
+ resp = requests.get(incident_url, timeout=10)
130
+ resp.raise_for_status()
131
+ pagerduty_payload = resp.json()
132
+
133
+ # Import β†’ scenario
134
+ scenario = import_pagerduty_incident(pagerduty_payload)
135
+
136
+ # Run agent
137
+ postmortem = generate_postmortem(scenario)
138
+
139
+ # Format Slack response
140
+ message = (
141
+ f":memo: *Post-Mortem for {scenario.get('incident_id')}*\n"
142
+ f"_{scenario.get('incident_title')}_\n\n"
143
+ f"{postmortem}\n\n"
144
+ f"_Generated by incident-postmortem-writer agent (Llama 3.1 8B)_"
145
+ )
146
+
147
+ post_to_slack(response_url, message)
148
+
149
+ except Exception as e:
150
+ post_to_slack(
151
+ response_url,
152
+ f":warning: Failed to generate post-mortem: `{type(e).__name__}: {str(e)[:200]}`"
153
+ )
154
+
155
+
156
+ # ─── Routes ───
157
+
158
+ @app.get("/health")
159
+ async def health():
160
+ return {"status": "ok"}
161
+
162
+
163
+ @app.post("/slack/postmortem")
164
+ async def slack_postmortem(request: Request, background_tasks: BackgroundTasks):
165
+ """Slack slash command handler. MUST respond within 3 seconds."""
166
+ body = await request.body()
167
+
168
+ # Verify it's actually from Slack
169
+ timestamp = request.headers.get("X-Slack-Request-Timestamp", "")
170
+ signature = request.headers.get("X-Slack-Signature", "")
171
+ if not verify_slack_signature(body, timestamp, signature):
172
+ return JSONResponse({"error": "invalid signature"}, status_code=401)
173
+
174
+ # Parse form data (Slack sends URL-encoded form, not JSON)
175
+ form = await request.form()
176
+ text = form.get("text", "").strip()
177
+ response_url = form.get("response_url", "")
178
+
179
+ # Validate input
180
+ if not text.startswith("http"):
181
+ return JSONResponse({
182
+ "response_type": "ephemeral",
183
+ "text": (
184
+ ":x: *Usage:* `/postmortem <url-to-incident-json>`\n"
185
+ "Example: `/postmortem https://raw.githubusercontent.com/Jeevan2798/incident-postmortem-writer/main/samples/pagerduty/incident_payments_outage.json`"
186
+ ),
187
+ })
188
+
189
+ # Acknowledge immediately, then process in background
190
+ background_tasks.add_task(process_incident_async, response_url, text)
191
+
192
+ return JSONResponse({
193
+ "response_type": "in_channel",
194
+ "text": f":mag: Generating post-mortem from `{text[:80]}`... (takes ~15-30s)",
195
+ })
196
+
197
+
198
+ if __name__ == "__main__":
199
+ import uvicorn
200
+ uvicorn.run(app, host="0.0.0.0", port=8000)
slackbot/requirements.txt ADDED
@@ -0,0 +1,5 @@
 
 
 
 
 
 
1
+ fastapi>=0.115.0
2
+ uvicorn>=0.30.0
3
+ python-multipart>=0.0.9
4
+ requests>=2.31.0
5
+ openai>=1.50.0