File size: 7,545 Bytes
7009aba
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
# πŸ› οΈ Development Guide

## Quick Start

### πŸš€ **Initial Setup**
```bash
# Clone and install
git clone <repo-url>
cd sap-docs-mcp
npm install

# Run enhanced setup (submodules + build)
npm run setup

# Start development server
npm run start:http
```

### πŸ§ͺ **Run Tests**
```bash
npm run test:smoke    # Quick validation
npm run test:fast     # Skip build, test only
npm run test          # Full build + test
```

## Common Commands

### πŸ“¦ **Build Commands**
```bash
npm run build:tsc       # Compile TypeScript
npm run build:index     # Build documentation index
npm run build:fts       # Build FTS5 search database  
npm run build           # Complete build pipeline (tsc + index + fts)
```

### πŸ–₯️ **Server Commands**
```bash
npm start                    # MCP stdio server (for Claude)
npm run start:http           # HTTP development server (port 3001)
npm run start:streamable     # Streamable HTTP server (port 3122)
```

### πŸ§ͺ **Test Commands**  
```bash
npm run test:smoke      # Quick smoke tests
npm run test:fast       # Test without rebuild
npm run test            # Full test suite
npm run test:community  # SAP Community search tests
npm run inspect         # MCP protocol inspector
```

## Environment Variables

### πŸ”§ **Core Configuration**
```bash
RETURN_K=25                    # Number of search results (default: 25)
LOG_LEVEL=INFO                 # Logging level (ERROR, WARN, INFO, DEBUG)
LOG_FORMAT=json                # Log format (json or text)
NODE_ENV=production            # Environment mode
```

### πŸ—„οΈ **Database & Paths**
```bash
DB_PATH=dist/data/docs.sqlite  # FTS5 database path
METADATA_PATH=src/metadata.json # Metadata configuration path
```

### 🌐 **Server Configuration**
```bash
PORT=3001                      # HTTP server port
MCP_PORT=3122                  # Streamable HTTP MCP port
```

## Development Servers

### πŸ“‘ **1. Stdio MCP Server** (Main)
```bash
npm run start:stdio
# For Claude/LLM integration via stdio transport
```

### 🌐 **2. HTTP Development Server**
```bash
npm run start:http
# Access: http://localhost:3001
# Endpoints: /status, /healthz, /readyz, /mcp
```

### πŸ”„ **3. Streamable HTTP Server**
```bash
npm run start:streamable  
# Access: http://localhost:3122
# Endpoints: /mcp, /health
```

## Where to Change Things

### πŸ” **Search Behavior**
- **Query Processing**: `src/lib/searchDb.ts` β†’ `toMatchQuery()`
- **Search Logic**: `src/lib/search.ts` β†’ `search()`
- **Result Formatting**: `src/lib/localDocs.ts` β†’ `searchLibraries()`

### βš™οΈ **Configuration**
- **Source Settings**: `src/metadata.json` β†’ Add/modify sources
- **Core Config**: `src/lib/config.ts` β†’ System settings
- **Metadata APIs**: `src/lib/metadata.ts` β†’ Configuration access

### πŸ› οΈ **MCP Tools**
- **Tool Definitions**: `src/server.ts` β†’ `ListToolsRequestSchema`
- **Tool Handlers**: `src/server.ts` β†’ `CallToolRequestSchema`
- **HTTP Endpoints**: `src/http-server.ts` β†’ `/mcp` handler

### πŸ—οΈ **Build Process**
- **Index Building**: `scripts/build-index.ts`
- **FTS Database**: `scripts/build-fts.ts`
- **Source Processing**: Modify build scripts for new source types

### πŸ§ͺ **Tests**
- **Test Cases**: `test/tools/search/` β†’ Add new test files
- **Test Runner**: `test/tools/run-tests.js` β†’ Modify test execution
- **Output Parsing**: `test/_utils/parseResults.js` β†’ Update format expectations

### πŸš€ **Deployment**
- **PM2 Config**: `ecosystem.config.cjs` β†’ Process configuration
- **GitHub Actions**: `.github/workflows/deploy-mcp-sap-docs.yml`
- **Setup Script**: `setup.sh` β†’ Deployment automation

## Adding New Documentation Sources

### 1. **Update Metadata** (`src/metadata.json`)
```json
{
  "id": "new-source",
  "type": "documentation",
  "libraryId": "/new-source",
  "sourcePath": "new-source/docs",
  "baseUrl": "https://example.com/docs",
  "pathPattern": "/{file}",
  "anchorStyle": "github",
  "boost": 0.05,
  "tags": ["new", "documentation"],
  "description": "New documentation source"
}
```

### 2. **Add Context Boosts** (if needed)
```json
"contextBoosts": {
  "New Context": {
    "/new-source": 1.0,
    "/other-source": 0.3
  }
}
```

### 3. **Add Library Mapping** (if needed)
```json
"libraryMappings": {
  "new-source-alias": "new-source"
}
```

### 4. **No Code Changes Required!**
The metadata APIs automatically handle the new source.

## Debugging

### πŸ” **Search Issues**
```bash
# Test specific queries
node -e "
import { search } from './dist/src/lib/search.js';
const results = await search('your query');
console.log(JSON.stringify(results, null, 2));
"

# Check FTS database
sqlite3 dist/data/docs.sqlite "SELECT * FROM docs WHERE docs MATCH 'your query' LIMIT 5;"
```

### πŸ“Š **Metadata Issues**
```bash
# Test metadata loading
node -e "
import { loadMetadata, getSourceBoosts } from './dist/src/lib/metadata.js';
loadMetadata();
console.log('Boosts:', getSourceBoosts());
"
```

### 🌐 **Server Issues**
```bash
# Check server health
curl http://localhost:3001/status
curl http://localhost:3122/health

# Test search endpoint
curl -X POST http://localhost:3001/mcp \
  -H "Content-Type: application/json" \
  -d '{"role": "user", "content": "wizard"}'
```

## Performance Optimization

### ⚑ **Search Performance**
- **FTS5 Tuning**: Modify `scripts/build-fts.ts` for different indexing strategies
- **Query Optimization**: Adjust `toMatchQuery()` in `src/lib/searchDb.ts`
- **Result Limits**: Configure `RETURN_K` environment variable

### πŸ’Ύ **Memory Usage**
- **Index Size**: Monitor `dist/data/` artifact sizes
- **Metadata Loading**: Lazy loading in `src/lib/metadata.ts`
- **Process Monitoring**: Use PM2 monitoring features

## Common Issues

### ❌ **Build Failures**
```bash
# Clean and rebuild
rm -rf dist/
npm run build:all
```

### ❌ **Search Returns No Results**
```bash
# Check if database exists
ls -la dist/data/docs.sqlite

# Verify index content
sqlite3 dist/data/docs.sqlite "SELECT COUNT(*) FROM docs;"
```

### ❌ **Metadata Loading Errors**
```bash
# Validate JSON syntax
node -e "JSON.parse(require('fs').readFileSync('src/metadata.json', 'utf8'))"

# Check file permissions
ls -la src/metadata.json
```

### ❌ **Server Won't Start**
```bash
# Check port availability
lsof -i :3001
lsof -i :3122

# Kill conflicting processes
lsof -ti:3001 | xargs kill -9
```

## Best Practices

### πŸ“ **Code Changes**
1. **Update Cursor Rules**: Modify `.cursor/rules/` when changing architecture
2. **Test First**: Run smoke tests before committing
3. **Metadata Over Code**: Use metadata.json for configuration changes
4. **Type Safety**: Use metadata APIs, never direct JSON access

### πŸ§ͺ **Testing**
1. **Smoke Tests**: Always run before deployment
2. **Integration Tests**: Test full MCP tool workflows
3. **Performance Tests**: Monitor search response times
4. **Output Validation**: Ensure format consistency

### πŸš€ **Deployment**
1. **Build Validation**: Ensure all artifacts generated
2. **Health Checks**: Verify all endpoints after deployment
3. **Rollback Plan**: Keep previous artifacts for quick rollback
4. **Monitoring**: Watch logs and performance metrics

## Useful Development Tools

### πŸ”§ **VS Code Extensions**
- **REST Client**: Use `test-search.http` for API testing
- **SQLite Viewer**: Inspect FTS5 database content
- **JSON Schema**: Validate metadata.json structure

### πŸ“Š **Monitoring**
```bash
# PM2 monitoring
pm2 monit

# Log streaming
pm2 logs mcp-sap-http --lines 100

# Process status
pm2 status
```