PolyHedra / docs /MANUAL_TESTING.md
yf176
Deploy Polyhedra to HF Spaces
891fe52
|
Raw History Blame Contribute Delete
6.3 kB
# Manual IDE Integration Testing Checklist
This checklist guides manual testing of Polyhedra in real IDE environments.
## Test Environment Setup
**IDEs to Test** (minimum 2 required):
- [ ] Cursor
- [ ] VS Code with GitHub Copilot
- [ ] Windsurf
- [ ] VS Code with MCP extension
**Prerequisites:**
- [ ] Polyhedra installed (`pip install polyhedra` or `uv pip install polyhedra`)
- [ ] IDE configured per `docs/SETUP.md`
- [ ] IDE restarted after configuration
## Test Suite
### Test 1: Tool Discovery
**Objective:** Verify all tools are available in the IDE
**Steps:**
1. Open IDE chat/copilot
2. Ask: "What Polyhedra tools are available?"
3. Verify response lists all 10 tools
**Expected Result:**
- [ ] search_papers
- [ ] get_paper
- [ ] query_similar_papers
- [ ] index_papers
- [ ] add_citation
- [ ] get_citations
- [ ] save_file
- [ ] get_context
- [ ] get_project_status
- [ ] init_project
**Status:** ☐ Pass ☐ Fail
---
### Test 2: Project Initialization
**Objective:** Initialize a new research project
**Steps:**
1. Create new empty directory
2. Open directory in IDE
3. In IDE chat: "Initialize a new research project named 'test-research'"
4. Verify directory structure created
**Expected Result:**
```
literature/
ideas/
method/
paper/
references.bib
.poly/
config.yaml
embeddings/
```
**Status:** ☐ Pass ☐ Fail
---
### Test 3: Paper Search
**Objective:** Search for academic papers
**Steps:**
1. In IDE chat: "Search for 5 recent papers on 'transformer models' from 2023-2024"
2. Wait for results
3. Verify papers returned with metadata
**Expected Result:**
- [ ] Returns 5 or fewer papers
- [ ] Papers have titles, authors, years
- [ ] Years are within 2023-2024
- [ ] Response time < 3 seconds
**Status:** ☐ Pass ☐ Fail
---
### Test 4: Citation Management
**Objective:** Add and retrieve citations
**Steps:**
1. In IDE chat: "Add this BibTeX entry to my bibliography:"
```bibtex
@article{vaswani2017attention,
title={Attention is all you need},
author={Vaswani, Ashish and Shazeer, Noam},
journal={Advances in neural information processing systems},
year={2017}
}
```
2. In IDE chat: "Show me all my citations"
3. Verify citation appears
**Expected Result:**
- [ ] Citation added successfully
- [ ] Citation appears in list
- [ ] references.bib file updated
**Status:** ☐ Pass ☐ Fail
---
### Test 5: File Operations
**Objective:** Write and read project files
**Steps:**
1. In IDE chat: "Create a file ideas/hypothesis.md with content: '# Research Hypothesis\n\nWe hypothesize that...'"
2. Verify file created
3. In IDE chat: "Read the contents of ideas/hypothesis.md"
4. Verify content matches
**Expected Result:**
- [ ] File created in correct location
- [ ] Content is correct
- [ ] Read operation returns same content
**Status:** ☐ Pass ☐ Fail
---
### Test 6: Project Status
**Objective:** Get comprehensive project status
**Steps:**
1. In IDE chat: "Show me my project status"
2. Verify status information returned
**Expected Result:**
- [ ] Shows papers_count
- [ ] Shows citations_count
- [ ] Shows rag_indexed status
- [ ] Lists standard_files existence
- [ ] Response time < 100ms
**Status:** ☐ Pass ☐ Fail
---
### Test 7: Complete Workflow
**Objective:** Execute a complete research workflow
**Steps:**
1. "Initialize a new project called 'ml-survey'"
2. "Search for 3 papers on 'deep learning'"
3. "Save these papers to literature/papers.json"
4. "Add citations from these papers to my bibliography"
5. "Get my project status"
6. "Create a file literature/review.md with notes about these papers"
**Expected Result:**
- [ ] All steps complete without errors
- [ ] Files created correctly
- [ ] Citations added successfully
- [ ] Status shows correct counts
**Status:** ☐ Pass ☐ Fail
---
### Test 8: Error Handling
**Objective:** Verify graceful error handling
**Steps:**
1. In IDE chat: "Read the file nonexistent.md"
2. Verify error message is clear
3. In IDE chat: "Query similar papers about 'test'" (before indexing)
4. Verify error message is helpful
**Expected Result:**
- [ ] Missing file error clearly explains issue
- [ ] Not indexed error suggests running index_papers
- [ ] No crashes or unclear errors
**Status:** ☐ Pass ☐ Fail
---
### Test 9: Semantic Search (RAG)
**Objective:** Index and query papers semantically
**Steps:**
1. Create literature/papers.json with 3+ papers
2. In IDE chat: "Index the papers in literature/papers.json"
3. Wait for indexing (may take 30-60s first time)
4. In IDE chat: "Find papers similar to 'attention mechanisms'"
5. Verify results with similarity scores
**Expected Result:**
- [ ] Indexing completes successfully
- [ ] Returns similar papers
- [ ] Each result has similarity score
- [ ] Query time < 1 second (after initial model download)
**Status:** ☐ Pass ☐ Fail
---
### Test 10: Performance Validation
**Objective:** Verify performance targets
**Measurements:**
- [ ] Paper search: < 2 seconds
- [ ] Local file ops: < 100ms
- [ ] RAG query: < 500ms (after indexing)
- [ ] Project status: < 100ms
**Status:** ☐ Pass ☐ Fail
---
## Test Results Summary
**IDE:** _____________
**Date:** _____________
**Tester:** _____________
**Results:**
- Tests Passed: _____ / 10
- Tests Failed: _____ / 10
- Critical Issues: _____________
**Notes:**
_________________________________________________________________
_________________________________________________________________
_________________________________________________________________
## Critical Issues Found
| Test | Issue | Severity | Notes |
|------|-------|----------|-------|
| | | | |
## Performance Benchmarks
| Operation | Target | Actual | Status |
|-----------|--------|--------|--------|
| Paper search | < 2s | ___s | ☐ Pass ☐ Fail |
| File write | < 100ms | ___ms | ☐ Pass ☐ Fail |
| File read | < 100ms | ___ms | ☐ Pass ☐ Fail |
| Project status | < 100ms | ___ms | ☐ Pass ☐ Fail |
| RAG query | < 500ms | ___ms | ☐ Pass ☐ Fail |
## Sign-off
**Tested By:** _____________
**Date:** _____________
**Approved:** ☐ Yes ☐ No (explain below)
**Comments:**
_________________________________________________________________
_________________________________________________________________