|
Download docs/MANUAL_TESTING.md from polyHedra/PolyHedra: direct link, hf CLI and curl.
- Browser
- Download file 6.3 kB
-
https://huggingface.co/spaces/polyHedra/PolyHedra/resolve/main/docs/MANUAL_TESTING.md
- Command line
-
hf download hf://spaces/polyHedra/PolyHedra/docs/MANUAL_TESTING.md
-
curl -L -o MANUAL_TESTING.md https://huggingface.co/spaces/polyHedra/PolyHedra/resolve/main/docs/MANUAL_TESTING.md
6.3 kB
| # Manual IDE Integration Testing Checklist | |
| This checklist guides manual testing of Polyhedra in real IDE environments. | |
| ## Test Environment Setup | |
| **IDEs to Test** (minimum 2 required): | |
| - [ ] Cursor | |
| - [ ] VS Code with GitHub Copilot | |
| - [ ] Windsurf | |
| - [ ] VS Code with MCP extension | |
| **Prerequisites:** | |
| - [ ] Polyhedra installed (`pip install polyhedra` or `uv pip install polyhedra`) | |
| - [ ] IDE configured per `docs/SETUP.md` | |
| - [ ] IDE restarted after configuration | |
| ## Test Suite | |
| ### Test 1: Tool Discovery | |
| **Objective:** Verify all tools are available in the IDE | |
| **Steps:** | |
| 1. Open IDE chat/copilot | |
| 2. Ask: "What Polyhedra tools are available?" | |
| 3. Verify response lists all 10 tools | |
| **Expected Result:** | |
| - [ ] search_papers | |
| - [ ] get_paper | |
| - [ ] query_similar_papers | |
| - [ ] index_papers | |
| - [ ] add_citation | |
| - [ ] get_citations | |
| - [ ] save_file | |
| - [ ] get_context | |
| - [ ] get_project_status | |
| - [ ] init_project | |
| **Status:** β Pass β Fail | |
| --- | |
| ### Test 2: Project Initialization | |
| **Objective:** Initialize a new research project | |
| **Steps:** | |
| 1. Create new empty directory | |
| 2. Open directory in IDE | |
| 3. In IDE chat: "Initialize a new research project named 'test-research'" | |
| 4. Verify directory structure created | |
| **Expected Result:** | |
| ``` | |
| literature/ | |
| ideas/ | |
| method/ | |
| paper/ | |
| references.bib | |
| .poly/ | |
| config.yaml | |
| embeddings/ | |
| ``` | |
| **Status:** β Pass β Fail | |
| --- | |
| ### Test 3: Paper Search | |
| **Objective:** Search for academic papers | |
| **Steps:** | |
| 1. In IDE chat: "Search for 5 recent papers on 'transformer models' from 2023-2024" | |
| 2. Wait for results | |
| 3. Verify papers returned with metadata | |
| **Expected Result:** | |
| - [ ] Returns 5 or fewer papers | |
| - [ ] Papers have titles, authors, years | |
| - [ ] Years are within 2023-2024 | |
| - [ ] Response time < 3 seconds | |
| **Status:** β Pass β Fail | |
| --- | |
| ### Test 4: Citation Management | |
| **Objective:** Add and retrieve citations | |
| **Steps:** | |
| 1. In IDE chat: "Add this BibTeX entry to my bibliography:" | |
| ```bibtex | |
| @article{vaswani2017attention, | |
| title={Attention is all you need}, | |
| author={Vaswani, Ashish and Shazeer, Noam}, | |
| journal={Advances in neural information processing systems}, | |
| year={2017} | |
| } | |
| ``` | |
| 2. In IDE chat: "Show me all my citations" | |
| 3. Verify citation appears | |
| **Expected Result:** | |
| - [ ] Citation added successfully | |
| - [ ] Citation appears in list | |
| - [ ] references.bib file updated | |
| **Status:** β Pass β Fail | |
| --- | |
| ### Test 5: File Operations | |
| **Objective:** Write and read project files | |
| **Steps:** | |
| 1. In IDE chat: "Create a file ideas/hypothesis.md with content: '# Research Hypothesis\n\nWe hypothesize that...'" | |
| 2. Verify file created | |
| 3. In IDE chat: "Read the contents of ideas/hypothesis.md" | |
| 4. Verify content matches | |
| **Expected Result:** | |
| - [ ] File created in correct location | |
| - [ ] Content is correct | |
| - [ ] Read operation returns same content | |
| **Status:** β Pass β Fail | |
| --- | |
| ### Test 6: Project Status | |
| **Objective:** Get comprehensive project status | |
| **Steps:** | |
| 1. In IDE chat: "Show me my project status" | |
| 2. Verify status information returned | |
| **Expected Result:** | |
| - [ ] Shows papers_count | |
| - [ ] Shows citations_count | |
| - [ ] Shows rag_indexed status | |
| - [ ] Lists standard_files existence | |
| - [ ] Response time < 100ms | |
| **Status:** β Pass β Fail | |
| --- | |
| ### Test 7: Complete Workflow | |
| **Objective:** Execute a complete research workflow | |
| **Steps:** | |
| 1. "Initialize a new project called 'ml-survey'" | |
| 2. "Search for 3 papers on 'deep learning'" | |
| 3. "Save these papers to literature/papers.json" | |
| 4. "Add citations from these papers to my bibliography" | |
| 5. "Get my project status" | |
| 6. "Create a file literature/review.md with notes about these papers" | |
| **Expected Result:** | |
| - [ ] All steps complete without errors | |
| - [ ] Files created correctly | |
| - [ ] Citations added successfully | |
| - [ ] Status shows correct counts | |
| **Status:** β Pass β Fail | |
| --- | |
| ### Test 8: Error Handling | |
| **Objective:** Verify graceful error handling | |
| **Steps:** | |
| 1. In IDE chat: "Read the file nonexistent.md" | |
| 2. Verify error message is clear | |
| 3. In IDE chat: "Query similar papers about 'test'" (before indexing) | |
| 4. Verify error message is helpful | |
| **Expected Result:** | |
| - [ ] Missing file error clearly explains issue | |
| - [ ] Not indexed error suggests running index_papers | |
| - [ ] No crashes or unclear errors | |
| **Status:** β Pass β Fail | |
| --- | |
| ### Test 9: Semantic Search (RAG) | |
| **Objective:** Index and query papers semantically | |
| **Steps:** | |
| 1. Create literature/papers.json with 3+ papers | |
| 2. In IDE chat: "Index the papers in literature/papers.json" | |
| 3. Wait for indexing (may take 30-60s first time) | |
| 4. In IDE chat: "Find papers similar to 'attention mechanisms'" | |
| 5. Verify results with similarity scores | |
| **Expected Result:** | |
| - [ ] Indexing completes successfully | |
| - [ ] Returns similar papers | |
| - [ ] Each result has similarity score | |
| - [ ] Query time < 1 second (after initial model download) | |
| **Status:** β Pass β Fail | |
| --- | |
| ### Test 10: Performance Validation | |
| **Objective:** Verify performance targets | |
| **Measurements:** | |
| - [ ] Paper search: < 2 seconds | |
| - [ ] Local file ops: < 100ms | |
| - [ ] RAG query: < 500ms (after indexing) | |
| - [ ] Project status: < 100ms | |
| **Status:** β Pass β Fail | |
| --- | |
| ## Test Results Summary | |
| **IDE:** _____________ | |
| **Date:** _____________ | |
| **Tester:** _____________ | |
| **Results:** | |
| - Tests Passed: _____ / 10 | |
| - Tests Failed: _____ / 10 | |
| - Critical Issues: _____________ | |
| **Notes:** | |
| _________________________________________________________________ | |
| _________________________________________________________________ | |
| _________________________________________________________________ | |
| ## Critical Issues Found | |
| | Test | Issue | Severity | Notes | | |
| |------|-------|----------|-------| | |
| | | | | | | |
| ## Performance Benchmarks | |
| | Operation | Target | Actual | Status | | |
| |-----------|--------|--------|--------| | |
| | Paper search | < 2s | ___s | β Pass β Fail | | |
| | File write | < 100ms | ___ms | β Pass β Fail | | |
| | File read | < 100ms | ___ms | β Pass β Fail | | |
| | Project status | < 100ms | ___ms | β Pass β Fail | | |
| | RAG query | < 500ms | ___ms | β Pass β Fail | | |
| ## Sign-off | |
| **Tested By:** _____________ | |
| **Date:** _____________ | |
| **Approved:** β Yes β No (explain below) | |
| **Comments:** | |
| _________________________________________________________________ | |
| _________________________________________________________________ | |