PolyHedra / docs /MANUAL_TESTING.md
yf176
Deploy Polyhedra to HF Spaces
891fe52
|
Raw History Blame Contribute Delete
6.3 kB

Manual IDE Integration Testing Checklist

This checklist guides manual testing of Polyhedra in real IDE environments.

Test Environment Setup

IDEs to Test (minimum 2 required):

  • Cursor
  • VS Code with GitHub Copilot
  • Windsurf
  • VS Code with MCP extension

Prerequisites:

  • Polyhedra installed (pip install polyhedra or uv pip install polyhedra)
  • IDE configured per docs/SETUP.md
  • IDE restarted after configuration

Test Suite

Test 1: Tool Discovery

Objective: Verify all tools are available in the IDE

Steps:

  1. Open IDE chat/copilot
  2. Ask: "What Polyhedra tools are available?"
  3. Verify response lists all 10 tools

Expected Result:

  • search_papers
  • get_paper
  • query_similar_papers
  • index_papers
  • add_citation
  • get_citations
  • save_file
  • get_context
  • get_project_status
  • init_project

Status: ☐ Pass ☐ Fail


Test 2: Project Initialization

Objective: Initialize a new research project

Steps:

  1. Create new empty directory
  2. Open directory in IDE
  3. In IDE chat: "Initialize a new research project named 'test-research'"
  4. Verify directory structure created

Expected Result:

literature/
ideas/
method/
paper/
references.bib
.poly/
  config.yaml
  embeddings/

Status: ☐ Pass ☐ Fail


Test 3: Paper Search

Objective: Search for academic papers

Steps:

  1. In IDE chat: "Search for 5 recent papers on 'transformer models' from 2023-2024"
  2. Wait for results
  3. Verify papers returned with metadata

Expected Result:

  • Returns 5 or fewer papers
  • Papers have titles, authors, years
  • Years are within 2023-2024
  • Response time < 3 seconds

Status: ☐ Pass ☐ Fail


Test 4: Citation Management

Objective: Add and retrieve citations

Steps:

  1. In IDE chat: "Add this BibTeX entry to my bibliography:"
@article{vaswani2017attention,
  title={Attention is all you need},
  author={Vaswani, Ashish and Shazeer, Noam},
  journal={Advances in neural information processing systems},
  year={2017}
}
  1. In IDE chat: "Show me all my citations"
  2. Verify citation appears

Expected Result:

  • Citation added successfully
  • Citation appears in list
  • references.bib file updated

Status: ☐ Pass ☐ Fail


Test 5: File Operations

Objective: Write and read project files

Steps:

  1. In IDE chat: "Create a file ideas/hypothesis.md with content: '# Research Hypothesis\n\nWe hypothesize that...'"
  2. Verify file created
  3. In IDE chat: "Read the contents of ideas/hypothesis.md"
  4. Verify content matches

Expected Result:

  • File created in correct location
  • Content is correct
  • Read operation returns same content

Status: ☐ Pass ☐ Fail


Test 6: Project Status

Objective: Get comprehensive project status

Steps:

  1. In IDE chat: "Show me my project status"
  2. Verify status information returned

Expected Result:

  • Shows papers_count
  • Shows citations_count
  • Shows rag_indexed status
  • Lists standard_files existence
  • Response time < 100ms

Status: ☐ Pass ☐ Fail


Test 7: Complete Workflow

Objective: Execute a complete research workflow

Steps:

  1. "Initialize a new project called 'ml-survey'"
  2. "Search for 3 papers on 'deep learning'"
  3. "Save these papers to literature/papers.json"
  4. "Add citations from these papers to my bibliography"
  5. "Get my project status"
  6. "Create a file literature/review.md with notes about these papers"

Expected Result:

  • All steps complete without errors
  • Files created correctly
  • Citations added successfully
  • Status shows correct counts

Status: ☐ Pass ☐ Fail


Test 8: Error Handling

Objective: Verify graceful error handling

Steps:

  1. In IDE chat: "Read the file nonexistent.md"
  2. Verify error message is clear
  3. In IDE chat: "Query similar papers about 'test'" (before indexing)
  4. Verify error message is helpful

Expected Result:

  • Missing file error clearly explains issue
  • Not indexed error suggests running index_papers
  • No crashes or unclear errors

Status: ☐ Pass ☐ Fail


Test 9: Semantic Search (RAG)

Objective: Index and query papers semantically

Steps:

  1. Create literature/papers.json with 3+ papers
  2. In IDE chat: "Index the papers in literature/papers.json"
  3. Wait for indexing (may take 30-60s first time)
  4. In IDE chat: "Find papers similar to 'attention mechanisms'"
  5. Verify results with similarity scores

Expected Result:

  • Indexing completes successfully
  • Returns similar papers
  • Each result has similarity score
  • Query time < 1 second (after initial model download)

Status: ☐ Pass ☐ Fail


Test 10: Performance Validation

Objective: Verify performance targets

Measurements:

  • Paper search: < 2 seconds
  • Local file ops: < 100ms
  • RAG query: < 500ms (after indexing)
  • Project status: < 100ms

Status: ☐ Pass ☐ Fail


Test Results Summary

IDE: _____________ Date: _____________ Tester: _____________

Results:

  • Tests Passed: _____ / 10
  • Tests Failed: _____ / 10
  • Critical Issues: _____________

Notes:




Critical Issues Found

Test Issue Severity Notes

Performance Benchmarks

Operation Target Actual Status
Paper search < 2s ___s ☐ Pass ☐ Fail
File write < 100ms ___ms ☐ Pass ☐ Fail
File read < 100ms ___ms ☐ Pass ☐ Fail
Project status < 100ms ___ms ☐ Pass ☐ Fail
RAG query < 500ms ___ms ☐ Pass ☐ Fail

Sign-off

Tested By: _____________ Date: _____________ Approved: ☐ Yes ☐ No (explain below)

Comments: