diff --git a/.github/ISSUE_TEMPLATE/bug_report.yml b/.github/ISSUE_TEMPLATE/bug_report.yml new file mode 100644 index 0000000000000000000000000000000000000000..e0a1fd1fe6e355c32c7a6146f2c3cb72df5c6189 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/bug_report.yml @@ -0,0 +1,117 @@ +name: ๐Ÿ› Bug Report +description: Report a bug or unexpected behavior in Forge +title: "[Bug]: " +labels: ["type: bug"] +body: + - type: markdown + attributes: + value: | + Thanks for taking the time to report a bug! Please fill out the sections below to help us reproduce and fix the issue. + + - type: textarea + id: description + attributes: + label: Bug Description + description: A clear and concise description of what the bug is. + placeholder: Describe the bug... + validations: + required: true + + - type: textarea + id: reproduction + attributes: + label: Steps to Reproduce + description: Steps to reproduce the behavior + placeholder: | + 1. Run command '...' + 2. Enter prompt '...' + 3. See error + validations: + required: true + + - type: textarea + id: expected + attributes: + label: Expected Behavior + description: What did you expect to happen? + placeholder: Describe what should happen... + validations: + required: true + + - type: textarea + id: actual + attributes: + label: Actual Behavior + description: What actually happened? + placeholder: Describe what actually happened... + validations: + required: true + + - type: input + id: version + attributes: + label: Forge Version + description: What version of Forge are you running? (`forge --version`) + placeholder: e.g., 0.1.0 + validations: + required: true + + - type: input + id: os + attributes: + label: Operating System & Version + description: What operating system and version are you using? + placeholder: e.g., macOS 14.1, Ubuntu 22.04, Windows 11 + validations: + required: false + + - type: dropdown + id: provider + attributes: + label: AI Provider + description: Which AI provider are you using? + options: + - OpenAI + - Anthropic + - OpenRouter + - Google Vertex AI + - Cerebras + - x-ai + - z.ai + - Requesty + - IO Intelligence + - Groq + - Amazon Bedrock + - OpenAI-compatible + - Other + validations: + required: false + + - type: input + id: model + attributes: + label: Model + description: Which model are you using? + placeholder: e.g., claude-3.7-sonnet, gpt-4, o3-mini-high + + - type: dropdown + id: installation + attributes: + label: Installation Method + description: How did you install Forge? + options: + - npx forgecode@latest + - npm install -g forgecode + - curl -fsSL https://forgecode.dev/cli | sh + - Built from source + - Other + validations: + required: true + + - type: textarea + id: config + attributes: + label: Configuration + description: | + Please share relevant parts of your `forge.yaml` configuration (remove sensitive data like API keys). + render: yaml diff --git a/.github/ISSUE_TEMPLATE/config.yml b/.github/ISSUE_TEMPLATE/config.yml new file mode 100644 index 0000000000000000000000000000000000000000..96b03849fee7a645f4b77be34778de924d1c175e --- /dev/null +++ b/.github/ISSUE_TEMPLATE/config.yml @@ -0,0 +1,11 @@ +blank_issues_enabled: false +contact_links: + - name: ๐Ÿ’ฌ Discord Community + url: https://discord.gg/kRZBPpkgwq + about: Join our Discord server for questions, discussions, and community support + - name: ๐Ÿ“– Documentation + url: https://forgecode.dev/docs/ + about: Check out our comprehensive documentation + - name: โ“ Discussions + url: https://github.com/antinomyhq/forge/discussions + about: Ask questions and discuss ideas with the community diff --git a/.github/ISSUE_TEMPLATE/documentation.yml b/.github/ISSUE_TEMPLATE/documentation.yml new file mode 100644 index 0000000000000000000000000000000000000000..d04a25a289e3e29fca11881a4c0c0dd27abdd42e --- /dev/null +++ b/.github/ISSUE_TEMPLATE/documentation.yml @@ -0,0 +1,100 @@ +name: ๐Ÿ“š Documentation Issue +description: Report problems with documentation or suggest improvements +title: "[Docs]: " +labels: ["type: docs"] +body: + - type: markdown + attributes: + value: | + Thanks for helping improve our documentation! Clear docs help everyone. + + - type: dropdown + id: issue-type + attributes: + label: Documentation Issue Type + description: What type of documentation issue is this? + options: + - Missing documentation + - Incorrect/outdated information + - Unclear explanation + - Broken link + - Typo/grammar + - Code example doesn't work + - Need more examples + - Need better organization + - Other + validations: + required: true + + - type: input + id: location + attributes: + label: Documentation Location + description: Where is the documentation issue? (URL, file path, or section name) + placeholder: https://github.com/antinomyhq/forge/blob/main/docs/... + validations: + required: true + + - type: textarea + id: current-content + attributes: + label: Current Content + description: What does the current documentation say? (Quote or describe) + placeholder: Currently, the documentation says... + + - type: textarea + id: issue-description + attributes: + label: Issue Description + description: What is wrong or confusing about the current documentation? + placeholder: The issue is that... + validations: + required: true + + - type: textarea + id: suggested-improvement + attributes: + label: Suggested Improvement + description: How would you improve this documentation? + placeholder: It would be better if... + validations: + required: true + + - type: dropdown + id: audience + attributes: + label: Target Audience + description: Who would benefit from this documentation improvement? + options: + - New users/beginners + - Experienced users + - Contributors/developers + - All users + validations: + required: true + + - type: textarea + id: examples + attributes: + label: Example or Reference + description: Provide examples or references that could help improve the documentation + placeholder: | + For example: + ```yaml + # forge.yaml example + model: claude-3.7-sonnet + ``` + + - type: checkboxes + id: contribution + attributes: + label: Contribution + description: Can you help improve this documentation? + options: + - label: I'm willing to submit a PR with the documentation improvements + + - type: textarea + id: additional + attributes: + label: Additional Context + description: Any other information that would help improve the documentation diff --git a/.github/ISSUE_TEMPLATE/feature_request.yml b/.github/ISSUE_TEMPLATE/feature_request.yml new file mode 100644 index 0000000000000000000000000000000000000000..f92f4acc4553260e65597abfdfb3561857d833bd --- /dev/null +++ b/.github/ISSUE_TEMPLATE/feature_request.yml @@ -0,0 +1,155 @@ +name: โœจ Feature Request +description: Suggest a new feature or enhancement for Forge +title: "[Feature]: " +labels: ["type: feature"] +body: + - type: markdown + attributes: + value: | + Thanks for suggesting a new feature! + + **Before you continue, please ensure you can clearly answer:** + 1. **Why do you NEED this feature?** (Not just "nice to have") + 2. **What is NOT possible right now?** (Current limitations) + 3. **What WILL be possible with this feature?** (Concrete outcomes) + + Vague or incomplete requests will be closed. Take your time to write a compelling case. + + - type: textarea + id: why-needed + attributes: + label: "1. Why Do You NEED This Feature?" + description: | + **Be specific.** Explain the real-world problem or blocker you're facing. + Not "it would be cool if..." but "I cannot accomplish X because..." + Why is this important to your workflow? What is the impact of NOT having this? + placeholder: | + I NEED this feature because: + - Current workflow requires me to... + - This costs me X hours per week... + - Without this, I cannot accomplish... + - The business impact is... + validations: + required: true + + - type: textarea + id: current-limitation + attributes: + label: "2. What Is NOT Possible Right Now?" + description: | + **Describe the current limitations in detail.** + What have you tried? What doesn't work? Why can't you work around it? + Include specific examples, commands, or workflows that fail. + placeholder: | + Currently, I cannot: + - [Specific action/workflow that fails] + - [What happens when you try] + - [Why existing features don't solve this] + + I've tried: + - [Workaround 1 and why it doesn't work] + - [Workaround 2 and why it doesn't work] + validations: + required: true + + - type: textarea + id: future-possibilities + attributes: + label: "3. What WILL Be Possible With This Feature?" + description: | + **Describe concrete outcomes and capabilities.** + What specific tasks/workflows will this enable? + How will your work change? What new things can you build/do? + placeholder: | + With this feature, I will be able to: + - [Specific new capability 1] + - [Specific new capability 2] + - [Measurable improvement: time saved, errors reduced, etc.] + + Example workflow: + ``` + # Show the exact commands or steps you'll use + forge new-feature --example + ``` + validations: + required: true + + - type: textarea + id: solution + attributes: + label: Proposed Solution (User Experience) + description: | + **Focus on the user experience, not implementation details.** + How would a user interact with this feature? What would they see/do? + Focus on what the user experiences, not how it would be built. + placeholder: | + From a user's perspective, the feature should work like this: + - When I [action], I would see/get [result] + - The interaction would be: [step-by-step user flow] + - Example: "When I run `forge search docs`, I'd see matching results from Google Workspace" + validations: + required: true + + - type: textarea + id: alternatives + attributes: + label: Alternatives Considered + description: Have you considered any alternative solutions or workarounds? + placeholder: I've also thought about... + + - type: dropdown + id: category + attributes: + label: Feature Category + description: Which area does this feature relate to? + options: + - AI/LLM Integration + - Code Generation/Editing + - File Operations + - Configuration + - CLI/UX + - Documentation + - Performance + - Security + - MCP Integration + - Multi-Agent Workflows + - Shell Plugin + - Testing + - Other + validations: + required: true + + - type: dropdown + id: priority + attributes: + label: Priority/Impact + description: How important is this feature to your workflow? + options: + - Critical - Blocking my work + - High - Would significantly improve my workflow + - Medium - Would be nice to have + - Low - Minor improvement + validations: + required: true + + - type: textarea + id: examples + attributes: + label: Examples from Other Tools + description: Are there similar features in other tools that could serve as inspiration? + placeholder: Tool X does this by... + + - type: textarea + id: additional + attributes: + label: Additional Context + description: Add any other context, screenshots, or examples about the feature request. + + - type: checkboxes + id: requirements-check + attributes: + label: "Pre-submission Checklist" + options: + - label: "I have searched existing issues and confirmed this is not a duplicate" + required: true + - label: "I am willing to submit a PR to implement this feature" diff --git a/.github/ISSUE_TEMPLATE/performance_issue.yml b/.github/ISSUE_TEMPLATE/performance_issue.yml new file mode 100644 index 0000000000000000000000000000000000000000..cff6374548e365a24e3124c1573e00cf77e5724d --- /dev/null +++ b/.github/ISSUE_TEMPLATE/performance_issue.yml @@ -0,0 +1,150 @@ +name: โšก Performance Issue +description: Report performance problems, slowness, or resource usage issues +title: "[Performance]: " +labels: ["type: performance"] +body: + - type: markdown + attributes: + value: | + Thanks for reporting a performance issue! Please provide detailed information to help us investigate. + + - type: textarea + id: description + attributes: + label: Performance Issue Description + description: Describe the performance problem you're experiencing + placeholder: Forge becomes slow when... + validations: + required: true + + - type: dropdown + id: issue-type + attributes: + label: Issue Type + description: What type of performance issue are you experiencing? + options: + - Slow response time + - High CPU usage + - High memory usage + - Network latency + - File I/O slowness + - Startup time + - Tool execution time + - Other + validations: + required: true + + - type: textarea + id: reproduction + attributes: + label: Steps to Reproduce + description: What actions lead to the performance issue? + placeholder: | + 1. Run command '...' + 2. Process large file/directory + 3. Notice slowdown after... + validations: + required: true + + - type: textarea + id: expected-performance + attributes: + label: Expected Performance + description: What performance did you expect? + placeholder: I expected the operation to complete in... + validations: + required: true + + - type: textarea + id: actual-performance + attributes: + label: Actual Performance + description: What performance are you experiencing? + placeholder: The operation takes... and uses... + validations: + required: true + + - type: textarea + id: measurements + attributes: + label: Performance Measurements + description: | + If possible, provide specific measurements (time, memory, CPU usage, etc.) + You can use tools like `time`, `htop`, Activity Monitor, or Task Manager + placeholder: | + - Execution time: 45 seconds + - Memory usage: 2.5 GB + - CPU usage: 180% + render: markdown + + - type: input + id: project-size + attributes: + label: Project Size + description: Approximate size of your project (number of files, lines of code, etc.) + placeholder: e.g., 500 files, ~50k lines of code + + - type: textarea + id: environment + attributes: + label: Environment Details + description: | + Provide details about your environment: + - Forge version (`forge --version`) + - Operating system and version + - CPU, RAM + - AI provider and model + - Shell + placeholder: | + - Forge: 0.1.0 + - OS: macOS 14.1 + - CPU: M2 Pro + - RAM: 16GB + - Provider: Anthropic + - Model: claude-3.7-sonnet + - Shell: zsh + validations: + required: true + + - type: textarea + id: config + attributes: + label: Configuration + description: Share relevant parts of your `forge.yaml` (remove sensitive data) + render: yaml + + - type: textarea + id: profiling + attributes: + label: Profiling Data + description: | + If you have any profiling data or verbose logs, please share them here. + Run with `--verbose` flag for detailed output. + render: shell + + - type: dropdown + id: frequency + attributes: + label: Frequency + description: How often does this performance issue occur? + options: + - Always + - Often (>50% of the time) + - Sometimes (25-50% of the time) + - Rarely (<25% of the time) + - Only in specific scenarios + validations: + required: true + + - type: textarea + id: workarounds + attributes: + label: Workarounds + description: Have you found any workarounds for this issue? + placeholder: I can work around this by... + + - type: textarea + id: additional + attributes: + label: Additional Context + description: Any other relevant information about the performance issue diff --git a/.github/ISSUE_TEMPLATE/provider_integration.yml b/.github/ISSUE_TEMPLATE/provider_integration.yml new file mode 100644 index 0000000000000000000000000000000000000000..213d7f46b2666fa78cee5b2d9a4fb139e8b1af86 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/provider_integration.yml @@ -0,0 +1,129 @@ +name: ๐Ÿ”Œ Provider Integration Request +description: Request support for a new AI provider +title: "[Provider]: " +labels: ["type: feature"] +body: + - type: markdown + attributes: + value: | + Thanks for suggesting a new AI provider! Please provide detailed information about the provider. + + - type: input + id: provider-name + attributes: + label: Provider Name + description: What is the name of the AI provider? + placeholder: e.g., Cohere, Mistral AI, Together AI + validations: + required: true + + - type: input + id: website + attributes: + label: Provider Website + description: Link to the provider's website + placeholder: https://... + validations: + required: true + + - type: input + id: api-docs + attributes: + label: API Documentation + description: Link to the provider's API documentation + placeholder: https://docs... + + - type: dropdown + id: api-type + attributes: + label: API Compatibility + description: What type of API does this provider offer? + options: + - OpenAI-compatible + - Anthropic-compatible + - Custom API + - Other + validations: + required: true + + - type: textarea + id: models + attributes: + label: Available Models + description: List the models this provider offers that would be useful in Forge + placeholder: | + - model-1 (description) + - model-2 (description) + validations: + required: true + + - type: textarea + id: features + attributes: + label: Key Features + description: What are the key features or advantages of this provider? + placeholder: | + - Fast inference + - Specialized models + - Cost-effective + - Open source models + validations: + required: true + + - type: dropdown + id: authentication + attributes: + label: Authentication Method + description: How does authentication work for this provider? + options: + - API Key + - OAuth + - Bearer Token + - Basic Auth + - Other + validations: + required: true + + - type: textarea + id: pricing + attributes: + label: Pricing Information + description: Link to pricing information or describe the pricing model + placeholder: https://provider.com/pricing or "Free tier available, paid plans start at..." + + - type: dropdown + id: priority + attributes: + label: Community Interest + description: How much demand do you think there is for this provider? + options: + - High - Many users would benefit + - Medium - Some users would benefit + - Low - Niche use case + validations: + required: true + + - type: textarea + id: use-case + attributes: + label: Your Use Case + description: Why do you want to use this provider with Forge? + placeholder: I would use this provider because... + validations: + required: true + + - type: checkboxes + id: contribution + attributes: + label: Contribution + description: Are you willing to help with this integration? + options: + - label: I'm willing to submit a PR to implement this integration + - label: I can help with testing + - label: I have access to this provider's API for testing + + - type: textarea + id: additional + attributes: + label: Additional Context + description: Any other information about this provider that would be helpful diff --git a/.github/contribution.md b/.github/contribution.md new file mode 100644 index 0000000000000000000000000000000000000000..ca27a67dd6ec49a17dc2ff7176192b8004aefd73 --- /dev/null +++ b/.github/contribution.md @@ -0,0 +1,69 @@ +# Code-Forge ๐Ÿ› ๏ธ + +A sophisticated AI-powered coding assistant platform built in Rust, designed to provide intelligent code generation, manipulation, and analysis capabilities through a modular and extensible architecture. + +## โœจ Features + +- ๐Ÿค– **AI-Powered Code Generation** - Advanced code generation and manipulation using modern AI models +- ๐Ÿ” **Smart Code Analysis** - Language-aware parsing and analysis for multiple programming languages +- ๐Ÿ› ๏ธ **Extensive Tool System** - Rich set of development tools including file operations, shell commands, and code outline generation +- ๐Ÿ’พ **Persistent Conversations** - Maintain context and history across coding sessions +- ๐Ÿ”’ **Secure Operations** - Built-in security measures for file system and shell operations +- ๐Ÿ”Œ **Extensible Architecture** - Modular design supporting easy addition of new features and languages + +## ๐Ÿš€ Setup + +### Prerequisites + +- Rust toolchain (1.75+) +- SQLite +- Tree-sitter (for code analysis) + +### Installation + +```bash +# Build the project +cargo build --release + +# Run the server +cargo run --release +``` + +## ๐Ÿ—๏ธ Project Structure + +``` +code-forge/ +โ”œโ”€โ”€ crates/ +โ”‚ โ”œโ”€โ”€ forge_main/ # CLI and main application logic +โ”‚ โ”œโ”€โ”€ forge_domain/ # Core domain models and interfaces +โ”‚ โ”œโ”€โ”€ forge_services/ # HTTP API and database management +โ”‚ โ”œโ”€โ”€ forge_tool/ # Tool implementations +โ”‚ โ””โ”€โ”€ forge_walker/ # File system operations +``` + +## ๐Ÿ› ๏ธ Core Components + +- **Domain Layer** (`forge_domain`) - Core business logic and interfaces +- **Tool Layer** (`forge_tool`) - Development tools implementation +- **Server Layer** (`forge_services`) - API endpoints and persistence +- **Main Application** (`forge_main`) - CLI and application coordination + +## ๐Ÿ”ง Configuration + +The application requires several environment variables for proper operation: + +```bash +# Required environment variables +DATABASE_URL="sqlite:path/to/database.db" +OPENROUTER_API_KEY="your-api-key" +``` + +## ๐Ÿ“š Documentation + +Internal documentation: +- [Onboarding Guide](docs/onboarding.md) +- [Architecture Overview](docs/architecture.md) + +## ๐Ÿ”’ Proprietary Software + +This is proprietary software. All rights reserved. \ No newline at end of file diff --git a/.github/dependabot.yml b/.github/dependabot.yml new file mode 100644 index 0000000000000000000000000000000000000000..801d6c923bf02f16a6f2d6d321b35dc8706374f0 --- /dev/null +++ b/.github/dependabot.yml @@ -0,0 +1,26 @@ +# https://docs.github.com/en/code-security/dependabot/dependabot-version-updates/optimizing-pr-creation-version-updates#setting-up-a-cooldown-period-for-dependency-updates + +version: 2 +updates: + - package-ecosystem: cargo + directory: / + schedule: + interval: weekly + labels: ["type: chore"] + # https://docs.github.com/en/code-security/dependabot/working-with-dependabot/dependabot-options-reference#groups-- + groups: + major: + update-types: ["major"] + minor: + update-types: ["minor"] + patch: + update-types: ["patch"] + - package-ecosystem: github-actions + directory: / + schedule: + interval: weekly + labels: ["type: chore"] + groups: + actions: + # Combine all images of last week + patterns: ["*"] diff --git a/.github/labels.json b/.github/labels.json new file mode 100644 index 0000000000000000000000000000000000000000..27f0e977c7053d18d17807c8f3eab1e52b1d44a4 --- /dev/null +++ b/.github/labels.json @@ -0,0 +1,349 @@ +[ + { + "name": "release: skip changelog", + "color": "d73a4a", + "description": "Skip this change from release notes.", + "aliases": ["skip"] + }, + { + "name": "release: breaking", + "color": "d73a4a", + "description": "Introduces a breaking change.", + "aliases": [ + "breaking change", + "breaking-change", + "Type: Breaking Change", + "type: breaking", + "breaking" + ] + }, + + { + "name": "good first issue", + "color": "7057ff", + "description": "Good for newcomers.", + "aliases": [ + "beginner-friendly", + "beginner", + "good for beginner", + "Good for beginners", + "good-starter-issue", + "starter-issue", + "starter", + "status: good starter issue" + ] + }, + { + "name": "help", + "color": "0e8a16", + "aliases": [ + "help wanted", + "Patch Welcome", + "status: help wanted", + "Status: PR Welcome", + "Status: Ready for PR" + ] + }, + { + "name": "state: approved", + "color": "C3F7F7", + "description": "Approved to proceed.", + "aliases": ["approved"] + }, + { + "name": "state: blocked", + "color": "C3F7F7", + "description": "Something is blocking action.", + "aliases": ["blocked", "Status: Blocked", "status: blocked", "blocker"] + }, + { + "name": "state: pending", + "color": "C3F7F7", + "description": "Action is underway but awaiting other tasks or data before completion.", + "aliases": [ + "in progress", + "on hold", + "pending", + "Status: In Progress", + "Status: Review Needed", + "watchlist" + ] + }, + { + "name": "state: inactive", + "color": "C3F7F7", + "description": "No current action needed/possible; issue fixed, out of scope, or superseded.", + "aliases": [ + "duplicate", + "invalid", + "stale", + "Status: Abandoned", + "status: duplicate", + "status: wontfix", + "wont-fix", + "wontfix" + ] + }, + { + "name": "type: bug", + "color": "FA1DD4", + "description": "Something isn't working.", + "aliases": ["bug", "Type: Bug"] + }, + { + "name": "type: chore", + "color": "FA1DD4", + "description": "Routine tasks like conversions, reorganization, and maintenance work.", + "aliases": [ + "chore", + "dependabot", + "dependencies", + "dependency", + "greenkeeper", + "legal", + "maintenance", + "Type: Maintenance", + "type: maintenance" + ] + }, + { + "name": "type: discussion", + "color": "FA1DD4", + "description": "Questions, proposals and info that requires discussion.", + "aliases": [ + "discussion", + "feedback", + "help wanted", + "idea", + "Idea", + "Proposal", + "proposal", + "question", + "type: design", + "Type: Question", + "type: question" + ] + }, + { + "name": "type: docs", + "color": "FA1DD4", + "description": "Related to documentation and information.", + "aliases": [ + "doc", + "docs", + "document", + "documentation", + "documents", + "Type: Documentation", + "type: documentation", + "specification" + ] + }, + { + "name": "type: feature", + "color": "FA1DD4", + "description": "Brand new functionality, features, pages, workflows, endpoints, etc.", + "aliases": [ + "addition", + "enhancement", + "feature", + "type: enhancement", + "Type: Feature", + "performance" + ] + }, + { + "name": "type: fix", + "color": "FA1DD4", + "description": "Iterations on existing features or infrastructure.", + "aliases": [ + "fix", + "improvement", + "optimization", + "Type: Fix", + "bug" + ] + }, + { + "name": "type: refactor", + "color": "FA1DD4", + "description": "Code refactoring and restructuring.", + "aliases": [ + "refactor", + "Type: Refactoring", + "refactoring" + ] + }, + { + "name": "type: security", + "color": "FA1DD4", + "description": "Something is vulnerable or not secure.", + "aliases": ["security"] + }, + { + "name": "type: performance", + "color": "FA1DD4", + "description": "Improved performance." + }, + { + "name": "type: testing", + "color": "FA1DD4", + "description": "Related to testing.", + "aliases": ["test", "testing"] + }, + { + "name": "work: chaotic", + "color": "0288d1", + "description": "The situation is chaotic, novel practices used.", + "aliases": ["chaotic"] + }, + { + "name": "work: complex", + "color": "4fc3f7", + "description": "The situation is complex, emergent practices used.", + "aliases": ["complex"] + }, + { + "name": "work: complicated", + "color": "81d4fa", + "description": "The situation is complicated, good practices used.", + "aliases": ["complicated"] + }, + { + "name": "work: obvious", + "color": "e1f5fe", + "description": "The situation is obvious, best practices used.", + "aliases": ["obvious"] + }, + { + "name": "ci: lint", + "color": "F9D0C4", + "description": "Automatically fix the linters issues and make a commit", + "aliases": ["ci:lint"] + }, + { + "name": "ci: benchmark", + "color": "F9D0C4", + "description": "Runs benchmarks", + "aliases": ["ci:benchmark"] + }, + { + "name": "type: provider", + "color": "0075ca", + "description": "Updates provider.json configuration.", + "aliases": ["provider", "update: provider"] + }, + { + "name": "ci: test-jit", + "color": "F9D0C4", + "description": "Run all integration tests for the JIT optimized engine", + "aliases": ["ci:test-jit"] + }, + { + "name": "os: windows", + "color": "0078d4", + "description": "Windows-specific issue or feature.", + "aliases": ["windows", "win32", "platform: windows"] + }, + { + "name": "onlydust-wave", + "color": "FF6D00", + "description": "Related to OnlyDust Wave Hackathon" + }, + { + "name": "ci: build all targets", + "color": "F9D0C4", + "description": "Build release binaries for all target platforms in CI", + "aliases": ["ci:build-all-targets", "build-all-targets"] + }, + { + "name": "bounty", + "color": "F9C74F", + "description": "This issue has a bounty attached.", + "aliases": ["bounty:open"] + }, + { + "name": "bounty: $100", + "color": "F9C74F", + "description": "This issue has a $100 bounty attached.", + "aliases": ["bounty:100", "bounty: ๐Ÿ’ฐ $100"] + }, + { + "name": "bounty: $200", + "color": "F9C74F", + "description": "This issue has a $200 bounty attached.", + "aliases": ["bounty:200", "bounty: ๐Ÿ’ฐ $200"] + }, + { + "name": "bounty: $300", + "color": "F9C74F", + "description": "This issue has a $300 bounty attached.", + "aliases": ["bounty:300", "bounty: ๐Ÿ’ฐ $300"] + }, + { + "name": "bounty: $500", + "color": "F9C74F", + "description": "This issue has a $500 bounty attached.", + "aliases": ["bounty:500", "bounty: ๐Ÿ’ฐ $500"] + }, + { + "name": "bounty: $800", + "color": "F9C74F", + "description": "This issue has a $800 bounty attached.", + "aliases": ["bounty:800", "bounty: ๐Ÿ’ฐ $800"] + }, + { + "name": "bounty: $1300", + "color": "F9C74F", + "description": "This issue has a $1300 bounty attached.", + "aliases": ["bounty:1300", "bounty: ๐Ÿ’ฐ $1300"] + }, + { + "name": "bounty: $2100", + "color": "F9C74F", + "description": "This issue has a $2100 bounty attached.", + "aliases": ["bounty:2100", "bounty: ๐Ÿ’ฐ $2100"] + }, + { + "name": "bounty: $3400", + "color": "F9C74F", + "description": "This issue has a $3400 bounty attached.", + "aliases": ["bounty:3400", "bounty: ๐Ÿ’ฐ $3400"] + }, + { + "name": "bounty: $5500", + "color": "F9C74F", + "description": "This issue has a $5500 bounty attached.", + "aliases": ["bounty:5500", "bounty: ๐Ÿ’ฐ $5500"] + }, + { + "name": "bounty: claimed", + "color": "F9C74F", + "description": "This bounty has been claimed and is being worked on.", + "aliases": ["bounty:claimed"] + }, + { + "name": "bounty: rewarded", + "color": "F9C74F", + "description": "This bounty has been completed and the reward paid out.", + "aliases": ["bounty:rewarded", "bounty:earned"] + }, + { + "name": "severity: low", + "color": "FFEB3B", + "description": "Minor impact; does not significantly affect functionality.", + "aliases": ["severity:low"] + }, + { + "name": "severity: high", + "color": "FF9800", + "description": "Significant impact; core functionality is impaired.", + "aliases": ["severity:high"] + }, + { + "name": "severity: critical", + "color": "B71C1C", + "description": "Severe impact; system is unusable or data is at risk.", + "aliases": ["severity:critical", "critical"] + } +] diff --git a/.github/release-drafter.yml b/.github/release-drafter.yml new file mode 100644 index 0000000000000000000000000000000000000000..933354e054b22fab33337ce894884c96155efbc5 --- /dev/null +++ b/.github/release-drafter.yml @@ -0,0 +1,132 @@ +name-template: "v$RESOLVED_VERSION" +tag-template: "v$RESOLVED_VERSION" +categories: + - title: "๐Ÿš€ Features" + labels: + - "type: feature" + + - title: "๐Ÿ“ Documentation" + labels: + - "type: docs" + + - title: "๐Ÿ› Bug Fixes" + labels: + - "type: bug" + - "type: fix" + - "type: provider" + + - title: "๐Ÿš€ Performance" + labels: + - "type: performance" + + - title: "๐Ÿงฐ Maintenance" + labels: + - "type: chore" + - "type: testing" + - "type: refactor" + - "ratatui" + + - title: "๐Ÿ’ฅ Breaking Changes" + labels: + - "release: breaking" + + - title: "๐Ÿ”’ Security" + labels: + - "type: security" + +change-template: "- $TITLE @$AUTHOR (#$NUMBER)" +change-title-escapes: '\<*_&' # You can add # and @ to disable mentions, and add ` to disable code blocks. + +version-resolver: + major: + labels: + - "release: breaking" + minor: + labels: + - "type: feature" + # Add direct commit message pattern matching for features + commits: + - "feat:" + - "feature:" + patch: + labels: + - "type: fix" + - "type: performance" + - "type: chore" + - "type: docs" + - "type: bug" + - "type: testing" + - "type: refactor" + - "type: provider" + # Add direct commit message pattern matching for patches + commits: + - "fix:" + - "perf:" + - "chore:" + - "docs:" + - "refactor:" + default: patch + +exclude-labels: + - "release: skip changelog" + +template: | + ## Changes + + $CHANGES +autolabeler: + - label: "type: chore" + title: + - /chore/i + + - label: "type: refactor" + title: + - /refactor/i + - /refactoring/i + + - label: "type: fix" + title: + - /fix/i + - /hotfix/i + - /bugfix/i + - /patch/i + + - label: "type: feature" + title: + - /feat/i + - /feature/i + - /enhancement/i + - /new/i + + - label: "type: docs" + title: + - /doc/i + - /docs/i + + - label: "type: performance" + title: + - /perf/i + + - label: "ci: benchmark" + title: + - /perf/i + files: + - "/benches/" + + - label: "type: provider" + files: + - "provider.json" + + - label: "os: windows" + title: + - /windows/i + - /win32/i + - /winapi/i + - /powershell/i + - /scoop/i + - /chocolatey/i + - /winget/i + body: + - /windows[ -](only|specific|terminal|path|installer|build)/i + - /multiline paste.*windows/i + - /windows.*multiline paste/i diff --git a/.github/scripts/bounty/README.md b/.github/scripts/bounty/README.md new file mode 100644 index 0000000000000000000000000000000000000000..5794e530bfd83fff3bbef68ed914d160acbc58b2 --- /dev/null +++ b/.github/scripts/bounty/README.md @@ -0,0 +1,155 @@ +# Bounty Automation + +Automates the full lifecycle of issue bounties โ€” from label propagation when a PR is opened, through claiming when work begins, to rewarding when a PR is merged. + +## Flow + +``` +Issue created +โ””โ”€โ”€ maintainer adds bounty: $N label + โ”‚ + โ”œโ”€โ”€ parse-sync-generic-bounty.ts โ†’ adds generic bounty label + โ”‚ + โ–ผ +Issue assigned to contributor +โ””โ”€โ”€ parse-sync-claimed.ts โ†’ adds bounty: claimed + โ”‚ + โ–ผ +Contributor opens PR with "Closes #N" / "Fixes #N" / "Resolves #N" +โ””โ”€โ”€ parse-propagate-label.ts โ†’ copies bounty: $N to PR + โ†’ posts comment on issue: "PR #X opened by @author" + โ”‚ + โ–ผ +PR is merged +โ””โ”€โ”€ parse-mark-rewarded.ts โ†’ adds bounty: rewarded to PR + โ†’ adds bounty: rewarded to linked issue(s) + โ†’ removes bounty: claimed from linked issue(s) + โ”‚ + โ–ผ +Bounty lifecycle complete + +Removal path: +maintainer removes last bounty: $N label +โ””โ”€โ”€ parse-sync-generic-bounty.ts โ†’ removes generic bounty label +``` + +## Labels + +| Label | Applied to | Set by | +| -------------------------------- | ---------- | ----------------------------------------------------------- | +| `bounty: $100` โ€ฆ `bounty: $5500` | Issue | Maintainer (manually) | +| `bounty` | Issue | `parse-sync-generic-bounty.ts` on value label add/remove | +| `bounty: claimed` | Issue | `parse-sync-claimed.ts` on assignment | +| `bounty: rewarded` | Issue + PR | `parse-mark-rewarded.ts` on merge | + +Bounty values follow the Fibonacci sequence: **$100, $200, $300, $500, $800, $1300, $2100, $3400, $5500**. + +## Pipeline + +Each workflow job runs a single Node process (`run.ts`) that executes all three stages in memory โ€” no temp files, no shell piping: + +``` +parse โ†’ ParsedIntent (in memory) โ†’ plan โ†’ BatchPlan (in memory) โ†’ execute +``` + +**Stage 1 โ€” parse** (`parse-*.ts`): Reads the GitHub Actions event payload from `GITHUB_EVENT_PATH` and returns a `ParsedIntent` object. Pure โ€” makes no API calls. + +**Stage 2 โ€” plan** (`plan.ts`): Receives the `ParsedIntent`, fetches current labels only for targets not already known from the event payload, diffs desired vs actual state, and returns a `BatchPlan`. Minimises API calls by reusing label data already present in the event. + +**Stage 3 โ€” execute** (`execute.ts`): Receives the `BatchPlan` and applies all mutations. Label additions per target are sent as a single batched `POST /labels` call. Each removal is a separate `DELETE` (GitHub has no bulk-remove endpoint). Comments are posted last. + +This design means: +- The parse stage is trivially unit-testable (pure function, no mocks needed). +- The plan stage is testable with a minimal mock that only needs `getLabels`. +- The execute stage is testable with a mock that tracks calls โ€” no HTTP. +- API calls are minimised: additions are batched per target; label state already in the event payload is never re-fetched. + +## Scripts + +The workflow invokes a single orchestrator script (`run.ts`) per job. The parse, plan, and execute modules are called directly in the same Node process, passing objects in memory. + +### `run.ts` + +The entrypoint used by `bounty.yml`. Accepts a `--script` flag to select which parse module to run, then calls `plan` and `execute` in sequence โ€” all in-process. + +```sh +npx tsx .github/scripts/bounty/run.ts \ + --script \ + --repo \ + --token \ + [--pr | --issue ] +``` + +### `parse-propagate-label.ts` + +Triggered by: `pull_request` โ€” opened, edited, reopened. + +1. Parses the PR body for closing keywords (`closes`, `fixes`, `resolves`, case-insensitive). +2. Returns a `ParsedIntent` with a `labelCopies` field โ€” the plan stage fetches each linked issue's labels and copies any `bounty: $N` ones onto the PR. +3. Includes a comment mutation per linked issue (the plan stage drops it if the issue has no bounty labels). + +### `parse-sync-claimed.ts` + +Triggered by: `issues` โ€” assigned, unassigned. + +- **assigned**: returns `add: ["bounty: claimed"]` if the issue has a `bounty: $N` label. +- **unassigned**: returns `remove: ["bounty: claimed"]` only when no assignees remain. +- Issue labels are already in the event payload and supplied as `knownLabels` โ€” no extra fetch. + +### `parse-sync-generic-bounty.ts` + +Triggered by: `issues` โ€” labeled, unlabeled. + +Keeps the generic `bounty` label in sync with value labels. Inspects `event.label` (the label that just changed) and only acts when it matches `bounty: $`. + +- **labeled**: returns `add: ["bounty"]`. +- **unlabeled**: returns `remove: ["bounty"]` only when no value labels remain (guards against mid-tier-swap removal when a maintainer swaps one value label for another). +- Issue labels from the event are supplied as `knownLabels` โ€” no extra fetch. + +### `parse-mark-rewarded.ts` + +Triggered by: `pull_request_target` โ€” closed (merged only). + +1. Returns empty intent if the PR was not merged or has no `bounty: $N` label. +2. Returns `add: ["bounty: rewarded"]` for the PR (labels known from event, no fetch). +3. Parses the PR body for linked issues; returns `add: ["bounty: rewarded"], remove: ["bounty: claimed"]` for each (plan stage fetches their labels). + +Uses `pull_request_target` so the job has write access to issues and PRs from forks. + +### `plan.ts` + +Receives a `ParsedIntent` from the parse stage. Resolves `labelCopies` by fetching source issue labels. Fetches current labels for any target not already in `knownLabels`. Filters out no-op adds and removes. Returns a `BatchPlan`. + +### `execute.ts` + +Receives a `BatchPlan` from the plan stage. For each mutation: one batched `POST` for all additions, one `DELETE` per removal, one `POST` per comment. + +## Shared Module + +`github-api.ts` defines: +- Event payload types (`PullRequestEvent`, `IssuesEvent`) +- Pipeline types (`ParsedIntent`, `BatchPlan`, `TargetMutation`) +- The `GitHubApi` interface (injectable for testing) +- `GitHubRestApi` โ€” the production implementation using `node:https` + +All scripts import types from `github-api.ts` and accept a `GitHubApi` instance in their `run()` / `plan()` / `execute()` function signature, making every step independently mockable. + +## Tests + +Unit tests live alongside each script (`*.test.ts`) and use Node's built-in `node:test` runner. + +```sh +npm run test:bounty +``` + +- Parse tests: pure โ€” no mock needed, just call `parse()` with a synthetic event. +- Plan and execute tests: use a mock `GitHubApi` that tracks calls and returns preset label lists. +- The CLI entrypoint in each script (yargs parsing + `GITHUB_EVENT_PATH` read) is guarded behind an `import.meta.url` check so it does not execute on import. + +## Workflow Source + +`bounty.yml` is auto-generated from Rust source in `crates/forge_ci`. Do not edit it by hand โ€” modify `crates/forge_ci/src/workflows/bounty.rs` and regenerate with: + +```sh +cargo test -p forge_ci +``` diff --git a/.github/scripts/bounty/src/api.ts b/.github/scripts/bounty/src/api.ts new file mode 100644 index 0000000000000000000000000000000000000000..cb5447981e199265e367667b6c92a1daf62b1612 --- /dev/null +++ b/.github/scripts/bounty/src/api.ts @@ -0,0 +1,110 @@ +// GitHub API abstraction for v2 bounty sync. +// Uses the native `fetch` API (available in Node 18+). + +import type { Issue, Label, PullRequest } from "./types.js"; + +// --------------------------------------------------------------------------- +// Interface โ€” injectable for testing +// --------------------------------------------------------------------------- + +/// Minimal GitHub REST API surface needed by the bounty sync commands. +export interface GitHubApi { + /// Fetch a full issue by number. Throws on non-2xx. + getIssue(number: number): Promise; + /// Fetch a full pull request by number. Throws on non-2xx. + getPullRequest(number: number): Promise; + /// Fetch all open issues that have any label matching the given prefix. + /// Handles pagination automatically and excludes pull requests. + listIssuesWithLabelPrefix(prefix: string): Promise; + /// Add one or more labels to an issue or PR. Batched into a single request. + addLabels(target: number, labels: string[]): Promise; + /// Remove a single label from an issue or PR. + removeLabel(target: number, label: string): Promise; + /// Post a comment on an issue or PR. + addComment(target: number, body: string): Promise; +} + +// --------------------------------------------------------------------------- +// Production implementation +// --------------------------------------------------------------------------- + +/// Calls the real GitHub REST API v3 using `fetch`. +export class GitHubRestApi implements GitHubApi { + private readonly base: string; + private readonly headers: Record; + + constructor( + private readonly owner: string, + private readonly repo: string, + token: string + ) { + this.base = `https://api.github.com/repos/${owner}/${repo}`; + this.headers = { + Authorization: `Bearer ${token}`, + Accept: "application/vnd.github+json", + "X-GitHub-Api-Version": "2022-11-28", + "User-Agent": "bounty-bot/v2", + "Content-Type": "application/json", + }; + } + + private async request(method: string, path: string, body?: unknown): Promise { + const res = await fetch(`${this.base}${path}`, { + method, + headers: this.headers, + body: body !== undefined ? JSON.stringify(body) : undefined, + }); + + if (!res.ok) { + const text = await res.text().catch(() => ""); + throw new Error(`GitHub API ${method} ${path} โ†’ ${res.status}: ${text}`); + } + + // 204 No Content + if (res.status === 204) return {} as T; + + return res.json() as Promise; + } + + async getIssue(number: number): Promise { + return this.request("GET", `/issues/${number}`); + } + + async getPullRequest(number: number): Promise { + return this.request("GET", `/pulls/${number}`); + } + + async listIssuesWithLabelPrefix(prefix: string): Promise { + const results: Issue[] = []; + let page = 1; + while (true) { + const batch = await this.request( + "GET", + `/issues?state=open&per_page=100&page=${page}` + ); + if (batch.length === 0) break; + for (const issue of batch) { + // Exclude pull requests (GitHub returns them in /issues) + if (issue.pull_request !== undefined) continue; + if (issue.labels.some((l) => l.name.startsWith(prefix))) { + results.push(issue); + } + } + if (batch.length < 100) break; + page++; + } + return results; + } + + async addLabels(target: number, labels: string[]): Promise { + await this.request("POST", `/issues/${target}/labels`, { labels }); + } + + async removeLabel(target: number, label: string): Promise { + await this.request("DELETE", `/issues/${target}/labels/${encodeURIComponent(label)}`); + } + + async addComment(target: number, body: string): Promise { + await this.request("POST", `/issues/${target}/comments`, { body }); + } +} diff --git a/.github/scripts/bounty/src/rules.ts b/.github/scripts/bounty/src/rules.ts new file mode 100644 index 0000000000000000000000000000000000000000..72ba7980372d90e1bbbee3ce23e46fe3a01f8d1f --- /dev/null +++ b/.github/scripts/bounty/src/rules.ts @@ -0,0 +1,187 @@ +// Rules engine for v2 bounty sync. +// +// Both exported functions are pure: they take current state and return the +// minimal Patch needed to reconcile it with the desired state. +// No I/O, no side effects โ€” fully unit-testable without mocks. + +import type { IssueState, PrState, Patch, LabelOp, Issue } from "./types.js"; + +// --------------------------------------------------------------------------- +// Label name constants +// --------------------------------------------------------------------------- + +export const BOUNTY_GENERIC = "bounty"; +export const BOUNTY_CLAIMED = "bounty: claimed"; +export const BOUNTY_REWARDED = "bounty: rewarded"; + +/// Matches bounty value labels in both formats: +/// - new: "bounty: ๐Ÿ’ฐ $100", "bounty: ๐Ÿ’ฐ $5500" +/// - legacy: "bounty: $300", "bounty: $500" +export const VALUE_LABEL_RE = /^bounty: (?:๐Ÿ’ฐ )?\$\d/; + +// --------------------------------------------------------------------------- +// Helpers +// --------------------------------------------------------------------------- + +/// Returns true when the label name is a bounty value label. +export function isBountyValue(label: string): boolean { + return VALUE_LABEL_RE.test(label); +} + +/// Extract all issue numbers referenced by closing keywords in a PR body. +/// A new RegExp is created on each call to avoid stale `lastIndex` state. +export function linkedIssueNumbers(body: string | null): number[] { + const numbers: number[] = []; + const text = body ?? ""; + const re = /(?:closes?|fix(?:es)?|resolves?)\s+#(\d+)/gi; + let m: RegExpExecArray | null; + while ((m = re.exec(text)) !== null) { + numbers.push(parseInt(m[1]!, 10)); + } + return numbers; +} + +/// Build a LabelOp that only includes labels that actually need to change. +function diff( + target: number, + current: Set, + desired: Set, + comment?: string, + meta?: { title?: string; url?: string } +): LabelOp | null { + const add = [...desired].filter((l) => !current.has(l)); + const remove = [...current].filter((l) => !desired.has(l)); + if (add.length === 0 && remove.length === 0 && !comment) return null; + return { target, title: meta?.title, url: meta?.url, add, remove, comment }; +} + +// --------------------------------------------------------------------------- +// Issue rules +// --------------------------------------------------------------------------- + +/// Compute the desired label set for an issue given its full current state. +/// +/// Rules (all applied simultaneously, not sequentially): +/// +/// 1. **Generic bounty** โ€” `bounty` must be present iff any value label is present. +/// 2. **Claimed** โ€” `bounty: claimed` must be present iff the issue has a value +/// label AND at least one assignee. It must be absent otherwise. +/// 3. **Value labels and `bounty: rewarded`** โ€” these are set by maintainers or +/// the PR merge job and are left untouched by the issue sync. +/// 4. All non-bounty labels are preserved as-is. +export function computeIssuePatch({ issue, currentLabels }: IssueState): Patch { + const desired = new Set(currentLabels); + + const hasValueLabel = [...currentLabels].some(isBountyValue); + const isAssigned = issue.assignees.length > 0; + + // Rule 1 โ€” generic label mirrors value label presence. + if (hasValueLabel) { + desired.add(BOUNTY_GENERIC); + } else { + desired.delete(BOUNTY_GENERIC); + } + + // Rule 2 โ€” claimed mirrors assignment presence (only when bounty exists). + if (hasValueLabel && isAssigned) { + desired.add(BOUNTY_CLAIMED); + } else { + desired.delete(BOUNTY_CLAIMED); + } + + const op = diff(issue.number, currentLabels, desired, undefined, { + title: issue.title, + url: issue.html_url, + }); + return { ops: op ? [op] : [] }; +} + +// --------------------------------------------------------------------------- +// PR rules +// --------------------------------------------------------------------------- + +/// Compute the desired state for a PR and its linked issues. +/// +/// Rules: +/// +/// 1. **Label propagation** โ€” for each linked issue that has a value label, +/// copy that value label to the PR (if the PR doesn't already have it). +/// 2. **Rewarded on merge** โ€” if the PR is merged and has a value label: +/// a. Add `bounty: rewarded` to the PR. +/// b. Add `bounty: rewarded` to each linked issue. +/// c. Remove `bounty: claimed` from each linked issue. +/// 3. **Comment on propagation** โ€” when a value label is first added to the PR +/// via rule 1, post a comment on the source issue. +export function computePrPatch({ pr, currentLabels, linkedIssues }: PrState): Patch { + const ops: LabelOp[] = []; + + const prDesired = new Set(currentLabels); + + // Collect value labels to propagate from linked issues. + const labelsToPropagate: string[] = []; + for (const issue of linkedIssues) { + const issueValueLabels = issue.labels.map((l) => l.name).filter(isBountyValue); + for (const label of issueValueLabels) { + if (!prDesired.has(label)) { + labelsToPropagate.push(label); + prDesired.add(label); + } + } + } + + const prHasValueLabel = [...prDesired].some(isBountyValue); + + if (pr.merged && prHasValueLabel) { + // Rule 2 โ€” rewarded lifecycle. + prDesired.add(BOUNTY_REWARDED); + + const prOp = diff(pr.number, currentLabels, prDesired, undefined, { + title: pr.title, + url: pr.html_url, + }); + if (prOp) ops.push(prOp); + + // Update linked issues โ€” only those that already have a bounty value label. + for (const issue of linkedIssues) { + const issueCurrent = new Set(issue.labels.map((l) => l.name)); + const issueHasValue = [...issueCurrent].some(isBountyValue); + if (!issueHasValue) continue; + + const issueDesired = new Set(issueCurrent); + issueDesired.add(BOUNTY_REWARDED); + issueDesired.delete(BOUNTY_CLAIMED); + + const op = diff(issue.number, issueCurrent, issueDesired, undefined, { + title: issue.title, + url: issue.html_url, + }); + if (op) ops.push(op); + } + } else { + // Rule 1 โ€” label propagation (pre-merge). + const prOp = diff(pr.number, currentLabels, prDesired, undefined, { + title: pr.title, + url: pr.html_url, + }); + if (prOp) ops.push(prOp); + + // Rule 3 โ€” comment on each issue whose value label was newly propagated. + if (labelsToPropagate.length > 0) { + for (const issue of linkedIssues) { + const hadValueLabel = issue.labels.map((l) => l.name).some(isBountyValue); + if (hadValueLabel) { + ops.push({ + target: issue.number, + title: issue.title, + url: issue.html_url, + add: [], + remove: [], + comment: `PR [#${pr.number}](${pr.html_url}) has been opened for this bounty by @${pr.user.login}.`, + }); + } + } + } + } + + return { ops }; +} diff --git a/.github/scripts/bounty/src/sync-all-issues.ts b/.github/scripts/bounty/src/sync-all-issues.ts new file mode 100644 index 0000000000000000000000000000000000000000..5c0d84faa8ea2e523ba581ec7b285f13de3e2f48 --- /dev/null +++ b/.github/scripts/bounty/src/sync-all-issues.ts @@ -0,0 +1,77 @@ +#!/usr/bin/env tsx +// Syncs bounty labels across ALL open issues that carry any "bounty" label. +// +// Without --execute: fetches all matching issues, computes the combined patch, +// and prints a plan showing exactly what would change. No writes are made. +// +// With --execute: fetches, computes, and applies the patch for every issue. +// +// Usage: +// tsx sync-all-issues.ts --repo --token [--execute] + +import * as url from "url"; +import yargs from "yargs"; +import { hideBin } from "yargs/helpers"; +import { GitHubRestApi, type GitHubApi } from "./api.js"; +import { computeIssuePatch } from "./rules.js"; +import { applyPatch, printPlan, resolveToken } from "./sync-issue.js"; +import type { Patch } from "./types.js"; + +const BOUNTY_LABEL_PREFIX = "bounty"; + +export interface PlanAllIssuesInput { + api: GitHubApi; +} + +/// Fetches all open issues with any bounty label and computes the combined patch. +/// Makes no writes โ€” safe to call at any time. +export async function planAllIssues({ api }: PlanAllIssuesInput): Promise { + const issues = await api.listIssuesWithLabelPrefix(BOUNTY_LABEL_PREFIX); + const ops = issues.flatMap((issue) => { + const currentLabels = new Set(issue.labels.map((l) => l.name)); + return computeIssuePatch({ issue, currentLabels }).ops; + }); + return { ops }; +} + +/// Fetches, computes, and applies the label patch for all bounty issues. +/// Returns the patch that was applied. +export async function syncAllIssues({ api }: PlanAllIssuesInput): Promise { + const patch = await planAllIssues({ api }); + await applyPatch(patch, api); + return patch; +} + +// --------------------------------------------------------------------------- +// CLI entrypoint +// --------------------------------------------------------------------------- + +if (process.argv[1] === url.fileURLToPath(import.meta.url)) { + const argv = await yargs(hideBin(process.argv)) + .option("repo", { type: "string", demandOption: true, description: "owner/repo" }) + .option("token", { + type: "string", + description: "GitHub token (falls back to GITHUB_TOKEN env var or `gh auth token`)", + }) + .option("execute", { + type: "boolean", + default: false, + description: "Apply the patch. Without this flag only the plan is printed.", + }) + .strict() + .parseAsync(); + + const [owner, repo] = argv.repo.split("/") as [string, string]; + const token = resolveToken(argv.token); + const api = new GitHubRestApi(owner, repo, token); + + if (argv.execute) { + const patch = await syncAllIssues({ api }); + if (patch.ops.length === 0) { + console.log("All issues already in sync โ€” no changes needed."); + } + } else { + const patch = await planAllIssues({ api }); + printPlan(patch, "All bounty issues"); + } +} diff --git a/.github/scripts/bounty/src/sync-issue.ts b/.github/scripts/bounty/src/sync-issue.ts new file mode 100644 index 0000000000000000000000000000000000000000..802744592adcf2f4eb15d5596b1dcccabdacdef0 --- /dev/null +++ b/.github/scripts/bounty/src/sync-issue.ts @@ -0,0 +1,133 @@ +#!/usr/bin/env tsx +// Syncs all bounty labels on a single issue. +// +// Without --execute: fetches current state, computes the patch, and prints +// a plan showing exactly what would change. No writes are made. +// +// With --execute: fetches, computes, and applies the patch. +// +// Usage: +// tsx sync-issue.ts --issue --repo --token [--execute] + +import * as url from "url"; +import { execSync } from "child_process"; +import chalk from "chalk"; +import yargs from "yargs"; +import { hideBin } from "yargs/helpers"; +import { GitHubRestApi, type GitHubApi } from "./api.js"; +import { computeIssuePatch } from "./rules.js"; +import type { Patch } from "./types.js"; + +/// Resolves a GitHub token from the provided value, the GITHUB_TOKEN env var, +/// or by invoking `gh auth token` as a fallback. Exits with an error if none +/// can be found. +export function resolveToken(flag: string | undefined): string { + const token = flag || process.env["GITHUB_TOKEN"] || (() => { + try { + return execSync("gh auth token", { encoding: "utf8" }).trim(); + } catch { + return ""; + } + })(); + if (!token) { + console.error( + "Error: no GitHub token found. Pass --token, set GITHUB_TOKEN, or run `gh auth login`." + ); + process.exit(1); + } + return token; +} + +export interface PlanIssueInput { + issueNumber: number; + api: GitHubApi; +} + +/// Fetches the current issue state and computes the minimal label patch. +/// Makes no writes โ€” safe to call at any time. +export async function planIssue({ issueNumber, api }: PlanIssueInput): Promise { + const issue = await api.getIssue(issueNumber); + const currentLabels = new Set(issue.labels.map((l) => l.name)); + return computeIssuePatch({ issue, currentLabels }); +} + +/// Fetches, computes, and applies the label patch for a single issue. +/// Returns the patch that was applied (useful for tests and logging). +export async function syncIssue({ issueNumber, api }: PlanIssueInput): Promise { + const patch = await planIssue({ issueNumber, api }); + await applyPatch(patch, api); + return patch; +} + +/// Applies a Patch against the GitHub API. +/// Each target gets one batched addLabels call; each removal is individual. +export async function applyPatch(patch: Patch, api: GitHubApi): Promise { + for (const op of patch.ops) { + const ref = chalk.cyan(`#${op.target}`); + if (op.add.length > 0) { + await api.addLabels(op.target, op.add); + console.log(`${chalk.green("โœ”")} ${ref}: added [${chalk.green(op.add.join(", "))}]`); + } + for (const label of op.remove) { + await api.removeLabel(op.target, label); + console.log(`${chalk.red("โœ–")} ${ref}: removed "${chalk.red(label)}"`); + } + if (op.comment) { + await api.addComment(op.target, op.comment); + console.log(`${chalk.yellow("โœ‰")} ${ref}: posted comment`); + } + } +} + +/// Prints a human-readable plan to stdout without making any API calls. +export function printPlan(patch: Patch, subject: string): void { + if (patch.ops.length === 0) { + console.log(`${chalk.green("โœ”")} ${chalk.bold(subject)}: already in sync โ€” no changes needed.`); + return; + } + console.log(`${chalk.yellow("โ—")} ${chalk.bold(subject)}: plan (${chalk.bold(String(patch.ops.length))} target(s) to update)\n`); + for (const op of patch.ops) { + const title = op.title ? chalk.bold(` ${op.title}`) : ""; + const href = op.url ? `\n ${chalk.dim(chalk.blue(op.url))}` : ""; + console.log(` ${chalk.cyan(`#${op.target}`)}${title}${href}`); + if (op.add.length > 0) + console.log(` ${chalk.green("+")} add: ${chalk.green(op.add.join(", "))}`); + if (op.remove.length > 0) + console.log(` ${chalk.red("-")} remove: ${chalk.red(op.remove.join(", "))}`); + if (op.comment) + console.log(` ${chalk.yellow("~")} comment: ${chalk.dim(op.comment.slice(0, 80))}โ€ฆ`); + } + console.log(`\n${chalk.dim("Run with --execute to apply.")}`); +} + +// --------------------------------------------------------------------------- +// CLI entrypoint +// --------------------------------------------------------------------------- + +if (process.argv[1] === url.fileURLToPath(import.meta.url)) { + const argv = await yargs(hideBin(process.argv)) + .option("issue", { type: "number", demandOption: true, description: "Issue number" }) + .option("repo", { type: "string", demandOption: true, description: "owner/repo" }) + .option("token", { type: "string", description: "GitHub token (falls back to GITHUB_TOKEN env var or `gh auth token`)" }) + .option("execute", { + type: "boolean", + default: false, + description: "Apply the patch. Without this flag only the plan is printed.", + }) + .strict() + .parseAsync(); + + const [owner, repo] = argv.repo.split("/") as [string, string]; + const token = resolveToken(argv.token); + const api = new GitHubRestApi(owner, repo, token); + + if (argv.execute) { + const patch = await syncIssue({ issueNumber: argv.issue, api }); + if (patch.ops.length === 0) { + console.log(`Issue #${argv.issue}: already in sync, no changes needed.`); + } + } else { + const patch = await planIssue({ issueNumber: argv.issue, api }); + printPlan(patch, `Issue #${argv.issue}`); + } +} diff --git a/.github/scripts/bounty/src/sync-pr.ts b/.github/scripts/bounty/src/sync-pr.ts new file mode 100644 index 0000000000000000000000000000000000000000..200a2839ac26e3600b00450426b921deb1a261e1 --- /dev/null +++ b/.github/scripts/bounty/src/sync-pr.ts @@ -0,0 +1,82 @@ +#!/usr/bin/env tsx +// Syncs bounty labels on a PR (and its linked issues). +// +// Without --execute: fetches current state, computes the patch, and prints +// a plan showing exactly what would change. No writes are made. +// +// With --execute: fetches, computes, and applies the patch. +// +// Usage: +// tsx sync-pr.ts --pr --repo --token [--execute] + +import * as url from "url"; +import yargs from "yargs"; +import { hideBin } from "yargs/helpers"; +import { GitHubRestApi, type GitHubApi } from "./api.js"; +import { computePrPatch, linkedIssueNumbers } from "./rules.js"; +import { applyPatch, printPlan, resolveToken } from "./sync-issue.js"; +import type { Patch } from "./types.js"; + +export interface PlanPrInput { + prNumber: number; + api: GitHubApi; +} + +/// Fetches the current PR + linked issue state and computes the minimal label patch. +/// Makes no writes โ€” safe to call at any time. +export async function planPr({ prNumber, api }: PlanPrInput): Promise { + const pr = await api.getPullRequest(prNumber); + const currentLabels = new Set(pr.labels.map((l) => l.name)); + const issueNumbers = linkedIssueNumbers(pr.body); + + const linkedIssues = await Promise.all( + issueNumbers.map((n) => + api.getIssue(n).catch((err) => { + console.warn(`Could not fetch linked issue #${n}: ${String(err)}, skipping.`); + return null; + }) + ) + ).then((results) => results.filter((i): i is NonNullable => i !== null)); + + return computePrPatch({ pr, currentLabels, linkedIssues }); +} + +/// Fetches, computes, and applies the label patch for a PR and its linked issues. +/// Returns the patch that was applied (useful for tests and logging). +export async function syncPr({ prNumber, api }: PlanPrInput): Promise { + const patch = await planPr({ prNumber, api }); + await applyPatch(patch, api); + return patch; +} + +// --------------------------------------------------------------------------- +// CLI entrypoint +// --------------------------------------------------------------------------- + +if (process.argv[1] === url.fileURLToPath(import.meta.url)) { + const argv = await yargs(hideBin(process.argv)) + .option("pr", { type: "number", demandOption: true, description: "PR number" }) + .option("repo", { type: "string", demandOption: true, description: "owner/repo" }) + .option("token", { type: "string", description: "GitHub token (falls back to GITHUB_TOKEN env var or `gh auth token`)" }) + .option("execute", { + type: "boolean", + default: false, + description: "Apply the patch. Without this flag only the plan is printed.", + }) + .strict() + .parseAsync(); + + const [owner, repo] = argv.repo.split("/") as [string, string]; + const token = resolveToken(argv.token); + const api = new GitHubRestApi(owner, repo, token); + + if (argv.execute) { + const patch = await syncPr({ prNumber: argv.pr, api }); + if (patch.ops.length === 0) { + console.log(`PR #${argv.pr}: already in sync, no changes needed.`); + } + } else { + const patch = await planPr({ prNumber: argv.pr, api }); + printPlan(patch, `PR #${argv.pr}`); + } +} diff --git a/.github/scripts/bounty/src/types.ts b/.github/scripts/bounty/src/types.ts new file mode 100644 index 0000000000000000000000000000000000000000..a519235496eb5377f99a072caea4c95bc7af98fb --- /dev/null +++ b/.github/scripts/bounty/src/types.ts @@ -0,0 +1,82 @@ +// Core domain types for v2 bounty sync. +// +// Design: both sync commands work by fetching the complete current state of an +// issue or PR from GitHub, deriving the *desired* label set from the rules, and +// computing a minimal patch (add / remove) to reconcile the two. + +// --------------------------------------------------------------------------- +// GitHub resource types (as returned by the REST API) +// --------------------------------------------------------------------------- + +export interface Label { + name: string; +} + +export interface User { + login: string; +} + +/// Full issue as returned by GET /repos/:owner/:repo/issues/:number +export interface Issue { + number: number; + title: string; + html_url: string; + state: "open" | "closed"; + labels: Label[]; + assignees: User[]; + pull_request?: unknown; // present when the issue is actually a PR +} + +/// Full pull request as returned by GET /repos/:owner/:repo/pulls/:number +export interface PullRequest { + number: number; + title: string; + state: "open" | "closed"; + merged: boolean; + body: string | null; + labels: Label[]; + user: User; + html_url: string; +} + +// --------------------------------------------------------------------------- +// Derived state used by the rules engine +// --------------------------------------------------------------------------- + +/// Everything the rules engine needs to know about an issue. +export interface IssueState { + issue: Issue; + /// Current label names on the issue. + currentLabels: Set; +} + +/// Everything the rules engine needs to know about a PR and its linked issues. +export interface PrState { + pr: PullRequest; + /// Current label names on the PR. + currentLabels: Set; + /// Full state of each issue linked via "closes/fixes/resolves #N" in the PR body. + linkedIssues: Issue[]; +} + +// --------------------------------------------------------------------------- +// Patch types โ€” the minimal diff to apply +// --------------------------------------------------------------------------- + +/// A label operation on a single target (issue or PR number). +export interface LabelOp { + target: number; + /// Human-readable title of the issue or PR, for display in plan output. + title?: string; + /// URL of the issue or PR, for display in plan output. + url?: string; + add: string[]; + remove: string[]; + /// Optional comment to post on the target after label ops. + comment?: string; +} + +/// The complete set of operations to apply. +export interface Patch { + ops: LabelOp[]; +} diff --git a/.github/scripts/bounty/tests/rules.test.ts b/.github/scripts/bounty/tests/rules.test.ts new file mode 100644 index 0000000000000000000000000000000000000000..7d985df87930f95b2f0af7e86822dbfd771d34df --- /dev/null +++ b/.github/scripts/bounty/tests/rules.test.ts @@ -0,0 +1,369 @@ +import { describe, it } from "node:test"; +import assert from "node:assert/strict"; +import { + computeIssuePatch, + computePrPatch, + linkedIssueNumbers, + BOUNTY_GENERIC, + BOUNTY_CLAIMED, + BOUNTY_REWARDED, +} from "../src/rules.js"; +import type { Issue, PullRequest, IssueState, PrState } from "../src/types.js"; + +// --------------------------------------------------------------------------- +// Fixtures +// --------------------------------------------------------------------------- + +function makeIssue(overrides: Partial & { number: number }): Issue { + return { + title: "Test issue", + html_url: `https://github.com/owner/repo/issues/${overrides.number}`, + state: "open", + labels: [], + assignees: [], + ...overrides, + }; +} + +function makePr(overrides: Partial & { number: number }): PullRequest { + return { + title: "Test PR", + state: "open", + merged: false, + body: null, + labels: [], + user: { login: "contributor" }, + html_url: `https://github.com/owner/repo/pull/${overrides.number}`, + ...overrides, + }; +} + +function labelNames(...names: string[]): { name: string }[] { + return names.map((name) => ({ name })); +} + +function issueState(issue: Issue): IssueState { + return { issue, currentLabels: new Set(issue.labels.map((l) => l.name)) }; +} + +// --------------------------------------------------------------------------- +// computeIssuePatch โ€” issue rules +// --------------------------------------------------------------------------- + +describe("computeIssuePatch", () => { + describe("no bounty label", () => { + it("produces no patch when issue has no bounty labels at all", () => { + const issue = makeIssue({ number: 1, labels: labelNames("type: bug") }); + const patch = computeIssuePatch(issueState(issue)); + assert.deepEqual(patch.ops, []); + }); + + it("removes generic bounty label when no value label present", () => { + const issue = makeIssue({ number: 1, labels: labelNames(BOUNTY_GENERIC) }); + const patch = computeIssuePatch(issueState(issue)); + assert.equal(patch.ops.length, 1); + assert.deepEqual(patch.ops[0]!.remove, [BOUNTY_GENERIC]); + assert.deepEqual(patch.ops[0]!.add, []); + }); + + it("removes claimed label when no value label present", () => { + const issue = makeIssue({ + number: 1, + labels: labelNames(BOUNTY_CLAIMED), + assignees: [{ login: "alice" }], + }); + const patch = computeIssuePatch(issueState(issue)); + assert.equal(patch.ops.length, 1); + assert.deepEqual(patch.ops[0]!.remove, [BOUNTY_CLAIMED]); + }); + }); + + describe("has value label, no assignees", () => { + it("adds generic label when missing", () => { + const issue = makeIssue({ number: 1, labels: labelNames("bounty: ๐Ÿ’ฐ $500") }); + const patch = computeIssuePatch(issueState(issue)); + assert.equal(patch.ops.length, 1); + assert.deepEqual(patch.ops[0]!.add, [BOUNTY_GENERIC]); + assert.deepEqual(patch.ops[0]!.remove, []); + }); + + it("recognises legacy label format (no emoji) as a value label", () => { + const issue = makeIssue({ number: 1, labels: labelNames("bounty: $300") }); + const patch = computeIssuePatch(issueState(issue)); + assert.equal(patch.ops.length, 1); + assert.deepEqual(patch.ops[0]!.add, [BOUNTY_GENERIC]); + assert.deepEqual(patch.ops[0]!.remove, []); + }); + + it("produces no patch when generic already present and no assignees", () => { + const issue = makeIssue({ + number: 1, + labels: labelNames("bounty: ๐Ÿ’ฐ $500", BOUNTY_GENERIC), + }); + const patch = computeIssuePatch(issueState(issue)); + assert.deepEqual(patch.ops, []); + }); + + it("removes claimed when no assignees", () => { + const issue = makeIssue({ + number: 1, + labels: labelNames("bounty: ๐Ÿ’ฐ $500", BOUNTY_GENERIC, BOUNTY_CLAIMED), + assignees: [], + }); + const patch = computeIssuePatch(issueState(issue)); + assert.equal(patch.ops.length, 1); + assert.deepEqual(patch.ops[0]!.remove, [BOUNTY_CLAIMED]); + assert.deepEqual(patch.ops[0]!.add, []); + }); + }); + + describe("has value label and assignees", () => { + it("adds both generic and claimed when both missing", () => { + const issue = makeIssue({ + number: 1, + labels: labelNames("bounty: ๐Ÿ’ฐ $300"), + assignees: [{ login: "alice" }], + }); + const patch = computeIssuePatch(issueState(issue)); + assert.equal(patch.ops.length, 1); + assert.deepEqual(patch.ops[0]!.add.sort(), [BOUNTY_CLAIMED, BOUNTY_GENERIC].sort()); + assert.deepEqual(patch.ops[0]!.remove, []); + }); + + it("produces no patch when generic and claimed both already present", () => { + const issue = makeIssue({ + number: 1, + labels: labelNames("bounty: ๐Ÿ’ฐ $300", BOUNTY_GENERIC, BOUNTY_CLAIMED), + assignees: [{ login: "alice" }], + }); + const patch = computeIssuePatch(issueState(issue)); + assert.deepEqual(patch.ops, []); + }); + + it("preserves non-bounty labels untouched", () => { + const issue = makeIssue({ + number: 1, + labels: labelNames("type: bug", "bounty: ๐Ÿ’ฐ $100", BOUNTY_GENERIC, BOUNTY_CLAIMED), + assignees: [{ login: "alice" }], + }); + const patch = computeIssuePatch(issueState(issue)); + assert.deepEqual(patch.ops, []); + }); + }); + + describe("rewarded state", () => { + it("leaves rewarded label alone (owned by PR sync)", () => { + const issue = makeIssue({ + number: 1, + labels: labelNames("bounty: ๐Ÿ’ฐ $500", BOUNTY_GENERIC, BOUNTY_REWARDED), + assignees: [], + }); + const patch = computeIssuePatch(issueState(issue)); + // rewarded is not touched; only generic should be kept since value label present + assert.deepEqual(patch.ops, []); + }); + }); +}); + +// --------------------------------------------------------------------------- +// linkedIssueNumbers +// --------------------------------------------------------------------------- + +describe("linkedIssueNumbers", () => { + it("returns empty array for null body", () => { + assert.deepEqual(linkedIssueNumbers(null), []); + }); + + it("parses closes/fixes/resolves keywords", () => { + const body = "Closes #10\nFixes #20\nResolves #30"; + assert.deepEqual(linkedIssueNumbers(body), [10, 20, 30]); + }); + + it("is case-insensitive", () => { + const body = "CLOSES #5 fixes #6 RESOLVES #7"; + assert.deepEqual(linkedIssueNumbers(body), [5, 6, 7]); + }); + + it("handles plural forms: close/fix/resolve", () => { + const body = "close #1 fix #2 resolve #3"; + assert.deepEqual(linkedIssueNumbers(body), [1, 2, 3]); + }); + + it("returns empty when no keywords present", () => { + assert.deepEqual(linkedIssueNumbers("See issue #99 for context."), []); + }); +}); + +// --------------------------------------------------------------------------- +// computePrPatch โ€” PR rules +// --------------------------------------------------------------------------- + +describe("computePrPatch", () => { + describe("label propagation (pre-merge)", () => { + it("produces no patch when PR body has no linked issues", () => { + const pr = makePr({ number: 10, body: "No references here." }); + const patch = computePrPatch({ pr, currentLabels: new Set(), linkedIssues: [] }); + assert.deepEqual(patch.ops, []); + }); + + it("produces no patch when linked issues have no bounty labels", () => { + const pr = makePr({ number: 10, body: "Closes #1" }); + const issue = makeIssue({ number: 1, labels: labelNames("type: bug") }); + const patch = computePrPatch({ pr, currentLabels: new Set(), linkedIssues: [issue] }); + assert.deepEqual(patch.ops, []); + }); + + it("copies bounty value label from linked issue to PR", () => { + const pr = makePr({ number: 10, body: "Closes #1" }); + const issue = makeIssue({ number: 1, labels: labelNames("bounty: ๐Ÿ’ฐ $500") }); + const patch = computePrPatch({ pr, currentLabels: new Set(), linkedIssues: [issue] }); + + const prOp = patch.ops.find((o) => o.target === 10); + assert.ok(prOp); + assert.deepEqual(prOp.add, ["bounty: ๐Ÿ’ฐ $500"]); + assert.deepEqual(prOp.remove, []); + }); + + it("does not duplicate a value label already on the PR", () => { + const pr = makePr({ number: 10, body: "Closes #1" }); + const issue = makeIssue({ number: 1, labels: labelNames("bounty: ๐Ÿ’ฐ $500") }); + const patch = computePrPatch({ + pr, + currentLabels: new Set(["bounty: ๐Ÿ’ฐ $500"]), + linkedIssues: [issue], + }); + assert.deepEqual(patch.ops, []); + }); + + it("posts a comment on the linked issue when label is propagated", () => { + const pr = makePr({ number: 10, body: "Closes #1" }); + const issue = makeIssue({ number: 1, labels: labelNames("bounty: ๐Ÿ’ฐ $200") }); + const patch = computePrPatch({ pr, currentLabels: new Set(), linkedIssues: [issue] }); + + const issueOp = patch.ops.find((o) => o.target === 1); + assert.ok(issueOp); + assert.ok(issueOp.comment?.includes("PR")); + assert.ok(issueOp.comment?.includes("#10")); + assert.ok(issueOp.comment?.includes("contributor")); + }); + + it("does not post a comment when label was already on the PR", () => { + const pr = makePr({ number: 10, body: "Closes #1" }); + const issue = makeIssue({ number: 1, labels: labelNames("bounty: ๐Ÿ’ฐ $200") }); + const patch = computePrPatch({ + pr, + currentLabels: new Set(["bounty: ๐Ÿ’ฐ $200"]), + linkedIssues: [issue], + }); + const issueOp = patch.ops.find((o) => o.target === 1); + assert.ok(!issueOp); + }); + + it("merges value labels from multiple linked issues onto the PR", () => { + const pr = makePr({ number: 10, body: "Closes #1\nCloses #2" }); + const issue1 = makeIssue({ number: 1, labels: labelNames("bounty: ๐Ÿ’ฐ $100") }); + const issue2 = makeIssue({ number: 2, labels: labelNames("bounty: ๐Ÿ’ฐ $200") }); + const patch = computePrPatch({ + pr, + currentLabels: new Set(), + linkedIssues: [issue1, issue2], + }); + + const prOp = patch.ops.find((o) => o.target === 10); + assert.ok(prOp); + assert.deepEqual(prOp.add.sort(), ["bounty: ๐Ÿ’ฐ $100", "bounty: ๐Ÿ’ฐ $200"].sort()); + }); + }); + + describe("rewarded on merge", () => { + it("produces no patch when PR was closed but not merged", () => { + const pr = makePr({ number: 10, state: "closed", merged: false, body: "Closes #1" }); + const issue = makeIssue({ number: 1, labels: labelNames("bounty: ๐Ÿ’ฐ $500") }); + const patch = computePrPatch({ + pr, + currentLabels: new Set(["bounty: ๐Ÿ’ฐ $500"]), + linkedIssues: [issue], + }); + const prOp = patch.ops.find((o) => o.target === 10); + assert.ok(!prOp?.add.includes(BOUNTY_REWARDED)); + }); + + it("adds rewarded to PR on merge", () => { + const pr = makePr({ + number: 10, + merged: true, + body: "Closes #1", + labels: labelNames("bounty: ๐Ÿ’ฐ $500"), + }); + const issue = makeIssue({ number: 1, labels: labelNames("bounty: ๐Ÿ’ฐ $500") }); + const patch = computePrPatch({ + pr, + currentLabels: new Set(["bounty: ๐Ÿ’ฐ $500"]), + linkedIssues: [issue], + }); + + const prOp = patch.ops.find((o) => o.target === 10); + assert.ok(prOp); + assert.ok(prOp.add.includes(BOUNTY_REWARDED)); + }); + + it("adds rewarded to linked issue on merge", () => { + const pr = makePr({ + number: 10, + merged: true, + body: "Closes #1", + labels: labelNames("bounty: ๐Ÿ’ฐ $500"), + }); + const issue = makeIssue({ number: 1, labels: labelNames("bounty: ๐Ÿ’ฐ $500", BOUNTY_CLAIMED) }); + const patch = computePrPatch({ + pr, + currentLabels: new Set(["bounty: ๐Ÿ’ฐ $500"]), + linkedIssues: [issue], + }); + + const issueOp = patch.ops.find((o) => o.target === 1); + assert.ok(issueOp); + assert.ok(issueOp.add.includes(BOUNTY_REWARDED)); + assert.ok(issueOp.remove.includes(BOUNTY_CLAIMED)); + }); + + it("does not add rewarded again when already present", () => { + const pr = makePr({ + number: 10, + merged: true, + body: "Closes #1", + labels: labelNames("bounty: ๐Ÿ’ฐ $500", BOUNTY_REWARDED), + }); + const issue = makeIssue({ + number: 1, + labels: labelNames("bounty: ๐Ÿ’ฐ $500", BOUNTY_REWARDED), + }); + const patch = computePrPatch({ + pr, + currentLabels: new Set(["bounty: ๐Ÿ’ฐ $500", BOUNTY_REWARDED]), + linkedIssues: [issue], + }); + assert.deepEqual(patch.ops, []); + }); + + it("skips issues with no bounty label when merged PR has a bounty", () => { + const pr = makePr({ + number: 10, + merged: true, + body: "Closes #1\nCloses #2", + labels: labelNames("bounty: ๐Ÿ’ฐ $500"), + }); + const issue1 = makeIssue({ number: 1, labels: labelNames("bounty: ๐Ÿ’ฐ $500") }); + const issue2 = makeIssue({ number: 2, labels: labelNames("type: bug") }); + const patch = computePrPatch({ + pr, + currentLabels: new Set(["bounty: ๐Ÿ’ฐ $500"]), + linkedIssues: [issue1, issue2], + }); + + // issue2 has no bounty so its desired state equals current: no op + const issue2Op = patch.ops.find((o) => o.target === 2); + assert.ok(!issue2Op); + }); + }); +}); diff --git a/.github/scripts/bounty/tests/sync-all-issues.test.ts b/.github/scripts/bounty/tests/sync-all-issues.test.ts new file mode 100644 index 0000000000000000000000000000000000000000..fdb2e6fd80a10e9b3aeb6c81dba0b85e6d910685 --- /dev/null +++ b/.github/scripts/bounty/tests/sync-all-issues.test.ts @@ -0,0 +1,138 @@ +import { describe, it } from "node:test"; +import assert from "node:assert/strict"; +import { planAllIssues, syncAllIssues } from "../src/sync-all-issues.js"; +import type { GitHubApi } from "../src/api.js"; +import type { Issue, PullRequest } from "../src/types.js"; + +// --------------------------------------------------------------------------- +// Mock API +// --------------------------------------------------------------------------- + +function makeMockApi(issues: Issue[]): GitHubApi & { + added: Map; + removed: Map; +} { + const added = new Map(); + const removed = new Map(); + + return { + added, + removed, + async getIssue(number) { + const found = issues.find((i) => i.number === number); + if (!found) throw new Error(`Issue #${number} not found in mock`); + return found; + }, + async getPullRequest(): Promise { + throw new Error("not used"); + }, + async listIssuesWithLabelPrefix(_prefix) { + // Return only real issues (not PRs โ€” pull_request field absent) + return issues.filter((i) => i.pull_request === undefined); + }, + async addLabels(target, labels) { + if (!added.has(target)) added.set(target, []); + added.get(target)!.push(labels); + }, + async removeLabel(target, label) { + if (!removed.has(target)) removed.set(target, []); + removed.get(target)!.push(label); + }, + async addComment() {}, + }; +} + +function makeIssue(overrides: Partial & { number: number }): Issue { + return { title: "Test issue", html_url: `https://github.com/owner/repo/issues/${overrides.number}`, state: "open", labels: [], assignees: [], ...overrides }; +} + +function labelNames(...names: string[]): { name: string }[] { + return names.map((name) => ({ name })); +} + +// --------------------------------------------------------------------------- +// Tests +// --------------------------------------------------------------------------- + +describe("planAllIssues", () => { + it("returns empty patch when no issues need changes", async () => { + const issues = [ + makeIssue({ number: 1, labels: labelNames("bounty: $100", "bounty") }), + makeIssue({ number: 2, labels: labelNames("type: bug") }), + ]; + const api = makeMockApi(issues); + const patch = await planAllIssues({ api }); + assert.deepEqual(patch.ops, []); + }); + + it("plans adds for issues missing the generic bounty label", async () => { + const issues = [ + makeIssue({ number: 1, labels: labelNames("bounty: $100") }), + makeIssue({ number: 2, labels: labelNames("bounty: $200") }), + ]; + const api = makeMockApi(issues); + const patch = await planAllIssues({ api }); + assert.equal(patch.ops.length, 2); + assert.ok(patch.ops.every((op) => op.add.includes("bounty"))); + }); + + it("plans removal of stale claimed label when issue is unassigned", async () => { + const issues = [ + makeIssue({ + number: 3, + labels: labelNames("bounty: $500", "bounty", "bounty: claimed"), + assignees: [], + }), + ]; + const api = makeMockApi(issues); + const patch = await planAllIssues({ api }); + assert.equal(patch.ops.length, 1); + assert.deepEqual(patch.ops[0]!.remove, ["bounty: claimed"]); + }); + + it("collects ops for multiple issues with different needs", async () => { + const issues = [ + makeIssue({ number: 1, labels: labelNames("bounty: $100") }), // missing generic + makeIssue({ number: 2, labels: labelNames("bounty: $200", "bounty") }), // already in sync + makeIssue({ + number: 3, + labels: labelNames("bounty: $300", "bounty", "bounty: claimed"), + assignees: [], + }), // stale claimed + ]; + const api = makeMockApi(issues); + const patch = await planAllIssues({ api }); + assert.equal(patch.ops.length, 2); + const targets = patch.ops.map((op) => op.target).sort(); + assert.deepEqual(targets, [1, 3]); + }); +}); + +describe("syncAllIssues", () => { + it("applies adds and removals across all issues", async () => { + const issues = [ + makeIssue({ number: 1, labels: labelNames("bounty: $100") }), + makeIssue({ + number: 2, + labels: labelNames("bounty: $200", "bounty", "bounty: claimed"), + assignees: [], + }), + ]; + const api = makeMockApi(issues); + await syncAllIssues({ api }); + + assert.ok(api.added.get(1)?.[0]?.includes("bounty")); + assert.deepEqual(api.removed.get(2), ["bounty: claimed"]); + }); + + it("returns empty patch when all issues are already in sync", async () => { + const issues = [ + makeIssue({ number: 1, labels: labelNames("bounty: $100", "bounty") }), + ]; + const api = makeMockApi(issues); + const patch = await syncAllIssues({ api }); + assert.deepEqual(patch.ops, []); + assert.equal(api.added.size, 0); + assert.equal(api.removed.size, 0); + }); +}); diff --git a/.github/scripts/bounty/tests/sync-issue.test.ts b/.github/scripts/bounty/tests/sync-issue.test.ts new file mode 100644 index 0000000000000000000000000000000000000000..b864d67f69cf03cb2aa09b76e1fe9e6928cb542e --- /dev/null +++ b/.github/scripts/bounty/tests/sync-issue.test.ts @@ -0,0 +1,144 @@ +import { describe, it } from "node:test"; +import assert from "node:assert/strict"; +import { syncIssue } from "../src/sync-issue.js"; +import type { GitHubApi } from "../src/api.js"; +import type { Issue, PullRequest } from "../src/types.js"; + +// --------------------------------------------------------------------------- +// Mock API +// --------------------------------------------------------------------------- + +function makeMockApi(issue: Issue): GitHubApi & { + added: string[][]; + removed: string[]; + comments: string[]; +} { + const added: string[][] = []; + const removed: string[] = []; + const comments: string[] = []; + + return { + added, + removed, + comments, + async getIssue() { + return issue; + }, + async getPullRequest(): Promise { + throw new Error("not used"); + }, + async listIssuesWithLabelPrefix(): Promise { + throw new Error("not used"); + }, + async addLabels(_target, labels) { + added.push(labels); + }, + async removeLabel(_target, label) { + removed.push(label); + }, + async addComment(_target, body) { + comments.push(body); + }, + }; +} + +function makeIssue(overrides: Partial & { number: number }): Issue { + return { + title: "Test issue", + html_url: `https://github.com/owner/repo/issues/${overrides.number}`, + state: "open", + labels: [], + assignees: [], + ...overrides, + }; +} + +function labelNames(...names: string[]): { name: string }[] { + return names.map((name) => ({ name })); +} + +// --------------------------------------------------------------------------- +// Tests +// --------------------------------------------------------------------------- + +describe("syncIssue", () => { + it("returns empty patch and makes no API calls when issue is already in sync", async () => { + const issue = makeIssue({ + number: 42, + labels: labelNames("bounty: ๐Ÿ’ฐ $500", "bounty", "bounty: claimed"), + assignees: [{ login: "alice" }], + }); + const api = makeMockApi(issue); + const patch = await syncIssue({ issueNumber: 42, api }); + + assert.deepEqual(patch.ops, []); + assert.deepEqual(api.added, []); + assert.deepEqual(api.removed, []); + }); + + it("adds generic bounty label when value label is present but generic is missing", async () => { + const issue = makeIssue({ + number: 42, + labels: labelNames("bounty: ๐Ÿ’ฐ $300"), + }); + const api = makeMockApi(issue); + const patch = await syncIssue({ issueNumber: 42, api }); + + assert.equal(patch.ops.length, 1); + assert.deepEqual(api.added, [["bounty"]]); + assert.deepEqual(api.removed, []); + }); + + it("adds both generic and claimed when value label present and issue assigned", async () => { + const issue = makeIssue({ + number: 42, + labels: labelNames("bounty: ๐Ÿ’ฐ $800"), + assignees: [{ login: "bob" }], + }); + const api = makeMockApi(issue); + await syncIssue({ issueNumber: 42, api }); + + assert.equal(api.added.length, 1); + assert.deepEqual(api.added[0]!.sort(), ["bounty", "bounty: claimed"].sort()); + }); + + it("removes claimed when last assignee is removed", async () => { + const issue = makeIssue({ + number: 42, + labels: labelNames("bounty: ๐Ÿ’ฐ $500", "bounty", "bounty: claimed"), + assignees: [], + }); + const api = makeMockApi(issue); + await syncIssue({ issueNumber: 42, api }); + + assert.deepEqual(api.removed, ["bounty: claimed"]); + assert.deepEqual(api.added, []); + }); + + it("removes both generic and claimed when value label is removed", async () => { + const issue = makeIssue({ + number: 42, + labels: labelNames("bounty", "bounty: claimed"), + assignees: [{ login: "alice" }], + }); + const api = makeMockApi(issue); + await syncIssue({ issueNumber: 42, api }); + + assert.deepEqual(api.removed.sort(), ["bounty", "bounty: claimed"].sort()); + assert.deepEqual(api.added, []); + }); + + it("does a single batched addLabels call for multiple adds", async () => { + const issue = makeIssue({ + number: 42, + labels: labelNames("bounty: ๐Ÿ’ฐ $100"), + assignees: [{ login: "carol" }], + }); + const api = makeMockApi(issue); + await syncIssue({ issueNumber: 42, api }); + + // One call, two labels + assert.equal(api.added.length, 1); + assert.equal(api.added[0]!.length, 2); + }); +}); diff --git a/.github/scripts/bounty/tests/sync-pr.test.ts b/.github/scripts/bounty/tests/sync-pr.test.ts new file mode 100644 index 0000000000000000000000000000000000000000..d17e975d5ff4c30fb2fd68aefb7435edb0b86ec4 --- /dev/null +++ b/.github/scripts/bounty/tests/sync-pr.test.ts @@ -0,0 +1,241 @@ +import { describe, it } from "node:test"; +import assert from "node:assert/strict"; +import { syncPr } from "../src/sync-pr.js"; +import type { GitHubApi } from "../src/api.js"; +import type { Issue, PullRequest } from "../src/types.js"; +import { BOUNTY_CLAIMED, BOUNTY_REWARDED } from "../src/rules.js"; + +// --------------------------------------------------------------------------- +// Mock API +// --------------------------------------------------------------------------- + +interface MockApi extends GitHubApi { + addedByTarget: Map; + removedByTarget: Map; + commentsByTarget: Map; +} + +function makeMockApi(pr: PullRequest, issues: Issue[]): MockApi { + const addedByTarget = new Map(); + const removedByTarget = new Map(); + const commentsByTarget = new Map(); + + function recordAdd(target: number, labels: string[]) { + if (!addedByTarget.has(target)) addedByTarget.set(target, []); + addedByTarget.get(target)!.push(labels); + } + function recordRemove(target: number, label: string) { + if (!removedByTarget.has(target)) removedByTarget.set(target, []); + removedByTarget.get(target)!.push(label); + } + function recordComment(target: number, body: string) { + if (!commentsByTarget.has(target)) commentsByTarget.set(target, []); + commentsByTarget.get(target)!.push(body); + } + + return { + addedByTarget, + removedByTarget, + commentsByTarget, + async getIssue(number) { + const found = issues.find((i) => i.number === number); + if (!found) throw new Error(`Issue #${number} not found in mock`); + return found; + }, + async getPullRequest() { + return pr; + }, + async listIssuesWithLabelPrefix(): Promise { + throw new Error("not used"); + }, + async addLabels(target, labels) { + recordAdd(target, labels); + }, + async removeLabel(target, label) { + recordRemove(target, label); + }, + async addComment(target, body) { + recordComment(target, body); + }, + }; +} + +function makeIssue(overrides: Partial & { number: number }): Issue { + return { title: "issue", html_url: `https://github.com/owner/repo/issues/${overrides.number}`, state: "open", labels: [], assignees: [], ...overrides }; +} + +function makePr(overrides: Partial & { number: number }): PullRequest { + return { + title: "Test PR", + state: "open", + merged: false, + body: null, + labels: [], + user: { login: "dev" }, + html_url: `https://github.com/owner/repo/pull/${overrides.number}`, + ...overrides, + }; +} + +function labelNames(...names: string[]): { name: string }[] { + return names.map((name) => ({ name })); +} + +// --------------------------------------------------------------------------- +// Tests +// --------------------------------------------------------------------------- + +describe("syncPr", () => { + describe("label propagation", () => { + it("returns empty patch when PR has no linked issues", async () => { + const pr = makePr({ number: 10, body: "No links" }); + const api = makeMockApi(pr, []); + const patch = await syncPr({ prNumber: 10, api }); + assert.deepEqual(patch.ops, []); + }); + + it("returns empty patch when linked issue has no bounty label", async () => { + const pr = makePr({ number: 10, body: "Closes #1" }); + const issue = makeIssue({ number: 1, labels: labelNames("type: bug") }); + const api = makeMockApi(pr, [issue]); + const patch = await syncPr({ prNumber: 10, api }); + assert.deepEqual(patch.ops, []); + }); + + it("copies bounty label from linked issue to PR", async () => { + const pr = makePr({ number: 10, body: "Closes #1" }); + const issue = makeIssue({ number: 1, labels: labelNames("bounty: ๐Ÿ’ฐ $500") }); + const api = makeMockApi(pr, [issue]); + await syncPr({ prNumber: 10, api }); + + assert.ok(api.addedByTarget.get(10)?.[0]?.includes("bounty: ๐Ÿ’ฐ $500")); + }); + + it("skips label already on the PR", async () => { + const pr = makePr({ + number: 10, + body: "Closes #1", + labels: labelNames("bounty: ๐Ÿ’ฐ $500"), + }); + const issue = makeIssue({ number: 1, labels: labelNames("bounty: ๐Ÿ’ฐ $500") }); + const api = makeMockApi(pr, [issue]); + const patch = await syncPr({ prNumber: 10, api }); + assert.deepEqual(patch.ops, []); + }); + + it("posts comment on linked issue when label is propagated", async () => { + const pr = makePr({ number: 10, body: "Closes #1" }); + const issue = makeIssue({ number: 1, labels: labelNames("bounty: ๐Ÿ’ฐ $200") }); + const api = makeMockApi(pr, [issue]); + await syncPr({ prNumber: 10, api }); + + const comments = api.commentsByTarget.get(1); + assert.ok(comments && comments.length === 1); + assert.ok(comments[0]!.includes("#10")); + assert.ok(comments[0]!.includes("dev")); + }); + + it("does not post comment when label was already on the PR", async () => { + const pr = makePr({ + number: 10, + body: "Closes #1", + labels: labelNames("bounty: ๐Ÿ’ฐ $200"), + }); + const issue = makeIssue({ number: 1, labels: labelNames("bounty: ๐Ÿ’ฐ $200") }); + const api = makeMockApi(pr, [issue]); + await syncPr({ prNumber: 10, api }); + + assert.ok(!api.commentsByTarget.has(1)); + }); + + it("fetches multiple linked issues in parallel and merges labels", async () => { + const pr = makePr({ number: 10, body: "Closes #1\nCloses #2" }); + const issue1 = makeIssue({ number: 1, labels: labelNames("bounty: ๐Ÿ’ฐ $100") }); + const issue2 = makeIssue({ number: 2, labels: labelNames("bounty: ๐Ÿ’ฐ $200") }); + const api = makeMockApi(pr, [issue1, issue2]); + await syncPr({ prNumber: 10, api }); + + const added = api.addedByTarget.get(10)?.[0] ?? []; + assert.ok(added.includes("bounty: ๐Ÿ’ฐ $100")); + assert.ok(added.includes("bounty: ๐Ÿ’ฐ $200")); + }); + + it("skips a linked issue that fails to fetch without aborting", async () => { + const pr = makePr({ number: 10, body: "Closes #1\nCloses #999" }); + const issue1 = makeIssue({ number: 1, labels: labelNames("bounty: ๐Ÿ’ฐ $300") }); + // issue #999 will throw (not in mock store) + const api = makeMockApi(pr, [issue1]); + const patch = await syncPr({ prNumber: 10, api }); + + // Issue #1 label still propagated + assert.ok(api.addedByTarget.get(10)?.[0]?.includes("bounty: ๐Ÿ’ฐ $300")); + }); + }); + + describe("rewarded on merge", () => { + it("adds rewarded to PR and linked issue on merge", async () => { + const pr = makePr({ + number: 10, + merged: true, + body: "Closes #1", + labels: labelNames("bounty: ๐Ÿ’ฐ $500"), + }); + const issue = makeIssue({ + number: 1, + labels: labelNames("bounty: ๐Ÿ’ฐ $500", BOUNTY_CLAIMED), + }); + const api = makeMockApi(pr, [issue]); + await syncPr({ prNumber: 10, api }); + + assert.ok(api.addedByTarget.get(10)?.[0]?.includes(BOUNTY_REWARDED)); + assert.ok(api.addedByTarget.get(1)?.[0]?.includes(BOUNTY_REWARDED)); + assert.ok(api.removedByTarget.get(1)?.includes(BOUNTY_CLAIMED)); + }); + + it("does not add rewarded again when already present on PR and issue", async () => { + const pr = makePr({ + number: 10, + merged: true, + body: "Closes #1", + labels: labelNames("bounty: ๐Ÿ’ฐ $500", BOUNTY_REWARDED), + }); + const issue = makeIssue({ + number: 1, + labels: labelNames("bounty: ๐Ÿ’ฐ $500", BOUNTY_REWARDED), + }); + const api = makeMockApi(pr, [issue]); + const patch = await syncPr({ prNumber: 10, api }); + assert.deepEqual(patch.ops, []); + }); + + it("does not add rewarded when PR was closed but not merged", async () => { + const pr = makePr({ + number: 10, + state: "closed", + merged: false, + body: "Closes #1", + labels: labelNames("bounty: ๐Ÿ’ฐ $500"), + }); + const issue = makeIssue({ number: 1, labels: labelNames("bounty: ๐Ÿ’ฐ $500") }); + const api = makeMockApi(pr, [issue]); + await syncPr({ prNumber: 10, api }); + + assert.ok(!api.addedByTarget.get(10)?.[0]?.includes(BOUNTY_REWARDED)); + }); + + it("handles merged PR with no linked issues gracefully", async () => { + const pr = makePr({ + number: 10, + merged: true, + body: "No issue links", + labels: labelNames("bounty: ๐Ÿ’ฐ $500"), + }); + const api = makeMockApi(pr, []); + const patch = await syncPr({ prNumber: 10, api }); + + // Only the PR op + assert.ok(api.addedByTarget.get(10)?.[0]?.includes(BOUNTY_REWARDED)); + assert.equal(patch.ops.length, 1); + }); + }); +}); diff --git a/.github/workflows/autofix.yml b/.github/workflows/autofix.yml new file mode 100644 index 0000000000000000000000000000000000000000..f4862d10398437c68bf067112648c1c4d791b51e --- /dev/null +++ b/.github/workflows/autofix.yml @@ -0,0 +1,62 @@ +# ------------------------------------------------------------------- +# ------------------------------- WARNING --------------------------- +# ------------------------------------------------------------------- +# +# This file was automatically generated by gh-workflows using the +# gh-workflow-gen bin. You should add and commit this file to your +# git repository. **DO NOT EDIT THIS FILE BY HAND!** Any manual changes +# will be lost if the file is regenerated. +# +# To make modifications, update your `build.rs` configuration to adjust +# the workflow description as needed, then regenerate this file to apply +# those changes. +# +# ------------------------------------------------------------------- +# ----------------------------- END WARNING ------------------------- +# ------------------------------------------------------------------- + +name: autofix.ci +env: + RUSTFLAGS: '-Dwarnings' +'on': + pull_request: + types: + - opened + - synchronize + - reopened + branches: + - main + push: + branches: + - main +jobs: + lint: + name: Lint Fix + runs-on: ubuntu-latest + permissions: + contents: read + steps: + - name: Checkout Code + uses: actions/checkout@v6 + - name: Install SQLite + run: sudo apt-get install -y libsqlite3-dev + - name: Setup Protobuf Compiler + uses: arduino/setup-protoc@v3 + with: + repo-token: ${{ secrets.GITHUB_TOKEN }} + - name: Setup Rust Toolchain + uses: actions-rust-lang/setup-rust-toolchain@v1 + with: + toolchain: nightly + components: clippy, rustfmt + - name: Cargo Fmt + run: cargo +nightly fmt --all + - name: Cargo Clippy + run: cargo +nightly clippy --all-features --workspace --all-targets --fix --allow-dirty -- -D warnings + - name: Cargo Clippy String Safety + run: cargo +nightly clippy --all-features --workspace -- -D clippy::string_slice -D clippy::indexing_slicing -D clippy::disallowed_methods + - name: Autofix + uses: autofix-ci/action@7a166d7532b277f34e16238930461bf77f9d7ed8 +concurrency: + group: autofix-${{github.ref}} + cancel-in-progress: false diff --git a/.github/workflows/bounty.yml b/.github/workflows/bounty.yml new file mode 100644 index 0000000000000000000000000000000000000000..e41e7c79e3b0189b1bad3699a65feb9f8c2b67cf --- /dev/null +++ b/.github/workflows/bounty.yml @@ -0,0 +1,64 @@ +# ------------------------------------------------------------------- +# ------------------------------- WARNING --------------------------- +# ------------------------------------------------------------------- +# +# This file was automatically generated by gh-workflows using the +# gh-workflow-gen bin. You should add and commit this file to your +# git repository. **DO NOT EDIT THIS FILE BY HAND!** Any manual changes +# will be lost if the file is regenerated. +# +# To make modifications, update your `build.rs` configuration to adjust +# the workflow description as needed, then regenerate this file to apply +# those changes. +# +# ------------------------------------------------------------------- +# ----------------------------- END WARNING ------------------------- +# ------------------------------------------------------------------- + +name: Bounty Management +'on': + issues: + types: + - assigned + - unassigned + - labeled + - unlabeled + pull_request: + types: + - opened + - edited + - reopened + pull_request_target: + types: + - closed + schedule: + - cron: '0 2 * * *' +permissions: + issues: write + pull-requests: write +jobs: + sync-all-issues: + name: Sync all bounty issues + runs-on: ubuntu-latest + permissions: + issues: write + steps: + - name: Checkout + uses: actions/checkout@v6 + - name: Install npm packages + run: npm install + - name: Sync all bounty labels + run: npx tsx .github/scripts/bounty/src/sync-all-issues.ts --repo ${{ github.repository }} --token ${{ secrets.GITHUB_TOKEN }} --execute + sync-pr: + name: Sync PR bounty labels + runs-on: ubuntu-latest + permissions: + issues: write + pull-requests: write + steps: + - name: Checkout + uses: actions/checkout@v6 + - name: Install npm packages + run: npm install + - name: Sync bounty labels + run: npx tsx .github/scripts/bounty/src/sync-pr.ts --pr ${{ github.event.pull_request.number }} --repo ${{ github.repository }} --token ${{ secrets.GITHUB_TOKEN }} diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml new file mode 100644 index 0000000000000000000000000000000000000000..778df922e80bf344d1b593584ad336390a0aeb26 --- /dev/null +++ b/.github/workflows/ci.yml @@ -0,0 +1,296 @@ +# ------------------------------------------------------------------- +# ------------------------------- WARNING --------------------------- +# ------------------------------------------------------------------- +# +# This file was automatically generated by gh-workflows using the +# gh-workflow-gen bin. You should add and commit this file to your +# git repository. **DO NOT EDIT THIS FILE BY HAND!** Any manual changes +# will be lost if the file is regenerated. +# +# To make modifications, update your `build.rs` configuration to adjust +# the workflow description as needed, then regenerate this file to apply +# those changes. +# +# ------------------------------------------------------------------- +# ----------------------------- END WARNING ------------------------- +# ------------------------------------------------------------------- + +name: ci +env: + RUSTFLAGS: '-Dwarnings' + OPENROUTER_API_KEY: ${{secrets.OPENROUTER_API_KEY}} +'on': + pull_request: + types: + - opened + - synchronize + - reopened + - labeled + branches: + - main + push: + branches: + - main + tags: + - v* +jobs: + build: + name: Build and Test + runs-on: ubuntu-latest + permissions: + contents: read + steps: + - name: Checkout Code + uses: actions/checkout@v6 + - name: Setup Protobuf Compiler + uses: arduino/setup-protoc@v3 + with: + repo-token: ${{ secrets.GITHUB_TOKEN }} + - name: Setup Rust Toolchain + uses: actions-rust-lang/setup-rust-toolchain@v1 + with: + toolchain: stable + - name: Install cargo-llvm-cov + run: cargo install cargo-llvm-cov + - name: Generate coverage + run: cargo llvm-cov --all-features --workspace --lcov --output-path lcov.info + zsh_rprompt_perf: + name: 'Performance: zsh rprompt' + runs-on: ubuntu-latest + permissions: + contents: read + steps: + - name: Checkout Code + uses: actions/checkout@v6 + - name: Setup Protobuf Compiler + uses: arduino/setup-protoc@v3 + with: + repo-token: ${{ secrets.GITHUB_TOKEN }} + - name: Setup Rust Toolchain + uses: actions-rust-lang/setup-rust-toolchain@v1 + with: + toolchain: stable + - name: Run performance benchmark + run: './scripts/benchmark.sh --threshold 60 zsh rprompt' + draft_release: + needs: + - build + if: github.event_name == 'push' && github.ref == 'refs/heads/main' + name: Draft Release + runs-on: ubuntu-latest + permissions: + contents: write + pull-requests: write + outputs: + crate_release_name: ${{ steps.set_output.outputs.crate_release_name }} + crate_release_id: ${{ steps.set_output.outputs.crate_release_id }} + steps: + - name: Checkout Code + uses: actions/checkout@v6 + - id: create_release + name: Draft Release + uses: release-drafter/release-drafter@v7 + with: + config-name: release-drafter.yml + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + - id: set_output + name: Export Outputs + run: echo "crate_release_id=${{ steps.create_release.outputs.id }}" >> $GITHUB_OUTPUT && echo "crate_release_name=${{ steps.create_release.outputs.tag_name }}" >> $GITHUB_OUTPUT + draft_release_pr: + if: 'github.event_name == ''pull_request'' && contains(github.event.pull_request.labels.*.name, ''ci: build all targets'')' + name: Draft Release for PR + runs-on: ubuntu-latest + outputs: + crate_release_name: ${{ steps.set_output.outputs.crate_release_name }} + crate_release_id: ${{ steps.set_output.outputs.crate_release_id }} + steps: + - name: Checkout Code + uses: actions/checkout@v6 + - id: set_output + name: Set Release Version + run: echo "crate_release_name=pr-build-${{ github.event.number }}" >> $GITHUB_OUTPUT && echo "crate_release_id=pr-build-${{ github.event.number }}" >> $GITHUB_OUTPUT + build_release: + needs: + - draft_release + if: github.event_name == 'push' && github.ref == 'refs/heads/main' + name: build-release + runs-on: ${{ matrix.os }} + permissions: + contents: write + pull-requests: write + strategy: + matrix: + include: + - binary_name: forge-x86_64-unknown-linux-musl + binary_path: target/x86_64-unknown-linux-musl/release/forge + cross: 'true' + os: ubuntu-latest + target: x86_64-unknown-linux-musl + - binary_name: forge-aarch64-unknown-linux-musl + binary_path: target/aarch64-unknown-linux-musl/release/forge + cross: 'true' + os: ubuntu-latest + target: aarch64-unknown-linux-musl + - binary_name: forge-x86_64-unknown-linux-gnu + binary_path: target/x86_64-unknown-linux-gnu/release/forge + cross: 'false' + os: ubuntu-latest + target: x86_64-unknown-linux-gnu + - binary_name: forge-aarch64-unknown-linux-gnu + binary_path: target/aarch64-unknown-linux-gnu/release/forge + cross: 'true' + os: ubuntu-latest + target: aarch64-unknown-linux-gnu + - binary_name: forge-x86_64-apple-darwin + binary_path: target/x86_64-apple-darwin/release/forge + cross: 'false' + os: macos-latest + target: x86_64-apple-darwin + - binary_name: forge-aarch64-apple-darwin + binary_path: target/aarch64-apple-darwin/release/forge + cross: 'false' + os: macos-latest + target: aarch64-apple-darwin + - binary_name: forge-x86_64-pc-windows-msvc.exe + binary_path: target/x86_64-pc-windows-msvc/release/forge.exe + cross: 'false' + os: windows-latest + target: x86_64-pc-windows-msvc + - binary_name: forge-aarch64-pc-windows-msvc.exe + binary_path: target/aarch64-pc-windows-msvc/release/forge.exe + cross: 'false' + os: windows-latest + target: aarch64-pc-windows-msvc + - binary_name: forge-aarch64-linux-android + binary_path: target/aarch64-linux-android/release/forge + cross: 'true' + os: ubuntu-latest + target: aarch64-linux-android + steps: + - name: Checkout Code + uses: actions/checkout@v6 + - name: Setup Protobuf Compiler + if: ${{ matrix.cross == 'false' }} + uses: arduino/setup-protoc@v3 + with: + repo-token: ${{ secrets.GITHUB_TOKEN }} + - name: Setup Cross Toolchain + if: ${{ matrix.cross == 'false' }} + uses: taiki-e/setup-cross-toolchain-action@v1 + with: + target: ${{ matrix.target }} + - name: Add Rust target + if: ${{ matrix.cross == 'false' }} + run: rustup target add ${{ matrix.target }} + - name: Set Rust Flags + if: '!(contains(matrix.target, ''-unknown-linux-'') || contains(matrix.target, ''-android''))' + run: echo "RUSTFLAGS=-C target-feature=+crt-static" >> $GITHUB_ENV + - name: Build Binary + uses: ClementTsang/cargo-action@v0.0.7 + with: + command: build --release + args: '--target ${{ matrix.target }}' + use-cross: ${{ matrix.cross }} + cross-version: '0.2.5' + env: + RUSTFLAGS: ${{ env.RUSTFLAGS }} + POSTHOG_API_SECRET: ${{secrets.POSTHOG_API_SECRET}} + APP_VERSION: ${{ needs.draft_release.outputs.crate_release_name }} + - name: Copy Binary + run: cp ${{ matrix.binary_path }} ${{ matrix.binary_name }} + - name: Upload to Release + uses: xresloader/upload-to-github-release@v1 + with: + release_id: ${{ needs.draft_release.outputs.crate_release_id }} + file: ${{ matrix.binary_name }} + overwrite: 'true' + build_release_pr: + needs: + - draft_release_pr + if: 'github.event_name == ''pull_request'' && contains(github.event.pull_request.labels.*.name, ''ci: build all targets'')' + name: build-release + runs-on: ${{ matrix.os }} + permissions: + contents: write + pull-requests: write + strategy: + matrix: + include: + - binary_name: forge-x86_64-unknown-linux-musl + binary_path: target/x86_64-unknown-linux-musl/release/forge + cross: 'true' + os: ubuntu-latest + target: x86_64-unknown-linux-musl + - binary_name: forge-aarch64-unknown-linux-musl + binary_path: target/aarch64-unknown-linux-musl/release/forge + cross: 'true' + os: ubuntu-latest + target: aarch64-unknown-linux-musl + - binary_name: forge-x86_64-unknown-linux-gnu + binary_path: target/x86_64-unknown-linux-gnu/release/forge + cross: 'false' + os: ubuntu-latest + target: x86_64-unknown-linux-gnu + - binary_name: forge-aarch64-unknown-linux-gnu + binary_path: target/aarch64-unknown-linux-gnu/release/forge + cross: 'true' + os: ubuntu-latest + target: aarch64-unknown-linux-gnu + - binary_name: forge-x86_64-apple-darwin + binary_path: target/x86_64-apple-darwin/release/forge + cross: 'false' + os: macos-latest + target: x86_64-apple-darwin + - binary_name: forge-aarch64-apple-darwin + binary_path: target/aarch64-apple-darwin/release/forge + cross: 'false' + os: macos-latest + target: aarch64-apple-darwin + - binary_name: forge-x86_64-pc-windows-msvc.exe + binary_path: target/x86_64-pc-windows-msvc/release/forge.exe + cross: 'false' + os: windows-latest + target: x86_64-pc-windows-msvc + - binary_name: forge-aarch64-pc-windows-msvc.exe + binary_path: target/aarch64-pc-windows-msvc/release/forge.exe + cross: 'false' + os: windows-latest + target: aarch64-pc-windows-msvc + - binary_name: forge-aarch64-linux-android + binary_path: target/aarch64-linux-android/release/forge + cross: 'true' + os: ubuntu-latest + target: aarch64-linux-android + steps: + - name: Checkout Code + uses: actions/checkout@v6 + - name: Setup Protobuf Compiler + if: ${{ matrix.cross == 'false' }} + uses: arduino/setup-protoc@v3 + with: + repo-token: ${{ secrets.GITHUB_TOKEN }} + - name: Setup Cross Toolchain + if: ${{ matrix.cross == 'false' }} + uses: taiki-e/setup-cross-toolchain-action@v1 + with: + target: ${{ matrix.target }} + - name: Add Rust target + if: ${{ matrix.cross == 'false' }} + run: rustup target add ${{ matrix.target }} + - name: Set Rust Flags + if: '!(contains(matrix.target, ''-unknown-linux-'') || contains(matrix.target, ''-android''))' + run: echo "RUSTFLAGS=-C target-feature=+crt-static" >> $GITHUB_ENV + - name: Build Binary + uses: ClementTsang/cargo-action@v0.0.7 + with: + command: build --release + args: '--target ${{ matrix.target }}' + use-cross: ${{ matrix.cross }} + cross-version: '0.2.5' + env: + RUSTFLAGS: ${{ env.RUSTFLAGS }} + POSTHOG_API_SECRET: ${{secrets.POSTHOG_API_SECRET}} + APP_VERSION: ${{ needs.draft_release_pr.outputs.crate_release_name }} +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} diff --git a/.github/workflows/labels.yml b/.github/workflows/labels.yml new file mode 100644 index 0000000000000000000000000000000000000000..6c8f168130a766d83e0cdcf5729b0f60d758495b --- /dev/null +++ b/.github/workflows/labels.yml @@ -0,0 +1,39 @@ +# ------------------------------------------------------------------- +# ------------------------------- WARNING --------------------------- +# ------------------------------------------------------------------- +# +# This file was automatically generated by gh-workflows using the +# gh-workflow-gen bin. You should add and commit this file to your +# git repository. **DO NOT EDIT THIS FILE BY HAND!** Any manual changes +# will be lost if the file is regenerated. +# +# To make modifications, update your `build.rs` configuration to adjust +# the workflow description as needed, then regenerate this file to apply +# those changes. +# +# ------------------------------------------------------------------- +# ----------------------------- END WARNING ------------------------- +# ------------------------------------------------------------------- + +name: Github Label Sync +'on': + push: + branches: + - main +permissions: + contents: write +jobs: + label-sync: + name: label-sync + runs-on: ubuntu-latest + permissions: + issues: write + steps: + - name: Checkout + uses: actions/checkout@v6 + - name: Sync labels + run: |- + npx github-label-sync \ + --access-token ${{ secrets.GITHUB_TOKEN }} \ + --labels ".github/labels.json" \ + ${{ github.repository }} diff --git a/.github/workflows/release-drafter.yml b/.github/workflows/release-drafter.yml new file mode 100644 index 0000000000000000000000000000000000000000..53cddb0648b2d0c0979c6f62c9e4494f6c5bddfa --- /dev/null +++ b/.github/workflows/release-drafter.yml @@ -0,0 +1,53 @@ +# ------------------------------------------------------------------- +# ------------------------------- WARNING --------------------------- +# ------------------------------------------------------------------- +# +# This file was automatically generated by gh-workflows using the +# gh-workflow-gen bin. You should add and commit this file to your +# git repository. **DO NOT EDIT THIS FILE BY HAND!** Any manual changes +# will be lost if the file is regenerated. +# +# To make modifications, update your `build.rs` configuration to adjust +# the workflow description as needed, then regenerate this file to apply +# those changes. +# +# ------------------------------------------------------------------- +# ----------------------------- END WARNING ------------------------- +# ------------------------------------------------------------------- + +name: Release Drafter +'on': + pull_request_target: + types: + - opened + - reopened + - synchronize + - labeled + - unlabeled + - closed + branches: + - main + push: + branches: + - main +permissions: + contents: write + pull-requests: write +jobs: + update_release_draft: + name: update_release_draft + runs-on: ubuntu-latest + steps: + - name: Auto Labeler + if: github.event_name == 'pull_request_target' + uses: release-drafter/release-drafter/autolabeler@v7 + with: + config-name: release-drafter.yml + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + - name: Release Drafter + uses: release-drafter/release-drafter@v7 + with: + config-name: release-drafter.yml + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml new file mode 100644 index 0000000000000000000000000000000000000000..01574fb0313ce2e7913ab3ff349af161702b150d --- /dev/null +++ b/.github/workflows/release.yml @@ -0,0 +1,155 @@ +# ------------------------------------------------------------------- +# ------------------------------- WARNING --------------------------- +# ------------------------------------------------------------------- +# +# This file was automatically generated by gh-workflows using the +# gh-workflow-gen bin. You should add and commit this file to your +# git repository. **DO NOT EDIT THIS FILE BY HAND!** Any manual changes +# will be lost if the file is regenerated. +# +# To make modifications, update your `build.rs` configuration to adjust +# the workflow description as needed, then regenerate this file to apply +# those changes. +# +# ------------------------------------------------------------------- +# ----------------------------- END WARNING ------------------------- +# ------------------------------------------------------------------- + +name: Multi Channel Release +'on': + release: + types: + - published +permissions: + contents: write + pull-requests: write +jobs: + build_release: + name: build-release + runs-on: ${{ matrix.os }} + permissions: + contents: write + pull-requests: write + strategy: + matrix: + include: + - binary_name: forge-x86_64-unknown-linux-musl + binary_path: target/x86_64-unknown-linux-musl/release/forge + cross: 'true' + os: ubuntu-latest + target: x86_64-unknown-linux-musl + - binary_name: forge-aarch64-unknown-linux-musl + binary_path: target/aarch64-unknown-linux-musl/release/forge + cross: 'true' + os: ubuntu-latest + target: aarch64-unknown-linux-musl + - binary_name: forge-x86_64-unknown-linux-gnu + binary_path: target/x86_64-unknown-linux-gnu/release/forge + cross: 'false' + os: ubuntu-latest + target: x86_64-unknown-linux-gnu + - binary_name: forge-aarch64-unknown-linux-gnu + binary_path: target/aarch64-unknown-linux-gnu/release/forge + cross: 'true' + os: ubuntu-latest + target: aarch64-unknown-linux-gnu + - binary_name: forge-x86_64-apple-darwin + binary_path: target/x86_64-apple-darwin/release/forge + cross: 'false' + os: macos-latest + target: x86_64-apple-darwin + - binary_name: forge-aarch64-apple-darwin + binary_path: target/aarch64-apple-darwin/release/forge + cross: 'false' + os: macos-latest + target: aarch64-apple-darwin + - binary_name: forge-x86_64-pc-windows-msvc.exe + binary_path: target/x86_64-pc-windows-msvc/release/forge.exe + cross: 'false' + os: windows-latest + target: x86_64-pc-windows-msvc + - binary_name: forge-aarch64-pc-windows-msvc.exe + binary_path: target/aarch64-pc-windows-msvc/release/forge.exe + cross: 'false' + os: windows-latest + target: aarch64-pc-windows-msvc + - binary_name: forge-aarch64-linux-android + binary_path: target/aarch64-linux-android/release/forge + cross: 'true' + os: ubuntu-latest + target: aarch64-linux-android + steps: + - name: Checkout Code + uses: actions/checkout@v6 + - name: Setup Protobuf Compiler + if: ${{ matrix.cross == 'false' }} + uses: arduino/setup-protoc@v3 + with: + repo-token: ${{ secrets.GITHUB_TOKEN }} + - name: Setup Cross Toolchain + if: ${{ matrix.cross == 'false' }} + uses: taiki-e/setup-cross-toolchain-action@v1 + with: + target: ${{ matrix.target }} + - name: Add Rust target + if: ${{ matrix.cross == 'false' }} + run: rustup target add ${{ matrix.target }} + - name: Set Rust Flags + if: '!(contains(matrix.target, ''-unknown-linux-'') || contains(matrix.target, ''-android''))' + run: echo "RUSTFLAGS=-C target-feature=+crt-static" >> $GITHUB_ENV + - name: Build Binary + uses: ClementTsang/cargo-action@v0.0.7 + with: + command: build --release + args: '--target ${{ matrix.target }}' + use-cross: ${{ matrix.cross }} + cross-version: '0.2.5' + env: + RUSTFLAGS: ${{ env.RUSTFLAGS }} + POSTHOG_API_SECRET: ${{secrets.POSTHOG_API_SECRET}} + APP_VERSION: ${{ github.event.release.tag_name }} + - name: Copy Binary + run: cp ${{ matrix.binary_path }} ${{ matrix.binary_name }} + - name: Upload to Release + uses: xresloader/upload-to-github-release@v1 + with: + release_id: ${{ github.event.release.id }} + file: ${{ matrix.binary_name }} + overwrite: 'true' + npm_release: + needs: + - build_release + name: npm_release + runs-on: ubuntu-latest + strategy: + matrix: + repository: + - antinomyhq/npm-code-forge + - antinomyhq/npm-forgecode + steps: + - name: Checkout Code + uses: actions/checkout@v6 + with: + repository: ${{ matrix.repository }} + ref: main + token: ${{ secrets.NPM_ACCESS }} + - name: Update NPM Package + run: './update-package.sh ${{ github.event.release.tag_name }}' + env: + AUTO_PUSH: 'true' + CI: 'true' + NPM_TOKEN: ${{ secrets.NPM_TOKEN }} + homebrew_release: + needs: + - build_release + name: homebrew_release + runs-on: ubuntu-latest + steps: + - name: Checkout Code + uses: actions/checkout@v6 + with: + repository: antinomyhq/homebrew-code-forge + ref: main + token: ${{ secrets.HOMEBREW_ACCESS }} + - name: Update Homebrew Formula + run: GITHUB_TOKEN="${{ secrets.HOMEBREW_ACCESS }}" ./update-formula.sh ${{ github.event.release.tag_name }} diff --git a/.github/workflows/stale.yml b/.github/workflows/stale.yml new file mode 100644 index 0000000000000000000000000000000000000000..2d795379e0ef30c1a9f95f82b672b7a89e09dd1f --- /dev/null +++ b/.github/workflows/stale.yml @@ -0,0 +1,51 @@ +# ------------------------------------------------------------------- +# ------------------------------- WARNING --------------------------- +# ------------------------------------------------------------------- +# +# This file was automatically generated by gh-workflows using the +# gh-workflow-gen bin. You should add and commit this file to your +# git repository. **DO NOT EDIT THIS FILE BY HAND!** Any manual changes +# will be lost if the file is regenerated. +# +# To make modifications, update your `build.rs` configuration to adjust +# the workflow description as needed, then regenerate this file to apply +# those changes. +# +# ------------------------------------------------------------------- +# ----------------------------- END WARNING ------------------------- +# ------------------------------------------------------------------- + +name: Close Stale Issues and PR +env: + DAYS_BEFORE_ISSUE_STALE: '30' + DAYS_BEFORE_ISSUE_CLOSE: '7' + DAYS_BEFORE_PR_STALE: '5' + DAYS_BEFORE_PR_CLOSE: '10' +'on': + schedule: + - cron: '0 * * * *' +permissions: + issues: write + pull-requests: write +jobs: + stale: + name: Stale Issues + runs-on: ubuntu-latest + steps: + - name: Mark Stale Issues + uses: actions/stale@v10 + with: + stale-issue-label: 'state: inactive' + stale-pr-label: 'state: inactive' + stale-issue-message: |- + **Action required:** Issue inactive for ${{ env.DAYS_BEFORE_ISSUE_STALE }} days. + Status update or closure in ${{ env.DAYS_BEFORE_ISSUE_CLOSE }} days. + close-issue-message: Issue closed after ${{ env.DAYS_BEFORE_ISSUE_CLOSE }} days of inactivity. + stale-pr-message: |- + **Action required:** PR inactive for ${{ env.DAYS_BEFORE_PR_STALE }} days. + Status update or closure in ${{ env.DAYS_BEFORE_PR_CLOSE }} days. + close-pr-message: PR closed after ${{ env.DAYS_BEFORE_PR_CLOSE }} days of inactivity. + days-before-issue-stale: ${{ env.DAYS_BEFORE_ISSUE_STALE }} + days-before-issue-close: ${{ env.DAYS_BEFORE_ISSUE_CLOSE }} + days-before-pr-stale: ${{ env.DAYS_BEFORE_PR_STALE }} + days-before-pr-close: ${{ env.DAYS_BEFORE_PR_CLOSE }} diff --git a/benchmarks/README.md b/benchmarks/README.md new file mode 100644 index 0000000000000000000000000000000000000000..80a4cff7c604d756d0ad753c0d9e49610021afbd --- /dev/null +++ b/benchmarks/README.md @@ -0,0 +1,320 @@ +# Forge Code Evaluations + +A flexible evaluation framework for running automated tests and benchmarks against Forge Code commands. + +## Quick Start + +### Setup + +Before running evaluations, create a `forgee` symlink to the debug binary: + +```bash +# Create symlink in your PATH (e.g., ~/bin or /usr/local/bin) +ln -sf /path/to/code-forge/target/debug/forge ~/forgee + +# Or if ~/bin is in your PATH +ln -sf $(pwd)/target/debug/forge ~/bin/forgee +``` + +**Why is this needed?** Tasks execute in temporary directories, so relative paths like `../../target/debug/forge` won't work. The `forgee` symlink provides a stable absolute path that works from any directory. + +### Running Evaluations + +```bash +# Run an evaluation +npm run eval ./evals/create_skill/task.yml + +# Set custom log level +LOG_LEVEL=debug npm run eval ./evals/create_skill/task.yml +``` + +## How It Works + +The evaluation system executes commands based on task definitions and validates their output. It supports: + +- **Parallel execution** with configurable concurrency +- **Timeout handling** for long-running tasks +- **Data-driven testing** using CSV files +- **Output validation** using regex patterns +- **Debug artifacts** stored with timestamps + +## Project Structure + +``` +benchmarks/ +โ”œโ”€โ”€ cli.ts # Main CLI entry point +โ”œโ”€โ”€ command-generator.ts # Command template rendering +โ”œโ”€โ”€ task-executor.ts # Task execution with timeout support +โ”œโ”€โ”€ model.ts # TypeScript types for tasks +โ”œโ”€โ”€ parse.ts # CLI argument parsing +โ””โ”€โ”€ evals/ # Evaluation definitions + โ””โ”€โ”€ create_skill/ + โ”œโ”€โ”€ task.yml # Task definition + โ”œโ”€โ”€ create_skill_tasks.csv # Test data + โ””โ”€โ”€ debug/ # Debug outputs (timestamped) +``` + +## Creating an Evaluation + +### 1. Create an Evaluation Directory + +```bash +mkdir -p evals/my_eval +``` + +### 2. Create a `task.yml` File + +The task file defines how to run your evaluation: + +```yaml +# Optional: Commands to run before evaluation starts +before_run: + - cargo build + - npm install + +# Required: Command(s) to execute for each test case +# Single command +run: forgee -p '{{prompt}}' + +# Or multiple commands (executed sequentially) +run: + - echo "Step 1: {{task}}" + - forgee -p '{{prompt}}' + - echo "Step 2: Complete" + +# Execution configuration +parallelism: 10 # Number of tasks to run in parallel (default: 1) +timeout: 60 # Timeout in seconds (optional) +early_exit: true # Stop execution when validations pass (optional) + +# Optional: Validations to run on output +validations: + - name: "Check success message" + type: regex + regex: \[[0-9:]*\] Skill create-skill + +# Required: Data sources for test cases +sources: + - csv: my_tasks.csv +``` + +#### Task File Schema + +**`before_run`** (optional): Array of shell commands to execute before running tasks +- Runs sequentially before the main command execution +- Executes in a temporary directory created for the evaluation run +- Useful for building binaries or setting up dependencies + +**`run`** (required): Command(s) to execute for each test case +- Can be a single string or an array of strings +- Commands support template placeholders (e.g., `{{variable}}`) +- Multiple commands are executed sequentially +- If any command fails, subsequent commands are skipped + +**`parallelism`** (optional): Number of tasks to run concurrently (default: 1) + +**`timeout`** (optional): Maximum execution time in seconds per task + +**`early_exit`** (optional): Stop command execution when all validations pass + +**`validations`** (optional): Array of validation rules +- `name`: Human-readable description +- `type`: Validation type. Supported values: + - `regex`: Match output against a regular expression pattern + - `shell`: Execute a shell command with output as stdin +- For `regex` type: + - `regex`: Regular expression pattern to match in output +- For `shell` type: + - `command`: Shell command to execute (receives task output via stdin) + - `exit_code`: Expected exit code (default: 0) + +**`sources`** (required): Array of data sources +- Currently supports CSV files: `- csv: filename.csv` +- Future: Command output: `- cmd: command` + +### 3. Create Test Data (CSV) + +Create a CSV file with columns matching your template variables: + +```csv +prompt,expected_output +"Create a backup script","backup.sh" +"Generate API client","api_client.py" +``` + +The column names (e.g., `prompt`, `expected_output`) become template variables you can use in your command with `{{column_name}}`. + +### 4. Run the Evaluation + +```bash +npm run eval ./evals/my_eval/task.yml +``` + +## Template Variables + +Commands support Handlebars template syntax. Variables are populated from CSV columns: + +```yaml +run: + command: ./tool --input '{{input_file}}' --format {{format}} --verbose +``` + +With CSV: +```csv +input_file,format +data1.txt,json +data2.txt,yaml +``` + +## Output and Debugging + +### Debug Artifacts + +Each run creates a timestamped debug directory: + +``` +evals/my_eval/debug/2025-11-23T10-30-45-123Z/ +โ”œโ”€โ”€ task-1.log +โ”œโ”€โ”€ task-2.log +โ””โ”€โ”€ task-3.log +``` + +Each log file contains the full output (stdout + stderr) from the command execution. + +### Task Status + +Tasks can have four statuses: + +- **`passed`**: Task completed successfully and passed all validations +- **`validation_failed`**: Task completed but failed one or more validations +- **`timeout`**: Task exceeded the timeout limit +- **`failed`**: Task execution failed (non-zero exit code) + +### Logging + +**Human-readable output (default):** +```bash +npm run eval ./evals/my_eval/task.yml +``` + +**Machine-readable JSON output:** +```bash +LOG_JSON=1 npm run eval ./evals/my_eval/task.yml | jq . +``` + +**Debug logging:** +```bash +LOG_LEVEL=debug npm run eval ./evals/my_eval/task.yml +``` + +## Examples + +### Example 1: Simple Sequential Execution + +```yaml +run: echo "Processing {{name}}" +sources: + - csv: names.csv +``` + +```csv +name +Alice +Bob +Charlie +``` + +### Example 2: Parallel Execution with Timeout + +```yaml +run: ./slow_task --id {{task_id}} +parallelism: 5 +timeout: 30 +sources: + - csv: tasks.csv +``` + +### Example 3: Multiple Commands + +```yaml +run: + - echo "Starting task {{id}}" + - ./process --input {{file}} + - echo "Task {{id}} complete" +parallelism: 3 +timeout: 120 +sources: + - csv: tasks.csv +``` + +### Example 4: Shell Command Validation + +```yaml +run: echo "{{message}}" +parallelism: 3 +validations: + # Using grep to check if output contains specific text + - name: "Contains 'test' word" + type: shell + command: grep -q "test" + exit_code: 0 + + # Count words and ensure it's greater than 2 + - name: "More than 2 words" + type: shell + command: test $(wc -w | awk '{print $1}') -gt 2 + exit_code: 0 + + # Traditional regex validation (for comparison) + - name: "Contains test or validation" + type: regex + regex: "test|validation" +sources: + - csv: messages.csv +``` + +### Example 5: Regex Validation + +```yaml +run: cargo test {{test_name}} +validations: + - name: "All tests passed" + type: regex + regex: test result:\s+ok +sources: + - csv: tests.csv +``` + +## Tips + +1. **Use quotes in commands**: When passing CSV values with spaces, wrap them in quotes: + ```yaml + command: forge -p '{{prompt}}' + ``` + +2. **Build before running**: Use `before_run` to ensure binaries are up-to-date: + ```yaml + before_run: + - cargo build + ``` + +3. **Start with low parallelism**: Test with `parallelism: 1` first, then increase: + ```yaml + parallelism: 1 # Start here + ``` + +4. **Set appropriate timeouts**: Add timeouts to prevent hanging: + ```yaml + timeout: 60 # seconds + ``` + +5. **Check debug logs**: When tasks fail, check the debug directory for full output: + ```bash + cat evals/my_eval/debug/*/task-1.log + ``` + +## Exit Codes + +The CLI exits with: +- **0**: All tasks passed +- **1**: One or more tasks failed (excluding timeouts and validation failures) diff --git a/benchmarks/cli.ts b/benchmarks/cli.ts new file mode 100644 index 0000000000000000000000000000000000000000..3f681ce9adcefe1af05335e7173356e5947dbfc5 --- /dev/null +++ b/benchmarks/cli.ts @@ -0,0 +1,394 @@ +#!/usr/bin/env node + +// Handle EPIPE errors gracefully (e.g., when piping to `head` or `jq` that closes early) +process.stdout.on("error", (error: NodeJS.ErrnoException) => { + if (error.code === "EPIPE") { + process.exit(0); + } + throw error; +}); + +import * as fs from "fs/promises"; +import * as path from "path"; +import { fileURLToPath } from "url"; +import { parse as parseYaml } from "yaml"; +import { exec } from "child_process"; +import { promisify } from "util"; +import pLimit from "p-limit"; +import pino from "pino"; +import { TaskStatus, type Task } from "./model.js"; +import { + getContextsFromSources, + generateCommand, +} from "./command-generator.js"; +import { parseCliArgs } from "./parse.js"; +import { executeTask, type TaskExecutionResult } from "./task-executor.js"; +import { processValidations, type ValidationResult } from "./verification.js"; +import { createTempDir, parseCsvAsync } from "./utils.js"; + +const execAsync = promisify(exec); + +export type TaskResult = { + index: number; + status: TaskStatus; + command: string; + duration: number; + validationResults: ValidationResult[]; +}; + +// ESM compatibility for __dirname +const __filename = fileURLToPath(import.meta.url); +const __dirname = path.dirname(__filename); + +/** + * Create logger instance + * - Human-readable CLI output by default + * - Set LOG_JSON=1 for machine-readable JSON output (for piping to jq, log aggregators, etc.) + */ +const logger = + process.env.LOG_JSON === "1" + ? pino({ + level: process.env.LOG_LEVEL || "info", + formatters: { + level: (label) => ({ level: label }), + }, + timestamp: pino.stdTimeFunctions.isoTime, + }) + : pino({ + level: process.env.LOG_LEVEL || "info", + transport: { + target: "pino-pretty", + options: { + colorize: true, + translateTime: "HH:MM:ss", + ignore: "pid,hostname", + messageFormat: "{msg}", + }, + }, + formatters: { + level: (label) => ({ level: label }), + }, + timestamp: pino.stdTimeFunctions.isoTime, + }); + +async function main() { + // Parse command line arguments + let args; + try { + args = await parseCliArgs(__dirname); + } catch (error) { + const message = error instanceof Error ? error.message : "Unknown error"; + logger.error({ error: message }, "Failed to parse CLI arguments"); + process.exit(1); + } + + const { evalName, evalDir, taskFile } = args; + + // Check if eval directory and task file exist + try { + await fs.access(evalDir); + } catch { + logger.error({ evalDir }, "Eval directory not found"); + process.exit(1); + } + + try { + await fs.access(taskFile); + } catch { + logger.error({ evalDir }, "task.yml not found"); + process.exit(1); + } + + // Read and parse task.yml + const taskContent = await fs.readFile(taskFile, "utf-8"); + const task: Task = parseYaml(taskContent); + + // Display header + const displayName = path.relative(__dirname, evalDir) || evalName; + + // Create debug directory with timestamp + const timestamp = new Date().toISOString().replace(/[:.]/g, "-"); + const debugDir = path.join(evalDir, "debug", timestamp); + await fs.mkdir(debugDir, { recursive: true }); + + // Create a temp directory for setup commands + const setupTmpDir = await createTempDir("forge-setup-"); + + // Execute before_run commands + if (task.before_run && task.before_run.length > 0) { + for (const cmd of task.before_run) { + try { + logger.info( + { dir: setupTmpDir.name, command: cmd }, + "Running setup command", + ); + // Small delay to allow logger to flush before command output + await new Promise((resolve) => setTimeout(resolve, 0)); + await execAsync(cmd, { + cwd: setupTmpDir.name, + }); + } catch (error) { + logger.error({ command: cmd }, "Setup command failed"); + process.exit(1); + } + } + } + + // Load data from sources and create cross product + const sourcesData: Record[][] = []; + + for (const source of task.sources) { + if ("csv" in source) { + const csvPath = path.join(evalDir, source.csv); + try { + await fs.access(csvPath); + } catch { + logger.error({ csvPath }, "CSV file not found"); + process.exit(1); + } + + const csvContent = await fs.readFile(csvPath, "utf-8"); + const csvData = await parseCsvAsync(csvContent, { + columns: true, + skip_empty_lines: true, + }); + sourcesData.push(csvData); + } else if ("cmd" in source) { + logger.error("cmd source type not yet implemented"); + process.exit(1); + } else if ("value" in source) { + sourcesData.push(source.value); + } + } + + // Create cross product of all sources + if (sourcesData.length === 0) { + logger.error("No sources configured"); + process.exit(1); + } + + // Get contexts from sources using pure function + const data = getContextsFromSources(sourcesData); + + const results: TaskResult[] = []; + + // Get parallelism setting (default to 1 for sequential execution) + const parallelism = task.parallelism ?? 1; + const limit = pLimit(parallelism); + + // Execute run command for each data row + // Create promises for all tasks + const taskPromises = data.map((row, i) => { + return limit(async () => { + // Create a unique temp directory for this task + const taskTmpDir = await createTempDir(`forge-task-${i + 1}-`); + + // Create a 'task' subdirectory for running commands + const taskWorkDir = path.join(taskTmpDir.name, 'task'); + await fs.mkdir(taskWorkDir, { recursive: true }); + + const logFile = path.join(taskTmpDir.name, `task.log`); + + // Context for command interpolation and validations + const context = { ...row, dir: taskTmpDir.name }; + + // Support both single command and multiple commands + const commands = Array.isArray(task.run) ? task.run : [task.run]; + + // Filter out empty or non-string commands + const validCommands = commands.filter(cmd => typeof cmd === 'string' && cmd.trim().length > 0); + + // If no valid commands, skip this task + if (validCommands.length === 0) { + logger.warn({ task_id: i + 1 }, "No valid commands found, skipping task"); + return { + index: i + 1, + status: TaskStatus.Failed, + command: "No valid commands", + duration: 0, + validationResults: [], + }; + } + + let combinedOutput = ""; + let totalDuration = 0; + let lastError: string | undefined; + let hasTimeout = false; + let hasEarlyExit = false; + + // Log task launch once before executing commands + logger.info( + { + task_id: i + 1, + total_commands: validCommands.length, + log: logFile, + dir: taskTmpDir.name, + work_dir: taskWorkDir, + parameters: context, + }, + "Launching task", + ); + + // Execute commands sequentially + for (let cmdIdx = 0; cmdIdx < validCommands.length; cmdIdx++) { + const commandTemplate = validCommands[cmdIdx]!; // Non-null assertion safe after filter + + const command = generateCommand(commandTemplate, context); + + logger.info( + { + command, + task_id: i + 1, + command_id: cmdIdx + 1, + total_commands: validCommands.length, + }, + "Executing command", + ); + + const executionResult = await executeTask( + command, + i + 1, + logFile, + taskWorkDir, // Use taskWorkDir instead of taskTmpDir.name + task, + context, + cmdIdx > 0, // append if this is not the first command + ); + + totalDuration += executionResult.duration; + + if (executionResult.output) { + combinedOutput += executionResult.output; + } + + if (executionResult.earlyExit) { + hasEarlyExit = true; + } + + // If execution failed or timed out, stop executing remaining commands + if (executionResult.error) { + lastError = executionResult.error; + hasTimeout = executionResult.isTimeout; + + logger.warn( + { + task_id: executionResult.index, + command: executionResult.command, + command_id: cmdIdx + 1, + duration: executionResult.duration, + error: executionResult.error, + is_timeout: executionResult.isTimeout, + }, + executionResult.isTimeout ? "Task timed out" : "Task failed", + ); + break; + } + } + + // If any command failed, return failure result + if (lastError) { + const { validationResults } = await processValidations( + combinedOutput, + task, + logger, + i + 1, + totalDuration, + logFile, + context, + ); + + return { + index: i + 1, + status: hasTimeout ? TaskStatus.Timeout : TaskStatus.Failed, + command: validCommands.length === 1 ? validCommands[0]! : `${validCommands.length} commands`, + duration: totalDuration, + validationResults, + }; + } + + // Run validations on the combined output + const { validationResults, status: validationStatus } = + await processValidations( + combinedOutput, + task, + logger, + i + 1, + totalDuration, + logFile, + context, + ); + + return { + index: i + 1, + status: + validationStatus === "passed" + ? TaskStatus.Passed + : TaskStatus.ValidationFailed, + command: validCommands.length === 1 ? validCommands[0]! : `${validCommands.length} commands`, + duration: totalDuration, + validationResults, + }; + }); + }); + + // Wait for all tasks to complete + const taskResults = await Promise.all(taskPromises); + results.push(...taskResults); + + // Calculate summary statistics + const successCount = results.filter( + (r) => r.status === TaskStatus.Passed, + ).length; + const warningCount = results.filter( + (r) => r.status === TaskStatus.ValidationFailed, + ).length; + const timeoutCount = results.filter( + (r) => r.status === TaskStatus.Timeout, + ).length; + const failCount = results.filter( + (r) => r.status === TaskStatus.Failed, + ).length; + const totalDuration = results.reduce((sum, r) => sum + r.duration, 0); + + // Calculate validation statistics + const totalValidations = results.reduce( + (sum, r) => sum + r.validationResults.length, + 0, + ); + const passedValidations = results.reduce( + (sum, r) => sum + r.validationResults.filter((v) => v.passed).length, + 0, + ); + + // Print summary + logger.info( + { + total: results.length, + passed: successCount, + validation_failed: warningCount, + timeout: timeoutCount, + failed: failCount, + total_duration: totalDuration, + validations: { + total: totalValidations, + passed: passedValidations, + failed: totalValidations - passedValidations, + }, + dir: setupTmpDir.name, + }, + "Evaluation completed", + ); + + // Exit with error code if any task failed (excluding timeouts and validation failures) + if (failCount > 0) { + process.exit(1); + } + + // Exit successfully - ensures process terminates even with open handles + process.exit(0); +} + +main().catch((error) => { + logger.error({ error: error.message }, "Fatal error"); + process.exit(1); +}); diff --git a/benchmarks/command-generator.ts b/benchmarks/command-generator.ts new file mode 100644 index 0000000000000000000000000000000000000000..c3d87325873c03e91eb299b9d8943a2683421adb --- /dev/null +++ b/benchmarks/command-generator.ts @@ -0,0 +1,98 @@ +import Handlebars from "handlebars"; + +/** + * Loads data from CSV content string (internal helper) + */ +function loadCsvData(csvContent: string): Record[] { + const lines = csvContent.trim().split("\n"); + if (lines.length === 0) { + return []; + } + + const firstLine = lines[0]; + if (!firstLine) { + return []; + } + + const headers = firstLine.split(",").map((h) => h.trim()); + const data: Record[] = []; + + for (let i = 1; i < lines.length; i++) { + const line = lines[i]; + if (!line || !line.trim()) continue; + + const values = line.split(",").map((v) => v.trim()); + const row: Record = {}; + + for (let j = 0; j < headers.length; j++) { + const header = headers[j]; + if (header) { + row[header] = values[j] || ""; + } + } + + data.push(row); + } + + return data; +} + +/** + * Creates a cross product of multiple data arrays (internal helper) + */ +function createCrossProduct( + sourcesData: Record[][] +): Record[] { + if (sourcesData.length === 0) { + return []; + } + + return sourcesData.reduce((acc, sourceData) => { + if (acc.length === 0) { + return sourceData; + } + + const result: Record[] = []; + for (const accItem of acc) { + for (const sourceItem of sourceData) { + result.push({ ...accItem, ...sourceItem }); + } + } + return result; + }, [] as Record[]); +} + +/** + * Generates a command from a template and data context + */ +export function generateCommand( + commandTemplate: string, + context: Record +): string { + const template = Handlebars.compile(commandTemplate, { + strict: true, + }); + return template(context); +} + +/** + * Generates all commands from a template and cross product data (internal helper) + */ +function generateCommands( + commandTemplate: string, + sourcesData: Record[][] +): string[] { + const crossProduct = createCrossProduct(sourcesData); + return crossProduct.map((context) => + generateCommand(commandTemplate, context) + ); +} + +/** + * Pure function to get contexts from sources data + */ +export function getContextsFromSources( + sourcesData: Record[][] +): Record[] { + return createCrossProduct(sourcesData); +} diff --git a/benchmarks/evals/echo/task.yml b/benchmarks/evals/echo/task.yml new file mode 100644 index 0000000000000000000000000000000000000000..85ef1b130bdf1840142ff35568474aa92af95bb1 --- /dev/null +++ b/benchmarks/evals/echo/task.yml @@ -0,0 +1,26 @@ +run: + - git clone --depth=1 --branch main https://github.com/antinomyhq/forge + - ls -la + - echo "Forge deployment: $(readlink $(which forgee))" + - echo '{{message}}' +parallelism: 5 +timeout: 10 +validations: + - name: "Output contains Hello" + type: regex + regex: "Hello" + - name: "Contains Hello using grep" + type: shell + command: 'grep -q "Hello"' + exit_code: 0 + - name: "Output is not empty" + type: shell + command: 'grep -q "."' + exit_code: 0 +sources: + - value: + - message: "Hello, World!" + - message: "Hello from test message 123" + - message: "Hello with special chars: !@#$%" + - message: "Hello multi word message with spaces" + - message: "Hello with numbers: 42 and 3.14" diff --git a/benchmarks/evals/multi_file_patch/multi_file_patch_tasks.csv b/benchmarks/evals/multi_file_patch/multi_file_patch_tasks.csv new file mode 100644 index 0000000000000000000000000000000000000000..c03d4f60e8c5ec9fb67978b6a6e0a0183e2be609 --- /dev/null +++ b/benchmarks/evals/multi_file_patch/multi_file_patch_tasks.csv @@ -0,0 +1,5 @@ +task +"Add a comment // TODO: Refactor this module at the top of these three files: crates/forge_domain/src/lib.rs, crates/forge_app/src/lib.rs, and crates/forge_infra/src/lib.rs" +"Add the line edition = ""2021"" to the [package] section in these three Cargo.toml files: crates/forge_domain/Cargo.toml, crates/forge_app/Cargo.toml, and crates/forge_infra/Cargo.toml" +"Add ## License as a new section at the end of these markdown files: README.md, AGENTS.md, and docs/tool-guidelines.md" +"Add the import use anyhow::Result; at the top of these files (after existing imports): crates/forge_domain/src/lib.rs, crates/forge_app/src/lib.rs, and crates/forge_repo/src/lib.rs" diff --git a/benchmarks/evals/multi_file_patch/task.yml b/benchmarks/evals/multi_file_patch/task.yml new file mode 100644 index 0000000000000000000000000000000000000000..047af145cef90e7285f66afa4f2c74e0a265bb71 --- /dev/null +++ b/benchmarks/evals/multi_file_patch/task.yml @@ -0,0 +1,21 @@ +run: + # Clone into `tmp/task` dir + - git clone --depth=1 --branch main https://github.com/tailcallhq/forgecode . + - FORGE_DEBUG_REQUESTS='{{dir}}/context.json' forgee --provider open_router --model {{model}} -p '{{task}}' +parallelism: 50 +timeout: 120 +early_exit: true +validations: + - name: "Should use patch tool (not git apply)" + type: shell + command: "jq -e '[.messages[]?.tool_calls[]? | select(.function.name == \"patch\")] | length > 0' {{dir}}/context.json" + - name: "Should NOT use shell with git apply" + type: shell + command: "jq -e '[.messages[]?.tool_calls[]? | select(.function.name == \"shell\" and (.function.arguments | contains(\"git apply\") or contains(\"git patch\")))] | length == 0' {{dir}}/context.json" + - name: "Should call patch tool multiple times for multi-file changes" + type: shell + command: "jq -e '[.messages[]?.tool_calls[]? | select(.function.name == \"patch\")] | length >= 3' {{dir}}/context.json" +sources: + - value: + - model: "openai/gpt-5.2" + - csv: multi_file_patch_tasks.csv diff --git a/benchmarks/evals/patch_exact_match/patch_exact_match_tasks.csv b/benchmarks/evals/patch_exact_match/patch_exact_match_tasks.csv new file mode 100644 index 0000000000000000000000000000000000000000..fdaeccc3457b8fe543c52cfaae5f1c9f4ae24a7d --- /dev/null +++ b/benchmarks/evals/patch_exact_match/patch_exact_match_tasks.csv @@ -0,0 +1,7 @@ +task +"Remove all backticks from test_files/plan.md. Make multiple separate patches to remove backticks from different sections." +"Update test_files/plan.md to remove markdown code formatting. Apply patches to remove backticks from each task rationale." +"Fix test_files/plan.md by converting backtick-wrapped identifiers. Use patch for each occurrence." +"Clean up test_files/plan.md by removing backticks. Patch each task description separately." +"In test_files/plan.md, convert backtick-enclosed terms. Use patch operations for different lines." +"Remove markdown code formatting (backticks) from test_files/plan.md. Apply multiple patches." diff --git a/benchmarks/evals/patch_exact_match/task.yml b/benchmarks/evals/patch_exact_match/task.yml new file mode 100644 index 0000000000000000000000000000000000000000..ab9d069a901b623511b102a5cb76558e1db2ffc5 --- /dev/null +++ b/benchmarks/evals/patch_exact_match/task.yml @@ -0,0 +1,217 @@ +run: + - mkdir -p test_files + - | + cat > test_files/plan.md << 'EOF' + # Support for Remote Tools (Provider-Executed Tools) + + ## Objective + + Enable support for remote tools (also known as provider-executed tools) such as websearch, which are executed by the AI provider's infrastructure rather than by the client application. This allows providers like OpenAI to offer specialized capabilities like web search, code execution, and file retrieval that run on their servers and return results directly to the application. + + ## Implementation Plan + + - [ ] 1. Extend the ToolDefinition structure to support provider-executed tools + Add a new field to the `ToolDefinition` struct to distinguish between client-executed tools (current behavior) and provider-executed tools (new capability). This field will indicate that the tool should be passed directly to the provider without local execution logic. + + Rationale: The current `ToolDefinition` at `crates/forge_domain/src/tools/definition/tool_definition.rs:12` only contains name, description, and input_schema. To support provider-executed tools, we need a way to mark which tools are handled by the provider infrastructure. This is similar to how Vercel AI SDK distinguishes between `function` type tools and `provider` type tools. + + Affected files: + - `crates/forge_domain/src/tools/definition/tool_definition.rs` - Add field to ToolDefinition + - `crates/forge_repo/src/conversation/conversation_record.rs` - Update serialization/deserialization + + - [ ] 2. Create provider-specific tool definitions for OpenAI Responses API + Define a set of provider-executed tools for OpenAI's Responses API, including `web_search`, `code_interpreter`, `file_search`, and other provider-native capabilities. These will be represented as `ToolDefinition` instances marked as provider-executed. + + Rationale: OpenAI's Responses API provides built-in tools like `web_search` that are executed by OpenAI's infrastructure. These tools need to be defined with their specific schemas and parameters (e.g., searchContextSize, userLocation for web_search) similar to how Vercel AI SDK's `prepareResponsesTools` function handles these tools. This allows the application to offer these capabilities without implementing the execution logic. + + Affected files: + - `crates/forge_app/src/dto/openai/provider_tools.rs` - New file for provider tool definitions + - `crates/forge_app/src/dto/openai/mod.rs` - Export new module + + - [ ] 3. Implement tool conversion logic for provider-executed tools + Create a conversion function that transforms provider-executed `ToolDefinition` instances into the provider-specific format required by the API. For OpenAI, this means converting provider tools to the appropriate structure (e.g., `{type: "web_search", ...}`) instead of the standard function tool format. + + Rationale: The current `From for Tool` implementation at `crates/forge_app/src/dto/openai/request.rs:297` converts all tools to the standard function format. Provider-executed tools need special handling to be converted to their provider-specific formats. For OpenAI, this means converting provider tools to the appropriate structure with a type field indicating the specific tool type instead of using the standard function tool format. This mirrors Vercel AI SDK's `prepareResponsesTools` approach where provider tools are handled differently from function tools. + + Affected files: + - `crates/forge_app/src/dto/openai/request.rs` - Modify Tool conversion logic + - `crates/forge_app/src/dto/openai/provider_tools.rs` - Implement conversion functions + + - [ ] 4. Extend the Context to Tool request conversion + Update the `From for Request` implementation at `crates/forge_app/src/dto/openai/request.rs:323` to handle provider-executed tools. This involves filtering and converting provider tools separately from standard function tools when building the request. + + Rationale: The current implementation treats all tools uniformly and converts them using the same logic. Provider-executed tools need to be identified and converted using the new provider-specific conversion logic before being included in the API request. This ensures that tools like `web_search` are sent in the correct format to OpenAI. + + Affected files: + - `crates/forge_app/src/dto/openai/request.rs` - Update Context to Request conversion + + - [ ] 5. Update the tool registry to handle provider-executed tool results + Modify the `ToolRegistry` at `crates/forge_app/src/tool_registry.rs:21` to recognize that provider-executed tools do not require local execution. When a provider tool is invoked, the tool registry should pass the result directly from the provider response without attempting local execution. + + Rationale: Currently, the tool registry attempts to execute all tools locally through `ToolExecutor`, `AgentExecutor`, or `McpExecutor`. Provider-executed tools are different - the provider executes them and returns the result. The tool registry needs to distinguish between these cases and handle provider tool results appropriately. + + Affected files: + - `crates/forge_app/src/tool_registry.rs` - Add logic to skip execution for provider tools + - `crates/forge_app/src/tool_registry.rs` - Handle provider tool results from responses + + - [ ] 6. Add provider tool configuration to agent definitions + Extend the agent definition to include optional provider tool configurations. This allows agents to be configured with specific provider-executed tools (e.g., `web_search` enabled with specific parameters) while maintaining compatibility with existing agent configurations. + + Rationale: Different agents may require different provider tools. For example, a research agent might need `web_search` enabled, while a coding agent might need `code_interpreter`. Adding this configuration to agent definitions provides fine-grained control over which provider tools are available to each agent. + + Affected files: + - `crates/forge_domain/src/agent_definition.rs` - Add provider tools configuration + - `crates/forge_app/src/dto/tools_overview.rs` - Include provider tools in overview + + - [ ] 7. Create provider tool factory and registration + Implement a factory pattern for registering provider tools with the system. This factory will create provider tool definitions for different providers (OpenAI, Anthropic, etc.) and make them available to agents that request them. + + Rationale: Different providers support different provider-executed tools. A factory pattern allows the system to create the appropriate tool definitions based on the provider being used. This is extensible - new providers can be added by implementing their tool factory without modifying core logic. + + Affected files: + - `crates/forge_app/src/provider_tools/factory.rs` - New file for tool factory + - `crates/forge_app/src/provider_tools/mod.rs` - New module + - `crates/forge_app/src/dto/tools_overview.rs` - Integrate with factory + + - [ ] 8. Update the response parsing to handle provider tool results + Modify the response parsing logic in `crates/forge_app/src/dto/openai/response.rs` to correctly parse and handle tool results from provider-executed tools. This includes recognizing when a tool result comes from a provider tool and ensuring it's properly formatted. + + Rationale: Provider tool results may have different structures than standard tool results. The response parser needs to handle these differences and convert them to the internal `ToolResult` format consistently. This ensures that tool results are processed correctly regardless of whether they came from a client-executed or provider-executed tool. + + Affected files: + - `crates/forge_app/src/dto/openai/response.rs` - Update tool result parsing + - `crates/forge_app/src/dto/openai/response.rs` - Handle provider-specific result formats + + - [ ] 9. Add provider tool support to the tool overview API + Update the `ToolsOverview` structure and related API endpoints to include provider-executed tools alongside system tools, agent tools, and MCP tools. This allows clients to discover which provider tools are available. + + Rationale: The current `ToolsOverview` at `crates/forge_app/src/dto/tools_overview.rs:9` includes system tools, agents, and MCP tools. Provider tools should be included in this overview so clients can see all available tools, including those executed by providers. This maintains consistency with the existing tool discovery mechanism. + + Affected files: + - `crates/forge_app/src/dto/tools_overview.rs` - Add provider tools field + - `crates/forge_app/src/tool_registry.rs` - Update tools_overview method + + - [ ] 10. Implement Anthropic provider tools support + Create provider tool definitions for Anthropic's provider-executed tools (e.g., `computer_20241022`, `textEditor_20241022`, `bash_20241022`, `webSearch_20250305`, `codeExecution_20250522`). This demonstrates the extensibility of the provider tool system to multiple providers. + + Rationale: Anthropic offers provider-executed tools similar to OpenAI but with different names and capabilities. Implementing support for Anthropic provider tools validates that the architecture is provider-agnostic and extensible. It follows the same pattern established for OpenAI tools. + + Affected files: + - `crates/forge_app/src/dto/anthropic/provider_tools.rs` - New file for Anthropic provider tools + - `crates/forge_app/src/dto/anthropic/request.rs` - Update conversion logic + - `crates/forge_app/src/provider_tools/factory.rs` - Add Anthropic tool factory + + - [ ] 11. Add tests for provider tool conversion and execution + Create comprehensive tests covering provider tool definition, conversion to provider formats, integration with Context to Request conversion, and handling of provider tool results. Tests should verify that provider tools are correctly formatted and that client-executed tools remain unaffected. + + Rationale: Provider tools introduce new code paths and conversion logic that need thorough testing. Tests should ensure that provider tools are correctly converted to their provider-specific formats, that they don't interfere with standard tool execution, and that the system handles mixed scenarios (both provider and client tools) correctly. + + Affected files: + - `crates/forge_app/src/dto/openai/request.rs` - Add conversion tests + - `crates/forge_app/src/dto/anthropic/request.rs` - Add conversion tests + - `crates/forge_app/src/tool_registry.rs` - Add execution handling tests + + - [ ] 12. Update documentation and examples + Document the provider tool system, including how to define provider tools, how they are converted to provider formats, and how agents can be configured to use them. Include examples of using `web_search` and other provider tools. + + Rationale: Provider tools are a new concept in the system. Documentation is essential for developers to understand when and how to use provider-executed tools versus client-executed tools. Examples should demonstrate practical usage patterns and configuration options. + + Affected files: + - `docs/provider-tools.md` - New documentation file + - `examples/` - Add example configurations + + ## Verification Criteria + + - [ ] Provider tool definitions can be created and marked as provider-executed + - [ ] Provider tools are correctly converted to OpenAI-specific format (for example, using the type field to specify web search instead of function format) + - [ ] Provider tools are correctly converted to Anthropic-specific format + - [ ] Context to Request conversion includes provider tools in the correct format + - [ ] Tool registry recognizes provider tools and skips local execution + - [ ] Provider tool results from API responses are correctly parsed and handled + - [ ] Agent definitions can be configured with provider tools + - [ ] Tool overview API includes provider tools alongside other tool types + - [ ] All existing tests pass without modification + - [ ] New tests cover provider tool conversion, execution, and result handling + - [ ] Documentation describes the provider tool system with examples + + ## Potential Risks and Mitigations + + 1. **Breaking changes to existing tool definitions** + Risk: Adding a new field to `ToolDefinition` could break existing serialization/deserialization or require migrations. + + Mitigation: Make the new field optional with a default value of `false` (client-executed). This ensures backward compatibility with existing tool definitions. Run all existing tests to verify no regressions. + + 2. **Provider tool format incompatibility** + Risk: Different providers may have incompatible tool formats, making it difficult to create a unified abstraction. + + Mitigation: Use provider-specific conversion modules (e.g., `openai_provider_tools`, `anthropic_provider_tools`) that handle format conversion independently. The common `ToolDefinition` serves as an abstraction, but each provider has its own conversion logic. + + 3. **Tool result handling complexity** + Risk: Provider tool results may have different structures that are difficult to normalize into a common `ToolResult` format. + + Mitigation: Create provider-specific result parsers that normalize results to the internal format. Handle unknown or unexpected result formats gracefully with appropriate error messages. + + 4. **Execution model confusion** + Risk: Developers may be confused about which tools are executed locally versus by the provider, leading to unexpected behavior. + + Mitigation: Clearly document the distinction in the `ToolDefinition` structure and provide tool overview information that indicates execution location (client vs. provider). Use descriptive naming conventions. + + 5. **Testing complexity** + Risk: Testing provider tools requires mocking provider responses, which can be complex and brittle. + + Mitigation: Create integration tests that use real provider APIs for critical paths, supplemented by unit tests with mocked responses. Use snapshot testing to verify correct request formatting. + + ## Alternative Approaches + + 1. **Separate tool type enum instead of boolean flag** + Instead of adding a boolean field to `ToolDefinition`, create an enum `ToolExecutionType` with variants `Client` and `Provider`. This is more explicit and extensible, allowing future execution types (e.g., `Hybrid`, `Delegated`). + + Trade-offs: More explicit and type-safe, but adds complexity. The boolean flag is simpler for the current use case and can be extended to an enum later if needed. + + 2. **Provider-specific tool registries** + Create separate tool registries for each provider (OpenAI, Anthropic, etc.) rather than a unified registry. Each provider would manage its own tools, including provider-executed tools. + + Trade-offs: Cleaner separation of concerns and easier provider-specific logic, but increases code duplication and makes it harder to share tools across providers. + + 3. **Dynamic tool discovery from provider APIs** + Instead of hard-coding provider tool definitions, query provider APIs to discover available tools and their schemas dynamically. This would automatically support new provider tools as they are added. + + Trade-offs: More flexible and future-proof, but introduces runtime dependencies on provider APIs and may be slower. Requires robust error handling for API failures. + + 4. **Tool execution delegation pattern** + Implement a delegation pattern where tools specify an executor (client, provider, MCP, etc.) through a trait. The tool registry would delegate execution based on the executor specified by the tool. + + Trade-offs: More flexible and extensible architecture, but adds abstraction layers and complexity. The current approach of checking tool type is simpler for the immediate use case. + EOF + - | + FORGE_DEBUG_REQUESTS='{{dir}}/context.json' forgee --provider open_router --model {{model}} -p '{{task}}' +parallelism: 2 +timeout: 180 +early_exit: true +validations: + - name: "Should use patch tool" + type: shell + command: "jq -e '[.messages[]?.tool_calls[]? | select(.function.name == \"patch\")] | length > 0' {{dir}}/context.json" + + - name: "Should NOT have patch failures due to missing operation field" + type: shell + command: "! jq -e '[.messages[]? | select(.role == \"tool\" and .name == \"patch\" and (.content | contains(\"missing field\") and contains(\"operation\")))] | length > 0' {{dir}}/context.json" + + - name: "Should NOT have patch failures due to text mismatch" + type: shell + command: "! jq -e '[.messages[]? | select(.role == \"tool\" and .name == \"patch\" and (.content | contains(\"Could not find match for search text\")))] | length > 0' {{dir}}/context.json" + + - name: "Patch search text should match file content exactly" + type: shell + command: | + # Extract all patch tool calls and verify search text exists in the file + # This is a heuristic check - we look for patch calls that should have succeeded + jq -e '[.messages[]?.tool_calls[]? | select(.function.name == "patch")] | length > 0' {{dir}}/context.json + +sources: + - value: + - model: "anthropic/claude-sonnet-4.5" + - model: "z-ai/glm-4.6:exacto" + - model: "minimax/minimax-m2.1" + + - csv: patch_exact_match_tasks.csv diff --git a/benchmarks/evals/redundant_cd_with_cwd/task.yml b/benchmarks/evals/redundant_cd_with_cwd/task.yml new file mode 100644 index 0000000000000000000000000000000000000000..74adb9b3980b3aebf085b97dd2ab09115e1659a7 --- /dev/null +++ b/benchmarks/evals/redundant_cd_with_cwd/task.yml @@ -0,0 +1,45 @@ +run: + - git clone --depth=1 --branch main https://github.com/tailcallhq/forgecode . + - mkdir ../test-dir && cd ../test-dir + - FORGE_DEBUG_REQUESTS='{{dir}}/context.json' forgee --provider open_router --model {{model}} -p '{{task}}' +parallelism: 5 +timeout: 120 +early_exit: false # We want to see all cases, not just the first success + +validations: + # Check if any shell tool call contains "cd" command when cwd is also provided + - name: "No redundant cd with cwd" + type: shell + command: | + jq -e ' + [.messages[]?.tool_calls[]? | + select(.function.name == "shell") | + .function.arguments | fromjson | + select((.command | contains("cd "))) | + {command: .command, cwd: .cwd} + ] | length == 0 + ' {{dir}}/context.json + + # Detect specific pattern: "cd && " with cwd set + - name: "No cd && pattern with cwd" + type: shell + command: | + jq -e ' + [.messages[]?.tool_calls[]? | + select(.function.name == "shell") | + .function.arguments | fromjson | + select((.command | test("cd\\s+[^;]+\\s*&&"))) | + {command: .command, cwd: .cwd, issue: "redundant_cd_with_cwd"} + ] | length == 0 + ' {{dir}}/context.json + +sources: + - value: + - model: "anthropic/claude-sonnet-4.5" + - model: "z-ai/glm-4.6:exacto" + - value: + - task: "clean ~/tailcall" + - task: "format code in ~/task/crates/forge_domain" + - task: "install dependencies in ~/task/benchmarks" + - task: "cd to ~/task/crates and run cargo test" + - task: "go to ~/task/docs and list files" diff --git a/benchmarks/evals/search_over_find/task.yml b/benchmarks/evals/search_over_find/task.yml new file mode 100644 index 0000000000000000000000000000000000000000..9f1dd61e297b8a1cb9a49e9883de673133fd05bb --- /dev/null +++ b/benchmarks/evals/search_over_find/task.yml @@ -0,0 +1,58 @@ +run: + - git clone --depth=1 --branch main https://github.com/tailcallhq/forgecode . + - FORGE_DEBUG_REQUESTS='{{dir}}/context.json' forgee --provider open_router --model {{model}} -p '{{task}}' +parallelism: 10 +timeout: 180 +early_exit: true + +validations: + # Primary: Model should NOT use shell find command for searching code + - name: "No shell find command usage" + type: shell + command: | + jq -e ' + [.messages[]?.tool_calls[]? | + select(.function.name == "shell") | + .function.arguments | fromjson | + select(.command | test("\\bfind\\s+")) + ] | length == 0 + ' {{dir}}/context.json + + # Model SHOULD use builtin search tools instead + - name: "Uses builtin search tools" + type: shell + command: | + jq -e ' + [.messages[]?.tool_calls[]? | + select(.function.name == "search" or + .function.name == "fs_search" or + .function.name == "sem_search") + ] | length > 0 + ' {{dir}}/context.json + + # Detect specific anti-pattern: find with grep pipeline + - name: "No find + grep anti-pattern" + type: shell + command: | + jq -e ' + [.messages[]?.tool_calls[]? | + select(.function.name == "shell") | + .function.arguments | fromjson | + select(.command | test("find.*\\|.*grep")) + ] | length == 0 + ' {{dir}}/context.json + +sources: + - value: + - model: "anthropic/claude-sonnet-4.5" + - model: "z-ai/glm-4.6:exacto" + - value: + # File discovery tasks - should use fs_search, NOT find command + - task: "I need to merge service and app crates. Tell me what files exist in the crates/ directory." + - task: "Find all Rust source files in the crates/service directory." + - task: "List all Rust files in crates/app to understand the structure." + - task: "Show me all .rs files in the project so I can plan refactoring." + + # Structure understanding - should use fs_search or sem_search + - task: "Find all source files under crates/service and crates/app for analysis." + - task: "List all Rust source files in the workspace for documentation purposes." diff --git a/benchmarks/evals/sem_search/README.md b/benchmarks/evals/sem_search/README.md new file mode 100644 index 0000000000000000000000000000000000000000..f7b4dc9d369302123f08a3bd29f8953600667698 --- /dev/null +++ b/benchmarks/evals/sem_search/README.md @@ -0,0 +1,33 @@ +# Semantic Search Tool Selection Evaluation + +This evaluation validates that the Forge agent correctly identifies and uses the `sem_search` (Codebase Search) tool when presented with conceptual, functionality-based code location queries. + +## What We're Testing + +The agent's ability to recognize when a user query requires semantic understanding rather than exact pattern matching. Specifically, we test whether the agent: + +1. **Correctly invokes `sem_search`** for conceptual queries about code functionality +2. **Understands the distinction** between semantic search (conceptual) vs regex search (exact patterns) +3. **Recognizes appropriate use cases** such as: + - Finding code by its purpose/behavior (e.g., "retry logic with exponential backoff") + - Locating implementation patterns (e.g., "authentication token validation") + - Discovering architectural components (e.g., "message transformation between AI providers") + - Identifying system behaviors (e.g., "rate limiting for API requests") + +## Test Scenarios + +The evaluation uses real-world queries that describe: +- **Message transformation** between different AI provider formats +- **Retry mechanisms** with exponential backoff patterns +- **Authentication flows** including token validation and refresh +- **Error handling** for network failures +- **Caching strategies** for API responses +- **Streaming response** handling from LLM APIs +- **File validation** including upload and size checks +- **Rate limiting** implementations for API requests +- **Tool registration** and availability management +- **Context management** including conversation history truncation + +## Expected Behavior + +For all test queries, the agent should invoke the `sem_search` tool (displayed as "Codebase Search" in logs) rather than falling back to regex-based `search` tool, demonstrating understanding that these are conceptual queries requiring semantic code comprehension. diff --git a/benchmarks/evals/sem_search/task.yml b/benchmarks/evals/sem_search/task.yml new file mode 100644 index 0000000000000000000000000000000000000000..96fe117ebc3c760bba58b0be221431e61b96c34f --- /dev/null +++ b/benchmarks/evals/sem_search/task.yml @@ -0,0 +1,138 @@ +run: + # Clone into `tmp/task` dir + - git clone --depth=1 --branch main https://github.com/tailcallhq/forgecode . + - forge workspace init --yes + - forge workspace sync + - FORGE_DEBUG_REQUESTS='{{dir}}/context.jsonl' FORGE_SESSION__PROVIDER_ID=open_router FORGE_SESSION__MODEL_ID={{model}} forge -p '{{task}}' +parallelism: 8 +timeout: 60 +early_exit: true +validations: + - name: "Uses codebase search tool" + type: shell + command: grep -q "Codebase Search" '{{dir}}/task.log' + - name: "Does not call task tool before sem_search" + type: shell + command: "python3 -c \"\nimport sys\nlog = open('{{dir}}/task.log').read().splitlines()\ntask_line = next((i for i, l in enumerate(log) if '[Agent]' in l), None)\nsearch_line = next((i for i, l in enumerate(log) if 'Codebase Search' in l), None)\nif task_line is None: sys.exit(0)\nif search_line is None: sys.exit(1)\nsys.exit(0 if search_line < task_line else 1)\n\"" +sources: + - value: + # - model: "x-ai/grok-code-fast-1" + - model: "anthropic/claude-sonnet-4.6" + - value: + # # Location questions - "where is" + # - task: "Where is the code that transforms messages between different AI provider formats?" + # - task: "Where is the error handling code for network failures?" + # - task: "Where is file upload validation and size checking implemented?" + # - task: "Where are environment variables parsed and configuration loaded?" + # - task: "Where is the logic for selecting which AI model to use for a task?" + + # # Discovery questions - "find" + # - task: "Find the code that caches API responses to reduce costs" + # - task: "Find where tool definitions are registered and made available to the AI" + # - task: "Find the code that manages API keys, tokens, and credentials securely" + # - task: "Find where Git operations like commit, diff, and status are implemented" + + # # Understanding questions - "how does" + # - task: "How does the system handle differences between OpenAI and Anthropic APIs?" + # - task: "How does the tool execution pipeline process and validate tool calls?" + # - task: "How are streaming responses from LLMs processed?" + + # # Need-based questions - "I need to" + # - task: "I need to find retry logic with exponential backoff for API calls" + # - task: "I need to understand how streaming responses from LLM APIs work" + # - task: "I need to see how conversation history is truncated when context is too large" + # - task: "I need to find examples of rate limiting implementations" + + # # Show me questions + # - task: "Show me where authentication tokens are validated and refreshed" + # - task: "Show me the code that handles rate limiting for API requests" + # - task: "Show me the code that compacts conversation context when tokens exceed limits" + # - task: "Show me how the system handles request timeouts and cancellation" + # - task: "Show me the permission checking and security policy enforcement code" + + # # Strategy/pattern questions + # - task: "I want to understand the caching strategy for semantic search embeddings" + # - task: "What's the strategy for managing concurrent API requests?" + # - task: "What's the approach for handling API errors and retries?" + + # # Troubleshooting questions + # - task: "What could cause API requests to fail?" + # - task: "How are timeout errors handled in the system?" + + # # Architecture questions + # - task: "How do the different AI provider adapters integrate with the main system?" + # - task: "What's the flow of a user request from input to response?" + # - task: "How are tools registered and executed?" + + # # Capability questions + # - task: "How does the system track token usage?" + # - task: "What mechanism prevents infinite loops in tool execution?" + + # Human review tasks - integrated from human-review-sem-search-tasks.json + - task: "Now that we support piping, how can I use an external editor to type a large command and pipe it to the forge cli?" + - task: "Is there a type-safe version of commander that can make sure the command is parsed into a type-safe object" + - task: "Search for ZAI-related model fetching code in the codebase" + - task: "Look for API endpoints and authentication patterns used for ZAI" + - task: "Starting a new session always loads the older node version" + - task: "Examine any existing curl examples or HTTP client implementations" + - task: "Before sorting the tools we will dedupe all the tools" + - task: "Identify the current embedding implementation and how it interfaces with OpenRouter" + - task: "Is there a place in the code base where we are subscribing to events?" + - task: "There is a bug, where when I paste text the syntax highlighting is lost for a moment until some key is pressed again." + - task: "Search for any existing validation patterns in the forge_domain crate, particularly looking for validation methods, validate functions, or error types used for domain validation" + - task: "Show me the temperature type" + - task: "Search for cancellation mechanisms including cancellation tokens, abort signals, task cancellation, and how the system handles user interrupts or cancellation of in-flight requests across all crates" + - task: "In the workspace info, we should return the creation time of the workspace too." + - task: "token cost is being sent but we are not able to show it on the ui. Our cost conversation in some place is getting missed." + - task: "Look at workflow-related files to understand how model should be obtained from API" + - task: "Lets use the newtype crate for all the type in domain that are newtypes so that we don't need to implement all this boilerplate" + - task: "Look for any existing patterns in the codebase for ensuring migrations run exactly once across all tests, such as global mutexes or initialization guards" + - task: "The clear conversation doesn't happen on CMD+K." + - task: "Use plimit and prompt.all to run it concurrently" + - task: "Like infra I need another crate called - test_infra that looks similar and has a ForgeInfra struct that has test implementations of stuff." + - task: "Infra query nodes should take Query instead of each param as a separate arg." + - task: "Add health check endpoints for monitoring" + - task: "Examine conversation ID generation and management in domain layer" + - task: "Lets recursively derive Dummy from fake, on Node and its children." + - task: "Add a new infra component called Reranker. It takes a list of nodes and reranks them." + - task: "Update the implementations of Cache based on the new trait implementation." + - task: "In skill lets add description also" + - task: "Investigate signal handling and graceful shutdown mechanisms in /Volumes/Bran/code-forge-workspace/code-forge, particularly looking for SIGINT, SIGTERM handling and how they affect ongoing operations" + - task: "Lets add a new range.rs under arbitrary/ and implement Arbitrary for all the std Range types eg: Range, RangeFrom etc." + - task: "Make `-v` optional. It should not be required." + - task: "in ProviderRegistry I need a few more methods - get_active_provider, set_active_provider and get_all_providers" + - task: "Lets use derive_setters on Config. Drop the handwritten setters" + - task: "Lets drop path and env from the `info` command. Its a bit too noisy." + - task: "The Secret type should print the last 4 chars of the secret in debug." + - task: "What's the purpose of ModelId?" + - task: "Where is the Agent struct defined?" + - task: "**Questions**\n\n1. Switching agent happens via CLI `forge config --set-agent muse` or `forge config -i` opens up an interactive configuration manager (RATATUI)\n2. How do I know the active agent and model? `forge config --get-agent` show in via a ZSH theme\n3. Additional CLI commands to be added\n 1. `forge conversation --set-active `\n 2. `forge info` Show current info including agent, model etc." + - task: "Find all places where event names are constructed or used, particularly in relation to agent IDs. Look for patterns like 'forge', 'muse', 'sage' in event names and how they map to agents." + - task: "Change the structure to be a simple - Vec of Vec of Option of String." + - task: "Where are the fixtures located?" + - task: "Move McpToolCache to forge_app dto" + - task: "based on your understanding of the project, where are the tables created in supbase?" + - task: "Examine how the truncation functions are called and where the results are used. Identify the call sites and understand the data flow from truncation to final output." + - task: "Like ShowCommands, I need ShowModels and ShowConversations" + - task: "Lets implement a function on SummaryMessage that can merge consecutive calls together. If the nth can be merged with nth-1, then we should remove the nth-1 and insert the new value." + - task: "Lets have upload take a glob pattern for upload" + - task: "Check this for security issues: eval(user_input)" + - task: "Update the vector dimension to be 4096" + - task: "Fix this - 'docker run tailcallhq/forgecode'" + - task: "Document the adapter registration and discovery mechanism" + - task: "There are multiple places where we throw - \"Conversation not found\" how do u suggest we consolidate and reuse." + - task: "Is there a crate for humanizing date time format that we are using?" + - task: "Lets move the domain compaction transformers into app/compact/transformers. Make sure you move them in bulk." + - task: "Identify the main entry point (main.rs or equivalent) and understand how the application initializes and starts the server." + - task: "Lets use chrono-humanize instead of humantime in forge_main everywhere." + - task: "Look at all the docs in .rs files and remove the args and return info from the docs." + - task: "Investigate why each approach (1-4) has failed - what are the technical limitations we're hitting with ZLE widgets, terminal control sequences, and shell execution contexts" + - task: "as per rust best practices, what would you recommend should we call arbit? and how should the file be structured. Show me the best examples - purest form" + - task: "Explain the flow of authentication in the project. I need to understand it deeply, going from high level to micro details." + - task: "Research xml.rs in forge_domain and create a plan to add helper functions for common XML operations" + - task: "Add JWT-based authentication system" + - task: "How does FinishReason work?" + - task: "Check what changes were made to api_key.rs conversions and identify why the key field might be getting the wrong value" + - task: "Trace the flow from skill listing to system prompt generation to identify where the integration is missing or broken" + - task: "What could cause memory leaks in long-running sessions?" + - task: "I am not able to make send of these name - DefinitionBatchGenerator DefinitionBatch. I need something much better," diff --git a/benchmarks/evals/semantic_search_quality/README.md b/benchmarks/evals/semantic_search_quality/README.md new file mode 100644 index 0000000000000000000000000000000000000000..2e4cf555a38d200ffaaf7b0f829a8fdd974861a0 --- /dev/null +++ b/benchmarks/evals/semantic_search_quality/README.md @@ -0,0 +1,197 @@ +# Semantic Search Quality Evaluation + +This evaluation measures the quality of semantic search queries generated by Forge using an LLM-as-judge approach. + +## Quick Start + +### Test Your Queries Manually + +Before running the full eval, test your queries to get immediate feedback: + +```bash +# Interactive mode +npx tsx benchmarks/evals/semantic_search_quality/test_queries.ts + +# Non-interactive mode +npx tsx benchmarks/evals/semantic_search_quality/test_queries.ts \ + "retry mechanism with exponential backoff" \ + "I need to find the implementation code for retry logic" +``` + +The query tester provides: +- Detailed scores for embedding and reranking queries +- Specific strengths and weaknesses +- Actionable suggestions for improvement +- Differentiation analysis + +### Run the Full Evaluation + +```bash +# Run the entire semantic search quality eval +npm run eval semantic_search_quality + +# Run with specific parallelism +npm run eval semantic_search_quality -- --parallelism 10 + +# Run with specific model +npm run eval semantic_search_quality -- --model "anthropic/claude-sonnet-4.5" +``` + +## Purpose + +The semantic search feature uses two types of queries: +1. **Embedding Query (`query`)**: Converted to vector embeddings for semantic similarity search +2. **Reranking Query (`use_case`)**: Used by the reranker to filter results based on intent + +This eval verifies that: +- Queries are well-formed and contain sufficient context +- The two queries are different and serve distinct purposes +- Results match the user's intent (implementation vs docs vs tests, etc.) +- Correct file types are returned based on intent +- Unwanted file types are avoided + +## How It Works + +### 1. Test Execution +The eval runs various semantic search tasks covering different intents: +- **Implementation**: Find actual code, not documentation +- **Flow Understanding**: Understand how systems work +- **Architecture**: Understand structural design +- **Tests**: Find test examples and fixtures +- **Documentation**: Find docs and guides +- **Debugging**: Investigate issues +- **Modification**: Find code to modify +- **Configuration**: Find config files and schemas + +### 2. LLM Judge Evaluation +After each task, an LLM judge (Gemini 3 Pro) evaluates: + +#### Query Quality (40 points) +- **Embedding Query (15pts)**: Domain terms, technical context, behavior description +- **Reranking Query (15pts)**: Intent clarity, context about WHY, specificity +- **Query Differentiation (10pts)**: Queries serve different purposes + +#### Result Relevance (60 points) +- **Intent Matching (25pts)**: Results match the stated intent +- **File Type Accuracy (20pts)**: Correct file types returned +- **Avoidance Compliance (15pts)**: Unwanted file types avoided + +#### Pass Criteria +- Total score >= 70/100 +- No critical issues (e.g., identical queries, severe intent mismatch) + +### 3. Structured Output +The judge uses structured output (Zod schema) to provide: +- Detailed scores for each dimension +- Specific feedback on what's good/bad +- List of critical issues +- Overall pass/fail determination + +## Environment Variables + +Required: +- `GOOGLE_APPLICATION_CREDENTIALS`: Path to Google Cloud service account JSON credentials file + +Or authenticate using `gcloud auth application-default login`. + +The LLM judge uses Vertex AI to run Gemini 3 Pro. It supports Google's Application Default Credentials through the `google-auth-library`. The most common authentication method is to set the path to a JSON credentials file in the `GOOGLE_APPLICATION_CREDENTIALS` environment variable. + +## Test Cases + +The eval includes diverse test cases: + +### Implementation Queries +```yaml +- task: "Find the retry mechanism with exponential backoff implementation" + intent: "implementation" + expected_file_types: "rust,typescript,javascript" + should_avoid: "markdown,txt,documentation" +``` + +### Flow Understanding +```yaml +- task: "How does the authentication flow work from request to response?" + intent: "flow_understanding" + expected_file_types: "rust,typescript" + should_avoid: "test,markdown" +``` + +### Documentation +```yaml +- task: "Find documentation on how to configure semantic search" + intent: "documentation" + expected_file_types: "markdown,txt" + should_avoid: "rust,typescript,javascript" +``` + +## Interpreting Results + +### High Scores (>80) +Queries are excellent - domain-specific, well-differentiated, and would return highly relevant results. + +### Medium Scores (60-80) +Queries are functional but have room for improvement. Check feedback for specific suggestions. + +### Low Scores (<60) +Queries have significant issues: +- Too generic or vague +- Embedding and reranking queries too similar +- Intent mismatch +- Would return wrong file types + +## Common Issues + +1. **Identical Queries**: Embedding and reranking queries are the same or too similar + - **Fix**: Make reranking query focus on intent, not just rephrasing + +2. **Generic Queries**: Lack domain-specific context + - **Fix**: Add technical terms, describe behavior, include framework names + +3. **Intent Mismatch**: Queries would return docs when implementation is needed + - **Fix**: Be explicit in reranking query about what type of code is needed + +4. **Poor Differentiation**: Both queries try to do the same thing + - **Fix**: Embedding = WHAT code does, Reranking = WHY you need it + +## Architecture + +``` +task.yml +โ”œโ”€โ”€ Runs Forge with semantic search task +โ”œโ”€โ”€ Captures debug output (context.json) +โ””โ”€โ”€ Validates with llm_judge.ts + โ”œโ”€โ”€ Extracts sem_search calls + โ”œโ”€โ”€ Calls Gemini 3 Pro with structured output + โ””โ”€โ”€ Returns evaluation with scores +``` + +## Extending the Eval + +To add new test cases: + +1. Add to `task.yml` under `sources`: +```yaml +- task: "Your task description" + intent: "implementation|flow_understanding|tests|documentation|debugging|modification|architecture|configuration" + expected_file_types: "comma,separated,extensions" + should_avoid: "comma,separated,extensions" +``` + +2. Choose appropriate intent: + - `implementation`: Actual code implementation + - `flow_understanding`: Understanding how things work + - `architecture`: Structural/design understanding + - `tests`: Test files and fixtures + - `documentation`: Docs, README, guides + - `debugging`: Finding bugs/issues + - `modification`: Code to modify/extend + - `configuration`: Config files, schemas + +## Future Improvements + +- [ ] Add actual result verification (not just query quality) +- [ ] Measure embedding quality directly +- [ ] Test reranking effectiveness with real results +- [ ] Add precision/recall metrics +- [ ] Compare against human-labeled ground truth +- [ ] Test edge cases (very short queries, multi-lingual code, etc.) diff --git a/benchmarks/evals/semantic_search_quality/llm_judge.ts b/benchmarks/evals/semantic_search_quality/llm_judge.ts new file mode 100644 index 0000000000000000000000000000000000000000..d36c224bf8bc940634cb4a89f1ef674137f17a27 --- /dev/null +++ b/benchmarks/evals/semantic_search_quality/llm_judge.ts @@ -0,0 +1,400 @@ +#!/usr/bin/env tsx + +/** + * LLM-as-Judge for Semantic Search Quality Evaluation + * + * This script evaluates the quality of semantic search queries and results + * using Google's Gemini 3 Pro via Vertex AI as a judge. It verifies: + * 1. Query quality - Are the embedding and reranking queries well-formed? + * 2. Result relevance - Do the results match the stated intent? + * 3. File type accuracy - Are the correct file types returned? + * 4. Avoidance compliance - Are unwanted file types properly avoided? + */ + +import { readFileSync } from "fs"; +import { vertex } from "@ai-sdk/google-vertex"; +import { generateObject } from "ai"; +import { z } from "zod"; + +// Evaluation result schema +const EvaluationSchema = z.object({ + query_quality: z.object({ + embedding_query_score: z + .number() + .min(0) + .max(15) + .describe("Score for the embedding query quality (0-15)"), + embedding_query_feedback: z + .string() + .describe("Specific feedback on the embedding query"), + reranking_query_score: z + .number() + .min(0) + .max(15) + .describe("Score for the reranking query quality (0-15)"), + reranking_query_feedback: z + .string() + .describe("Specific feedback on the reranking query"), + construct_keywords_score: z + .number() + .min(0) + .max(10) + .describe( + "Score for codebase construct keywords in reranking query (0-10)", + ), + construct_keywords_feedback: z + .string() + .describe( + "Feedback on use of codebase constructs (struct, trait, impl, interface, class, function, definition, implementation, declaration)", + ), + queries_differentiated: z + .boolean() + .describe( + "Whether the embedding and reranking queries are sufficiently different", + ), + }), + result_relevance: z.object({ + intent_match_score: z + .number() + .min(0) + .max(25) + .describe("How well results match the stated intent (0-25)"), + intent_match_feedback: z + .string() + .describe("Explanation of intent matching"), + file_type_accuracy_score: z + .number() + .min(0) + .max(20) + .describe("Correctness of file types in results (0-20)"), + file_type_feedback: z.string().describe("Feedback on file type selection"), + avoidance_compliance_score: z + .number() + .min(0) + .max(15) + .describe("How well unwanted file types were avoided (0-15)"), + avoidance_feedback: z.string().describe("Feedback on file type avoidance"), + }), + overall: z.object({ + passed: z + .boolean() + .describe("Whether the search quality passes the evaluation"), + total_score: z + .number() + .min(0) + .max(100) + .describe("Total score (0-100)"), + summary: z + .string() + .describe("Brief summary of the evaluation (2-3 sentences)"), + critical_issues: z + .array(z.string()) + .describe("List of critical issues that caused failure"), + }), +}); + +type Evaluation = z.infer; + +interface ToolCall { + id: string; + type: string; + function: { + name: string; + arguments: string; + }; +} + +interface Message { + tool_calls?: ToolCall[]; +} + +interface ContextData { + messages: Message[]; +} + +interface Args { + context: string; + intent: string; + expectedFileTypes: string; + shouldAvoid: string; +} + +function parseArgs(): Args { + const args = process.argv.slice(2); + const parsed: Record = {}; + + for (let i = 0; i < args.length; i += 2) { + const key = args[i]?.replace(/^--/, "").replace(/-/g, "_"); + const value = args[i + 1]; + if (key && value) { + parsed[key] = value; + } + } + + if ( + !parsed.context || + !parsed.intent || + !parsed.expected_file_types || + !parsed.should_avoid + ) { + console.error("Missing required arguments"); + console.error( + "Usage: llm_judge.ts --context --intent --expected-file-types --should-avoid ", + ); + process.exit(1); + } + + return { + context: parsed.context, + intent: parsed.intent, + expectedFileTypes: parsed.expected_file_types, + shouldAvoid: parsed.should_avoid, + }; +} + +function extractSemanticSearchCalls(contextData: ContextData): Array<{ + query: string; + use_case: string; +}> { + const searches: Array<{ query: string; use_case: string }> = []; + + for (const message of contextData.messages) { + if (message.tool_calls) { + for (const toolCall of message.tool_calls) { + if (toolCall.function.name === "sem_search") { + try { + const args = JSON.parse(toolCall.function.arguments); + if (args.queries && Array.isArray(args.queries)) { + for (const q of args.queries) { + if (q.query && q.use_case) { + searches.push({ + query: q.query, + use_case: q.use_case, + }); + } + } + } + } catch (e) { + // Skip invalid JSON + } + } + } + } + } + + return searches; +} + +async function evaluateWithLLM( + searches: Array<{ query: string; use_case: string }>, + intent: string, + expectedFileTypes: string, + shouldAvoid: string, +): Promise { + const prompt = `You are an expert evaluator of semantic code search quality. Your task is to evaluate the quality of semantic search queries and assess whether they would return relevant results based on the user's intent. + +## User's Intent +The user wants to: ${intent} + +## Expected File Types +The search should primarily return files of these types: ${expectedFileTypes} + +## File Types to Avoid +The search should avoid returning files of these types: ${shouldAvoid} + +## Semantic Search Queries Generated +${searches.map((s, i) => `### Query ${i + 1} +**Embedding Query (for vector search):** ${s.query} +**Reranking Query (for result filtering):** ${s.use_case} +`).join("\n")} + +## Evaluation Criteria + +### Query Quality (50 points) +1. **Embedding Query (15 points):** + - Contains domain-specific terms and technical context + - Describes behavior and functionality, not meta-descriptions + - Is complete and descriptive, not overly generic + - Would generate embeddings that match similar code + +2. **Reranking Query (15 points):** + - Expresses the specific intent and use case + - Provides context about WHY the code is needed + - Specifies whether implementation/docs/tests are needed + - Is different from the embedding query (not verbatim) + +3. **Codebase Construct Keywords (10 points):** + - **CRITICAL**: When looking for code (not documentation), the reranking query MUST include specific construct keywords + - These keywords get HIGH WEIGHTAGE in reranking: struct, trait, impl, interface, class, function, definition, implementation, declaration, type + - Examples of GOOD use_case queries: + * "I need the struct definition for User authentication" + * "Show me the trait implementation for DatabaseConnection" + * "Find the function implementation that handles file patching" + * "I need the type declarations and interface definitions for the tool registry" + - Examples of BAD use_case queries (missing construct keywords): + * "I need code that handles authentication" (missing: struct/trait/impl/function) + * "Show me the database logic" (missing: trait/impl/function) + * "Find file patching code" (missing: function/impl/struct) + - **Scoring**: + * 10 points: Multiple specific construct keywords (struct, trait, impl, function, etc.) + * 7-9 points: At least one specific construct keyword + * 4-6 points: Generic keywords (code, implementation) without specifics + * 0-3 points: No construct keywords at all + - **Exception**: Documentation/config intents don't need construct keywords + +4. **Query Differentiation (10 points):** + - The two queries serve different purposes + - Embedding query focuses on WHAT (semantic similarity) + - Reranking query focuses on INTENT + CONSTRUCTS (contextual relevance + code structure) + +### Result Relevance (50 points) +Based on the queries, evaluate whether they would return appropriate results: + +1. **Intent Matching (25 points):** + - Would the queries find code matching the user's intent? + - For "implementation" intent: would it find actual code, not docs? + - For "documentation" intent: would it find docs, not code? + - For "tests" intent: would it find test files? + - For "flow_understanding": would it find the relevant code flow? + - For "debugging": would it find the problematic areas? + - For "modification": would it find the right code to modify? + +2. **File Type Accuracy (20 points):** + - Would the queries naturally return the expected file types? + - Are the queries specific enough to filter for the right extensions? + +3. **Avoidance Compliance (15 points):** + - Would the queries avoid returning the unwanted file types? + - Is the intent clear enough to prevent irrelevant results? + +## Scoring Guidelines +- **9-10:** Excellent - Professional quality, no significant issues +- **7-8:** Good - Minor improvements possible but functional +- **5-6:** Adequate - Some issues but would work in most cases +- **3-4:** Poor - Significant issues that impact effectiveness +- **0-2:** Failing - Critical problems, would not work as intended + +## Pass/Fail Criteria +- **PASS:** Total score >= 70/100 AND no critical issues +- **FAIL:** Total score < 70/100 OR has critical issues + +Critical issues include: +- Embedding and reranking queries are identical or nearly identical +- Query is too generic to return relevant results +- Intent mismatch is severe (e.g., asking for implementation but would return docs) +- Would definitely return wrong file types for the intent + +Please evaluate these queries and provide detailed, constructive feedback.`; + + try { + // @ts-ignore - Type instantiation depth issue with complex Zod schemas + const result = await generateObject({ + model: vertex("gemini-3-pro-preview"), + schema: EvaluationSchema, + prompt, + temperature: 0.3, // Lower temperature for more consistent evaluations + }); + + return result.object as Evaluation; + } catch (error) { + console.error("Error calling Gemini API:", error); + throw error; + } +} + +function formatEvaluation(evaluation: Evaluation): string { + const lines: string[] = []; + + lines.push("=== SEMANTIC SEARCH QUALITY EVALUATION ===\n"); + + // Query Quality + lines.push("## Query Quality"); + lines.push( + `Embedding Query: ${evaluation.query_quality.embedding_query_score}/15`, + ); + lines.push(` ${evaluation.query_quality.embedding_query_feedback}`); + lines.push( + `Reranking Query: ${evaluation.query_quality.reranking_query_score}/15`, + ); + lines.push(` ${evaluation.query_quality.reranking_query_feedback}`); + lines.push( + `Construct Keywords: ${evaluation.query_quality.construct_keywords_score}/10`, + ); + lines.push(` ${evaluation.query_quality.construct_keywords_feedback}`); + lines.push( + `Queries Differentiated: ${evaluation.query_quality.queries_differentiated ? "โœ“ Yes" : "โœ— No"}`, + ); + lines.push(""); + + // Result Relevance + lines.push("## Result Relevance"); + lines.push( + `Intent Match: ${evaluation.result_relevance.intent_match_score}/25`, + ); + lines.push(` ${evaluation.result_relevance.intent_match_feedback}`); + lines.push( + `File Type Accuracy: ${evaluation.result_relevance.file_type_accuracy_score}/20`, + ); + lines.push(` ${evaluation.result_relevance.file_type_feedback}`); + lines.push( + `Avoidance Compliance: ${evaluation.result_relevance.avoidance_compliance_score}/15`, + ); + lines.push(` ${evaluation.result_relevance.avoidance_feedback}`); + lines.push(""); + + // Overall + lines.push("## Overall Assessment"); + lines.push(`Total Score: ${evaluation.overall.total_score}/100`); + lines.push(`Status: ${evaluation.overall.passed ? "โœ“ PASS" : "โœ— FAIL"}`); + lines.push(`Summary: ${evaluation.overall.summary}`); + + if (evaluation.overall.critical_issues.length > 0) { + lines.push("\n## Critical Issues:"); + for (const issue of evaluation.overall.critical_issues) { + lines.push(` - ${issue}`); + } + } + + return lines.join("\n"); +} + +async function main() { + const args = parseArgs(); + + // Read context file + let contextData: ContextData; + try { + const contextContent = readFileSync(args.context, "utf-8"); + contextData = JSON.parse(contextContent); + } catch (error) { + console.error(`Error reading context file: ${error}`); + process.exit(1); + } + + // Extract semantic search calls + const searches = extractSemanticSearchCalls(contextData); + + if (searches.length === 0) { + console.error("No semantic search calls found in context"); + process.exit(1); + } + + // Evaluate with LLM + const evaluation = await evaluateWithLLM( + searches, + args.intent, + args.expectedFileTypes, + args.shouldAvoid, + ); + + // Output results + console.log(formatEvaluation(evaluation)); + + // Exit with appropriate code + process.exit(evaluation.overall.passed ? 0 : 1); +} + +main().catch((error) => { + console.error("Unexpected error:", error); + process.exit(1); +}); diff --git a/benchmarks/evals/semantic_search_quality/run_eval.sh b/benchmarks/evals/semantic_search_quality/run_eval.sh new file mode 100644 index 0000000000000000000000000000000000000000..7e3fb6185e9635130459e5eea96ec605eda4cff1 --- /dev/null +++ b/benchmarks/evals/semantic_search_quality/run_eval.sh @@ -0,0 +1,95 @@ +#!/bin/bash + +set -e + +CONTEXT_FILE="$1" +if [ -z "$CONTEXT_FILE" ]; then + echo "Usage: $0 " + exit 1 +fi + +echo "==========================================" +echo "Running Semantic Search Quality Eval" +echo "==========================================" +echo "" + +# Step 1: Check if sem_search was used +echo "Step 1: Checking for sem_search tool usage..." +if cat "$CONTEXT_FILE" | jq -e '[.messages[]?.tool_calls[]? | select(.function.name == "sem_search")] | any' > /dev/null 2>&1; then + echo "โœ“ sem_search tool found in context" +else + echo "โœ— FAILED: sem_search tool not used" + exit 1 +fi + +# Step 2: Extract intent and queries +echo "" +echo "Step 2: Extracting task intent and queries..." +INTENT=$(cat "$CONTEXT_FILE" | jq -r '.task.intent // "implementation"') +echo " Intent: $INTENT" + +# Extract expected file types and avoid list from task +EXPECTED_TYPES=$(cat "$CONTEXT_FILE" | jq -r '.task.expected_file_types // [] | join(",")') +SHOULD_AVOID=$(cat "$CONTEXT_FILE" | jq -r '.task.should_avoid // [] | join(",")') + +# Extract queries from tool calls +QUERIES=$(cat "$CONTEXT_FILE" | jq -r '[.messages[]?.tool_calls[]? | select(.function.name == "sem_search") | .function.arguments] | .[0]' 2>/dev/null || echo '{}') +echo " Queries extracted: $(echo "$QUERIES" | jq -r '.queries | length // 0') query pairs" + +# Step 3: Check authentication +echo "" +echo "Step 3: Checking Google Cloud authentication..." +if gcloud auth application-default print-access-token > /dev/null 2>&1; then + echo "โœ“ Google Cloud authentication available" + RUN_LLM_JUDGE=true +elif [ -n "$GOOGLE_APPLICATION_CREDENTIALS" ] && [ -f "$GOOGLE_APPLICATION_CREDENTIALS" ]; then + echo "โœ“ Service account credentials found" + RUN_LLM_JUDGE=true +else + echo "โš  Google Cloud authentication not configured" + echo " Skipping LLM judge evaluation" + echo " To enable: run 'gcloud auth application-default login'" + RUN_LLM_JUDGE=false +fi + +# Step 4: Run LLM judge (conditional) +echo "" +if [ "$RUN_LLM_JUDGE" = true ]; then + echo "Step 4: Running LLM judge evaluation..." + + # Build command with parameters (use empty strings if not set) + if [ -z "$EXPECTED_TYPES" ]; then + EXPECTED_TYPES="" + fi + + if [ -z "$SHOULD_AVOID" ]; then + SHOULD_AVOID="" + fi + + CMD="npx tsx llm_judge.ts --context \"$CONTEXT_FILE\" --intent \"$INTENT\" --expected-file-types \"$EXPECTED_TYPES\" --should-avoid \"$SHOULD_AVOID\"" + + eval $CMD + + if [ $? -eq 0 ]; then + echo "" + echo "==========================================" + echo "โœ“ EVALUATION PASSED" + echo "==========================================" + exit 0 + else + echo "" + echo "==========================================" + echo "โœ— EVALUATION FAILED" + echo "==========================================" + exit 1 + fi +else + echo "Step 4: Skipped (authentication required)" + echo "" + echo "==========================================" + echo "โœ“ VALIDATION PASSED (LLM judge skipped)" + echo "==========================================" + echo "" + echo "Note: Full evaluation requires Google Cloud auth" + exit 0 +fi diff --git a/benchmarks/evals/semantic_search_quality/test_context.json b/benchmarks/evals/semantic_search_quality/test_context.json new file mode 100644 index 0000000000000000000000000000000000000000..19b18d4f6b8a931ef8145f2120fe9f8a84a427da --- /dev/null +++ b/benchmarks/evals/semantic_search_quality/test_context.json @@ -0,0 +1,22 @@ +{ + "messages": [ + { + "role": "user", + "content": "Find the retry mechanism with exponential backoff implementation" + }, + { + "role": "assistant", + "content": "I'll search for the retry mechanism implementation.", + "tool_calls": [ + { + "id": "call_1", + "type": "function", + "function": { + "name": "sem_search", + "arguments": "{\"queries\":[{\"query\":\"retry mechanism with exponential backoff for API calls handling network failures\",\"use_case\":\"I need to find the implementation code for retry logic with backoff, not documentation or test files\"}]}" + } + } + ] + } + ] +} diff --git a/benchmarks/evals/semantic_search_quality/test_queries.ts b/benchmarks/evals/semantic_search_quality/test_queries.ts new file mode 100644 index 0000000000000000000000000000000000000000..6206130cb63677ffbaa0f864a4e983ce4e0fd8d8 --- /dev/null +++ b/benchmarks/evals/semantic_search_quality/test_queries.ts @@ -0,0 +1,232 @@ +#!/usr/bin/env tsx + +/** + * Manual Query Tester for Semantic Search + * + * This script helps developers test and improve their semantic search queries + * before committing them to the codebase using Gemini 3 Pro via Vertex AI. + */ + +import { vertex } from "@ai-sdk/google-vertex"; +import { generateObject } from "ai"; +import { z } from "zod"; +import * as readline from "readline"; + +const QueryEvaluationSchema = z.object({ + embedding_query: z.object({ + score: z.number().min(0).max(10), + strengths: z.array(z.string()), + weaknesses: z.array(z.string()), + suggestions: z.array(z.string()), + }), + reranking_query: z.object({ + score: z.number().min(0).max(10), + strengths: z.array(z.string()), + weaknesses: z.array(z.string()), + suggestions: z.array(z.string()), + }), + differentiation: z.object({ + are_different: z.boolean(), + explanation: z.string(), + suggestion: z.string().optional(), + }), + overall_recommendation: z.string(), +}); + +async function evaluateQueries( + embeddingQuery: string, + rerankingQuery: string, +): Promise> { + const prompt = `You are an expert in semantic code search. Evaluate these two queries: + +**Embedding Query (for vector search):** +${embeddingQuery} + +**Reranking Query (for result filtering):** +${rerankingQuery} + +Provide detailed, constructive feedback on: +1. The quality of each query +2. Whether they are sufficiently different +3. How to improve them + +Scoring (0-10): +- 9-10: Excellent +- 7-8: Good +- 5-6: Adequate +- 3-4: Poor +- 0-2: Failing + +Focus on practical, actionable suggestions.`; + + // @ts-ignore - Type instantiation depth issue with complex Zod schemas + const result = await generateObject({ + model: vertex("gemini-3-pro-preview"), + schema: QueryEvaluationSchema, + prompt, + temperature: 0.3, + }); + return result.object as z.infer; +} + +function formatEvaluation(evaluation: z.infer): string { + const lines: string[] = []; + + lines.push("\nโ•”โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•—"); + lines.push("โ•‘ SEMANTIC SEARCH QUERY EVALUATION โ•‘"); + lines.push("โ•šโ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•\n"); + + // Embedding Query + lines.push("๐Ÿ“Š EMBEDDING QUERY ANALYSIS"); + lines.push(` Score: ${evaluation.embedding_query.score}/10 ${getScoreEmoji(evaluation.embedding_query.score)}\n`); + + if (evaluation.embedding_query.strengths.length > 0) { + lines.push(" โœ… Strengths:"); + for (const strength of evaluation.embedding_query.strengths) { + lines.push(` โ€ข ${strength}`); + } + lines.push(""); + } + + if (evaluation.embedding_query.weaknesses.length > 0) { + lines.push(" โš ๏ธ Weaknesses:"); + for (const weakness of evaluation.embedding_query.weaknesses) { + lines.push(` โ€ข ${weakness}`); + } + lines.push(""); + } + + if (evaluation.embedding_query.suggestions.length > 0) { + lines.push(" ๐Ÿ’ก Suggestions:"); + for (const suggestion of evaluation.embedding_query.suggestions) { + lines.push(` โ€ข ${suggestion}`); + } + lines.push(""); + } + + // Reranking Query + lines.push("๐Ÿ“Š RERANKING QUERY ANALYSIS"); + lines.push(` Score: ${evaluation.reranking_query.score}/10 ${getScoreEmoji(evaluation.reranking_query.score)}\n`); + + if (evaluation.reranking_query.strengths.length > 0) { + lines.push(" โœ… Strengths:"); + for (const strength of evaluation.reranking_query.strengths) { + lines.push(` โ€ข ${strength}`); + } + lines.push(""); + } + + if (evaluation.reranking_query.weaknesses.length > 0) { + lines.push(" โš ๏ธ Weaknesses:"); + for (const weakness of evaluation.reranking_query.weaknesses) { + lines.push(` โ€ข ${weakness}`); + } + lines.push(""); + } + + if (evaluation.reranking_query.suggestions.length > 0) { + lines.push(" ๐Ÿ’ก Suggestions:"); + for (const suggestion of evaluation.reranking_query.suggestions) { + lines.push(` โ€ข ${suggestion}`); + } + lines.push(""); + } + + // Differentiation + lines.push("๐Ÿ”„ QUERY DIFFERENTIATION"); + lines.push(` ${evaluation.differentiation.are_different ? "โœ…" : "โŒ"} Queries are ${evaluation.differentiation.are_different ? "sufficiently different" : "too similar"}`); + lines.push(` ${evaluation.differentiation.explanation}\n`); + + if (evaluation.differentiation.suggestion) { + lines.push(` ๐Ÿ’ก ${evaluation.differentiation.suggestion}\n`); + } + + // Overall + lines.push("๐ŸŽฏ OVERALL RECOMMENDATION"); + lines.push(` ${evaluation.overall_recommendation}\n`); + + return lines.join("\n"); +} + +function getScoreEmoji(score: number): string { + if (score >= 9) return "๐ŸŒŸ"; + if (score >= 7) return "๐Ÿ‘"; + if (score >= 5) return "๐Ÿ‘Œ"; + if (score >= 3) return "โš ๏ธ"; + return "โŒ"; +} + +async function promptUser(question: string): Promise { + const rl = readline.createInterface({ + input: process.stdin, + output: process.stdout, + }); + + return new Promise((resolve) => { + rl.question(question, (answer) => { + rl.close(); + resolve(answer.trim()); + }); + }); +} + +async function main() { + console.log("\nโ•”โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•—"); + console.log("โ•‘ SEMANTIC SEARCH QUERY TESTER โ•‘"); + console.log("โ•‘ Test and improve your queries with AI feedback โ•‘"); + console.log("โ•šโ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•\n"); + + // Get queries from user + const embeddingQuery = await promptUser( + "Enter your embedding query (for vector search):\n> ", + ); + + if (!embeddingQuery) { + console.error("Embedding query is required"); + process.exit(1); + } + + console.log(""); + const rerankingQuery = await promptUser( + "Enter your reranking query (for result filtering):\n> ", + ); + + if (!rerankingQuery) { + console.error("Reranking query is required"); + process.exit(1); + } + + console.log("\n๐Ÿ”„ Evaluating your queries...\n"); + + try { + const evaluation = await evaluateQueries(embeddingQuery, rerankingQuery); + console.log(formatEvaluation(evaluation)); + } catch (error) { + console.error("Error evaluating queries:", error); + process.exit(1); + } +} + +// Check if running in interactive mode or with args +const args = process.argv.slice(2); +if (args.length >= 2) { + // Non-interactive mode + const embeddingQuery = args[0]; + const rerankingQuery = args[1]; + + console.log("\n๐Ÿ”„ Evaluating your queries...\n"); + evaluateQueries(embeddingQuery!, rerankingQuery!) + .then((evaluation) => { + console.log(formatEvaluation(evaluation)); + }) + .catch((error) => { + console.error("Error evaluating queries:", error); + process.exit(1); + }); +} else { + // Interactive mode + main().catch((error) => { + console.error("Unexpected error:", error); + process.exit(1); + }); +} diff --git a/benchmarks/evals/suggest/task.yml b/benchmarks/evals/suggest/task.yml new file mode 100644 index 0000000000000000000000000000000000000000..98d829040d2a21e094a35023a46d9e4b1bcdf486 --- /dev/null +++ b/benchmarks/evals/suggest/task.yml @@ -0,0 +1,47 @@ +run: + - mkdir -p test_dir + - cd test_dir && touch file1.txt file2.py README.md + - cd test_dir && forgee --provider {{provider}} --model {{model}} suggest "{{prompt}}" + +parallelism: 2 +timeout: 60 +early_exit: true + +validations: + - name: "Should generate output (non-empty)" + type: shell + command: "test $(wc -c | awk '{print $1}') -gt 0" + + - name: "Should NOT contain markdown code blocks" + type: shell + command: "! grep -qE '^```'" + + - name: "Should NOT contain JSON formatting tags" + type: shell + command: "! grep -qE '\\{\\s*\"command\"\\s*:'" + +sources: + - value: + - provider: "anthropic" + model: "claude-sonnet-4-5-20250929" + - provider: "open_router" + model: "z-ai/glm-4.7" + - provider: "open_router" + model: "minimax/minimax-m2.1" + + - value: + - prompt: "list all files" + - prompt: "find all python files" + - prompt: "show disk usage" + - prompt: "git status" + - prompt: "count lines in all files" + - prompt: "search for TODO in code" + - prompt: "check running processes" + - prompt: "compress this directory" + - prompt: "show last 20 commits" + - prompt: "find files larger than 1MB" + - prompt: "kill process on port 3000" + - prompt: "show system memory usage" + - prompt: "download file from url" + - prompt: "create git branch" + - prompt: "watch log file changes" diff --git a/benchmarks/model.ts b/benchmarks/model.ts new file mode 100644 index 0000000000000000000000000000000000000000..501a967d00b12549bb09701ae9e29d44737efc98 --- /dev/null +++ b/benchmarks/model.ts @@ -0,0 +1,38 @@ +export type Task = { + before_run: Array; + run: string | Array; + parallelism?: number; + timeout?: number; + early_exit?: boolean; + validations?: Array; + sources: Array; +}; + +export type Validation = + | { + name: string; + type: "regex"; + regex: string; + } + | { + name: string; + type: "shell"; + command: string; + exit_code?: number; + } + | { + name: string; + type: "llm"; + evaluator: string; + criteria: Record; + }; + +export type Source = { csv: string } | { cmd: string } | { value: Record[] }; + + +export enum TaskStatus { + Passed = "passed", + ValidationFailed = "validation_failed", + Timeout = "timeout", + Failed = "failed", +} diff --git a/benchmarks/parse.ts b/benchmarks/parse.ts new file mode 100644 index 0000000000000000000000000000000000000000..786c9a7542077acb8f2425e9329aca2ae62807bf --- /dev/null +++ b/benchmarks/parse.ts @@ -0,0 +1,52 @@ +import yargs from "yargs"; +import { hideBin } from "yargs/helpers"; +import path from "path"; + +export type CliArgs = { + evalName: string; + evalDir: string; + taskFile: string; +}; + +/** + * Parses command line arguments and resolves paths + */ +export async function parseCliArgs(dirname: string): Promise { + const argv = await yargs(hideBin(process.argv)) + .usage("Usage: $0 [options]") + .command("$0 ", "Run an evaluation") + .positional("eval-name", { + describe: "Name of the evaluation to run", + type: "string", + }) + .help() + .alias("h", "help") + .parseAsync(); + + const evalName = argv["eval-name"]; + + // Ensure evalName is provided + if (!evalName) { + throw new Error("eval-name is required"); + } + + // Support both directory path and direct task.yml path + let evalDir: string; + let taskFile: string; + + if (evalName.endsWith("task.yml") || evalName.endsWith(".yml") || evalName.endsWith(".yaml")) { + // Direct path to task file + taskFile = path.isAbsolute(evalName) ? evalName : path.join(dirname, evalName); + evalDir = path.dirname(taskFile); + } else { + // Directory path (original behavior) + evalDir = path.join(dirname, evalName); + taskFile = path.join(evalDir, "task.yml"); + } + + return { + evalName, + evalDir, + taskFile, + }; +} diff --git a/benchmarks/task-executor.ts b/benchmarks/task-executor.ts new file mode 100644 index 0000000000000000000000000000000000000000..d1d076a02264a5ef746b768cb2f59c2a757b3cf0 --- /dev/null +++ b/benchmarks/task-executor.ts @@ -0,0 +1,184 @@ +import * as fs from "fs"; +import * as path from "path"; +import { spawn } from "child_process"; +import stripAnsi from "strip-ansi"; +import type { Validation, Task } from "./model.js"; +import { runValidations, allValidationsPassed } from "./verification.js"; +import { formatTimestamp } from "./utils.js"; + +export type TaskExecutionResult = { + index: number; + command: string; + duration: number; + output?: string; + error?: string; + isTimeout: boolean; + earlyExit?: boolean; +}; + +/** + * Executes a single task command and returns the result + */ +export async function executeTask( + command: string, + index: number, + logFile: string, + cwd: string, + task: Task, + context?: Record, + append: boolean = false, +): Promise { + const startTime = Date.now(); + + // Create log stream for this task (append if this is not the first command) + const logStream = fs.createWriteStream(logFile, { + flags: append ? "a" : "w", + }); + + // Write command at the top of the log file + logStream.write(`Command: ${command}\n`); + logStream.write(`Started: ${formatTimestamp(new Date())}\n`); + logStream.write(`${"=".repeat(80)}\n\n`); + + try { + // Track timeout state outside the promise + let timedOut = false; + let exitedEarly = false; + + // Execute command and stream output to log file + const output = await new Promise((resolve, reject) => { + const child = spawn(command, { + shell: true, + cwd: cwd, + stdio: ["ignore", "pipe", "pipe"], + }); + + let stdout = ""; + let stderr = ""; + let timeoutId: NodeJS.Timeout | null = null; + + // Helper function to check validations after each write + const checkValidations = async () => { + if (exitedEarly || timedOut) return; + + if ( + task.early_exit && + task.validations && + task.validations.length > 0 + ) { + const currentOutput = stdout + stderr; + if (currentOutput) { + const results = await runValidations( + currentOutput, + task.validations, + context, + ); + if (allValidationsPassed(results)) { + exitedEarly = true; + if (timeoutId) clearTimeout(timeoutId); + if (logStream.writable) { + logStream.write( + `\n${"=".repeat(80)}\nEarly exit: All validations passed\nKilling process...\n`, + () => { + logStream.end(); + }, + ); + } + child.kill("SIGTERM"); + resolve(currentOutput); + } + } + } + }; + + // Set up timeout if configured + if (task.timeout) { + timeoutId = setTimeout(() => { + timedOut = true; + if (logStream.writable) { + logStream.write(`\n${"=".repeat(80)}\n`); + logStream.write(`Timeout: ${task.timeout}s exceeded\n`); + logStream.write(`Killing process...\n`); + logStream.end(); + } + child.kill("SIGKILL"); + // Resolve with captured output so far + resolve(stdout + stderr); + }, task.timeout * 1000); + } + + // Stream stdout to both log file and capture for validation + child.stdout?.on("data", (data) => { + const text = data.toString(); + stdout += text; + if (logStream.writable) { + logStream.write(stripAnsi(text)); + } + checkValidations(); + }); + + // Stream stderr to both log file and capture for validation + child.stderr?.on("data", (data) => { + const text = data.toString(); + stderr += text; + if (logStream.writable) { + logStream.write(stripAnsi(text)); + } + checkValidations(); + }); + + child.on("close", (code) => { + if (timeoutId) clearTimeout(timeoutId); + + // Don't log or resolve if already timed out or exited early + if (timedOut || exitedEarly) return; + + logStream.write(`\n${"=".repeat(80)}\n`); + logStream.write(`Finished: ${formatTimestamp(new Date())}\n`); + logStream.write(`Exit Code: ${code}\n`); + logStream.end(); + + if (code === 0) { + resolve(stdout + stderr); + } else { + reject(new Error(`Command failed with exit code ${code}`)); + } + }); + + child.on("error", (err) => { + if (timeoutId) clearTimeout(timeoutId); + + // Don't log if already timed out or exited early + if (timedOut || exitedEarly) return; + + logStream.write(`\nError: ${err.message}\n`); + logStream.end(); + reject(err); + }); + }); + + const duration = Date.now() - startTime; + + return { + index, + command, + duration, + output, + isTimeout: timedOut, + earlyExit: exitedEarly, + }; + } catch (error) { + const duration = Date.now() - startTime; + const errorMessage = + error instanceof Error ? error.message : "Command failed"; + + return { + index, + command, + duration, + error: errorMessage, + isTimeout: false, + earlyExit: false, + }; + } +} diff --git a/benchmarks/tsconfig.json b/benchmarks/tsconfig.json new file mode 100644 index 0000000000000000000000000000000000000000..1e1b330fc179588d3c34eccc9086f4188451d731 --- /dev/null +++ b/benchmarks/tsconfig.json @@ -0,0 +1,47 @@ +{ + // Visit https://aka.ms/tsconfig to read more about this file + "compilerOptions": { + // File Layout + "rootDir": ".", + "outDir": "dist", + + // Environment Settings + // See also https://aka.ms/tsconfig/module + "module": "nodenext", + "target": "esnext", + "types": [], + // For nodejs: + // "lib": ["esnext"], + // "types": ["node"], + // and npm install -D @types/node + + // Other Outputs + "sourceMap": true, + "declaration": true, + "declarationMap": true, + + // Stricter Typechecking Options + "noUncheckedIndexedAccess": true, + "exactOptionalPropertyTypes": true, + + // Style Options + // "noImplicitReturns": true, + // "noImplicitOverride": true, + // "noUnusedLocals": true, + // "noUnusedParameters": true, + // "noFallthroughCasesInSwitch": true, + // "noPropertyAccessFromIndexSignature": true, + + // Recommended Options + "strict": true, + "jsx": "react-jsx", + "verbatimModuleSyntax": true, + "isolatedModules": true, + "noUncheckedSideEffectImports": true, + "moduleDetection": "force", + "skipLibCheck": true, + + "esModuleInterop": true, + "resolveJsonModule": true, + } +} diff --git a/benchmarks/utils.ts b/benchmarks/utils.ts new file mode 100644 index 0000000000000000000000000000000000000000..827b944bb3796de7b32ac94a7cc78de4883250ec --- /dev/null +++ b/benchmarks/utils.ts @@ -0,0 +1,56 @@ +import tmp from "tmp"; +import { parse as parseCsv } from "csv-parse"; + +/** + * Formats a date with local timezone information + */ +export function formatTimestamp(date: Date): string { + const offset = -date.getTimezoneOffset(); + const sign = offset >= 0 ? "+" : "-"; + const hours = Math.floor(Math.abs(offset) / 60) + .toString() + .padStart(2, "0"); + const minutes = (Math.abs(offset) % 60).toString().padStart(2, "0"); + const timezone = `${sign}${hours}:${minutes}`; + + return `${date.toISOString().replace("Z", "")}${timezone}`; +} + +/** + * Escapes special regex characters in a string + */ +export function escapeRegex(str: string): string { + return str.replace(/[.*+?^${}()|[\]\\]/g, "\\$&"); +} + +/** + * Creates a temporary directory asynchronously + */ +export async function createTempDir( + prefix: string, +): Promise<{ name: string; removeCallback: () => void }> { + return new Promise((resolve, reject) => { + tmp.dir({ prefix }, (err, path, cleanupCallback) => { + if (err) reject(err); + else resolve({ name: path, removeCallback: cleanupCallback }); + }); + }); +} + +/** + * Parses CSV content asynchronously + */ +export async function parseCsvAsync( + content: string, + options: { + columns: boolean; + skip_empty_lines: boolean; + }, +): Promise[]> { + return new Promise((resolve, reject) => { + parseCsv(content, options, (err, records: unknown) => { + if (err) reject(err); + else resolve(records as Record[]); + }); + }); +} diff --git a/benchmarks/verification.ts b/benchmarks/verification.ts new file mode 100644 index 0000000000000000000000000000000000000000..749619083402134b20582b0d08c9d0c583caeb84 --- /dev/null +++ b/benchmarks/verification.ts @@ -0,0 +1,235 @@ +import { exec } from "child_process"; +import { promisify } from "util"; +import Handlebars from "handlebars"; +import type { Task, Validation } from "./model.js"; +import { escapeRegex } from "./utils.js"; + +const execAsync = promisify(exec); + +// Register Handlebars helper for escaping regex +Handlebars.registerHelper("escapeRegex", escapeRegex); + +export type ValidationResult = { + name: string; + passed: boolean; + message: string; +}; + +/** + * Validates output against a regex pattern + */ +function validateRegex( + output: string, + regex: string, + name: string, +): ValidationResult { + const pattern = new RegExp(regex); + const passed = pattern.test(output); + + return { + name, + passed, + message: passed ? `Matched: ${regex}` : `Did not match: ${regex}`, + }; +} + +/** + * Validates output using a shell command + */ +async function validateShellCommand( + output: string, + command: string, + expectedExitCode: number, + name: string, +): Promise { + try { + const { spawn } = await import("child_process"); + + // Use spawn to pipe stdin properly + const result = await new Promise<{ code: number }>((resolve, reject) => { + const child = spawn(command, { + shell: true, + stdio: ["pipe", "pipe", "pipe"], + }); + + let processExited = false; + + // Handle stdin errors (EPIPE when process exits early) + child.stdin.on("error", (err: NodeJS.ErrnoException) => { + // Ignore EPIPE errors - they happen when the child process + // exits before we finish writing, which is expected behavior + if (err.code !== "EPIPE") { + reject(err); + } + }); + + child.on("close", (code) => { + processExited = true; + resolve({ code: code ?? 0 }); + }); + + child.on("error", (err) => { + reject(err); + }); + + // Write output to stdin + // Use setImmediate to ensure event handlers are attached first + setImmediate(() => { + if (!processExited && child.stdin.writable) { + child.stdin.write(output, (err?: Error | null) => { + if (err && (err as NodeJS.ErrnoException).code !== "EPIPE") { + // Only reject on non-EPIPE errors + reject(err); + } else { + child.stdin.end(); + } + }); + } else { + // Process already exited or stdin not writable + child.stdin.end(); + } + }); + }); + + // Command succeeded (exit code 0) + const passed = result.code === expectedExitCode; + return { + name, + passed, + message: passed + ? `Command succeeded with exit code ${result.code}` + : `Expected exit code ${expectedExitCode}, got ${result.code}`, + }; + } catch (error: any) { + // Command failed with error + return { + name, + passed: false, + message: `Command failed: ${error.message}`, + }; + } +} + +/** + * Runs all validations on output and returns results + */ +export async function runValidations( + output: string, + validations: Array, + context?: Record, +): Promise { + const results: ValidationResult[] = []; + + for (const validation of validations) { + if (validation.type === "regex") { + // Interpolate regex with context if provided + let regex = validation.regex; + if (context) { + const template = Handlebars.compile(regex, { strict: true }); + regex = template(context); + } + results.push(validateRegex(output, regex, validation.name)); + } else if (validation.type === "shell") { + // Interpolate command with context if provided + let command = validation.command; + if (context) { + const template = Handlebars.compile(command, { strict: true }); + command = template(context); + } + const expectedExitCode = validation.exit_code ?? 0; + results.push( + await validateShellCommand( + output, + command, + expectedExitCode, + validation.name, + ), + ); + } + } + + return results; +} + +/** + * Checks if all validation results passed + */ +export function allValidationsPassed(results: ValidationResult[]): boolean { + return results.every((result) => result.passed); +} + +/** + * Counts how many validations passed + */ +export function countPassed(results: ValidationResult[]): number { + return results.filter((result) => result.passed).length; +} + +export type ProcessValidationsResult = { + validationResults: ValidationResult[]; + status: "passed" | "validation_failed"; +}; + +/** + * Processes validations and returns results with status + */ +export async function processValidations( + output: string | undefined, + task: Task, + logger: { + info: (data: any, message: string) => void; + warn: (data: any, message: string) => void; + error: (data: any, message: string) => void; + }, + task_id: number, + duration: number, + logFile: string, + context?: Record, +): Promise { + // Run validations if configured and output is available + const validationResults = + task.validations && task.validations.length > 0 && output + ? await runValidations(output, task.validations, context) + : []; + + const allPassed = allValidationsPassed(validationResults); + const status = allPassed ? "passed" : "validation_failed"; + + // Log all validation results + if (validationResults.length > 0) { + const passedCount = countPassed(validationResults); + const totalCount = validationResults.length; + + if (allPassed) { + logger.info( + { + task_id, + duration, + log: logFile, + parameters: context, + passed: validationResults.map((r) => r.name), + }, + "Validation passed", + ); + } else { + logger.error( + { + task_id, + duration, + log_file: logFile, + parameters: context, + failed: validationResults + .filter((r) => !r.passed) + .map((r) => ({ + name: r.name, + message: r.message, + })), + summary: `${passedCount}/${totalCount} passed`, + }, + "Validation Failed", + ); + } + } + + return { validationResults, status }; +} diff --git a/commands/github-pr-description.md b/commands/github-pr-description.md new file mode 100644 index 0000000000000000000000000000000000000000..391a1760f464ed05e7f0ceefe23422ec003693ed --- /dev/null +++ b/commands/github-pr-description.md @@ -0,0 +1,8 @@ +--- +name: github-pr-description +description: Updates the description of the PR +--- + +Create / Update the pull-request description and title using the pull-request skill. + +{{parameters}} diff --git a/docs/tool-guidelines.md b/docs/tool-guidelines.md new file mode 100644 index 0000000000000000000000000000000000000000..90ea70072867a98d03aae683ff0b19b375637801 --- /dev/null +++ b/docs/tool-guidelines.md @@ -0,0 +1,94 @@ +# Claude Tool Description Best Practices + +## Key Principles + +When defining tools for Claude to use, the quality of your tool descriptions dramatically affects performance. Here are the most important best practices extracted from Anthropic's documentation: + +### 1. Provide Extremely Detailed Descriptions + +This is by far the most important factor in tool performance. Your descriptions should explain every detail about the tool, including: + +- **What the tool does** - Be explicit about the tool's primary function and purpose +- **When it should be used** - Specify the scenarios where this tool is appropriate +- **When it should NOT be used** - Clarify scenarios where the tool is not suitable +- **What each parameter means** - Explain how each parameter affects the tool's behavior +- **Limitations and caveats** - Note what information the tool does not return or handle +- **Additional context** - If the tool name is unclear, provide extra clarification + +### 2. Use Comprehensive But Concise Descriptions + +The more context you can give Claude about your tools, the better it will be at deciding when and how to use them. Aim for 3-4 sentences per tool description, or more if the tool is complex. + +> **IMPORTANT**: Tool descriptions must never exceed 1024 characters. This is enforced by tests to ensure compatibility with LLM API constraints. + +### 3. Prioritize Descriptions Over Examples + +While you can include examples of how to use a tool in its description or in the accompanying prompt, this is less important than having a clear and comprehensive explanation of the tool's purpose and parameters. Only add examples after you've fully fleshed out the description. + +### 4. Register All Tools in the Registry + +Every tool must be registered in the `crates/forge_services/src/tools/registry.rs` file to be available for use. The `ToolRegistry::tools()` method returns all available tools configured with the given infrastructure. + +## Example Comparison + +### Good Tool Description Example: + +```json +{ + "name": "get_stock_price", + "description": "Retrieves the current stock price for a given ticker symbol. The ticker symbol must be a valid symbol for a publicly traded company on a major US stock exchange like NYSE or NASDAQ. The tool will return the latest trade price in USD. It should be used when the user asks about the current or most recent price of a specific stock. It will not provide any other information about the stock or company.", + "input_schema": { + "type": "object", + "properties": { + "ticker": { + "type": "string", + "description": "The stock ticker symbol, e.g. AAPL for Apple Inc." + } + }, + "required": ["ticker"] + } +} +``` + +### Poor Tool Description Example: + +```json +{ + "name": "get_stock_price", + "description": "Gets the stock price for a ticker.", + "input_schema": { + "type": "object", + "properties": { + "ticker": { + "type": "string" + } + }, + "required": ["ticker"] + } +} +``` + +## Why the Good Description Works Better + +The good description clearly explains: +- What the tool does (retrieves the current stock price) +- What data format it returns (latest trade price in USD) +- When to use it (when the user asks about current/recent price) +- What limitations it has (won't provide other company info) +- What the parameter means (ticker symbol for publicly traded companies) + +The poor description is too brief and leaves Claude with many open questions about the tool's behavior and usage. It doesn't explain what kind of data is returned, what format it's in, when to use it, or what parameters are expected. + +## Practical Tips + +1. **Be specific about data formats** - Explain what format the data will be returned in +2. **Clarify parameter constraints** - Note any restrictions on parameter values +3. **Explain tool selection criteria** - Help Claude understand when to pick this tool over others +4. **Describe error handling** - Note how the tool behaves with invalid inputs +5. **Include domain-specific details** - Add contextual information related to the tool's domain +6. **Keep descriptions under 1024 characters** - Ensure compatibility with LLM API constraints +7. **Register your tool in the registry** - Add your tool to the registry.rs file for availability + +Thorough yet concise tool descriptions lead to more accurate tool selection, fewer clarification questions, and better overall performance when using Claude with tools. + +Source: [Anthropic Documentation on Tool Use](https://docs.anthropic.com/en/docs/build-with-claude/tool-use/overview#example-poor-tool-description) \ No newline at end of file diff --git a/scripts/benchmark.sh b/scripts/benchmark.sh new file mode 100644 index 0000000000000000000000000000000000000000..b6d4aa3a443fe26d356f2328bec01131c189d581 --- /dev/null +++ b/scripts/benchmark.sh @@ -0,0 +1,133 @@ +#!/usr/bin/env bash + +# Performance test script for forge commands +# Runs the command 10 times and collects timing statistics +# Usage: ./benchmark.sh [--threshold MS] [command args...] +# Example: ./benchmark.sh info +# Example: ./benchmark.sh --threshold 50 zsh rprompt +# Example: ./benchmark.sh --version + +set -euo pipefail + +# Colors and styling +BOLD='\033[1m' +DIM='\033[2m' +RESET='\033[0m' +GREEN='\033[32m' +YELLOW='\033[33m' +CYAN='\033[36m' +GRAY='\033[90m' + +# Configuration +BASE_COMMAND="target/debug/forge" +THRESHOLD="" + +# Parse arguments +ARGS=() +while [[ $# -gt 0 ]]; do + case $1 in + --threshold) + THRESHOLD="$2" + shift 2 + ;; + *) + ARGS+=("$1") + shift + ;; + esac +done + +# Build command +if [ ${#ARGS[@]} -gt 0 ]; then + COMMAND="$BASE_COMMAND ${ARGS[*]}" +else + COMMAND="$BASE_COMMAND" +fi +ITERATIONS=10 +TIMES=() + +# Extract command name for display +DISPLAY_CMD=$(echo "$COMMAND" | sed "s|target/debug/||") + +# Header +echo "" +echo -e "๐Ÿš€ ${BOLD}Performance Test${RESET} ${DIM}โ€”${RESET} ${CYAN}${DISPLAY_CMD}${RESET}" +echo "" + +# Build step +echo -e "${GRAY}๐Ÿ“ฆ Building...${RESET}" +cargo build 2>&1 | grep -E "Compiling|Finished" | tail -1 +echo "" + +# Show sample output +echo -e "${GRAY}๐Ÿ“‹ Sample output:${RESET}" +echo "" +$COMMAND +echo "" + +# Run performance tests +echo -e "${GRAY}โฑ๏ธ Running ${YELLOW}$ITERATIONS${GRAY} iterations...${RESET}" +echo "" + +for i in $(seq 1 $ITERATIONS); do + # Measure execution time + START=$(date +%s%N) + $COMMAND > /dev/null 2>&1 + END=$(date +%s%N) + + # Calculate duration in milliseconds + DURATION=$(( (END - START) / 1000000 )) + TIMES+=($DURATION) + + # Color code based on performance + if [ $DURATION -lt 50 ]; then + COLOR=$GREEN + elif [ $DURATION -lt 100 ]; then + COLOR=$YELLOW + else + COLOR=$GRAY + fi + + printf " ${DIM}%2d${RESET} ${COLOR}%5d${RESET} ${DIM}ms${RESET}\n" $i $DURATION +done + +echo "" + +# Calculate statistics +TOTAL=0 +MIN=${TIMES[0]} +MAX=${TIMES[0]} + +for time in "${TIMES[@]}"; do + TOTAL=$((TOTAL + time)) + if [ $time -lt $MIN ]; then + MIN=$time + fi + if [ $time -gt $MAX ]; then + MAX=$time + fi +done + +AVG=$((TOTAL / ITERATIONS)) + +# Results summary +echo -e "๐Ÿ“Š ${BOLD}Summary${RESET}" +echo "" +printf " ${DIM}avg${RESET} ${CYAN}%5d${RESET} ${DIM}ms${RESET}\n" $AVG +printf " ${DIM}min${RESET} ${GREEN}%5d${RESET} ${DIM}ms${RESET}\n" $MIN +printf " ${DIM}max${RESET} ${YELLOW}%5d${RESET} ${DIM}ms${RESET}\n" $MAX +echo "" + +# Check threshold if provided +if [ -n "$THRESHOLD" ]; then + if [ $AVG -gt $THRESHOLD ]; then + echo -e "โŒ ${BOLD}Performance regression detected!${RESET}" + echo -e " Average time ${CYAN}${AVG}ms${RESET} exceeds threshold ${YELLOW}${THRESHOLD}ms${RESET}" + echo "" + exit 1 + else + echo -e "โœ… ${BOLD}Performance check passed!${RESET}" + echo -e " Average time ${CYAN}${AVG}ms${RESET} is within threshold ${YELLOW}${THRESHOLD}ms${RESET}" + echo "" + fi +fi diff --git a/scripts/list-all-porcelain.sh b/scripts/list-all-porcelain.sh new file mode 100644 index 0000000000000000000000000000000000000000..ae5589bea6e116df144a1be1e04e34c6182be51a --- /dev/null +++ b/scripts/list-all-porcelain.sh @@ -0,0 +1,114 @@ +#!/usr/bin/env bash +# Script to run all 'forge list' commands with --porcelain flag +# This helps visualize which list types contain $ID columns + +set -e + +FORGE_BIN="${1:-./target/debug/forge}" + +if [ ! -f "$FORGE_BIN" ]; then + echo "Error: forge binary not found at $FORGE_BIN" + echo "Usage: $0 [path-to-forge-binary]" + exit 1 +fi + +# Color codes for better visibility +BLUE='\033[0;34m' +GREEN='\033[0;32m' +YELLOW='\033[1;33m' +NC='\033[0m' # No Color + +print_section() { + echo "" + echo -e "${BLUE}โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•${NC}" + echo -e "${GREEN}$1${NC}" + echo -e "${BLUE}โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•${NC}" +} + +print_command() { + echo -e "${YELLOW}Command: $1${NC}" + echo "" +} + +print_runtime() { + local start="$1" + local end=$(date +%s%N) + local runtime_ms=$(( (end - start) / 1000000 )) + + if [ $runtime_ms -lt 1000 ]; then + echo -e "${YELLOW}Runtime: ${runtime_ms}ms${NC}" + else + local runtime_s=$(echo "scale=2; $runtime_ms / 1000" | bc 2>/dev/null || echo "$((runtime_ms / 1000))") + echo -e "${YELLOW}Runtime: ${runtime_s}s${NC}" + fi + echo "" +} + +# Agent list +print_section "AGENTS" +print_command "$FORGE_BIN list agent --porcelain" +start=$(date +%s%N) +$FORGE_BIN list agent --porcelain 2>&1 | head -20 || echo "No agents found" +print_runtime "$start" + +# Provider list +print_section "PROVIDERS" +print_command "$FORGE_BIN list provider --porcelain" +start=$(date +%s%N) +$FORGE_BIN list provider --porcelain 2>&1 | head -20 || echo "No providers found" +print_runtime "$start" + +# # Model list +# print_section "MODELS" +# print_command "$FORGE_BIN list model --porcelain" +# $FORGE_BIN list model --porcelain 2>&1 | head -20 || echo "No models found" + +# Config list +print_section "CONFIG" +print_command "$FORGE_BIN list config --porcelain" +start=$(date +%s%N) +$FORGE_BIN list config --porcelain 2>&1 | head -20 || echo "No config found" +print_runtime "$start" + +# MCP servers list +print_section "MCP SERVERS" +print_command "$FORGE_BIN list mcp --porcelain" +start=$(date +%s%N) +$FORGE_BIN list mcp --porcelain 2>&1 | head -20 || echo "No MCP servers found" +print_runtime "$start" + +# Conversation list +print_section "CONVERSATIONS" +print_command "$FORGE_BIN list conversation --porcelain" +start=$(date +%s%N) +$FORGE_BIN list conversation --porcelain 2>&1 | head -20 || echo "No conversations found" +print_runtime "$start" + +# Custom commands list +print_section "CUSTOM COMMANDS" +print_command "$FORGE_BIN list cmd --porcelain" +start=$(date +%s%N) +$FORGE_BIN list cmd --porcelain 2>&1 | head -20 || echo "No custom commands found" +print_runtime "$start" + +# Skills list +print_section "SKILLS" +print_command "$FORGE_BIN list skill --porcelain" +start=$(date +%s%N) +$FORGE_BIN list skill --porcelain 2>&1 | head -20 || echo "No skills found" +print_runtime "$start" + +# Summary +print_section "SUMMARY" +echo "List types WITH \$ID column:" +echo " โœ“ models - Second column (\$ID) contains display name" +echo " โœ“ mcp - First column (\$ID) contains server name" +echo " โœ“ cmd - First column (\$ID) contains command name" +echo " โœ“ skill - First column (\$ID) contains skill title" +echo "" +echo "List types WITHOUT \$ID column:" +echo " โœ— agent - Uses 'ID' (without $) instead" +echo " โœ— provider - Column 0 dropped, no \$ID" +echo " โœ— config - Uses \$FIELD/\$VALUE format (long format)" +echo " โœ— conversation - Direct UUID without \$ID label" +echo "" diff --git a/scripts/test-400-error-message.sh b/scripts/test-400-error-message.sh new file mode 100644 index 0000000000000000000000000000000000000000..98ba7f5bf561b34dfc46b6b3a1aa26089da48bc1 --- /dev/null +++ b/scripts/test-400-error-message.sh @@ -0,0 +1,39 @@ +#!/usr/bin/env bash +# Test that a 400 error from an invalid model shows the full response body. +# +# Usage: +# ./scripts/test-400-error-message.sh [provider_id] [model_id] +# +# Defaults to github_copilot / gpt-5.5 which routes through the Responses API +# path (model contains "gpt-5") and triggers a 400 for an unsupported model. + +set -euo pipefail + +PROVIDER="${1:-github_copilot}" +MODEL="${2:-gpt-5.5}" +BINARY="target/debug/forge" + +echo "Building debug binary..." +cargo build -p forge_main 2>&1 | tail -3 + +echo "" +echo "Running: FORGE_SESSION__PROVIDER_ID=$PROVIDER FORGE_SESSION__MODEL_ID=$MODEL $BINARY -p 'Hi'" +echo "---" + +# Capture stderr (where forge writes errors) and stdout separately. +ERROR_OUTPUT=$( + FORGE_SESSION__PROVIDER_ID="$PROVIDER" \ + FORGE_SESSION__MODEL_ID="$MODEL" \ + "$BINARY" -p "Hi" 2>&1 || true +) + +echo "$ERROR_OUTPUT" +echo "---" + +# Check that the error output contains something more than just a status code. +if echo "$ERROR_OUTPUT" | grep -qiE "Reason:|message|error|body"; then + echo "PASS: Error output contains a descriptive reason (body or message)." +else + echo "FAIL: Error output does not contain a reason. Only status code shown." + exit 1 +fi diff --git a/scripts/test-zsh-utils.sh b/scripts/test-zsh-utils.sh new file mode 100644 index 0000000000000000000000000000000000000000..9aa16926a5325dd0f7c665b0c0763b09ec4f29d3 --- /dev/null +++ b/scripts/test-zsh-utils.sh @@ -0,0 +1,211 @@ +#!/usr/bin/env zsh + +# Correctness and performance tests for `forge zsh format` which wraps bare +# file paths in @[...] syntax. All parsing logic now lives in Rust; these +# tests exercise the CLI subcommand end-to-end. +# +# Usage: zsh scripts/test-zsh-utils.sh + +set -euo pipefail +zmodload zsh/datetime + +# Colors +BOLD='\033[1m' +DIM='\033[2m' +RESET='\033[0m' +GREEN='\033[32m' +RED='\033[31m' +CYAN='\033[36m' + +PASS=0 +FAIL=0 + +# Resolve the forge binary (prefer local debug build) +SCRIPT_DIR="${0:A:h}" +FORGE_BIN="${FORGE_BIN:-${SCRIPT_DIR}/../target/debug/forge}" + +if [[ ! -x "$FORGE_BIN" ]]; then + echo "${RED}forge binary not found at ${FORGE_BIN}${RESET}" + echo "Run: cargo build -p forge_main" + exit 1 +fi + +# Wrapper that calls the Rust formatter +function format() { + "$FORGE_BIN" zsh format --buffer "$1" +} + +# Create temporary files for testing paths with spaces +TMPDIR_TEST=$(mktemp -d) +mkdir -p "${TMPDIR_TEST}/my folder" +touch "${TMPDIR_TEST}/my folder/test file.txt" +touch "${TMPDIR_TEST}/simple.txt" + +# --- Test harness ----------------------------------------------------------- + +function assert_eq() { + local test_name="$1" + local actual="$2" + local expected="$3" + + if [[ "$actual" == "$expected" ]]; then + printf " ${GREEN}โœ“${RESET} %s\n" "$test_name" + PASS=$(( PASS + 1 )) + else + printf " ${RED}โœ—${RESET} %s\n" "$test_name" + printf " ${DIM}expected:${RESET} %s\n" "$expected" + printf " ${DIM} actual:${RESET} %s\n" "$actual" + FAIL=$(( FAIL + 1 )) + fi +} + +# --- Correctness tests ------------------------------------------------------ + +echo "" +echo -e "${BOLD}Correctness Tests${RESET} ${DIM}โ€” forge zsh format${RESET}" +echo "" + +# Basic wrapping +assert_eq "bare existing path" \ + "$(format "/usr/bin/env")" \ + "@[/usr/bin/env]" + +assert_eq "path in sentence" \ + "$(format "look at /usr/bin/env please")" \ + "look at @[/usr/bin/env] please" + +# Non-existent paths left untouched +assert_eq "nonexistent path untouched" \ + "$(format "check /nonexistent/foo.rs")" \ + "check /nonexistent/foo.rs" + +# Already wrapped left untouched +assert_eq "already wrapped @[...] untouched" \ + "$(format "check @[/usr/bin/env] ok")" \ + "check @[/usr/bin/env] ok" + +# Plain text (no paths) +assert_eq "plain text no paths" \ + "$(format "hello world")" \ + "hello world" + +# Paths with spaces +assert_eq "bare path with spaces" \ + "$(format "${TMPDIR_TEST}/my folder/test file.txt")" \ + "@[${TMPDIR_TEST}/my folder/test file.txt]" + +# Quoted paths with spaces +assert_eq "single-quoted path with spaces" \ + "$(format "'${TMPDIR_TEST}/my folder/test file.txt'")" \ + "@[${TMPDIR_TEST}/my folder/test file.txt]" + +assert_eq "double-quoted path with spaces" \ + "$(format "\"${TMPDIR_TEST}/my folder/test file.txt\"")" \ + "@[${TMPDIR_TEST}/my folder/test file.txt]" + +assert_eq "single-quoted path with spaces in sentence" \ + "$(format "check '${TMPDIR_TEST}/my folder/test file.txt' please")" \ + "check @[${TMPDIR_TEST}/my folder/test file.txt] please" + +# Simple path (no spaces) +assert_eq "simple path no spaces" \ + "$(format "${TMPDIR_TEST}/simple.txt")" \ + "@[${TMPDIR_TEST}/simple.txt]" + +# Multiple paths +assert_eq "multiple existing paths" \ + "$(format "compare /usr/bin/env and ${TMPDIR_TEST}/simple.txt")" \ + "compare @[/usr/bin/env] and @[${TMPDIR_TEST}/simple.txt]" + +assert_eq "mixed existing and nonexistent" \ + "$(format "check /usr/bin/env and /nonexistent/foo.rs")" \ + "check @[/usr/bin/env] and /nonexistent/foo.rs" + +# Empty input +assert_eq "empty input" \ + "$(format "")" \ + "" + +# Backslash-escaped paths (terminals like Ghostty send /path/my\ file.txt) +local escaped_path="${TMPDIR_TEST}/my\ folder/test\ file.txt" +assert_eq "backslash-escaped path (whole paste)" \ + "$(format "$escaped_path")" \ + "@[${TMPDIR_TEST}/my folder/test file.txt]" + +assert_eq "backslash-escaped path in sentence" \ + "$(format "check $escaped_path please")" \ + "check @[${TMPDIR_TEST}/my folder/test file.txt] please" + +local escaped_simple="${TMPDIR_TEST}/simple.txt" +assert_eq "path without spaces (no escaping needed)" \ + "$(format "$escaped_simple")" \ + "@[${TMPDIR_TEST}/simple.txt]" + +assert_eq "backslash-escaped nonexistent path untouched" \ + "$(format "/nonexistent/my\ folder/file.txt")" \ + "/nonexistent/my\ folder/file.txt" + +# --- Performance tests ------------------------------------------------------- + +echo "" +echo -e "${BOLD}Performance Tests${RESET} ${DIM}โ€” forge zsh format${RESET}" +echo "" + +ITERATIONS=20 + +# Benchmark: simple path +START=$EPOCHREALTIME +for i in $(seq 1 $ITERATIONS); do + format "look at /usr/bin/env please" > /dev/null +done +END=$EPOCHREALTIME +ELAPSED=$(( (END - START) * 1000 )) +AVG=$(( ELAPSED / ITERATIONS )) +printf " ${DIM}simple path ${RESET} ${CYAN}%.2f${RESET} ${DIM}ms avg (${ITERATIONS} iterations)${RESET}\n" $AVG + +# Benchmark: path with spaces +START=$EPOCHREALTIME +for i in $(seq 1 $ITERATIONS); do + format "check '${TMPDIR_TEST}/my folder/test file.txt' please" > /dev/null +done +END=$EPOCHREALTIME +ELAPSED=$(( (END - START) * 1000 )) +AVG=$(( ELAPSED / ITERATIONS )) +printf " ${DIM}quoted path spaces ${RESET} ${CYAN}%.2f${RESET} ${DIM}ms avg (${ITERATIONS} iterations)${RESET}\n" $AVG + +# Benchmark: plain text (no paths) +START=$EPOCHREALTIME +for i in $(seq 1 $ITERATIONS); do + format "explain how this works in detail" > /dev/null +done +END=$EPOCHREALTIME +ELAPSED=$(( (END - START) * 1000 )) +AVG=$(( ELAPSED / ITERATIONS )) +printf " ${DIM}plain text ${RESET} ${CYAN}%.2f${RESET} ${DIM}ms avg (${ITERATIONS} iterations)${RESET}\n" $AVG + +# Benchmark: already wrapped +START=$EPOCHREALTIME +for i in $(seq 1 $ITERATIONS); do + format "check @[/usr/bin/env] and explain" > /dev/null +done +END=$EPOCHREALTIME +ELAPSED=$(( (END - START) * 1000 )) +AVG=$(( ELAPSED / ITERATIONS )) +printf " ${DIM}already wrapped ${RESET} ${CYAN}%.2f${RESET} ${DIM}ms avg (${ITERATIONS} iterations)${RESET}\n" $AVG + +# --- Cleanup ----------------------------------------------------------------- + +rm -rf "$TMPDIR_TEST" + +# --- Summary ----------------------------------------------------------------- + +echo "" +TOTAL=$(( PASS + FAIL )) +if (( FAIL > 0 )); then + echo -e "${RED}${BOLD}FAILED${RESET} ${PASS}/${TOTAL} passed, ${FAIL} failed" + echo "" + exit 1 +else + echo -e "${GREEN}${BOLD}ALL PASSED${RESET} ${PASS}/${TOTAL}" + echo "" +fi diff --git a/shell-plugin/README.md b/shell-plugin/README.md new file mode 100644 index 0000000000000000000000000000000000000000..a70c0bbb15147d5aff433f3d276317b742b30965 --- /dev/null +++ b/shell-plugin/README.md @@ -0,0 +1,312 @@ +# Forge ZSH Plugin + +A powerful ZSH plugin that provides intelligent command transformation, file tagging, and conversation management for the Forge AI assistant. + +## Features + +- **Smart Command Transformation**: Convert `:command` syntax into forge executions +- **Agent Selection**: Tab completion for available agents using `:agent_name` +- **File Tagging**: Interactive file selection with `@[filename]` syntax +- **Syntax Highlighting**: Visual feedback for commands and tagged files +- **Conversation Continuity**: Automatic session management across commands +- **Interactive Completion**: Fuzzy finding for files and agents via built-in picker + +## Prerequisites + +Before using this plugin, ensure you have the following tools installed: + +- **fd** - Fast file finder (alternative to find) +- **forge** - The Forge CLI tool + +### Installation of Prerequisites + +```bash +# macOS (using Homebrew) +brew install fd + +# Ubuntu/Debian +sudo apt install fd-find + +# Arch Linux +sudo pacman -S fd +``` + +## Usage + +### Starting a Conversation + +Begin any command with `:` followed by your prompt: + +```bash +: Get the current time +``` + +This automatically starts a new conversation with the default Forge agent. + +### Using Specific Agents + +Specify an agent by name after the colon: + +```bash +:sage How does caching work in this system? +:muse Create a deployment strategy for my app +``` + +**Tab Completion**: Type `:` followed by partial agent name and press `TAB` for interactive selection. + +### File Tagging + +Tag files in your commands using the `@[filename]` syntax: + +```bash +: Review this code @[src/main.rs] +: Explain the configuration in @[config.yaml] +``` + +**Interactive Selection**: Type `@` and press `TAB` to search and select files interactively using fuzzy finder. + +### Conversation Continuity + +Commands within the same session maintain context: + +```bash +# First command +: My project uses React and TypeScript + +# Second command (remembers previous context) +: How can I optimize the build process? +``` + +The plugin automatically manages conversation IDs to maintain context across related commands. + +### Command Naming + +Shell commands should follow the **Object-Action** format. + +Examples: +- `:provider-login` +- `:sync-status` + +For backward compatibility, `:login` remains available as an alias for `:provider-login`. + +### Session Management + +#### Starting New Sessions + +Clear the current conversation context and start fresh: + +```bash +:new +# or use the alias +:n +``` + +This will: + +- Clear the current conversation ID +- Show the banner with helpful information +- Reset the session state +- Display a confirmation message with timestamp + +#### System Information + +View system and project information: + +```bash +:info +# or use the alias +:i +``` + +This displays: + +- System information +- Project details +- Current configuration + +- Current configuration + +#### Switching Conversations + +Browse and switch between conversations interactively: + +```bash +:conversation +# or use the alias +:c +``` + +This will display an interactive list of all conversations with preview, allowing you to select and switch. + +Switch to a specific conversation by ID: + +```bash +:conversation +``` + +Toggle between current and previous conversation (like `cd -`): + +```bash +:conversation - +# or +:c - +``` + +The plugin remembers your previous conversation, allowing you to quickly toggle back and forth. This works just like `cd -` in your shell, and **also works with `:new`** - when you start a new conversation, you can toggle back to your previous one. + +If there's no previous conversation tracked (e.g., first time using the plugin), `:c -` will show the conversation list popup, allowing you to select a conversation. + +This is useful when: +- You need to temporarily check another conversation and come back +- You're comparing or referencing information between two conversations +- You want to quickly switch context between related tasks +- You started a new conversation but want to reference the previous one + +#### Cloning Conversations + +Create a copy of an existing conversation with interactive selection: + +```bash +:clone +``` + +This will: +- Display an interactive list of all conversations with preview +- Allow you to select a conversation to clone +- Create a new conversation with the same content +- Automatically switch to the cloned conversation +- Show the cloned conversation content and details + +You can also clone a specific conversation by providing its ID: + +```bash +:clone +``` + +This is useful when you want to: +- Create a backup before making significant changes +- Start a new conversation branch from an existing context +- Experiment with different approaches while preserving the original + +#### Session Status + +The plugin automatically displays session information including: +- Conversation ID when starting new sessions +- Active agent information +- New session confirmations with timestamps + +## Syntax Highlighting + +The plugin provides visual feedback through syntax highlighting: + +- **Tagged Files** (`@[filename]`): Displayed in **green bold** +- **Agent Commands** (`:agent`): Agent names in **yellow bold** +- **Command Text**: Remaining text in **white bold** + +## Configuration + +Customize the plugin behavior by setting these variables before loading the plugin: + +```bash +# Custom forge binary location +export FORGE_BIN="/path/to/custom/forge" +``` + +### Available Configuration Variables + +- `FORGE_BIN`: Path to the forge executable (default: `forge`) +- `FORGE_EDITOR`: Editor command to use for `:edit` command (default: `$EDITOR` or `nano`) +- `FORGE_SYNC_ENABLED`: Enable/disable automatic workspace sync (default: `true`) +- `FORGE_MAX_COMMIT_DIFF`: Maximum diff size for commit message generation in bytes (default: `100000`) +- `FORGE_SKIP_INTERACTIVE`: Skip interactive prompts (internal use) +- `FORGE_CURRENCY_SYMBOL`: Currency symbol for cost display in ZSH theme (default: `"$"`) +- `FORGE_CURRENCY_CONVERSION_RATE`: Conversion rate for currency display (default: `1.0`) +- `NERD_FONT`: Enable Nerd Font icons in ZSH theme (default: auto-detected, set to `"1"` or `"true"` to enable, `"0"` or `"false"` to disable) +- `USE_NERD_FONT`: Alternative variable for enabling Nerd Font icons (same behavior as `NERD_FONT`) +- Internal pattern matching for conversation syntax (`:`) +- New session command keyword: `:new` or `:n` + +### Codebase Indexing + +Sync your codebase for semantic search: + +```bash +:sync +``` + +This will index the current directory for semantic code search. + +### Environment Diagnostics + +Run comprehensive environment diagnostics to check your Forge setup: + +```bash +:doctor +``` + +This will check: +- ZSH version and terminal information +- Forge installation and version +- Plugin and theme loading status +- Completions availability +- Dependencies (fd, bat) +- ZSH plugins (autosuggestions, syntax-highlighting) +- Editor configuration and PATH setup +- Nerd Font support for icons + +### .forge Directory + +The plugin creates a `.forge` directory in your current working directory (similar to `.git`) for temporary files: + +- `FORGE_EDITMSG.md`: Temporary file used when opening an external editor with `:edit` + +## Advanced Features + +### Command History + +All transformed commands are properly saved to ZSH history, allowing you to: +- Navigate command history with arrow keys +- Search previous forge commands with `Ctrl+R` +- Reuse complex commands with file tags + +### Keyboard Shortcuts + +- **Tab**: Interactive completion for files (`@`) and agents (`:`) +- **Enter**: Transform and execute `:commands` +- **Ctrl+C**: Interrupt running forge commands + +## Examples + +### Basic Usage + +```bash +: What's the weather like? +:sage Explain the MVC pattern +:planner Help me structure this project +``` + +### With File Tagging + +```bash +: Review this implementation @[src/auth.rs] +: Debug the issue in @[logs/error.log] @[config/app.yml] +``` + +### Session Flow + +```bash +: I'm working on a Rust web API +: What are the best practices for error handling? +: Show me an example with @[src/errors.rs] +:info +:new +: New conversation starts here +``` + + +### Codebase Indexing + +```bash +# Sync current directory for semantic search +:sync +``` \ No newline at end of file diff --git a/shell-plugin/doctor.zsh b/shell-plugin/doctor.zsh new file mode 100644 index 0000000000000000000000000000000000000000..ebb03309388fc3d65a307298eac9a0f8ab77e396 --- /dev/null +++ b/shell-plugin/doctor.zsh @@ -0,0 +1,569 @@ +#!/usr/bin/env zsh + +# ZSH Doctor - Diagnostic tool for Forge shell environment +# Checks for common configuration issues and environment setup + +# Source user's .zshrc to get their environment (suppress errors from non-interactive mode) +if [[ -f "${ZDOTDIR:-$HOME}/.zshrc" ]]; then + source "${ZDOTDIR:-$HOME}/.zshrc" 2>/dev/null +fi + +# ANSI codes +local RESET='\033[0m' +local _BOLD='\033[1m' +local _DIM='\033[2m' +local _GREEN='\033[0;32m' +local _RED='\033[0;31m' +local _YELLOW='\033[0;33m' +local _CYAN='\033[0;36m' + +# Text formatting helpers - auto-reset +function bold() { echo "${_BOLD}${1}${RESET}"; } +function dim() { echo "${_DIM}${1}${RESET}"; } +function green() { echo "${_GREEN}${1}${RESET}"; } +function red() { echo "${_RED}${1}${RESET}"; } +function yellow() { echo "${_YELLOW}${1}${RESET}"; } +function cyan() { echo "${_CYAN}${1}${RESET}"; } + +# Simple ASCII symbols +local PASS="[OK]" +local FAIL="[ERROR]" +local WARN="[WARN]" + +# Counters +local passed=0 +local failed=0 +local warnings=0 + +# Helper function to print section headers +function print_section() { + echo "" + echo "$(bold "$1")" +} + +# Helper function to print results +function print_result() { + local result_status=$1 + local message=$2 + local detail=$3 + + case $result_status in + pass) + echo " $(green "${PASS}") ${message}" + ((passed++)) + ;; + fail) + echo " $(red "${FAIL}") ${message}" + [[ -n "$detail" ]] && echo " $(dim "ยท ${detail}")" + ((failed++)) + ;; + warn) + echo " $(yellow "${WARN}") ${message}" + [[ -n "$detail" ]] && echo " $(dim "ยท ${detail}")" + ((warnings++)) + ;; + info) + echo " $(dim "ยท ${message}")" + ;; + code) + echo " $(dim "ยท ${message}")" + ;; + instruction) + echo " $(dim "ยท ${message}")" + ;; + esac +} + +echo "$(bold "FORGE ENVIRONMENT DIAGNOSTICS")" + +# 1. Check ZSH version +print_section "Shell Environment" +local zsh_version="${ZSH_VERSION}" +if [[ -n "$zsh_version" ]]; then + local major=$(echo $zsh_version | cut -d. -f1) + local minor=$(echo $zsh_version | cut -d. -f2) + if [[ $major -ge 5 ]] && [[ $minor -ge 0 ]]; then + print_result pass "zsh: ${zsh_version}" + else + print_result warn "zsh: ${zsh_version}" "Recommended: 5.0+" + fi +else + print_result fail "Unable to detect ZSH version" +fi + +# Check terminal information +if [[ -n "$TERM_PROGRAM" ]]; then + if [[ -n "$TERM_PROGRAM_VERSION" ]]; then + print_result pass "Terminal: ${TERM_PROGRAM} ${TERM_PROGRAM_VERSION}" + else + print_result pass "Terminal: ${TERM_PROGRAM}" + fi +elif [[ -n "$TERM" ]]; then + print_result pass "Terminal: ${TERM}" +else + print_result info "Terminal: unknown" +fi + + +# Check if Oh My Zsh is installed +if [[ -n "$ZSH" ]] && [[ -d "$ZSH" ]]; then + local omz_version="" + # Try to get OMZ version from version file if it exists + if [[ -f "$ZSH/.git/refs/heads/master" ]]; then + omz_version=$(cd "$ZSH" && git describe --tags 2>/dev/null || echo "custom") + fi + + if [[ -n "$omz_version" ]]; then + print_result pass "Oh My Zsh: ${omz_version}" + else + print_result pass "Oh My Zsh: installed" + fi + print_result info "${ZSH}" +else + print_result warn "Oh My Zsh not found" "Install: sh -c \"\$(curl -fsSL https://raw.githubusercontent.com/ohmyzsh/ohmyzsh/master/tools/install.sh)\"" +fi + +# 2. Check if forge is installed and in PATH +print_section "Forge Installation" + +# Check if forge is in PATH +if command -v forge &> /dev/null; then + local forge_path=$(command -v forge) + + # Get forge version and extract just the version number + local forge_version=$(forge --version 2>&1 | head -n1 | awk '{print $2}') + if [[ -n "$forge_version" ]]; then + print_result pass "forge: ${forge_version}" + print_result info "${forge_path}" + else + print_result pass "forge: installed" + print_result info "${forge_path}" + fi +else + print_result fail "Forge binary not found in PATH" "Installation: curl -fsSL https://forgecode.dev/cli | sh" +fi + +# 3. Check shell plugin +print_section "Plugin" + +# Check if forge plugin is loaded by checking environment variable +if [[ -n "$_FORGE_PLUGIN_LOADED" ]]; then + print_result pass "Forge plugin loaded" +else + print_result fail "Forge plugin not loaded" + print_result instruction "Add to your ~/.zshrc:" + print_result code "eval \"\$(forge zsh plugin)\"" + print_result instruction "Or run: forge zsh setup" +fi + + +# Check plugin loading order in .zshrc +local zshrc_file="${ZDOTDIR:-$HOME}/.zshrc" +if [[ -f "$zshrc_file" ]] && [[ -n "$_FORGE_PLUGIN_LOADED" ]]; then + # Extract line numbers for plugin declarations and forge plugin eval + local plugins_line=$(grep -n "^[[:space:]]*plugins=(" "$zshrc_file" 2>/dev/null | head -n1 | cut -d: -f1) + local forge_plugin_line=$(grep -n "eval.*forge.*zsh plugin" "$zshrc_file" 2>/dev/null | head -n1 | cut -d: -f1) + + if [[ -n "$plugins_line" ]] && [[ -n "$forge_plugin_line" ]]; then + if [[ $forge_plugin_line -lt $plugins_line ]]; then + print_result fail "Plugin loading order incorrect" + print_result instruction "Forge plugin (line ${forge_plugin_line}) should be loaded AFTER plugins=() (line ${plugins_line})" + print_result instruction "Move the forge plugin eval statement after the plugins=() array in ~/.zshrc" + else + print_result pass "Plugin loading order correct" + fi + elif [[ -n "$forge_plugin_line" ]] && [[ -z "$plugins_line" ]]; then + # Forge plugin found but no plugins=() array - check for individual plugin sources + local has_other_plugins=false + if grep -q "source.*zsh-autosuggestions" "$zshrc_file" 2>/dev/null || \ + grep -q "source.*zsh-syntax-highlighting" "$zshrc_file" 2>/dev/null; then + has_other_plugins=true + fi + + if [[ "$has_other_plugins" == "true" ]]; then + print_result warn "Manual plugin loading detected" + print_result info "Ensure forge plugin is sourced AFTER zsh-autosuggestions and zsh-syntax-highlighting" + fi + fi +fi + +# 4. Check ZSH theme RPROMPT +print_section "FORGE RIGHT PROMPT" + +# Check if forge theme is loaded by checking environment variable +if [[ -n "$_FORGE_THEME_LOADED" ]]; then + print_result pass "Forge theme loaded" +elif (( $+functions[p10k] )); then + print_result info "Powerlevel10k detected (not using Forge theme)" +elif [[ -n "$ZSH_THEME" ]]; then + print_result warn "Using theme: ${ZSH_THEME}" + print_result instruction "To use Forge theme, add to ~/.zshrc:" + print_result code "eval \"\$(forge zsh theme)\"" +else + print_result warn "No theme loaded" + print_result instruction "To use Forge theme, add to ~/.zshrc:" + print_result code "eval \"\$(forge zsh theme)\"" +fi + +# Helper function to compare versions +# Returns 0 if version1 >= version2, 1 otherwise +function version_gte() { + local version1=$1 + local version2=$2 + + # Remove 'v' prefix if present + version1=${version1#v} + version2=${version2#v} + + # Split versions into arrays + local -a ver1_parts=(${(s:.:)version1}) + local -a ver2_parts=(${(s:.:)version2}) + + # Compare each part + for i in {1..3}; do + local v1=${ver1_parts[$i]:-0} + local v2=${ver2_parts[$i]:-0} + + # Remove any non-numeric suffix (e.g., "0-rc1" -> "0") + v1=${v1%%[^0-9]*} + v2=${v2%%[^0-9]*} + + if [[ $v1 -gt $v2 ]]; then + return 0 + elif [[ $v1 -lt $v2 ]]; then + return 1 + fi + done + + return 0 # versions are equal +} + +# 5. Check dependencies +print_section "Dependencies" + +# Forge uses its built-in nucleo-picker for interactive selection +# No external fuzzy finder (like fzf) is required +print_result pass "Interactive picker: built-in (nucleo-picker)" + +# Check for fd/fdfind - used for file discovery +if command -v fd &> /dev/null; then + local fd_version=$(fd --version 2>&1 | awk '{print $2}') + if [[ -n "$fd_version" ]]; then + if version_gte "$fd_version" "10.0.0"; then + print_result pass "fd: ${fd_version}" + else + print_result fail "fd: ${fd_version}" "Version 10.0.0 or higher required. Update: https://github.com/sharkdp/fd#installation" + fi + else + print_result pass "fd: installed" + fi +elif command -v fdfind &> /dev/null; then + local fd_version=$(fdfind --version 2>&1 | awk '{print $2}') + if [[ -n "$fd_version" ]]; then + if version_gte "$fd_version" "10.0.0"; then + print_result pass "fdfind: ${fd_version}" + else + print_result fail "fdfind: ${fd_version}" "Version 10.0.0 or higher required. Update: https://github.com/sharkdp/fd#installation" + fi + else + print_result pass "fdfind: installed" + fi +else + print_result warn "fd/fdfind not found" "Enhanced file discovery. See installation: https://github.com/sharkdp/fd#installation" +fi + +# Check for bat - used for syntax highlighting +if command -v bat &> /dev/null; then + local bat_version=$(bat --version 2>&1 | awk '{print $2}') + if [[ -n "$bat_version" ]]; then + if version_gte "$bat_version" "0.20.0"; then + print_result pass "bat: ${bat_version}" + else + print_result fail "bat: ${bat_version}" "Version 0.20.0 or higher required. Update: https://github.com/sharkdp/bat#installation" + fi + else + print_result pass "bat: installed" + fi +else + print_result warn "bat not found" "Enhanced preview. See installation: https://github.com/sharkdp/bat#installation" +fi + +# 6. Check required ZSH plugins +print_section "Required Plugins" + +# Check for zsh-autosuggestions +if [[ " ${plugins[*]} " =~ " zsh-autosuggestions " ]] || \ + [[ -n "$fpath[(r)*zsh-autosuggestions*]" ]] || \ + (( $+functions[_zsh_autosuggest_accept] )); then + print_result pass "zsh-autosuggestions loaded" +else + print_result warn "zsh-autosuggestions not found" + print_result info "Install plugin and add to plugins=() in .zshrc" + print_result info "Installation guide: https://github.com/zsh-users/zsh-autosuggestions/blob/master/INSTALL.md" +fi + +# Check for zsh-syntax-highlighting +if [[ " ${plugins[*]} " =~ " zsh-syntax-highlighting " ]] || \ + [[ -n "$fpath[(r)*zsh-syntax-highlighting*]" ]] || \ + (( $+functions[_zsh_highlight] )); then + print_result pass "zsh-syntax-highlighting loaded" +else + print_result warn "zsh-syntax-highlighting not found" + print_result info "Install plugin and add to plugins=() in .zshrc" + print_result info "Installation guide: https://github.com/zsh-users/zsh-syntax-highlighting/blob/master/INSTALL.md" +fi + +# 7. Check system configuration +print_section "System" + +# Check editor configuration (FORGE_EDITOR takes precedence over EDITOR) +if [[ -n "$FORGE_EDITOR" ]]; then + print_result pass "FORGE_EDITOR: ${FORGE_EDITOR}" + if [[ -n "$EDITOR" ]]; then + print_result info "EDITOR also set: ${EDITOR} (ignored)" + fi +elif [[ -n "$EDITOR" ]]; then + print_result pass "EDITOR: ${EDITOR}" + print_result info "TIP: Set FORGE_EDITOR for forge-specific editor" +else + print_result warn "No editor configured" "export EDITOR=vim or export FORGE_EDITOR=vim" +fi + +# Check PATH for common issues +if [[ "$PATH" == *"/usr/local/bin"* ]] || [[ "$PATH" == *"/usr/bin"* ]]; then + print_result pass "PATH: configured" +else + print_result warn "PATH may need common directories" "Ensure /usr/local/bin or /usr/bin is in PATH" +fi + +# 7. Check keyboard configuration (Alt/Option key as Meta) +print_section "Keyboard Configuration" + +local platform=$(uname) +local meta_key_ok=false +local check_performed=false + +if [[ "$platform" == "Darwin" ]]; then + # macOS checks + if [[ "$TERM_PROGRAM" == "vscode" ]]; then + check_performed=true + # Check VS Code settings + local vscode_settings="${HOME}/Library/Application Support/Code/User/settings.json" + if [[ -f "$vscode_settings" ]]; then + if grep -q '"terminal.integrated.macOptionIsMeta"[[:space:]]*:[[:space:]]*true' "$vscode_settings" 2>/dev/null; then + print_result pass "VS Code: Option key configured as Meta" + meta_key_ok=true + else + print_result warn "VS Code: Option key NOT configured as Meta" + print_result instruction "Option+F and Option+B shortcuts won't work for word navigation" + print_result instruction "Add to VS Code settings.json:" + print_result code '"terminal.integrated.macOptionIsMeta": true' + print_result instruction "Then reload VS Code: Cmd+Shift+P โ†’ Reload Window" + fi + else + print_result warn "VS Code settings file not found" + print_result info "Expected: ${vscode_settings}" + fi + elif [[ "$TERM_PROGRAM" == "iTerm.app" ]]; then + check_performed=true + # Check iTerm2 preferences + local iterm_prefs="${HOME}/Library/Preferences/com.googlecode.iterm2.plist" + if [[ -f "$iterm_prefs" ]]; then + # Check if either Left or Right Option key is set to Esc+ (value 2) + local option_setting=$(defaults read com.googlecode.iterm2 2>/dev/null | grep -E '"(Left |Right )?Option Key Sends"' | grep -o '[0-9]' | head -1) + if [[ "$option_setting" == "2" ]]; then + print_result pass "iTerm2: Option key configured as Esc+" + meta_key_ok=true + else + print_result warn "iTerm2: Option key NOT configured as Esc+" + print_result instruction "Option+F and Option+B shortcuts won't work for word navigation" + print_result instruction "Configure in iTerm2:" + print_result info "Preferences โ†’ Profiles โ†’ Keys โ†’ Left/Right Option Key โ†’ Esc+" + fi + else + print_result warn "iTerm2 preferences not found" + print_result info "Expected: ${iterm_prefs}" + fi + elif [[ "$TERM_PROGRAM" == "Apple_Terminal" ]]; then + check_performed=true + # Check Terminal.app preferences + local terminal_prefs="${HOME}/Library/Preferences/com.apple.Terminal.plist" + if [[ -f "$terminal_prefs" ]]; then + local use_option=$(defaults read com.apple.Terminal 2>/dev/null | grep -E 'useOptionAsMetaKey' | grep -o '[0-9]' | head -1) + if [[ "$use_option" == "1" ]]; then + print_result pass "Terminal.app: Option key configured as Meta" + meta_key_ok=true + else + print_result warn "Terminal.app: Option key NOT configured as Meta" + print_result instruction "Option+F and Option+B shortcuts won't work for word navigation" + print_result instruction "Configure in Terminal.app:" + print_result info "Preferences โ†’ Profiles โ†’ Keyboard โ†’ โœ“ Use Option as Meta key" + fi + else + print_result warn "Terminal.app preferences not found" + print_result info "Expected: ${terminal_prefs}" + fi + fi + + # If no specific terminal detected, provide general guidance for macOS + if [[ "$check_performed" == "false" ]]; then + print_result info "Terminal: ${TERM_PROGRAM:-unknown}" + print_result info "For Option key shortcuts (word navigation) to work:" + print_result info "โ€ข VS Code: Settings โ†’ terminal.integrated.macOptionIsMeta โ†’ true" + print_result info "โ€ข iTerm2: Preferences โ†’ Profiles โ†’ Keys โ†’ Option Key โ†’ Esc+" + print_result info "โ€ข Terminal.app: Preferences โ†’ Profiles โ†’ Keyboard โ†’ Use Option as Meta" + print_result info "Run 'forge zsh keyboard' for detailed keyboard shortcuts" + fi + +elif [[ "$platform" == "Linux" ]]; then + # Linux checks + if [[ "$TERM_PROGRAM" == "vscode" ]]; then + check_performed=true + # Check VS Code settings on Linux + local vscode_settings="${HOME}/.config/Code/User/settings.json" + if [[ -f "$vscode_settings" ]]; then + # On Linux, check for sendAltAsMetaKey (deprecated but still works) or macOptionIsMeta + if grep -q '"terminal.integrated.sendAltAsMetaKey"[[:space:]]*:[[:space:]]*true' "$vscode_settings" 2>/dev/null || \ + grep -q '"terminal.integrated.macOptionIsMeta"[[:space:]]*:[[:space:]]*true' "$vscode_settings" 2>/dev/null; then + print_result pass "VS Code: Alt key configured as Meta" + meta_key_ok=true + else + print_result warn "VS Code: Alt key NOT configured as Meta" + print_result instruction "Alt+F and Alt+B shortcuts won't work for word navigation" + print_result instruction "Add to VS Code settings.json:" + print_result code '"terminal.integrated.sendAltAsMetaKey": true' + print_result instruction "Then reload VS Code: Ctrl+Shift+P โ†’ Reload Window" + fi + else + print_result warn "VS Code settings file not found" + print_result info "Expected: ${vscode_settings}" + fi + elif [[ -n "$GNOME_TERMINAL_SERVICE" ]] || [[ "$COLORTERM" == "gnome-terminal" ]]; then + check_performed=true + # GNOME Terminal check + local profile_id=$(gsettings get org.gnome.Terminal.ProfilesList default 2>/dev/null | tr -d "'") + if [[ -n "$profile_id" ]]; then + local alt_sends_escape=$(gsettings get org.gnome.Terminal.Legacy.Profile:/org/gnome/terminal/legacy/profiles:/:${profile_id}/ use-theme-colors 2>/dev/null) + # GNOME Terminal doesn't have a simple setting check - most distributions enable Alt by default + print_result pass "GNOME Terminal: Alt key typically works by default" + print_result info "If Alt+F/B don't work, check: Preferences โ†’ Profile โ†’ Keyboard" + meta_key_ok=true + else + print_result info "GNOME Terminal detected" + print_result info "Alt key typically works by default for word navigation" + fi + elif [[ "$COLORTERM" == "truecolor" ]] && command -v konsole &> /dev/null; then + check_performed=true + # Konsole (KDE) - Alt typically works by default + print_result pass "Konsole: Alt key typically works by default" + print_result info "If Alt+F/B don't work, check: Settings โ†’ Edit Profile โ†’ Keyboard" + meta_key_ok=true + elif [[ -n "$ALACRITTY_SOCKET" ]] || [[ "$TERM" == "alacritty" ]]; then + check_performed=true + # Alacritty check - look for config file + local alacritty_config="${HOME}/.config/alacritty/alacritty.yml" + local alacritty_config_toml="${HOME}/.config/alacritty/alacritty.toml" + + if [[ -f "$alacritty_config" ]] || [[ -f "$alacritty_config_toml" ]]; then + print_result pass "Alacritty: Alt key typically works by default" + print_result info "If Alt+F/B don't work, ensure no conflicting key bindings" + meta_key_ok=true + else + print_result pass "Alacritty: Alt key typically works by default" + print_result info "Config: ${alacritty_config} or ${alacritty_config_toml}" + fi + elif [[ "$TERM" == "xterm" ]] || [[ "$TERM" == "xterm-256color" ]]; then + check_performed=true + # xterm - check .Xresources + local xresources="${HOME}/.Xresources" + if [[ -f "$xresources" ]]; then + if grep -q "XTerm\*metaSendsEscape:[[:space:]]*true" "$xresources" 2>/dev/null || \ + grep -q "XTerm\*eightBitInput:[[:space:]]*false" "$xresources" 2>/dev/null; then + print_result pass "xterm: Meta key configured" + meta_key_ok=true + else + print_result warn "xterm: Meta key may not be configured" + print_result instruction "Add to ~/.Xresources:" + print_result code "XTerm*metaSendsEscape: true" + print_result instruction "Then reload: xrdb ~/.Xresources" + fi + else + print_result info "xterm detected" + print_result info "To enable Alt as Meta, add to ~/.Xresources:" + print_result info "XTerm*metaSendsEscape: true" + fi + fi + + # If no specific terminal detected, provide general guidance for Linux + if [[ "$check_performed" == "false" ]]; then + print_result info "Terminal: ${TERM_PROGRAM:-$TERM}" + print_result info "For Alt key shortcuts (word navigation) to work:" + print_result info "โ€ข VS Code: Settings โ†’ terminal.integrated.sendAltAsMetaKey โ†’ true" + print_result info "โ€ข GNOME Terminal: Usually works by default" + print_result info "โ€ข Konsole: Usually works by default" + print_result info "โ€ข xterm: Add 'XTerm*metaSendsEscape: true' to ~/.Xresources" + print_result info "Run 'forge zsh keyboard' for detailed keyboard shortcuts" + fi +else + # Other platforms (BSD, etc.) + print_result info "Keyboard check: Platform ${platform} - manual verification needed" + print_result info "Ensure Alt/Meta key is configured for word navigation shortcuts" +fi + +# 8. Check font and Nerd Font support +print_section "Nerd Font" + +# Check if Nerd Font is enabled via environment variables +if [[ -n "$NERD_FONT" ]]; then + if [[ "$NERD_FONT" == "1" || "$NERD_FONT" == "true" ]]; then + print_result pass "NERD_FONT: enabled" + else + print_result warn "NERD_FONT: disabled (${NERD_FONT})" + print_result instruction "Enable Nerd Font by setting:" + print_result code "export NERD_FONT=1" + fi +elif [[ -n "$USE_NERD_FONT" ]]; then + if [[ "$USE_NERD_FONT" == "1" || "$USE_NERD_FONT" == "true" ]]; then + print_result pass "USE_NERD_FONT: enabled" + else + print_result warn "USE_NERD_FONT: disabled (${USE_NERD_FONT})" + print_result instruction "Enable Nerd Font by setting:" + print_result code "export NERD_FONT=1" + fi +else + print_result pass "Nerd Font: enabled (default)" + print_result info "Forge will auto-detect based on terminal capabilities" +fi + +# Show actual icons used in Forge theme for manual verification (skip if explicitly disabled) +local nerd_font_disabled=false +if [[ -n "$NERD_FONT" && "$NERD_FONT" != "1" && "$NERD_FONT" != "true" ]]; then + nerd_font_disabled=true +elif [[ -n "$USE_NERD_FONT" && "$USE_NERD_FONT" != "1" && "$USE_NERD_FONT" != "true" ]]; then + nerd_font_disabled=true +fi + +if [[ "$nerd_font_disabled" == "false" ]]; then + echo "" + echo "$(yellow "Visual Check [Manual Verification Required]")" +echo " $(bold "๓ฑ™บ FORGE 33.0k") $(cyan "๎ฐ™ tonic-1.0")" + echo "" + echo " Forge uses Nerd Fonts to enrich cli experience, can you see all the icons clearly without any overlap?" + echo " If you see boxes (โ–ก) or question marks (?), install a Nerd Font from:" + echo " $(dim "https://www.nerdfonts.com/")" + echo "" +fi + +# Summary +echo "" + +if [[ $failed -eq 0 && $warnings -eq 0 ]]; then + echo "$(green "${PASS}") $(bold "All checks passed") $(dim "(${passed})")" + exit 0 +elif [[ $failed -eq 0 ]]; then + echo "$(yellow "${WARN}") $(bold "${warnings} warnings") $(dim "(${passed} passed)")" + exit 0 +else + echo "$(red "${FAIL}") $(bold "${failed} failed") $(dim "(${warnings} warnings, ${passed} passed)")" + exit 1 +fi diff --git a/shell-plugin/forge.plugin.zsh b/shell-plugin/forge.plugin.zsh new file mode 100644 index 0000000000000000000000000000000000000000..d988364bac412907632ce540bd8ee9f9e141e2dc --- /dev/null +++ b/shell-plugin/forge.plugin.zsh @@ -0,0 +1,35 @@ +#!/usr/bin/env zsh + +# Documentation in [README.md](./README.md) + +# Modular forge plugin - sources all required modules + +# Configuration variables +source "${0:A:h}/lib/config.zsh" + +# Syntax highlighting +source "${0:A:h}/lib/highlight.zsh" + +# Core utilities (includes logging) +source "${0:A:h}/lib/helpers.zsh" + +# Terminal context capture (preexec/precmd hooks, OSC 133) +source "${0:A:h}/lib/context.zsh" + +# Completion widget +source "${0:A:h}/lib/completion.zsh" + +# Action handlers +source "${0:A:h}/lib/actions/core.zsh" +source "${0:A:h}/lib/actions/config.zsh" +source "${0:A:h}/lib/actions/conversation.zsh" +source "${0:A:h}/lib/actions/git.zsh" +source "${0:A:h}/lib/actions/auth.zsh" +source "${0:A:h}/lib/actions/editor.zsh" +source "${0:A:h}/lib/actions/provider.zsh" + +# Main dispatcher and widget registration +source "${0:A:h}/lib/dispatcher.zsh" + +# Key bindings and widget registration +source "${0:A:h}/lib/bindings.zsh" diff --git a/shell-plugin/forge.setup.zsh b/shell-plugin/forge.setup.zsh new file mode 100644 index 0000000000000000000000000000000000000000..76e203990529a6b61fc5c61abdd06c85d6c6df07 --- /dev/null +++ b/shell-plugin/forge.setup.zsh @@ -0,0 +1,20 @@ +# !! Contents within this block are managed by 'forge zsh setup' !! +# !! Do not edit manually - changes will be overwritten !! + +# Add required zsh plugins if not already present +if [[ ! " ${plugins[@]} " =~ " zsh-autosuggestions " ]]; then + plugins+=(zsh-autosuggestions) +fi +if [[ ! " ${plugins[@]} " =~ " zsh-syntax-highlighting " ]]; then + plugins+=(zsh-syntax-highlighting) +fi + +# Load forge shell plugin (commands, completions, keybindings) if not already loaded +if [[ -z "$_FORGE_PLUGIN_LOADED" ]]; then + eval "$(forge zsh plugin)" +fi + +# Load forge shell theme (prompt with AI context) if not already loaded +if [[ -z "$_FORGE_THEME_LOADED" ]]; then + eval "$(forge zsh theme)" +fi diff --git a/shell-plugin/keyboard.zsh b/shell-plugin/keyboard.zsh new file mode 100644 index 0000000000000000000000000000000000000000..9b2288b03526c14883561d17de1a80333a3288c8 --- /dev/null +++ b/shell-plugin/keyboard.zsh @@ -0,0 +1,160 @@ +#!/usr/bin/env zsh + +# ZSH Keyboard Shortcuts - Display ZLE keyboard shortcuts +# Shows platform-specific keyboard shortcuts for ZSH Line Editor + +# ANSI codes +RESET='\033[0m' +BOLD='\033[1m' +DIM='\033[2m' +CYAN='\033[0;36m' + +# Text formatting helpers - auto-reset +function bold() { echo "${BOLD}${1}${RESET}"; } +function dim() { echo "${DIM}${1}${RESET}"; } +function cyan() { echo "${CYAN}${1}${RESET}"; } + +# Helper function to print section headers +function print_section() { + echo "" + echo "$(bold "$1")" +} + +# Helper function to print shortcuts with automatic padding +# Usage: print_shortcut "key" "description" +# If only one argument, prints it as-is (for special messages) +function print_shortcut() { + local key=$1 + local description=$2 + + if [[ -z "$description" ]]; then + # Single argument - print as-is (for configuration lines) + echo " $(dim "${key}")" + else + # Two arguments - pad the key and align descriptions + # Calculate padding based on key length (max 20 chars for alignment) + local padding=20 + local key_len=${#key} + local pad_len=$((padding - key_len)) + local pad="" + if [[ $pad_len -gt 0 ]]; then + printf -v pad "%${pad_len}s" "" + fi + printf " $(cyan "%s")%s%s\n" "$key" "$pad" "$description" + fi +} + +# Detect platform +platform="unknown" +alt_key="Alt" +if [[ "$(uname)" == "Darwin" ]]; then + platform="macOS" + alt_key="Option" +elif [[ "$(uname)" == "Linux" ]]; then + platform="Linux" +elif [[ "$(uname)" =~ "MINGW" ]] || [[ "$(uname)" =~ "MSYS" ]] || [[ "$(uname)" =~ "CYGWIN" ]] || [[ -n "$WINDIR" ]]; then + platform="Windows" +fi + +# Detect if vi/vim mode is enabled +vi_mode=false +# Check if main keymap is bound to viins or vicmd +if bindkey -lL main 2>/dev/null | grep -q "bindkey -A viins main\|bindkey -A vicmd main"; then + vi_mode=true +fi + +# Show platform and mode info +print_section "Configuration" +if [[ "$platform" != "unknown" ]]; then + print_shortcut "Platform: ${platform}" +fi +if [[ "$vi_mode" == "true" ]]; then + print_shortcut "Mode: Vi/Vim keybindings" +else + print_shortcut "Mode: Emacs keybindings (default)" +fi + +if [[ "$vi_mode" == "true" ]]; then + # Vim mode shortcuts + print_section "Mode Switching" + print_shortcut "ESC / Ctrl+[" "Enter command mode (normal)" + print_shortcut "i" "Enter insert mode" + print_shortcut "a" "Enter insert mode (after cursor)" + print_shortcut "A" "Enter insert mode (end of line)" + print_shortcut "I" "Enter insert mode (start of line)" + + print_section "Navigation (Command Mode)" + print_shortcut "w" "Move forward one word" + print_shortcut "b" "Move backward one word" + print_shortcut "0 / ^" "Move to beginning of line" + print_shortcut "\$" "Move to end of line" + + print_section "Editing (Command Mode)" + print_shortcut "dd" "Delete entire line" + print_shortcut "D" "Delete from cursor to end of line" + print_shortcut "cc" "Change entire line" + print_shortcut "C" "Change from cursor to end of line" + print_shortcut "cw" "Change word" + print_shortcut "dw" "Delete word" + print_shortcut "u" "Undo" + print_shortcut "p" "Paste after cursor" + print_shortcut "P" "Paste before cursor" + + print_section "History (Command Mode)" + print_shortcut "k / โ†‘" "Previous command" + print_shortcut "j / โ†“" "Next command" + print_shortcut "/" "Search history backward" + print_shortcut "?" "Search history forward" + print_shortcut "n" "Next search match" + print_shortcut "N" "Previous search match" + + print_section "Insert Mode" + print_shortcut "Ctrl+W" "Delete word before cursor" + print_shortcut "Ctrl+U" "Delete from cursor to start" + + print_section "Other" + print_shortcut "v" "Edit command in \$EDITOR" + print_shortcut "Ctrl+L" "Clear screen" + print_shortcut "Ctrl+C" "Cancel current command" + print_shortcut "Ctrl+Z" "Suspend current command" + print_shortcut "Tab" "Complete command/path" +else + # Emacs mode shortcuts (default) + print_section "Line Navigation" + print_shortcut "Ctrl+A" "Move to beginning of line" + print_shortcut "Ctrl+E" "Move to end of line" + print_shortcut "${alt_key}+F" "Move forward one word" + print_shortcut "${alt_key}+B" "Move backward one word" + + print_section "Editing" + print_shortcut "Ctrl+U" "Kill line before cursor" + print_shortcut "Ctrl+K" "Kill line after cursor" + print_shortcut "Ctrl+W" "Kill word before cursor" + print_shortcut "${alt_key}+D" "Kill word after cursor" + print_shortcut "Ctrl+Y" "Yank (paste) killed text" + print_shortcut "Ctrl+_" "Undo last edit" + + print_section "History" + print_shortcut "Ctrl+R" "Search command history backward" + print_shortcut "Ctrl+S" "Search command history forward" + print_shortcut "Ctrl+P / โ†‘" "Previous command" + print_shortcut "Ctrl+N / โ†“" "Next command" + print_shortcut "${alt_key}+<" "Move to first history entry" + print_shortcut "${alt_key}+>" "Move to last history entry" + + print_section "Other" + print_shortcut "Ctrl+L" "Clear screen" + print_shortcut "Ctrl+C" "Cancel current command" + print_shortcut "Ctrl+Z" "Suspend current command" + print_shortcut "Tab" "Complete command/path" + + echo "" + if [[ "$platform" == "macOS" ]]; then + echo " $(dim "If Option key shortcuts don't work, run: forge zsh doctor")" + elif [[ "$platform" == "Linux" ]]; then + echo " $(dim "If Alt key shortcuts don't work, run: forge zsh doctor")" + fi + echo " $(dim "To enable Vi mode, add to ~/.zshrc: bindkey -v")" +fi + +echo "" diff --git a/templates/forge-command-generator-prompt.md b/templates/forge-command-generator-prompt.md new file mode 100644 index 0000000000000000000000000000000000000000..8fab156a054d7ee7f90023e69fd48bdc9a694ed0 --- /dev/null +++ b/templates/forge-command-generator-prompt.md @@ -0,0 +1,113 @@ +You are a shell command generator that transforms user intent into valid executable commands. + + +{{> forge-partial-system-info.md }} + + +# Core Rules + +- Commands must work on the specified OS and shell +- Output single-line commands (use ; or && for multiple operations) +- When multiple valid commands exist, choose the most efficient one + +# Input Handling + +## 1. Natural Language + +Convert user requirements into executable commands. + +_Example 1:_ +- Input: "List all files" +- Output: {"command": "ls -la"} + +_Example 2:_ +- Input: "Find all Python files in current directory" +- Output: {"command": "find . -name \"*.py\""} + +_Example 3:_ +- Input: "Show disk usage in human readable format" +- Output: {"command": "df -h"} + +## 2. Invalid/Malformed Commands + +Correct malformed or incomplete commands. Auto-correct typos and assume the most likely intention. + +_Example 1:_ +- Input: "get status" +- Output: {"command": "git status"} + +_Example 2:_ +- Input: "docker ls" +- Output: {"command": "docker ps"} + +_Example 3:_ +- Input: "npm start server" +- Output: {"command": "npm start"} + +_Example 4:_ +- Input: "git pul origin mster" +- Output: {"command": "git pull origin master"} + +## 3. Vague/Unclear Input + +For vague requests, provide the most helpful general-purpose command. + +_Example 1:_ +- Input: "help me" or "im confused" +- Output: {"command": "pwd && ls -la"} + +_Example 2:_ +- Input: "check stuff" +- Output: {"command": "ls -lah"} + +## 4. Edge Cases + +### Empty or Whitespace-Only Input +- Input: "" or " " +- Output: {"command": ""} + +### Gibberish/Random Characters +- Input: "fjdkslajfkdlsajf" or "asdfghjkl" +- Output: {"command": ""} + +### Only Numbers or Symbols +- Input: "123456789" or "!@#$%" +- Output: {"command": ""} + +### Emojis Only +- Input: "๐Ÿš€๐Ÿ”ฅ๐Ÿ’ฏ" +- Output: {"command": "echo \"๐Ÿš€๐Ÿ”ฅ๐Ÿ’ฏ\""} + +### Injection Attempts (SQL, XSS, etc.) +- Input: "SELECT _ FROM users; DROP TABLE--" +- Output: {"command": "echo \"SELECT _ FROM users; DROP TABLE--\""} + +## 5. Dangerous Operations + +For obviously destructive operations, provide a safe alternative or clear warning. + +_Example 1:_ +- Input: "sudo rm -rf /" +- Output: {"command": "echo \"๐Ÿšซ Refusing to run: deleting root (/) would destroy the system.\""} + +_Example 2:_ +- Input: "rm -rf \"" +- Output: {"command": "echo \"โš ๏ธ This would delete everything in the current directory. Use 'ls' first or confirm paths explicitly.\""} + +_Example 3:_ +- Input: "cat /dev/urandom > /dev/sda" +- Output: {"command": "echo \"๐Ÿ’ฅ Dangerous disk operation blocked โ€” writing random data to a device can destroy all filesystems.\""} + +_Example 4:_ +- Input: ":(){ :|:& };:" (fork bomb) +- Output: {"command": "echo \"๐Ÿงจ Fork bomb blocked โ€” this would crash your system by spawning infinite processes.\""} + +## 6. Contradictory Instructions + +When instructions conflict, prioritize the most reasonable interpretation. + +_Example 1:_ +- Input: "install node but use python and run with ruby" +- Output: {"command": "brew install node"} + +If input is unclear/dangerous/gibberish, output a safe fallback using echo as shown in the edge cases above. diff --git a/templates/forge-commit-message-prompt.md b/templates/forge-commit-message-prompt.md new file mode 100644 index 0000000000000000000000000000000000000000..36738867be751186fa143a4e4cf362aef343854f --- /dev/null +++ b/templates/forge-commit-message-prompt.md @@ -0,0 +1,40 @@ +You are a commit message generator that creates concise, conventional commit messages from git diffs. + +IMPORTANT: Return ONLY raw Text. No markdown. No code blocks. No ``` markers. + +# Commit Message Format +Structure: type(scope): description +- **Type**: feat, fix, refactor, perf, docs, style, test, chore, ci, build, revert +- **Scope**: optional, component/module name (lowercase, no spaces) +- **Description**: imperative mood, lowercase, no period, 10-72 characters +- **Breaking changes**: add ! after type/scope (e.g., refactor!: or feat(api)!:) + +# Rules +1. **Single line only** - never use multiple lines or bullet points +2. **Focus on what changed** - describe the primary change, not implementation details +3. **Be specific** - mention the affected component/module when relevant +4. **Exclude issue/PR references** - never include issue or PR numbers like (#1234) in the commit message +5. **Match project style** - analyze recent_commit_messages for patterns (scope usage, verbosity), but ignore any issue/PR references +6. **Imperative mood** - use "add" not "adds" or "added" +7. **Conciseness** - shorter is better; avoid redundant words like "improve", "update", "enhance" unless necessary + +# Input Analysis Priority +1. **git_diff** - primary source for understanding the actual changes +2. **additional_context** - user-provided context to help structure the commit message (if provided, use this information to guide the commit message structure and focus) +3. **recent_commit_messages** - reference for project's commit message style and conventions +4. **branch_name** - additional context hint (feature/, fix/, etc.) + +# Examples +Good commit messages: +- feat(auth): add OAuth2 login support +- fix(api): handle null response in user endpoint +- refactor(db): simplify query builder interface +- docs(readme): update installation instructions +- perf(parser): optimize token scanning algorithm + +Bad commit messages (avoid these): +- refactor: improve the authentication system by adding new OAuth2 support and updating the login flow (too verbose) +- fix: fix bug (too vague) +- Add new feature (not lowercase, missing type) + +REMINDER: Output raw text directly. Do NOT use ```json or ``` or any markdown. \ No newline at end of file diff --git a/templates/forge-custom-agent-template.md b/templates/forge-custom-agent-template.md new file mode 100644 index 0000000000000000000000000000000000000000..8544b0ba4cd64f6e1bb81335bb35f135ef30f857 --- /dev/null +++ b/templates/forge-custom-agent-template.md @@ -0,0 +1,58 @@ + +{{> forge-partial-system-info.md }} + + +{{#if (not tool_supported)}} + +{{tool_information}} + + +{{> forge-partial-tool-use-example.md }} + +{{/if}} + + +{{#if (not tool_supported)}} +- You have access to set of tools as described in the tag. +- You can use one tool per message, and will receive the result of that tool use in the user's response. +- You use tools step-by-step to accomplish a given task, with each tool use informed by the result of the previous tool use. +{{else}} +- For maximum efficiency, whenever you need to perform multiple independent operations, invoke all relevant tools (for eg: `patch`, `read`) simultaneously rather than sequentially. +{{/if}} +- NEVER ever refer to tool names when speaking to the USER even when user has asked for it. For example, instead of saying 'I need to use the edit_file tool to edit your file', just say 'I will edit your file'. +- If you need to read a file, prefer to read larger sections of the file at once over multiple smaller calls. + + +{{#if custom_rules}} + +{{custom_rules}} + +{{/if}} + + +- ALWAYS present the result of your work in a neatly structured format (using markdown syntax in your response) to the user at the end of every task. +- Do what has been asked; nothing more, nothing less. +- NEVER create files unless they're absolutely necessary for achieving your goal. +- ALWAYS prefer editing an existing file to creating a new one. +- NEVER create documentation files (\*.md, \*.txt, README, CHANGELOG, CONTRIBUTING, etc.) unless explicitly requested by the user. Includes summaries/overviews, architecture docs, migration guides/HOWTOs, or any explanatory file about work just completed. Instead, explain in your reply in the final response or use code comments. "Explicitly requested" means the user asks for a specific document by name or purpose. +- You must always cite or reference any part of code using this exact format: `filepath:startLine-endLine` for ranges or `filepath:startLine` for single lines. Do not use any other format. +- The conversation has unlimited context through automatic summarization, so do not stop until the objective is fully achieved. + + **Good examples:** + + - `src/main.rs:10` (single line) + - `src/utils/helper.rs:25-30` (range) + - `lib/core.rs:100-150` (larger range) + + **Bad examples:** + + - "line 10 of main.rs" + - "see src/main.rs lines 25-30" + - "check main.rs" + - "in the helper.rs file around line 25" + - `crates/app/src/lib.rs` (lines 1-4) + +- User may tag files using the format @[] and send it as a part of the message. Do not attempt to reread those files. +- Only use emojis if the user explicitly requests it. Avoid using emojis in all communication unless asked. +{{#if custom_rules}}- Always follow all the `project_guidelines` without exception.{{/if}} + diff --git a/templates/forge-doom-loop-reminder.md b/templates/forge-doom-loop-reminder.md new file mode 100644 index 0000000000000000000000000000000000000000..833be5600dfded826d6780503833b152f146bf40 --- /dev/null +++ b/templates/forge-doom-loop-reminder.md @@ -0,0 +1,6 @@ +You appear to be stuck in a repetitive loop, having made {{consecutive_calls}} similar calls. +This indicates you are not making progress. Please: + +1. Reconsider your approach to solving this problem +2. Try a different tool or different arguments +3. If you're stuck, explain what you're trying to accomplish and ask for clarification diff --git a/templates/forge-partial-skill-instructions.md b/templates/forge-partial-skill-instructions.md new file mode 100644 index 0000000000000000000000000000000000000000..700b359f837c14de17eb0b59574a552d33aaad49 --- /dev/null +++ b/templates/forge-partial-skill-instructions.md @@ -0,0 +1,45 @@ +## Skill Instructions: + +**CRITICAL**: Before attempting any task, ALWAYS check if a skill exists for it in the available_skills list below. Skills are specialized workflows that must be invoked when their trigger conditions match the user's request. + +How skills work: + +1. **Invocation**: Use the `skill` tool with just the skill name parameter + + - Example: Call skill tool with `{"name": "mock-calculator"}` + - No additional arguments needed + +2. **Response**: The tool returns the skill's details wrapped in `` containing: + + - `` - The complete SKILL.md file content with the skill's path + - `` tags - List of additional resource files available in the skill directory + - Includes usage guidelines, instructions, and any domain-specific knowledge + +3. **Action**: Read and follow the instructions provided in the skill content + - The skill instructions will tell you exactly what to do and how to use the resources + - Some skills provide workflows, others provide reference information + - Apply the skill's guidance to complete the user's task + +Examples of skill invocation: + +- To invoke calculator skill: use skill tool with name "calculator" +- To invoke weather skill: use skill tool with name "weather" +- For namespaced skills: use skill tool with name "office-suite:pdf" + +Important: + +- Only invoke skills listed in `` below +- Do not invoke a skill that is already active/loaded +- Skills are not CLI commands - use the skill tool to load them +- After loading a skill, follow its specific instructions to help the user + + +{{#each skills}} + +{{this.name}} + +{{this.description}} + + +{{/each}} + diff --git a/templates/forge-partial-summary-frame.md b/templates/forge-partial-summary-frame.md new file mode 100644 index 0000000000000000000000000000000000000000..e48d69ca40404ed08d23096a01f3735b13bb9efc --- /dev/null +++ b/templates/forge-partial-summary-frame.md @@ -0,0 +1,63 @@ +Use the following summary frames as the authoritative reference for all coding suggestions and decisions. Do not re-explain or revisit it unless I ask. Additional summary frames will be added as the conversation progresses. + +## Summary + +{{#each messages}} +### {{inc @index}}. {{role}} + +{{#each contents}} +{{#if text}} +```` +{{text}} +```` +{{/if}} +{{~#if tool_call}} +{{#if tool_call.tool.file_update}} +**Update:** `{{tool_call.tool.file_update.path}}` +{{else if tool_call.tool.file_read}} +**Read:** `{{tool_call.tool.file_read.path}}` +{{else if tool_call.tool.file_remove}} +**Delete:** `{{tool_call.tool.file_remove.path}}` +{{else if tool_call.tool.search}} +**Search:** `{{tool_call.tool.search.pattern}}` +{{else if tool_call.tool.skill}} +**Skill:** `{{tool_call.tool.skill.name}}` +{{else if tool_call.tool.sem_search}} +**Semantic Search:** +{{#each tool_call.tool.sem_search.queries}} +- `{{use_case}}` +{{/each}} +{{else if tool_call.tool.shell}} +**Execute:** +``` +{{tool_call.tool.shell.command}} +``` +{{else if tool_call.tool.mcp}} +**MCP:** `{{tool_call.tool.mcp.name}}` +{{else if tool_call.tool.todo_write}} +**Task Plan:** +{{#each tool_call.tool.todo_write.changes}} +{{#if (eq kind "added")}} +- [ADD] {{todo.content}} +{{else if (eq kind "updated")}} +{{#if (eq todo.status "completed")}} +- [DONE] ~~{{todo.content}}~~ +{{else if (eq todo.status "in_progress")}} +- [IN_PROGRESS] {{todo.content}} +{{else}} +- [UPDATE] {{todo.content}} +{{/if}} +{{else if (eq kind "removed")}} +- [CANCELLED] ~~{{todo.content}}~~ +{{/if}} +{{/each}} +{{/if~}} +{{/if~}} + +{{/each}} + +{{/each}} + +--- + +Proceed with implementation based on this context. diff --git a/templates/forge-partial-system-info.md b/templates/forge-partial-system-info.md new file mode 100644 index 0000000000000000000000000000000000000000..47a59ef81ca5ac7cc131dc50705b8a7b9efd23fe --- /dev/null +++ b/templates/forge-partial-system-info.md @@ -0,0 +1,15 @@ +{{env.os}} +{{env.cwd}} +{{env.shell}} +{{env.home}} +{{#if files}} + +{{#each files}} - {{path}}{{#if is_dir}}/{{/if}} +{{/each}} +{{/if}} +{{#if extensions}} + +{{#each extensions.extension_stats}} - .{{extension}}: {{count}} files ({{percentage}}%) +{{/each}}{{#if (gt extensions.total_extensions extensions.max_extensions)}}(showing top {{extensions.max_extensions}} of {{extensions.total_extensions}} extensions; other extensions account for {{extensions.remaining_percentage}}% of files) +{{/if}} +{{/if}} \ No newline at end of file diff --git a/templates/forge-partial-tool-error-reflection.md b/templates/forge-partial-tool-error-reflection.md new file mode 100644 index 0000000000000000000000000000000000000000..6ecc3ce217d6497f86619adb157f19e45c88354f --- /dev/null +++ b/templates/forge-partial-tool-error-reflection.md @@ -0,0 +1,6 @@ +You must now deeply reflect on the error above: +1. Pinpoint exactly what was wrong with the tool call โ€” was it the wrong tool, incorrect or missing parameters, or malformed structure? +2. Explain why that mistake happened. Did you misunderstand the tool's schema? Miss a required field? Misread the context? +3. Make the correct tool call as it should have been made. + +Do NOT skip this reflection. \ No newline at end of file diff --git a/templates/forge-partial-tool-use-example.md b/templates/forge-partial-tool-use-example.md new file mode 100644 index 0000000000000000000000000000000000000000..baa06d9357196e32fb373beb82b679f471988cc9 --- /dev/null +++ b/templates/forge-partial-tool-use-example.md @@ -0,0 +1,26 @@ +1. You can only make one tool call per message. +2. Each tool call must be wrapped in `` tags. +3. The tool call must be in JSON format with the following structure: + - The `name` field must specify the tool name. + - The `arguments` field must contain the required parameters for the tool. + +Here's a correct example structure: + +Example 1: + +{"name": "read", "arguments": {"path": "/a/b/c.txt"}} + + +Example 2: + +{"name": "write", "arguments": {"path": "/a/b/c.txt", "content": "Hello World!"}} + + +Important: +1. ALWAYS use JSON format inside `forge_tool_call` tags. +2. Specify the name of tool in the `name` field. +3. Specify the tool arguments in the `arguments` field. +4. If you need to make multiple tool calls, send them in separate messages. + +Before using a tool, ensure all required arguments are available. +If any required arguments are missing, do not attempt to use the tool. diff --git a/templates/forge-pending-todos-reminder.md b/templates/forge-pending-todos-reminder.md new file mode 100644 index 0000000000000000000000000000000000000000..4cbcc00cc59f8ab4acc839681888b2084003f80f --- /dev/null +++ b/templates/forge-pending-todos-reminder.md @@ -0,0 +1,7 @@ +You have pending todo items that must be completed before finishing the task: + +{{#each todos}} +- [{{this.status}}] {{this.content}} +{{/each}} + +Please complete all pending items before finishing. \ No newline at end of file diff --git a/templates/forge-system-prompt-title-generation.md b/templates/forge-system-prompt-title-generation.md new file mode 100644 index 0000000000000000000000000000000000000000..4ef76d30f1381390dd3786908534c90c1dcf38bb --- /dev/null +++ b/templates/forge-system-prompt-title-generation.md @@ -0,0 +1,8 @@ +You are Title Generator, an expert assistant that analyzes user tasks and generates precise, impactful titles for user prompts. + +## Core Requirements: + +- **Length**: 3โ€“7 words preferred +- **Format**: Title case (e.g., "Advanced File Processing System") without Markdown Formatting +- **Style**: Technical, clear, and informative +- **Focus**: Capture core functionality without marketing language. diff --git a/templates/forge-tool-retry-message.md b/templates/forge-tool-retry-message.md new file mode 100644 index 0000000000000000000000000000000000000000..4f8ad046d9aeddc2bd5179166c9902ef636f0575 --- /dev/null +++ b/templates/forge-tool-retry-message.md @@ -0,0 +1,3 @@ +Tool call failed +- **Attempts remaining:** {{attempts_left}} +- **Next steps:** Analyze the error, identify the root cause, and adjust your approach before retrying. \ No newline at end of file