Skip to content

Conversation

@matthewgoodman13
Copy link
Member

Summary

Integrate AWS Knowledge MCP Server as a proxy to provide up-to-date AWS documentation access, including new ECS features beyond standard LLM training data. Add tool transformations to append ECS-specific guidance and register prompt patterns for new features.

Changes

  • Add aws_knowledge_proxy module with proxy setup and tool transformations
  • Enhance server with lifespan management for async operations
  • Add comprehensive integration tests for the new tools
    • These test that the upstream tools are accessible and have not changed. We validate if there is exactly 3 tools, and that the description is exactly what we expect. This way, if a change happens upstream, we will know about it.
  • Update dependencies to support pytest-asyncio for async testing
  • Document new AWS Knowledge tools and usage examples in README
  • Update the troubleshooting tools integ test helper due to changes in mcp-inspector 0.16.6 w.r.t. Parse tool args from CLI as JsonValue instead of just strings

User experience

Customers can retrieve up-to-date ECS documentation, best practices, and knowledge of new tools that models may not be trained on, such as Blue-Green deployments.

Testing

  • Achieved 99%+ coverage on new changes with added UTs
  • Unit Tests Pass
  • Integ. Tests Pass
  • All pre-commit Tests Pass

Checklist

If your change doesn't seem to apply, please leave them unchecked.

  • [Y] I have reviewed the contributing guidelines
  • [Y] I have performed a self-review of this change
  • [Y] Changes have been tested
  • [Y] Changes are documented

Is this a breaking change? (Y/N) N

RFC issue number:

Checklist:

  • [NA] Migration process documented
  • [NA] Implement warnings (if it can live side by side)

Acknowledgment

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of the project license.

Copy link
Contributor

@guitar80ep guitar80ep left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Directionally, it all looks correct. My comment areas:

  1. Can we provide a signal in the tools doc to the LLM that this proxy is duplicative if the customer has the AWS KB MCP installed? It’s not the end of the world to repeat work, but it’s less than ideal
  2. will we plan to update the prompts on any new release?
  3. What kinds of manual tests did we do?

Smaller comments:

  1. did we have to do the main.py refactor in this CR?
  2. Any best practices for Fast MCP Proxy that we discovered?

@matthewgoodman13
Copy link
Member Author

matthewgoodman13 commented Sep 22, 2025

1. Can we provide a signal in the tools doc to the LLM that this proxy is duplicative if the customer has the AWS KB MCP installed?
=> See response above

2. will we plan to update the prompts on any new release?
=> I think we should, unless we have proof from testing, that we do not need to.

3. What kinds of manual tests did we do?
=> Asked about AWS knowledge, asked about ECS knowledge (ie. best guidance on setting up an ECS cluster, best guidance on security), asked different questions about Blue-Green deployments, including how to setup, and LLM reliably used the proxied tools once they had the updated descriptions and prompt patterns.

4. did we have to do the main.py refactor in this CR?
=> I had to update the main.py to use the async lifespan management feature to allow us to asynchronously update the tool descriptions once they are loaded. The current file structure made it hard to unit test, so I did a minor refactor. I didn't consider that a big change as code is the same, but just now in methods. Cleaner and we also increase unit test coverage on this file by 50-60%

5. Any best practices for Fast MCP Proxy that we discovered?**
=> To implement the proxy, I looked into what the docs say, and also what other teams have done in the awslabs. My implementation seems like the generic one people use, including adding the lifespan param above.

Copy link
Contributor

@guitar80ep guitar80ep left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved! We can iterate from here - but I think we have raised the bar for testing compared to previous CRs. In addition, it'll be nice to get this in the hands of customers to see what other feedback we should address.

@matthewgoodman13 matthewgoodman13 added this pull request to the merge queue Sep 22, 2025
Merged via the queue into main with commit d6dca9a Sep 22, 2025
202 checks passed
@matthewgoodman13 matthewgoodman13 deleted the ecs-mcp-server/proxy-kb branch September 22, 2025 18:25
@github-project-automation github-project-automation bot moved this from To triage to Done in awslabs/mcp Project Sep 22, 2025
@codecov
Copy link

codecov bot commented Sep 22, 2025

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 88.90%. Comparing base (34f4302) to head (7191c6a).
⚠️ Report is 3 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1332      +/-   ##
==========================================
- Coverage   89.40%   88.90%   -0.50%     
==========================================
  Files         708      460     -248     
  Lines       49392    26992   -22400     
  Branches     7748     4316    -3432     
==========================================
- Hits        44160    23998   -20162     
+ Misses       3398     1932    -1466     
+ Partials     1834     1062     -772     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

3 participants