A production-grade Kubernetes-based tool for tracing and monitoring errors in MiniUdm deployments. This tool helps collect symptoms, apply patches, and monitor async error patterns in Kubernetes clusters.
- Deployment Monitoring: List and monitor Kubernetes deployments with detailed status
- Error Tracing: Collect async error traces from pods with configurable keyword detection
- Patch Management: Apply patches to services with MD5 validation and health monitoring
- Symptom Collection: Monitor multiple log files and collect error symptoms in parallel
- Production-Ready: Comprehensive error handling, logging, configuration management, and CI/CD
The project follows Go best practices with a clear separation of concerns:
.
├── cmd/ # Command-line applications
│ ├── list-deployments/ # List Kubernetes deployments
│ ├── symptom-collection/ # Start symptom collection
│ └── apply-patch/ # Apply patches to services
├── pkg/ # Reusable packages
│ ├── kubernetes/ # Kubernetes client wrapper
│ ├── utils/ # Utility functions (file, hash, command)
│ ├── patch/ # Patch application logic
│ ├── symptom/ # Symptom collection logic
│ └── config/ # Configuration management
├── internal/ # Internal packages (not for external use)
│ ├── logger/ # Logging utilities
│ └── validator/ # Validation utilities
├── configs/ # Configuration files
├── examples/ # Example code
├── docs/ # Documentation
└── scripts/ # Build and utility scripts
- Go 1.21 or higher
- Kubernetes cluster access
- kubectl configured with cluster access
- Make (optional, for using Makefile)
git clone https://github.com/Ricky512227/MiniUdmAsyncErrorTracing.git
cd MiniUdmAsyncErrorTracing
# Initial setup: Update BUILD files with Gazelle
bazel run //:gazelle
# Build all binaries
make build
# or
bazel build //cmd/...The binaries will be available in the bazel-bin/ directory.
Note: On first setup, run bazel run //:gazelle to ensure BUILD files are properly configured with dependencies from go.mod.
# Build all binaries
bazel build //cmd/list-deployments:list-deployments
bazel build //cmd/symptom-collection:symptom-collection
bazel build //cmd/apply-patch:apply-patch
# Run tests
bazel test //...
# Run a specific binary
bazel run //cmd/list-deployments:list-deployments -- -n default# Build all binaries
make build
# Build specific binary
make build-list
make build-symptom
make build-patch
# Run tests
make test
# Update BUILD files with Gazelle
make gazelle
# Format code (Go)
make format
# Run linter
make lintThe tool can be configured via:
- Configuration file (
configs/config.yaml) - Environment variables (prefixed with
MINIUDM_) - Command-line flags
See configs/config.yaml for a complete example:
kubernetes:
namespace: "default"
timeout: "30s"
paths:
tcn_vol_path: "/tcnVol"
lib64_path: "/opt/SMAW/INTP/lib64"
logging:
level: "info"
format: "json"
symptom:
error_keywords:
- "error"
- "ERROR"
- "fatal"
check_interval: "1s"
collection_timeout: "10m"
patch:
backup_enabled: true
health_timeout: "30s"List all deployments in a namespace:
./bin/list-deployments -n default
# Or with custom config
./bin/list-deployments -n my-namespace -c /path/to/config.yamlStart symptom collection for specific pods:
./bin/symptom-collection -n default -p "uecm nim"
# With custom config
./bin/symptom-collection -n miniudm -p "uecm" -c /path/to/config.yamlThe symptom collection process:
- Enables traces for processes
- Enables pcap capture
- Executes test commands (pybot)
- Monitors log files for errors:
/cmconfig.log/logstore/TspCore/RTPTraceError/Envoy/dumplog
- Collects and stores traces for analysis
Apply a patch file to a service:
./bin/apply-patch -p /path/to/patch.so -s service-name
# With custom config
./bin/apply-patch -p /path/to/patch.so -s uecm -c /path/to/config.yamlThe patch application process:
- Validates patch file (MD5 checksum)
- Copies patch to
/tcnVol - Links library files to
/opt/SMAW/INTP/lib64 - Restarts service processes
- Monitors process health
- Logs success/failure status
# Run all tests with Bazel
bazel test //...
# or
make test
# Run tests with coverage
make test-cover# Format code
make format
# Check formatting
make fmt-check
# Run linter
make lint
# Run all checks
make verify
# Update BUILD files after adding dependencies
make gazelle- cmd/: Each subdirectory contains a main package for a command-line tool
- pkg/: Public packages that can be imported by other projects
- internal/: Private packages specific to this project
- configs/: Configuration file templates
- examples/: Example code demonstrating usage
Contributions are welcome! Please follow these guidelines:
- Fork the repository
- Create a feature branch (
git checkout -b feature/amazing-feature) - Make your changes
- Add tests for new functionality
- Ensure all tests pass (
make test) - Format code (
make format) - Commit your changes (
git commit -m 'Add amazing feature') - Push to the branch (
git push origin feature/amazing-feature) - Open a Pull Request
- Follow Go code review comments: https://github.com/golang/go/wiki/CodeReviewComments
- Use
gofmtfor formatting - Add comments for exported functions and types
- Write tests for new features
The project uses GitHub Actions for continuous integration:
- Test: Runs tests on every push and pull request
- Lint: Validates code quality using golangci-lint
- Build: Builds binaries for multiple platforms
This project is open source and available for use. See LICENSE file for details.
Ricky512227
Built with:
- Kubernetes client-go
- Cobra - CLI framework
- Viper - Configuration management
- Zap - Logging