Create input validation utility for environment variables

[coderabbitai]

Description

Create a utility function in services/utils/ to validate required environment variables at service startup, providing clear error messages when config is missing.

Context

Services currently fail with cryptic errors when environment variables are missing. A validation utility would provide better error messages and fail-fast behavior.

Tasks

• Create services/utils/env_validator.py
• Implement validate_env_vars(required_vars: List[str]) function
• Function should:
  • Check if required env vars are set
  • Raise helpful error if missing (list all missing vars)
  • Optionally validate format (e.g., port numbers are integers)
  • Return a dict of validated values
• Add unit tests in tests/unit/test_env_validator.py
• Update services/ingest/app.py to use the validator

Acceptance Criteria

• New module created with proper docstrings
• Function has >90% test coverage
• Provides clear error messages
• Example integration in ingest service
• Does not break existing functionality

Resources

• Environment template: .env.template
• Existing utils: services/utils/logging_config.py
• Python os.environ docs

Estimated Time

90-120 minutes

[S3DFX-CYBER]

/tenet fix

[github-actions]

🤖 TENET Agent is on it! Analyzing the issue and generating a fix...

Triggered by @S3DFX-CYBER

[github-actions]

🤖 TENET Agent - Fix Generation Failed

TENET Agent was unable to parse a valid code fix from the LLM response for issue #43. The LLM may have misunderstood the request.

Analysis:

Part 1: Analysis

1. Root Cause / Requirement

Root Cause: Services within the TENET AI project currently rely on direct os.getenv() calls for configuration, leading to two main issues:

1. Cryptic Errors: When required environment variables are missing, services fail with generic Python errors (e.g., TypeError if None is used where a string is expected, or ValueError during type conversion), making debugging difficult.
2. Lack of Fail-Fast: Services might start up partially before encountering a missing variable, delaying the detection of configuration issues.

Requirement: Implement a centralized, reusable utility function validate_env_vars in services/utils/env_validator.py. This function must:

• Verify the presence of a list of specified environment variables.
• Support optional type conversion (e.g., to int, float).
• Aggregate all missing or invalid variables and raise a single, clear error message detailing the issues.
• Return a dictionary of the validated environment variable names and their converted values.
• Be integrated into services/ingest/app.py and services/analyzer/app.py to ensure early and clear validation at service startup.

2. Files to Modify

• services/ingest/app.py: To integrate the new env_validator utility at service startup, replacing existing os.getenv calls and any manual validation. (This file was not provided, but is explicitly mentioned in the tasks).
• services/analyzer/app.py: To integrate the new env_validator utility at service startup, replacing existing os.getenv calls, type conversions, and the explicit API_KEY check.
• tests/unit/test_ingest.py: Potentially, if the startup logic of services/ingest/app.py changes such that the TestClient initialization fails due to missing environment variables in the test context. This might require mocking environment variables for tests.

3. Files to Create

• services/utils/env_validator.py: This will contain the validate_env_vars function and any related helper functions or custom exceptions.
• tests/unit/test_env_validator.py: This will contain unit tests to ensure the validate_env_vars function works correctly for various scenarios (missing variables, invalid types, successful validation).

4. Approach

I will create services/utils/env_validator.py with a validate_env_vars function that accepts a list of dictionaries, where each dictionary defines an environment variable's name, its required status, and an optional type converter (e.g., int, float). The function will iterate through this list, checking os.getenv() for each variable. If a required variable is missing or fails type conversion, it will collect the error. After checking all variables, if any errors were collected, it will raise a custom exception (e.g., EnvironmentValidationError) listing all issues. Otherwise, it will return a dictionary of the validated variables and their converted values. Unit tests will cover successful validation, missing required variables, and invalid formats. Finally, I will update services/ingest/app.py and services/analyzer/app.py to call this utility at startup, using the returned dictionary to configure their respective services.

5. Risks

• Breaking Service Startup: Introducing a fail-fast validation mechanism means services will now immediately halt if required environment variables are missing or malformed. This is the desired behavior but requires ensuring all deployment and development environments have their .env files or system environment variables correctly configured.
• Test Environment Impact: Existing unit tests for services/ingest/app.py and services/analyzer/app.py (e.g., those using TestClient) might fail if they don't explicitly set or mock the environment variables now required by the new validation logic. Adjustments to test setups might be necessary.
• Type Conversion Edge Cases: The type conversion logic within validate_env_vars needs to be robust to handle various input formats (e.g., non-numeric strings for integer ports).
• Backward Compatibility: While the goal is to improve, ensure that the new validation doesn't inadvertently break any existing, albeit "cryptic," startup paths that might have been implicitly relied upon in some non-standard configurations.

---

TENET Agent 🛡️

[S3DFX-CYBER]

/tenet fix

[github-actions]

🤖 TENET Agent is on it! Analyzing the issue and generating a fix...

Triggered by @S3DFX-CYBER

[github-actions]

🤖 TENET Agent - Fix Generation Failed

TENET Agent was unable to parse a valid code fix from the LLM response for issue #43. The LLM may have misunderstood the request.

Analysis:

TENET Agent Analysis - Issue #43

1. Root Cause / Requirement

The current TENET AI services, such as the Analyzer and Ingest services, retrieve environment variables directly using os.getenv() and perform manual, often ad-hoc, validation checks. This leads to inconsistent error handling, cryptic error messages when configurations are missing or malformed, and a lack of a centralized "fail-fast" mechanism. The core requirement is to establish a robust, reusable utility for validating essential environment variables at service startup, ensuring clear error reporting and immediate failure if critical configurations are absent or invalid.

2. Files to Modify

• services/ingest/app.py: To integrate the new environment variable validation utility at service startup.
• services/analyzer/app.py: To integrate the new environment variable validation utility at service startup, replacing existing manual checks for API_KEY, REDIS_HOST, REDIS_PORT, etc. (This is an extension beyond the explicit task list but aligns with the issue's spirit and the provided relevant source file).

3. Files to Create

• services/utils/env_validator.py: This will contain the validate_env_vars utility function.
• tests/unit/test_env_validator.py: This will house the unit tests for the env_validator module.

4. Approach

I will create services/utils/env_validator.py with a function validate_env_vars. This function will accept a list of required environment variable names and an optional dictionary for type conversion/validation rules (e.g., {"PORT": int}). It will iterate through the required variables, checking their presence using os.getenv(). If a variable is missing, it will be added to a list of missing variables. If type conversion/validation is specified, it will attempt to convert the value and raise an error if it fails (e.g., non-integer port). If any required variables are missing or invalid, a ValueError will be raised, listing all issues clearly. Otherwise, it will return a dictionary of validated and converted environment variables. I will then update services/ingest/app.py and services/analyzer/app.py to call this utility at their respective startup events, replacing their current os.getenv calls and manual validation with the new centralized mechanism. Comprehensive unit tests will be developed for env_validator.py to ensure high coverage and reliability.

5. Risks

• Service Startup Failure: The primary risk is that services will now explicitly fail to start if required environment variables are missing or malformed. While this is the intended "fail-fast" behavior, it could disrupt existing deployments that might have implicitly relied on default values or partial configurations. Clear documentation and communication of required environment variables (e.g., updating .env.template) will mitigate this.
• Over-validation/Complexity: The optional format validation needs to be carefully designed to avoid being overly complex or restrictive. It should focus on common types (integers for ports, booleans, etc.) without introducing excessive parsing logic.
• Backward Compatibility: The new utility should not introduce new external dependencies that are not already managed by requirements.txt.
• Refactoring Impact: Modifying services/ingest/app.py and services/analyzer/app.py requires careful refactoring to ensure existing functionality is not broken and all environment variables are correctly migrated to the new validation scheme.

---

TENET Agent 🛡️

[S3DFX-CYBER]

/tenet fix

[github-actions]

🤖 TENET Agent is on it! Analyzing the issue and generating a fix...

Triggered by @S3DFX-CYBER

[github-actions]

🤖 TENET Agent - PR Creation Failed

Branch tenet/fix-issue-43 was pushed, but opening the PR failed:

GitHub Actions is not permitted to create or approve pull requests.: 403 {"message": "GitHub Actions is not permitted to create or approve pull requests.", "documentation_url": "https://docs.github.com/rest/pulls/pulls#create-a-pull-request", "status": "403"}

Please open a PR from that branch manually.

---

TENET Agent 🛡️

[S3DFX-CYBER]

/tenet fix