fix: validate header names/values and prevent CRLF injection (#1817)

[SaumyaT-21]

Description

Validates HTTP header names and values in crawler.py to ensure they conform to HTTP specs.
Specifically:

• Enforces header name and value validation against expected HTTP token grammar.
• Checks for and rejects \r (Carriage Return) and \n (Line Feed) characters to prevent header injection (CRLF) vulnerabilities.
• Raises a ValueError with clear messaging when an invalid header format or illegal character is detected.

Related Issues

Closes #1817

Type of Change

• [x] Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation update

How Has This Been Tested?

Tested manually via a local test block in crawler.py using Python in the project's virtual environment:

1. Valid Headers Test: Verified compliant header keys and values execute cleanly without errors.
2. CRLF Injection Tests: Verified that passing \r or \n characters in either the header name or the header value correctly raises a ValueError.

Checklist

• [x] My code follows the code style of this project.
• [x] I have performed a self-review of my own code.
• I have commented my code, particularly in hard-to-understand areas.
• I have made corresponding changes to the documentation.
• [x] My changes generate no new warnings.

[SaumyaT-21]

Hey @utksh1 ! Added the fix for CRLF injection and HTTP token validation. All green on CI checks. Please go ahead and review the changes!

[utksh1]

Request changes: _HEADER_NAME_re.match() is not anchored, so a value such as X-Test@invalid can pass by matching only the valid prefix. Use a full match/anchors and add tests for trailing invalid characters. Also remove the unrelated frontend/package-lock.json update and add coverage for invalid field values through the actual header-building path.