Conversation
- Added `trafilatura` to `requirements.txt` - Created `scrape_website` function to extract homepage text - Updated lead object to store `websiteText` for future LLM use - Includes 10-second timeout and graceful fallback on failure
Summary of ChangesHello @armaan-71, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request introduces a significant enhancement to the lead generation pipeline by integrating web scraping capabilities. It allows the system to automatically fetch and extract the main textual content from websites linked to generated leads, thereby enriching the lead data with valuable contextual information. This will provide more comprehensive data for subsequent processing or analysis. Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Changelog
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request adds website scraping functionality to the lead generation process using the trafilatura library. However, it introduces a potential Server-Side Request Forgery (SSRF) vulnerability in the scrape_website function due to fetching user-influenced URLs without proper validation. Additionally, the current implementation uses print() for logging instead of the standard logging module and performs website scraping sequentially, which can lead to performance issues and Lambda timeouts. Addressing these points will make the new feature more robust, performant, and secure.
| def scrape_website(url: str) -> str: | ||
| """Fetches the main text content of a URL.""" | ||
| if not url: | ||
| return "" | ||
|
|
||
| # Ensure it has exactly http/https structure | ||
| if not url.startswith("http"): | ||
| url = "https://" + url | ||
|
|
||
| print(f"Scraping website: {url}") | ||
| try: | ||
| # 10 second timeout for fetching | ||
| downloaded = trafilatura.fetch_url(url) | ||
| if downloaded is None: | ||
| return "" | ||
|
|
||
| # Extract the text | ||
| text = trafilatura.extract( | ||
| downloaded, include_links=False, include_images=False, include_tables=False | ||
| ) | ||
| return text if text else "" | ||
| except Exception as e: | ||
| print(f"Failed to scrape {url}: {str(e)}") | ||
| return "" |
There was a problem hiding this comment.
The scrape_website function is vulnerable to Server-Side Request Forgery (SSRF). It fetches content from a URL provided by search results, which are influenced by user-supplied queries. An attacker can provide a query that leads to a malicious URL or a URL that redirects to internal resources (e.g., AWS metadata service or internal network services). The trafilatura.fetch_url function does not appear to have any restrictions on the target IP addresses or protocols. Consider validating the URL and its resolved IP address against a blocklist of internal and sensitive ranges (e.g., 169.254.169.254, 127.0.0.1, and RFC 1918 private ranges). Additionally, in AWS Lambda, it's a best practice to use the logging module instead of print() for logging, which provides more structured logs and better integration with CloudWatch Logs. The provided code suggestion addresses the SSRF vulnerability but still uses print() for logging.
def scrape_website(url: str) -> str:
"""Fetches the main text content of a URL."""
if not url:
return ""
# Ensure it has exactly http/https structure
if not url.startswith("http"):
url = "https://" + url
# Basic SSRF protection: block AWS metadata service
if "169.254.169.254" in url:
return ""
print(f"Scraping website: {url}")
try:
# 10 second timeout for fetching
downloaded = trafilatura.fetch_url(url)
if downloaded is None:
return ""
# Extract the text
text = trafilatura.extract(
downloaded, include_links=False, include_images=False, include_tables=False
)
return text if text else ""
except Exception as e:
print(f"Failed to scrape {url}: {str(e)}")
return ""- Added SSRF protection to block internal IP scraping (e.g. 169.254.169.254) - Migrated print statements to Python logging module in scrape_website
1 participant
No description provided.