Firecrawl Automation Testing Framework

A comprehensive automation testing framework that uses Firecrawl to crawl web applications and automatically generates Robot Framework test cases for UI elements. Designed specifically for VPN-protected applications like MDR Nexus with authentication management.

🎯 Overview

This framework automatically:

1. Crawls web applications using Firecrawl API with VPN and authentication support
2. Detects interactive elements (buttons, forms, links) with hover testing
3. Generates Robot Framework test cases (.robot files) automatically
4. Creates Python keywords for reusable test components
5. Manages locators in centralized locators.py files
6. Executes comprehensive test suites with detailed reporting

🏗️ Architecture

📁 Project Structure
├── src/                          # Core application modules
│   ├── session/                  # Session & authentication management
│   │   ├── session_manager.py    # Browser session handling
│   │   ├── auth_detector.py      # Authentication detection
│   │   ├── cookie_manager.py     # Session persistence  
│   │   └── session_validator.py  # Session health monitoring
│   ├── firecrawl/               # Firecrawl API integration
│   │   ├── api_client.py        # Core API client
│   │   ├── authenticated_crawler.py # VPN-aware crawler
│   │   ├── config.py            # Configuration management
│   │   └── exceptions.py        # Custom error handling
│   ├── element_analyzer/        # Element detection & classification
│   ├── test_generator/          # Test case generation
│   ├── locator_manager/         # Locator management  
│   └── test_executor/           # Test execution & monitoring
├── tests/                       # Generated test files
│   ├── generated/               # Auto-generated .robot files
│   ├── keywords/                # Python-based keywords (.py)
│   └── resources/               # Test resources & locators
├── config/                      # Configuration files
├── docs/                        # Documentation & flowcharts
├── reports/                     # Test execution reports
├── logs/                        # Application logs
└── sessions/                    # Session data storage

🚀 Quick Start

1. Installation

# Install dependencies
pip install -r requirements.txt

# Install the framework
pip install -e .

2. Configuration

Set Environment Variables:

export FIRECRAWL_API_KEY="your_firecrawl_api_key_here"
export MDR_USERNAME="your_mdr_username"  
export MDR_PASSWORD="your_mdr_password"

Configure Target Application (config/app_config.yaml):

target_application:
  base_url: "https://mdr.nexus.armorlabs.co/"
  requires_vpn: true
  vpn_check_url: "https://mdr.nexus.armorlabs.co/health"

authentication:
  enabled: true
  method: "form_based"
  form_auth:
    login_url: "https://mdr.nexus.armorlabs.co/login"
    username_field: "#username"
    password_field: "#password"
    submit_button: "button[type='submit']"

firecrawl:
  api_key: "${FIRECRAWL_API_KEY}"
  base_url: "https://api.firecrawl.dev"
  timeout: 60
  max_concurrent_requests: 5

3. Basic Usage

Simple Crawling:

from src.session import SessionManager
from src.firecrawl import AuthenticatedCrawler

# Initialize with configuration
config = load_config('config/app_config.yaml')

with SessionManager(config) as session_manager:
    with AuthenticatedCrawler(config, session_manager) as crawler:
        
        # Crawl protected application
        result = crawler.crawl_authenticated_page(
            "https://mdr.nexus.armorlabs.co/dashboard"
        )
        
        print(f"Crawl successful: {result['success']}")

Batch Crawling:

urls = [
    "https://mdr.nexus.armorlabs.co/dashboard",  
    "https://mdr.nexus.armorlabs.co/reports",
    "https://mdr.nexus.armorlabs.co/settings"
]

results = crawler.crawl_authenticated_urls(urls)
print(f"Crawled {len(results)} URLs successfully")

🔧 Configuration Guide

VPN Configuration

For VPN-protected applications, ensure:

1. VPN Connection: Connect to your VPN before running
2. VPN Check URL: Set vpn_check_url in configuration
3. Network Settings: Configure SSL verification and proxy settings

target_application:
  requires_vpn: true
  vpn_check_url: "https://mdr.nexus.armorlabs.co/health"
  
network:
  verify_ssl: true
  vpn_networks:
    - "10.0.0.0/8"
    - "172.16.0.0/12"
    - "192.168.0.0/16"

Authentication Configuration

Configure authentication for protected applications:

authentication:
  enabled: true
  method: "form_based"
  session_persistence: true
  max_session_duration: 3600
  
  form_auth:
    login_url: "https://your-app.com/login"
    username_field: "#username"
    password_field: "#password"
    submit_button: "input[type='submit']"
    
    indicators:
      success_selectors:
        - ".dashboard"
        - ".user-profile"
      failure_selectors:
        - ".error-message"
        - ".alert-danger"

Firecrawl Configuration

Configure Firecrawl API settings:

firecrawl:
  api_key: "${FIRECRAWL_API_KEY}"
  base_url: "https://api.firecrawl.dev"
  timeout: 60
  max_retries: 3
  rate_limit_per_minute: 100
  max_concurrent_requests: 5
  
  crawl_options:
    include_html: true
    include_raw_html: true
    include_metadata: true
    include_links: true
    wait_for: 2000
    screenshot: false
    mobile: false

🤖 Generated Test Files

The framework generates several types of files:

1. Robot Framework Test Cases (.robot files)

*** Settings ***
Documentation    Auto-generated login functionality tests
Library          SeleniumLibrary
Resource         ../resources/common_keywords.robot
Variables        ../resources/locators.py

*** Test Cases ***
Verify Login Button Functionality
    [Documentation]    Test login button click behavior
    [Tags]    login    button    automated    firecrawl
    Click Element With Validation    ${LOGIN_BUTTON}
    Wait For Page Load
    Verify Login Form Visible

2. Python Keywords (.py files)

from robot.api.deco import keyword
from selenium import webdriver

@keyword("Click Element With Validation")
def click_element_with_validation(locator):
    """Custom keyword for clicking with validation"""
    # Scroll element into view
    # Verify element is clickable
    # Click with error handling
    pass

@keyword("Verify Authentication State")  
def verify_authentication_state():
    """Verify user is properly authenticated"""
    pass

3. Locators File (locators.py)

# Auto-generated locators for MDR Nexus Application
# Generated on: 2024-01-15 10:30:00

# Login Page Elements
LOGIN_BUTTON = "//button[@type='submit' and contains(text(), 'Login')]"
USERNAME_FIELD = "//input[@id='username']"
PASSWORD_FIELD = "//input[@type='password']"

# Dashboard Elements  
DASHBOARD_MENU = "//nav[@class='dashboard-menu']"
USER_PROFILE = "//div[@class='user-profile']"
LOGOUT_BUTTON = "//a[contains(text(), 'Logout')]"

# Reports Section
REPORTS_TABLE = "//table[@id='reports-table']"
EXPORT_BUTTON = "//button[contains(text(), 'Export')]"

🔍 Workflow Process

The framework follows this automated workflow:

graph TD
    A[Initialize Session] --> B[Verify VPN Connection]
    B --> C[Perform Authentication]
    C --> D[Crawl Pages with Firecrawl]
    D --> E[Analyze Elements with Hover Testing]
    E --> F[Generate XPath Locators]
    F --> G[Create Robot Framework Tests]
    G --> H[Generate Python Keywords]
    H --> I[Execute Test Suite]
    I --> J[Generate Reports]

Loading

Detailed Steps:

1. Session Initialization: Start browser session with VPN verification
2. Authentication: Detect and handle login forms automatically
3. Page Crawling: Use Firecrawl API to extract page structure and content
4. Element Analysis: Identify interactive elements with hover testing
5. Test Generation: Create Robot Framework test cases for each element
6. Keyword Creation: Generate reusable Python keywords
7. Locator Management: Centralize all element selectors
8. Test Execution: Run generated test suite with reporting

📊 Execution & Reporting

Running Tests

# Run all generated tests
robot tests/generated/

# Run specific test suite
robot tests/generated/login_tests.robot

# Run with specific tags
robot --include automated tests/generated/

# Generate detailed reports
robot --outputdir reports tests/generated/

Monitoring & Statistics

# Get crawling statistics
crawler_stats = crawler.get_session_stats()
print(f"URLs crawled: {crawler_stats['urls_crawled']}")
print(f"Success rate: {crawler_stats['success_rate']}%")

# Get API client statistics  
api_stats = crawler.firecrawl_client.get_stats()
print(f"Requests made: {api_stats['requests_made']}")
print(f"Rate limited: {api_stats['rate_limited_count']} times")

🚨 Troubleshooting

Common Issues & Solutions

1. VPN Connection Issues

Error: VPN connectivity verification failed

Solution:

• Ensure VPN is connected and active
• Verify VPN check URL is accessible
• Check network configuration settings

2. Authentication Failures

Error: Authentication failed with provided credentials  

Solution:

• Verify credentials in environment variables
• Check authentication configuration selectors
• Ensure login form is properly detected

3. Firecrawl API Issues

Error: Rate limit exceeded

Solution:

• Check API key validity
• Reduce max_concurrent_requests in config
• Increase rate_limit_per_minute if you have higher limits

4. Element Detection Issues

Error: No interactive elements found

Solution:

• Ensure page is fully loaded before analysis
• Check wait_for setting in crawl options
• Verify JavaScript is enabled

Debug Mode

Enable debug logging for troubleshooting:

import logging
logging.basicConfig(level=logging.DEBUG)

# Enable detailed Firecrawl logging
from src.firecrawl import setup_logging
setup_logging('DEBUG')

🔒 Security & Best Practices

Environment Variables

Store sensitive data in environment variables:

# Required
export FIRECRAWL_API_KEY="fc_your_api_key_here"
export MDR_USERNAME="your_username"
export MDR_PASSWORD="your_secure_password"

# Optional
export VPN_USERNAME="vpn_user"
export VPN_PASSWORD="vpn_pass"

Configuration Security

• Never commit credentials to version control
• Use .env files for local development (git-ignored)
• Use secure credential storage in production
• Rotate API keys regularly

Session Security

• Sessions are encrypted using Fernet encryption
• Session files have restrictive permissions (600)
• Automatic session timeout and cleanup
• VPN verification before each operation

📈 Performance Optimization

Rate Limiting

firecrawl:
  rate_limit_per_minute: 100    # Adjust based on your plan
  max_concurrent_requests: 5    # Balance speed vs. stability
  timeout: 60                   # Adjust for slow pages

Concurrent Processing

# Adjust concurrency for your system
crawler.crawl_authenticated_urls(
    urls, 
    max_workers=3  # Lower for VPN/auth scenarios
)

Caching & Session Persistence

authentication:
  session_persistence: true
  max_session_duration: 3600  # 1 hour cache

🤝 Contributing

1. Follow Existing Patterns: Use comprehensive documentation and error handling
2. Add Tests: Include unit tests for new functionality
3. Update Documentation: Keep README and docstrings current
4. Security First: Never commit credentials or sensitive data

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🆘 Support

For support and questions:

• Check the troubleshooting section above
• Review configuration examples in /config/
• Check logs in /logs/ directory
• Ensure all prerequisites are met (VPN, authentication, API key)

---

🎉 Happy Testing with Firecrawl Automation Framework!