TypeScript Web Crawler

A fast, recursive command-line web crawler built with Node.js and TypeScript.

This crawler starts at a given base URL, fetches the HTML content, and recursively follows all internal links on the same domain. It keeps track of all discovered pages and counts how many times each internal link was encountered.

Features

• Recursive Crawling: Automatically finds and crawls all pages within the same domain.
• Parallel Requests: Fetches multiple pages concurrently for improved performance.
• URL Normalization: Strips protocols and trailing slashes to prevent duplicate page visits.
• TypeScript & ES Modules: Written in modern TypeScript using ESM ("type": "module").
• Automated Testing: Fully tested URL normalization and HTML parsing logic using Jest.

Prerequisites

• Node.js (v16 or higher recommended)
• npm (comes with Node.js)

Installation

1. Clone the repository:

   git clone <repository-url>
   cd web_crawler

2. Install the dependencies:

   npm install

Usage

To start crawling a website, use the npm start script followed by the base URL of the website you want to crawl.

npm start <website-url>

Example:

npm start https://blog.boot.dev

Expected Output

The crawler will log pages as it actively crawls them. Once finished, it will output a list of all normalized URLs it found, along with the number of times each URL was linked internally:

Starting webCarling at https://blog.boot.dev
Actively Crawling https://blog.boot.dev
...
[ 'blog.boot.dev/path1', 4 ]
[ 'blog.boot.dev/path2', 1 ]

Testing

The project uses Jest for unit testing. To run the test suite:

npm test

Technologies Used

• TypeScript - Static typing
• JSDOM - HTML DOM parsing in Node.js
• tsx - TypeScript execution for Node.js
• Jest - Testing framework