Skip to content

Update crawler.ts: Add hostname check to keep crawler on the same domain. #15

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Update crawler.ts: Add hostname check to keep crawler on the same domain. #15

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
aab60d8
Update crawler.ts
dougwithseismic Sep 15, 2023
1b7e03f
Remove Linting Changes
dougwithseismic Sep 25, 2023
e934889
Removed additional Code
dougwithseismic Sep 25, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 41 additions & 7 deletions src/app/api/crawl/crawler.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,9 +11,17 @@ class Crawler {
private pages: Page[] = [];
private queue: { url: string; depth: number }[] = [];

constructor(private maxDepth = 2, private maxPages = 1) { }
constructor(private maxDepth = 2, private maxPages = 1) {}

async crawl(startUrl: string): Promise<Page[]> {
// Capture the hostname of the start URL
let startHostname;
try {
startHostname = new URL(startUrl).hostname;
} catch (error) {
throw new Error(`Invalid URL: ${startUrl}`);
}

// Add the start URL to the queue
this.addToQueue(startUrl);

Expand All @@ -34,8 +42,8 @@ class Crawler {
// Parse the HTML and add the page to the list of crawled pages
this.pages.push({ url, content: this.parseHtml(html) });

// Extract new URLs from the page HTML and add them to the queue
this.addNewUrlsToQueue(this.extractUrls(html, url), depth);
// Pass startHostname to addNewUrlsToQueue
this.addNewUrlsToQueue(this.extractUrls(html, url), depth, startHostname);
}

// Return the list of crawled pages
Expand All @@ -58,8 +66,22 @@ class Crawler {
this.queue.push({ url, depth });
}

private addNewUrlsToQueue(urls: string[], depth: number) {
this.queue.push(...urls.map(url => ({ url, depth: depth + 1 })));
private addNewUrlsToQueue(
urls: string[],
depth: number,
startHostname: string
) {
const filteredUrls = urls.filter((url) => {
try {
const hostname = new URL(url).hostname;
return hostname === startHostname;
} catch (e) {
console.error(`Invalid URL: ${url}`);
return false;
}
});

this.queue.push(...filteredUrls.map((url) => ({ url, depth: depth + 1 })));
}

private async fetchPage(url: string): Promise<string> {
Expand All @@ -80,8 +102,20 @@ class Crawler {

private extractUrls(html: string, baseUrl: string): string[] {
const $ = cheerio.load(html);
Copy link

@athrael-soju athrael-soju Oct 9, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor observation: This will give a 'deprecated' warning. You can do import { load } from 'cheerio'; and use the load by itself instead.

const relativeUrls = $('a').map((_, link) => $(link).attr('href')).get() as string[];
return relativeUrls.map(relativeUrl => new URL(relativeUrl, baseUrl).href);
const relativeUrls = $('a')
.map((_, link) => $(link).attr('href'))
.get() as string[];

try {
const relativeMap = relativeUrls.map(
(relativeUrl) => new URL(relativeUrl, baseUrl).href
);

return relativeMap;
} catch (error) {
console.error('Error extracting URLs:', error);
return [];
}
}
}

Expand Down