Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,9 @@ decodo scrape https://ip.decodo.com
decodo https://ip.decodo.com
decodo search "decodo scraping api"
decodo search "shoes" --engine bing --geo us --limit 2
decodo screenshot https://ip.decodo.com -o shot.png
decodo screenshot https://ip.decodo.com -o ./shots
decodo screenshot https://ip.decodo.com > shot.png
decodo universal --help
decodo universal https://ip.decodo.com
decodo google-search "decodo scraping api"
Expand Down
5 changes: 3 additions & 2 deletions src/cli/register.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3,12 +3,13 @@ import { resetCommand } from "../auth/commands/reset.js";
import { setupCommand } from "../auth/commands/setup.js";
import { whoamiCommand } from "../auth/commands/whoami.js";
import { createScrapeCommands } from "../scrape/register.js";
import { sortCommandsByOrder } from "./services/sort-commands-by-order.js";

export async function createCommands(): Promise<Command[]> {
return [
return sortCommandsByOrder([
setupCommand,
resetCommand,
whoamiCommand,
...(await createScrapeCommands()),
];
]);
}
30 changes: 30 additions & 0 deletions src/cli/services/sort-commands-by-order.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
import type { Command } from "commander";

export const ROOT_COMMAND_ORDER = [

@aurimas-angladagis-dcd aurimas-angladagis-dcd Jun 4, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure if possible, but maybe we can create groups, like:

User:

  • setup;
  • reset;
  • etc;

Scrape:

  • scrape;
  • search;
  • screenshot
  • etc;

and when you will use --help it shows as grouped commands.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code is more about sorting commands which favor top level over target specific ones. Grouping is possible and can be done but I think it's product desicion, separate task

"scrape",
"search",
"screenshot",
"setup",
"reset",
"whoami",
"targets",
] as const;

export function sortCommandsByOrder(
commands: Command[],
order: readonly string[] = ROOT_COMMAND_ORDER
): Command[] {
const orderIndex = new Map(order.map((name, index) => [name, index]));
const fallbackIndex = order.length;

return [...commands].sort((a, b) => {
const aIndex = orderIndex.get(a.name()) ?? fallbackIndex;
const bIndex = orderIndex.get(b.name()) ?? fallbackIndex;

if (aIndex !== bIndex) {
return aIndex - bIndex;
}

return a.name().localeCompare(b.name());
Comment on lines +17 to +28

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm having a hard time understanding reasoning here, but as I've understand the idea is to have ROOT_COMMAND_ORDER as first commands, then everything else?

@paulius-krutkis-dcd paulius-krutkis-dcd Jun 4, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, you go it right. Not sure how to better write this, since there is no option for sort order in commander. Maybe better would be to have an export SORT_ORDER = ... for each command but that still requires a registry to map all the commands so it's kind of the same as it is right now

});
}
17 changes: 17 additions & 0 deletions src/platform/services/resolve-output-file.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
import { statSync } from "node:fs";
import { join } from "node:path";

export function resolveOutputFilePath(
outputPath: string,
defaultFileName: string
): string {
try {
if (statSync(outputPath).isDirectory()) {
return join(outputPath, defaultFileName);
}
} catch {
// Path missing or inaccessible — treat as a file path.
}

return outputPath;
}
29 changes: 29 additions & 0 deletions src/platform/services/write-binary.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
import { writeFileSync } from "node:fs";
import { resolve } from "node:path";
import { EXIT } from "../constants.js";
import { resolveOutputFilePath } from "./resolve-output-file.js";

export const BINARY_TTY_ERROR =
"Refusing to write binary data to a TTY. Use -o <file> or pipe to a file.";

export function writeBinaryOutput(
bytes: Buffer,
options: { output?: string; defaultFileName?: string }
): void {
if (options.output !== undefined) {
const filePath =
options.defaultFileName === undefined
? options.output
: resolveOutputFilePath(options.output, options.defaultFileName);
writeFileSync(filePath, bytes);
console.error(`Wrote ${resolve(filePath)}`);
return;
}

if (process.stdout.isTTY) {
console.error(`Error: ${BINARY_TTY_ERROR}`);
process.exit(EXIT.USAGE);
}

process.stdout.write(bytes);
}
2 changes: 1 addition & 1 deletion src/scrape/commands/codegen-target-commands.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ import type { DecodoSchema } from "@decodo/sdk-ts";
import { Command } from "commander";
import { configureTargetCommand } from "../services/command-builder.js";
import { snakeToKebab } from "../services/naming.js";
import { createTargetAction } from "./run-target-scrape.js";
import { createTargetAction } from "../services/run-target-scrape.js";

export function createCodegenTargetCommands(schema: DecodoSchema): Command[] {
const commands: Command[] = [];
Expand Down
2 changes: 1 addition & 1 deletion src/scrape/commands/scrape.ts
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
import { type DecodoSchema, Target, ValidationError } from "@decodo/sdk-ts";
import { Command } from "commander";
import { resolveTarget } from "../services/resolve-target.js";
import { createTargetAction } from "../services/run-target-scrape.js";
import type { ScrapeOptions } from "../types/scrape-command.js";
import { createTargetAction } from "./run-target-scrape.js";

function parseHeadersJson(json: string): Record<string, unknown> {
try {
Expand Down
53 changes: 53 additions & 0 deletions src/scrape/commands/screenshot.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
import { type DecodoSchema, Target } from "@decodo/sdk-ts";
import { Command } from "commander";
import { writeBinaryOutput } from "../../platform/services/write-binary.js";
import { extractPngFromResponse } from "../services/extract-png.js";
import { resolveTarget } from "../services/resolve-target.js";
import { createTargetAction } from "../services/run-target-scrape.js";
import { defaultScreenshotFilename } from "../services/screenshot-output-filename.js";
import type { ScreenshotOptions } from "../types/screenshot-command.js";

export function createScreenshotCommand(schema: DecodoSchema): Command {
return new Command("screenshot")
.description(
"Capture a PNG screenshot (universal, headless). Use decodo universal --headless png for full options."
)
.argument("<url>", "URL to screenshot")
.option("--country <code>", "Geo / country code (maps to geo)")
.option(
"-o, --output <path>",
"Write PNG to file or directory (default name: <host>.png)"
)
.option("--target <name>", "Scrape target override (default: universal)")
.action(
createTargetAction(
Target.Universal,
schema,
(url, options) => {
if (url === undefined) {
throw new Error("Missing required URL.");
}

const opts = options as ScreenshotOptions;
const body: Record<string, unknown> = {
target: resolveTarget(opts.target, schema, Target.Universal),
url,
headless: "png",
};

if (opts.country !== undefined) {
body.geo = opts.country;
}

return body;
},
(response, options, url) => {
const opts = options as ScreenshotOptions;
writeBinaryOutput(extractPngFromResponse(response), {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about filename? Maybe we should have option for this too.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

-o, --output already accepts a full file path (see line 16). If filename only variant is needed it can be added but I think the full path covers that use case

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what about if path is a directory?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good point, with previous changes it just throws which is bad ux, if folder is specified then by default it will save to a predefined filepath, updated pr description on how it works with new changes

output: opts.output,
defaultFileName: defaultScreenshotFilename(url),
});
}
)
);
}
2 changes: 1 addition & 1 deletion src/scrape/commands/search.ts
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
import { type DecodoSchema, Target, ValidationError } from "@decodo/sdk-ts";
import { Command, Option } from "commander";
import { resolveTarget } from "../services/resolve-target.js";
import { createTargetAction } from "../services/run-target-scrape.js";
import type { SearchOptions } from "../types/search-command.js";
import { createTargetAction } from "./run-target-scrape.js";

const ENGINE_TARGETS = {
google: Target.GoogleSearch,
Expand Down
2 changes: 2 additions & 0 deletions src/scrape/register.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ import type { Command } from "commander";
import { createCodegenTargetCommands } from "./commands/codegen-target-commands.js";
import { createListTargetsCommand } from "./commands/list-targets.js";
import { createScrapeCommand } from "./commands/scrape.js";
import { createScreenshotCommand } from "./commands/screenshot.js";
import { createSearchCommand } from "./commands/search.js";
import { loadSchema } from "./services/schema-loader.js";

Expand All @@ -11,6 +12,7 @@ export async function createScrapeCommands(): Promise<Command[]> {
return [
createScrapeCommand(schema),
createSearchCommand(schema),
createScreenshotCommand(schema),
createListTargetsCommand(schema),
...createCodegenTargetCommands(schema),
];
Expand Down
31 changes: 31 additions & 0 deletions src/scrape/services/extract-png.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
import { type SyncResponse, ValidationError } from "@decodo/sdk-ts";

const PNG_SIGNATURE = Buffer.from([
0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a,
]);

function isPngBuffer(buffer: Buffer): boolean {
return (
buffer.length >= PNG_SIGNATURE.length &&
buffer.subarray(0, PNG_SIGNATURE.length).equals(PNG_SIGNATURE)
);
}

export function extractPngFromResponse(response: SyncResponse): Buffer {
const entry = response.results[0];

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if result will be empty.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is handled in this check:

 if (entry === undefined) {
    throw new ValidationError("Screenshot response has no results.");
  }

if (entry === undefined) {
throw new ValidationError("Screenshot response has no results.");
}

const { content } = entry;
if (typeof content !== "string" || content.length === 0) {
throw new ValidationError("Screenshot response has no PNG content.");
}

const bytes = Buffer.from(content, "base64");
if (bytes.length === 0 || !isPngBuffer(bytes)) {
throw new ValidationError("Screenshot response content is not valid PNG.");
}

return bytes;
}
Original file line number Diff line number Diff line change
Expand Up @@ -10,18 +10,14 @@ import { AuthRequiredError } from "../../auth/errors/auth-required-error.js";
import { getRootOpts } from "../../auth/services/global-opts.js";
import { requireAuthToken } from "../../auth/services/resolve-token.js";
import { EXIT } from "../../platform/constants.js";
import { createDecodoClient } from "../services/client.js";
import {
buildScrapeBody,
getTargetCommandConfig,
} from "../services/command-builder.js";

export type ScrapeBodyBuilder = (
input: string | undefined,
options: Record<string, unknown>
) => Record<string, unknown>;
import type {
ScrapeBodyBuilder,
ScrapeResponseHandler,
} from "../types/run-target-scrape.js";
import { createDecodoClient } from "./client.js";
import { buildScrapeBody, getTargetCommandConfig } from "./command-builder.js";

function handleScrapeError(err: unknown): never {
export function handleScrapeError(err: unknown): never {
if (err instanceof Error && err.message.startsWith("process.exit:")) {
throw err;
}
Expand All @@ -47,23 +43,31 @@ function handleScrapeError(err: unknown): never {
process.exit(EXIT.ERROR);
}

async function executeScrape(
export async function executeScrape(
token: string,
schema: DecodoSchema,
body: Record<string, unknown>
body: Record<string, unknown>,
options: Record<string, unknown>,
onResponse?: ScrapeResponseHandler,
input?: string
): Promise<void> {
const client = createDecodoClient(token, schema);
const response = await client.webScrapingApi.scrape(
body as unknown as ScrapeRequest
);

console.log(JSON.stringify(response, null, 2));
if (onResponse) {
await onResponse(response, options, input);
} else {
console.log(JSON.stringify(response, null, 2));
}
}

export function createTargetAction(
target: string,
schema: DecodoSchema,
buildBody?: ScrapeBodyBuilder
buildBody?: ScrapeBodyBuilder,
onResponse?: ScrapeResponseHandler
) {
const config = getTargetCommandConfig(target, schema);
const resolveBody =
Expand All @@ -80,7 +84,7 @@ export function createTargetAction(
try {
const token = await requireAuthToken({ token: rootOpts.token });
const body = resolveBody(input, options);
await executeScrape(token, schema, body);
await executeScrape(token, schema, body, options, onResponse, input);
} catch (err) {
if (err instanceof AuthRequiredError) {
console.error(err.message);
Expand Down
19 changes: 19 additions & 0 deletions src/scrape/services/screenshot-output-filename.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
const DEFAULT_SCREENSHOT_FILENAME = "screenshot.png";

export function defaultScreenshotFilename(url: string | undefined): string {
if (url === undefined || url.length === 0) {
return DEFAULT_SCREENSHOT_FILENAME;
}

try {
const hostname = new URL(url).hostname;
const safe = hostname.replace(/[^a-zA-Z0-9.-]+/g, "-");
if (safe.length > 0) {
return `${safe}.png`;
}
} catch {
// Fall through to default.
}

return DEFAULT_SCREENSHOT_FILENAME;
}
12 changes: 12 additions & 0 deletions src/scrape/types/run-target-scrape.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
import type { SyncResponse } from "@decodo/sdk-ts";

export type ScrapeBodyBuilder = (
input: string | undefined,
options: Record<string, unknown>
) => Record<string, unknown>;

export type ScrapeResponseHandler = (
response: SyncResponse,
options: Record<string, unknown>,
input?: string
) => void | Promise<void>;
5 changes: 5 additions & 0 deletions src/scrape/types/screenshot-command.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
export interface ScreenshotOptions {
country?: string;
output?: string;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you sure all three are optional?

@paulius-krutkis-dcd paulius-krutkis-dcd Jun 4, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this mirrors command options:

    .option("--country <code>", "Geo / country code (maps to geo)")
    .option("-o, --output <path>", "Write PNG to file instead of stdout")
    .option("--target <name>", "Scrape target override (default: universal)")

output is only needed when stdout is a TTY
country and target have defaults in the command action (resolveTarget defaults to universal; geo is omitted if unset)

target?: string;
}
37 changes: 37 additions & 0 deletions tests/cli/services/sort-commands-by-order.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
import { Command } from "commander";
import { describe, expect, it } from "vitest";
import { sortCommandsByOrder } from "../../../src/cli/services/sort-commands-by-order.js";

describe("sortCommandsByOrder", () => {
it("sorts known commands by explicit order", () => {
const commands = [
new Command("whoami"),
new Command("scrape"),
new Command("universal"),
new Command("search"),
new Command("setup"),
];

expect(sortCommandsByOrder(commands).map((c) => c.name())).toEqual([
"scrape",
"search",
"setup",
"whoami",
"universal",
]);
});

it("sorts unlisted commands after known ones, alphabetically", () => {
const commands = [
new Command("zebra-target"),
new Command("scrape"),
new Command("alpha-target"),
];

expect(sortCommandsByOrder(commands).map((c) => c.name())).toEqual([
"scrape",
"alpha-target",
"zebra-target",
]);
});
});
Loading
Loading