Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 8 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -145,6 +145,9 @@ omk-crawl https://example.com -v
omk-crawl reddit://r/programming
omk-crawl 'appstore://search?q=요기요'
omk-crawl 'baemin://shops?lat=36.83&lng=127.13&limit=40'
omk-crawl 'reach://github/search?q=agent%20reach' --json
omk-crawl 'reach://bilibili/hot?limit=5' --json
omk-crawl 'reach://xiaoyuzhou/transcribe?url=<episode-url>' --json

# Offline capture and mobile surfaces
omk-crawl capture.har --json
Expand All @@ -153,6 +156,8 @@ omk-crawl app.ipa --json
omk-crawl android://
```

`reach://` uses the separately installed `agent-reach` executable (`OMK_AGENT_REACH_BIN` overrides its path). Raw response envelopes and per-item `raw` data are omitted by default; set `OMK_AGENT_REACH_RAW=1` only when sensitive upstream data is intentionally needed. See [Agent Reach adapter](references/agent-reach.md).

### Deep crawl controls

| Control | Meaning |
Expand Down Expand Up @@ -200,8 +205,8 @@ print(rotated.metadata.get("rotation"))
┌──────────────────────────┼──────────────────────────┐
▼ ▼ ▼
single URL DeepCrawler target adapters
escalation BFS + sitemap Reddit / Baemin /
CrawlResult bounded frontier Android / iOS / HAR
escalation BFS + sitemap Agent Reach / Reddit /
CrawlResult bounded frontier Baemin / Android / iOS / HAR
│
▼
┌──────────────────┐
Expand All @@ -227,6 +232,7 @@ print(rotated.metadata.get("rotation"))
| agent-driven interaction | `browser-use` | last-resort public workflow |
| many pages | `omk-crawl --deep` | bounded BFS and sitemap seeding |
| public proxy rotation | `--proxy-pool` | harvest, score, rotate, refill |
| platform search/read/hot/transcribe | `agent_reach`, `reach://` | bounded Agent Reach CLI routing |
| files and captures | `markitdown`, `har` | local conversion and inspection |

See the detailed router at [references/routing.md](references/routing.md) and adapter notes in [references/tools](references/tools).
Expand Down
21 changes: 20 additions & 1 deletion SKILL.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
name: omk-crawling
description: 'OMK 웹·데이터 수집/추출 툴���스 — 13개 도구 + 돌파(breakthrough) 레이어. crawl4ai(웹→LLM
description: 'OMK 웹·데이터 수집/추출 툴���스 — 14개 도구 + 돌파(breakthrough) 레이어. agent-reach(플랫폼 검색·읽기·hot·transcribe), crawl4ai(웹→LLM
Markdown·딥크롤·MCP), scrapy(대규모 클래식), crawlee(큐·오토스케일·프록시), browser-use(LLM
브라우저 에이전트), curl-impersonate/curl_cffi(TLS·JA3 위장 403 우회), autoscraper(예시→규칙
학습), markitdown(파일→Markdown), scrcpy(Android 미러·제어), scrapling(스텔스·안티봇),
Expand All @@ -21,6 +21,7 @@ metadata:
family: web-access
siblings: [insane-search, scrapling]
tools:
- { name: agent-reach, repo: https://github.com/Panniantong/agent-reach, pkg: 'CLI / built-in agent_reach adapter', license: MIT }
- { name: crawl4ai, repo: https://github.com/unclecode/crawl4ai, pkg: 'pip:crawl4ai==0.9.2', license: Apache-2.0 }
- { name: scrapy, repo: https://github.com/scrapy/scrapy, pkg: 'pip:scrapy==2.17.0', license: BSD-3-Clause }
- { name: crawlee, repo: https://github.com/apify/crawlee, pkg: 'npm:crawlee@3.17.0 / pip:crawlee==1.8.3', license: Apache-2.0 }
Expand All @@ -47,6 +48,7 @@ metadata:
## 레이어별 지도

```
⓪ 플랫폼 읽기 agent-reach(검색·read·hot·subtitle/transcribe — 실제 channel/action만)
① Fetch / 안티핑거프린트 curl-impersonate·curl_cffi(TLS·JA3) · scrapling(스텔스) · insane-search(단일 URL)
② Crawl 프레임워크 scrapy(클���식·생태계) · crawlee(큐·오토스케일·프록시) · crawl4ai(LLM·딥크롤·MCP)
③ 브라우저 자동화 camoufox(안티디텍트) · nodriver(CDP) · patchright(패치PW) · browser-use(LLM) · crawl4ai/scrapling
Expand Down Expand Up @@ -86,6 +88,7 @@ metadata:
| 예시만 주면 **추출 규칙 학습** (브라우저 X, 초경량) | **autoscraper** | [tools/autoscraper.md](references/tools/autoscraper.md) |
| PDF·Office·이미지·오디오·HTML **파일 → Markdown** | **markitdown** | [tools/markitdown.md](references/tools/markitdown.md) |
| 웹/API 없이 **Android 앱에만** 있는 데이터 | **scrcpy** | [tools/scrcpy.md](references/tools/scrcpy.md) |
| GitHub·YouTube·Bilibili·V2EX·Xueqiu·Xiaoyuzhou 등 플랫폼 읽기 | **agent-reach** / `reach://` | [agent-reach.md](references/agent-reach.md) |
| **Reddit** 서브레딧/검색 JSON | **reddit** / `RedditClient` | [tools/reddit.md](references/tools/reddit.md) |
| **배달의민족** 근처 가게 리스트 (geo) | **baemin** / `BaeminClient` | [tools/baemin.md](references/tools/baemin.md) |
| **iOS App Store** 메타 (IPA 없을 때) | **appstore** / `AppStoreClient` | [tools/appstore.md](references/tools/appstore.md) |
Expand Down Expand Up @@ -262,11 +265,27 @@ scrcpy # 미러링+제어
scrcpy --record session.mp4 --no-audio # 세션 녹화
```

### agent_reach — 실제 플랫폼 channel/action 읽기

별도 설치된 `agent-reach` 실행 파일을 셸 없이 한 번만 호출한다. hot 계열은 positional query 없이
실행하고, 상위가 지원하는 경우 `limit`만 `--limit`으로 전달한다.

```bash
omk-crawl 'reach://github/search?q=agent%20reach' --json
omk-crawl 'reach://bilibili/hot?limit=5' --json
omk-crawl 'reach://xueqiu/hot_posts' --json
omk-crawl 'reach://xiaoyuzhou/transcribe?url=<episode-url>' --json
```

`raw_response`와 item의 `raw`는 기본 제외다. 민감한 상위 원본이 필요할 때만
`OMK_AGENT_REACH_RAW=1`을 명시적으로 설정한다. 실행 경로는 `OMK_AGENT_REACH_BIN`으로 바꾼다.

## references/

| 경로 | 내용 |
| ------ | ------ |
| [`references/routing.md`](references/routing.md) | **크로스-툴 결정 트리** (무엇을 언제) |
| [`references/agent-reach.md`](references/agent-reach.md) | Agent Reach channel/action·raw·프로세스 계약 |
| [`references/tools/*.md`](references/tools/) | 도구별 상세 (10개) |
| [`references/choosing.md`](references/choosing.md) · [`extraction`](references/extraction.md) · [`deep-crawl`](references/deep-crawl.md) · [`docker-mcp`](references/docker-mcp.md) · [`cli`](references/cli.md) | crawl4ai 세부 |
| `examples/*.py` | crawl4ai·markitdown·autoscraper·curl_cffi 실행 예제 |
Expand Down
6 changes: 5 additions & 1 deletion omk_crawl/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,9 @@
omk-crawl ios://Freeform # iOS-only style search
omk-crawl reddit://r/programming # Reddit listing JSON
omk-crawl 'reddit://search?q=rust+async&limit=10'
omk-crawl 'reach://github/search?q=agent%20reach' --json
omk-crawl 'reach://bilibili/hot?limit=5' --json
omk-crawl 'reach://xiaoyuzhou/transcribe?url=<episode-url>' --json
"""

from __future__ import annotations
Expand Down Expand Up @@ -137,14 +140,15 @@ def build_parser() -> argparse.ArgumentParser:
description=("Smart crawling toolbox — web auto-escalation + Android/iOS surfaces"),
epilog=(
"Web: curl_cffi → crawl4ai → scrapling → browser-use. "
"Agent Reach: reach://<channel>/<action> (agent-reach binary). "
"Mobile: apk / ipa / android:// (adb). "
"Crawl4AI: https://github.com/unclecode/crawl4ai"
),
)
parser.add_argument(
"url",
nargs="?",
help="URL, file path, .har/.apk/.ipa, or android:// scheme",
help="URL, file path, .har/.apk/.ipa, reach:// or android:// scheme",
)
parser.add_argument("--tool", "-t", help="Force a specific tool (skip auto-escalation)")
parser.add_argument("--output", "-o", help="Save output to file")
Expand Down
10 changes: 9 additions & 1 deletion omk_crawl/detect.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@
from __future__ import annotations

import importlib.util
import os
import shutil
from dataclasses import dataclass
from enum import Flag, auto
from functools import lru_cache
Expand Down Expand Up @@ -73,6 +75,7 @@ class Detection:
"ios": "always",
"scrcpy": "adb", # needs adb on PATH
"android": "adb",
"agent_reach": "agent-reach",
}


Expand All @@ -84,6 +87,9 @@ def _host_tool_available(name: str) -> bool:
from omk_crawl.mobile.device import which_bin

return which_bin("adb") is not None
if kind == "agent-reach":
executable = os.environ.get("OMK_AGENT_REACH_BIN", kind)
return os.name == "posix" and shutil.which(executable) is not None
return False


Expand Down Expand Up @@ -112,7 +118,9 @@ def available_tools() -> list[str]:

def missing_tools() -> list[str]:
"""List tools that need install (pip or host)."""
names = [n for n in list(_TOOL_MODULES) + ["apk", "ipa", "scrcpy"] if n != "android"]
names = [
n for n in list(_TOOL_MODULES) + ["apk", "ipa", "scrcpy", "agent_reach"] if n != "android"
]
return [name for name in names if not tool_available(name)]


Expand Down
6 changes: 4 additions & 2 deletions omk_crawl/request_runner.py
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,8 @@ def fetch_sync(
router: SmartRouter, execution: Execution, tool: BaseTool, url: str, kwargs: dict[str, Any]
) -> CrawlResult:
result = CrawlResult(url=url, tool=tool.name, status=CrawlStatus.ERROR)
for retry in range(router.max_retries + 1):
max_retries = 0 if tool.name == "agent_reach" else router.max_retries

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

agent_reach는 구조화된 오류 코드와 retry-after를 자체 보유하므로 공통 재시도 루프에서 제외했습니다. 따라서 NEED_LOGIN, RATE_LIMITED, TIMEOUT 등에서 같은 호출을 반복하지 않고 원래 종료 사유가 유지됩니다. async 경로에도 동일하게 적용됐습니다.

for retry in range(max_retries + 1):
if execution.stopped():
break
if retry:
Expand Down Expand Up @@ -60,7 +61,8 @@ async def fetch_async(
router: SmartRouter, execution: Execution, tool: BaseTool, url: str, kwargs: dict[str, Any]
) -> CrawlResult:
result = CrawlResult(url=url, tool=tool.name, status=CrawlStatus.ERROR)
for retry in range(router.max_retries + 1):
max_retries = 0 if tool.name == "agent_reach" else router.max_retries
for retry in range(max_retries + 1):
if execution.stopped():
break
if retry:
Expand Down
14 changes: 13 additions & 1 deletion omk_crawl/route_engine.py
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,10 @@ def _prepare(router: SmartRouter, url: str, kwargs: dict[str, Any]) -> _Plan:


def _stop_reason(result: CrawlResult, detection: Detection, method: str) -> str | None:
if result.tool == "agent_reach":
reason = result.metadata.get("stop_reason")
if isinstance(reason, str) and reason:
return reason
if result.status_code in (401, 407) or is_auth_block(detection.block):
result.metadata["auth_stop"] = True
return "auth_required"
Expand Down Expand Up @@ -159,7 +163,15 @@ def _exhausted(
) -> CrawlResult:
result = best or CrawlResult(url=plan.target.value, status=CrawlStatus.ERROR)
result.metadata["escalation_exhausted"] = True
return _finish(router, plan, result, attempts, "max_attempts" if pending else "tools_exhausted")
adapter_reason = result.metadata.get("stop_reason") if result.tool == "agent_reach" else None
reason = (
adapter_reason
if isinstance(adapter_reason, str) and adapter_reason
else "max_attempts"
if pending
else "tools_exhausted"
)
return _finish(router, plan, result, attempts, reason)


def diagnose(router: SmartRouter, url: str, kwargs: dict[str, Any]) -> dict[str, Any]:
Expand Down
51 changes: 46 additions & 5 deletions omk_crawl/targets.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

from dataclasses import dataclass
from pathlib import Path
from urllib.parse import urlsplit
from urllib.parse import unquote, urlsplit

_SCHEMES = {
"android": "scrcpy",
Expand All @@ -13,11 +13,35 @@
"appstore": "appstore",
"ios": "appstore",
"reddit": "reddit",
"reach": "agent_reach",
"baemin": "baemin",
"apk": "apk",
"ipa": "ipa",
"proxy": "proxy_pool",
}

# Official Agent Reach base-domain → channel mapping. Exact HTTPS hosts and their
# subdomains use agent_reach; route_for_target() performs the channel dispatch.
# Reddit is intentionally absent so its specialized adapter keeps precedence.
_AGENT_REACH_HOST_CHANNELS = {
"github.com": "github",
"youtube.com": "youtube",
"youtu.be": "youtube",
"bilibili.com": "bilibili",
"b23.tv": "bilibili",
"v2ex.com": "v2ex",
"twitter.com": "twitter",
"x.com": "twitter",
"facebook.com": "facebook",
"instagram.com": "instagram",
"xiaohongshu.com": "xiaohongshu",
"xhslink.com": "xiaohongshu",
"linkedin.com": "linkedin",
"zhipin.com": "boss",
"xueqiu.com": "xueqiu",
"xiaoyuzhoufm.com": "xiaoyuzhou",
}

_FILES = {
".har": "har",
".apk": "apk",
Expand All @@ -43,20 +67,37 @@ class Target:
is_web: bool


def _has_control_chars(value: str) -> bool:
return any(ord(char) < 32 or 127 <= ord(char) <= 159 for char in value)


def resolve_target(value: str, *, explicit: bool = False) -> Target:
"""Classify without fetching, opening a device, or reading file contents."""
if not isinstance(value, str) or not value.strip():
raise ValueError("Target must be a non-empty string")
if _has_control_chars(value):
raise ValueError("Target contains control characters")
parsed = urlsplit(value)
if parsed.scheme == "reach" and _has_control_chars(unquote(value)):
raise ValueError("Target contains decoded control characters")
if parsed.scheme in {"http", "https"}:
if not parsed.hostname or any(ord(char) < 32 for char in value):
if not parsed.hostname:
raise ValueError("Invalid HTTP(S) target")
# Validate the port before choosing a network adapter.
_ = parsed.port
port = parsed.port
if port is not None and not 1 <= port <= 65535:
raise ValueError("Invalid HTTP(S) port")
host = parsed.hostname.lower()
tool = None
if not explicit and (host == "reddit.com" or host.endswith(".reddit.com")):
tool = "reddit"
if not explicit:
# Specialized adapters take precedence over generic host routing.
if host == "reddit.com" or host.endswith(".reddit.com"):
tool = "reddit"
elif parsed.scheme == "https" and any(

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

자동 Agent Reach 선택은 HTTPS에만 적용됩니다. plain HTTP는 기존 ordinary web 경로를 유지하고, Reddit은 계속 전용 reddit 어댑터가 우선합니다. 이 조합으로 기존 라우팅 회귀를 막았습니다.

host == domain or host.endswith(f".{domain}")
for domain in _AGENT_REACH_HOST_CHANNELS
):
tool = "agent_reach"
return Target(value, tool, tool is None)
if parsed.scheme in _SCHEMES:
return Target(value, _SCHEMES[parsed.scheme], False)
Expand Down
4 changes: 3 additions & 1 deletion omk_crawl/tools/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@

from __future__ import annotations

from omk_crawl.tools.agent_reach_tool import AgentReachTool
from omk_crawl.tools.apk_tool import ApkTool
from omk_crawl.tools.appstore_tool import AppStoreTool
from omk_crawl.tools.autoscraper_tool import AutoscraperTool
Expand Down Expand Up @@ -56,6 +57,7 @@
"proxy_pool": ProxyPoolTool,
"proxy": ProxyPoolTool, # alias — omk-crawl proxy://…
"reddit": RedditTool,
"agent_reach": AgentReachTool,
# Mobile / native (layer 5)
"apk": ApkTool,
"ipa": IpaTool,
Expand All @@ -66,7 +68,7 @@
}

MOBILE_TOOLS: frozenset[str] = frozenset({"apk", "ipa", "appstore", "ios", "scrcpy", "android"})
TARGET_TOOLS: frozenset[str] = frozenset({"baemin", "reddit"})
TARGET_TOOLS: frozenset[str] = frozenset({"baemin", "reddit", "agent_reach"})


def get_tool(name: str, **kwargs) -> BaseTool:
Expand Down
Loading
Loading