Skip to content

domain-skill: google-maps reviews scraping #489

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
theoutsider04 wants to merge 1 commit into
base: main
Choose a base branch
from
Open

domain-skill: google-maps reviews scraping #489

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 36 additions & 0 deletions domain-skills/google-maps/reviews.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
# Google Maps — business profile reviews

Scrape all reviews for a business (name, stars, date, text) without an API key.

## URL patterns

- Search: `https://www.google.com/maps/search/<business>+<city>` — a unique match redirects straight to the place page.
- Direct by place_id: `https://www.google.com/maps/place/?q=place_id:<PLACE_ID>` — resolves and redirects to the canonical place URL.
- **Language matters:** Google auto-translates reviews into the UI language. Add `hl=fr` (etc.) and re-scrape to get originals — for each review, one language is the reviewer's original and the other is Google's translation. Scrape both if you need the pair.

@cubic-dev-ai cubic-dev-ai Bot Jul 6, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: The guidance overstates the reliability of the two-language hl heuristic. It claims that scraping in two UI languages always yields one original and one translation per review, but this only holds when a review's original language matches one of the two hl values. For reviews in a third language, both scrapes produce translations — so downstream consumers may mislabel translated text as "original." Consider qualifying this (e.g., "one scrape may match the reviewer's language, yielding the original, while the other is translated") or noting that the approach only works when the review language is known to match one of your chosen hl values. If you can access per-review language metadata (analogous to the Places API's textLanguageCode), that would be a more reliable signal.

Prompt for AI agents 
Check if this issue is valid — if so, understand the root cause and fix it. At domain-skills/google-maps/reviews.md, line 9:

<comment>The guidance overstates the reliability of the two-language `hl` heuristic. It claims that scraping in two UI languages always yields one original and one translation per review, but this only holds when a review's original language matches one of the two `hl` values. For reviews in a third language, both scrapes produce translations — so downstream consumers may mislabel translated text as "original." Consider qualifying this (e.g., "one scrape may match the reviewer's language, yielding the original, while the other is translated") or noting that the approach only works when the review language is known to match one of your chosen `hl` values. If you can access per-review language metadata (analogous to the Places API's `textLanguageCode`), that would be a more reliable signal.</comment>

<file context>
@@ -0,0 +1,36 @@
+
+- Search: `https://www.google.com/maps/search/<business>+<city>` — a unique match redirects straight to the place page.
+- Direct by place_id: `https://www.google.com/maps/place/?q=place_id:<PLACE_ID>` — resolves and redirects to the canonical place URL.
+- **Language matters:** Google auto-translates reviews into the UI language. Add `hl=fr` (etc.) and re-scrape to get originals — for each review, one language is the reviewer's original and the other is Google's translation. Scrape both if you need the pair.
+- The place URL contains the ftid `!1s0x...:0x<hex>` — the hex after the colon is the CID: `int(hex, 16)` → `https://maps.google.com/?cid=<decimal>`.
+- All-reviews / write-review deep links (no Maps UI): `https://search.google.com/local/reviews?placeid=<PLACE_ID>` and `https://search.google.com/local/writereview?placeid=<PLACE_ID>`.
</file context>
Suggested change
- **Language matters:** Google auto-translates reviews into the UI language. Add `hl=fr` (etc.) and re-scrape to get originals — for each review, one language is the reviewer's original and the other is Google's translation. Scrape both if you need the pair.
- **Language matters:** Google auto-translates reviews into the UI language. Add `hl=fr` (etc.) and re-scrape — if the reviewer's original language matches this `hl`, you'll get the original text; otherwise you'll get a Google translation. For reviews in languages other than your chosen `hl` values, both scrapes may be translations, so this heuristic only reliably yields an original/translation pair when the review language is known to match one of your `hl` settings. Scrape both if you need the pair.
Fix with cubic

- The place URL contains the ftid `!1s0x...:0x<hex>` — the hex after the colon is the CID: `int(hex, 16)` → `https://maps.google.com/?cid=<decimal>`.
- All-reviews / write-review deep links (no Maps UI): `https://search.google.com/local/reviews?placeid=<PLACE_ID>` and `https://search.google.com/local/writereview?placeid=<PLACE_ID>`.

## Flow

1. Navigate to the place. Click the reviews tab: `[...document.querySelectorAll('button[role=tab]')].find(b => /Avis|Reviews/i.test(b.textContent)).click()`.
2. Reviews live in a lazy feed. Scroll the feed container (NOT the window): `document.querySelector('div[role=feed], div.m6QErb.DxyBCb').scrollBy(0, 3000)` in a loop with ~1 s sleeps until no new cards appear.
3. Expand truncated texts: click all `button.w8nwRe` ("More"/"Plus").
4. Extract per card:

```js
document.querySelectorAll('div[data-review-id][jsaction]').forEach(el => {

@cubic-dev-ai cubic-dev-ai Bot Jul 6, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: The extraction example doesn't implement the deduping guard that its own documentation recommends. The 'Traps' section says duplicate div[data-review-id] nodes are removed by '[jsaction] filter plus name-presence check,' but the code block only applies the [jsaction] selector and never checks that name is truthy before processing the element. If a user copies this example, duplicate card/expanded-state nodes will be emitted as separate reviews. Consider adding a if (!name) return; guard (or a data-review-id Set) so the sample matches the described deduping logic.

Prompt for AI agents 
Check if this issue is valid — if so, understand the root cause and fix it. At domain-skills/google-maps/reviews.md, line 21:

<comment>The extraction example doesn't implement the deduping guard that its own documentation recommends. The 'Traps' section says duplicate `div[data-review-id]` nodes are removed by '[jsaction] filter **plus** name-presence check,' but the code block only applies the `[jsaction]` selector and never checks that `name` is truthy before processing the element. If a user copies this example, duplicate card/expanded-state nodes will be emitted as separate reviews. Consider adding a `if (!name) return;` guard (or a `data-review-id` Set) so the sample matches the described deduping logic.</comment>

<file context>
@@ -0,0 +1,36 @@
+4. Extract per card:
+
+```js
+document.querySelectorAll('div[data-review-id][jsaction]').forEach(el => {
+  const name  = el.querySelector('.d4r55')?.textContent?.trim();   // reviewer name
+  const stars = el.querySelector('.kvMYJc')?.getAttribute('aria-label'); // "5 stars"/"5 étoiles"
</file context>
Fix with cubic

const name = el.querySelector('.d4r55')?.textContent?.trim(); // reviewer name
const stars = el.querySelector('.kvMYJc')?.getAttribute('aria-label'); // "5 stars"/"5 étoiles"
const date = el.querySelector('.rsqaWe')?.textContent?.trim(); // relative ("2 months ago")
const text = el.querySelector('.wiI7pd')?.textContent?.trim(); // review body
const meta = el.querySelector('.RfnDt')?.textContent?.trim(); // "Local Guide · 15 reviews"
});
```

## Traps

- **Rating-only reviews leak the owner's response as the text.** If a reviewer left stars but no text, `.wiI7pd` inside that card can be the *owner's reply* ("Thank you X for..."). Detect: text addresses the reviewer by name / thanks them. Treat as empty.
- Dates are relative ("a day ago") — convert against scrape date immediately, they age.
- Duplicate `div[data-review-id]` nodes exist (card + expanded state); the `[jsaction]` filter plus name-presence check dedupes.
- Class names (`.d4r55`, `.wiI7pd`, `.rsqaWe`, `.kvMYJc`, `.RfnDt`, `.w8nwRe`) are obfuscated CSS-modules — they've been stable for a while but verify with a screenshot before a long scrape; `div[data-review-id]` and `div[role=feed]` are the durable anchors.
- No consent-wall in CA/US normally; EU IPs may get a consent interstitial before Maps loads.