# Rakuten Scraper — Ichiba Products, Prices and Reviews in Python [![release](https://img.shields.io/github/v/release/2scraper/rakuten-scraper?sort=semver)](https://github.com/2scraper/rakuten-scraper/releases) [![tests](https://github.com/2scraper/rakuten-scraper/actions/workflows/tests.yml/badge.svg)](https://github.com/2scraper/rakuten-scraper/actions/workflows/tests.yml) [![canary](https://github.com/2scraper/rakuten-scraper/actions/workflows/canary.yml/badge.svg)](https://github.com/2scraper/rakuten-scraper/actions/workflows/canary.yml) [![python](https://img.shields.io/badge/python-3.9%20%7C%203.13-blue)](pyproject.toml) [![licence](https://img.shields.io/badge/licence-MIT-green)](LICENSE) [![engines](https://img.shields.io/badge/engines-Playwright%20%7C%20Selenium%20%7C%20pyppeteer%20%7C%20CDP-informational)](#engines) [![runs without an account](https://img.shields.io/badge/runs%20without-an%20account-brightgreen)](#do-you-need-any-of-the-paid-products) **Rakuten scraper** — an open-source (MIT) Python tool that reads Rakuten Ichiba (rakuten.co.jp) keyword searches, genre listings and item pages, and writes products, prices, Rakuten points, shops and review scores to JSON or CSV. It needs no key, no proxy and no account: measured on 2026-09-21 from a datacentre address, a 3-page keyword search returned 135 rows, complete, exit 0. Use it to build a Japanese e-commerce price dataset, or to compare two runs of the same search with the bundled `diff_runs.py`. Scrapes [Rakuten Ichiba](https://www.rakuten.co.jp) — Japan's main domestic marketplace — to JSON or CSV: products, prices, Rakuten points, shipping, shops, review scores and per-variant pricing, with four interchangeable browser back ends and one row schema shared with the rest of the [2scraper](https://github.com/2scraper) family. ## Example output Two records cut from [`sample_output.json`](sample_output.json), a real run of the keyword search below on 2026-09-21. 13 of the 42 columns are shown (titles shortened); every column is listed under [What comes out](#what-comes-out). ```json [ { "url": "https://item.rakuten.co.jp/ajinomoto/r4901111371248/", "sku": "ajinomoto:r4901111371248", "title": "【9/24まで P10倍!】味の素AGF ブレンディ スティックブラック100本…", "price": 2447.0, "currency": "JPY", "points": 220, "point_rate": 10, "rating": 4.76, "review_count": 475, "shop_name": "味の素グループ公式ショップ", "category": "水・ソフトドリンク > コーヒー > インスタントコーヒー", "genre_path": "/0/100316/100356/303017", "scraped_at": "2026-09-21T09:22:22.033307+00:00" }, { "url": "https://item.rakuten.co.jp/sawaicoffee-tea/ac-sale-1960/", "sku": "sawaicoffee-tea:ac-sale-1960", "title": "\先着300円OFFクーポン配布/ 送料無料 コーヒー コーヒー豆 コーヒー粉…", "price": 2788.0, "currency": "JPY", "points": 50, "point_rate": 2, "rating": 4.7, "review_count": 50175, "shop_name": "澤井珈琲Beans&Leaf", "category": "水・ソフトドリンク > コーヒー > コーヒー豆", "genre_path": "/0/100316/100356/201402", "scraped_at": "2026-09-21T09:22:22.033563+00:00" } ] ``` Full sample: [`sample_output.json`](sample_output.json) · 9 rows, collected 2026-09-21 (listing mode, the keyword search from the Quick start). `price` is in yen and `points` is what the purchase earns on top of it — see the FAQ on [Rakuten points](#what-are-rakuten-points-in-the-output-and-how-do-they-change-the-real-price). `genre_path` is Rakuten's genre ids; `category` is the same path in Rakuten's own (Japanese) genre names. ## Requirements and access | | | |---|---| | Runtime | Python 3.9 or newer (the offline suite runs on 3.9 and 3.13 in CI) | | Engine | Playwright with its Chromium (recommended); Selenium or pyppeteer as alternatives, one per virtualenv | | Coverage | rakuten.co.jp (Rakuten Ichiba, Japan, prices in JPY): keyword searches, genre listings and item pages. Merchant storefronts and `ranking.rakuten.co.jp` are not read — see [Coverage and limitations](#coverage-and-limitations) | | Required services | **Nothing.** No key, no proxy, no account | | Measured | 2026-11-17 at commit `fe0d03d`, from a datacentre address with no key, proxy or account (see [Verification](#verification)); first on 2026-09-21 | | Optional: a proxy | For long runs. Rakuten answers a client going too fast with HTTP 503, which is a throttle, not a block; raise `--delay` first, then spread the run across exits. See [503 is a throttle, not a block](#503-is-a-throttle-not-a-block) | | Optional: 2Captcha Scraping Browser API | Not needed on this site. What it adds: no local browser, a chosen exit country and persistent cookies per profile | | Captcha solving | Not needed: no captcha is configured on the routes this scraper reads. See [There is no captcha on this site](#there-is-no-captcha-on-this-site) | ## Quick start ```bash python3 -m venv .venv && . .venv/bin/activate pip install -r requirements.txt -r requirements-playwright.txt playwright install chromium python3 playwright_scraper.py \ --url "https://search.rakuten.co.jp/search/mall/コーヒー/" --pages 3 ``` That is the whole setup. No key, no proxy, no account, no `.env`. It writes three files to the current directory (choose the prefix with `--out`): * `rakuten_products.json` and `rakuten_products.csv` — one row per product, 45 per page; * `rakuten_products.meta.json` — the run's status (`complete` or not), stop reason, failed pages and the site's own result cap. If the run does not end with exit 0: * **exit 3, refused** — remove anything you added to the client's identity (a custom user agent, or `--fingerprint` together with `--cdp-endpoint`); see [The one thing to know first](#the-one-thing-to-know-first). * **exit 4, zero products** — the page was served but matched nothing; check the keyword and any filters in the URL. Nothing is written over earlier output. * **exit 5 or 6, throttled or partial** — raise `--delay`; `pages_failed` in the `.meta.json` names the pages that were not read. See [503 is a throttle, not a block](#503-is-a-throttle-not-a-block). All exit codes: [Exit codes](#exit-codes). Setup, `.env` and exit codes for the whole family: [How to run the 2scraper scrapers](https://2captcha.com/h/how-to-run-2scraper-open-source-scrapers). ## Coverage and limitations What the scraper reads, what it does not, and what the site itself caps. The short version: listings and item pages, never storefronts or rankings; at most 6,750 results per query; no was-price on listing rows; merchant text stays in Japanese. ### Pass a listing or an item, not a shop Three routes carry products and one looks as though it should: | URL | Reads | |---|---| | `search.rakuten.co.jp/search/mall/{keyword}/` | ✅ `--mode listing` | | `search.rakuten.co.jp/search/mall/-/{genreId}/` | ✅ `--mode listing` | | `www.rakuten.co.jp/category/{genreId}/` | ✅ `--mode listing` | | `item.rakuten.co.jp/{shop}/{code}/` | ✅ `--mode product` | | `www.rakuten.co.jp/{shopCode}/` | ❌ refused, with the reason | | `ranking.rakuten.co.jp/…` | ❌ not implemented | A **merchant storefront** is refused rather than attempted because its `__INITIAL_STATE__.state.data` is *empty* — it carries no product payload at all, so reading it would need a second parser written against markup nobody has captured. A mode that ships untested is worse than a mode that is absent. `ranking.rakuten.co.jp` is a different application: no payload, one JSON-LD `BreadcrumbList`, 80 product links in hand-rolled markup. It is also the one route on this site that cares about the window — 403 to plain HTTP and to headless Chromium (3 of 3 each), **200 to headful Chromium** (3 of 3). Not unreachable, just not implemented. ### Complete is not exhaustive Rakuten serves at most **6,750 results per query — 150 pages of 45 — however many it matched**, and it states both numbers itself. One measured query reported: ``` numFound: 3,053,682 subset: 6,750 pageSize: 45 ``` So a full 150-page run of that query is genuinely `status: complete` — it fetched everything the site will serve — and it is a **0.2% sample**. A sidecar that said only "complete" would be lying by omission, so every run records `total_results`, `reachable_max`, `page_size`, `capped_by_site` and `pages_beyond_cap` beside the status, and the engine logs the percentage. It also gives `diff_runs.py` a third meaning for `removed`: not delisted, not un-fetched, but *outside this run's slice of a capped result set*. To reach more, narrow the query with the site's own filters — genre, price band, shop, tag — and run each slice. ### Traps that look like bugs * **`rating` and `review_count` null on some rows.** Nobody has reviewed that product. Rakuten writes that as `{score: 0, numReviews: 0}` rather than as a null, on 28 of 405 rows, and a zero written through as a rating drags every average a consumer computes — so both columns are nulled together, keyed on the count. * **`original_price` and `discount_pct` null on every listing row.** The listing payload publishes no was-price of any kind: zero occurrences of `doublePrice`, `listPrice`, `referencePrice` or `strike` across 405 rows. The nearest thing it has is `sale`, on 32 of 405, which is a campaign *window* with no prices in it — read as a discount it would invent one. A verified was-price does exist on the detail page, gated on Rakuten's own `doublePrice.referencePriceVerified` flag, which is the outcome of Japan's double-pricing substantiation rule. Verified on 2 of 8 item pages (¥10,398 → ¥5,199, and ¥3,476 → ¥1,738). * **`in_stock` true on every row.** The search excludes sold-out products rather than marking them — checked specifically on deep pages (80, 120, 149, 150 of a 142,000-hit genre), where stale rows would live: 0 sold out. So a `false` on the listing route is *unproven*; the detail route reads the page's own `availability` microdata, which is the source that can say otherwise. * **45 rows from a payload with 52 entries.** Seven were sponsored placements. Rakuten injects paid slots into its own result list; they carry a click-tracking redirect instead of a product URL, so there is nothing to key a row on. The count varies between fetches of the same URL — two engines seconds apart saw 45 and 52 — which is why `position` counts the rows emitted rather than the payload's slots. * **`--locale en` does not translate anything you care about.** `?lang=en` comes back with `locale: "en"` in the payload and an English navigation bar, and product names, shop names and tag values are merchant-authored Japanese and come back **byte-identical**. `?lang=zh-tw` and `?lang=ko` are accepted with HTTP 200 and silently fall back to `ja`. * **`genre_rank` is not a position in your results.** It is the item's standing in its genre's *ranking* listing, which this run never fetched. Sibling repos in this family use published ranks to prove that cards went missing; that arithmetic does not work here. * **A readiness wait that times out costs nothing.** Every column comes out of the payload, which is in the first response with no JavaScript run at all — the same URL fetched by plain HTTP and by a real browser parses to the same 45 rows with the same skus and prices. * **Never wait on `networkidle`.** Rakuten's ad and tracking beacons keep firing indefinitely — 46 fetches and 13 pings still going after load — so that wait always runs to its full timeout. All three engines wait for `domcontentloaded`. More in [TROUBLESHOOTING.md](TROUBLESHOOTING.md). ## Examples Step by step: [How to scrape Rakuten](https://2captcha.com/h/how-to-scrape-rakuten) · the [Rakuten scraper page on 2scraper.com](https://2scraper.com/scrapers/rakuten-scraper/). Each command is a complete run once the [Quick start](#quick-start) setup is done. ```bash # A keyword search, three pages, JSON and CSV python3 playwright_scraper.py \ --url "https://search.rakuten.co.jp/search/mall/コーヒー/" --pages 3 # A genre listing, by Rakuten's own genre id python3 playwright_scraper.py \ --url "https://www.rakuten.co.jp/category/100356/" --pages 5 --format csv # One product: per-variant prices and a variant count (and a was-price while # the shop's own sale is verified: see the note under "Measured") python3 playwright_scraper.py --mode product \ --url "https://item.rakuten.co.jp/sawaicoffee-tea/solandluna/" # The site's own filters are part of the URL and survive pagination python3 playwright_scraper.py \ --url "https://search.rakuten.co.jp/search/mall/コーヒー/?min=1000&max=3000" \ --pages 3 # Spread a long run across exits (and see the note on 503 below). Put one # proxy URL per line in the list; exits*.txt, proxies*.txt and proxylist*.txt # are git-ignored, because that file holds logins and passwords. python3 playwright_scraper.py \ --url "https://search.rakuten.co.jp/search/mall/コーヒー/" --pages 50 \ --proxy-file exits.txt --proxy-rotate per-page --delay 4 ``` ## Verification Last checked on **2026-11-17** at commit `fe0d03d`. The scraper's code is unchanged since `ab9babd`, where the full three-page Quick start was run on 2026-11-09. | Date, commit | Check | Conditions | Result | |---|---|---|---| | 2026-11-17, `fe0d03d` | Quick start command with `--pages 1` (one page of 45; kept short on purpose) | a fresh virtualenv, Python 3.13, `requirements.txt` + `requirements-playwright.txt` (Playwright 1.65.2), headless Chromium, no `.env`, one datacentre address | **45 rows**, 45 unique `sku`, `status: complete`, exit 0; `price` and `currency` (`JPY`) on 45 of 45; same 42 columns in the same order as `sample_output.json` | | 2026-11-17, `fe0d03d` | Site cap reported | the same run | `total_results` 3,233,987, `reachable_max` 6,750 — a complete run is a sample, as described [above](#complete-is-not-exhaustive) | | 2026-11-09, `ab9babd` | Quick start, as written above (`--pages 3`) | a fresh virtualenv, Python 3.13, `requirements.txt` + `requirements-playwright.txt`, headless Chromium, no `.env`, one Hetzner datacentre address | **135 rows**, 135 unique `sku`, `status: complete`, exit 0; `price` and `currency` (`JPY`) on 135 of 135; `total_results` 3,164,855 | | 2026-11-17, `fe0d03d` | Offline suite | `python3 smoke_test.py` and `python3 .github/ci_checks.py --all`, core requirements only | passed (engine groups skipped, as designed, without their libraries) | | 2026-10-01 to 2026-11-16 | [Canary](https://github.com/2scraper/rakuten-scraper/actions/workflows/canary.yml) (daily, weekly since 2026-10-16) | a 3-page genre listing from a GitHub runner with no secrets | green on every run, the last one on 2026-11-16 at `fe0d03d` | ## The one thing to know first **You do not need any of the paid products to read this site's listings, and the reason is not what you would guess.** Rakuten is fronted by Akamai, and what Akamai refuses here is not a datacentre address — it is a client whose claimed identity and TLS fingerprint disagree. Measured 2026-09-21 from one Hetzner datacentre address in Helsinki, nothing else changed between the rows: | Client | Result | |---|---| | `curl`, curl's own User-Agent | **200**, 919 KB, 45 products | | `curl`, a Chrome User-Agent | **200**, **42-43 bytes**, `Reference #…` | | headless Chromium (real UA, real TLS) | **200**, 855 KB, 45 products | | headful Chromium | **200**, 969 KB, 45 products | The two `curl` rows were repeated on 2026-10-29 from a Hetzner datacentre address and gave the same answer (919,197 bytes and 45 parsed products; 42 bytes, the length varying with the reference id), which also replaced the size this table first carried for the first row. The two Chromium rows are from 2026-09-21 and were not repeated. Read it by column: the address is identical in all four and the *consistency of the client* decides everything. curl claiming to be curl is served; curl claiming to be Chrome is refused. So the practical advice is the opposite of most sites in this family — **if you are being refused, look for a disguise you added before you buy an exit**: * do not pass `--fingerprint` or a custom user agent together with `--cdp-endpoint`, because the remote browser already brings its own identity and stacking a second one manufactures exactly this mismatch; * do not put a browser user agent on an HTTP client. Two more things follow from that table, and both are easy to get wrong: * **A refusal arrives under HTTP 200.** The status code is not the signal. This scraper decides "was this served at all" from the reference-id shape and from the page being built out of Rakuten's own `r10s.jp` asset hosts (90–343 references on every served page, 0–1 on every refusal). * **Headless is fine.** It is the default here. The browser even announces itself — a tracking beacon from the headless run carried `HeadlessChrome/153.0.8010.12` — and Akamai served it anyway. The canary in this repository is that claim under test: a real three-page scrape, weekly (Mondays), from a bare GitHub runner, with **no secrets**. If Rakuten ever puts the listing routes behind something, the badge above goes red on the next weekly run: a blocked, partial or empty run fails the job rather than passing with a warning. ## Install One engine, not all three. They declare mutually unsatisfiable pins — playwright and pyppeteer disagree on `pyee`, pyppeteer and selenium on `urllib3` — so use a virtualenv per engine if you need more than one. ```bash python3 -m venv .venv && . .venv/bin/activate pip install -r requirements.txt -r requirements-playwright.txt playwright install chromium # the recommended engine ``` ```bash pip install -r requirements.txt -r requirements-selenium.txt # or pip install -r requirements.txt -r requirements-puppeteer.txt # or ``` `pip install .[playwright]` is equivalent. pyppeteer downloads a Chromium of its own, and that build does not start on every machine (seen on an Apple Silicon Mac, where it is an x86_64 build and ends with `Browser closed unexpectedly`). Pass `--chromium-path` with a Chrome or Chromium that you installed, e.g. `--chromium-path "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome"`. Without it the run now ends with that advice and exit 5 rather than a traceback; the Apple Silicon case itself was reported by an audit and was not reproduced here (Linux, a browser that exits at once, gave the same error and the same handling). ## What comes out 42 columns. The first 18 are the family prefix, byte-identical and in the same order as every other repo in this family, so one consumer reads a rakuten run and a mediamarkt run with the same code. ```json { "source": "rakuten.co.jp", "scraped_at": "2026-09-21T09:22:22+00:00", "url": "https://item.rakuten.co.jp/ajinomoto/r4901111371248/", "sku": "ajinomoto:r4901111371248", "title": "【9/24まで P10倍!】味の素AGF ブレンディ スティックブラック100本…", "brand": "味の素AGF", "price": 2447.0, "currency": "JPY", "original_price": null, "discount_pct": null, "rating": 4.76, "review_count": 475, "in_stock": true, "image_url": "https://thumbnail.image.rakuten.co.jp/@0_mall/ajinomoto/…", "category": "水・ソフトドリンク > コーヒー > インスタントコーヒー", "price_source": "state", "page": 1, "position": 1, "variant_id": "r4901111371248", "item_number": "10000338", "price_max": 6975.0, "has_price_range": true, "subscription_price": 2324.0, "points": 220, "point_rate": 10, "shipping_fee": 0.0, "free_shipping": true, "delivery_estimate": "最短9/23お届け", "shop_name": "味の素グループ公式ショップ", "shop_code": "ajinomoto", "shop_id": 383220, "shop_url": "https://www.rakuten.co.jp/ajinomoto/", "genre_id": 303017, "genre_path": "/0/100316/100356/303017", "genre_rank": 2, "is_super_deal": false, "is_39shop": true, "is_official_shop": true, "variant_count": null, "original_price_label": null, "tags": {"ブランド": ["味の素AGF"], "単品重量": ["~ 99g"]}, "variants": null } ``` `sample_output.json` and `sample_output.csv` are cut from a real run, not written by hand. Three columns are worth explaining because this site is unusual: * **`sku` is `{shopCode}:{manageNumber}`** — Rakuten's own item code, exactly as a detail page states it in ``. It is the same id on both routes, so you can join a listing run to a product run on it. Deliberately *not* the payload's `variantId`, which names the pre-selected SKU inside the item and matches the URL's own code on only 39 of 180 measured rows. * **`points` is half the price story.** Rakuten is a points marketplace: a 10× point campaign is effectively a 10% discount that never touches the price column, so a monitor watching `price` alone would call a product unchanged through an entire Super Sale. `diff_runs.py` tracks both. * **`shop_*` is not run metadata.** This is a mall — 74 distinct merchants across 405 measured rows — so the same physical article is sold by many shops at many prices under many ids. ### Exit codes | Code | Means | |---|---| | 0 | products written | | 1 | crash | | 2 | bad usage | | 3 | blocked before parsing | | 4 | served a page, zero products | | 5 | the content was never obtained: a remote API error, a dead exit, a load that never completed, a page that stayed throttled after every wait | | 6 | partial run | A run that finds nothing **writes nothing** — `--allow-empty` is the opt-out — so a failed run cannot replace last night's good output with `[]`. With the opt-out the empty file IS written, but the exit code does not change: a run that found nothing still exits 4 and its sidecar says `status: failed`. `--out` may name a directory that does not exist yet: it is created before the first request, and one that cannot be created is a usage error (exit 2) instead of a traceback after the whole run. A page that stays HTTP 503 after every wait is a **failed page** (`stop_reason: throttled`, `pages_failed` names it): exit 6 if other pages produced rows, 5 if none did. It is never read as a page with no products. **All three files are replaced atomically.** Each is written to a temporary file beside it and renamed over the old one, so a crash, a kill or a full disk leaves the previous run's output exactly as it was rather than a half-written file where a complete one had been. Measured before the fix: a 2,084-byte good `out.json` came back **0 bytes and invalid JSON** after an interrupted write, because `open(path, "w")` truncates before writing a single byte. That matters most for the sidecar, which is the file a consumer branches on. Every run also writes `.meta.json`, and on this site it carries the site's own arithmetic as well as the status. See [complete is not exhaustive](#complete-is-not-exhaustive). **A CSV cell cannot be a formula.** A spreadsheet executes a cell beginning `=`, `+`, `-` or `@`, and every string here was written by a merchant, not by this code. Such a cell is prefixed with an apostrophe in the **CSV only** — the JSON keeps the site's bytes exactly as served — and `csv_cells_escaped` in the sidecar declares how many cells the two files therefore differ in. Only strings are touched: escaping a number would turn `-5` into text and break every sum written over the column. This does not fire on today's data and is not claimed to. Measured 2026-09-30: **0 of 42,592 string cells across 2,662 stored rows** begin with any of those characters. It is here because a Rakuten title routinely opens with a discount badge, and `-16%OFF ...` is one merchant away from being a leading minus. ## Where the data actually is **The JSON-LD on a listing page is a trap.** A search page has exactly one `application/ld+json` block, it is an `ItemList`, and it holds **ten** items whose every url carries `?scid=seo-carousel-search` — it is the SEO recommendation carousel — while the page itself holds **45** products. A JSON-LD-primary parser returns ten rows of the wrong products and reports success. The real sources: | Route | Source | |---|---| | listing | `window.__INITIAL_STATE__.state.data.ichibaSearch` — `items` (45/page) + `pagination` | | product | `