# Rakuten Scraper — Ichiba Products, Prices and Reviews in Python
[](https://github.com/2scraper/rakuten-scraper/releases)
[](https://github.com/2scraper/rakuten-scraper/actions/workflows/tests.yml)
[](https://github.com/2scraper/rakuten-scraper/actions/workflows/canary.yml)
[](pyproject.toml)
[](LICENSE)
[](#engines)
[](#do-you-need-any-of-the-paid-products)
**Rakuten scraper** — an open-source (MIT) Python tool that reads Rakuten Ichiba (rakuten.co.jp) keyword searches, genre listings and item pages, and writes products, prices, Rakuten points, shops and review scores to JSON or CSV. It needs no key, no proxy and no account: measured on 2026-09-21 from a datacentre address, a 3-page keyword search returned 135 rows, complete, exit 0. Use it to build a Japanese e-commerce price dataset, or to compare two runs of the same search with the bundled `diff_runs.py`.
Scrapes [Rakuten Ichiba](https://www.rakuten.co.jp) — Japan's main domestic
marketplace — to JSON or CSV: products, prices, Rakuten points, shipping,
shops, review scores and per-variant pricing, with four interchangeable
browser back ends and one row schema shared with the rest of the
[2scraper](https://github.com/2scraper) family.
## Example output
Two records cut from [`sample_output.json`](sample_output.json), a real run
of the keyword search below on 2026-09-21. 13 of the 42 columns are shown
(titles shortened); every column is listed under
[What comes out](#what-comes-out).
```json
[
{
"url": "https://item.rakuten.co.jp/ajinomoto/r4901111371248/",
"sku": "ajinomoto:r4901111371248",
"title": "【9/24まで P10倍!】味の素AGF ブレンディ スティックブラック100本…",
"price": 2447.0,
"currency": "JPY",
"points": 220,
"point_rate": 10,
"rating": 4.76,
"review_count": 475,
"shop_name": "味の素グループ公式ショップ",
"category": "水・ソフトドリンク > コーヒー > インスタントコーヒー",
"genre_path": "/0/100316/100356/303017",
"scraped_at": "2026-09-21T09:22:22.033307+00:00"
},
{
"url": "https://item.rakuten.co.jp/sawaicoffee-tea/ac-sale-1960/",
"sku": "sawaicoffee-tea:ac-sale-1960",
"title": "\先着300円OFFクーポン配布/ 送料無料 コーヒー コーヒー豆 コーヒー粉…",
"price": 2788.0,
"currency": "JPY",
"points": 50,
"point_rate": 2,
"rating": 4.7,
"review_count": 50175,
"shop_name": "澤井珈琲Beans&Leaf",
"category": "水・ソフトドリンク > コーヒー > コーヒー豆",
"genre_path": "/0/100316/100356/201402",
"scraped_at": "2026-09-21T09:22:22.033563+00:00"
}
]
```
Full sample: [`sample_output.json`](sample_output.json) · 9 rows, collected
2026-09-21 (listing mode, the keyword search from the Quick start).
`price` is in yen and `points` is what the purchase earns on top of it — see
the FAQ on [Rakuten points](#what-are-rakuten-points-in-the-output-and-how-do-they-change-the-real-price).
`genre_path` is Rakuten's genre ids; `category` is the same path in
Rakuten's own (Japanese) genre names.
## Requirements and access
| | |
|---|---|
| Runtime | Python 3.9 or newer (the offline suite runs on 3.9 and 3.13 in CI) |
| Engine | Playwright with its Chromium (recommended); Selenium or pyppeteer as alternatives, one per virtualenv |
| Coverage | rakuten.co.jp (Rakuten Ichiba, Japan, prices in JPY): keyword searches, genre listings and item pages. Merchant storefronts and `ranking.rakuten.co.jp` are not read — see [Coverage and limitations](#coverage-and-limitations) |
| Required services | **Nothing.** No key, no proxy, no account |
| Measured | 2026-11-17 at commit `fe0d03d`, from a datacentre address with no key, proxy or account (see [Verification](#verification)); first on 2026-09-21 |
| Optional: a proxy | For long runs. Rakuten answers a client going too fast with HTTP 503, which is a throttle, not a block; raise `--delay` first, then spread the run across exits. See [503 is a throttle, not a block](#503-is-a-throttle-not-a-block) |
| Optional: 2Captcha Scraping Browser API | Not needed on this site. What it adds: no local browser, a chosen exit country and persistent cookies per profile |
| Captcha solving | Not needed: no captcha is configured on the routes this scraper reads. See [There is no captcha on this site](#there-is-no-captcha-on-this-site) |
## Quick start
```bash
python3 -m venv .venv && . .venv/bin/activate
pip install -r requirements.txt -r requirements-playwright.txt
playwright install chromium
python3 playwright_scraper.py \
--url "https://search.rakuten.co.jp/search/mall/コーヒー/" --pages 3
```
That is the whole setup. No key, no proxy, no account, no `.env`.
It writes three files to the current directory (choose the prefix with
`--out`):
* `rakuten_products.json` and `rakuten_products.csv` — one row per product,
45 per page;
* `rakuten_products.meta.json` — the run's status (`complete` or not), stop
reason, failed pages and the site's own result cap.
If the run does not end with exit 0:
* **exit 3, refused** — remove anything you added to the client's identity (a
custom user agent, or `--fingerprint` together with `--cdp-endpoint`); see
[The one thing to know first](#the-one-thing-to-know-first).
* **exit 4, zero products** — the page was served but matched nothing; check
the keyword and any filters in the URL. Nothing is written over earlier
output.
* **exit 5 or 6, throttled or partial** — raise `--delay`; `pages_failed` in
the `.meta.json` names the pages that were not read. See
[503 is a throttle, not a block](#503-is-a-throttle-not-a-block).
All exit codes: [Exit codes](#exit-codes). Setup, `.env` and exit codes for
the whole family: [How to run the 2scraper scrapers](https://2captcha.com/h/how-to-run-2scraper-open-source-scrapers).
## Coverage and limitations
What the scraper reads, what it does not, and what the site itself caps. The
short version: listings and item pages, never storefronts or rankings; at
most 6,750 results per query; no was-price on listing rows; merchant text
stays in Japanese.
### Pass a listing or an item, not a shop
Three routes carry products and one looks as though it should:
| URL | Reads |
|---|---|
| `search.rakuten.co.jp/search/mall/{keyword}/` | ✅ `--mode listing` |
| `search.rakuten.co.jp/search/mall/-/{genreId}/` | ✅ `--mode listing` |
| `www.rakuten.co.jp/category/{genreId}/` | ✅ `--mode listing` |
| `item.rakuten.co.jp/{shop}/{code}/` | ✅ `--mode product` |
| `www.rakuten.co.jp/{shopCode}/` | ❌ refused, with the reason |
| `ranking.rakuten.co.jp/…` | ❌ not implemented |
A **merchant storefront** is refused rather than attempted because its
`__INITIAL_STATE__.state.data` is *empty* — it carries no product payload at
all, so reading it would need a second parser written against markup nobody
has captured. A mode that ships untested is worse than a mode that is absent.
`ranking.rakuten.co.jp` is a different application: no payload, one JSON-LD
`BreadcrumbList`, 80 product links in hand-rolled markup. It is also the one
route on this site that cares about the window — 403 to plain HTTP and to
headless Chromium (3 of 3 each), **200 to headful Chromium** (3 of 3). Not
unreachable, just not implemented.
### Complete is not exhaustive
Rakuten serves at most **6,750 results per query — 150 pages of 45 —
however many it matched**, and it states both numbers itself. One measured
query reported:
```
numFound: 3,053,682 subset: 6,750 pageSize: 45
```
So a full 150-page run of that query is genuinely `status: complete` — it
fetched everything the site will serve — and it is a **0.2% sample**. A
sidecar that said only "complete" would be lying by omission, so every run
records `total_results`, `reachable_max`, `page_size`, `capped_by_site` and
`pages_beyond_cap` beside the status, and the engine logs the percentage.
It also gives `diff_runs.py` a third meaning for `removed`: not delisted, not
un-fetched, but *outside this run's slice of a capped result set*.
To reach more, narrow the query with the site's own filters — genre, price
band, shop, tag — and run each slice.
### Traps that look like bugs
* **`rating` and `review_count` null on some rows.** Nobody has reviewed that
product. Rakuten writes that as `{score: 0, numReviews: 0}` rather than as
a null, on 28 of 405 rows, and a zero written through as a rating drags
every average a consumer computes — so both columns are nulled together,
keyed on the count.
* **`original_price` and `discount_pct` null on every listing row.** The
listing payload publishes no was-price of any kind: zero occurrences of
`doublePrice`, `listPrice`, `referencePrice` or `strike` across 405 rows.
The nearest thing it has is `sale`, on 32 of 405, which is a campaign
*window* with no prices in it — read as a discount it would invent one. A
verified was-price does exist on the detail page, gated on Rakuten's own
`doublePrice.referencePriceVerified` flag, which is the outcome of Japan's
double-pricing substantiation rule. Verified on 2 of 8 item pages
(¥10,398 → ¥5,199, and ¥3,476 → ¥1,738).
* **`in_stock` true on every row.** The search excludes sold-out products
rather than marking them — checked specifically on deep pages (80, 120,
149, 150 of a 142,000-hit genre), where stale rows would live: 0 sold out.
So a `false` on the listing route is *unproven*; the detail route reads the
page's own `availability` microdata, which is the source that can say
otherwise.
* **45 rows from a payload with 52 entries.** Seven were sponsored
placements. Rakuten injects paid slots into its own result list; they carry
a click-tracking redirect instead of a product URL, so there is nothing to
key a row on. The count varies between fetches of the same URL — two
engines seconds apart saw 45 and 52 — which is why `position` counts the
rows emitted rather than the payload's slots.
* **`--locale en` does not translate anything you care about.** `?lang=en`
comes back with `locale: "en"` in the payload and an English navigation
bar, and product names, shop names and tag values are merchant-authored
Japanese and come back **byte-identical**. `?lang=zh-tw` and `?lang=ko` are
accepted with HTTP 200 and silently fall back to `ja`.
* **`genre_rank` is not a position in your results.** It is the item's
standing in its genre's *ranking* listing, which this run never fetched.
Sibling repos in this family use published ranks to prove that cards went
missing; that arithmetic does not work here.
* **A readiness wait that times out costs nothing.** Every column comes out
of the payload, which is in the first response with no JavaScript run at
all — the same URL fetched by plain HTTP and by a real browser parses to
the same 45 rows with the same skus and prices.
* **Never wait on `networkidle`.** Rakuten's ad and tracking beacons keep
firing indefinitely — 46 fetches and 13 pings still going after load — so
that wait always runs to its full timeout. All three engines wait for
`domcontentloaded`.
More in [TROUBLESHOOTING.md](TROUBLESHOOTING.md).
## Examples
Step by step:
[How to scrape Rakuten](https://2captcha.com/h/how-to-scrape-rakuten) ·
the [Rakuten scraper page on 2scraper.com](https://2scraper.com/scrapers/rakuten-scraper/).
Each command is a complete run once the [Quick start](#quick-start) setup is
done.
```bash
# A keyword search, three pages, JSON and CSV
python3 playwright_scraper.py \
--url "https://search.rakuten.co.jp/search/mall/コーヒー/" --pages 3
# A genre listing, by Rakuten's own genre id
python3 playwright_scraper.py \
--url "https://www.rakuten.co.jp/category/100356/" --pages 5 --format csv
# One product: per-variant prices and a variant count (and a was-price while
# the shop's own sale is verified: see the note under "Measured")
python3 playwright_scraper.py --mode product \
--url "https://item.rakuten.co.jp/sawaicoffee-tea/solandluna/"
# The site's own filters are part of the URL and survive pagination
python3 playwright_scraper.py \
--url "https://search.rakuten.co.jp/search/mall/コーヒー/?min=1000&max=3000" \
--pages 3
# Spread a long run across exits (and see the note on 503 below). Put one
# proxy URL per line in the list; exits*.txt, proxies*.txt and proxylist*.txt
# are git-ignored, because that file holds logins and passwords.
python3 playwright_scraper.py \
--url "https://search.rakuten.co.jp/search/mall/コーヒー/" --pages 50 \
--proxy-file exits.txt --proxy-rotate per-page --delay 4
```
## Verification
Last checked on **2026-11-17** at commit `fe0d03d`. The scraper's code is
unchanged since `ab9babd`, where the full three-page Quick start was run on
2026-11-09.
| Date, commit | Check | Conditions | Result |
|---|---|---|---|
| 2026-11-17, `fe0d03d` | Quick start command with `--pages 1` (one page of 45; kept short on purpose) | a fresh virtualenv, Python 3.13, `requirements.txt` + `requirements-playwright.txt` (Playwright 1.65.2), headless Chromium, no `.env`, one datacentre address | **45 rows**, 45 unique `sku`, `status: complete`, exit 0; `price` and `currency` (`JPY`) on 45 of 45; same 42 columns in the same order as `sample_output.json` |
| 2026-11-17, `fe0d03d` | Site cap reported | the same run | `total_results` 3,233,987, `reachable_max` 6,750 — a complete run is a sample, as described [above](#complete-is-not-exhaustive) |
| 2026-11-09, `ab9babd` | Quick start, as written above (`--pages 3`) | a fresh virtualenv, Python 3.13, `requirements.txt` + `requirements-playwright.txt`, headless Chromium, no `.env`, one Hetzner datacentre address | **135 rows**, 135 unique `sku`, `status: complete`, exit 0; `price` and `currency` (`JPY`) on 135 of 135; `total_results` 3,164,855 |
| 2026-11-17, `fe0d03d` | Offline suite | `python3 smoke_test.py` and `python3 .github/ci_checks.py --all`, core requirements only | passed (engine groups skipped, as designed, without their libraries) |
| 2026-10-01 to 2026-11-16 | [Canary](https://github.com/2scraper/rakuten-scraper/actions/workflows/canary.yml) (daily, weekly since 2026-10-16) | a 3-page genre listing from a GitHub runner with no secrets | green on every run, the last one on 2026-11-16 at `fe0d03d` |
## The one thing to know first
**You do not need any of the paid products to read this site's listings, and
the reason is not what you would guess.**
Rakuten is fronted by Akamai, and what Akamai refuses here is not a
datacentre address — it is a client whose claimed identity and TLS
fingerprint disagree. Measured 2026-09-21 from one Hetzner datacentre address
in Helsinki, nothing else changed between the rows:
| Client | Result |
|---|---|
| `curl`, curl's own User-Agent | **200**, 919 KB, 45 products |
| `curl`, a Chrome User-Agent | **200**, **42-43 bytes**, `Reference #…` |
| headless Chromium (real UA, real TLS) | **200**, 855 KB, 45 products |
| headful Chromium | **200**, 969 KB, 45 products |
The two `curl` rows were repeated on 2026-10-29 from a Hetzner datacentre
address and gave the same answer (919,197 bytes and 45 parsed products; 42
bytes, the length varying with the reference id), which also replaced the
size this table first carried for the first row. The two Chromium rows are
from 2026-09-21 and were not repeated.
Read it by column: the address is identical in all four and the *consistency
of the client* decides everything. curl claiming to be curl is served; curl
claiming to be Chrome is refused. So the practical advice is the opposite of
most sites in this family — **if you are being refused, look for a disguise
you added before you buy an exit**:
* do not pass `--fingerprint` or a custom user agent together with
`--cdp-endpoint`, because the remote browser already brings its own
identity and stacking a second one manufactures exactly this mismatch;
* do not put a browser user agent on an HTTP client.
Two more things follow from that table, and both are easy to get wrong:
* **A refusal arrives under HTTP 200.** The status code is not the signal.
This scraper decides "was this served at all" from the reference-id shape
and from the page being built out of Rakuten's own `r10s.jp` asset hosts
(90–343 references on every served page, 0–1 on every refusal).
* **Headless is fine.** It is the default here. The browser even announces
itself — a tracking beacon from the headless run carried
`HeadlessChrome/153.0.8010.12` — and Akamai served it anyway.
The canary in this repository is that claim under test: a real three-page
scrape, weekly (Mondays), from a bare GitHub runner, with **no secrets**. If Rakuten ever
puts the listing routes behind something, the badge above goes red on the next weekly run: a blocked, partial or empty run fails the job rather than passing
with a warning.
## Install
One engine, not all three. They declare mutually unsatisfiable pins —
playwright and pyppeteer disagree on `pyee`, pyppeteer and selenium on
`urllib3` — so use a virtualenv per engine if you need more than one.
```bash
python3 -m venv .venv && . .venv/bin/activate
pip install -r requirements.txt -r requirements-playwright.txt
playwright install chromium # the recommended engine
```
```bash
pip install -r requirements.txt -r requirements-selenium.txt # or
pip install -r requirements.txt -r requirements-puppeteer.txt # or
```
`pip install .[playwright]` is equivalent.
pyppeteer downloads a Chromium of its own, and that build does not start on
every machine (seen on an Apple Silicon Mac, where it is an x86_64 build and
ends with `Browser closed unexpectedly`). Pass `--chromium-path` with a Chrome
or Chromium that you installed, e.g. `--chromium-path "/Applications/Google
Chrome.app/Contents/MacOS/Google Chrome"`. Without it the run now ends with
that advice and exit 5 rather than a traceback; the Apple Silicon case itself
was reported by an audit and was not reproduced here (Linux, a browser that
exits at once, gave the same error and the same handling).
## What comes out
42 columns. The first 18 are the family prefix, byte-identical and in the
same order as every other repo in this family, so one consumer reads a
rakuten run and a mediamarkt run with the same code.
```json
{
"source": "rakuten.co.jp",
"scraped_at": "2026-09-21T09:22:22+00:00",
"url": "https://item.rakuten.co.jp/ajinomoto/r4901111371248/",
"sku": "ajinomoto:r4901111371248",
"title": "【9/24まで P10倍!】味の素AGF ブレンディ スティックブラック100本…",
"brand": "味の素AGF",
"price": 2447.0,
"currency": "JPY",
"original_price": null,
"discount_pct": null,
"rating": 4.76,
"review_count": 475,
"in_stock": true,
"image_url": "https://thumbnail.image.rakuten.co.jp/@0_mall/ajinomoto/…",
"category": "水・ソフトドリンク > コーヒー > インスタントコーヒー",
"price_source": "state",
"page": 1,
"position": 1,
"variant_id": "r4901111371248",
"item_number": "10000338",
"price_max": 6975.0,
"has_price_range": true,
"subscription_price": 2324.0,
"points": 220,
"point_rate": 10,
"shipping_fee": 0.0,
"free_shipping": true,
"delivery_estimate": "最短9/23お届け",
"shop_name": "味の素グループ公式ショップ",
"shop_code": "ajinomoto",
"shop_id": 383220,
"shop_url": "https://www.rakuten.co.jp/ajinomoto/",
"genre_id": 303017,
"genre_path": "/0/100316/100356/303017",
"genre_rank": 2,
"is_super_deal": false,
"is_39shop": true,
"is_official_shop": true,
"variant_count": null,
"original_price_label": null,
"tags": {"ブランド": ["味の素AGF"], "単品重量": ["~ 99g"]},
"variants": null
}
```
`sample_output.json` and `sample_output.csv` are cut from a real run, not
written by hand.
Three columns are worth explaining because this site is unusual:
* **`sku` is `{shopCode}:{manageNumber}`** — Rakuten's own item code, exactly
as a detail page states it in ``. It is the same id on
both routes, so you can join a listing run to a product run on it.
Deliberately *not* the payload's `variantId`, which names the pre-selected
SKU inside the item and matches the URL's own code on only 39 of 180
measured rows.
* **`points` is half the price story.** Rakuten is a points marketplace: a
10× point campaign is effectively a 10% discount that never touches the
price column, so a monitor watching `price` alone would call a product
unchanged through an entire Super Sale. `diff_runs.py` tracks both.
* **`shop_*` is not run metadata.** This is a mall — 74 distinct merchants
across 405 measured rows — so the same physical article is sold by many
shops at many prices under many ids.
### Exit codes
| Code | Means |
|---|---|
| 0 | products written |
| 1 | crash |
| 2 | bad usage |
| 3 | blocked before parsing |
| 4 | served a page, zero products |
| 5 | the content was never obtained: a remote API error, a dead exit, a load that never completed, a page that stayed throttled after every wait |
| 6 | partial run |
A run that finds nothing **writes nothing** — `--allow-empty` is the opt-out
— so a failed run cannot replace last night's good output with `[]`. With
the opt-out the empty file IS written, but the exit code does not change: a
run that found nothing still exits 4 and its sidecar says `status: failed`.
`--out` may name a directory that does not exist yet: it is created before
the first request, and one that cannot be created is a usage error (exit 2)
instead of a traceback after the whole run.
A page that stays HTTP 503 after every wait is a **failed page**
(`stop_reason: throttled`, `pages_failed` names it): exit 6 if other pages
produced rows, 5 if none did. It is never read as a page with no products.
**All three files are replaced atomically.** Each is written to a temporary
file beside it and renamed over the old one, so a crash, a kill or a full
disk leaves the previous run's output exactly as it was rather than a
half-written file where a complete one had been. Measured before the fix: a
2,084-byte good `out.json` came back **0 bytes and invalid JSON** after an
interrupted write, because `open(path, "w")` truncates before writing a
single byte. That matters most for the sidecar, which is the file a consumer
branches on.
Every run also writes `.meta.json`, and on this site it carries the
site's own arithmetic as well as the status. See
[complete is not exhaustive](#complete-is-not-exhaustive).
**A CSV cell cannot be a formula.** A spreadsheet executes a cell beginning
`=`, `+`, `-` or `@`, and every string here was written by a merchant, not by
this code. Such a cell is prefixed with an apostrophe in the **CSV only** —
the JSON keeps the site's bytes exactly as served — and `csv_cells_escaped`
in the sidecar declares how many cells the two files therefore differ in.
Only strings are touched: escaping a number would turn `-5` into text and
break every sum written over the column.
This does not fire on today's data and is not claimed to. Measured
2026-09-30: **0 of 42,592 string cells across 2,662 stored rows** begin with
any of those characters. It is here because a Rakuten title routinely opens
with a discount badge, and `-16%OFF ...` is one merchant away from being a
leading minus.
## Where the data actually is
**The JSON-LD on a listing page is a trap.** A search page has exactly one
`application/ld+json` block, it is an `ItemList`, and it holds **ten** items
whose every url carries `?scid=seo-carousel-search` — it is the SEO
recommendation carousel — while the page itself holds **45** products. A
JSON-LD-primary parser returns ten rows of the wrong products and reports
success.
The real sources:
| Route | Source |
|---|---|
| listing | `window.__INITIAL_STATE__.state.data.ichibaSearch` — `items` (45/page) + `pagination` |
| product | `