---
name: marketing-reddit
description: "Mine Reddit posts, threads, comments, and question clusters through a bot-challenge-resistant browser. Use for subreddit research, content gaps, unanswered questions, post data, or Reddit 403 blocks."
---

# marketing-reddit
Reddit hard-blocks the public `.json` API for datacenter/cloud IPs and serves a
JS bot-challenge to anything automated. The decisive signal is the
**`HeadlessChrome` token in the User-Agent** — override it to a clean
`Chrome/...` UA and headless Chrome passes the challenge. The same-origin in-page
`fetch()` then carries the clearance cookie and returns real JSON.

This tool drives `agent-browser` (headless) with a clean UA — optionally through a
residential proxy — clears the challenge once per run, then fetches `.json` via
in-page `fetch()`, metering real wire bytes (proxy bandwidth) and cost. Question
detection + clustering are ported in-tool (no Python dependency).

Written in TypeScript, run with **bun** (runs `.ts` directly, zero install).

## Why headless works here

| Signal | Headless default | What we send | Verdict |
|--------|------------------|--------------|---------|
| User-Agent | `…HeadlessChrome/…` | `…Chrome/…` (clean) | **decisive** — clean UA passes |
| navigator.webdriver | true | true (unchanged) | not enough alone to block |

`--proxy`/`--user-agent` apply at agent-browser **daemon launch**, so the tool
runs `agent-browser close --all` at the start of a run to guarantee they take
effect (this also closes other agent-browser sessions).

## Setup

- `bun` installed (https://bun.sh) and `agent-browser` (`npm i -g agent-browser`).
- **Proxy is optional.** From a clean/residential IP it works direct. From a
  flagged datacenter/cloud IP (e.g. a CI box) you need a residential proxy.

```bash
# Store a proxy in machine-local config (chmod 600, OUTSIDE the skill repo — repo is public)
bun <this-skill-dir>/cli.ts setup --proxy "http://user:pass@host:port"

# Verify everything needed to run
bun <this-skill-dir>/cli.ts doctor
```

Proxy resolution order: `--proxy` flag → `REDDIT_PROXY` env → `~/.config/reddit-miner/config.json`.
The proxy secret is never read from or written to the skill repo. Use `--no-proxy` to force direct.

## Setup flow — guide the user through this

When the user first asks to set this up, don't dump manual steps — drive it. Run
`doctor`, then fix each FAIL yourself (ask before any global install or before
handling a proxy secret). `<cli>` below is `bun <this-skill-dir>/cli.ts`.

1. **Run `<cli> doctor` first.** It reports each requirement as PASS/FAIL with the
   exact fix. Work down the failures in order.

2. **`bun runtime` FAIL** → bun isn't installed. Offer to run:
   ```bash
   curl -fsSL https://bun.sh/install | bash
   ```

3. **`agent-browser installed` FAIL** → offer to run:
   ```bash
   npm i -g agent-browser
   ```

4. **`browser engine (Chrome) reachable` FAIL** → agent-browser has no browser to
   drive. Offer to run (downloads a Chromium it controls; existing Chrome/Brave is
   auto-detected too):
   ```bash
   agent-browser install
   ```

5. **`proxy credential resolved` is INFO, not a failure.** Ask the user: do they
   have a residential/ISP proxy? Reddit blocks datacenter IPs, so a cloud/CI box
   needs one; a clean home IP often works direct.
   - If yes, take the proxy URL and store it for them — never paste it into the repo:
     ```bash
     <cli> setup --proxy "http://user:pass@host:port"
     ```
   - If no, tell them it'll run direct and only works from a clean IP. They can add
     a proxy later with `setup`.

6. **Re-run `<cli> doctor` until it prints `RESULT: READY`.** Don't proceed past a
   FAIL — a failing engine or a flagged IP means every mine returns nothing.

7. **Smoke test** once READY:
   ```bash
   <cli> mine --subreddit <their-subreddit> --threads 5
   ```
   Confirm it returns clustered questions and a small bandwidth/cost line.

For convenience, suggest the user alias the CLI:
`alias reddit-miner='bun <this-skill-dir>/cli.ts'`.

## Usage

```bash
# Busiest posts (title, selftext, score, comments, flair, author)
bun <this-skill-dir>/cli.ts posts --subreddit Retatrutide --sort top --time month --limit 30

# A single thread's full comment tree (raw Reddit JSON)
bun <this-skill-dir>/cli.ts thread --url https://www.reddit.com/r/Retatrutide/comments/<id>/<slug>/

# Mine questions from POSTS + COMMENTS, mark answered/unanswered, cluster the gaps
bun <this-skill-dir>/cli.ts mine --subreddit Retatrutide --time month --threads 30 \
  --topic-keywords "reta|retatrutide|dose|side effect"
```

`--sort`: `top|hot|new|rising|controversial` (`--time` applies to `top`/`controversial`).
`--keep-open`: leave the browser session running for inspection.
`--no-proxy`: force a direct connection even if a proxy is configured.
`--no-rotate`: keep the proxy's stored `sessid` instead of minting a fresh one (see below).

## Sticky-session auto-rotation + exit-IP failover

A sticky exit IP is required so the clearance cookie stays valid for every `.json`
fetch in the run. Two vendor shapes:

| Vendor | Sticky shape | What we mint |
|---|---|---|
| **Oxylabs** | `sessid-…` in the URL/userinfo | fresh `sessid-<id>` per run |
| **DataImpulse sticky** (`gw.dataimpulse.com:1000X`, user `login__cr.us`) | `;sessid.<id>` on the username | inject/refresh `;sessid.<id>` per run |

Without a sessid, DataImpulse pins **one IP per sticky port** — a flagged exit then
fails every morning until someone hand-rotates. The tool now:

1. **Mints a fresh sessid at process start** (stable within the run).
2. **On challenge fail, rotates immediately** — new sessid → new residential IP →
   kill the agent-browser daemon (so the new `--proxy` takes effect) → retry.
   Default **5 exit IPs** (`REDDIT_MAX_EXIT_IPS`); 2 page loads per exit.

Stored config is never mutated. `--no-rotate` keeps the configured URL as-is (no
mint at start; clear still rotates on fail if the vendor shape is known).

## Concurrent callers

The tool drives one agent-browser daemon per session name (default `reddit-miner`)
and kills its own session's daemon whenever the proxy rotates. Two callers sharing
the default session interfere: the second caller's `--proxy`/`--user-agent` are
silently ignored ("daemon already running"), and each caller's daemon kill takes
down the other's in-flight challenge run — which then looks exactly like a flagged
proxy pool. When callers can overlap (parallel cron loops), give each its own
session: `REDDIT_MINER_SESSION=<caller-name>`. Serializing callers with a lock
works too, but per-session isolation is the fix.

## Output

- `posts` → `{subreddit, sort, posts: [...], bandwidth}`
- `thread` → `{thread: <raw Reddit listing JSON>, bandwidth}`
- `mine` → `{threads_scanned, posts_considered, questions, n_questions, n_unanswered, clusters_unanswered, top_questions, bandwidth}`
  - `questions[].source` is `reddit` (comment) or `reddit-post` (selftext) — "more than just comments".

Every run also prints a one-line bandwidth/cost summary to stderr.

## Bandwidth & cost

`bandwidth` reports real wire bytes (`transferSize`, post-gzip, incl. headers) — what
a metered proxy bills. Set `REDDIT_PROXY_USD_PER_GB` (default 8) for cost estimates.
Reference: a 30-thread `mine` ≈ **1 MB wire ≈ $0.008** at $8/GB (~955 runs per GB).

## Failure modes

- `could not clear Reddit JS challenge after N exit IP(s)` → the tool already
  rotated through `REDDIT_MAX_EXIT_IPS` sticky sessions and every one was flagged
  (or the vendor shape is unknown so it couldn't rotate). Check the proxy balance /
  country filter, or bump a dead DataImpulse sticky **port** (`:10001`→`:10002` —
  ports die independently of sessid rotation).
- `fetchJson failed … http 403 / blocked` → challenge expired mid-run; the tool
  retries the fetch. A fresh process re-clears (and re-mints a sessid).