# People Also Ask — Google PAA Scraper for Python

> Extract **Google "People Also Ask" questions**, answers, source links, and related metadata at scale. Built with Playwright for reliable, captcha-resistant scraping that works with Google's modern JavaScript-rendered SERPs.

[![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Playwright](https://img.shields.io/badge/powered%20by-Playwright-green.svg)](https://playwright.dev/)

## Why This Scraper Exists

Most existing Google PAA scrapers are broken. Here's why:

- **HTTP-only scrapers fail** — Google now requires JavaScript to render search results. A simple `requests.get()` returns an empty shell, not the PAA box.
- **Selenium scrapers break** — Google's anti-bot systems detect ChromeDriver easily, and internal `jsname` selectors change without warning.
- **No answer extraction** — Most scrapers collect question text only, missing the rich answer content (AI Overview, featured snippets, source links) that appears when you click a question.

This scraper solves all three problems:

1. **Playwright with real Chrome** (`channel="chrome"`) renders the full SERP including the PAA box.
2. **Persistent browser profile** retains cookies/history, dramatically reducing captcha triggers.
3. **Click-to-expand** extracts answers, source links, and discovers new questions via BFS recursion.
4. **Resilient selectors** with multiple fallback strategies — uses `data-q` attribute (stable) as primary key, not fragile `jsname` tokens.
5. **Captcha detection** with clear error messages instead of silently parsing garbage.

## Features

- **Questions + Answers** — Extracts not just PAA questions, but the full answer content (AI Overview, featured snippets) and source links.
- **BFS Recursion** — Clicks questions to expand them, discovering 2–3 new questions per click. Depth-controlled graph traversal.
- **Multi-language & Geo** — Supports `hl` (language) and `gl` (country) parameters for localized results.
- **SQLite Caching** — Optional persistent cache to avoid re-scraping the same queries.
- **Anti-Bot Defense** — Persistent Chrome profile, User-Agent rotation, stealth scripts, captcha detection.
- **CLI + Library** — Use as a command-line tool or import as a Python library.
- **JSON & CSV Output** — Structured output with question, answer, sources, depth, and related questions.

## Quick Start

### Installation

```bash
pip install people-also-ask
playwright install chromium
```

> **Note:** This scraper uses `channel="chrome"` by default, which requires Google Chrome to be installed on your system. If you prefer Chromium, pass `channel=None` to the `PaaScraper` constructor.

### CLI Usage

```bash
# Basic — get questions and answers
paa "climate change"

# Limit questions and depth
paa "python programming" --max-questions 50 --max-depth 3

# Questions only (faster)
paa "netflix stock" --no-answers

# Localized results
paa "melhor linguagem de programação" --hl pt --gl br

# Save to file
paa "best laptop 2024" --output results.json
paa "best laptop 2024" --output results.csv --format csv

# Headless mode
paa "machine learning" --headless --max-questions 20
```

### Library Usage

```python
from paa import get_questions, get_answers, scrape

# Get questions only
questions = get_questions("climate change", max_questions=20)
# → ["What is climate change?", "How hot will Earth be in 2050?", ...]

# Get questions with answers and metadata
results = get_answers("climate change", max_questions=10, max_depth=2)
for r in results:
    print(f"Q: {r.question}")
    print(f"A: {r.answer[:100]}...")
    print(f"Sources: {len(r.sources)}")
    print(f"Related: {r.related_questions}")
    print()

# Full control with output file
data = scrape(
    "renewable energy",
    max_questions=30,
    with_answers=True,
    hl="en",
    gl="us",
    max_depth=2,
    output="paa_results.json",
)
```

### Async API (for high-volume scraping)

```python
import asyncio
from paa.scraper import PaaScraper

async def main():
    scraper = PaaScraper(hl="en", gl="us", headless=True)

    # Single query
    results = await scraper.scrape(
        "artificial intelligence",
        max_questions=50,
        with_answers=True,
        max_depth=3,
    )

    # Multiple queries (reuses browser session)
    all_results = await scraper.scrape_many(
        ["python", "javascript", "rust"],
        max_questions=20,
        with_answers=False,
    )

    scraper.close()

asyncio.run(main())
```

## Output Format

Each result is a dictionary with the following structure:

```json
{
  "question": "How hot will Earth be in 2050?",
  "parent_query": "climate change",
  "depth": 0,
  "answer": "By 2050, Earth's global average temperature is projected to be...",
  "sources": [
    {
      "title": "Climate Predictions - NOAA",
      "url": "https://www.noaa.gov/...",
      "displayed_url": "noaa.gov"
    }
  ],
  "related_questions": [
    "What will no longer exist in 2050?",
    "Why is 2035 the year of no return?"
  ],
  "has_answer": true
}
```

## How It Works

```
Query "climate change"
         │
         ▼
┌─────────────────────┐
│  Playwright (Chrome) │  ← Persistent profile avoids captcha
│  Render Google SERP  │
└────────┬────────────┘
         │
         ▼
┌─────────────────────┐
│  Parse PAA Section   │  ← div.cUnQKe / div[jsname="bq0EGf"]
│  Extract questions   │  ← div.related-question-pair[data-q]
└────────┬────────────┘
         │
         ▼
┌─────────────────────┐
│  BFS Expansion       │  ← Click each question → discover new ones
│  (depth-limited)     │  ← 2-3 new questions per click
└────────┬────────────┘
         │
         ▼
┌─────────────────────┐
│  Extract Answers     │  ← AI Overview / featured snippet
│  + Source Links      │  ← <a href> + <cite> within expanded pair
└────────┬────────────┘
         │
         ▼
  Results (JSON / CSV)
```

### Selector Strategy

Google changes internal `jsname` attributes frequently. This scraper uses a **layered selector strategy**:

| Element | Primary Selector | Fallback |
|---------|-----------------|----------|
| PAA section | `div.cUnQKe` | `div[jsname="bq0EGf"]`, heading text search |
| Question pair | `div.related-question-pair` | `div[jsname="xGrnfe"]` |
| Question text | `data-q` attribute | `span[jsname="r4nke"]`, `span.CSkcDe` |
| Click target | `div[jsname="tJHJj"]` | `[role="button"]` within pair |
| Answer container | `div[jsname="YrZdPb"]` | `div.roMIYb` |

The `data-q` attribute is the most stable — it contains the question text directly and doesn't depend on Google's internal naming.

## Comparison with Other PAA Scrapers

| Feature | This Scraper | `sundios/people-also-ask` | `dataxquare/people-also-ask` |
|---------|:-----------:|:-------------------------:|:----------------------------:|
| Engine | Playwright + Chrome | Selenium + undetected-chromedriver | HTTP (requests + BeautifulSoup) |
| JS rendering | ✅ Full | ✅ Full | ❌ Empty shell |
| Answer extraction | ✅ AI Overview + sources | ❌ Questions only | ⚠️ Featured snippets (broken) |
| BFS recursion | ✅ Depth-controlled | ❌ Linear only | ✅ But limited by HTTP approach |
| Captcha handling | ✅ Detection + persistent profile | ⚠️ No detection | ❌ No detection |
| Selector resilience | ✅ Multi-strategy + `data-q` | ❌ Fragile `jsname` | ❌ Broken `related-question-pair` class |
| Geo/localization | ✅ `hl` + `gl` | ⚠️ `hl` only | ✅ `hl` + `gl` + `uule` |
| Caching | ✅ SQLite | ❌ | ❌ |
| CLI | ✅ Rich tables | ❌ Interactive prompts | ❌ API only |
| Output formats | JSON + CSV | CSV | JSON |
| Anti-bot | ✅ Persistent profile + UA rotation | ⚠️ undetected-chromedriver | ⚠️ UA rotation + proxy |

## Configuration

### PaaScraper Parameters

| Parameter | Default | Description |
|-----------|---------|-------------|
| `hl` | `"en"` | Language code (`en`, `pt`, `es`, `fr`, `de`, ...) |
| `gl` | `"us"` | Country code (`us`, `br`, `es`, `uk`, ...) |
| `headless` | `False` | Run browser without visible window |
| `channel` | `"chrome"` | Browser channel (`"chrome"`, `"msedge"`, or `None` for Chromium) |
| `click_delay` | `2.0` | Seconds to wait after clicking a question |
| `page_delay` | `3.0` | Seconds to wait for page load |
| `timeout` | `30000` | Playwright navigation timeout (ms) |
| `cache_db` | `None` | Path to SQLite cache database |
| `profile_dir` | `{tempdir}/paa_chrome_profile` | Chrome profile directory (cookies/history) |

### Avoiding Captchas

Google may serve captchas if you scrape too aggressively. To minimize this:

1. **Use the persistent profile** (default) — cookies and history build trust over time.
2. **Increase `click_delay`** — slower scraping looks more human-like.
3. **Use `headless=False`** — visible browsers are less suspicious to Google.
4. **Limit `max_depth`** — depth 0–2 is usually safe; higher values trigger more requests.
5. **Add delays between queries** — the `scrape_many` method does this automatically.

## Use Cases

- **SEO Content Research** — Discover what questions people ask about your target keywords.
- **FAQ Generation** — Build FAQ pages from real Google PAA data.
- **Content Gap Analysis** — Find questions your competitors aren't answering.
- **Voice Search Optimization** — PAA questions mirror natural language voice queries.
- **Market Research** — Understand user intent and common concerns in any niche.
- **AI Training Data** — Collect question-answer pairs for model training.

## Limitations

- Google's HTML structure changes periodically. If the scraper stops finding questions, the selectors may need updating.
- Scraping rate is limited by Google's anti-bot systems. This is a browser-based scraper, not a high-speed API.
- PAA questions are personalized and may vary by location, language, and search history.

## Contributing

Contributions are welcome! If you find a bug or want to add a feature:

1. Fork the repository
2. Create a feature branch (`git checkout -b feature/amazing-feature`)
3. Commit your changes (`git commit -m 'Add amazing feature'`)
4. Push to the branch (`git push origin feature/amazing-feature`)
5. Open a Pull Request

## License

MIT License — see [LICENSE](LICENSE) for details.

## Disclaimer

This scraper is for educational and research purposes only. Scraping Google Search results may violate Google's Terms of Service. Use responsibly and consider using official APIs (e.g., Google Custom Search API, SerpAPI) for production applications.