# Applied AI PM Evaluation Framework

[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Python](https://img.shields.io/badge/Python-3.7%2B-blue.svg)](https://www.python.org/)
[![Platform](https://img.shields.io/badge/Platform-macOS%20%7C%20Linux%20%7C%20Windows-green.svg)](#installation)

**Analyze Applied AI PM resumes against the 2026 6-pillar framework** - Get instant, consistent evaluations using OpenAI GPT-5, Anthropic Claude Sonnet 4.5, or Google Gemini 2.5 Pro.

Built for hiring teams evaluating AI PM candidates, recruiters standardizing their process, and **especially for people from non-traditional backgrounds** looking to transition into AI Product Management roles.

---

## 🎯 Why This Exists

> "In disaster relief, you don't map safe routes and hide them. You share them."

[This conversation](https://www.youtube.com/watch?v=rcz39Y0qFxw) between [Aakash Gupta](https://x.com/aakashg0) and [Jaclyn Konzelmann](https://x.com/jacalulu) (Google's Director of AI Product) truly inspired this project. Jaclyn's AI PM evaluation framework was so clear and rigorous that it immediately became obvious: **this shouldn't be a private checklist**.

This tool exists to give **everyone**—especially people from non-traditional backgrounds—an actionable guide on:
- What to build to demonstrate AI PM capabilities
- Where to focus your learning and projects
- How to position yourself for AI PM roles
- How hiring teams actually evaluate candidates

**For Job Seekers**: Use this to self-assess, identify gaps, and build projects that matter.
**For Hiring Teams**: Use this to standardize evaluations and find builders, not just coordinators.

Built by [@abe238](https://x.com/abe238) | Full framework at [aipmframework.com](https://aipmframework.com)

---

## ✨ Features

- 🎯 **6-Pillar Framework** - Evaluates Technical Skills, Product Thinking, AI/ML Knowledge, Communication, Strategic Thinking, and Execution
- 🤖 **Latest AI Models** - GPT-5, Claude Sonnet 4.5, Gemini 2.5 Pro (with cheaper alternatives)
- 🔄 **Multiple Model Options** - Choose from 8 different models across 3 providers
- 📄 **Multiple File Formats** - Supports PDF, DOC, and DOCX resume files
- 📊 **Detailed Scoring** - 0-10 scores per pillar with evidence and level assessment
- 📝 **Beautiful Reports** - Generates markdown and HTML outputs with clean design
- ⚡ **Fast Analysis** - Get results in 30 seconds vs 10+ minutes manual review
- 💰 **Cost-Effective** - $0.10-0.60 per resume depending on model choice
- 🎨 **Professional Design** - HTML reports styled with Stripe/Tailwind-inspired CSS

---

## 📦 Quick Start

### Prerequisites

- **Python 3.7+** installed on your computer
- **Pandoc** (optional, for .doc/.docx support) - Install with:
  - macOS: `brew install pandoc`
  - Linux: `sudo apt-get install pandoc`
  - Windows: Download from [pandoc.org](https://pandoc.org/installing.html)
- **API key** from ONE of these providers:
  - [OpenAI](https://platform.openai.com/api-keys) (GPT-5, GPT-5 Mini, GPT-4o)
  - [Anthropic](https://console.anthropic.com/settings/keys) (Claude Sonnet 4.5, Haiku 4.5, Opus 4.1)
  - [Google](https://aistudio.google.com/app/apikey) (Gemini 2.5 Pro, Gemini 2.5 Flash)

### Installation & Setup

**1. Download or clone this repository:**
```bash
git clone https://github.com/abe238/aipm-resume-analyzer.git
cd aipm-resume-analyzer
```

**2. Install dependencies:**

**macOS/Linux:**
```bash
./scripts/install.sh
```

**Windows:**
```cmd
pip install -r requirements.txt
```

**3. ⚠️ REQUIRED: Create your .env file with API key**

**IMPORTANT**: The `.env` file is **hidden** by default. Follow these steps carefully:

**Option A: Copy from example (recommended)**
```bash
# Copy the example file to create your .env
cp examples/example.env .env

# Then edit .env and add your API key
nano .env  # or use any text editor
```

**Option B: Let the tool create it**
```bash
# Run the analyzer once - it will create .env automatically
./analyze resume.pdf

# You'll see instructions showing where the .env file was created
# Then edit .env and add your API key
```

**📁 Where is the .env file?**
- **Exact location**: `aipm-resume-analyzer/.env` (in the same folder as the `analyze` script)
- **Full path example**: `/Users/yourname/aipm-resume-analyzer/.env`

**🔍 Can't see the .env file?**

The `.env` file starts with a dot (.), making it hidden by default:

- **macOS Finder**: Press `Cmd + Shift + .` to show hidden files
- **macOS Terminal**: Use `ls -la` instead of `ls`
- **Windows Explorer**: Go to View → Show → Hidden items
- **Windows Command Prompt**: Use `dir /a` instead of `dir`
- **Linux Terminal**: Use `ls -la` instead of `ls`

**✏️ Editing the .env file:**

Open `.env` in any text editor and add your API key:

```bash
# For OpenAI (example)
OPENAI_API_KEY=sk-proj-your-actual-key-here

# OR for Anthropic
ANTHROPIC_API_KEY=sk-ant-your-actual-key-here

# OR for Google
GOOGLE_API_KEY=your-actual-key-here

# Default provider
DEFAULT_PROVIDER=openai
```

**See `examples/example.env` for a complete template with all options.**

**4. You're ready!** Run your first analysis:

---

## 🔑 API Setup (First Time Only)

### Step 1: Get an API Key

Choose **ONE** provider and get your API key:

#### Option A: OpenAI (GPT-5) - Most Advanced Reasoning
1. Go to https://platform.openai.com/api-keys
2. Click "+ Create new secret key"
3. Name it "AI PM Resume Analyzer"
4. Copy the key (starts with `sk-...`)
5. **Models**: GPT-5 (default), GPT-5 Mini, GPT-4o
6. **Cost**: $0.15-0.50 per resume

#### Option B: Anthropic (Claude Sonnet 4.5) - Best for Complex Analysis
1. Go to https://console.anthropic.com/settings/keys
2. Click "Create Key"
3. Name it "Resume Analyzer"
4. Copy the key (starts with `sk-ant-...`)
5. **Models**: Claude Sonnet 4.5 (default), Claude Haiku 4.5, Claude Opus 4.1
6. **Cost**: $0.20-0.60 per resume

#### Option C: Google (Gemini 2.5 Pro) - Advanced Thinking
1. Go to https://aistudio.google.com/app/apikey
2. Click "Create API Key"
3. Select or create a project
4. Copy the key
5. **Models**: Gemini 2.5 Pro (default), Gemini 2.5 Flash
6. **Cost**: $0.10-0.30 per resume (often free tier available)

### Step 2: Run the Analyzer (It Will Create .env File)

```bash
./bin/analyze resume.pdf
```

The tool will create a `.env` file and show you where to add your API key.

### Step 3: Add Your API Key

Open the `.env` file (created automatically) and add your key:

```bash
# For OpenAI
OPENAI_API_KEY=sk-your-key-here

# OR for Anthropic
ANTHROPIC_API_KEY=sk-ant-your-key-here

# OR for Google
GOOGLE_API_KEY=your-key-here

# Default provider (optional)
DEFAULT_PROVIDER=openai
```

Save the file. **Done!** You're ready to analyze resumes.

---

## 🚀 Usage

**Supported file formats:** `.pdf`, `.doc`, `.docx`

### Basic Analysis

```bash
# Analyze resume (uses default provider from .env)
./bin/analyze resume.pdf
# Or with Word documents:
./bin/analyze resume.docx
./bin/analyze resume.doc
```

Output files will be created in `./output/`:
- `Candidate_Name_20241025_143022.md` (Markdown report)
- `Candidate_Name_20241025_143022.html` (HTML report)
- `Candidate_Name_20241025_143022.json` (Raw JSON data)

### Choose Specific Provider

```bash
# Use Claude
./bin/analyze resume.pdf --provider anthropic

# Use Gemini
./bin/analyze resume.pdf --provider google

# Use OpenAI (default)
./bin/analyze resume.pdf --provider openai
```

### Choose Specific Model

Each provider has multiple models with different speed/cost tradeoffs:

```bash
# List all available models
./bin/analyze --list-models

# OpenAI models
./bin/analyze resume.pdf --provider openai --model gpt-5          # Best reasoning (default)
./bin/analyze resume.pdf --provider openai --model gpt-5-mini     # Faster, cheaper
./bin/analyze resume.pdf --provider openai --model gpt-4o         # Budget option

# Anthropic models
./bin/analyze resume.pdf --provider anthropic --model claude-sonnet-4-5-20250929  # Best (default)
./bin/analyze resume.pdf --provider anthropic --model claude-haiku-4-5            # Fast & cheap
./bin/analyze resume.pdf --provider anthropic --model claude-opus-4-1             # Most capable

# Google models
./bin/analyze resume.pdf --provider google --model gemini-2.5-pro    # Advanced (default)
./bin/analyze resume.pdf --provider google --model gemini-2.5-flash  # Fast & affordable
```

### Custom Output Location

```bash
# Save to custom directory
./bin/analyze resume.pdf --output ./reports/

# Different formats
./bin/analyze resume.pdf --format markdown  # MD only
./bin/analyze resume.pdf --format html      # HTML only
./bin/analyze resume.pdf --format both      # Both (default)
```

### Deep Analysis Mode (Maximum Feedback)

For the most comprehensive analysis, use `--deep-analysis` to run ALL available providers and get aggregated insights:

```bash
# Run analysis with all providers (requires API keys for all)
./bin/analyze resume.pdf --deep-analysis
```

**What Deep Analysis Does:**
- ✅ Runs analysis with **all providers** you have API keys for (GPT-5, Claude Sonnet 4.5, Gemini 2.5 Pro)
- ✅ Generates **consensus scores** showing agreement/disagreement across models
- ✅ Aggregates **all strengths and concerns** from each provider
- ✅ Shows **detailed comparison** table with min/max/avg scores per pillar
- ✅ Provides **multiple perspectives** on the same candidate

**When to Use:**
- Making final hiring decisions
- Screening senior/principal candidates
- When you want maximum confidence in the evaluation
- To identify areas where models agree (high confidence) vs. disagree (needs human judgment)

**Cost:** $0.45-$1.50 per resume (runs 3 analyses instead of 1)

**Example Output:**
```
Consensus Score: 52.3/60

Individual Scores:
- GPT-5: 53/60 - Strong Screen
- Claude Sonnet 4.5: 48/60 - Strong Screen
- Gemini 2.5 Pro: 56/60 - Strong Screen

Pillar Scores (showing agreement/disagreement):
Technical Skills: 8.0/10 (range: 7-9)
Communication: 9.3/10 (range: 9-10)  ← High consensus
AI/ML Knowledge: 7.7/10 (range: 6-9)  ← More variation
```

---

## 📊 Understanding the Output

### Scoring System

Each resume gets scored across **6 pillars**:

| Pillar | Weight | What It Measures |
|--------|--------|------------------|
| **Technical Skills** | 20% | Engineering background, coding, technical decision-making |
| **Product Thinking** | 25% | User empathy, prioritization, product strategy, metrics |
| **AI/ML Knowledge** | 20% | ML fundamentals, AI projects, responsible AI awareness |
| **Communication** | 10% | Writing quality, stakeholder management, presentations |
| **Strategic Thinking** | 15% | Vision setting, market analysis, long-term planning |
| **Execution** | 10% | Shipping track record, project management, results |

### Score Ranges

| Total Score | Decision | Action |
|-------------|----------|--------|
| **8.0 - 10.0** | **Strong Screen** | Top candidate - prioritize interview |
| **6.5 - 7.9** | **Screen** | Solid candidate - invite to interview |
| **5.0 - 6.4** | **Maybe** | Borderline - use additional criteria |
| **Below 5.0** | **No Screen** | Does not meet bar |

### Sample Output

**Markdown Report** (`.md` file):
```markdown
# AI PM Resume Analysis: John Smith

**Total Score**: 42/60 (70/100 weighted)
**Decision**: **Screen**

## Top 3 Strengths
1. Strong product management background with 5 years experience
2. Excellent communication skills - resume is clear and well-organized
3. Proven execution track record with quantified outcomes

## Top 3 Concerns
1. Limited technical background - no engineering experience
2. No AI/ML projects or education
3. Strategic thinking appears tactical (short-term focused)

[... detailed pillar-by-pillar analysis ...]
```

**HTML Report** (`.html` file):
- Beautiful, print-friendly design
- Color-coded scores and decision badges
- Progress bars for each pillar
- Professional styling inspired by Stripe Docs + Tailwind CSS

---

## 🛠️ Troubleshooting

### "Module not found" Errors

```bash
# Reinstall dependencies
pip3 install -r requirements.txt
```

### "API key not found"

**Problem**: The tool can't find your API key.

**Solutions**:

1. **Check `.env` file exists**:
   ```bash
   # Show all files including hidden ones
   ls -la .env

   # Should show: -rw-r--r--  1 yourname  staff  XXX Oct 25 XX:XX .env
   ```

2. **Verify `.env` file location**:
   - Must be in the **root directory** of the project
   - Same folder as the `analyze` script
   - NOT in `examples/`, `bin/`, or any subfolder

3. **Check file contents**:
   ```bash
   # View the .env file
   cat .env

   # Make sure you see your actual API key, not just empty values
   ```

4. **Common mistakes**:
   - ❌ Extra spaces: `OPENAI_API_KEY = sk-...` (wrong)
   - ✅ No spaces: `OPENAI_API_KEY=sk-...` (correct)
   - ❌ Quotes around key: `OPENAI_API_KEY="sk-..."` (wrong)
   - ✅ No quotes: `OPENAI_API_KEY=sk-...` (correct)
   - ❌ Wrong filename: `.env.txt` or `env` (wrong)
   - ✅ Exact name: `.env` (correct)

5. **Start fresh**:
   ```bash
   # Delete old .env and copy from example
   rm .env
   cp examples/example.env .env
   nano .env  # Add your key
   ```

### "Resume file not found"

```bash
# Use full path to resume
./bin/analyze /full/path/to/resume.pdf

# Or move resume to current directory
cp ~/Downloads/resume.pdf ./
./bin/analyze resume.pdf
```

### "Unsupported file format" Error

- Ensure file extension is .pdf, .doc, or .docx
- For .doc/.docx files, make sure pandoc is installed (see Prerequisites)
- Try re-exporting from the original application if file appears corrupted
- Some password-protected files may not work - remove protection first

### Python Version Issues

```bash
# Check Python version (need 3.7+)
python3 --version

# If too old, install newer Python:
# macOS: brew install python3
# Linux: sudo apt install python3.9
```

### API Errors

**"Insufficient credits"**: Add credits to your AI provider account
- OpenAI: https://platform.openai.com/account/billing
- Anthropic: https://console.anthropic.com/settings/billing
- Google: Usually has generous free tier

**"Rate limit exceeded"**: Wait 60 seconds and try again (or upgrade API plan)

---

## 💡 Pro Tips

### Cost Optimization

**Budget Options** (Best for high-volume screening):
- **Gemini 2.5 Flash**: $0.10-0.15/resume - Fast and affordable
- **Claude Haiku 4.5**: $0.15-0.25/resume - Fast with great quality
- **GPT-5 Mini**: $0.20-0.30/resume - Faster GPT-5 variant
- **GPT-4o**: $0.15-0.25/resume - Budget-friendly OpenAI option

**Premium Options** (Best for final round evaluations):
- **GPT-5**: $0.40-0.50/resume - Best reasoning capabilities
- **Claude Sonnet 4.5**: $0.45-0.60/resume - Best for complex analysis
- **Gemini 2.5 Pro**: $0.25-0.30/resume - Advanced thinking model
- **Claude Opus 4.1**: $0.50-0.70/resume - Most capable (highest cost)

**Recommendation**: Use Gemini Flash or Claude Haiku for initial screening (100+ resumes), then use GPT-5 or Claude Sonnet for top candidates in final rounds.

### Batch Processing

```bash
# Analyze multiple resumes
for resume in resumes/*.pdf; do
  ./bin/analyze "$resume" --output ./batch_reports/
done
```

### Integration with ATS

```bash
# Generate JSON output for ATS integration
./bin/analyze resume.pdf --format markdown
# Then parse the .json file programmatically
```

### Compare Candidates

Open multiple HTML reports in browser tabs for side-by-side comparison.

---

## 📖 Examples

See the `examples/` folder for:
- **`example.env`** - Complete template for configuring your API keys (copy this to create your `.env` file)

---

## 🏗️ Project Structure

```
aipm-resume-analyzer/
├── bin/
│   └── analyze                    # Main analyzer script
├── scripts/
│   └── install.sh                 # Installation script
├── templates/
│   └── output_generator.py        # Report generation
├── examples/
│   └── example.env                # ⭐ Template for your .env file (copy this!)
├── output/                        # Generated reports (created on first run)
├── requirements.txt               # Python dependencies
├── analyze                        # Convenience wrapper script
├── .env                          # ⚠️ YOUR API KEYS (create from example.env)
├── .gitignore
├── LICENSE
└── README.md
```

**Key files:**
- **`examples/example.env`** - Template showing all API configuration options
- **`.env`** - Your actual API keys (you must create this from the example)

---

## 🔒 Privacy & Security

**How your data is handled:**
- ⚠️ **Resume data is sent to your chosen AI provider** (OpenAI, Anthropic, or Google) for analysis
- ✅ **No data stored by this tool** - we don't have servers, only your local machine
- ✅ **You control which provider** receives your data via API selection
- ✅ **API keys stored locally** in `.env` file on your machine
- 📋 **Review provider policies**: Each AI provider has their own data retention and privacy policies

**Best practices:**
- Never commit `.env` file to git (it's in `.gitignore`)
- Don't share your API keys
- Rotate keys if accidentally exposed
- Review your chosen AI provider's data policies before analyzing sensitive resumes
- Keep this tool updated for security patches

---

## ❓ FAQ

**Q: Can I use this for non-AI PM roles?**
A: The framework is AI PM-specific, but you can fork and customize the prompts for other roles.

**Q: How accurate is the AI scoring?**
A: Treat it as a screening tool, not a final decision. Always validate with human review.

**Q: Can I customize the evaluation criteria?**
A: Yes! Edit `bin/analyze` to modify the scoring prompts and framework.

**Q: Which AI provider is most accurate?**
A: Claude Sonnet 4.5 and GPT-5 give the most nuanced analysis. Use `--list-models` to see all options. Gemini 2.5 Flash is fastest/cheapest for high-volume screening.

**Q: What file formats are supported?**
A: PDF, DOC, and DOCX files are supported. For .doc/.docx files, pandoc must be installed (see Prerequisites).

**Q: Can I analyze my own resume?**
A: Absolutely! Great for self-assessment before applying to AI PM roles.

---

## 🤝 Contributing

Contributions welcome! This tool helps standardize AI PM hiring and provides a roadmap for people transitioning into AI PM roles.

### To Contribute

1. Fork the repository
2. Create a feature branch: `git checkout -b feature-name`
3. Make your changes
4. Test with sample resumes
5. Submit a pull request

### Ideas for Contributions

- [ ] Additional output formats (PDF export)
- [ ] Batch processing UI
- [ ] Integration with popular ATS systems
- [ ] Customizable scoring frameworks
- [ ] Multi-language support

---

## 📄 License

MIT License - see LICENSE file for details.

---

## 🙏 Acknowledgments

**Inspiration:**
- [Jaclyn Konzelmann](https://x.com/jacalulu) - Google's Director of AI Product, whose evaluation framework inspired this project
- [Aakash Gupta](https://x.com/aakashg0) - Product Growth expert whose [podcast conversation](https://www.youtube.com/watch?v=rcz39Y0qFxw) sparked the idea

**Built with:**
- [OpenAI GPT-5](https://openai.com/) - Most advanced reasoning model
- [Anthropic Claude Sonnet 4.5](https://anthropic.com/) - Best-in-class AI assistant
- [Google Gemini 2.5 Pro](https://ai.google.dev/) - Fast and affordable AI
- [PyPDF2](https://github.com/py-pdf/pypdf2) - PDF text extraction
- [Pandoc](https://pandoc.org/) - Universal document converter (.doc/.docx support)

**Design inspired by:**
- [Stripe Docs](https://stripe.com/docs) - Clean documentation design
- [Tailwind CSS](https://tailwindcss.com/) - Modern utility-first CSS
- [Anthropic Docs](https://docs.anthropic.com/) - Warm, professional color palette

---

## 📚 Related Resources

- **Framework Website**: [aipmframework.com](https://aipmframework.com) - Full evaluation framework and philosophy
- **Source Code**: [GitHub Repository](https://github.com/abe238/aipm-resume-analyzer)
- **Twitter**: [@abe238](https://x.com/abe238)

---

## 🚨 Important Note

**This tool is a screening aid, not a replacement for human judgment.**

- ✅ Use it to: Standardize initial screening, identify promising candidates, save time, self-assess your readiness
- ❌ Don't use it to: Make final hiring decisions, replace interviews, avoid talking to candidates

The best hiring process combines AI-assisted screening with thoughtful human evaluation.

---

**Built for hiring managers, recruiters, and candidates—especially those from non-traditional backgrounds—who want standardized, data-driven AI PM evaluations.**

⭐ Star this repo if you find it useful!

---

## Need Help?

- **Issues**: https://github.com/abe238/aipm-resume-analyzer/issues
- **Twitter**: [@abe238](https://x.com/abe238)
- **Website**: [aipmframework.com](https://aipmframework.com)

---

## About the Builder

Hola. I'm **[Abe Diaz](https://abediaz.ai)**. By day, I'm Sr. Manager of Disaster Relief at Amazon, leading a team that uses AI and logistics to deliver 26+ million relief items and response technology across 200+ disasters worldwide. On weekends, I build artisanal developer tools like this :-P

- 💼 [linkedin.com/in/abediaz](https://linkedin.com/in/abediaz)
- 🌐 [abediaz.ai](https://abediaz.ai)
- 🐙 [github.com/abe238](https://github.com/abe238)
- 🐦 [@abe238](https://twitter.com/abe238)

### Other Open Source Projects

| Project | Description |
|---------|-------------|
| [gg-deploy](https://github.com/abe238/gg-deploy) | Domain → GitHub Pages in 60 seconds |
| [gemini-deep-research](https://github.com/abe238/gemini-deep-research) | CLI for Gemini's Deep Research Agent |
| [project-kickoff](https://github.com/abe238/project-kickoff) | Scaffold AI production-ready projects |