--- name: data-analysis description: > Comprehensive data analysis agent skill for loading, cleaning, exploring, visualizing, and reporting on structured datasets. Supports CSV, JSON, Excel, and SQL data sources. Produces statistical summaries, correlation matrices, time series analysis, regression models, hypothesis tests, and publication-quality visualizations. version: 1.0.0 author: Skill Foundry platforms: - claude-code - codex - cursor - gemini-cli - openclaw - copilot tags: - data-analysis - visualization - statistics - pandas - matplotlib - seaborn - plotly - reporting - csv - json - excel - sql --- # Data Analysis Agent Skill ## Overview A production-grade agent skill for end-to-end data analysis workflows. Use this skill when you need to load datasets, inspect structure, clean messy data, perform statistical analyses, create visualizations, or generate structured reports from tabular data. ## When to Trigger Activate this skill when the user asks to: - **Analyze data**: "Analyze this CSV", "What patterns do you see in sales.csv?", "Explore this dataset", "Summarize the data in users.json" - **Create charts/graphs**: "Make a bar chart of…", "Plot revenue over time", "Visualize the correlation matrix", "Show me a heatmap of…" - **Find patterns**: "Find trends in this data", "Is there a correlation between X and Y?", "Cluster customers from this data", "Detect anomalies in…" - **Generate reports**: "Create a report from survey_results.xlsx", "Summarize quarterly metrics", "Build a dashboard from sales data" - **Clean data**: "Clean this messy dataset", "Fix missing values in…", "Normalize these columns", "Deduplicate this CSV" - **Statistical testing**: "Run a t-test on group A vs B", "Check if this distribution is normal", "Perform regression analysis", "Calculate confidence intervals" ### Near-Miss Negatives — Do NOT Trigger - Questions about **database schema design** without actual data (e.g., "What columns should my users table have?") - Questions about **spreadsheet software UI** (e.g., "How do I freeze a row in Google Sheets?") - General **math / statistics theory** questions without a dataset context (e.g., "Explain the central limit theorem") - Pure **SQL query writing** without a data-analysis intent (e.g., "Write a query to join three tables" — use a SQL skill instead) - Questions about **ETL pipeline architecture** or data engineering (e.g., "Design a data ingestion pipeline") ## Step-by-Step Workflow Follow these phases in order. Skip phases that don't apply (e.g., if data is already clean) but always **state that you're skipping** and why. ### Phase 1: Load Determine the data source and load it into a DataFrame. ``` CSV → pd.read_csv(filepath, ...) JSON → pd.read_json(filepath, ...) Excel → pd.read_excel(filepath, sheet_name=...) SQL → pd.read_sql_query(query, connection) ``` **Checklist:** - [ ] Identify encoding (try UTF-8, Latin-1, detect automatically) - [ ] For CSV: inspect delimiter (comma, tab, semicolon), quote character - [ ] For Excel: list available sheets, load the right one - [ ] For JSON: handle nested structures with `pd.json_normalize()` if needed - [ ] For SQL: confirm read-only access, never run destructive queries - [ ] Load a sample first if the dataset is large (>100k rows) ### Phase 2: Inspect Understand what you're working with before touching anything. ```python df.shape # rows × columns df.info() # dtypes, non-null counts, memory df.head(10) # first rows df.tail(5) # last rows df.describe() # numeric summary stats df.describe(include='object') # categorical summary df.dtypes # column types df.columns.tolist() # column names ``` **Checklist:** - [ ] Report shape: rows × columns - [ ] List all columns with their dtypes - [ ] Show summary statistics for numeric columns - [ ] Show value counts for low-cardinality categorical columns - [ ] Flag potential issues: wrong dtypes, placeholder values, suspicious zeros ### Phase 3: Clean Address data quality issues. **Never modify the source file** without prompting the user first. Work on a copy. **Common operations:** | Issue | Approach | |-------|----------| | Missing values | `df.isnull().sum()` → decide drop vs impute | | Wrong dtypes | `pd.to_numeric()`, `pd.to_datetime()`, `astype()` | | Outliers | IQR method, Z-score, domain-specific thresholds | | Duplicates | `df.duplicated().sum()` → `df.drop_duplicates()` | | Inconsistent strings | `.str.strip()`, `.str.lower()`, `.str.replace()` | | Date parsing | `pd.to_datetime()` with format or infer | | Normalization | Min-max scaling, Z-score standardization | | Categorical encoding | One-hot, label encoding for ML prep | **Rules:** 1. Always work on `df_clean = df.copy()`, never mutate the original in-place without explicit user consent. 2. Report every change: "Dropped 47 duplicate rows (2.3% of data)", "Imputed missing age values with median (142 cells)". 3. Flag suspicious patterns even if you don't fix them: "Column 'salary' has 340 zero values — verify if these are legitimate." 4. If a cleaning decision is irreversible, ask first. ### Phase 4: Analyze Apply appropriate analytical methods based on the question. **Exploratory Data Analysis (EDA):** - Univariate: histograms, box plots, value counts per column - Bivariate: scatter plots, correlation coefficients, grouped means - Multivariate: pair plots, correlation matrix heatmap, PCA **Statistical Methods (see `references/statistical-methods.md`):** | Goal | Method | |------|--------| | Compare two groups | Independent t-test, Mann-Whitney U | | Compare 3+ groups | One-way ANOVA, Kruskal-Wallis | | Relationship between two continuous vars | Pearson/Spearman correlation | | Predict continuous outcome | Linear regression, polynomial regression | | Predict categorical outcome | Logistic regression | | Check normality | Shapiro-Wilk test, Q-Q plot | | Detect time trends | Moving averages, decomposition, stationarity tests | | Find clusters | K-means, hierarchical clustering, DBSCAN | | Reduce dimensions | PCA, t-SNE (visualization only) | **Time Series specifics:** - Set datetime index: `df.set_index('date', inplace=True)` - Resample: `df.resample('M').mean()` - Rolling windows: `df['value'].rolling(7).mean()` - Decomposition: trend, seasonal, residual ### Phase 5: Visualize Choose the right chart for the data and question. See `references/visualization-patterns.md` for the full guide. **Library selection:** - **Static, publication-quality**: `matplotlib` + `seaborn` - **Interactive, exploratory**: `plotly` - **Statistical plots**: `seaborn` (box, violin, pair, joint, heatmap) **Quick reference:** | Data Type | Question | Chart | |-----------|----------|-------| | Categorical × Numeric | Compare amounts | Bar chart, box plot | | Numeric × Numeric | Relationship | Scatter plot, line chart | | Time × Numeric | Trend over time | Line chart, area chart | | Categorical × Categorical | Cross-tabulation | Heatmap, stacked bar | | Distribution | Shape of data | Histogram, KDE, violin | | Part-to-whole | Proportions | Pie chart* (≤5 categories), treemap | | Correlation matrix | Relationships | Heatmap | | Rankings | Order | Horizontal bar chart | *Pie charts: use only when ≤5 categories and values sum to a meaningful whole. Prefer bar charts otherwise. **Best practices:** - Always label axes and add a title - Use accessible color palettes (avoid red-green for colorblind users) - Sort bar charts by value unless categories have a natural order - Add data source and date to chart footnotes - For interactive charts, include hover tooltips ### Phase 6: Report Synthesize findings into a structured report. **Report structure:** 1. **Executive Summary** — 2–3 sentences with the key finding 2. **Data Overview** — source, shape, date range, columns 3. **Data Quality** — issues found, actions taken 4. **Key Findings** — bullet points with numbers, ranked by importance 5. **Visualizations** — inline charts with captions 6. **Statistical Results** — test statistics, p-values, effect sizes 7. **Limitations & Caveats** — data gaps, assumptions, edge cases 8. **Recommendations** — actionable next steps or further analysis **Output formats:** - **Quick answer**: plain text summary in chat with key numbers - **Detailed report**: Markdown document with embedded charts - **Dashboard**: interactive HTML with plotly (offer if >5 charts) - **Export**: offer to save cleaned data and charts as files ## Tool-Aware Implementation ### Python Libraries This skill assumes Python 3.9+ with the following libraries available. Check availability before use; install missing packages as needed. ```python # Core import pandas as pd import numpy as np # Visualization import matplotlib.pyplot as plt import seaborn as sns # Interactive import plotly.express as px import plotly.graph_objects as go # Statistics from scipy import stats from scipy.stats import norm, ttest_ind, f_oneway, pearsonr, spearmanr # Optional: machine learning from sklearn.preprocessing import StandardScaler, LabelEncoder from sklearn.cluster import KMeans from sklearn.decomposition import PCA from sklearn.linear_model import LinearRegression, LogisticRegression ``` **Matplotlib setup for non-interactive environments:** ```python import matplotlib matplotlib.use('Agg') # headless rendering ``` **Plotly in notebooks vs scripts:** ```python # In Jupyter/notebook environments: import plotly.io as pio pio.renderers.default = 'notebook' # For saving to files: fig.write_html('chart.html') fig.write_image('chart.png') ``` ### Platform-Specific Notes | Platform | Matplotlib backend | File output | Notes | |----------|-------------------|-------------|-------| | Claude Code | Agg | write to files | Save charts as PNG/HTML, display from disk | | Codex | Agg | write to files | Same approach | | Cursor | Agg or interactive | write to files | Can open HTML in preview | | Gemini CLI | Agg | write to files | Save charts, display paths | | OpenClaw | Agg | write to files | Use canvas for HTML output | | Copilot | Agg | write to files | Standard file-based approach | ## Safety & Guardrails 1. **Never modify source data in-place** — always create a copy or backup before transformations. Offer to save cleaned data as a new file. 2. **Flag data quality issues** — don't silently fix problems. Report missing values, outliers, and type inconsistencies before and after cleaning. 3. **Statistical honesty** — report p-values and effect sizes, not just "significant" or "not significant". Don't p-hack by running multiple tests without correction. Mention when sample sizes are too small for reliable inference. 4. **Privacy awareness** — if a dataset appears to contain PII (emails, phone numbers, names), warn the user and suggest anonymization before analysis. 5. **Large file handling** — for files >100MB, use chunked reading (`chunksize` parameter) or sample before full analysis. Warn about memory constraints. 6. **SQL safety** — use read-only connections. Never run INSERT, UPDATE, DELETE, DROP, or ALTER. Use transactions or connection strings that enforce read-only mode. 7. **Deterministic results** — set random seeds for reproducible analysis: `np.random.seed(42)`. ## Scripts ### `scripts/validate_dataset.py` PEP 723 compliant data quality validation script. Run with: ```bash python scripts/validate_dataset.py path/to/dataset.csv # or python scripts/validate_dataset.py path/to/dataset.json # or python scripts/validate_dataset.py path/to/dataset.xlsx --sheet "Sheet1" ``` Produces a structured quality report covering missing values, outliers, type consistency, duplicates, and basic statistics. See script docstring for details. ## References - **[data-cleaning-guide.md](references/data-cleaning-guide.md)** — Handling nulls, outliers, type coercion, deduplication, and normalization patterns. - **[visualization-patterns.md](references/visualization-patterns.md)** — Chart type selection guide, color best practices, accessibility considerations. - **[statistical-methods.md](references/statistical-methods.md)** — Descriptive statistics, hypothesis testing, regression, correlation, confidence intervals, and when to use each method.