# CBO's Open Data Repository

Standardized, versioned data from public Congressional Budget Office (CBO) releases.

This repository transforms CBO spreadsheets and tables into CSV outputs with consistent variable names, schemas, and vintage-aware organization. It is designed to make this data more machine-readable and allow for easier reproducible analysis and cross-vintage comparison. 

The canoncial source of data remains [Budget and Economic Data](https://www.cbo.gov/data/budget-economic-data).


## Feedback
This is an initial attempt at making CBO's datasets more machine-readable and easier to process programattically. CBO welcomes feedback on the structure of this repository, how to best format the individual datasets, other datasets the agency should publish, and any other feedback you have. Please send comments to CBO’s Office of Communications at communications@cbo.gov.

---

## Data Format

The exact column structure varies by dataset type.

### Standard Long Format (most datasets)

Most economic and budget datasets use a **3-column long format**:

| Column | Type | Description |
|--------|------|-------------|
| `date` | string | Time period: `1949q1` (quarterly), `FY2026` (fiscal year), `CY2026` (calendar year) |
| `variable` | string | Snake_case variable name matching a key in `schema.json` |
| `value` | numeric | The observed or projected value |

Example:
```csv
date,variable,value
1949q1,gdp,275.0
1949q1,real_gdp,2260.8
1949q1,unemployment_rate,4.6667
```
**Estimate_type column**
Where source formatting distinguishes actual and projected values, outputs may include an estimate_type column:

- actual
- projected


### Multi-Column Format (`spending_detail`)

The `spending_detail` dataset uses a **wide format** — one row per budget account per fiscal year, with numeric values in separate columns rather than pivoted into a `variable`/`value` pair. This reflects the source data's structure (~2,000 named accounts with distinct metadata).

| Column | Description |
|--------|-------------|
| `date` | Fiscal year (e.g., `FY2026`) |
| `tin` | Treasury Identification Number (unique account ID) |
| `title` | Budget account name |
| `disc_or_mand` | Discretionary or Mandatory |
| `agency`, `bureau` | Agency and bureau labels |
| `function_code`, `subfunction_code` | Budget function classification |
| `budget_authority` | Budget authority (millions of dollars) |
| `outlays` | Outlays (millions of dollars) |

Example:
```csv
date,tin,title,disc_or_mand,agency,bureau,function_code,subfunction_code,budget_authority,outlays
FY2026,017-0730-0-1-051,"Family housing construction, Navy",Discretionary,Dept of Defense,Family Housing,050,051,178.0,239.0
```

To find a specific account: filter on `tin` or match `title` text, then read `budget_authority` and `outlays` directly.

### Multi-Dimensional Format (`demographic`)

Demographic files are also multi-column, but their key structure differs: instead of a single `date` + `variable` pair, each row is uniquely identified by a combination of `year` and demographic dimensions (age, sex, place of birth, immigration status, etc.). The measure column varies by domain.

| File | Key columns | Measure column |
|------|-------------|----------------|
| `population_bls_*.csv` | `year`, `age`, `sex` | `number_of_people` |
| `fertility_*.csv` | `year`, `age`, `place_of_birth` | `births_per_1000_females` |
| `mortality_*.csv` | `year`, `age`, `sex` | `deaths_per_1000_people` |
| `migration_*.csv` | `year`, `age`, `sex`, `immigration_status`, `migration_flow` | `number_of_people` |

Example:
```csv
year,age,sex,number_of_people
2021,16,female,2161882.0
2021,16,male,2218989.0
```

To look up a specific cohort: filter on all key columns, then read the measure column directly.

### Frequency

Frequency varies by dataset type:

**Economic data** — **quarterly** or **annual (FY or CY)** 

**Budget data** — **annual (FY or CY)**. CBO publishes budget data as annual fiscal-year or calendar-year tables. There is no quarterly source to derive from.
- Most budget datasets use fiscal-year dates (`FY2025`, `FY2036`)
- Tax parameters and some revenue detail tables use calendar-year dates (`CY2022`, `CY2036`)
- Revenue detail produces both FY and CY output files from a single source workbook

---

## Machine-Readable Catalog

A `catalog.json` file in the repository root provides [DCAT](https://www.w3.org/TR/vocab-dcat-3/)-compatible metadata for every dataset, including file paths, vintages, temporal coverage, and field counts. Regenerate it with:

```bash
python scripts/build_catalog.py
```

---

## Data Catalog

### Economic Data

| Dataset | Publication | Description | Frequency | Status |
|---------|-------------|-------------|-----------|--------|
| `historical_economic` | [55022](https://www.cbo.gov/publication/55022) | Output, prices, labor, interest rates, income, potential GDP (1949–present + projections) | Quarterly | **Available** |
| `economic_projections` | [51135](https://www.cbo.gov/publication/51135) | 10-year projections of economic variables  | Quarterly + Annual| **Available** |
| `long_term_economic` | [57054](https://www.cbo.gov/publication/57054) | Multi-decade economic projections (growth rates, levels, LFP by age/sex) | Annual | **Available** |
| `potential_gdp` | [51137](https://www.cbo.gov/publication/51137) | Potential GDP and underlying inputs (pre-2020 standalone, now in 55022/51135) | Quarterly + Annual | **Available** |
| `demographic` | [57059](https://www.cbo.gov/publication/57059) | Population, fertility, mortality, migration projections by age/sex | Annual | **Available** |

### Budget Data

| Dataset | Publication | Description | Frequency | Status |
|---------|-------------|-------------|-----------|--------|
| `historical_budget` | [51134](https://www.cbo.gov/publication/51134) | Revenue, outlays, deficit/surplus, and debt since FY1962  | Annual FY | **Available** |
| `ten_year_budget` | [51118](https://www.cbo.gov/publication/51118) | 10-year spending/revenue/deficit/debt projections | Annual FY | **Available** |
| `long_term_budget` | [51119](https://www.cbo.gov/publication/51119) | 30-year budget projections as % of GDP | Annual FY | **Available** |
| `trust_fund` | [51136](https://www.cbo.gov/publication/51136) | Trust fund balances, surpluses, and OASI/DI/HI detail | Annual FY | **Available** |
| `revenue_detail` | [51138](https://www.cbo.gov/publication/51138) | Revenue by source with IIT/cap gains/corporate detail | Annual FY + CY | **Available** |
| `spending_detail` | [51142](https://www.cbo.gov/publication/51142) | Budget authority and outlays for ~2,000 budget accounts | Annual FY | **Available** |
| `automatic_stabilizers` | [51139](https://www.cbo.gov/publication/51139) | GDP/unemployment gaps, stabilizer effects on budget | Annual FY | **Available** |
| `tax_parameters` | [53724](https://www.cbo.gov/publication/53724) | Tax brackets, deductions, credits, EMTRs, tax wedges | Annual CY | **Available** |

---

## Versioning

Each CBO release is stored as a **separate file** named with the vintage date:

```
data/economic/historical_economic/quarterly_2026-02.csv
data/economic/historical_economic/quarterly_2025-01.csv
```

This mirrors CBO's own convention, keeps git diffs clean, and lets consumers request a specific vintage.

---

## Schema Files

Each dataset directory contains a `schema.json` that describes the dataset and its variables. The exact fields vary by dataset type.

**Economic datasets** include per-variable metadata under `fields`:

```json
{
  "fields": {
    "gdp": {
      "description": "Gross Domestic Product",
      "unit": "SAAR, billions_of_dollars",
      "aggregation": "mean",
      "source_frequency": "quarterly",
      "category": "output"
    }
  }
}
```

- `description` — Human-readable name
- `unit` — Measurement units
- `aggregation` — How to derive annual values from quarterly (`mean`, `sum`, `eop`) — economic only
- `source_frequency` — Native granularity (`quarterly` or `annual`)
- `category` — Thematic grouping (output, prices, labor, income, etc.)

**Budget datasets** include dataset-level metadata (`title`, `description`, `frequency`, `date_format`, `notes`) and per-variable `fields` where applicable. The `spending_detail` schema describes columns rather than variables, since its format is multi-column rather than long. The `revenue_detail` schema includes `variable_note` where applicable. Those `variable_notes` corrpespond to the variable-specific footnotes included in the most recent vintage of the revenue_detail data released on CBO's data page.  

---

## How to Access the Data Files

```python
# Read a file directly from GitHub in Python
import pandas as pd

url = "https://raw.githubusercontent.com/US-CBO/cbo-data/main/data/budget/revenue_detail/annual_fy_2026-02.csv"
df = pd.read_csv(url)

# Get deficit and debt projections from the file above

series = ["proj_deficit_total", "proj_debt_held_by_public"]
result = df[df["variable"].isin(series)].pivot(index="date", columns="variable", values="value")
print(result)
```

```r
# Read a file directly from GitHub in R
url <- "https://raw.githubusercontent.com/US-CBO/cbo-data/main/data/budget/revenue_detail/annual_fy_2026-02.csv"
df <- read.csv(url)
```

```bash
# Download a file using curl
curl -O https://raw.githubusercontent.com/US-CBO/cbo-data/main/data/budget/revenue_detail/annual_fy_2026-02.csv
```

## How to Install and Run This Code

Follow these three steps to replicate CBO's results:

### 1. Install the Anaconda Distribution of Python

The Anaconda distribution of Python contains many useful Python packages for data analysis and statistical modeling. You can download and install it from [Anaconda's Installation page](https://docs.anaconda.com/anaconda/install/).

This code was developed using Python 3.10 on computers running Windows 10, although it should run on other operating systems as well.

CBO used Anaconda's built-in package manager, `conda`, to manage external packages. To replicate CBO's results, you will need to use `conda` to create a virtual environment with the appropriate version of Python and all the same package versions. All external packages and their versions are documented in the `environment.yml` file in the project's root directory.

### 2. Download the Code from GitHub

There are several options for getting the code to your computer:

- **With git installed:** Clone the repo with:
  ```
  git clone https://github.com/us-cbo/cbo-data.git
  ```
- **With a GitHub account:** Fork the repo to your own account, then clone it to your computer.
- **Without git:** Download a zipped file of the repo from GitHub and unzip it.

### 3. Create the `cbo-data` Virtual Environment

Once you have Anaconda installed and the repo downloaded, follow these steps:

1. Open the **Anaconda Prompt** application.
   > **Note:** If the repo is on a network drive, use **Anaconda PowerShell Prompt** instead.

2. Navigate to the root directory of the repo:
   ```
   cd your/path/to/cbo-data
   ```
   > **Note:** If any directory names contain spaces, wrap the path in quotation marks.

3. Create the virtual environment:
   ```
   conda env create -f environment.yml
   ```
   > **Note:** This may take several minutes to complete.

4. Activate the virtual environment:
   ```
   conda activate cbo-data
   ```
   All code should be run from within this virtual environment to ensure results are replicable.

5. When finished, deactivate the virtual environment:
   ```
   conda deactivate
   ```

## Examples of How To Run the Code

```bash
# Download CBO data
python -m etl.download historical_economic 2026-02

# Transform to long format (economic)
python -m etl.transform_historical_econ 2026-02

# Transform budget data
python -m etl.transform_historical_budget 2026-02
python -m etl.transform_ten_year_budget 2026-02
python -m etl.transform_long_term_budget 2026-02
python -m etl.transform_trust_fund 2026-02
python -m etl.transform_revenue_detail 2026-02
python -m etl.transform_spending_detail 2026-02
python -m etl.transform_automatic_stabilizers 2024-11
python -m etl.transform_tax_parameters 2026-02

# Validate
python -m etl.validate data/economic/historical_economic/quarterly_2026-02.csv
python -m etl.validate data/budget/historical_budget/annual_fy_2026-02.csv

# Generate annual views (economic data only)
python scripts/generate_annual.py data/economic/historical_economic/quarterly_2026-02.csv

# Full pipeline (download → transform → validate all)
python scripts/build_all.py
```


## Contact

Questions may be directed to CBO's Office of Communications at communications@cbo.gov. CBO will respond to inquiries as its workload permits.