---
layout: default
title: Key concepts and common tasks
nav_order: 3
parent: How to use Data Commons
redirect_from: 
  - /bigquery/data_in_bq
  - /bigquery/unique_identifiers
---

{: .no_toc}
# Key concepts and common tasks

Whether you're just exploring the data on [datacommons.org](http://datacommons.org), using the programmatic APIs, or contributing data, it's helpful to have a basic understanding of some of the key concepts in Data Commons. 

{:toc}
* TOC

## Schema 

To allow data from hundreds of organizations around the world, in a myriad of models and formats to be interoperable and queryable in a unified way, Data Commons needs to have a common way of understanding and representing this data. To do so, it applies a schema, or vocabulary to all its data, that is largely derived from earlier schemes developed for semantic understanding of web pages – most notably, the data models and schemas of [Schema.org](http://Schema.org) (which were in turn based on earlier schemes such as Microformats and Resource Description Framework (RDF)). 

The Data Commons schema is in fact a superset of [Schema.org schemas](https://schema.org/docs/schemas.html){: target="_blank"}, with a particular emphasis on time series and statistical data. Every data set must have an associated schema, written in [Meta Content Format](https://en.wikipedia.org/wiki/Meta_Content_Framework){: target="_blank"} (MCF) language, that maps the provider's data to existing concepts in the Data Commons knowledge graph.

## Knowledge Graph

Data Commons models the world as a directed labeled graph, consisting of a set of _nodes_ and edges with labels, known as _properties_. This general framework allows Data Commons to represent information about a wide range of domains: from time series about demographics and employment, to hurricanes, to protein structures. 

As a simple example, here are a set of nodes and edges that represent the following statements:

*  California is a state
*  Santa Clara county and Berkeley are contained in the state of California
*  The latitude of Berkeley, CA is 37.8703

![knowledge graph](/assets/images/dc/concept1.png){: width="600"}

Each node consists of some kind of entity or value, and each edge describes some kind of property. More specifically, each node consists of the following objects:

*  One or more [types](#type): an [entity](#entity), [event](#event), [statistical variable](#statistical-variable), or [statistical observation](#observation)
*  A [unique identifier](#unique-identifier-dcid), known as a DCID
*  Various [properties](#property)
*  A [provenance](#provenance-source-dataset)

As in other knowledge graphs, each pair of connected nodes is a _triple_ consisting of a subject node, predicate (or "edge") and object node. The Data Commons knowledge graph is made up of billions of triples. The triple is not generally exposed in Data Commons as a concept that you need to know.

You can get information about a node and its edges by looking at the [Knowledge Graph browser](https://datacommons.org/browser){: target="_blank"}. If you know the [DCID](#unique-identifier-dcid) for a node, you can access it directly by typing <code>https://datacommons.org/browser/<var>DCID</var></code>. For example, here is the entry for the `City` node, available at [https://datacommons.org/browser/City](https://datacommons.org/browser/City){: target="_blank"}:

![KG browser](/assets/images/dc/concept2.png){: width="900"}

Every node entry shows a list of outgoing edges, or _properties,_ and incoming edges. [Properties](#property) are discussed in more detail below.

## Type

Every node has at least one type, where each type may be a sub-class of multiple types. For entities and events, their type is typically another entity. For example, `Berkeley` is a type of `City`. At the root, all types are instances of the Class type. For statistical variables and observations, their type is always `StatisticalVariable` and `StatVarObservation`, respectively.

## Entity

An _entity_ represents a persistent, physical thing in the real world. While Data Commons has information about a wide variety of types of entities (cities, states, countries, schools, companies, facilities, etc.), most of the information today is about _places_. Data Commons contains a catalog of about 2.9 million places. In addition to basic metadata like the location, type and containment information, many places also contain information about their shape, area, etc. For a list of available place types, take a look at the [place types page](/place_types.html).

## Event

An _event_ is what it sounds like: an occurrence at a specific point in time, such as an extreme weather event, a criminal incident, an election, etc. 

## Statistical variable

In Data Commons, even statistical measurements and time series data are modeled as nodes. A _statistical variable_ represents any type of metric, statistic, or measurement that can be taken at a place and time, such as a count, an average, a percentage, etc. A statistical variable for a specific place is a _time series_, consisting of a set of observed values over a time period.

Data Commons comprises hundreds of thousands of statistical variables, which you can view using the [Statistical Variable Explorer](https://datacommons.org/tools/statvar){: target="_blank"}. 

The type of a statistical variable is always the special sub-class [`StatisticalVariable`](https://datacommons.org/browser/StatisticalVariable){: target="_blank"}. For example, the metric [`Median Age of Female Population`](https://datacommons.org/browser/Median_Age_Person_Female){: target="_blank"} is a node whose type is a statistical variable. 

A statistical variable can be simple, such as [`Total Population`](https://datacommons.org/browser/Count_Person){: target="_blank"}, or more complex, such as [`Hispanic Female Population`](https://datacommons.org/tools/statvar#Count_Household_NoHealthInsurance=&sv=Count_Person_Female_HispanicOrLatino){: target="_blank"}. Complex variables may be broken down into constituent parts, or not. 

The constituent parts, or dimensions, are represented as "constraint properties" or "observation properties". (The latter is a newer feature of the Data Commons graph). With _constraint properties_, the actual values of each property are specified in the statistical variable itself. For example, with the aforementioned [`Hispanic Female Population`](https://datacommons.org/browser/Count_Person_Female_HispanicOrLatino){: target="_blank"} variable, there are two constraint properties: `gender` and `race`. Both have defined values, `Female` and `HispanicOrLatino`. With constraint properties, a separate variable is required for each combination of property values. This feature is best used with properties that have low cardinality, such as gender.

With _observation properties_, the values of each property are not specified in the variable itself, but in the [observations](#observation). Instead of needing a separate variable for every combination of all the values of two or more properties, this feature allows you to limit the number of total variables needed. It is most useful for properties that have high cardinality. As an example, imagine a variable that measures the number of teachers in schools, broken down by the age group and foreign languages spoken by the teachers. With constraint properties, there would need to be a separate variable for every combination of every age group and language spoken: `Count_Teachers_Age25To30_SpanishSpeaker` and so on. This would result in an explosion of hundreds of variables. With observation properties, a single variable can be defined, such as `Count_Teachers`. `languagesSpoken` and `ageGroup` become columns within a set of observations (timeseries). Each individual observation specifies the values of the relevant values for that observation.

### Task: Find places available for a statistical variable {#find-places}

Note that not all statistical variables have observations for all places or other entities. To find out which places have data for a given variable, you have to use the APIs to do so.

> Tip: You can also use an agent to query the [MCP server](/mcp/run_tools.html) for this information.

To use the APIs to look up places available for statistical variables:

* Use the REST v2 [Observation API](/api/rest/v2/observation.html) with an entity expression and `containedInPlace` filter, either interactively (e.g. using curl or the browser address bar) or programmatically.
* Use the Python v2 [`observation.fetch`](/api/python/v2/observation.html#fetch) API method with an entity expression, either interactively or programmatically.
* Use the SDMX [Availability API](/api/sdmx/availability.html) with a place ID or `containedInPlace` filter, either interactively or programmatically.

## Unique identifier: DCID

Every node has a unique identifier, called a Data Commons ID, or DCID. In the [Knowledge Graph browser](https://datacommons.org/browser/){: target="_blank"}, you can view the DCID for any node or edge. For example, the DCID for the city of Berkeley is `geoid/0606000`:

![KG browser](/assets/images/dc/concept6.png){: width="600"}

DCIDs are not restricted to entities; statistical variables also have DCIDs. For example, the DCID for the Gini Index of Economic Activity is `GiniIndex_EconomicActivity`:

![Stat Var Explorer](/assets/images/dc/concept7.png){: width="900"}

### Task: Find a DCID for an entity or variable {#find-dcid}

Many Data Commons tools and APIs require that you provide a DCID as input for a query. There are a few ways to do this.

To find the DCID for a place using the datacommons.org website:

1. Go to [https://docs.datacommons.org/place_types.html](https://docs.datacommons.org/place_types.html){: target="_blank"}. 
1. Click the link on the place type of interest and click its link to open the Knowledge Graph page for the entity; for example, **City**.
1. Scroll to the **In Arcs** section to look up the places of interest. 
1. If necessary, continue to drill down on links until you find the place of interest. 

![KG browser](/assets/images/dc/concept8.png){: width="900"}

To find the DCID for a place using other methods:

* Use the REST v2 [Resolve API](/api/rest/v2/resolve.html), either interactively (e.g. using curl or the browser address bar) or programmatically.
* Use the Python v2 [`resolve.fetch_dcids_by_name`](/api/python/v2/resolve.html#fetch_dcids_by_name) API method, either interactively or programmatically.

To find the DCID for a statistical variable using the datacommons.org website:

1. Open the Statistical Variable Explorer.
1. Search for the variable of interest, and optionally filter by data source and dataset.
1. Look under the heading for the DCID.

![Stat Var Explorer](/assets/images/dc/concept9.png){: width="900"}

To find the DCID for a statistical variable using other methods:

* Use the REST v2 [Resolve API](/api/rest/v2/resolve.html) with the `resolver=indicator` option.
* Use the Python v2 [`resolve.fetch_indicators`](/api/python/v2/resolve.html#fetch_indicators) API method.

## Property

Every node also contains properties or characteristics that describe its entity, event, or statistical variable. Each property is actually an edge to another node, with a label. If the object node is a primitive type, such as a string, date, or number, it is a "leaf", or terminal node, which we call an _attribute_. Examples are properties such as `latitude`, `year`, various unique IDs and so on. 

Other properties are links to other entities/events/ etc. In the Knowledge Graph, you can click through links to non-terminal nodes. 

For example, in this node for the city of Addis Ababa, Ethiopia, the `typeOf` and `containedInPlace` edges link to other entities, namely `City` and `Ethiopia`, whereas all the other values are terminal.

![KG browser](/assets/images/dc/concept10.png){: width="600"}

Note that the DCID for a property is the same as its name.

## Observation

An _observation_ is a single measured value for a statistical variable, at or during a specified period of time, for a specific entity (or entities).

For example, the value of the statistical variable [`Median Age of Female Population`](https://datacommons.org/browser/Median_Age_Person_Female){: target="_blank"} for the city of San Antonio, Texas in 2014 could have an observation `Observation_Median_Age_Person_Female_SanAntonio_TX_2014`. The type of an observation is always the special sub-class [`StatVarObservation`](https://datacommons.org/browser/StatVarObservation){: target="_blank"}.

Time series made up of many observations underlie the data available in the [Timeline Explorer](https://datacommons.org/tools/timeline){: target="_blank"} and timeline graphs. For example, here is the [median income in Berkeley, CA over a period of ten years](https://datacommons.org/tools/timeline#place=geoId%2F0606000&statsVar=Median_Income_Person){: target="_blank"}, according to the US Census Bureau:

![Timeline Explorer](/assets/images/dc/concept11.png){: width="900"}

## Provenance, Source, Dataset
{: #sources}

Every node and triple also have some important properties that indicate the origin of the data. 

*  [`Source`](https://datacommons.org/browser/Source){: target="_blank"}: This is the organization that provides the data, and is usually specified as the name of the organization; for example, [Australian Bureau of Statistics](https://datacommons.org/browser/dc/s/AustralianBureauOfStatistics){: target="_blank"}.
*  [`Dataset`](https://datacommons.org/browser/Dataset){: target="_blank"}: This is the name of a specific dataset provided by a source. In relational database terminology, a `dataset` roughly corresponds to a "database". Many sources provide multiple datasets. For example, the source Australian Bureau of Statistics provides two datasets, [Australia Statistics](https://datacommons.org/browser/dc/d/AustralianBureauOfStatistics_AustraliaStatistics){: target="_blank"}, and [Australia Subnational Administrative Boundaries](https://datacommons.org/browser/dc/d/AustralianBureauOfStatistics_AustraliaSubnationalAdministrativeBoundaries){: target="_blank"}.
*  [`Provenance`](https://datacommons.org/browser/Provenance){: target="_blank"}: A provenance is a subset of a dataset. For small datasets, it may represent the entire dataset. For example, Sweden Census is both a [dataset](https://datacommons.org/browser/dc/d/StatisticsSweden_SwedenCensus){: target="_blank"} and a [provenance](https://datacommons.org/browser/dc/base/Sweden_Census){: target="_blank"}.
  
   For larger datasets, a provenance usually represents a subset of the dataset, roughly corresponding to a "table" in relational database terminology. Thus, there may be several provenances for a given dataset. For example, [Brazil VIS DATA 3](https://datacommons.org/browser/dc/d/BrazilMinistryOfDevelopmentAndSocialAssistanceFamilyAndFightAgainstHunger_BrazilVisData3){: target="_blank"} is a dataset that comprises 2 provenances: [Brazil Food Distribution](https://datacommons.org/browser/dc/base/Brazil_FoodDsitribution){: target="_blank"} and [Brazil Rural Development Program](https://datacommons.org/browser/dc/base/Brazil_RuralDevelopmentProgram){: target="_blank"}. 
  
    In Data Commons, a provenance is the physical unit of an import, and thus contains detailed import information. A dataset is more of an abstract concept; each provenance has a property that points to the dataset it belongs to.

![Knowledge graph](/assets/images/dc/concept12.png){: width="600"}

Note that a given statistical variable may have multiple provenances, since many data sets define the same variables. You can see the list of all the data sources for a given statistical variable in the Statistical Variable Explorer. For example, the explorer shows multiple sources (Censuses from India, Mexico, Vietnam, OECD, World Bank, etc.) for the variable [Life Expectancy](https://datacommons.org/tools/statvar#LifeExpectancy_Person=&sv=LifeExpectancy_Person){: target="_blank"}:

![Stat Var Explorer](/assets/images/dc/concept13.png){: width="900"}

You can see a list of all sources and data sets in several places:

*  The [Data Sources](https://datacommons.org/data/){: target="_blank"} pages
*  The **Data source** and **Dataset** drop-down menus in the Statistical Variable Explorer

![Stat Var Explorer](/assets/images/dc/concept14.png){: width="600"}