![\"Open](\"https://colab.research.google.com/assets/colab-badge.svg\"){kind=icon}
"
]
},
{
"cell_type": "markdown",
"source": [
"# Cracking Transit Data — Calgary 2025\n",
"## How to Decode and Leverage GTFS for Real-Time Transit Insights\n",
"\n",
"\n",
"\n",
"\n",
"\n",
" "
],
"metadata": {
"id": "9HGnYbLacYhV"
}
},
{
"cell_type": "markdown",
"source": [
"# Introduction\n",
"In the digital age, public transportation systems increasingly leverage technology to provide passengers with real-time data, enabling a more seamless travel experience. Transit systems share this data primarily through the General Transit Feed Specification (GTFS), a standard that provides data on schedules, routes, and real-time updates. However, while this data can be incredibly valuable, accessing and interpreting it can be challenging, especially when combining static and real-time feeds.\n",
"\n",
"This article shows you how to use Calgary Transit’s data about bus locations and routes to get useful information. If you work with data, understanding these datasets can help improve transit planning and make it better for users. We’ll cover the essential steps for fetching and processing Calgary Transit’s static and real-time data, including troubleshooting common issues you might encounter. By the end of this guide, you’ll be well-equipped to tap into Calgary’s transit data to solve real-world problems.\n",
"\n",
"---"
],
"metadata": {
"id": "AgUWBNxpeVOB"
}
},
{
"cell_type": "markdown",
"source": [
"# GTFS\n",
"The General Transit Feed Specification (GTFS) is an open standard that formats public transport schedules and geographic data. GTFS allows public transit agencies to publish their data in a format that various software applications can consume, such as trip planners and API developers. This allows users easy access to travel information on smartphones and other devices.\n",
"\n",
""
],
"metadata": {
"id": "7ztDPf_wePoF"
}
},
{
"cell_type": "markdown",
"source": [
"GTFS includes information such as\n",
"\n",
"* Trip Name\n",
"* Stops\n",
"* Bus Routes\n",
"* Fares\n",
"* Time and more.\n",
"\n",
"When working with GTFS data, it’s important to understand the source and format of the data you are using. To fully explore and use the transit’s GTFS feeds, including the static and real-time data, visit their official website.\n",
"\n",
"[What is GTFS - General Transit Feed Specification](https://gtfs.org/getting-started/what-is-GTFS/?source=post_page-----49baea30833e--------------------------------)\n",
"\n",
"---"
],
"metadata": {
"id": "4uEO0li5e_D5"
}
},
{
"cell_type": "markdown",
"source": [
"# Data Portal\n",
"We can use Calgary’s Open Data Portal to get real-time transit vehicle locations. Calgary Transit provides real-time data through the General Transit Feed Specification Realtime (GTFS-RT) format. These feeds offer live information on vehicle positions, trip updates, and service alerts.\n",
"\n",
"## Available GTFS-RT Feeds\n",
"* Vehicle Positions: Provides real-time locations of transit vehicles.\n",
"* Trip Updates: Offers real-time updates on scheduled trips, including delays and cancellations.\n",
"* Service Alerts: Contains information on disruptions or changes in service.\n",
"\n",
"## Accessing the Feeds\n",
"These feeds are accessible via Calgary’s Open Data Portal:\n",
"\n",
"* Vehicle Positions Feed — [Calgary Open Data](https://data.calgary.ca/Transportation-Transit/Calgary-Transit-Realtime-Vehicle-Positions-GTFS-RT/am7c-qe3u?utm_source=chatgpt.com)\n",
"* Trip Updates Feed — [Calgary Open Data](https://data.calgary.ca/Transportation-Transit/Calgary-Transit-Realtime-Trip-Updates-GTFS-RT/gs4m-mdc2?utm_source=chatgpt.com)\n",
"* Service Alerts — [Calgary Open Data](https://data.calgary.ca/Transportation-Transit/Calgary-Transit-Realtime-Service-Alerts-GTFS-RT/jhgn-ynqj)\n",
"\n",
"---"
],
"metadata": {
"id": "DogTaGHIfZny"
}
},
{
"cell_type": "markdown",
"source": [
"# Vehicle Positions Feed\n",
"## Handling GTFS-RT Feeds with Python\n",
"Install Required Libraries"
],
"metadata": {
"id": "KQlP0WNOgFlk"
}
},
{
"cell_type": "code",
"source": [
"pip install requests protobuf"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "E2QVMut1gLbH",
"outputId": "2b84bd07-adc0-49e9-e907-64769e02144d"
},
"execution_count": 1,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Requirement already satisfied: requests in /usr/local/lib/python3.11/dist-packages (2.32.3)\n",
"Requirement already satisfied: protobuf in /usr/local/lib/python3.11/dist-packages (4.25.5)\n",
"Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.11/dist-packages (from requests) (3.4.1)\n",
"Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.11/dist-packages (from requests) (3.10)\n",
"Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.11/dist-packages (from requests) (2.3.0)\n",
"Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.11/dist-packages (from requests) (2024.12.14)\n"
]
}
]
},
{
"cell_type": "markdown",
"source": [
"The requests library is used to send HTTP requests in Python. Whereas the protobuf is a library for working with Protocol Buffers (Protobuf), a method developed by Google for serializing structured data. Long story short, if you are handling GTFS-RT feed, then requests is used to fetch and protobuf is used to parse the data from an API.\n",
"\n",
"## Generating Python code from a Protocol Buffer\n",
"We need to use the Protocol buffer compiler protoc1, a tool provided by Google for working with .proto files. We shall use the gtfs-realtime.proto Protobuf definition file. It describes the structure of GTFS-RT messages, including FeedMessage, FeedHeader, FeedEntity, etc. The technical documentation follows below.\n",
"\n",
"[Protobuf - General Transit Feed Specification](https://gtfs.org/documentation/realtime/proto/?source=post_page-----49baea30833e--------------------------------)"
],
"metadata": {
"id": "VNGi1KNQgQoI"
}
},
{
"cell_type": "code",
"source": [
"!protoc --python_out=. gtfs-realtime.proto"
],
"metadata": {
"id": "VBlxVP1RgdUA"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"Running this command creates a Python module, which isgtfs_realtime_pb2.py. If you don’t want the hassle, you can skip this step by just manually uploading the gtfs_realtime_pb2.py to your drive. Click [here](https://github.com/MobilityData/gtfs-realtime-bindings/blob/master/python/google/transit/gtfs_realtime_pb2.py) to access the file. To upload a file on Google Colab, you can write as:"
],
"metadata": {
"id": "K8aOcJWegfcG"
}
},
{
"cell_type": "code",
"source": [
"from google.colab import files\n",
"files.upload()"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 1000
},
"id": "FqZbVs1Tgncg",
"outputId": "aec00674-33c9-4ad5-86aa-96b768be5993"
},
"execution_count": 3,
"outputs": [
{
"output_type": "display_data",
"data": {
"text/plain": [
"| \n", " | Vehicle ID | \n", "Latitude | \n", "Longitude | \n", "
|---|---|---|---|
| 0 | \n", "8241 | \n", "51.071911 | \n", "-113.981918 | \n", "
| 1 | \n", "1200 | \n", "51.057251 | \n", "-114.020523 | \n", "
| 2 | \n", "8312 | \n", "51.121761 | \n", "-114.071648 | \n", "
| 3 | \n", "8274 | \n", "51.123436 | \n", "-114.071243 | \n", "
| 4 | \n", "8254 | \n", "51.092236 | \n", "-114.033279 | \n", "
| \n", " | Trip ID | \n", "Start Time | \n", "Start Date | \n", "Stop ID | \n", "Arrival Time | \n", "Departure Time | \n", "
|---|---|---|---|---|---|---|
| 0 | \n", "69082338 | \n", "\n", " | \n", " | 6212 | \n", "2025-01-17 04:33:00 | \n", "2025-01-17 04:33:00 | \n", "
| 1 | \n", "69082338 | \n", "\n", " | \n", " | 4343 | \n", "2025-01-17 04:34:00 | \n", "2025-01-17 04:34:00 | \n", "
| 2 | \n", "69082338 | \n", "\n", " | \n", " | 6674 | \n", "2025-01-17 04:35:00 | \n", "2025-01-17 04:35:00 | \n", "
| 3 | \n", "69082338 | \n", "\n", " | \n", " | 6213 | \n", "2025-01-17 04:36:00 | \n", "2025-01-17 04:36:00 | \n", "
| 4 | \n", "69082338 | \n", "\n", " | \n", " | 4342 | \n", "2025-01-17 04:37:00 | \n", "2025-01-17 04:37:00 | \n", "
| 5 | \n", "69082338 | \n", "\n", " | \n", " | 6214 | \n", "2025-01-17 04:38:00 | \n", "2025-01-17 04:38:00 | \n", "
| 6 | \n", "69082338 | \n", "\n", " | \n", " | 6962 | \n", "2025-01-17 04:39:00 | \n", "2025-01-17 04:39:00 | \n", "
| 7 | \n", "69082338 | \n", "\n", " | \n", " | 6705 | \n", "2025-01-17 04:41:00 | \n", "2025-01-17 04:41:00 | \n", "
| 8 | \n", "69082338 | \n", "\n", " | \n", " | 6963 | \n", "2025-01-17 04:41:00 | \n", "2025-01-17 04:41:00 | \n", "
| 9 | \n", "69082338 | \n", "\n", " | \n", " | 6964 | \n", "2025-01-17 04:42:00 | \n", "2025-01-17 04:42:00 | \n", "
Starting ... \n", "1 166174
Starting Th... \n", "2 166175
\\nStart... \n", "3 166177
\\nStarting ... \n", "4 166178
\\nStarting ... \n", "\n", " Severity Level \n", "0 1 \n", "1 1 \n", "2 1 \n", "3 1 \n", "4 1 \n" ] } ] }, { "cell_type": "markdown", "source": [ "You can read the documentation of the Service Alerts below.\n", "\n", "[Calgary Transit Realtime Service Alerts GTFS - RT](https://data.calgary.ca/Transportation-Transit/Calgary-Transit-Realtime-Service-Alerts-GTFS-RT/jhgn-ynqj/about_data?source=post_page-----49baea30833e--------------------------------)\n", "\n", "Upon executing the above code snippet, I noticed that the data was not in the readable format. The descriptions were not fully displayed and contained HTML tags. So that’s why I have used BeautifulSoup library to clean this up, you can trim leading and trailing spaces or newlines from the alert text.\n", "\n" ], "metadata": { "id": "u8RJ_PxxjOop" } }, { "cell_type": "code", "source": [ "from bs4 import BeautifulSoup # Import the BeautifulSoup library to parse and clean HTML\n", "\n", "def clean_html(raw_html):\n", " \"\"\"Removes HTML tags and returns plain text.\"\"\"\n", " # Create a BeautifulSoup object to parse the HTML content\n", " soup = BeautifulSoup(raw_html, 'html.parser')\n", " # Use the get_text() method to extract and return the plain text from the HTML\n", " return soup.get_text()\n", "# Loop through each alert and clean the \"Header\" and \"Description\" fields by removing HTML tags\n", "for alert in alerts:\n", " # Apply the clean_html function to the \"Header\" field\n", " alert[\"Header\"] = clean_html(alert[\"Header\"])\n", " # Apply the clean_html function to the \"Description\" field\n", " alert[\"Description\"] = clean_html(alert[\"Description\"])\n", "# Convert the cleaned alerts into a pandas DataFrame for easy viewing and manipulation\n", "df_alerts = pd.DataFrame(alerts)\n", "# Display the first 5 rows of the DataFrame to verify the cleaned alerts\n", "print(df_alerts.head(5))\n" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "VIEXmsrUjf6c", "outputId": "bad67a5e-265d-4c24-b644-1e2d2c455c23" }, "execution_count": 18, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ " Alert ID Header Description \\\n", "0 166173 \\nStarting Thursday, January 16 at 9 a.m. thro... \n", "1 166174 Starting Thursday, January 16 at 9 a.m. throug... \n", "2 166175 \\nStarting Thursday, January 16 at 9 a.m. thro... \n", "3 166177 \\nStarting Thursday, January 16 at 9 a.m. thro... \n", "4 166178 \\nStarting Thursday, January 16 at 9 a.m. thro... \n", "\n", " Severity Level \n", "0 1 \n", "1 1 \n", "2 1 \n", "3 1 \n", "4 1 \n" ] } ] }, { "cell_type": "markdown", "source": [ "# Output\n", "You might be asking yourself, “Can I associate this with the trip_id and vehicle_id”. The answer is Yes, by incorporating that information into your data processing, you can link the alerts to particular trip_id and vehicle_id. Every alert should be connected to the appropriate trip and vehicle. Once again, it’s a moving piece of the puzzle. Once the data is completely available for the public without encapsulation, this can be possible." ], "metadata": { "id": "i6yh1MbCjkL_" } }, { "cell_type": "code", "source": [ "df_alerts.head(5)" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 507 }, "id": "W_5wtfRCjlxa", "outputId": "4ff0413f-534f-4151-8920-4526d5fdac16" }, "execution_count": 19, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ " Alert ID Header Description \\\n", "0 166173 \\nStarting Thursday, January 16 at 9 a.m. thro... \n", "1 166174 Starting Thursday, January 16 at 9 a.m. throug... \n", "2 166175 \\nStarting Thursday, January 16 at 9 a.m. thro... \n", "3 166177 \\nStarting Thursday, January 16 at 9 a.m. thro... \n", "4 166178 \\nStarting Thursday, January 16 at 9 a.m. thro... \n", "\n", " Severity Level \n", "0 1 \n", "1 1 \n", "2 1 \n", "3 1 \n", "4 1 " ], "text/html": [ "\n", "
| \n", " | Alert ID | \n", "Header | \n", "Description | \n", "Severity Level | \n", "
|---|---|---|---|---|
| 0 | \n", "166173 | \n", "\n", " | \\nStarting Thursday, January 16 at 9 a.m. thro... | \n", "1 | \n", "
| 1 | \n", "166174 | \n", "\n", " | Starting Thursday, January 16 at 9 a.m. throug... | \n", "1 | \n", "
| 2 | \n", "166175 | \n", "\n", " | \\nStarting Thursday, January 16 at 9 a.m. thro... | \n", "1 | \n", "
| 3 | \n", "166177 | \n", "\n", " | \\nStarting Thursday, January 16 at 9 a.m. thro... | \n", "1 | \n", "
| 4 | \n", "166178 | \n", "\n", " | \\nStarting Thursday, January 16 at 9 a.m. thro... | \n", "1 | \n", "