{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "\n", "# Getting started with machine learning pipelines in PySpark\n", "> PySpark has built-in, cutting-edge machine learning routines, along with utilities to create full machine learning pipelines. You'll learn about them in this chapter. This is the Summary of lecture \"Introduction to PySpark\", via datacamp.\n", "\n", "- toc: true \n", "- badges: true\n", "- comments: true\n", "- author: Chanseok Kang\n", "- categories: [Python, Datacamp, PySpark, Machine_Learning]\n", "- image: images/spark.png" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import pyspark\n", "import numpy as np\n", "import pandas as pd" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Machine Learning Pipelines\n", "At the core of the `pyspark.ml` module are the `Transformer` and `Estimator` classes. Almost every other class in the module behaves similarly to these two basic classes.\n", "\n", "`Transformer` classes have a `.transform()` method that takes a DataFrame and returns a new DataFrame; usually the original one with a new column appended. For example, you might use the class `Bucketizer` to create discrete bins from a continuous feature or the class PCA to reduce the dimensionality of your dataset using principal component analysis.\n", "\n", "Estimator classes all implement a `.fit()` method. These methods also take a DataFrame, but instead of returning another DataFrame they return a model object. This can be something like a `StringIndexerModel` for including categorical data saved as strings in your models, or a `RandomForestModel` that uses the random forest algorithm for classification or regression." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Join the DataFrames\n", "In the next two chapters you'll be working to build a model that predicts whether or not a flight will be delayed based on the flights data we've been working with. This model will also include information about the plane that flew the route, so the first step is to join the two tables: `flights` and `planes`!" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [], "source": [ "from pyspark.sql import SparkSession\n", "\n", "spark = (SparkSession\n", " .builder\n", " .appName(\"flights\")\n", " .getOrCreate())\n", "\n", "# Read and create a temporary view\n", "# Infer schema (note that for larger files you \n", "# may want to specify the schema)\n", "flights = (spark.read.format(\"csv\")\n", " .option(\"inferSchema\", \"true\")\n", " .option(\"header\", \"true\")\n", " .load(\"./dataset/flights_small.csv\"))\n", "flights.createOrReplaceTempView(\"flights\")\n", "\n", "planes = (spark.read.format(\"csv\")\n", " .option(\"inferSchema\", \"true\")\n", " .option(\"header\", \"true\")\n", " .load('./dataset/planes.csv'))\n", "planes.createOrReplaceTempView(\"planes\")" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [], "source": [ "# Rename year column\n", "planes = planes.withColumnRenamed('year', 'plane_year')\n", "\n", "# Join the DataFrame\n", "model_data = flights.join(planes, on='tailnum', how='leftouter')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Data types\n", "Good work! Before you get started modeling, it's important to know that Spark only handles numeric data. That means all of the columns in your DataFrame must be either integers or decimals (called 'doubles' in Spark).\n", "\n", "When we imported our data, we let Spark guess what kind of information each column held. Unfortunately, Spark doesn't always guess right and you can see that some of the columns in our DataFrame are strings containing numbers as opposed to actual numeric values.\n", "\n", "To remedy this, you can use the `.cast()` method in combination with the `.withColumn()` method. It's important to note that `.cast()` works on columns, while `.withColumn()` works on DataFrames.\n", "\n", "The only argument you need to pass to `.cast()` is the kind of value you want to create, in string form. For example, to create integers, you'll pass the argument `\"integer\"` and for decimal numbers you'll use `\"double\"`." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## String to integer\n", "Now you'll use the `.cast()` method you learned in the previous exercise to convert all the appropriate columns from your DataFrame model_data to integers!\n", "\n", "To convert the type of a column using the `.cast()` method, you can write code like this:\n", "```python\n", "dataframe = dataframe.withColumn(\"col\", dataframe.col.cast(\"new_type\"))\n", "```" ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [], "source": [ "# Cast the columns to integers\n", "model_data = model_data.withColumn(\"arr_delay\", model_data.arr_delay.cast('integer'))\n", "model_data = model_data.withColumn('air_time', model_data.air_time.cast('integer'))\n", "model_data = model_data.withColumn('month', model_data.month.cast('integer'))\n", "model_data = model_data.withColumn('plane_year', model_data.plane_year.cast('integer'))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Create a new column\n", "In the last exercise, you converted the column `plane_year` to an integer. This column holds the year each plane was manufactured. However, your model will use the planes' age, which is slightly different from the year it was made!\n", "\n" ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [], "source": [ "# Create the column plane_age\n", "model_data = model_data.withColumn('plane_age', model_data.year - model_data.plane_year)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Making a Boolean\n", "Consider that you're modeling a yes or no question: is the flight late? However, your data contains the arrival delay in minutes for each flight. Thus, you'll need to create a boolean column which indicates whether the flight was late or not!\n", "\n" ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [], "source": [ "# Create is_late\n", "model_data = model_data.withColumn('is_late', model_data.arr_delay > 0)\n", "\n", "# Convert to an integer\n", "model_data = model_data.withColumn('label', model_data.is_late.cast('integer'))\n", "\n", "# Remove missing values\n", "model_data = model_data.filter('arr_delay is not NULL and dep_delay is not NULL and \\\n", " air_time is not NULL and plane_year is not NULL')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Strings and factors\n", "As you know, Spark requires numeric data for modeling. So far this hasn't been an issue; even boolean columns can easily be converted to integers without any trouble. But you'll also be using the airline and the plane's destination as features in your model. These are coded as strings and there isn't any obvious way to convert them to a numeric data type.\n", "\n", "Fortunately, PySpark has functions for handling this built into the `pyspark.ml.features` submodule. You can create what are called 'one-hot vectors' to represent the carrier and the destination of each flight. A one-hot vector is a way of representing a categorical feature where every observation has a vector in which all elements are zero except for at most one element, which has a value of one (1).\n", "\n", "Each element in the vector corresponds to a level of the feature, so it's possible to tell what the right level is by seeing which element of the vector is equal to one (1).\n", "\n", "The first step to encoding your categorical feature is to create a `StringIndexer`. Members of this class are `Estimators` that take a DataFrame with a column of strings and map each unique string to a number. Then, the `Estimator` returns a `Transformer` that takes a DataFrame, attaches the mapping to it as metadata, and returns a new DataFrame with a numeric column corresponding to the string column.\n", "\n", "The second step is to encode this numeric column as a one-hot vector using a `OneHotEncoder`. This works exactly the same way as the `StringIndexer` by creating an `Estimator` and then a `Transformer`. The end result is a column that encodes your categorical feature as a vector that's suitable for machine learning routines!\n", "\n", "This may seem complicated, but don't worry! All you have to remember is that you need to create a `StringIndexer` and a `OneHotEncoder`, and the Pipeline will take care of the rest." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Carrier\n", "In this exercise you'll create a `StringIndexer` and a `OneHotEncoder` to code the `carrier` column. To do this, you'll call the class constructors with the arguments `inputCol` and `outputCol`.\n", "\n", "The `inputCol` is the name of the column you want to index or encode, and the `outputCol` is the name of the new column that the `Transformer` should create." ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [], "source": [ "from pyspark.ml.feature import StringIndexer, OneHotEncoder\n", "\n", "# Create StringIndexer\n", "carr_indexer = StringIndexer(inputCol='carrier', outputCol='carrier_index')\n", "\n", "# Create a OneHotEncoder\n", "carr_encoder = OneHotEncoder(inputCol='carrier_index', outputCol='carrier_fact')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Destination\n", "Now you'll encode the `dest` column just like you did in the previous exercise.\n", "\n" ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [], "source": [ "# Create a StringIndexer\n", "dest_indexer = StringIndexer(inputCol='dest', outputCol='dest_index')\n", "\n", "# Create a OneHotEncoder\n", "dest_encoder = OneHotEncoder(inputCol='dest_index', outputCol='dest_fact')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Assemble a vector\n", "The last step in the `Pipeline` is to combine all of the columns containing our features into a single column. This has to be done before modeling can take place because every Spark modeling routine expects the data to be in this form. You can do this by storing each of the values from a column as an entry in a vector. Then, from the model's point of view, every observation is a vector that contains all of the information about it and a label that tells the modeler what value that observation corresponds to.\n", "\n", "Because of this, the `pyspark.ml.feature` submodule contains a class called `VectorAssembler`. This `Transformer` takes all of the columns you specify and combines them into a new vector column.\n", "\n" ] }, { "cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [], "source": [ "from pyspark.ml.feature import VectorAssembler\n", "\n", "# Make a VectorAssembler\n", "vec_assembler = VectorAssembler(inputCols=['month', 'air_time', 'carrier_fact', 'dest_fact', 'plane_age'],\n", " outputCol='features')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Create the pipeline\n", "You're finally ready to create a Pipeline!" ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [], "source": [ "from pyspark.ml import Pipeline\n", "\n", "# Make the pipeline\n", "flights_pipe = Pipeline(stages=[dest_indexer, dest_encoder, carr_indexer, carr_encoder, vec_assembler])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Test vs Train\n", "After you've cleaned your data and gotten it ready for modeling, one of the most important steps is to split the data into a test set and a train set. After that, don't touch your test data until you think you have a good model! As you're building models and forming hypotheses, you can test them on your training data to get an idea of their performance.\n", "\n", "Once you've got your favorite model, you can see how well it predicts the new data in your test set. This never-before-seen data will give you a much more realistic idea of your model's performance in the real world when you're trying to predict or classify new data.\n", "\n", "In Spark it's important to make sure you split the data after all the transformations. This is because operations like `StringIndexer` don't always produce the same index even when given the same list of strings." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Transform the data\n", "Hooray, now you're finally ready to pass your data through the `Pipeline` you created!\n", "\n" ] }, { "cell_type": "code", "execution_count": 27, "metadata": {}, "outputs": [], "source": [ "# Fit and transform the data\n", "piped_data = flights_pipe.fit(model_data).transform(model_data)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Split the data\n", "Now that you've done all your manipulations, the last step before modeling is to split the data!" ] }, { "cell_type": "code", "execution_count": 28, "metadata": {}, "outputs": [], "source": [ "# Split the data into training and test sets\n", "training, test = piped_data.randomSplit([.6, .4])" ] }, { "cell_type": "code", "execution_count": 29, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "+-------+----+-----+---+--------+---------+--------+---------+-------+------+------+----+--------+--------+----+------+----------+--------------------+----------------+--------+-------+-----+-----+---------+---------+-------+-----+----------+---------------+-------------+--------------+--------------------+\n", "|tailnum|year|month|day|dep_time|dep_delay|arr_time|arr_delay|carrier|flight|origin|dest|air_time|distance|hour|minute|plane_year| type| manufacturer| model|engines|seats|speed| engine|plane_age|is_late|label|dest_index| dest_fact|carrier_index| carrier_fact| features|\n", "+-------+----+-----+---+--------+---------+--------+---------+-------+------+------+----+--------+--------+----+------+----------+--------------------+----------------+--------+-------+-----+-----+---------+---------+-------+-----+----------+---------------+-------------+--------------+--------------------+\n", "| N102UW|2014| 5| 7| 1311| 6| 2115| 2| US| 1971| SEA| CLT| 274| 2279| 13| 11| 1998|Fixed wing multi ...|AIRBUS INDUSTRIE|A320-214| 2| 182| NA|Turbo-fan| 16| true| 1| 33.0|(68,[33],[1.0])| 5.0|(10,[5],[1.0])|(81,[0,1,7,45,80]...|\n", "| N102UW|2014| 11| 9| 2220| -5| 555| -11| US| 1930| PDX| CLT| 257| 2282| 22| 20| 1998|Fixed wing multi ...|AIRBUS INDUSTRIE|A320-214| 2| 182| NA|Turbo-fan| 16| false| 0| 33.0|(68,[33],[1.0])| 5.0|(10,[5],[1.0])|(81,[0,1,7,45,80]...|\n", "| N105UW|2014| 3| 13| 1325| 5| 2123| 13| US| 1805| SEA| CLT| 261| 2279| 13| 25| 1999|Fixed wing multi ...|AIRBUS INDUSTRIE|A320-214| 2| 182| NA|Turbo-fan| 15| true| 1| 33.0|(68,[33],[1.0])| 5.0|(10,[5],[1.0])|(81,[0,1,7,45,80]...|\n", "| N107US|2014| 4| 2| 1316| -4| 2114| 4| US| 1805| SEA| CLT| 267| 2279| 13| 16| 1999|Fixed wing multi ...|AIRBUS INDUSTRIE|A320-214| 2| 182| NA|Turbo-fan| 15| true| 1| 33.0|(68,[33],[1.0])| 5.0|(10,[5],[1.0])|(81,[0,1,7,45,80]...|\n", "| N107US|2014| 5| 15| 1058| -2| 1845| -23| US| 2092| SEA| CLT| 264| 2279| 10| 58| 1999|Fixed wing multi ...|AIRBUS INDUSTRIE|A320-214| 2| 182| NA|Turbo-fan| 15| false| 0| 33.0|(68,[33],[1.0])| 5.0|(10,[5],[1.0])|(81,[0,1,7,45,80]...|\n", "| N107US|2014| 9| 14| 509| -6| 751| -8| US| 480| SEA| PHX| 143| 1107| 5| 9| 1999|Fixed wing multi ...|AIRBUS INDUSTRIE|A320-214| 2| 182| NA|Turbo-fan| 15| false| 0| 4.0| (68,[4],[1.0])| 5.0|(10,[5],[1.0])|(81,[0,1,7,16,80]...|\n", "| N110UW|2014| 7| 16| 101| 16| 901| 15| US| 1917| SEA| PHL| 288| 2378| 1| 1| 1999|Fixed wing multi ...|AIRBUS INDUSTRIE|A320-214| 2| 182| NA|Turbo-fan| 15| true| 1| 32.0|(68,[32],[1.0])| 5.0|(10,[5],[1.0])|(81,[0,1,7,44,80]...|\n", "| N111US|2014| 3| 14| 1319| -1| 2101| -9| US| 1805| SEA| CLT| 256| 2279| 13| 19| 1999|Fixed wing multi ...|AIRBUS INDUSTRIE|A320-214| 2| 182| NA|Turbo-fan| 15| false| 0| 33.0|(68,[33],[1.0])| 5.0|(10,[5],[1.0])|(81,[0,1,7,45,80]...|\n", "| N111US|2014| 4| 29| 2216| 1| 620| 10| US| 1930| PDX| CLT| 271| 2282| 22| 16| 1999|Fixed wing multi ...|AIRBUS INDUSTRIE|A320-214| 2| 182| NA|Turbo-fan| 15| true| 1| 33.0|(68,[33],[1.0])| 5.0|(10,[5],[1.0])|(81,[0,1,7,45,80]...|\n", "| N112US|2014| 2| 1| 833| -2| 1612| -27| US| 788| SEA| PHL| 260| 2378| 8| 33| 1999|Fixed wing multi ...|AIRBUS INDUSTRIE|A320-214| 2| 182| NA|Turbo-fan| 15| false| 0| 32.0|(68,[32],[1.0])| 5.0|(10,[5],[1.0])|(81,[0,1,7,44,80]...|\n", "| N112US|2014| 6| 30| 1101| 1| 1851| -7| US| 2092| SEA| CLT| 269| 2279| 11| 1| 1999|Fixed wing multi ...|AIRBUS INDUSTRIE|A320-214| 2| 182| NA|Turbo-fan| 15| false| 0| 33.0|(68,[33],[1.0])| 5.0|(10,[5],[1.0])|(81,[0,1,7,45,80]...|\n", "| N112US|2014| 7| 19| 43| -2| 845| -1| US| 2051| SEA| PHL| 281| 2378| 0| 43| 1999|Fixed wing multi ...|AIRBUS INDUSTRIE|A320-214| 2| 182| NA|Turbo-fan| 15| false| 0| 32.0|(68,[32],[1.0])| 5.0|(10,[5],[1.0])|(81,[0,1,7,44,80]...|\n", "| N117UW|2014| 6| 28| 2209| -6| 605| 4| US| 1930| PDX| CLT| 282| 2282| 22| 9| 2000|Fixed wing multi ...|AIRBUS INDUSTRIE|A320-214| 2| 182| NA|Turbo-fan| 14| true| 1| 33.0|(68,[33],[1.0])| 5.0|(10,[5],[1.0])|(81,[0,1,7,45,80]...|\n", "| N1201P|2014| 7| 28| 1054| -1| 1836| -3| DL| 716| PDX| ATL| 239| 2172| 10| 54| 1998|Fixed wing multi ...| BOEING| 767-332| 2| 330| NA|Turbo-fan| 16| false| 0| 12.0|(68,[12],[1.0])| 3.0|(10,[3],[1.0])|(81,[0,1,5,24,80]...|\n", "| N1201P|2014| 9| 23| 1651| 370| 31| 371| DL| 1702| PDX| ATL| 249| 2172| 16| 51| 1998|Fixed wing multi ...| BOEING| 767-332| 2| 330| NA|Turbo-fan| 16| true| 1| 12.0|(68,[12],[1.0])| 3.0|(10,[3],[1.0])|(81,[0,1,5,24,80]...|\n", "| N121DE|2014| 9| 1| 2317| 2| 605| -22| DL| 910| SEA| DTW| 205| 1927| 23| 17| 1987|Fixed wing multi ...| BOEING| 767-332| 2| 330| NA|Turbo-fan| 27| false| 0| 25.0|(68,[25],[1.0])| 3.0|(10,[3],[1.0])|(81,[0,1,5,37,80]...|\n", "| N121UW|2014| 11| 6| 2226| 1| 552| -14| US| 1930| PDX| CLT| 246| 2282| 22| 26| 2000|Fixed wing multi ...|AIRBUS INDUSTRIE|A320-214| 2| 182| NA|Turbo-fan| 14| false| 0| 33.0|(68,[33],[1.0])| 5.0|(10,[5],[1.0])|(81,[0,1,7,45,80]...|\n", "| N12221|2014| 8| 30| 1142| -1| 1744| -17| UA| 1246| SEA| IAH| 218| 1874| 11| 42| 1998|Fixed wing multi ...| BOEING| 737-824| 2| 149| NA|Turbo-fan| 16| false| 0| 14.0|(68,[14],[1.0])| 4.0|(10,[4],[1.0])|(81,[0,1,6,26,80]...|\n", "| N12225|2014| 11| 24| 707| -3| 916| -16| UA| 1732| SEA| SFO| 89| 679| 7| 7| 1998|Fixed wing multi ...| BOEING| 737-824| 2| 149| NA|Turbo-fan| 16| false| 0| 0.0| (68,[0],[1.0])| 4.0|(10,[4],[1.0])|(81,[0,1,6,12,80]...|\n", "| N12238|2014| 9| 22| 2326| -4| 505| -13| UA| 1538| PDX| ORD| 203| 1739| 23| 26| 1999|Fixed wing multi ...| BOEING| 737-824| 2| 149| NA|Turbo-fan| 15| false| 0| 11.0|(68,[11],[1.0])| 4.0|(10,[4],[1.0])|(81,[0,1,6,23,80]...|\n", "+-------+----+-----+---+--------+---------+--------+---------+-------+------+------+----+--------+--------+----+------+----------+--------------------+----------------+--------+-------+-----+-----+---------+---------+-------+-----+----------+---------------+-------------+--------------+--------------------+\n", "only showing top 20 rows\n", "\n" ] } ], "source": [ "training.show()" ] }, { "cell_type": "code", "execution_count": 30, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "+-------+----+-----+---+--------+---------+--------+---------+-------+------+------+----+--------+--------+----+------+----------+--------------------+----------------+--------+-------+-----+-----+---------+---------+-------+-----+----------+---------------+-------------+--------------+--------------------+\n", "|tailnum|year|month|day|dep_time|dep_delay|arr_time|arr_delay|carrier|flight|origin|dest|air_time|distance|hour|minute|plane_year| type| manufacturer| model|engines|seats|speed| engine|plane_age|is_late|label|dest_index| dest_fact|carrier_index| carrier_fact| features|\n", "+-------+----+-----+---+--------+---------+--------+---------+-------+------+------+----+--------+--------+----+------+----------+--------------------+----------------+--------+-------+-----+-----+---------+---------+-------+-----+----------+---------------+-------------+--------------+--------------------+\n", "| N108UW|2014| 6| 17| 1107| 7| 1906| 8| US| 2092| SEA| CLT| 273| 2279| 11| 7| 1999|Fixed wing multi ...|AIRBUS INDUSTRIE|A320-214| 2| 182| NA|Turbo-fan| 15| true| 1| 33.0|(68,[33],[1.0])| 5.0|(10,[5],[1.0])|(81,[0,1,7,45,80]...|\n", "| N109UW|2014| 4| 20| 2209| -6| 548| -22| US| 1930| PDX| CLT| 266| 2282| 22| 9| 1999|Fixed wing multi ...|AIRBUS INDUSTRIE|A320-214| 2| 182| NA|Turbo-fan| 15| false| 0| 33.0|(68,[33],[1.0])| 5.0|(10,[5],[1.0])|(81,[0,1,7,45,80]...|\n", "| N109UW|2014| 5| 2| 2214| -1| 546| -24| US| 1930| PDX| CLT| 255| 2282| 22| 14| 1999|Fixed wing multi ...|AIRBUS INDUSTRIE|A320-214| 2| 182| NA|Turbo-fan| 15| false| 0| 33.0|(68,[33],[1.0])| 5.0|(10,[5],[1.0])|(81,[0,1,7,45,80]...|\n", "| N110UW|2014| 5| 24| 2220| 5| 602| 1| US| 1930| PDX| CLT| 263| 2282| 22| 20| 1999|Fixed wing multi ...|AIRBUS INDUSTRIE|A320-214| 2| 182| NA|Turbo-fan| 15| true| 1| 33.0|(68,[33],[1.0])| 5.0|(10,[5],[1.0])|(81,[0,1,7,45,80]...|\n", "| N112US|2014| 1| 30| 835| 0| 1630| -9| US| 1935| SEA| PHL| 276| 2378| 8| 35| 1999|Fixed wing multi ...|AIRBUS INDUSTRIE|A320-214| 2| 182| NA|Turbo-fan| 15| false| 0| 32.0|(68,[32],[1.0])| 5.0|(10,[5],[1.0])|(81,[0,1,7,44,80]...|\n", "| N117UW|2014| 8| 25| 833| -2| 1643| -1| US| 798| SEA| PHL| 290| 2378| 8| 33| 2000|Fixed wing multi ...|AIRBUS INDUSTRIE|A320-214| 2| 182| NA|Turbo-fan| 14| false| 0| 32.0|(68,[32],[1.0])| 5.0|(10,[5],[1.0])|(81,[0,1,7,44,80]...|\n", "| N119US|2014| 10| 2| 2208| -7| 548| -12| US| 1930| PDX| CLT| 260| 2282| 22| 8| 2000|Fixed wing multi ...|AIRBUS INDUSTRIE|A320-214| 2| 182| NA|Turbo-fan| 14| false| 0| 33.0|(68,[33],[1.0])| 5.0|(10,[5],[1.0])|(81,[0,1,7,45,80]...|\n", "| N121DE|2014| 8| 13| 2311| -4| 611| -15| DL| 1823| SEA| DTW| 216| 1927| 23| 11| 1987|Fixed wing multi ...| BOEING| 767-332| 2| 330| NA|Turbo-fan| 27| false| 0| 25.0|(68,[25],[1.0])| 3.0|(10,[3],[1.0])|(81,[0,1,5,37,80]...|\n", "| N12221|2014| 12| 30| 1554| 49| 1807| 54| UA| 1717| SEA| SFO| 99| 679| 15| 54| 1998|Fixed wing multi ...| BOEING| 737-824| 2| 149| NA|Turbo-fan| 16| true| 1| 0.0| (68,[0],[1.0])| 4.0|(10,[4],[1.0])|(81,[0,1,6,12,80]...|\n", "| N12238|2014| 6| 20| 734| 19| 1056| 11| UA| 1660| PDX| DEN| 123| 991| 7| 34| 1999|Fixed wing multi ...| BOEING| 737-824| 2| 149| NA|Turbo-fan| 15| true| 1| 2.0| (68,[2],[1.0])| 4.0|(10,[4],[1.0])|(81,[0,1,6,14,80]...|\n", "| N122US|2014| 5| 19| 1057| -3| 1847| -21| US| 2092| SEA| CLT| 260| 2279| 10| 57| 2000|Fixed wing multi ...|AIRBUS INDUSTRIE|A320-214| 2| 182| NA|Turbo-fan| 14| false| 0| 33.0|(68,[33],[1.0])| 5.0|(10,[5],[1.0])|(81,[0,1,7,45,80]...|\n", "| N123UW|2014| 2| 27| 1313| -7| 2049| -21| US| 1805| SEA| CLT| 258| 2279| 13| 13| 2000|Fixed wing multi ...|AIRBUS INDUSTRIE|A320-214| 2| 182| NA|Turbo-fan| 14| false| 0| 33.0|(68,[33],[1.0])| 5.0|(10,[5],[1.0])|(81,[0,1,7,45,80]...|\n", "| N124DE|2014| 6| 27| 1135| 5| 1641| -1| DL| 588| SEA| MSP| 168| 1399| 11| 35| 1987|Fixed wing multi ...| BOEING| 767-332| 2| 330| NA|Turbo-fan| 27| false| 0| 13.0|(68,[13],[1.0])| 3.0|(10,[3],[1.0])|(81,[0,1,5,25,80]...|\n", "| N125DL|2014| 6| 7| 42| -3| 556| 5| DL| 2440| SEA| MSP| 173| 1399| 0| 42| 1988|Fixed wing multi ...| BOEING| 767-332| 2| 330| NA|Turbo-fan| 26| true| 1| 13.0|(68,[13],[1.0])| 3.0|(10,[3],[1.0])|(81,[0,1,5,25,80]...|\n", "| N125DL|2014| 7| 21| 859| -1| 1641| -6| DL| 108| SEA| ATL| 250| 2182| 8| 59| 1988|Fixed wing multi ...| BOEING| 767-332| 2| 330| NA|Turbo-fan| 26| false| 0| 12.0|(68,[12],[1.0])| 3.0|(10,[3],[1.0])|(81,[0,1,5,24,80]...|\n", "| N125DL|2014| 9| 30| 731| 1| 1521| 5| DL| 1808| SEA| ATL| 251| 2182| 7| 31| 1988|Fixed wing multi ...| BOEING| 767-332| 2| 330| NA|Turbo-fan| 26| true| 1| 12.0|(68,[12],[1.0])| 3.0|(10,[3],[1.0])|(81,[0,1,5,24,80]...|\n", "| N126UW|2014| 5| 4| 1301| -4| 2031| -42| US| 1952| SEA| CLT| 241| 2279| 13| 1| 2009|Fixed wing multi ...| AIRBUS|A320-214| 2| 182| NA|Turbo-fan| 5| false| 0| 33.0|(68,[33],[1.0])| 5.0|(10,[5],[1.0])|(81,[0,1,7,45,80]...|\n", "| N126UW|2014| 5| 7| 1053| -7| 1852| -16| US| 2092| SEA| CLT| 273| 2279| 10| 53| 2009|Fixed wing multi ...| AIRBUS|A320-214| 2| 182| NA|Turbo-fan| 5| false| 0| 33.0|(68,[33],[1.0])| 5.0|(10,[5],[1.0])|(81,[0,1,7,45,80]...|\n", "| N127DL|2014| 7| 19| 1314| -4| 1824| -6| DL| 198| SEA| MSP| 165| 1399| 13| 14| 1988|Fixed wing multi ...| BOEING| 767-332| 2| 330| NA|Turbo-fan| 26| false| 0| 13.0|(68,[13],[1.0])| 3.0|(10,[3],[1.0])|(81,[0,1,5,25,80]...|\n", "| N127UW|2014| 3| 6| 2219| -6| 603| -1| US| 1930| PDX| CLT| 268| 2282| 22| 19| 2010|Fixed wing multi ...| AIRBUS|A320-214| 2| 182| NA|Turbo-fan| 4| false| 0| 33.0|(68,[33],[1.0])| 5.0|(10,[5],[1.0])|(81,[0,1,7,45,80]...|\n", "+-------+----+-----+---+--------+---------+--------+---------+-------+------+------+----+--------+--------+----+------+----------+--------------------+----------------+--------+-------+-----+-----+---------+---------+-------+-----+----------+---------------+-------------+--------------+--------------------+\n", "only showing top 20 rows\n", "\n" ] } ], "source": [ "test.show()" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.6" } }, "nbformat": 4, "nbformat_minor": 4 }