\n
Intro to Machine Learning
\n
with Linear Regression
\n
Level: Beginner
Language: Scala
Requirements:
- HDP 2.6 (or later) or HDCloud
- Spark 2.x
\n
Author: Robert Hryniewicz
Follow @RobH8z
\n
\n
In this lab we’ll cover basics of building a Linear Regression model using Apache Spark ML Pipeline API.
\n
\n - Starting from a simple 2 dim array
\n - Using Pipeline API to create vectorised version of features and build the model
\n - Using Pipeline API to calculate predictions
\n - Exchanging data between Scala and Python pandas via TempView (new API in 2.x)
\n - Simplified plotting by using pandas plot function (pandas similar as Spark DataFrame)
\n - Saving and loading back the model
\n
\n
\n
In this lab we will use basic Scala syntax. If you would like to learn more about Scala, here’s an excellent Tutorial.
\n
\n
To run a paragraph in a Zeppelin notebook you can either click the play button (blue triangle) on the right-hand side or simply press Shift + Enter.
\n
\n
A model is a mathematical formula with a number of parameters that need to be learned from the data. Fitting a model to the data is a process known as model training.
\n
Take, for instance one feature/variable linear regression, where a goal is to fit a line (described by the well know eqution y = ax + b) to a set of distributed data points.
\n
For example, assume that once model training is complete we get a model equation y = 2x + 5. Then for a set of inputs [1, 0, 7, 2, …] we would get a set of outputs [7, 5, 19, 9, …]. That’s it!
\n
In this notebook you will get a chance to learn a step-by-step process of training a one variable linear regression model with Spark.
\n
\n
We’re introducing Machine Learning with Linear Regression because it’s one of the more basic and commonly used predictive analytics method. It’s also easy to explain and grasp intuitively as you’ll make your way through the examples.
\n
Note, that we will not cover the details of how the underlying Linear Regression algorithm works. We will merely focus on applying the algorithm and generating a model. If you would like to learn more about Linear Regression and other algorithms check out this excellent Coursera Machine Learning Course taught by Andrew Ng.
\n
\n
Visualise the model and training data
\n
Note: The following paragraphs require the Python Pandas library which is not installed by default. Instead, we’ve ran the paragraphs for you and disabled run so you will avoid any errors.
\n
\n
In this lab we have looked at Linear Regression, but there are other popular algorithms. In the following labs we’ll begin exploring:
\n
\n
\n
We hope you’ve enjoyed this introductory lab. Below are additional resources that you should find useful:
\n
\n - Hortonworks Apache Spark Tutorials are your natural next step where you can explore Spark in more depth.
\n - Hortonworks Community Connection (HCC) is a great resource for questions and answers on Spark, Data Analytics/Science, and many more Big Data topics.
\n - Hortonworks Apache Spark Docs - official Spark documentation.
\n - Hortonworks Apache Zeppelin Docs - official Zeppelin documentation.
\n
\n
![\"HCC\"](\"http://hortonworks.com/wp-content/uploads/2016/03/logo-hcc.png\"){kind=logo}
\n\n