\n
![\"sv-image\"](\"https://raw.githubusercontent.com/roberthryniewicz/images/master/silicon_valley_corporation.jpg\")
\n
Apache Spark in 5 Minutes
\n
Exploring Silicon Valley Show Episodes Dataset
\n
Level: Beginner
Language: Scala
Requirements:
- HDP 2.6 (or later) or HDCloud
- Spark 2.x
\n
Author: Robert Hryniewicz
Follow @RobH8z
\n
\n
Welcome to a quick overview of Apache Spark with Sillicon Valley Episodes dataset. If you’ve never watched the Silicon Valley show you can learn more about it [here](https://en.wikipedia.org/wiki/Silicon_Valley_(TV_series)).
\n
In this notebook we will download the dataset (in JSON format) from an external github repository, ingest it into a Spark Dataset and perform basic analysis, filtering, and word count.
\n
\n
Throughout this lab we will use basic Scala syntax. If you would like to learn more about Scala, here’s an excellent introductory Tutorial.
\n
\n
If you haven’t already, checkout the Hortonworks Apache Zeppelin page as well as the Getting Started with Apache Zeppelin tutorial.
\n
You will find the official Apache Zeppelin page here.
\n
\n
Apache Spark is a fast, in-memory data processing engine with elegant and expressive development APIs to allow data workers to efficiently execute streaming, machine learning or SQL workloads that require fast iterative access to datasets.
\n
If you would like to learn more about Apache Spark visit:
- Official Apache Spark Page
- Hortonworks Apache Spark Page
- Hortonworks Apache Spark Docs
\n
\n
To run a paragraph in a Zeppelin notebook you can either click the play button (blue triangle) on the right-hand side or simply press Shift + Enter.
\n
\n
In the following paragraphs we are going to execute Spark code, run shell commands to download and move files, run sql queries etc. Each paragraph will start with % followed by an interpreter name, e.g. %spark2 for a Spark 2.x interpreter. Different interpreter names indicate what will be executed: code, markdown, html etc.This allows you to perform data ingestion, munging, wrangling, visualization, analysis, processing and more, all in one place!
\n
Throughtout this notebook we will use the following interpreters:
\n
\n %spark2 - Spark interpreter to run Spark 2.x code written in Scala%spark2.sql - Spark SQL interprter (to execute SQL queries against temporary tables in Spark)%sh - Shell interpreter to run shell commands%angular - Angular interpreter to run Angular and HTML code%md - Markdown for displaying formatted text, links, and images
\n
To learn more about Zeppelin interpreters check out this link.
\n
\n
Note: The first time you run spark.version in the paragraph below, several services will initialize in the background.
This may take 1~2 min so please be patient. Afterwards, each paragraph should run much more quickly since all the services will already be running.
\n
\n
Datasets and DataFrames are distributed collections of data created from a variety of sources: JSON and XML files, tables in Hive, external databases and more. Conceptually, they are equivalent to a table in a relational database or a DataFrame in R or Python. Key difference between the Dataset and the DataFrame is that Datasets are strongly typed.
\n
There are complex manipulations possible on Datasets and DataFrames, however they are beyond this quick guide.
\n
To learn more about Datasets and DataFrames checkout this link.
\n