[](https://github.com/groda/big_data/actions/workflows/run-jupyter-notebooks.yml)
[](https://github.com/groda/big_data/actions/workflows/link-check.yml)
[](https://github.com/groda/big_data/commits/main)
[](https://opensource.org/licenses/MIT)
# Big Data for beginners
Explore a variety of tutorials and interactive demonstrations focused on Big Data technologies like Hadoop, Spark, and more, primarily presented in the format of Jupyter notebooks. Most notebooks are self-contained, with instructions for installing all required services. They can be run on Google Colab or in a virtual Ubuntu machine/container.
## Setting Up Hadoop: Single-Node Configuration
- **[Hadoop_Setting_up_a_Single_Node_Cluster.ipynb](Hadoop_Setting_up_a_Single_Node_Cluster.ipynb)**
Set up a single-node Hadoop cluster on Google Colab and run some basic HDFS and MapReduce examples
- **[Hadoop_single_node_cluster_setup_Python.ipynb](Hadoop_single_node_cluster_setup_Python.ipynb)** Set up a single-node Hadoop cluster on Google Colab using Python 
- **[Hadoop_minicluster.ipynb](Hadoop_minicluster.ipynb)** Deploy a test Hadoop Cluster with a single command and no need for configuration. 
## Running Apache Spark in Standalone Mode
- **[Setting_up_Spark_Standalone_on_Google_Colab.ipynb](Setting_up_Spark_Standalone_on_Google_Colab.ipynb)** Set up a single-node Spark server on Google Colab and estimate „π“ with a Montecarlo method
- **[Setting_up_Spark_Standalone_on_Google_Colab_BigtopEdition.ipynb](Setting_up_Spark_Standalone_on_Google_Colab_BigtopEdition.ipynb)** Set up a single-node Spark server on Google Colab using the Bigtop distribution and utilities, estimate „π“ with a Montecarlo method and run another Java ML example. 
- **[Run_Spark_on_Google_Colab.ipynb](Run_Spark_on_Google_Colab.ipynb)** Set up a single-node standalone Spark server on Google Colab including Web UI and History Server - compact version 
- **[Spark_Standalone_Architecture_on_Google_Colab.ipynb](Spark_Standalone_Architecture_on_Google_Colab.ipynb)** Explore the Spark architecture through the immersive experience of deploying a standalone setup. 
- **[SparkOnSLURM.ipynb](SparkOnSLURM.ipynb)** Spark on SLURM: Single-Node Parallelism Guide. 
## MapReduce Tutorials
- **[MapReduce_Primer_HelloWorld.ipynb](MapReduce_Primer_HelloWorld.ipynb)** A MapReduce Primer with “Hello, World!” 
- **[MapReduce_Primer_HelloWorld_bash.ipynb](MapReduce_Primer_HelloWorld_bash.ipynb)** A MapReduce Primer with “Hello, World! in Bash with just a few lines of code” 
- **[mapreduce_with_bash.ipynb](mapreduce_with_bash.ipynb)** An introduction to MapReduce using MapReduce Streaming and bash to create mapper and reducer
- **[simplest_mapreduce_bash_wordcount.ipynb](simplest_mapreduce_bash_wordcount.ipynb)** A very basic MapReduce wordcount example
- **[mrjob_wordcount.ipynb](mrjob_wordcount.ipynb)** This demonstration showcases MapReduce using the `mrjob` library for performing the classic WordCount algorithm, which counts the frequency of every word in a text dataset. 
- **[getting_started_with_mrjob.ipynb](getting_started_with_mrjob.ipynb)** Getting started with `mrjob`. This demonstration showcases the power and flexibility of the `mrjob` Python framework for developing and executing scalable data processing jobs, supporting both MapReduce and Spark across different execution backends, culminating in a hybrid approach utilizing Spark on YARN.
- **[Hadoop_spilling.ipynb](Hadoop_spilling.ipynb)** Hadoop spilling explained
## PySpark Tutorials
- **[PySpark_On_Google_Colab.ipynb](PySpark_On_Google_Colab.ipynb)** Explore the inner workings of PySpark on Google Colab 
- **[PySpark_miscellanea.ipynb](PySpark_miscellanea.ipynb)** Tips, tricks, and insights related to PySpark. 
- **[getting_started_with_mrjob.ipynb](getting_started_with_mrjob.ipynb)** Getting started with `mrjob`. This demonstration showcases the power and flexibility of the `mrjob` Python framework for developing and executing scalable data processing jobs, supporting both MapReduce and Spark across different execution backends, culminating in a hybrid approach utilizing Spark on YARN.
- **[demoSparkSQLPython.ipynb](demoSparkSQLPython.ipynb)** A hands-on demo showcasing the fundamentals of PySpark SQL — how to create DataFrames, register temporary views, and query data using SQL syntax. 
- **[ngrams_with_pyspark.ipynb](ngrams_with_pyspark.ipynb)** Basic example of n-grams extraction with PySpark
- **[Ngrams.ipynb](Ngrams.ipynb)** This notebook demonstrates the power of Apache Spark (PySpark) for large-scale data processing through the computation of N-grams. It showcases fundamental Spark concepts, including transformations and actions, using two distinct and massive datasets: Common Crawl (Web Data) and Genomic Data 
- **[generate_data_with_Faker.ipynb](generate_data_with_Faker.ipynb)** Fake It Till You Make It: Generate Test Data with Faker. Create customizable fake data for testing and development using the Faker library. Useful for populating databases, simulating user activity, or prototyping applications without relying on real data. 
- **[Encoding_dataframe_columns.ipynb](Encoding_dataframe_columns.ipynb)** DataFrame Column Encoding with PySpark and Parquet Format 
- **[Apache_Sedona_with_PySpark.ipynb](Apache_Sedona_with_PySpark.ipynb)** Apache Sedona™ is a high-performance cluster computing system for processing large-scale spatial data, extending the capabilities of Apache Spark for advanced geospatial analytics. Run a basic example with PySpark on Google Colab 
## Miscellaneous Tutorials
- **[GutenbergBooks.ipynb](GutenbergBooks.ipynb)** Explore and download books from the Gutenberg books collection. 
- **[TestDFSio.ipynb](TestDFSio.ipynb)** Demo of TestDFSio for benchmarking Hadoop clusters
- **[Unicode.ipynb](Unicode.ipynb)** [](https://mybinder.org/v2/gh/groda/big_data/master?filepath=Unicode.ipynb) Exploring Unicode categories 
- **[polynomial_regression.ipynb](polynomial_regression.ipynb)** Worked out example of polynomial regression with numpy and matplotlib 
- **[downloadSpark.ipynb](downloadSpark.ipynb)** How to download and verify the Spark distribution. Learn about the Apache CDN distribution and visualize the Spark releases timeline. 
- **[SLURM.ipynb](SLURM.ipynb)** SLURM (Simple Linux Utility for Resource Management) is an open-source workload manager used to schedule and run jobs on shared compute resources—even on a single machine. This walkthrough shows how to set up and use SLURM, a popular workload manager, on a single Ubuntu virtual machine inside Google Colab. 
## Virtualization and Cloud Automation
- **[docker_for_beginners.md](docker_for_beginners.md)** Docker for beginners: an introduction to the world of containers
- **[Terraform for beginners.md](terraform_for_beginners.md)** Getting started with Terraform
- **[Terraform in 5 minutes](Terraform%20in%205%20minutes.md)** A short introduction to Terraform, the powerful and popular tool for infrastructure provisioning and management 
## Big Data Learning Pathways
- **[online_resources.md](online_resources.md)** Online resources for learning Big Data
# About this repository
## Notebooks Testing and CI
[](https://github.com/groda/big_data/actions/workflows/run-jupyter-notebooks.yml)
All Jupyter notebooks are tested on an Ubuntu virtual machine through a GitHub automated workflow. The executions results are saved in: [report.md](https://github.com/groda/big_data/blob/master/report.md).
## 💡 Keep Learning, Keep Sharing
If you find these tutorials useful, feel free to leave a ⭐ on the repository.