Data Engineering 101 - Spark Concepts

Shwetank Singh
GritSetGrow - GSGLearn.com
Data Engineering
SPARK
ALL CONCEPTS TO GET STARTED
Data Engineering 101 - Spark
Driver Program
The main program that runs the user's
application and creates the SparkContext.
Example
Initializing SparkContext: from pyspark import
SparkContext; sc = SparkContext("local",
"App").
Shwetank Singh
Executors
Worker nodes that execute tasks and store
data for the application.
Example
Executors process tasks and cache data in
memory.
Shwetank Singh
Cluster Manager
Manages resources and schedules jobs across
the cluster.
Example
Using YARN, Mesos, or Standalone mode for
resource management.
Shwetank Singh
SparkContext
The entry point to Spark functionality.
Example
sc = SparkContext("local", "App")
Shwetank Singh
SparkSession
The entry point for DataFrame and SQL
functionality.
Example
from pyspark.sql import SparkSession;
spark = SparkSession.builder.appName("App")
.getOrCreate()
Shwetank Singh
Resilient Distributed
Datasets (RDDs)
Immutable distributed collections of objects
that can be processed in parallel.
Example
rdd = sc.parallelize([1, 2, 3, 4, 5]).
Shwetank Singh
Transformations
Operations that create a new RDD from an
existing one.
Example
rdd2 = rdd.map(lambda x: x * 2).
Shwetank Singh
Actions
Operations that trigger computation and
return a result to the driver.
Example
count = rdd.count().
Shwetank Singh
Lazy Evaluation
Transformations are not executed until an
action is called.
Example
Defining transformations but no computation
until action:
rdd2 = rdd.map(x => x * 2);
rdd2.count().
Shwetank Singh
Directed Acyclic Graph

(DAG)
A sequence of computations represented as a
graph.
Example
Viewing the DAG in Spark UI.
Shwetank Singh
Spark SQL
A module for working with structured data
using SQL.
Example
df = spark.sql("SELECT * FROM table").
Shwetank Singh
DataFrame
A distributed collection of data organized into
named columns.
Example
df = spark.read.json("file.json").
Shwetank Singh
Dataset
A strongly-typed collection of objects.
Example
ds = spark.createDataset([(1, "Alice"), (2, "Bob")]).
Shwetank Singh
Catalyst Optimizer
Spark's query optimizer that generates
efficient execution plans.
Example
Analyzing query plans with df.explain().
Shwetank Singh
PySpark
The Python API for Spark, allowing Python
developers to use Spark's features.
Example
from pyspark.sql import SparkSession;
spark =
SparkSession.builder.appName("App").getOrCreate().
Shwetank Singh
SQL
Allows execution of SQL queries in PySpark.
Example
spark.sql("SELECT COUNT(*) FROM table").
Shwetank Singh
DataFrame
Similar to pandas DataFrame, but distributed
across a cluster.
Example
df = spark.read.csv("file.csv").
Shwetank Singh
RDD
Low-level API for distributed data
manipulation.
Example
rdd = sc.textFile("file.txt").
Shwetank Singh
UDF
User-defined functions for extending
PySpark's built-in functions.
Example
spark.udf.register("add_one", lambda x: x + 1,
IntegerType()).
Shwetank Singh
Window Functions
Functions that allow calculations across a set
of rows related to the current row.
Example
df.withColumn("rank", row_number()
.over(Window.partitionBy("dept")
.orderBy("salary"))).
Shwetank Singh
Machine Learning
Using MLlib for scalable machine learning in
PySpark.
Example
from pyspark.ml.classification import
LogisticRegression; lr = LogisticRegression().
Shwetank Singh
Streaming
Processing real-time data streams with Spark
Streaming.
Example
from pyspark.streaming import StreamingContext;
ssc = StreamingContext(sc, 1).
Shwetank Singh
GraphFrames
A library for graph processing using PySpark.
Example
from graphframes import GraphFrame; g =
GraphFrame(vertices, edges).
Shwetank Singh
Accumulators
Variables that can only be "added" to, used
for aggregating information across executors.
Example
accum = sc.accumulator(0).
Shwetank Singh
Broadcast Variables
Variables that are cached on each machine to
efficiently distribute large values.
Example
broadcastVar = sc.broadcast([1, 2, 3]).
Shwetank Singh
Partitioning
Dividing data into partitions for parallel
processing.
Example
rdd = rdd.repartition(4).
Shwetank Singh
Shuffling
Redistributing data across different nodes,
required by certain transformations like
reduceByKey.
Shwetank Singh
Caching
Storing RDDs in memory for faster access
during iterative algorithms.
Example
rdd.cache().
Shwetank Singh
Persisting
Storing RDDs with different storage levels
(memory, disk, etc.).
Example
rdd.persist(StorageLevel.MEMORY_AND_DISK).
Shwetank Singh
Checkpointing
Saving RDDs to reliable storage to truncate
lineage graphs and provide fault tolerance.
Example
rdd.checkpoint().
Shwetank Singh
Fault Tolerance
Recovering lost data using lineage
information.
Example
Handling node failures by recomputing lost
partitions.
Shwetank Singh
Lineage
Tracking the sequence of transformations that
produced an RDD.
Example
Viewing lineage with rdd.toDebugString.
Shwetank Singh
MapReduce
A programming model for processing large
datasets with a distributed algorithm on a
cluster.
Example
Implementing a simple MapReduce job:
rdd.map(...).reduceByKey(...).
Shwetank Singh
HDFS
Integration with Hadoop Distributed File
System for data storage.
Example
rdd = sc.textFile("hdfs://path/to/file").
Shwetank Singh
Data Sources
Reading and writing data from various sources
like HDFS, S3, JDBC, and more.
Example
df = spark.read.jdbc(url, table, properties).
Shwetank Singh
SQL Functions
Built-in functions for data manipulation and
analysis in Spark SQL.
Example
df.select(concat(col("name"), lit(" "),
col("surname"))).
Shwetank Singh
Column Pruning
An optimization technique that only reads
necessary columns from the data source.
Example
Using column pruning with DataFrame API.
Shwetank Singh
Predicate Pushdown
An optimization technique that pushes filter
conditions to the data source.
Shwetank Singh
Catalyst Optimization
Spark's internal optimization framework for
generating efficient execution plans.
Example
Viewing optimized logical plan with
df.queryExecution.optimizedPlan.
Shwetank Singh
Tungsten
An execution engine in Spark SQL for memory
management and binary processing.
Example
Enabling Tungsten optimization:
spark.sql.tungsten.enabled = true.
Shwetank Singh
Job
A complete computation expressed as a high-
level action like count or saveAsTextFile.
Example
rdd.count().
Shwetank Singh
Stage
A set of parallel tasks that will be executed as
a unit.
Example
Viewing stages in Spark UI for a job.
Shwetank Singh
Task
The smallest unit of work in Spark, executed
by an executor.
Example
Monitoring task execution in Spark UI.
Shwetank Singh
Cache
Storing RDDs in memory to speed up repeated
access.
Example
rdd.cache().
Shwetank Singh
Persist
Storing RDDs with different storage levels
(memory, disk, etc.).
Example
rdd.persist(StorageLevel.MEMORY_AND_DISK).
Shwetank Singh
Checkpoint
Saving an RDD to a reliable storage to truncate
the lineage graph and provide fault tolerance.
Example
rdd.checkpoint().
Shwetank Singh
Fault Tolerance
Spark's ability to recompute lost data using
lineage information.
Example
Handling node failures by recomputing lost
partitions.
Shwetank Singh
Lineage
The logical execution plan that Spark builds to
keep track of the transformations applied to
an RDD.
Example
Viewing lineage with rdd.toDebugString.
Shwetank Singh
MapReduce
A programming model for processing large
datasets with a parallel, distributed algorithm.
Example
Implementing a simple MapReduce job in
Spark: rdd.map(...).reduceByKey(...).
Shwetank Singh
HDFS
Integration with Hadoop Distributed File
System for data storage.
Example
Reading data from HDFS:
val rdd = sc.textFile("hdfs://path/to/file").
Shwetank Singh
Data Sources
Interfaces for reading and writing data from
various sources like HDFS, S3, JDBC, and more.
Example
Reading data from a JDBC source:
val df = spark.read.jdbc(url, table, properties).
Shwetank Singh
SQL Functions
Built-in functions for data manipulation and
analysis in Spark SQL.
Example
Using concat function:
df.select(concat(col("name"), lit(" "),
col("surname")).
Shwetank Singh
User-Defined Functions
(UDFs)
Custom functions defined by users to extend
Spark SQL capabilities.
Example
spark.udf.register("add_one", lambda x: x + 1,
IntegerType()).
Shwetank Singh
Window Functions
Functions that perform calculations across a
set of rows related to the current row.
Example
df.withColumn("rank", row_number()
.over(Window.partitionBy("dept")
.orderBy("salary"))).
Shwetank Singh
GroupBy
Grouping data by one or more columns for
aggregation.
Example
df.groupBy("department").agg(avg("salary")).
Shwetank Singh
Join
Combining rows from two or more
DataFrames based on a related column.
Example
df1.join(df2, df1("id") === df2("id")).
Shwetank Singh
Sorting
Ordering rows in a DataFrame based on
column values.
Example
df.sort("age", ascending=True).
Shwetank Singh
Aggregations
Computing summary statistics of data groups.
Example
df.groupBy("department").agg(sum("salary"))
Shwetank Singh
Pivot
Transforming unique values from one column
into multiple columns.
Example
df.groupBy("year").pivot("quarter").sum("rev
enue").
Shwetank Singh
DataFrame Operations
Common operations on DataFrames, such as
selecting, filtering, and grouping.
Example
df.select("name").filter(df("age") > 21).
Shwetank Singh
Schema
Defining the structure of a DataFrame with
column names and types.
Example
schema = StructType([StructField("name",
StringType(), True), StructField("age",
IntegerType(), True)]).
Shwetank Singh
DataFrame Creation
Creating DataFrames from various data
sources like CSV, JSON, and RDDs.
Example
df = spark.read.csv("file.csv").
Shwetank Singh
DataFrame
Transformation
Modifying DataFrames using operations like
withColumn, filter, groupBy, and agg.
Example
df.withColumn("new_col", df("col1") + df("col2"))
Shwetank Singh
DataFrame Actions
Triggering computations and returning results
to the driver.
Example
df.show().
Shwetank Singh
SQL Queries
Executing SQL queries on DataFrames.
Example
df.createOrReplaceTempView("table");
spark.sql("SELECT * FROM table").
Shwetank Singh
Joins
Combining DataFrames based on common
columns.
Example
df1.join(df2, df1("id") === df2("id")).
Shwetank Singh
Aggregations
Aggregating data in DataFrames using
functions like sum, avg, and count.
Example
df.groupBy("department").agg(sum("salary"))
Shwetank Singh
Filtering
Filtering rows based on column values.
Example
df.filter(df("age") > 21).
Shwetank Singh
Selection
Selecting specific columns from a DataFrame.
Example
df.select("name", "age").
Shwetank Singh
Renaming
Renaming columns in a DataFrame.
Example
df.withColumnRenamed("old_name",
"new_name")
Shwetank Singh
Dropping Columns
Removing columns from a DataFrame.
Example
df.drop("column_name").
Shwetank Singh
Merging
Combining multiple DataFrames into one.
Example
df1.union(df2).
Shwetank Singh
Pivoting
Transforming data to show unique values
from one column as multiple columns.
Example
df.groupBy("year")
.pivot("quarter")
.sum("revenue").
Shwetank Singh
Unpivoting
Transforming data from columns into rows.
Example
melted_df = df.selectExpr("stack(3, 'Q1', Q1,
'Q2', Q2, 'Q3', Q3) as (quarter, revenue)").
Shwetank Singh
Writing
Writing DataFrames to external storage
systems like HDFS, S3, or databases.
Example
df.write.csv("output.csv").
Shwetank Singh
Reading
Reading data from various external storage
systems into DataFrames.
Example
df = spark.read.json("input.json").
Shwetank Singh
Repartitioning
Changing the number of partitions of a
DataFrame.
Example
df.repartition(10).
Shwetank Singh
Coalescing
Reducing the number of partitions of a
DataFrame.
Example
df.coalesce(1).
Shwetank Singh
Serialization
Converting DataFrame rows into a format that
can be stored and transmitted.
Example
Using PySpark's built-in serializers like pickle
and Kryo.
Shwetank Singh
Deserialization
Converting serialized data back into
DataFrame rows.
Example
Reading serialized data and converting it back
to DataFrame.
Shwetank Singh
Functions
Built-in functions for performing operations
on DataFrames.
Example
from pyspark.sql.functions
import col, lit; df.select(col("name"), lit(1)).
Shwetank Singh
Column Operations
Performing operations on DataFrame
columns.
Example
df.withColumn("new_col", df("col1") + df("col2"))
Shwetank Singh
Exploding
Transforming a DataFrame with nested
structures into a flat DataFrame.
Example
df.select("name", explode("hobbies"))
Shwetank Singh
Aggregation Functions
Built-in functions for aggregating data in
DataFrames.
Example
from pyspark.sql.functions import sum, avg;
df.groupBy("dept")
.agg(sum("salary"), avg("salary"))
Shwetank Singh
Statistics
Computing summary statistics of DataFrames.
Example
df.describe().show().
Shwetank Singh
Union
Combining rows from two DataFrames into
one.
Example
df1.union(df2).
Shwetank Singh
Intersect
Finding common rows between two
DataFrames.
Example
df1.intersect(df2).
Shwetank Singh
Except
Finding rows in one DataFrame that are not in
another.
Example
df1.except(df2).
Shwetank Singh
Distinct
Removing duplicate rows from a DataFrame.
Example
df.distinct().
Shwetank Singh
Drop Duplicates
Removing duplicate rows based on specific
columns.
Example
df.dropDuplicates(["col1", "col2"]).
Shwetank Singh
Missing Data Handling

Handling missing values in DataFrames.
Example
df.fillna(0).
Shwetank Singh
Date Functions
Working with date and time data in
DataFrames.
Example
df.withColumn("year", year("date_col"))
Shwetank Singh
String Functions
Performing operations on string columns in
DataFrames.
Example
df.withColumn("uppercase_name",
upper("name"))
Shwetank Singh
Conversion
Converting between different data types in
DataFrames.
Example
df.withColumn("int_col", col("string_col")
.cast("int")).
Shwetank Singh
Explode
Converting an array column into multiple
rows, one for each element in the array.
Example
df.select("name", explode("hobbies")).
Shwetank Singh
JSON Functions
Working with JSON data in DataFrames.
Example
df.selectExpr("json_tuple(json_col, 'key1',
'key2')").
Shwetank Singh
Advanced Analytics
Using advanced analytical functions available
in PySpark.
Example
df.stat.freqItems(["col1", "col2"]).
Shwetank Singh
Vectorized UDFs
UDFs that operate on entire columns at once,
improving performance.
Example
from pyspark.sql.functions import pandas_udf;
@pandas_udf("int") def add_one(s: pd.Series) -
> pd.Series: return s + 1.
Shwetank Singh
THANK YOU

Data Engineering 101 - Spark Concepts

Uploaded by

Copyright:

Available Formats

Data Engineering 101 - Spark Concepts

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Data Engineering 101 - Spark Concepts

Uploaded by

Copyright:

Available Formats

Shwetank Singh

Directed Acyclic Graph

Missing Data Handling

You might also like