Spark Tutorials 3 - RDD

blairchen

spark

Publish：Sep 25, 2020

views

• very good Spark原理篇之RDD特征分析讲解
• spark 基础知识整理（一）
• spark 基础知识整理（二）- RDD专题
• Spark 分区(Partition)的认识、理解和应用

1. Create RDDs

2. RDD Persistence and Caching

What is RDD persistence, Why do we need to call cache or persist on an RDD, What is the Difference between Cache() and Persist() method in Spark

What is RDD Persistence and Caching

The difference between cache() and persist() is that using cache() the default storage level is MEMORY_ONLY while using persist() we can use various storage levels (described below).

• Time efficient
• Cost efficient
• Lessen the execution time.

Storage levels of Persisted RDDs

• MEMORY_ONLY
• MEMORY_AND_DISK
• MEMORY_ONLY_SER
• MEMORY_AND_DISK_SER
• DISK_ONLY

How to Unpersist RDD in Spark?

using RDD.unpersist() method.

3. RDD Features

3.1 In-memory computation

3.2 Lazy Evaluation

3.3 Fault Tolerance

Fault tolerance in Apache Spark – Reliable Spark Streaming

3.4 Immutability

3.5 Persistence

3.6 Partitioning

3.7 Parallel

Rdd, process the data parallelly over the cluster.

3.8 Location-Stickiness

3.9 Coarse-grained Operation

3.10 Typed

We can have RDD of various types like: RDD [int], RDD [long], RDD [string].

3.11 No limitation

• RDD Features

• good Spark基本工作原理和RDD特性

• Spark RDD 特征及其依赖

4. Paired RDD

Here transformation operations are:

1. groupByKey
2. reduceByKey
3. join
4. left outer join
5. right outer Join

Whereas actions like countByKey

4.1 Objective

in spark is designed as each dataset in RDD is divided into logical partitions. Further, we can say here each partition may be computed on different nodes of the cluster.

4.2 Spark Paired RDD

1
2
3
4
5
6
7
8
9
10
11

from pyspark import SparkConf , SparkContext
from operator import add
sc.version
lines22 = sc.textFile("/Users/blair/ghome/github/spark3.0/pyspark/spark-src/word_count.text", 2)
pairs22= lines22.map(lambda x: (x, 1))

#pairs22.take(2)

counts22 = pairs22.reduceByKey(add)

counts22

4.3 Create Spark Paired RDD

a. In Python language

1

pairs = lines.map(lambda x: (x.split(” “)[0], x))

4.4 Paired RDD Operations

PySpark 3.0.1 documentation »

(1). reduceByKey(fun) & groupByKey

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

lines = sc.textFile("/Users/blair/ghome/github/spark3.0/pyspark/spark-src/word_count.text", 2)

lines.take(3)
words = lines.flatMap(lambda x: x.split(' '))
print(words.take(5))

wco = words.map(lambda x: (x, 1))
print(wco.take(5))
# word_count = wco.reduceByKey(add)
# print("\nword_count:")
# print(word_count.take(5))
print("\ngroupByKey:")
test = wco.groupByKey()
print(test.take(2))
# gp = test.map(lambda x: (x[0], [i for i in x[1]]))
# gp.take(2)

(2). mapValues(fun)

1
2
3
4
5
6

rdd = sc.parallelize([("a", 1), ("b", 1), ("a", 1)])

sorted(rdd.groupByKey().mapValues(len).collect())
# [('a', 2), ('b', 1)]
sorted(rdd.groupByKey().mapValues(list).collect())
# [('a', [1, 1]), ('b', [1])]

(3). keys(), values()

1
2
3
4
5
6
7

m = sc.parallelize([(1, 2), (3, 4)]).keys()
m.collect()
[1, 3]

m = sc.parallelize([(1, 2), (3, 4)]).values()
m.collect()
[2, 4]

(4). sortBykey

• spark combinebykey？
• pyspark中combineByKey的两种理解方法

5. RDD limitations

Reference

• Spark原理篇之RDD特征分析讲解
• PySpark 3.0.1 documentation »
• data-flair.training/blogs
• Spark RDD Operations-Transformation & Action with Example
• Spark RDD常用算子学习笔记详解(python版)
• Spark常用的Transformation算子的简单例子

updated on：Jun 28, 2021

spark