[Spark、hadoop]spark Streaming的核心DStream

Bertha 。 2024-04-03 08:48 11阅读 0赞

**目录**

Spark Streaming的核心是DStream

一、DStream简介

二．DStream编程模型

三．DStream转换操作

--------------------

### **Spark Streaming的核心是DStream** ###

#### **一、DStream简介** ####

1.Spark Streaming提供了一个高级抽象的流，即DStream(离散流)。

2.DStream的内部结构是由一系列连续的RDD组成，每个RDD都是一小段由时间分隔开来的数据集。

#### **二．DStream编程模型** ####

![0fe737d0fe51471f86a0a8c85b0fb69b.png][]

#### **三．DStream转换操作** ####

![af73eb7e13394223a84bc68153f85d78.png][]

**transform（）**

**1．在3个节点启动zookeeper集群服务**

$ zkServer.sh start

![9e4323c2f10640969e8f875570223646.png][]

**2．启动kafka（3个节点都要）**

**$** /opt/module/kafka/bin/kafka-server-start.sh /opt/module/kafka/config/server.properties

![b8fd5e96bd61405d8067e2ae9594e56e.png][]

**3．克隆会话，jps查看是否有Kafka（3个节点）**

![6ec2632d396b41d58198dbcc9ea0012d.png][]

**4．进入/etc/resolv.conf加入如下内**

vi resolv.conf

![2c4bd758c63a467a953c83136399492a.png][]

**N****ame****server 114.114.114.114**

![b8f0b9c3b0c14976946f77fff32e032f.png][]

**5．下载nc**

**$** **yum install nc**

![2c6420b111944f5c90515f2fb7a368f1.png][]

**6．启动服务端且监听Socket服务，并输入Iamlearningsparkstreamingnow**

**$** **nc -lk 9999**

**I****am****learning****spark****streaming****now（回车）**

![7cf1e17894a845d6b14a2dd40b826abe.png][]

**7．创建Maven项目, 打开右上角的Fileànewàproject**

![6d7d0118fbe745b7aceccac24cf30aac.png][]

**8．选择mavenànext**

![42c8541a4dcb46a487d8de6f175505df.png][]

**9．填写groupld、artifctldànext**

![2b64115e0a9740d7baa40983fbdeb613.png][]

**10．添加依赖**

<dependencies>
    
        <dependency>
    
            <groupId>org.scala-lang</groupId>
    
            <artifactId>scala-library</artifactId>
    
            <version>2.11.8</version>
    
        </dependency>
    
        <dependency>
    
            <groupId>org.apache.spark</groupId>
    
            <artifactId>spark-core_2.11</artifactId>
    
            <version>2.0.2</version>
    
        </dependency>
    
        
    
        <dependency>
    
            <groupId>org.apache.spark</groupId>
    
            <artifactId>spark-streaming_2.11</artifactId>
    
            <version>2.0.2</version>
    
        </dependency>
    
        <dependency>
    
            <groupId>mysql</groupId>
    
            <artifactId>mysql-connector-java</artifactId>
    
            <version>5.1.38</version>
    
        </dependency>
    
        
    
        <dependency>
    
            <groupId>org.apache.spark</groupId>
    
            <artifactId>spark-streaming-kafka-0-8_2.11</artifactId>
    
            <version>2.0.2</version>
    
        </dependency>
    
    </dependencies>
    
    
    
    <build>
    
        <sourceDirectory>src/main/scala</sourceDirectory>
    
        <testSourceDirectory>src/test/scala</testSourceDirectory>
    
    </build>

![2cedff27b968468a9aa19545448498e8.png][]

**11 在main目录下创建scala文件，右击ànewàdirectory，命名为Scala**

![8c2930dd412748b4957e5eddeba00d73.png][]

![8f634fab663f48c391483697386ff3c8.png][]

**12 右击Scala文件，选择【Mark Directoryas】à【sourcesroot】表示将文件夹标记为资源文件夹类型（存放项目源码文件）**

![decbc50cfd3b41d78c600db52569b489.png][]

**13 同样在test目录下创建scala文件，右击ànewàdirectory，命名为Scala**

![a9f60c27b6da403c9aaa10509bc52aef.png][]

![382b070091ce47a1991bc5ce89d9d122.png][]

**14 右击Scala文件，选择【Mark Directoryas】à【testsourcesroot】表示将文件夹标记为测试资源文件夹类型（存放开发中测试的源码文件）**

![24bd00986af041fb930661a208f5e55f.png][]

**15 右击main下的Scala文件先创建一个package并命名为cn.itcast**

![a9d6d4cde8764a8abc3f6330b9657aa7.png][]

![ff5e514064e24614ba4c2d8ecfe9276f.png][]

**16 编写TransformTest.Scala**

注意红框内容！！！

![6e2aa4fc76da40f9ad1d6cdb404f919d.png][]

**17 运行程序，可以看到控制台输出结果**

![e4822f28036e44698bbca68e8a0b8951.png][]

**UpdateStateKey()**

**1编写UpdateStateKeyTest.scala**

注意红框内容！！！

![a556fee2ec92475c86cee28e781ac317.png][]

**2 运行程序并在master 9999 端口不断输入单词，观察到控制台输出内容**

$ nc -lk 9999

Hadoop spark itcast(回车)

Spark itcast

![7261189ca5fd45d6ba0afd5a89128d04.png][]

[0fe737d0fe51471f86a0a8c85b0fb69b.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/de6128f5aa7549be8334cc4841b8d9c3.png
[af73eb7e13394223a84bc68153f85d78.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/17dd865a5e894030924a35e46b15eaf4.png
[9e4323c2f10640969e8f875570223646.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/1335f499770949d9be31eaf85026891f.png
[b8fd5e96bd61405d8067e2ae9594e56e.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/dcbb1dfd8a4d49a9aefbe83b4ccd7f8b.png
[6ec2632d396b41d58198dbcc9ea0012d.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/3581885470e24b9c96bb4a056fe006e2.png
[2c4bd758c63a467a953c83136399492a.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/4748229f2cd64548918e64ccc94ac0f7.png
[b8f0b9c3b0c14976946f77fff32e032f.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/e0de1c23f7b94deaa3a3b17819ac79e1.png
[2c6420b111944f5c90515f2fb7a368f1.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/31dc6cd5669c4ca0abe3ff0aee152b5f.png
[7cf1e17894a845d6b14a2dd40b826abe.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/6b33eb082d45429eacb23e79ad24bd7f.png
[6d7d0118fbe745b7aceccac24cf30aac.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/64d5c2d4b4824db6875c69530421fc06.png
[42c8541a4dcb46a487d8de6f175505df.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/89ef86f2c5c8439686b75f8b1f2a7aa6.png
[2b64115e0a9740d7baa40983fbdeb613.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/461afe9bf0c640bb9fcf44a90e48d8c7.png
[2cedff27b968468a9aa19545448498e8.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/459d283e6fc0452787b36efb769418cf.png
[8c2930dd412748b4957e5eddeba00d73.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/06b63505079e4d64a3f3646ae61863c1.png
[8f634fab663f48c391483697386ff3c8.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/e73ed347dd2b48d7a952527f17107a5e.png
[decbc50cfd3b41d78c600db52569b489.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/aaae0ac260ac4e88800956b02b43b118.png
[a9f60c27b6da403c9aaa10509bc52aef.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/d1f9bd5c0f054bc3bc076cbc127ddd0d.png
[382b070091ce47a1991bc5ce89d9d122.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/b0b08ed8a5224635b2caad77035166d2.png
[24bd00986af041fb930661a208f5e55f.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/28f7898dda664c5c936570fc60650c81.png
[a9d6d4cde8764a8abc3f6330b9657aa7.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/5e1ac4569d744bc1b87fd4bf13b1187c.png
[ff5e514064e24614ba4c2d8ecfe9276f.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/379f4edde686456d9fe5104bc6475b14.png
[6e2aa4fc76da40f9ad1d6cdb404f919d.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/2c4db1b472234337ab53948116f04ae3.png
[e4822f28036e44698bbca68e8a0b8951.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/79291119d49a4edc87524055a1999eea.png
[a556fee2ec92475c86cee28e781ac317.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/83e4a917a50642f387225f535dbeb64a.png
[7261189ca5fd45d6ba0afd5a89128d04.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/b2292efbc73b443f9dad6473ba65225a.png