Apache Tez基本知识

不念不忘少年蓝@ 2022-03-24 20:36 144阅读 0赞

官方blog:

[http://hortonworks.com/blog/author/arun\_murthy/][http_hortonworks.com_blog_author_arun_murthy]

svn源码：

[http://hortonworks.com/blog/introducing-tez-faster-hadoop-processing/][http_hortonworks.com_blog_introducing-tez-faster-hadoop-processing]

看到一篇很不错的文章：

[http://dongxicheng.org/mapreduce-nextgen/apache-tez/][http_dongxicheng.org_mapreduce-nextgen_apache-tez]

Tez安装：

[http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.0.0.2/bk\_installing\_manually\_book/content/rpm-chap-tez.html][http_docs.hortonworks.com_HDPDocuments_HDP2_HDP-2.0.0.2_bk_installing_manually_book_content_rpm-chap-tez.html]

名称解释：

DAG：Directed Acyclic Graph 无环有向图

Tez将Map task和Reduce task进一步拆分，

![1366077189_7595.png][]

Tez的task由Input、processor、output阶段组成，可以表达所有复杂的map、reduce操作。

![1366077389_3057.png][]

这样，这些分解后的元操作可以任意灵活组合，产生新的操作单元Task.

举个例子：

在传统的MR如下左图所示，而采用Tez之后则将DAG的依赖关系去除，Tez将有依赖关系的作业转换为一个作业，如下右图所示。

![1366077154_3297.png][]![1366077171_3248.png][]

Pig/Hive -MR Pig/Hive -Tez

代码结构图如下：

![1366077275_6708.png][]

•tez-api：六个可编程组件

•tez-common

•tez-engine

•tez-mapreducetez-task-library: Tez为兼容MR提供的组件

•tez-yarn-application

•tez-yarn-client

[http_hortonworks.com_blog_author_arun_murthy]: http://hortonworks.com/blog/author/arun_murthy/
[http_hortonworks.com_blog_introducing-tez-faster-hadoop-processing]: http://hortonworks.com/blog/introducing-tez-faster-hadoop-processing/
[http_dongxicheng.org_mapreduce-nextgen_apache-tez]: http://dongxicheng.org/mapreduce-nextgen/apache-tez/
[http_docs.hortonworks.com_HDPDocuments_HDP2_HDP-2.0.0.2_bk_installing_manually_book_content_rpm-chap-tez.html]: http://blog.csdn.net/lijiajia81/article/details/8806931
[1366077189_7595.png]: https://img-my.csdn.net/uploads/201304/16/1366077189_7595.png
[1366077389_3057.png]: https://img-my.csdn.net/uploads/201304/16/1366077389_3057.png
[1366077154_3297.png]: https://img-my.csdn.net/uploads/201304/16/1366077154_3297.png
[1366077171_3248.png]: https://img-my.csdn.net/uploads/201304/16/1366077171_3248.png
[1366077275_6708.png]: https://img-my.csdn.net/uploads/201304/16/1366077275_6708.png