Apache Pig 安装

迷南。 2022-06-05 11:21 214阅读 0赞

![017f554168a2444e2556099ff9163703.png_wxfrom_5_wx_lazy_1][]

本章将介绍如何在系统中下载，安装和设置 Apache Pig 。 先决条件

--------------------

在你运行Apache Pig之前，必须在系统上安装好Hadoop和Java。因此，在安装Apache Pig之前，请按照步骤安装Hadoop和Java

步骤 1

--------------------

打开Apache Pig网站的主页。在News部分下，点击链接release page，如下面的快照所示。

![92c5182a2ed5908efad115601bce4e43.png_wxfrom_5_wx_lazy_1][]

步骤 2

--------------------

点击指定的链接后，你将被重定向到 Apache Pig Releases 页面。在此页面的Download部分下，单击链接，然后你将被重定向到具有一组镜像的页面。

![ded0dc657e0fbe95acf5693c8a35dd7f.png][]

步骤3 选择并单击这些镜像中的任一个，如下所示。

--------------------

![ab329a4e55c1102f8199323ea97fa932.png][]

步骤 4

--------------------

这些镜像将带您进入 Pig Releases 页面。 此页面包含Apache Pig的各种版本。 单击其中的最新版本。

![f7ef785c7094cdeb26c8e78f5f772f2a.png][]

步骤 5

--------------------

在这些文件夹中，有发行版中的Apache Pig的源文件和二进制文件。下载Apache Pig 0.16， pig0.16.0-src.tar.gz 和 pig-0.16.0.tar.gz 的源和二进制文件的tar文件。

![d06fbab98df45b5f23e4dfc777f1c652.png][]

安装Apache Pig

下载Apache Pig软件后，按照以下步骤将其安装在Linux环境中。

步骤1

在安装了 Hadoop，Java和其他软件的安装目录的同一目录中创建一个名为Pig的目录。（在我们的教程中，我们在名为Hadoop的用户中创建了Pig目录）。

$ mkdir Pig第2步 提取下载的tar文件，如下所示。

--------------------

$ cd Downloads/ $ tar zxvf pig-0.15.0-src.tar.gz $ tar zxvf pig-0.15.0.tar.gz 步骤3

将 pig-0.16.0-src.tar.gz 文件的内容移动到之前创建的 Pig 目录，如下所示。

$ mv pig-0.16.0-src.tar.gz/\* /home/Hadoop/Pig/配置Apache Pig

--------------------

安装Apache Pig后，我们必须配置它。要配置，我们需要编辑两个文件 - bashrc和pig.properties 。

.bashrc文件

在 .bashrc 文件中，设置以下变量

PIG\_HOME 文件夹复制到Apache Pig的安装文件夹

PATH 环境变量复制到bin文件夹

PIG\_CLASSPATH 环境变量复制到安装Hadoop的etc（配置）文件夹（包含core-site.xml，hdfs-site.xml和mapred-site.xml文件的目录）。

export PIG\_HOME = /home/Hadoop/Pig export PATH = PATH:/home/Hadoop/pig/bin export PIG\_CLASSPATH = $HADOOP\_HOME/confpig.properties文件

在Pig的 conf 文件夹中，我们有一个名为 pig.properties 的文件。在pig.properties文件中，可以设置如下所示的各种参数。

pig -h properties 支持以下属性:

--------------------

Logging: verbose = true|false; default is false. This property is the same as -v switch brief=true|false; default is false. This property is the same as -b switch debug=OFF|ERROR|WARN|INFO|DEBUG; default is INFO. This property is the same as -d switch aggregate.warning = true|false; default is true. If true, prints count of warnings of each type rather than logging each warning. Performance tuning: pig.cachedbag.memusage=<mem fraction>; default is 0.2 (20% of all memory). Note that this memory is shared across all large bags used by the application. pig.skewedjoin.reduce.memusagea=<mem fraction>; default is 0.3 (30% of all memory). Specifies the fraction of heap available for the reducer to perform the join. pig.exec.nocombiner = true|false; default is false. Only disable combiner as a temporary workaround for problems. opt.multiquery = true|false; multiquery is on by default. Only disable multiquery as a temporary workaround for problems. opt.fetch=true|false; fetch is on by default. Scripts containing Filter, Foreach, Limit, Stream, and Union can be dumped without MR jobs. pig.tmpfilecompression = true|false; compression is off by default. Determines whether output of intermediate jobs is compressed. pig.tmpfilecompression.codec = lzo|gzip; default is gzip. Used in conjunction with pig.tmpfilecompression. Defines compression type. pig.noSplitCombination = true|false. Split combination is on by default. Determines if multiple small files are combined into a single map. pig.exec.mapPartAgg = true|false. Default is false. Determines if partial aggregation is done within map phase, before records are sent to combiner. pig.exec.mapPartAgg.minReduction=<min aggregation factor>. Default is 10. If the in-map partial aggregation does not reduce the output num records by this factor, it gets disabled. Miscellaneous: exectype = mapreduce|tez|local; default is mapreduce. This property is the same as -x switch pig.additional.jars.uris=<comma seperated list of jars>. Used in place of register command. udf.import.list=<comma seperated list of imports>. Used to avoid package names in UDF. stop.on.failure = true|false; default is false. Set to true to terminate on the first error. pig.datetime.default.tz=<UTC time offset>. e.g. +08:00. Default is the default timezone of the host. Determines the timezone used to handle datetime datatype and UDFs. Additionally, any Hadoop property can be specified.验证安装

通过键入version命令验证Apache Pig的安装。如果安装成功，你将获得Apache Pig的正式版本，如下所示。

$ pig –version Apache Pig version 0.16.0 (r1682971) compiled Jun 01 2015, 11:44:35

![6e8b6c624b6448a4ebd266c6d81987d1.gif][]

[017f554168a2444e2556099ff9163703.png_wxfrom_5_wx_lazy_1]: https://img-blog.csdnimg.cn/img_convert/017f554168a2444e2556099ff9163703.png;wxfrom=5&wx_lazy=1
[92c5182a2ed5908efad115601bce4e43.png_wxfrom_5_wx_lazy_1]: https://img-blog.csdnimg.cn/img_convert/92c5182a2ed5908efad115601bce4e43.png;wxfrom=5&wx_lazy=1
[ded0dc657e0fbe95acf5693c8a35dd7f.png]: /images/20220605/92991484bf67422c891505c5db14c12f.png
[ab329a4e55c1102f8199323ea97fa932.png]: /images/20220605/0381d18a7c384febbebdc97cb408cf08.png
[f7ef785c7094cdeb26c8e78f5f772f2a.png]: /images/20220605/9678fac8414f436093c470afdd41f4ab.png
[d06fbab98df45b5f23e4dfc777f1c652.png]: /images/20220605/6027f9f50e0d4e0584ff1b9e542bea4f.png
[6e8b6c624b6448a4ebd266c6d81987d1.gif]: /images/20220605/7392ccc292ec4eaf9b0b3d3629ee8008.png