Linux上Elasticsearch及Logstash环境搭建

今天药忘吃喽~ 2022-05-31 05:40 161阅读 0赞

基本环境：CentOS 6.4、elasticsearch-6.1.2.tar.gz、logstash-6.1.3.tar.gz、JDK1.8

## 安装Elasticsearch ##

### 一、新建用户 ###

Elasticsearch不可以用root用户打开，所以要新建用户,执行：useradd ‘yourname’

设置密码后创建成功。

切换到刚创建的用户，执行：su ‘yourname’

输入密码。

### 二、解压安装 ###

压缩包可直接通过网络下载，执行：wget [https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.1.2.tar.gz][https_artifacts.elastic.co_downloads_elasticsearch_elasticsearch-6.1.2.tar.gz]

下载完成后执行：tar -zvxf elasticsearch-6.1.2.tar.gz

解压完毕后进入bin目录，执行：./elasticsearch

在内网通讯时：修改elasticsearch.yml文件 中的network.host:’yourip’

一定要确保该机器可使用9300-9400端口，同时打开9200端口

打开浏览器，在地址栏输入：http://‘yourip’:9200,看到如下界面，安装成功

![70][]

## 安装logstash ##

### 一、准备数据样本 ###

准备一段json格式的log数据。如下

\{"logType":"pnw","reallyIp":"101.40.49.166","requestURL":"http://\*\*\*\*/PublicQuery.do",

"operationYear":"2018","requestURI":"/PublicQuery.do","userType":"pnw1",

"operation":"pnw5","operationDay":"12","operationHour":"9",

"operationMonth":"2","searchKeyWord":"MA73TEPP2","accessPage":"pnw2"\}

### 二、下载解压 ###

压缩包可直接通过网络下载，执行：wget [https://artifacts.elastic.co/downloads/logstash/logstash-6.1.3.tar.gz][https_artifacts.elastic.co_downloads_logstash_logstash-6.1.3.tar.gz]

下载完成后执行：tar -zvxf logstash-6.1.3.tar.gz

解压完毕后进入/logstash-6.1.3/bin目录下，新建文件命名为：logstash.conf

将如下内容copy至该文件内。

input \{

file \{

path=> \["/usr/local/apache-tomcat-7.0.65/pnwLogs/\*"\]

start\_position=> beginning

codec=> json \{

charset =>"UTF-8"

filter \{

geoip\{

source => "reallyIp"

target => "geoip"

database => "GeoLite2-City.mmdb "

add\_field => \["\[geoip\]\[coordinates\]", "%\{\[geoip\]\[longitude\]\}" \]

add\_field => \["\[geoip\]\[coordinates\]", "%\{\[geoip\]\[latitude\]\}"  \]

output \{

elasticsearch \{

hosts=> "192.168.1.53:9200"

index=> "pnw"

**关键配置说明：**

**inpu下：**

path：文件路径，必须写绝对路径，支持多个路径，支持以\*为通配符，以数组的形式传递。例：\[“/xx/xx/xxx/\*.\*”,”/xxx/xxx/\*”\]

start\_position：从哪个位置开始读取文件

codec：建立索引时按照指定数据格式并指定字符编码

**filter**是logstash过滤器，其中可配置对各字段的条件过滤，示例中对ip进行定位过滤**。**

**output下：**

elasticsearch:es的集群名称

hosts：es的ip及端口号，

index：索引名称

更改完配置文件后，执行：./logstash -flogstash.conf   启动成功后如图：

![70 1][]

# 常见问题 #

## Elasticsearch： ##

1、max virtual memory areasvm.max\_map\_count \[65530\] likely too low

临时修改，执行：sysctl -w vm.max\_map\_count=262144

永久修改，：

\[root@localhost ~\]\# cat/etc/sysctl.conf | grep -v "vm.max\_map\_count" >/tmp/system\_sysctl.conf

\[root@localhost ~\]\# echo"vm.max\_map\_count=262144" >> /tmp/system\_sysctl.conf

\[root@localhost ~\]\# mv/tmp/system\_sysctl.conf /etc/sysctl.conf

mv：是否覆盖"/etc/sysctl.conf"？ y

\[root@localhost ~\]\# cat/etc/sysctl.conf

\# System defaultsettings live in /usr/lib/sysctl.d/00-system.conf.

\# To overridethose settings, enter new settings here, or in an/etc/sysctl.d/<name>.conf file

\# For moreinformation, see sysctl.conf(5) and sysctl.d(5).

vm.max\_map\_count= 655360

\[root@localhost ~\]\# sysctl -p

vm.max\_map\_count = 655360

2、 max file descriptors \[4096\] for elasticsearch process likely too low

\[root@localhost ~\]\# cp/etc/security/limits.conf /etc/security/limits.conf.bak

\[root@localhost ~\]\# cat/etc/security/limits.conf | grep -v "yourname" >/tmp/system\_limits.conf

\[root@localhost ~\]\# echo "‘yourname’hard nofile65536" >> /tmp/system\_limits.conf

\[root@localhost ~\]\# echo "‘yourname’soft nofile65536" >> /tmp/system\_limits.conf

\[root@localhost ~\]\# mv /tmp/system\_limits.conf /etc/security/limits.conf

3、max number of threads\[1024\] for user \[‘yourname’\] likely too low

执行：vi /etc/security/limits.conf

修改：\* soft nproc 4096

## Logstash： ##

JSONPARSEERROR：检查logstash.conf文件，保证指定编码和读取文件编码相同，保证源文件json格式正确。

startedFailed to install template相关错误可能是因为读取的源文件格式变化，可以删除/logstash/data文件夹下的所有文件,重新启动logstash

[https_artifacts.elastic.co_downloads_elasticsearch_elasticsearch-6.1.2.tar.gz]: https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.1.2.tar.gz
[70]: /images/20220531/4ef6a8e22a234dbba9a24d72252fe1f4.png
[https_artifacts.elastic.co_downloads_logstash_logstash-6.1.3.tar.gz]: https://artifacts.elastic.co/downloads/logstash/logstash-6.1.3.tar.gz
[70 1]: /images/20220531/4dca48fb91654a8e84775e2bb0368919.png