ELK在docker中的部署

朴灿烈づ我的快乐病毒、 2022-03-19 09:08 375阅读 0赞

ELK是ElasticSearch、Logstash、Kibana的简称，一般用于日志系统，从日志收集，日志转储，日志展示等入手，用以提供简洁高效的日志处理机制。  
![一个简易ELK系统的服务图][ELK]  
鉴于没有额外的机器，这里就用docker来简单模拟下一个简单ELK系统的部署和使用。

## 搭建Logstash ##

### 准备好镜像 ###

docker search logstash // 省略输出
    docker pull logstash // 省略输出
    ➜  logstash docker images
    REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
    es_test             latest              43cf30ef8591        2 days ago          349MB
    redis               latest              0f55cf3661e9        9 days ago          95MB
    logstash            5.5.2               98f8400d2944        17 months ago       724MB
    elasticsearch       2.3.5               1c3e7681c53c        2 years ago         346MB

### 运行实例 ###

➜  logstash docker run -i -t 98f8400d2944 /bin/bash
    root@36134d90b5cd:/# ls
    bin  boot  dev	docker-entrypoint.sh  docker-java-home	etc  home  lib	lib64  media  mnt  opt	proc  root  run  sbin  srv  sys  tmp  usr  var
    root@36134d90b5cd:/# cd /home
    root@36134d90b5cd:/home# ls
    root@36134d90b5cd:/home# cd ../
    root@36134d90b5cd:/# ls
    bin  boot  dev	docker-entrypoint.sh  docker-java-home	etc  home  lib	lib64  media  mnt  opt	proc  root  run  sbin  srv  sys  tmp  usr  var
    root@36134d90b5cd:/# cd /etc/logstash/
    root@36134d90b5cd:/etc/logstash# ls
    conf.d	jvm.options  log4j2.properties	log4j2.properties.dist	logstash.yml  startup.options
    root@36134d90b5cd:/etc/logstash# cat logstash.yml

发现没有logstash.conf（名字是随意的，logstash可以通过-f参数来指定对应的input，filter，output行为）

刚才pull到的镜像中没有vim等编辑器，所以不能直接手动编辑，所以为了使用自定义的配置文件，这里用docker的cp命令，把外部编辑好的配置文件共享到容器内部使用。

logstash docker ps -a
    CONTAINER ID        IMAGE               COMMAND                  CREATED              STATUS              PORTS                              NAMES
    36134d90b5cd        98f8400d2944        "/docker-entrypoint.…"   About a minute ago   Up About a minute                                      priceless_feistel
    687bf289bb43        43cf30ef8591        "/docker-entrypoint.…"   5 minutes ago        Up 5 minutes        0.0.0.0:9200->9200/tcp, 9300/tcp   my_elasticsearch
    ➜  logstash docker cp logstash.conf 36134d90b5cd:/etc/logstash
      logstash cat logstash.conf
    input { stdin {} }
    output {
      elasticsearch {
        action => "index"
        hosts => "http://172.17.0.2:9200"
        index => "my_log"#在es中的索引
      }
      # 这里输出调试，正式运行时可以注释掉
      stdout {
          codec => json_lines
      }
      #redis {
      #  codec => plain
      #  host => ["127.0.0.1:6379"]
      #  data_type => list
      #  key => logstash
      #}
    }
    ➜  logstash

配置文件中的172.17.0.2:9200 是elasticsearch在docker容器中的IP地址，具体的获取方式为：

➜  logstash docker ps -a
    CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                              NAMES
    36134d90b5cd        98f8400d2944        "/docker-entrypoint.…"   9 minutes ago       Up 9 minutes                                           priceless_feistel
    687bf289bb43        43cf30ef8591        "/docker-entrypoint.…"   13 minutes ago      Up 13 minutes       0.0.0.0:9200->9200/tcp, 9300/tcp   my_elasticsearch
    ➜  logstash docker inspect 687bf289bb43 | grep IP
                "LinkLocalIPv6Address": "",
                "LinkLocalIPv6PrefixLen": 0,
                "SecondaryIPAddresses": null,
                "SecondaryIPv6Addresses": null,
                "GlobalIPv6Address": "",
                "GlobalIPv6PrefixLen": 0,
                "IPAddress": "172.17.0.2",
                "IPPrefixLen": 16,
                "IPv6Gateway": "",
                        "IPAMConfig": null,
                        "IPAddress": "172.17.0.2",
                        "IPPrefixLen": 16,
                        "IPv6Gateway": "",
                        "GlobalIPv6Address": "",
                        "GlobalIPv6PrefixLen": 0,
    ➜  logstash

### 测试logstash ###

如此，关于Logstash的准备工作算是完成了，接下来就要部署logstash服务了。

➜  logstash docker run -i -t 98f8400d2944 /bin/bash
    root@36134d90b5cd:/etc/logstash# ls
    conf.d	jvm.options  log4j2.properties	log4j2.properties.dist	logstash.yml  startup.options
    root@36134d90b5cd:/etc/logstash# cd conf.d/
    root@36134d90b5cd:/etc/logstash/conf.d# ls
    root@36134d90b5cd:/etc/logstash/conf.d# cd ..
    root@36134d90b5cd:/etc/logstash# ls
    conf.d	jvm.options  log4j2.properties	log4j2.properties.dist	logstash.yml  startup.options
    root@36134d90b5cd:/etc/logstash# ls
    conf.d	jvm.options  log4j2.properties	log4j2.properties.dist	logstash.conf  logstash.yml  startup.options
    root@36134d90b5cd:/etc/logstash# logstash -f logstash.conf
    
    
    Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties
    03:00:36.516 [main] INFO  logstash.setting.writabledirectory - Creating directory {:setting=>"path.queue", :path=>"/var/lib/logstash/queue"}
    03:00:36.520 [main] INFO  logstash.setting.writabledirectory - Creating directory {:setting=>"path.dead_letter_queue", :path=>"/var/lib/logstash/dead_letter_queue"}
    03:00:36.552 [LogStash::Runner] INFO  logstash.agent - No persistent UUID file found. Generating new UUID {:uuid=>"fc54205f-077a-4c07-9ae4-85f689c970b1", :path=>"/var/lib/logstash/uuid"}
    03:00:37.093 [[main]-pipeline-manager] INFO  logstash.outputs.elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://172.17.0.2:9200/]}}
    03:00:37.094 [[main]-pipeline-manager] INFO  logstash.outputs.elasticsearch - Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://172.17.0.2:9200/, :path=>"/"}
    03:00:37.188 [[main]-pipeline-manager] WARN  logstash.outputs.elasticsearch - Restored connection to ES instance {:url=>"http://172.17.0.2:9200/"}
    03:00:37.200 [[main]-pipeline-manager] INFO  logstash.outputs.elasticsearch - Using mapping template from {:path=>nil}
    03:00:37.460 [[main]-pipeline-manager] INFO  logstash.outputs.elasticsearch - Attempting to install template {:manage_template=>{"template"=>"logstash-*", "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"_all"=>{"enabled"=>true, "omit_norms"=>true}, "dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"string", "index"=>"analyzed", "omit_norms"=>true, "fielddata"=>{"format"=>"disabled"}}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"string", "index"=>"analyzed", "omit_norms"=>true, "fielddata"=>{"format"=>"disabled"}, "fields"=>{"raw"=>{"type"=>"string", "index"=>"not_analyzed", "doc_values"=>true, "ignore_above"=>256}}}}}, {"float_fields"=>{"match"=>"*", "match_mapping_type"=>"float", "mapping"=>{"type"=>"float", "doc_values"=>true}}}, {"double_fields"=>{"match"=>"*", "match_mapping_type"=>"double", "mapping"=>{"type"=>"double", "doc_values"=>true}}}, {"byte_fields"=>{"match"=>"*", "match_mapping_type"=>"byte", "mapping"=>{"type"=>"byte", "doc_values"=>true}}}, {"short_fields"=>{"match"=>"*", "match_mapping_type"=>"short", "mapping"=>{"type"=>"short", "doc_values"=>true}}}, {"integer_fields"=>{"match"=>"*", "match_mapping_type"=>"integer", "mapping"=>{"type"=>"integer", "doc_values"=>true}}}, {"long_fields"=>{"match"=>"*", "match_mapping_type"=>"long", "mapping"=>{"type"=>"long", "doc_values"=>true}}}, {"date_fields"=>{"match"=>"*", "match_mapping_type"=>"date", "mapping"=>{"type"=>"date", "doc_values"=>true}}}, {"geo_point_fields"=>{"match"=>"*", "match_mapping_type"=>"geo_point", "mapping"=>{"type"=>"geo_point", "doc_values"=>true}}}], "properties"=>{"@timestamp"=>{"type"=>"date", "doc_values"=>true}, "@version"=>{"type"=>"string", "index"=>"not_analyzed", "doc_values"=>true}, "geoip"=>{"type"=>"object", "dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip", "doc_values"=>true}, "location"=>{"type"=>"geo_point", "doc_values"=>true}, "latitude"=>{"type"=>"float", "doc_values"=>true}, "longitude"=>{"type"=>"float", "doc_values"=>true}}}}}}}}
    03:00:37.477 [[main]-pipeline-manager] INFO  logstash.outputs.elasticsearch - Installing elasticsearch template to _template/logstash
    03:00:37.626 [[main]-pipeline-manager] INFO  logstash.outputs.elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://172.17.0.2:9200"]}
    03:00:37.629 [[main]-pipeline-manager] INFO  logstash.pipeline - Starting pipeline {"id"=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>250}
    03:00:37.648 [[main]-pipeline-manager] INFO  logstash.pipeline - Pipeline main started
    The stdin plugin is now waiting for input:
    03:00:37.814 [Api Webserver] INFO  logstash.agent - Successfully started Logstash API endpoint {:port=>9600}
    {"@timestamp":"2019-02-16T03:00:37.715Z","@version":"1","host":"36134d90b5cd","message":""}
    {"@timestamp":"2019-02-16T03:00:37.681Z","@version":"1","host":"36134d90b5cd","message":""}
    helloworld
    {"@timestamp":"2019-02-16T03:00:53.619Z","@version":"1","host":"36134d90b5cd","message":"helloworld"}
    what's up?
    {"@timestamp":"2019-02-16T03:01:41.383Z","@version":"1","host":"36134d90b5cd","message":"what's up?"}

示例中的helloworld和what’sup? 是通过input的stdin测试的，没有报错，说明logstash已经能正确运行了。

## 搭建ElasticSearch ##

### 准备镜像 ###

docker search elasticsearch // 省略输出
    docker pull elasticsearch // 省略输出

由于单纯的通过HTTP去查看elasticsearch运行的信息，有点繁琐，所以这里学着网上的教程，自己创建一个Dockerfile，然后直接安装好head插件，这样就可以很轻松的在webUI上操作elasticsearch

### Dockerfile ###

FROM elasticsearch:2.3.5
     
    RUN /usr/share/elasticsearch/bin/plugin install mobz/elasticsearch-head
     
    EXPOSE 9200

### build 自己的镜像 ###

# 进入到此Dockerfile所在的同级目录，然后开始创建镜像
     docker build --name "es_test" .

查看创建好的镜像

➜  elasticsearch ls
    Dockerfile        elasticsearch.yml operate.py
    ➜  elasticsearch docker images
    REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
    es_test             latest              43cf30ef8591        2 days ago          349MB
    redis               latest              0f55cf3661e9        9 days ago          95MB
    logstash            5.5.2               98f8400d2944        17 months ago       724MB
    elasticsearch       2.3.5               1c3e7681c53c        2 years ago         346MB
    ➜  elasticsearch

### 运行elasticsearch ###

elasticsearch docker ps -a
    CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
    ➜  elasticsearch docker run -d -p 9200:9200 --name="my_elasticsearch" 43cf30ef8591
    
    687bf289bb43d0a037293a2f308b59ce50f5fb475f0b50dd075a0f304be789a3
    ➜  elasticsearch docker ps -a
    CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                              NAMES
    687bf289bb43        43cf30ef8591        "/docker-entrypoint.…"   18 seconds ago      Up 17 seconds       0.0.0.0:9200->9200/tcp, 9300/tcp   my_elasticsearch
    ➜  elasticsearch docker inspect 687bf289bb43 | grep IP
                "LinkLocalIPv6Address": "",
                "LinkLocalIPv6PrefixLen": 0,
                "SecondaryIPAddresses": null,
                "SecondaryIPv6Addresses": null,
                "GlobalIPv6Address": "",
                "GlobalIPv6PrefixLen": 0,
                "IPAddress": "172.17.0.2",
                "IPPrefixLen": 16,
                "IPv6Gateway": "",
                        "IPAMConfig": null,
                        "IPAddress": "172.17.0.2",
                        "IPPrefixLen": 16,
                        "IPv6Gateway": "",
                        "GlobalIPv6Address": "",
                        "GlobalIPv6PrefixLen": 0,
    ➜  elasticsearch

docker的run命令中通过-p参数指定了宿主机端口9200 与容器的9200端口互通，所以我们就可以在宿主机的9200端口查看到具体的运行信息。  
![宿主机9200端口运行][9200]  
因为在镜像中已经安装好了elasticsearch的head插件，下面访问head下的elasticsearch的运行情况吧。  
![在elasticsearch的head插件中查看运行情况][elasticsearch_head]

### 在elasticsearch中查看logstash中的数据 ###

在开始查看之前，我们现在刚才logstash配置的终端中输入一些数据，作为数据源的积累。  
![随意加一些数据源][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L01hcmtzaW5vYmVyZw_size_16_color_FFFFFF_t_70]  
然后我们去elasticsearch中查看是否有对应的数据。  
![elasticsearch中查看是否有对应信息][elasticsearch]  
至此，elasticsearch服务也算是搭建完成了。

## 搭建kibana服务 ##

docker search kibana
    docker pull nshou/elasticsearch-kibana
    Using default tag: latest
    latest: Pulling from nshou/elasticsearch-kibana
    8e3ba11ec2a2: Pull complete
    311ad0da4533: Pull complete
    391a6a6b3651: Pull complete
    b80b8b42a95a: Pull complete
    0ae9073eaa12: Pull complete
    Digest: sha256:e504d283be8cd13f9e1d1ced9a67a140b40a10f5b39f4dde4010dbebcbdd6da0
    Status: Downloaded newer image for nshou/elasticsearch-kibana:latest

开始运行

➜  kibana docker images
    REPOSITORY                   TAG                 IMAGE ID            CREATED             SIZE
    es_test                      latest              43cf30ef8591        2 days ago          349MB
    redis                        latest              0f55cf3661e9        9 days ago          95MB
    nshou/elasticsearch-kibana   latest              1509f8ccdbf3        2 weeks ago         383MB
    logstash                     5.5.2               98f8400d2944        17 months ago       724MB
    elasticsearch                2.3.5               1c3e7681c53c        2 years ago         346MB
    ➜  kibana docker run --name my_kibana -p 5601:5601 -d -e ELASTICSEARCH_URL=http://172.17.0.2:9200 1509f8ccdbf3
    e0e7eab877ea3a1e603efbf041038f7e3c36e2b21e3e1934bf7e7d39606b8bac
    ➜  kibana docker ps -a
    CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                              NAMES
    e0e7eab877ea        1509f8ccdbf3        "/bin/sh -c 'sh elas…"   4 seconds ago       Up 4 seconds        0.0.0.0:5601->5601/tcp, 9200/tcp   my_kibana
    36134d90b5cd        98f8400d2944        "/docker-entrypoint.…"   33 minutes ago      Up 33 minutes                                          priceless_feistel
    687bf289bb43        43cf30ef8591        "/docker-entrypoint.…"   37 minutes ago      Up 37 minutes       0.0.0.0:9200->9200/tcp, 9300/tcp   my_elasticsearch
    ➜  kibana

注意上方的ELASTICSEARCH\_URL是根据之前部署好的elasticsearch的内网IP服务地址，否则可能导致连接不上。

![kibana运行示例][kibana]

## 总结 ##

到这里真的很庆幸，docker着实有用啊，虽说也可以在本机直接部署安装，但是那样污染了宿主机不说，还没办法做好很好的隔离效果。

不管我们有什么想法，把服务图画出来，然后就是把一个个容器跑起来，再进行对应的配置部署，一个demo服务体系就可以很轻松的被构建出来。

感谢docker，感恩技术。

[ELK]: /images/20220319/d93c8f11a6dc4e1e9e8258c44b59b9ec.png
[9200]: /images/20220319/850d4ea08deb451bb48c780f14e7d5e5.png
[elasticsearch_head]: /images/20220319/9d9ba12b3b4a4728af0a0659c05bf4dd.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L01hcmtzaW5vYmVyZw_size_16_color_FFFFFF_t_70]: /images/20220319/a5f033148da54ebf81df7b529c297fd3.png
[elasticsearch]: /images/20220319/b12f641645304bf4961aad7cef3a121a.png
[kibana]: /images/20220319/3b163e3a43d74b2baaeeb8bc787b8c7b.png