Elasticsearch同步数据库-Logstash（windows环境，linux环境大同小异）

Myth丶恋晨 2022-01-31 20:27 105阅读 0赞

### （一）安装elasticsearch（略） ###

### （二）安装head进行查看（略） ###

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDM2OTA3Ng_size_16_color_FFFFFF_t_70][]

### （三）logstash的安装配置 ###

（1）安装logstash

[下载地址][Link 1]  选择下载你对应的es的logstash版本，解压

（2）安装logstash-input-jdbc（连接数据库的）

进入解压后的bin目录，执行：logstash-plugin.bat install logstash-input-jdbc

（3）进行数据同步

Logstash 配置，bin下新建logstash.conf

input {
    jdbc {
            #数据库配置
            jdbc_connection_string => "jdbc:mysql://localhost:3306/test?useUnicode=true&useJDBCCompliantTimezoneShift=true&useLegacyDatetimeCode=false&serverTimezone=UTC"
            jdbc_user => "root"
            jdbc_password => "root"
            #驱动包
            jdbc_driver_library => "D:\soft\logstash-6.5.4\lib\mysql-connector-java-8.0.13.jar"
            jdbc_driver_class => "com.mysql.jdbc.Driver"
            jdbc_paging_enabled => "true"
            jdbc_page_size => "50000"
            schedule => "* * * * *"
            #sql语句
    		statement => "SELECT * from user"
            type => "jdbc"
        }
    }
    
    filter {
        json {
            source => "message"
            remove_field => ["message"]
    }
        #过滤字段
    	mutate {
    		remove_field => ["@version","@timestamp"]
        }
    }
    
    output {
        #es配置
        elasticsearch {
            hosts => ["es地址"]
            index => "user"
            document_type => "user"
            #以数据库的哪个字段作为es的id
            document_id => "%{id}"
        }
        stdout {
            codec => json_lines
        }
    }

（4）启动logstash

进入bin目录执行        logstash.bat -f logstash.conf

（5）进入head查看数据

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDM2OTA3Ng_size_16_color_FFFFFF_t_70 1][]

### 多表同步：**需要在input标签内写多个****jdbc** **标签，根据里面不同的sql语句进行查询数据，output标签内需要写：if\[type\]==“input标签内的type”进行判断对应的读取到的数据进行写入es。** ###

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDM2OTA3Ng_size_16_color_FFFFFF_t_70 2][]

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDM2OTA3Ng_size_16_color_FFFFFF_t_70]: /images/20220201/3934270147fc4d6ea6a7ca81cda92c6b.png
[Link 1]: https://www.elastic.co/cn/downloads/past-releases
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDM2OTA3Ng_size_16_color_FFFFFF_t_70 1]: /images/20220201/bfbbcdf9cdf441bcab9dabefd9da21fe.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDM2OTA3Ng_size_16_color_FFFFFF_t_70 2]: /images/20220201/0f10b392a23f446db71d20c92b7f77cd.png