Druid集群配置

妖狐艹你老母 2022-07-16 12:58 381阅读 0赞

**一、默认端口**

![Center][]

**二、集群配置思想**

（一）本集群有五个节点（master, slave1,slave2,slave3,slave4），节点安排情况如下：

master节点：Mysql server, coordinator node, overlord node

slave1节点:   historical node， middleManager node

slave2节点:   historical node， middleManager node

slave3节点:   broker node

slave4节点:   realtime node(没有配置，空着)

（二）配置deep storage（保存冷数据）为HDFS

**三、集群配置**

（一）配置所有节点（master, slave1,slave2,slave3,slave4）的\_common/common.runtime.properties文件

#
    # Licensed to Metamarkets Group Inc. (Metamarkets) under one
    # or more contributor license agreements. See the NOTICE file
    # distributed with this work for additional information
    # regarding copyright ownership. Metamarkets licenses this file
    # to you under the Apache License, Version 2.0 (the
    # "License"); you may not use this file except in compliance
    # with the License. You may obtain a copy of the License at
    #
    # http://www.apache.org/licenses/LICENSE-2.0
    #
    # Unless required by applicable law or agreed to in writing,
    # software distributed under the License is distributed on an
    # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    # KIND, either express or implied. See the License for the
    # specific language governing permissions and limitations
    # under the License.
    #
    
    #
    # Extensions
    #
    
    # This is not the full list of Druid extensions, but common ones that people often use. You may need to change this list
    # based on your particular setup.
    #
    # Licensed to Metamarkets Group Inc. (Metamarkets) under one
    # or more contributor license agreements. See the NOTICE file
    # distributed with this work for additional information
    # regarding copyright ownership. Metamarkets licenses this file
    # to you under the Apache License, Version 2.0 (the
    # "License"); you may not use this file except in compliance
    # with the License. You may obtain a copy of the License at
    #
    # http://www.apache.org/licenses/LICENSE-2.0
    #
    # Unless required by applicable law or agreed to in writing,
    # software distributed under the License is distributed on an
    # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    # KIND, either express or implied. See the License for the
    # specific language governing permissions and limitations
    # under the License.
    #
    
    #
    # Extensions
    #
    
    # This is not the full list of Druid extensions, but common ones that people often use. You may need to change this list
    # based on your particular setup.
    druid.extensions.loadList=["mysql-metadata-storage","druid-hdfs-storage"]
    
    
    # If you have a different version of Hadoop, place your Hadoop client jar files in your hadoop-dependencies directory
    # and uncomment the line below to point to your directory.
    #druid.extensions.hadoopDependenciesDir=/my/dir/hadoop-dependencies
    
    #
    # Logging
    #
    
    # Log all runtime properties on startup. Disable to avoid logging properties on startup:
    druid.startup.logging.logProperties=true
    
    #
    # Zookeeper
    #
    
    druid.zk.service.host=master:2181,slave1:2181,slave2:2181,slave3:2181,slave4:2181
    #druid.zk.service.host=master:2181
    druid.zk.paths.base=/druid
    
    #
    # Metadata storage
    #
    
    # For Derby server on your Druid Coordinator (only viable in a cluster with a single Coordinator, no fail-over):
    #druid.metadata.storage.type=derby
    #druid.metadata.storage.connector.connectURI=jdbc:derby://metadata.store.ip:1527/var/druid/metadata.db;create=true
    #druid.metadata.storage.connector.host=metadata.store.ip
    #druid.metadata.storage.connector.port=1527
    
    # For MySQL:
    druid.metadata.storage.type=mysql
    druid.metadata.storage.connector.connectURI=jdbc:mysql://localhost:3306/druid?characterEncoding=UTF-8
    druid.metadata.storage.connector.user=***
    druid.metadata.storage.connector.password=***
    
    # For PostgreSQL (make sure to additionally include the Postgres extension):
    #druid.metadata.storage.type=postgresql
    #druid.metadata.storage.connector.connectURI=jdbc:postgresql://db.example.com:5432/druid
    #druid.metadata.storage.connector.user=...
    #druid.metadata.storage.connector.password=...
    
    #
    # Deep storage
    #
    
    # For local disk (only viable in a cluster if this is a network mount):
    #druid.storage.type=local
    #druid.storage.storageDirectory=var/druid/segments
    
    # For HDFS (make sure to include the HDFS extension and that your Hadoop config files in the cp):
    druid.storage.type=hdfs
    druid.storage.storageDirectory=hdfs://master:9000/druid/segments
    
    # For S3:
    #druid.storage.type=s3
    #druid.storage.bucket=your-bucket
    #druid.storage.baseKey=druid/segments
    #druid.s3.accessKey=...
    #druid.s3.secretKey=...
    
    #
    # Indexing service logs
    #
    
    # For local disk (only viable in a cluster if this is a network mount):
    #druid.indexer.logs.type=file
    #druid.indexer.logs.directory=var/druid/indexing-logs
    
    # For HDFS (make sure to include the HDFS extension and that your Hadoop config files in the cp):
    druid.indexer.logs.type=hdfs
    druid.indexer.logs.directory=hdfs://master:9000/druid/indexing-logs
    
    # For S3:
    #druid.indexer.logs.type=s3
    #druid.indexer.logs.s3Bucket=your-bucket
    #druid.indexer.logs.s3Prefix=druid/indexing-logs
    
    #
    # Service discovery
    #
    
    druid.selectors.indexing.serviceName=druid/overlord
    druid.selectors.coordinator.serviceName=druid/coordinator
    
    #
    # Monitoring
    #
    
    druid.monitoring.monitors=["com.metamx.metrics.JvmMonitor"]
    druid.emitter=logging
    druid.emitter.logging.logLevel=info

（二）配置master节点的coordinator（在coordinator/runtime.properties文件里）

druid.service=druid/coordinator
    druid.port=8081
    
    druid.coordinator.startDelay=PT30S
    druid.coordinator.period=PT30S
    
    #Below it's append by fenghui for test
    druid.host=master

配置master节点的 overlord（在overlord/runtime.properties文件里）

druid.service=druid/overlord
    druid.port=8090
    
    #druid.indexer.queue.startDelay=PT30S
    
    druid.indexer.runner.type=remote
    druid.indexer.storage.type=metadata
    
    #below it is append by fenghui for test!
    druid.host=master
    
    #druid.indexer.logs.type=hdfs
    #druid.indexer.logs.directory=/druid/indexing-logs

（三）配置slave1节点的historical （在historical /runtime.properties文件里）

druid.service=druid/historical
    druid.port=8083
    
    # HTTP server threads
    druid.server.http.numThreads=25
    
    # Processing threads and buffers
    druid.processing.buffer.sizeBytes=6870912
    druid.processing.numThreads=7
    
    # Segment storage
    druid.segmentCache.locations=[{"path":"var/druid/segment-cache","maxSize"\:130000000000}]
    druid.server.maxSize=130000000000
    
    
    
    druid.host=slave1
    
    
    druid.historical.cache.useCache=false
    druid.historical.cache.populateCache=false

配置slave1节点的middleManager （在middleManager /runtime.properties文件里）

druid.service=druid/middleManager
    druid.port=8091
    
    # Number of tasks per middleManager
    druid.worker.capacity=3
    
    # Task launch parameters
    druid.indexer.runner.javaOpts=-server -Xmx256m -Duser.timezone=UTC -Dfile.encoding=UTF-8 -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager
    druid.indexer.task.baseTaskDir=var/druid/task
    
    # HTTP server threads
    druid.server.http.numThreads=25
    
    # Processing threads and buffers
    druid.processing.buffer.sizeBytes=65536
    druid.processing.numThreads=2
    
    # Hadoop indexing
    druid.indexer.task.hadoopWorkingPath=var/druid/hadoop-tmp
    druid.indexer.task.defaultHadoopCoordinates=["org.apache.hadoop:hadoop-client:2.3.0"]
    
    
    
    #below it's append by fenghui for test
    druid.host=slave1
    
    #druid.indexer.logs.type=hdfs
    #druid.indexer.logs.directory=/druid/indexing-logs

（四）配置slave2节点（同slave1节点）

（五）配置slave1节点的broker（在broker/runtime.properties文件里）

druid.service=druid/broker
    druid.port=8082
    
    # HTTP server threads
    druid.broker.http.numConnections=5
    druid.server.http.numThreads=25
    
    # Processing threads and buffers
    druid.processing.buffer.sizeBytes=32768
    druid.processing.numThreads=2
    
    # Query cache
    druid.broker.cache.useCache=true
    druid.broker.cache.populateCache=true
    druid.cache.type=local
    druid.cache.sizeInBytes=2000000000
    
    druid.host=slave3

**三、启动集群命令**

**   ** 分别到每个节点（master, slave1,slave2,slave3,slave4）去启动相应的服务，命令如下（下面是启动broker的命令）：     java -Xmx256m -Duser.timezone=UTC -Dfile.encoding=UTF-8 -classpath conf/druid/\_common:conf/druid/broker:lib/\* io.druid.cli.Main server broker &

**四、验证集群**

（一）打开web端

http://master:8090/console.html

![Center 1][]

http://master:8081/\#/

![Center 2][]

（二）下载[**测试文件**][Link 1]，把文件解压到每个节点（master, slave1,slave2,slave3,slave4）的druid-0.9.1.1目录下，然后再file目录下执行 bash submit\_task.sh命令，可以在http://master:8090/console.html下查看状态

在file目录下执行bash query\_task.sh命令

![Center 3][]

参考文件：

### [druid.io\_druid.io本地集群搭建 / 扩展集群搭建][druid.io_druid.io_ _] ###

**http://druid.io/docs/latest/tutorials/cluster.html**

# [Druid Cluster Setup][] #

# [海量数据实时OLAP分析系统-Druid.io安装配置和体验][OLAP_-Druid.io] #

[Center]: /images/20220715/8fb6a6e5566b4765881a6ebeb3704653.png
[Center 1]: /images/20220715/97f4dc0543f84daaa443ac23e36c37ff.png
[Center 2]: /images/20220715/a71347e62e794453ac5981268cf3d0b8.png
[Link 1]: http://download.csdn.net/detail/fenghuibian/9686740
[Center 3]: /images/20220715/24092e6eefbc40fc98c279000276b555.png
[druid.io_druid.io_ _]: http://blog.csdn.net/u010003835/article/details/52947743
[Druid Cluster Setup]: http://dongguo.me/blog/2015/03/02/druid-cluster-setup/
[OLAP_-Druid.io]: http://lxw1234.com/archives/2015/11/554.htm