MongoDB Sharding集群部署

Dear 丶 2022-08-07 05:58 276阅读 0赞

和Replica Set类似，都需要一个仲裁节点，但是Sharding还需要配置节点和路由节点。就三种集群搭建方式来说，这种是最复杂的。部署图如下：

![Center][]

MongoDB Sharding Cluster需要三种角色：  
（1）Shard Server：即存储实际数据的分片，每个Shard可以是一个mongod实例，也可以是一组mongod实例构成的Replica Set。  
（2）Config Server：用来存储所有shard节点的配置信息、每个chunk的shard key范围、chunk在各shard的分布情况、该集群中所有DB和collection的sharding配置信息。  
（3）Route Process：这是一个前端路由，客户端由此接入，然后询问Config Servers需要到哪个shard上查询或保存记录，再连接相应的shard进行操作，最后将结果返回给客户端，而这一切对客户端是透明的，客户端不用关心所操作的记录存储在哪个shard上。  
下面我们配置测试一下Sharding：  
**一、机器环境  
1.Shard Server**  
shard\_a  
192.168.110.71：10000  
192.168.100.90：10000  
192.168.100.110：10000 （Arbiter Server）  
shard\_b  
192.168.110.71：10001  
192.168.100.90：10001  
192.168.100.110：10001 （Arbiter Server）  
**2.Config Server**  
192.168.110.71：20000  
**3.Route Process**  
192.168.100.110：30000  
**二、配置  
1.创建数据目录**  
在这3台机器的mongodb安装目录下创建日志目录和数据目录  
mkdir -p logs  
mkdir -p conf  
mkdir -p data/shard\_a  
mkdir -p data/shard\_b  
在192.168.110.71创建config数据目录  
mkdir -p data/config

**2.创建配置文件  
shard\_a.conf**

port=10000
    pidfilepath=/home/slim/mongodb-2.6.8/data/shard_a.pid
    dbpath=/home/slim/mongodb-2.6.8/data/shard_a
    directoryperdb=true
    logpath=/home/slim/mongodb-2.6.8/logs/shard_a.log
    logappend=true
    fork=true
    profile=1
    slowms = 5
    noprealloc=false
    replSet=shard_a
    oplogSize=100
    shardsvr=true

**shard\_b.conf**

port=100001
    pidfilepath=/home/slim/mongodb-2.6.8/data/shard_b.pid
    dbpath=/home/slim/mongodb-2.6.8/data/shard_b
    directoryperdb=true
    logpath=/home/slim/mongodb-2.6.8/logs/shard_b.log
    logappend=true
    fork=true
    profile=1
    slowms = 5
    noprealloc=false
    replSet=shard_b
    oplogSize=100
    shardsvr=true

**config.conf**

port=20000
    pidfilepath=/home/slim/mongodb-2.6.8/data/config.pid
    dbpath=/home/slim/mongodb-2.6.8/data/config
    directoryperdb=true
    logpath=/home/slim/mongodb-2.6.8/logs/config.log
    logappend=true
    fork=true
    profile=0
    configsvr=true

**mongos.conf**

port=30000
    logpath=/home/slim/mongodb-2.6.8/logs/mongos.log
    logappend=true
    fork=true
    maxConns=1000
    chunkSize=1
    configdb=192.168.110.71:20000

这里面暂不配置 bind\_ip，方便配置文件维护。 一般来说一个数据节点对应一个配置节点，仲裁节点则不需要对应的配置节点。

**参数说明**：

bind\_ip:mongodb所绑定的ip地址

port:端口

pidfilepath：pid文件目录

dbpath:数据库存储文件目录

directoryperdb:设置每个数据库将被保存在一个单独的目录

logpath:日志文件路径

logappend:以追加方式写入,false:重新启动覆盖文件

fork:以守护进程的方式运行，创建服务器进程

profile:设置日志级别

\#0 - 关闭性能分析，测试环境可以打开，生成环境关闭，对性能有很大影响;

\#1 - 开启慢查询日志，执行时间大于100毫秒的语句

\#2 - 开启所有操作日志

slowms:慢查询的时间设置

noprealloc:预分配方式,默认false,true禁用预分配的数据文件

replSet:指定复制集名称

shardsvr:确定是否分片

oplogSize:操作日志文件的最大大小。单位为Mb，默认为硬盘剩余空间的5%

configsvr:声明这是一个集群的config服务,默认端口27019，默认目录/data/configdb

configdb:监听的配置服务器,只能有1个或者3个

chunkSize:单位 mb 生成环境请使用 100 或删除,删除后默认是64

maxConns:最大连接数,默认值：取决于系统（即的ulimit和文件描述符）限制,能设置该值大于20000。

**二、配置Replica Set**

启动每台机器的 Shard Server服务.

./bin/mongod -f conf/shard\_a.conf &

./bin/mongod -f conf/shard\_b.conf &

待全部启动完毕后，登录任意一台mongodb配置 replSet，如下：

**1.配置shard\_a**

./bin/mongo 192.168.100.90:10000
    MongoDB shell version: 2.0.6
    connecting to: 192.168.100.90:10000/test
    > use admin;
    switched to db admin
    > config_shard_a={_id : 'shard_a', members:[{_id:0, host:'192.168.110.71:10000'},{_id:1, host:'192.168.100.90:10000'},{_id:2,host:'192.168.100.110:10000',arbiterOnly:true}]};
    {
            "_id" : "shard_a",
            "members" : [
                    {
                            "_id" : 0,
                            "host" : "192.168.110.71:10000"
                    },
                    {
                            "_id" : 1,
                            "host" : "192.168.100.90:10000"
                    },
                    {
                            "_id" : 2,
                            "host" : "192.168.100.110:10000",
                            "arbiterOnly" : true
                    }
            ]
    }
    > rs.initiate(config_shard_a);
    {
            "info" : "Config now saved locally. Should come online in about a minute.",
            "ok" : 1
    }
    > rs.status();
    {
            "set" : "shard_a",
            "date" : ISODate("2015-03-17T05:10:38Z"),
            "myState" : 1,
            "members" : [
                    {
                            "_id" : 0,
                            "name" : "192.168.110.71:10000",
                            "health" : 1,
                            "state" : 2,
                            "stateStr" : "SECONDARY",
                            "uptime" : 88,
                            "optime" : {
                                    "t" : 1426568948000,
                                    "i" : 1
                            },
                            "optimeDate" : ISODate("2015-03-17T05:09:08Z"),
                            "lastHeartbeat" : ISODate("2015-03-17T05:10:37Z"),
                            "lastHeartbeatRecv" : ISODate("2015-03-17T05:10:37Z"),
                            "pingMs" : 0,
                            "syncingTo" : "192.168.100.90:10000"
                    },
                    {
                            "_id" : 1,
                            "name" : "192.168.100.90:10000",
                            "health" : 1,
                            "state" : 1,
                            "stateStr" : "PRIMARY",
                            "uptime" : 6289,
                            "optime" : {
                                    "t" : 1426568948000,
                                    "i" : 1
                            },
                            "optimeDate" : ISODate("2015-03-17T05:09:08Z"),
                            "electionTime" : {
                                    "t" : 1426568958000,
                                    "i" : 1
                            },
                            "electionDate" : ISODate("2015-03-17T05:09:18Z"),
                            "self" : true
                    },
                    {
                            "_id" : 2,
                            "name" : "192.168.100.110:10000",
                            "health" : 1,
                            "state" : 7,
                            "stateStr" : "ARBITER",
                            "uptime" : 88,
                            "lastHeartbeat" : ISODate("2015-03-17T05:10:37Z"),
                            "lastHeartbeatRecv" : ISODate("2015-03-17T05:10:37Z"),
                            "pingMs" : 0
                    }
            ],
            "ok" : 1
    }

**2.配置shard\_b**  
登录192.168.100.90:10001

./bin/mongo 192.168.100.90:10001
    MongoDB shell version: 2.0.6
    connecting to: 192.168.110.71:10001/test
    > use admin;
    switched to db admin
    > config_shard_b={_id : 'shard_b', members:[{_id:0, host:'192.168.110.71:10001'},{_id:1, host:'192.168.100.90:10001'},{_id:2,host:'192.168.100.110:10001',arbiterOnly:true}]};
    {
            "_id" : "shard_b",
            "members" : [
                    {
                            "_id" : 0,
                            "host" : "192.168.110.71:10001"
                    },
                    {
                            "_id" : 1,
                            "host" : "192.168.100.90:10001"
                    },
                    {
                            "_id" : 2,
                            "host" : "192.168.100.110:10001",
                            "arbiterOnly" : true
                    }
            ]
    }
    > rs.initiate(config_shard_b);
    {
            "info" : "Config now saved locally. Should come online in about a minute.",
            "ok" : 1
    }
    STARTUP2> rs.status();
    {
            "set" : "shard_b",
            "date" : ISODate("2015-03-17T05:17:48Z"),
            "myState" : 1,
            "members" : [
                    {
                            "_id" : 0,
                            "name" : "192.168.110.71:10001",
                            "health" : 1,
                            "state" : 1,
                            "stateStr" : "PRIMARY",
                            "uptime" : 128,
                            "optime" : {
                                    "t" : 1426569457000,
                                    "i" : 1
                            },
                            "optimeDate" : ISODate("2015-03-17T05:17:37Z"),
                            "electionTime" : {
                                    "t" : 1426569463000,
                                    "i" : 1
                            },
                            "electionDate" : ISODate("2015-03-17T05:17:43Z"),
                            "self" : true
                    },
                    {
                            "_id" : 1,
                            "name" : "192.168.100.90:10001",
                            "health" : 1,
                            "state" : 2,
                            "stateStr" : "SECONDARY",
                            "uptime" : 11,
                            "optime" : {
                                    "t" : 1426569457000,
                                    "i" : 1
                            },
                            "optimeDate" : ISODate("2015-03-17T05:17:37Z"),
                            "lastHeartbeat" : ISODate("2015-03-17T05:17:47Z"),
                            "lastHeartbeatRecv" : ISODate("2015-03-17T05:17:48Z"),
                            "pingMs" : 1,
                            "lastHeartbeatMessage" : "syncing to: 192.168.110.71:10001",
                            "syncingTo" : "192.168.110.71:10001"
                    },
                    {
                            "_id" : 2,
                            "name" : "192.168.100.110:10001",
                            "health" : 1,
                            "state" : 7,
                            "stateStr" : "ARBITER",
                            "uptime" : 11,
                            "lastHeartbeat" : ISODate("2015-03-17T05:17:47Z"),
                            "lastHeartbeatRecv" : ISODate("2015-03-17T05:17:47Z"),
                            "pingMs" : 0
                    }
            ],
            "ok" : 1
    }

**注**：配置shard时不能连接ARBITER配置，否则会抛出：  
"errmsg" : "couldn't initiate : initiation and reconfiguration of a replica set must be sent to a node that can become primary"  
**三、配置Sharding  
1.启动配置服务**  
./bin/mongod -f conf/config.conf &  
**2.启动路由服务**  
 ./bin/mongos -f conf/mongos.conf &  
**3. 路由节点配置**

./bin/mongo 192.168.100.110:30000
    MongoDB shell version: 2.0.6
    connecting to: 192.168.100.110:30000/test
    mongos> use admin;
    switched to db admin
    mongos> db.runCommand({addshard:"shard_a/192.168.110.71:10000,192.168.100.90:10000",name:"shard_a"});
    { "shardAdded" : "shard_a", "ok" : 1 }
    mongos> db.runCommand({addshard:"shard_b/192.168.110.71:10001,192.168.100.90:10001",name:"shard_b"});
    { "shardAdded" : "shard_b", "ok" : 1 }
    mongos> db.adminCommand({listshards:1});
    {
            "shards" : [
                    {
                            "_id" : "shard_a",
                            "host" : "shard_a/192.168.100.90:10000,192.168.110.71:10000" 
                    },
                    {
                            "_id" : "shard_b",
                            "host" : "shard_b/192.168.100.90:10001,192.168.110.71:10001" 
                    }
            ],
            "ok" : 1
    }

replica set + shard功能就配置好了，注意：虽然配置好了，但是还要声明库和表要进行分片。  
注：  
启动路由服务时，若报如下错误：  
BadValue need either 1 or 3 configdbs  
try './bin/mongos --help' for more information  
原因以及解决办法：  
mongos.conf文件中configdb参数有误，在这里路由个数，只能为 **1或3**。configdb =192.168.110.71:20000  
4.声明库和表要分片  
添加分片存储的数据库

mongos> db.runCommand({enablesharding:"shard"});
    { "ok" : 1 }

设置分片的集合名称，且必须指定Shard Key：

mongos> db.runCommand({shardcollection:"shard.user", key:{_id:1}});
    { "collectionsharded" : "shard.user", "ok" : 1 }

查看sharding状态：

mongos> db.printShardingStatus();
    --- Sharding Status ---
      sharding version: {
            "_id" : 1,
            "version" : 4,
            "minCompatibleVersion" : 4,
            "currentVersion" : 5,
            "clusterId" : ObjectId("5507bbdfd9b4a799a27e0120")
    }
      shards:
            { "_id" : "shard_a", "host" : "shard_a/192.168.100.90:10000,192.168.110.71:10000" }
            { "_id" : "shard_b", "host" : "shard_b/192.168.100.90:10001,192.168.110.71:10001" }
      databases:
            { "_id" : "admin", "partitioned" : false, "primary" : "config" }
            { "_id" : "shard", "partitioned" : true, "primary" : "shard_b" }
                    shard.user chunks:
                                    shard_b 1
                            { "_id" : { $minKey : 1 } } -->> { "_id" : { $maxKey : 1 } } on : shard_b { "t" : 1000, "i" : 0 }

查看数据库状态：

mongos> use shard;
    mongos> db.stats();
    {
            "raw" : {
                    "shard_b/192.168.100.90:10001,192.168.110.71:10001" : {
                            "db" : "shard",
                            "collections" : 4,
                            "objects" : 6,
                            "avgObjSize" : 122.66666666666667,
                            "dataSize" : 736,
                            "storageSize" : 1073152,
                            "numExtents" : 4,
                            "indexes" : 1,
                            "indexSize" : 8176,
                            "fileSize" : 67108864,
                            "nsSizeMB" : 16,
                            "dataFileVersion" : {
                                    "major" : 4,
                                    "minor" : 5
                            },
                            "extentFreeList" : {
                                    "num" : 0,
                                    "totalSize" : 0
                            },
                            "ok" : 1
                    }
            },
            "objects" : 6,
            "avgObjSize" : 122,
            "dataSize" : 736,
            "storageSize" : 1073152,
            "numExtents" : 4,
            "indexes" : 1,
            "indexSize" : 8176,
            "fileSize" : 67108864,
            "extentFreeList" : {
                    "num" : 0,
                    "totalSize" : 0
            },
            "ok" : 1
    }

查看表状态：

mongos> db.user.stats();
    {
            "sharded" : true,
            "systemFlags" : 1,
            "userFlags" : 1,
            "ns" : "shard.user",
            "count" : 0,
            "numExtents" : 1,
            "size" : 0,
            "storageSize" : 8192,
            "totalIndexSize" : 8176,
            "indexSizes" : {
                    "_id_" : 8176
            },
            "avgObjSize" : 0,
            "nindexes" : 1,
            "nchunks" : 1,
            "shards" : {
                    "shard_b" : {
                            "ns" : "shard.user",
                            "count" : 0,
                            "size" : 0,
                            "storageSize" : 8192,
                            "numExtents" : 1,
                            "nindexes" : 1,
                            "lastExtentSize" : 8192,
                            "paddingFactor" : 1,
                            "systemFlags" : 1,
                            "userFlags" : 1,
                            "totalIndexSize" : 8176,
                            "indexSizes" : {
                                    "_id_" : 8176
                            },
                            "ok" : 1
                    }
            },
            "ok" : 1
    }

注：  
1. 报以下错误，原因是库还不允许分片  
\{ "ok" : 0, "errmsg" : "sharding not enabled for db" \}  
2.报以下错误，原因是Shard Key设置有误  
"errmsg" : "Unsupported shard key pattern. Pattern must either be a single hashed field, or a list of ascending fields."  
**四、测试**  
登录mongos服务,执行如下脚本：  
for (var i = 1; i <= 20000; i++) db.user.save(\{id:i,name:"jack"+i,sex:"male",age:27,value:"test"\});  
查看结果：

mongos> db.user.stats();
    {
            "sharded" : true,
            "systemFlags" : 0,
            "userFlags" : 1,
            "ns" : "shard.user",
            "count" : 20000,
            "numExtents" : 9,
            "size" : 2240000,
            "storageSize" : 3489792,
            "totalIndexSize" : 670432,
            "indexSizes" : {
                    "_id_" : 670432
            },
            "avgObjSize" : 112,
            "nindexes" : 1,
            "nchunks" : 4,
            "shards" : {
                    "shard_a" : {
                            "ns" : "shard.user",
                            "count" : 15058,
                            "size" : 1686496,
                            "avgObjSize" : 112,
                            "storageSize" : 2793472,
                            "numExtents" : 5,
                            "nindexes" : 1,
                            "lastExtentSize" : 2097152,
                            "paddingFactor" : 1,
                            "systemFlags" : 0,
                            "userFlags" : 1,
                            "totalIndexSize" : 498736,
                            "indexSizes" : {
                                    "_id_" : 498736
                            },
                            "ok" : 1
                    },
                    "shard_b" : {
                            "ns" : "shard.user",
                            "count" : 4942,
                            "size" : 553504,
                            "avgObjSize" : 112,
                            "storageSize" : 696320,
                            "numExtents" : 4,
                            "nindexes" : 1,
                            "lastExtentSize" : 524288,
                            "paddingFactor" : 1,
                            "systemFlags" : 1,
                            "userFlags" : 1,
                            "totalIndexSize" : 171696,
                            "indexSizes" : {
                                    "_id_" : 171696
                            },
                            "ok" : 1
                    }
            },
            "ok" : 1
    }

从结果可以看到：  
shard\_a：15058  
shard\_b：4942

[Center]: /images/20220805/90182dcdaae54b92ac259608830f16a7.png