MongoDB复制集原理、环境配置及基本测试详解

Dear 丶 2022-03-28 14:52 194阅读 0赞

一、MongoDB复制集概述

MongoDB复制集实现了冗余备份和故障转移两大功能，这样能保证数据库的高可用性。在生产环境，复制集至少包括三个节点，其中一个必须为主节点，一个从节点，一个仲裁节点。其中每一个节点都是mongod进程对应的实例，节点间通过心跳检查对方的状态。  
primary节点：负责数据库的读写操作。  
secondary节点：备份primary节点上的数据，可以有多个。  
arbiter节点：主节点故障时，参与复制集剩下节点中选举一个新的primary节点。

二、搭建MongoDB复制集环境

1、环境准备：

1）主机环境

192.168.0.151 db01 \#\# primary节点  
192.168.0.152 db02 \#\# secondary节点  
192.168.0.153 db03 \#\# arbiter节点

2）创建搭建MongoDB用户

groupadd mongodb  
useradd -g mongodb mongodb

3）安装mongodb软件

tar -zxvf mongodb-linux-x86\_64-enterprise-rhel70-3.6.0.tgz  
mv mongodb-linux-x86\_64-enterprise-rhel70-3.6.0 mongodb-3.6.0

2、创建复制集中每个节点存放数据目录：

/home/mongodb/db\_rs0/data/rs0

3、创建复制集中每个节点的日志目录：

/home/mongodb/db\_rs0/logs

4、创建复制集中每个节点启动时所需的配置文件：

cat /home/mongodb/db\_rs0/config\_rs0/rs0.conf

dbpath = /home/mongodb/db\_rs0/data/rs0  
logpath = /home/mongodb/db\_rs0/logs/rs0.log  
logappend = true  
journal = true  
port = 40000  
fork = true  
maxConns = 5000  
bind\_ip = 0.0.0.0  
replSet = rs0

5、启动三个节点对应的mongodb实例

bin/mongod --config /home/mongodb/db\_rs0/config\_rs0/rs0.conf

6、db01下进入mongodb客户端：

bin/mongo --port 40000

7、初始化复制集primary节点、添加second节点和arbiter节点：

1）初始化primary节点：

MongoDB Enterprise > rs.initiate(\{\_id:'rs0',members:\[\{\_id:1,host:'db01:40000'\}\]\})  
\{  
"ok" : 1,  
"operationTime" : Timestamp(1513518326, 1),  
"$clusterTime" : \{  
"clusterTime" : Timestamp(1513518326, 1),  
"signature" : \{  
"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),  
"keyId" : NumberLong(0)  
\}  
\}  
\}

MongoDB Enterprise rs0:SECONDARY> rs.conf()  
\{  
"\_id" : "rs0",  
"version" : 1,  
"protocolVersion" : NumberLong(1),  
"members" : \[  
\{  
"\_id" : 1,  
"host" : "db01:40000",  
"arbiterOnly" : false,  
"buildIndexes" : true,  
"hidden" : false,  
"priority" : 1,  
"tags" : \{  
  
\},  
"slaveDelay" : NumberLong(0),  
"votes" : 1  
\}  
\],  
"settings" : \{  
"chainingAllowed" : true,  
"heartbeatIntervalMillis" : 2000,  
"heartbeatTimeoutSecs" : 10,  
"electionTimeoutMillis" : 10000,  
"catchUpTimeoutMillis" : -1,  
"catchUpTakeoverDelayMillis" : 30000,  
"getLastErrorModes" : \{  
  
\},  
"getLastErrorDefaults" : \{  
"w" : 1,  
"wtimeout" : 0  
\},  
"replicaSetId" : ObjectId("5a3674f6be9a0ef57aaf73ac")  
\}  
\}

2）添加second节点和arbiter节点：

MongoDB Enterprise rs0:PRIMARY> rs.add("db02:40000")  
\{  
"ok" : 1,  
"operationTime" : Timestamp(1513518502, 1),  
"$clusterTime" : \{  
"clusterTime" : Timestamp(1513518502, 1),  
"signature" : \{  
"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),  
"keyId" : NumberLong(0)  
\}  
\}  
\}  
MongoDB Enterprise rs0:PRIMARY> rs.addArb("db03:40000")  
\{  
"ok" : 1,  
"operationTime" : Timestamp(1513518525, 1),  
"$clusterTime" : \{  
"clusterTime" : Timestamp(1513518525, 1),  
"signature" : \{  
"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),  
"keyId" : NumberLong(0)  
\}  
\}  
\}

8、检查各个节点local库信息：

MongoDB Enterprise rs0:PRIMARY> use local  
switched to db local  
MongoDB Enterprise rs0:PRIMARY> show collections  
me  
oplog.rs  
replset.election  
replset.minvalid  
startup\_log  
system.replset  
system.rollback.id

MongoDB Enterprise rs0:SECONDARY> use local  
switched to db local  
MongoDB Enterprise rs0:SECONDARY>  
MongoDB Enterprise rs0:SECONDARY>  
MongoDB Enterprise rs0:SECONDARY> show collections  
me  
oplog.rs  
replset.minvalid  
replset.oplogTruncateAfterPoint  
startup\_log  
system.replset  
system.rollback.id

MongoDB Enterprise rs0:ARBITER> use local  
switched to db local  
MongoDB Enterprise rs0:ARBITER> show collections  
me  
replset.minvalid  
startup\_log  
system.replset  
system.rollback.id

9、检查复制集状态：

MongoDB Enterprise rs0:PRIMARY> rs.status()  
\{  
"set" : "rs0", //复制集名称  
"date" : ISODate("2017-12-17T13:49:23.671Z"), //当前实例所在服务器的时间  
"myState" : 1, //当前节点成员在复制集中的位置，如1表示primary，2表示secondry。  
"term" : NumberLong(1),  
"heartbeatIntervalMillis" : NumberLong(2000),  
"optimes" : \{  
"lastCommittedOpTime" : \{  
"ts" : Timestamp(1513518558, 1),  
"t" : NumberLong(1)  
\},  
"readConcernMajorityOpTime" : \{  
"ts" : Timestamp(1513518558, 1),  
"t" : NumberLong(1)  
\},  
"appliedOpTime" : \{  
"ts" : Timestamp(1513518558, 1),  
"t" : NumberLong(1)  
\},  
"durableOpTime" : \{  
"ts" : Timestamp(1513518558, 1),  
"t" : NumberLong(1)  
\}  
\},  
"members" : \[ //复制集所有成员信息  
\{  
"\_id" : 1, //成员编号  
"name" : "db01:40000", //成员所在服务器名称  
"health" : 1, //成员在复制集中是否运行，1表示运行，0表示失败  
"state" : 1, //成员在复制集中的状态，1表示主节点  
"stateStr" : "PRIMARY", //成员在复制集中状态名称  
"uptime" : 873, //成员在线时间，单位是秒  
"optime" : \{ //本实例最近一次更改数据库的时间以及每秒执行的操作数据库的次数。  
"ts" : Timestamp(1513518558, 1),  
"t" : NumberLong(1)  
\},  
"optimeDate" : ISODate("2017-12-17T13:49:18Z"),  
"electionTime" : Timestamp(1513518326, 2),  
"electionDate" : ISODate("2017-12-17T13:45:26Z"),  
"configVersion" : 3,  
"self" : true //成员为当前命令所在的服务器  
\},  
\{  
"\_id" : 2,  
"name" : "db02:40000",  
"health" : 1, //成员在复制集中是否运行，1表示运行，0表示失败  
"state" : 2, //成员在复制集中的状态，2表示second节点  
"stateStr" : "SECONDARY",  
"uptime" : 61,  
"optime" : \{  
"ts" : Timestamp(1513518558, 1),  
"t" : NumberLong(1)  
\},  
"optimeDurable" : \{  
"ts" : Timestamp(1513518558, 1),  
"t" : NumberLong(1)  
\},  
"optimeDate" : ISODate("2017-12-17T13:49:18Z"),  
"optimeDurableDate" : ISODate("2017-12-17T13:49:18Z"),  
"lastHeartbeat" : ISODate("2017-12-17T13:49:23.505Z"), //当前实例到此远端成员最近一次成功发送与接收心跳包的时间。  
"lastHeartbeatRecv" : ISODate("2017-12-17T13:49:23.504Z"),  
"pingMs" : NumberLong(0), //此远端成员到本实例间一个路由包的来回时间  
"syncingTo" : "db01:40000", //此成员从哪个实例同步数据  
"configVersion" : 3  
\},  
\{  
"\_id" : 3,  
"name" : "db03:40000",  
"health" : 1,  
"state" : 7, //成员在复制集中的状态，7表示arbiter节点  
"stateStr" : "ARBITER",  
"uptime" : 38,  
"lastHeartbeat" : ISODate("2017-12-17T13:49:23.505Z"),  
"lastHeartbeatRecv" : ISODate("2017-12-17T13:49:20.562Z"),  
"pingMs" : NumberLong(0),  
"configVersion" : 3  
\}  
\],  
"ok" : 1,  
"operationTime" : Timestamp(1513518558, 1),  
"$clusterTime" : \{  
"clusterTime" : Timestamp(1513518558, 1),  
"signature" : \{  
"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),  
"keyId" : NumberLong(0)  
\}  
\}  
\}

10、测试复制集

1）主节点插入数据：

MongoDB Enterprise rs0:PRIMARY> db  
test  
MongoDB Enterprise rs0:PRIMARY> for(var i =0; i <4; i ++)\{db.user.insert(\{userName:'gxt'+i,age:i\})\}  
WriteResult(\{ "nInserted" : 1 \})  
MongoDB Enterprise rs0:PRIMARY> show collections  
user  
MongoDB Enterprise rs0:PRIMARY> db.user.find()  
\{ "\_id" : ObjectId("5a3677b8041261917b9120a0"), "userName" : "gxt0", "age" : 0 \}  
\{ "\_id" : ObjectId("5a3677b8041261917b9120a1"), "userName" : "gxt1", "age" : 1 \}  
\{ "\_id" : ObjectId("5a3677b8041261917b9120a2"), "userName" : "gxt2", "age" : 2 \}  
\{ "\_id" : ObjectId("5a3677b8041261917b9120a3"), "userName" : "gxt3", "age" : 3 \}

2）从节点检查复制情况：

MongoDB Enterprise rs0:SECONDARY> db  
test  
MongoDB Enterprise rs0:SECONDARY> show collections  
2017-12-17T21:57:37.136+0800 E QUERY \[thread1\] Error: listCollections failed: \{  
"operationTime" : Timestamp(1513519048, 1),  
"ok" : 0,  
"errmsg" : "not master and slaveOk=false", //因为secondary是不允许读写的，如果非要解决，则执行：rs.slaveOk()  
"code" : 13435,  
"codeName" : "NotMasterNoSlaveOk",  
"$clusterTime" : \{  
"clusterTime" : Timestamp(1513519048, 1),  
"signature" : \{  
"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),  
"keyId" : NumberLong(0)  
\}  
\}  
\} :  
\_getErrorWithCode@src/mongo/shell/utils.js:25:13  
DB.prototype.\_getCollectionInfosCommand@src/mongo/shell/db.js:941:1  
DB.prototype.getCollectionInfos@src/mongo/shell/db.js:953:19  
DB.prototype.getCollectionNames@src/mongo/shell/db.js:964:16  
shellHelper.show@src/mongo/shell/utils.js:803:9  
shellHelper@src/mongo/shell/utils.js:700:15  
@(shellhelp2):1:1

MongoDB Enterprise rs0:SECONDARY> rs.slaveOk()  
MongoDB Enterprise rs0:SECONDARY> db  
test  
MongoDB Enterprise rs0:SECONDARY> show collections  
user  
MongoDB Enterprise rs0:SECONDARY> db.user.find()  
\{ "\_id" : ObjectId("5a3677b8041261917b9120a0"), "userName" : "gxt0", "age" : 0 \}  
\{ "\_id" : ObjectId("5a3677b8041261917b9120a1"), "userName" : "gxt1", "age" : 1 \}  
\{ "\_id" : ObjectId("5a3677b8041261917b9120a2"), "userName" : "gxt2", "age" : 2 \}  
\{ "\_id" : ObjectId("5a3677b8041261917b9120a3"), "userName" : "gxt3", "age" : 3 \}

通过db.user.find()查询到和主复制集上一样的数据，表示数据同步成功！

3）arbiter节点检查数据情况：

MongoDB Enterprise rs0:ARBITER> db  
test  
MongoDB Enterprise rs0:ARBITER> show collections  
2017-12-17T22:05:18.090+0800 E QUERY \[thread1\] Error: listCollections failed: \{  
"ok" : 0,  
"errmsg" : "not master and slaveOk=false",  
"code" : 13435,  
"codeName" : "NotMasterNoSlaveOk"  
\} :  
\_getErrorWithCode@src/mongo/shell/utils.js:25:13  
DB.prototype.\_getCollectionInfosCommand@src/mongo/shell/db.js:941:1  
DB.prototype.getCollectionInfos@src/mongo/shell/db.js:953:19  
DB.prototype.getCollectionNames@src/mongo/shell/db.js:964:16  
shellHelper.show@src/mongo/shell/utils.js:803:9  
shellHelper@src/mongo/shell/utils.js:700:15  
@(shellhelp2):1:1  
MongoDB Enterprise rs0:ARBITER> rs.slaveOk()  
MongoDB Enterprise rs0:ARBITER> show collections

arbiter并没有进行数据同步，因为仲裁节点只参与投票，不接收数据！

三、复制集工作原理

复制集通过local库下的oplog.rs集合进行数据同步。

四、测试MongoDB复制集自动故障转移功能

MongoDB复制集搭建完成后，相当于具备了备份的功能，上文中已经展示过，这里不再赘述，下面我们主要测试一下MongoDB的另一大功能，自动故障转移的功能。

1、MongoDB的second节点宕机

通过kill second节点进行测试：

1）查看集群状态，所有节点运行正常：

MongoDB Enterprise rs0:PRIMARY> rs.status()  
\{  
"set" : "rs0",  
"date" : ISODate("2017-12-17T14:43:37.210Z"),  
"myState" : 1,  
"term" : NumberLong(1),  
"heartbeatIntervalMillis" : NumberLong(2000),  
"optimes" : \{  
"lastCommittedOpTime" : \{  
"ts" : Timestamp(1513521808, 1),  
"t" : NumberLong(1)  
\},  
"readConcernMajorityOpTime" : \{  
"ts" : Timestamp(1513521808, 1),  
"t" : NumberLong(1)  
\},  
"appliedOpTime" : \{  
"ts" : Timestamp(1513521808, 1),  
"t" : NumberLong(1)  
\},  
"durableOpTime" : \{  
"ts" : Timestamp(1513521808, 1),  
"t" : NumberLong(1)  
\}  
\},  
"members" : \[  
\{  
"\_id" : 1,  
"name" : "db01:40000",  
"health" : 1,  
"state" : 1,  
"stateStr" : "PRIMARY",  
"uptime" : 4127,  
"optime" : \{  
"ts" : Timestamp(1513521808, 1),  
"t" : NumberLong(1)  
\},  
"optimeDate" : ISODate("2017-12-17T14:43:28Z"),  
"electionTime" : Timestamp(1513518326, 2),  
"electionDate" : ISODate("2017-12-17T13:45:26Z"),  
"configVersion" : 3,  
"self" : true  
\},  
\{  
"\_id" : 2,  
"name" : "db02:40000",  
"health" : 1,  
"state" : 2, //健康  
"stateStr" : "SECONDARY",  
"uptime" : 3314,  
"optime" : \{  
"ts" : Timestamp(1513521808, 1),  
"t" : NumberLong(1)  
\},  
"optimeDurable" : \{  
"ts" : Timestamp(1513521808, 1),  
"t" : NumberLong(1)  
\},  
"optimeDate" : ISODate("2017-12-17T14:43:28Z"),  
"optimeDurableDate" : ISODate("2017-12-17T14:43:28Z"),  
"lastHeartbeat" : ISODate("2017-12-17T14:43:35.254Z"),  
"lastHeartbeatRecv" : ISODate("2017-12-17T14:43:35.240Z"),  
"pingMs" : NumberLong(0),  
"syncingTo" : "db01:40000",  
"configVersion" : 3  
\},  
\{  
"\_id" : 3,  
"name" : "db03:40000",  
"health" : 1,  
"state" : 7,  
"stateStr" : "ARBITER",  
"uptime" : 3291,  
"lastHeartbeat" : ISODate("2017-12-17T14:43:35.254Z"),  
"lastHeartbeatRecv" : ISODate("2017-12-17T14:43:36.323Z"),  
"pingMs" : NumberLong(0),  
"configVersion" : 3  
\}  
\],  
"ok" : 1,  
"operationTime" : Timestamp(1513521808, 1),  
"$clusterTime" : \{  
"clusterTime" : Timestamp(1513521808, 1),  
"signature" : \{  
"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),  
"keyId" : NumberLong(0)  
\}  
\}  
\}

2）kill second节点进程：

\[mongodb@db02 mongodb-3.6.0\]$ ps -ef|grep mongo  
root 2313 2254 0 20:27 pts/0 00:00:00 su - mongodb  
mongodb 2314 2313 0 20:27 pts/0 00:00:00 -bash  
root 2625 2469 0 20:47 pts/1 00:00:00 su - mongodb  
mongodb 2626 2625 0 20:47 pts/1 00:00:00 -bash  
mongodb 3378 1 0 21:34 ? 00:00:24 bin/mongod --config /home/mongodb/db\_rs0/config\_rs0/rs0.conf  
mongodb 4083 2314 0 22:44 pts/0 00:00:00 ps -ef  
mongodb 4084 2314 0 22:44 pts/0 00:00:00 grep --color=auto mongo  
\[mongodb@db02 mongodb-3.6.0\]$ kill -9 3378  
\[mongodb@db02 mongodb-3.6.0\]$ ps -ef|grep mongo  
root 2313 2254 0 20:27 pts/0 00:00:00 su - mongodb  
mongodb 2314 2313 0 20:27 pts/0 00:00:00 -bash  
root 2625 2469 0 20:47 pts/1 00:00:00 su - mongodb  
mongodb 2626 2625 0 20:47 pts/1 00:00:00 -bash  
mongodb 4085 2314 0 22:44 pts/0 00:00:00 ps -ef  
mongodb 4086 2314 0 22:44 pts/0 00:00:00 grep --color=auto mongo

3）再次查看节点状态，发现second节点已经宕机：

MongoDB Enterprise rs0:PRIMARY> rs.status()  
\{  
"set" : "rs0",  
"date" : ISODate("2017-12-17T14:44:33.514Z"),  
"myState" : 1,  
"term" : NumberLong(1),  
"heartbeatIntervalMillis" : NumberLong(2000),  
"optimes" : \{  
"lastCommittedOpTime" : \{  
"ts" : Timestamp(1513521848, 1),  
"t" : NumberLong(1)  
\},  
"readConcernMajorityOpTime" : \{  
"ts" : Timestamp(1513521848, 1),  
"t" : NumberLong(1)  
\},  
"appliedOpTime" : \{  
"ts" : Timestamp(1513521868, 1),  
"t" : NumberLong(1)  
\},  
"durableOpTime" : \{  
"ts" : Timestamp(1513521868, 1),  
"t" : NumberLong(1)  
\}  
\},  
"members" : \[  
\{  
"\_id" : 1,  
"name" : "db01:40000",  
"health" : 1,  
"state" : 1,  
"stateStr" : "PRIMARY",  
"uptime" : 4183,  
"optime" : \{  
"ts" : Timestamp(1513521868, 1),  
"t" : NumberLong(1)  
\},  
"optimeDate" : ISODate("2017-12-17T14:44:28Z"),  
"electionTime" : Timestamp(1513518326, 2),  
"electionDate" : ISODate("2017-12-17T13:45:26Z"),  
"configVersion" : 3,  
"self" : true  
\},  
\{  
"\_id" : 2,  
"name" : "db02:40000",  
"health" : 0,  
"state" : 8, //second节点宕机  
"stateStr" : "(not reachable/healthy)",  
"uptime" : 0,  
"optime" : \{  
"ts" : Timestamp(0, 0),  
"t" : NumberLong(-1)  
\},  
"optimeDurable" : \{  
"ts" : Timestamp(0, 0),  
"t" : NumberLong(-1)  
\},  
"optimeDate" : ISODate("1970-01-01T00:00:00Z"),  
"optimeDurableDate" : ISODate("1970-01-01T00:00:00Z"),  
"lastHeartbeat" : ISODate("2017-12-17T14:44:33.301Z"),  
"lastHeartbeatRecv" : ISODate("2017-12-17T14:44:17.267Z"),  
"pingMs" : NumberLong(0),  
"lastHeartbeatMessage" : "Connection refused",  
"configVersion" : -1  
\},  
\{  
"\_id" : 3,  
"name" : "db03:40000",  
"health" : 1,  
"state" : 7,  
"stateStr" : "ARBITER",  
"uptime" : 3348,  
"lastHeartbeat" : ISODate("2017-12-17T14:44:33.288Z"),  
"lastHeartbeatRecv" : ISODate("2017-12-17T14:44:31.334Z"),  
"pingMs" : NumberLong(0),  
"configVersion" : 3  
\}  
\],  
"ok" : 1,  
"operationTime" : Timestamp(1513521868, 1),  
"$clusterTime" : \{  
"clusterTime" : Timestamp(1513521868, 1),  
"signature" : \{  
"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),  
"keyId" : NumberLong(0)  
\}  
\}  
\}

4）主节点向测试表插入一条记录，一切操作正常：

MongoDB Enterprise rs0:PRIMARY> db.scores.insert(\{stuid:1,subobject:"math",score:99\})  
WriteResult(\{ "nInserted" : 1 \})  
MongoDB Enterprise rs0:PRIMARY> db.scores.find()  
\{ "\_id" : ObjectId("5a36810c041261917b9120a5"), "stuid" : 1, "subobject" : "math", "score" : 99 \}  
\{ "\_id" : ObjectId("5a3682fe041261917b9120a7"), "stuid" : 1, "subobject" : "math", "score" : 99 \}

5）再次启动从节点，检查步骤4primary节点操作集合，发现数据正常同步到从节点：

MongoDB Enterprise rs0:SECONDARY> rs.slaveOk()  
MongoDB Enterprise rs0:SECONDARY> show collections  
scores  
user  
MongoDB Enterprise rs0:SECONDARY> db.scores.find()  
\{ "\_id" : ObjectId("5a36810c041261917b9120a5"), "stuid" : 1, "subobject" : "math", "score" : 99 \}  
\{ "\_id" : ObjectId("5a3682fe041261917b9120a7"), "stuid" : 1, "subobject" : "math", "score" : 99 \}

2、MongoDB的primary节点宕机

1）检查复制集运行状态

MongoDB Enterprise rs0:PRIMARY> rs.status()  
\{  
"set" : "rs0",  
"date" : ISODate("2017-12-17T14:56:52.953Z"),  
"myState" : 1,  
"term" : NumberLong(1),  
"heartbeatIntervalMillis" : NumberLong(2000),  
"optimes" : \{  
"lastCommittedOpTime" : \{  
"ts" : Timestamp(1513522608, 1),  
"t" : NumberLong(1)  
\},  
"readConcernMajorityOpTime" : \{  
"ts" : Timestamp(1513522608, 1),  
"t" : NumberLong(1)  
\},  
"appliedOpTime" : \{  
"ts" : Timestamp(1513522608, 1),  
"t" : NumberLong(1)  
\},  
"durableOpTime" : \{  
"ts" : Timestamp(1513522608, 1),  
"t" : NumberLong(1)  
\}  
\},  
"members" : \[  
\{  
"\_id" : 1,  
"name" : "db01:40000",  
"health" : 1,  
"state" : 1,  
"stateStr" : "PRIMARY",  
"uptime" : 4922,  
"optime" : \{  
"ts" : Timestamp(1513522608, 1),  
"t" : NumberLong(1)  
\},  
"optimeDate" : ISODate("2017-12-17T14:56:48Z"),  
"electionTime" : Timestamp(1513518326, 2),  
"electionDate" : ISODate("2017-12-17T13:45:26Z"),  
"configVersion" : 3,  
"self" : true  
\},  
\{  
"\_id" : 2,  
"name" : "db02:40000",  
"health" : 1,  
"state" : 2,  
"stateStr" : "SECONDARY",  
"uptime" : 649,  
"optime" : \{  
"ts" : Timestamp(1513522608, 1),  
"t" : NumberLong(1)  
\},  
"optimeDurable" : \{  
"ts" : Timestamp(1513522608, 1),  
"t" : NumberLong(1)  
\},  
"optimeDate" : ISODate("2017-12-17T14:56:48Z"),  
"optimeDurableDate" : ISODate("2017-12-17T14:56:48Z"),  
"lastHeartbeat" : ISODate("2017-12-17T14:56:51.820Z"),  
"lastHeartbeatRecv" : ISODate("2017-12-17T14:56:51.844Z"),  
"pingMs" : NumberLong(0),  
"syncingTo" : "db01:40000",  
"configVersion" : 3  
\},  
\{  
"\_id" : 3,  
"name" : "db03:40000",  
"health" : 1,  
"state" : 7,  
"stateStr" : "ARBITER",  
"uptime" : 4087,  
"lastHeartbeat" : ISODate("2017-12-17T14:56:51.743Z"),  
"lastHeartbeatRecv" : ISODate("2017-12-17T14:56:51.518Z"),  
"pingMs" : NumberLong(0),  
"configVersion" : 3  
\}  
\],  
"ok" : 1,  
"operationTime" : Timestamp(1513522608, 1),  
"$clusterTime" : \{  
"clusterTime" : Timestamp(1513522608, 1),  
"signature" : \{  
"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),  
"keyId" : NumberLong(0)  
\}  
\}  
\}

2）kill掉主节点

\[mongodb@db01 mongodb-3.6.0\]$ ps -ef|grep mongo  
root 3082 3056 0 20:26 pts/0 00:00:00 su - mongodb  
mongodb 3083 3082 0 20:26 pts/0 00:00:00 -bash  
root 3266 3139 0 20:46 pts/1 00:00:00 su - mongodb  
mongodb 3267 3266 0 20:46 pts/1 00:00:00 -bash  
root 3316 3299 0 20:50 pts/2 00:00:00 su - mongodb  
mongodb 3317 3316 0 20:50 pts/2 00:00:00 -bash  
mongodb 3771 1 0 21:34 ? 00:00:30 bin/mongod --config /home/mongodb/db\_rs0/config\_rs0/rs0.conf  
mongodb 3935 3083 0 22:57 pts/0 00:00:00 ps -ef  
mongodb 3936 3083 0 22:57 pts/0 00:00:00 grep --color=auto mongo  
\[mongodb@db01 mongodb-3.6.0\]$ kill -9 3771  
\[mongodb@db01 mongodb-3.6.0\]$ ps -ef|grep mongo  
root 3082 3056 0 20:26 pts/0 00:00:00 su - mongodb  
mongodb 3083 3082 0 20:26 pts/0 00:00:00 -bash  
root 3266 3139 0 20:46 pts/1 00:00:00 su - mongodb  
mongodb 3267 3266 0 20:46 pts/1 00:00:00 -bash  
root 3316 3299 0 20:50 pts/2 00:00:00 su - mongodb  
mongodb 3317 3316 0 20:50 pts/2 00:00:00 -bash  
mongodb 3937 3083 0 22:57 pts/0 00:00:00 ps -ef  
mongodb 3938 3083 0 22:57 pts/0 00:00:00 grep --color=auto mongo

3）再次检查集群运行状态

MongoDB Enterprise rs0:SECONDARY>  
MongoDB Enterprise rs0:PRIMARY> rs.status() //从提示我们可以看到secondary节点已经变成了主节点  
\{  
"set" : "rs0",  
"date" : ISODate("2017-12-17T14:58:24.922Z"),  
"myState" : 1,  
"term" : NumberLong(2),  
"heartbeatIntervalMillis" : NumberLong(2000),  
"optimes" : \{  
"lastCommittedOpTime" : \{  
"ts" : Timestamp(1513522668, 1),  
"t" : NumberLong(1)  
\},  
"readConcernMajorityOpTime" : \{  
"ts" : Timestamp(1513522668, 1),  
"t" : NumberLong(1)  
\},  
"appliedOpTime" : \{  
"ts" : Timestamp(1513522703, 1),  
"t" : NumberLong(2)  
\},  
"durableOpTime" : \{  
"ts" : Timestamp(1513522703, 1),  
"t" : NumberLong(2)  
\}  
\},  
"members" : \[  
\{  
"\_id" : 1,  
"name" : "db01:40000",  
"health" : 0,  
"state" : 8, //主节点宕机  
"stateStr" : "(not reachable/healthy)",  
"uptime" : 0,  
"optime" : \{  
"ts" : Timestamp(0, 0),  
"t" : NumberLong(-1)  
\},  
"optimeDurable" : \{  
"ts" : Timestamp(0, 0),  
"t" : NumberLong(-1)  
\},  
"optimeDate" : ISODate("1970-01-01T00:00:00Z"),  
"optimeDurableDate" : ISODate("1970-01-01T00:00:00Z"),  
"lastHeartbeat" : ISODate("2017-12-17T14:58:24.620Z"),  
"lastHeartbeatRecv" : ISODate("2017-12-17T14:57:51.856Z"),  
"pingMs" : NumberLong(0),  
"lastHeartbeatMessage" : "Connection refused",  
"configVersion" : -1  
\},  
\{  
"\_id" : 2,  
"name" : "db02:40000",  
"health" : 1,  
"state" : 1,  
"stateStr" : "PRIMARY", //原secondary节点已经变为主节点  
"uptime" : 744,  
"optime" : \{  
"ts" : Timestamp(1513522703, 1),  
"t" : NumberLong(2)  
\},  
"optimeDate" : ISODate("2017-12-17T14:58:23Z"),  
"infoMessage" : "could not find member to sync from",  
"electionTime" : Timestamp(1513522682, 1),  
"electionDate" : ISODate("2017-12-17T14:58:02Z"),  
"configVersion" : 3,  
"self" : true  
\},  
\{  
"\_id" : 3,  
"name" : "db03:40000",  
"health" : 1,  
"state" : 7,  
"stateStr" : "ARBITER",  
"uptime" : 743,  
"lastHeartbeat" : ISODate("2017-12-17T14:58:24.601Z"),  
"lastHeartbeatRecv" : ISODate("2017-12-17T14:58:21.549Z"),  
"pingMs" : NumberLong(0),  
"configVersion" : 3  
\}  
\],  
"ok" : 1,  
"operationTime" : Timestamp(1513522703, 1),  
"$clusterTime" : \{  
"clusterTime" : Timestamp(1513522703, 1),  
"signature" : \{  
"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),  
"keyId" : NumberLong(0)  
\}  
\}  
\}

4）新主节点插入测试数据，一切操作正常

MongoDB Enterprise rs0:PRIMARY> db  
test  
MongoDB Enterprise rs0:PRIMARY> show collections  
scores  
user  
MongoDB Enterprise rs0:PRIMARY> db.scores.insert(\{stuid:1,subobject:"math",score:99\})  
WriteResult(\{ "nInserted" : 1 \})  
MongoDB Enterprise rs0:PRIMARY> db.scores.find()  
\{ "\_id" : ObjectId("5a36810c041261917b9120a5"), "stuid" : 1, "subobject" : "math", "score" : 99 \}  
\{ "\_id" : ObjectId("5a3682fe041261917b9120a7"), "stuid" : 1, "subobject" : "math", "score" : 99 \}  
\{ "\_id" : ObjectId("5a3686bb1e50f07a6a2d207d"), "stuid" : 1, "subobject" : "math", "score" : 99 \}

5）重新启动原主节点，查看集群状态

MongoDB Enterprise rs0:PRIMARY> rs.status()  
\{  
"set" : "rs0",  
"date" : ISODate("2017-12-17T15:02:28.114Z"),  
"myState" : 1,  
"term" : NumberLong(2),  
"heartbeatIntervalMillis" : NumberLong(2000),  
"optimes" : \{  
"lastCommittedOpTime" : \{  
"ts" : Timestamp(1513522943, 1),  
"t" : NumberLong(2)  
\},  
"readConcernMajorityOpTime" : \{  
"ts" : Timestamp(1513522943, 1),  
"t" : NumberLong(2)  
\},  
"appliedOpTime" : \{  
"ts" : Timestamp(1513522943, 1),  
"t" : NumberLong(2)  
\},  
"durableOpTime" : \{  
"ts" : Timestamp(1513522943, 1),  
"t" : NumberLong(2)  
\}  
\},  
"members" : \[  
\{  
"\_id" : 1,  
"name" : "db01:40000",  
"health" : 1,  
"state" : 2,  
"stateStr" : "SECONDARY", //原主节点已经变为从节点  
"uptime" : 19,  
"optime" : \{  
"ts" : Timestamp(1513522943, 1),  
"t" : NumberLong(2)  
\},  
"optimeDurable" : \{  
"ts" : Timestamp(1513522943, 1),  
"t" : NumberLong(2)  
\},  
"optimeDate" : ISODate("2017-12-17T15:02:23Z"),  
"optimeDurableDate" : ISODate("2017-12-17T15:02:23Z"),  
"lastHeartbeat" : ISODate("2017-12-17T15:02:26.998Z"),  
"lastHeartbeatRecv" : ISODate("2017-12-17T15:02:27.971Z"),  
"pingMs" : NumberLong(0),  
"syncingTo" : "db02:40000",  
"configVersion" : 3  
\},  
\{  
"\_id" : 2,  
"name" : "db02:40000",  
"health" : 1,  
"state" : 1,  
"stateStr" : "PRIMARY",  
"uptime" : 988,  
"optime" : \{  
"ts" : Timestamp(1513522943, 1),  
"t" : NumberLong(2)  
\},  
"optimeDate" : ISODate("2017-12-17T15:02:23Z"),  
"electionTime" : Timestamp(1513522682, 1),  
"electionDate" : ISODate("2017-12-17T14:58:02Z"),  
"configVersion" : 3,  
"self" : true  
\},  
\{  
"\_id" : 3,  
"name" : "db03:40000",  
"health" : 1,  
"state" : 7,  
"stateStr" : "ARBITER",  
"uptime" : 986,  
"lastHeartbeat" : ISODate("2017-12-17T15:02:26.754Z"),  
"lastHeartbeatRecv" : ISODate("2017-12-17T15:02:26.603Z"),  
"pingMs" : NumberLong(0),  
"configVersion" : 3  
\}  
\],  
"ok" : 1,  
"operationTime" : Timestamp(1513522943, 1),  
"$clusterTime" : \{  
"clusterTime" : Timestamp(1513522943, 1),  
"signature" : \{  
"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),  
"keyId" : NumberLong(0)  
\}  
\}  
\}

6）查看步骤4插入测试数据信息

MongoDB Enterprise rs0:SECONDARY> db  
test  
MongoDB Enterprise rs0:SECONDARY> show collections  
2017-12-17T23:03:33.650+0800 E QUERY \[thread1\] Error: listCollections failed: \{  
"operationTime" : Timestamp(1513523003, 1),  
"ok" : 0,  
"errmsg" : "not master and slaveOk=false",  
"code" : 13435,  
"codeName" : "NotMasterNoSlaveOk",  
"$clusterTime" : \{  
"clusterTime" : Timestamp(1513523003, 1),  
"signature" : \{  
"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),  
"keyId" : NumberLong(0)  
\}  
\}  
\} :  
\_getErrorWithCode@src/mongo/shell/utils.js:25:13  
DB.prototype.\_getCollectionInfosCommand@src/mongo/shell/db.js:941:1  
DB.prototype.getCollectionInfos@src/mongo/shell/db.js:953:19  
DB.prototype.getCollectionNames@src/mongo/shell/db.js:964:16  
shellHelper.show@src/mongo/shell/utils.js:803:9  
shellHelper@src/mongo/shell/utils.js:700:15  
@(shellhelp2):1:1  
MongoDB Enterprise rs0:SECONDARY> rs.slaveOk()  
MongoDB Enterprise rs0:SECONDARY> show collections  
scores  
user  
MongoDB Enterprise rs0:SECONDARY> db.scores.find()  
\{ "\_id" : ObjectId("5a36810c041261917b9120a5"), "stuid" : 1, "subobject" : "math", "score" : 99 \}  
\{ "\_id" : ObjectId("5a3682fe041261917b9120a7"), "stuid" : 1, "subobject" : "math", "score" : 99 \}  
\{ "\_id" : ObjectId("5a3686bb1e50f07a6a2d207d"), "stuid" : 1, "subobject" : "math", "score" : 99 \}

集群运转正常，数据已经正常同步过来。