MongoDB调整oplog大小导致从库宕机

Posted on 本文总阅读量次

现象：

由于调大oplog的值后，导致从库宕机，日志报如下信息：

1
2
3
4
5
6
7
8
2019-08-10T08:32:40 I REPL [replication-2] could not find member to sync from
2019-08-10T08:32:40 E REPL [rsBackgroundSync] too stale to catch up -- entering maintenance mode
2019-08-10T08:32:40 I REPL [rsBackgroundSync] Our newest OpTime: {ts: Timestamp 1503977172000|27, t: 1}
2019-08-10T08:32:40 I REPL [rsBackgroundSync] Earliest OpTime available is {ts: Timestamp 1503998451000|1, t: -1}
2019-08-10T08:32:40 I REPL [rsBackgroundSync] See http://dochub.mongodb.org/core/resyncingaverystalereplicasetmember
2019-08-10T08:32:40 I REPL [rsBackgroundSync] going into maintenance mode with 1820 other maintenance mode tasks in progress
2019-08-10T08:32:40 I REPL [rsBackgroundSync] sync source candidate: 172.16.0.21:28000
2019-08-10T08:32:40 I REPL [replication-2] We are too stale to use 172.16.0.21:28000 as a sync source. Blacklisting this sync source because our last fetched timestamp: 59a4ded4:1b is before their earliest timestamp:59a524a9:21e for 1min until: 2019-08-10T08:34:19

原因：

由于存在延迟从库的问题，延迟从库由于延迟，还没有应用到oplog就被“drop”了，进而导致延迟从库处于recovering状态。

解决方案

  1. 先取消延迟配置，扩容延时从库的oplog大小，再扩容主库的oplog大小。
  2. 对主库先降级再进行升级操作。
oplog扩容流程
  1. 将需扩容oplog大小的库端口换成一个系统上未使用的新端口，如：28700。
  2. 关库
  3. 修改配置文件注释认证相关，修改oplog大小，如：将oplog扩容到50GB。
    1
    2
    vim /usr/local/mongodb/conf/mongodb.conf
    oplogSizeMB: 51200
  4. 重新启动库
    1
    /usr/local/mongodb/bin/mongod -f /usr/local/mongodb/conf/mongodb.conf
  5. 登录进库
    1
    mongo 127.0.0.1:28700/admin -u admin -p password --authenticationDatabase admin
  6. 执行oplog recreate操作
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    >use local
    >db=db.getSiblingDB('local')
    >db.temp.drop()
    >db.temp.save(db.oplog.rs.find({}, {ts: 1, h: 1}).sort({$natural: -1}).limit(1).next())
    >db.temp.find()
    >db=db.getSiblingDB('local')
    >db.oplog.rs.drop()
    >db.runCommand({create: "oplog.rs", capped: true, size: (80 *1024 * 1024 * 1024)})
    >db.oplog.rs.save(db.temp.findOne())
    >db.oplog.rs.find()
  7. 关库，修改mongodb.conf为原始值，启动。
    1
    mongo 127.0.0.1:28700/admin -u admin -p password --authenticationDatabase admin