Monday, September 17, 2018

Manipulating Member State

Manipulating Member State


Turning Primaries into Secondaries


rs.stepDown(300);

MongoDB Enterprise OurReplica:PRIMARY> db.printReplicationInfo()
configured oplog size:   1436.840576171875MB
log length start to end: 51327secs (14.26hrs)
oplog first event time:  Fri Sep 14 2018 12:52:15 GMT+0530 (IST)
oplog last event time:   Sat Sep 15 2018 03:07:42 GMT+0530 (IST)
now:                     Sat Sep 15 2018 03:07:50 GMT+0530 (IST)
MongoDB Enterprise OurReplica:PRIMARY>



MongoDB Enterprise OurReplica:PRIMARY> rs.stepDown(300)
2018-09-15T02:36:43.720+0530 E QUERY    [js] Error: error doing query: failed: network                                                                                                        error while attempting to run command 'replSetStepDown' on host '127.0.0.1:6666'  :
DB.prototype.runCommand@src/mongo/shell/db.js:168:1
DB.prototype.adminCommand@src/mongo/shell/db.js:186:16
rs.stepDown@src/mongo/shell/utils.js:1398:12
@(shell):1:1
2018-09-15T02:36:43.722+0530 I NETWORK  [js] trying reconnect to 127.0.0.1:6666 failed
2018-09-15T02:36:43.722+0530 I NETWORK  [js] reconnect 127.0.0.1:6666 ok
MongoDB Enterprise OurReplica:SECONDARY>

Server 2 failover to Primary state now:

Preventing Election

If you need to do some maintenance on the Primary but don't want to any of the other eligible members to become Primary in the interim, you can force them to stay secondaries by running freeze on each of them.

port: 7777

MongoDB Enterprise OurReplica:SECONDARY> rs.freeze(1200)
{
        "ok" : 1,
        "operationTime" : Timestamp(1536959629, 1),
        "$clusterTime" : {
                "clusterTime" : Timestamp(1536959629, 1),
                "signature" : {
                        "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
                        "keyId" : NumberLong(0)
                }
        }
}
MongoDB Enterprise OurReplica:SECONDARY>

port : 8888

MongoDB Enterprise OurReplica:SECONDARY> rs.freeze(1200)
{
        "ok" : 1,
        "operationTime" : Timestamp(1536959649, 1),
        "$clusterTime" : {
                "clusterTime" : Timestamp(1536959649, 1),
                "signature" : {
                        "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
                        "keyId" : NumberLong(0)
                }
        }
}
MongoDB Enterprise OurReplica:SECONDARY>

port: 6666(Primary)

now: I tried to stepDown Primary server 6666. Let see what will happen now.

MongoDB Enterprise OurReplica:PRIMARY> rs.stepDown(300)
2018-09-15T02:44:58.439+0530 E QUERY    [js] Error: error doing query: failed: network error while attempting to run command 'replSetStepDown' on host '127.0.0.1:6666'  :
DB.prototype.runCommand@src/mongo/shell/db.js:168:1
DB.prototype.adminCommand@src/mongo/shell/db.js:186:16
rs.stepDown@src/mongo/shell/utils.js:1398:12
@(shell):1:1
2018-09-15T02:44:58.440+0530 I NETWORK  [js] trying reconnect to 127.0.0.1:6666 failed
2018-09-15T02:44:58.441+0530 I NETWORK  [js] reconnect 127.0.0.1:6666 ok
MongoDB Enterprise OurReplica:SECONDARY>
MongoDB Enterprise OurReplica:SECONDARY>
MongoDB Enterprise OurReplica:SECONDARY>
MongoDB Enterprise OurReplica:SECONDARY>

Verify the status of the replication, All nodes are secondaries and there is no primary in the replica set for 5 minutes ((we gave command rs.stepDown(300)). If you verify after 5 minutes in current secondary server (6666) it will be back to primary state automatically after our maintenance activity.


MongoDB Enterprise OurReplica:SECONDARY> rs.status();
{
        "set" : "OurReplica",
        "date" : ISODate("2018-09-14T21:16:32.594Z"),
        "myState" : 2,
        "term" : NumberLong(12),
        "syncingTo" : "",
        "syncSourceHost" : "",
        "syncSourceId" : -1,
        "heartbeatIntervalMillis" : NumberLong(2000),
        "optimes" : {
                "lastCommittedOpTime" : {
                        "ts" : Timestamp(1536959689, 1),
                        "t" : NumberLong(12)
                },
                "readConcernMajorityOpTime" : {
                        "ts" : Timestamp(1536959689, 1),
                        "t" : NumberLong(12)
                },
                "appliedOpTime" : {
                        "ts" : Timestamp(1536959689, 1),
                        "t" : NumberLong(12)
                },
                "durableOpTime" : {
                        "ts" : Timestamp(1536959689, 1),
                        "t" : NumberLong(12)
                }
        },
        "lastStableCheckpointTimestamp" : Timestamp(1536959689, 1),
        "members" : [
                {
                        "_id" : 0,
                        "name" : "127.0.0.1:6666",
                        "health" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 4100,
                        "optime" : {
                                "ts" : Timestamp(1536959689, 1),
                                "t" : NumberLong(12)
                        },
                        "optimeDate" : ISODate("2018-09-14T21:14:49Z"),
                        "syncingTo" : "",
                        "syncSourceHost" : "",
                        "syncSourceId" : -1,
                        "infoMessage" : "could not find member to sync from",
                        "configVersion" : 22,
                        "self" : true,
                        "lastHeartbeatMessage" : ""
                },
                {
                        "_id" : 1,
                        "name" : "127.0.0.1:7777",
                        "health" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 4098,
                        "optime" : {
                                "ts" : Timestamp(1536959689, 1),
                                "t" : NumberLong(12)
                        },
                        "optimeDurable" : {
                                "ts" : Timestamp(1536959689, 1),
                                "t" : NumberLong(12)
                        },
                        "optimeDate" : ISODate("2018-09-14T21:14:49Z"),
                        "optimeDurableDate" : ISODate("2018-09-14T21:14:49Z"),
                        "lastHeartbeat" : ISODate("2018-09-14T21:16:32.485Z"),
                        "lastHeartbeatRecv" : ISODate("2018-09-14T21:16:32.469Z"),
                        "pingMs" : NumberLong(0),
                        "lastHeartbeatMessage" : "",
                        "syncingTo" : "",
                        "syncSourceHost" : "",
                        "syncSourceId" : -1,
                        "infoMessage" : "",
                        "configVersion" : 22
                },
                {
                        "_id" : 2,
                        "name" : "127.0.0.1:8888",
                        "health" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 3760,
                        "optime" : {
                                "ts" : Timestamp(1536959689, 1),
                                "t" : NumberLong(12)
                        },
                        "optimeDurable" : {
                                "ts" : Timestamp(1536959689, 1),
                                "t" : NumberLong(12)
                        },
                        "optimeDate" : ISODate("2018-09-14T21:14:49Z"),
                        "optimeDurableDate" : ISODate("2018-09-14T21:14:49Z"),
                        "lastHeartbeat" : ISODate("2018-09-14T21:16:32.485Z"),
                        "lastHeartbeatRecv" : ISODate("2018-09-14T21:16:32.486Z"),
                        "pingMs" : NumberLong(0),
                        "lastHeartbeatMessage" : "",
                        "syncingTo" : "",
                        "syncSourceHost" : "",
                        "syncSourceId" : -1,
                        "infoMessage" : "",
                        "configVersion" : 22
                },
                {
                        "_id" : 3,
                        "name" : "127.0.0.1:9999",
                        "health" : 1,
                        "state" : 7,
                        "stateStr" : "ARBITER",
                        "uptime" : 4098,
                        "lastHeartbeat" : ISODate("2018-09-14T21:16:32.485Z"),
                        "lastHeartbeatRecv" : ISODate("2018-09-14T21:16:32.485Z"),
                        "pingMs" : NumberLong(0),
                        "lastHeartbeatMessage" : "",
                        "syncingTo" : "",
                        "syncSourceHost" : "",
                        "syncSourceId" : -1,
                        "infoMessage" : "",
                        "configVersion" : 22
                }
        ],
        "ok" : 1,
        "operationTime" : Timestamp(1536959689, 1),
        "$clusterTime" : {
                "clusterTime" : Timestamp(1536959689, 1),
                "signature" : {
                        "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
                        "keyId" : NumberLong(0)
                }
        }
}
MongoDB Enterprise OurReplica:SECONDARY>

I reconnected after 5 minutes in primary. It is back to earlier state. i.e. PRIMARY

MongoDB Enterprise OurReplica:SECONDARY> exit
bye
[root@nosqlDB mongodb]# mongo --port 6666
MongoDB shell version v4.0.0
connecting to: mongodb://127.0.0.1:6666/
MongoDB server version: 4.0.0
Server has startup warnings:
2018-09-15T01:38:12.762+0530 I STORAGE  [initandlisten]
2018-09-15T01:38:12.762+0530 I STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2018-09-15T01:38:12.762+0530 I STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
2018-09-15T01:38:13.868+0530 I CONTROL  [initandlisten]
2018-09-15T01:38:13.868+0530 I CONTROL  [initandlisten] ** WARNING: Access control is not enabled for the database.
2018-09-15T01:38:13.868+0530 I CONTROL  [initandlisten] **          Read and write access to data and configuration is unrestricted.
2018-09-15T01:38:13.868+0530 I CONTROL  [initandlisten] ** WARNING: You are running this process as the root user, which is not recommended.
2018-09-15T01:38:13.868+0530 I CONTROL  [initandlisten]
2018-09-15T01:38:13.869+0530 I CONTROL  [initandlisten]
2018-09-15T01:38:13.869+0530 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
2018-09-15T01:38:13.869+0530 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2018-09-15T01:38:13.869+0530 I CONTROL  [initandlisten]
2018-09-15T01:38:13.869+0530 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2018-09-15T01:38:13.869+0530 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2018-09-15T01:38:13.869+0530 I CONTROL  [initandlisten]
MongoDB Enterprise OurReplica:PRIMARY


MongoDB Enterprise OurReplica:SECONDARY> rs.freeze(0);
{
        "info" : "unfreezing",
        "ok" : 1,
        "operationTime" : Timestamp(1536960082, 1),
        "$clusterTime" : {
                "clusterTime" : Timestamp(1536960082, 1),
                "signature" : {
                        "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
                        "keyId" : NumberLong(0)
                }
        }
}

MongoDB Enterprise OurReplica:SECONDARY>


MongoDB Enterprise OurReplica:SECONDARY> rs.freeze(0)
{
        "info" : "unfreezing",
        "ok" : 1,
        "operationTime" : Timestamp(1536960092, 1),
        "$clusterTime" : {
                "clusterTime" : Timestamp(1536960092, 1),
                "signature" : {
                        "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
                        "keyId" : NumberLong(0)
                }
        }
}

MongoDB Enterprise OurReplica:SECONDARY>





Step by Step YugabyteDB 2.11 (Open Source) Distributed DB - Multi-node Cluster Setup on RHEL

Scope - ·        Infrastructure planning and requirement for installation of the multi-node cluster database ·        Prerequisites Software...