Databases, distributed SQL, and the road to AI data architecture: YugabyteDB 2.11 Multi-Node Cluster Setup on RHEL: A Step-by-Step DBA Guide

YugabyteDB 2.11 Multi-Node Cluster Setup on RHEL: A Step-by-Step DBA Guide

==========================================================================

YugabyteDB is a distributed SQL database that keeps PostgreSQL wire

compatibility while adding automatic sharding, synchronous replication and

transparent failover. This guide builds a three-node YugabyteDB 2.11 cluster

on RHEL 7.9 from scratch and explains what every command actually does.

Tested on RHEL 7.9 with YugabyteDB 2.11.0.1. All hostnames and IP addresses

are lab values - substitute your own.

SCOPE

-----

- Infrastructure planning and sizing for a multi-node cluster

- OS prerequisites: kernel limits, huge pages, network ports

- Installing and starting yb-master and yb-tserver on all nodes

- Verifying leader election and overall cluster health

ARCHITECTURE OVERVIEW

YugabyteDB 2.11 Multi-Node Cluster Setup on RHEL: A Step-by-Step DBA Guide

==========================================================================

YugabyteDB is a distributed SQL database that keeps PostgreSQL wire

compatibility while adding automatic sharding, synchronous replication and

transparent failover. This guide builds a three-node YugabyteDB 2.11 cluster

on RHEL 7.9 from scratch and explains what every command actually does.

Tested on RHEL 7.9 with YugabyteDB 2.11.0.1. All hostnames and IP addresses

are lab values - substitute your own.

SCOPE

-----

- Infrastructure planning and sizing for a multi-node cluster

- OS prerequisites: kernel limits, huge pages, network ports

- Installing and starting yb-master and yb-tserver on all nodes

- Verifying leader election and overall cluster health

ARCHITECTURE OVERVIEW

---------------------

Every node runs two processes. The masters form a Raft group that stores

cluster metadata and elects one leader. The tablet servers hold the actual

data and serve client connections.

Client applications (psql / ysqlsh / JDBC / cqlsh)

+---------------------+---------------------+

| | |

+-----------+ +-----------+ +-----------+

| 10.0.0.11 | | 10.0.0.12 | | 10.0.0.13 |

+-----------+ +-----------+ +-----------+

| yb-master |<------->| yb-master |<------->| yb-master |

| :7100 | | :7100 | | :7100 |

+-----------+ +-----------+ +-----------+

| yb-tserver|<------->| yb-tserver|<------->| yb-tserver|

| :9100 RPC | | :9100 RPC | | :9100 RPC |

+-----------+ +-----------+ +-----------+

zone1 zone2 zone3

With three masters and replication factor 3, the cluster survives the loss

of any single node without data loss or manual intervention.

A. NODE LAYOUT

--------------

Role Hostname IP address

------ --------- -----------

Node 1 ybnode1 10.0.0.11

Node 2 ybnode2 10.0.0.12

Node 3 ybnode3 10.0.0.13

All three nodes run RHEL 7.9 and host both a master and a tablet server

process. This is normal for a small cluster; at larger scale the masters are

usually separated onto dedicated nodes.

B. NETWORK PORTS

----------------

Port Component Purpose

----- ------------- ------------------------------------------

7100 yb-master Master RPC - inter-node consensus traffic

9100 yb-tserver Tablet server RPC - inter-node data traffic

7000 yb-master Master admin web UI

9000 yb-tserver Tablet server admin web UI

5433 YSQL PostgreSQL-compatible SQL API

9042 YCQL Cassandra-compatible API

6379 YEDIS Redis-compatible API

13000 YSQL YSQL metrics endpoint

12000 YCQL YCQL metrics endpoint

11000 YEDIS YEDIS metrics endpoint

9300 node_exporter Prometheus metrics (optional)

22 sshd Administrative access

Also required:

- An NTP or chrony client on every node. Clock skew across nodes breaks

Raft consensus and can cause spurious leader elections.

- psycopg2, if you plan to drive YSQL from Python.

C. KERNEL RESOURCE LIMITS

-------------------------

As root, add the following to /etc/security/limits.conf on every node:

* - core unlimited

* - data unlimited

* - fsize unlimited

* - sigpending 119934

* - memlock 64

* - rss unlimited

* - nofile 1048576

* - msgqueue 819200

* - stack 8192

* - cpu unlimited

* - nproc 12000

* - locks unlimited

Why this matters: nofile is the critical one. YugabyteDB opens a file

descriptor per tablet per data directory, plus one per client connection.

The RHEL default of 1024 is exhausted almost immediately and the tablet

server dies with "Too many open files". nproc covers the thread count, which

scales with tablet count.

Log out and back in for the limits to take effect, then verify as the

yugabytedb user:

ulimit -a

The values only apply to new login sessions - an existing shell keeps the

old limits, which is a common reason this step appears not to have worked.

D. TRANSPARENT HUGE PAGES

-------------------------

YugabyteDB expects transparent huge pages to be enabled. Check the current

setting:

cat /sys/kernel/mm/transparent_hugepage/enabled

Expected output - the value in square brackets is the active one:

[always] madvise never

If the active value is madvise or never, it is usually being forced by the

kernel command line. Inspect the GRUB configuration:

grep -i "GRUB_CMDLINE_LINUX" /etc/default/grub

If you see transparent_hugepage=never in the output, remove that parameter,

rebuild the GRUB configuration and reboot:

grub2-mkconfig -o /boot/grub2/grub.cfg

reboot

Why this matters: huge pages reduce TLB misses for the large contiguous

memory regions YugabyteDB allocates for its block cache. With THP disabled

you get measurably higher CPU usage under load.

E. USERS, GROUPS AND DATA DIRECTORIES

-------------------------------------

Step 1 - Create the OS account on all three nodes

groupadd yugabytedb

useradd -d /home/yugabytedb -g yugabytedb yugabytedb

# verify the account and group were created

grep yugabytedb /etc/passwd /etc/group

What this does: creates a dedicated unprivileged service account. Never run

the database processes as root - a compromised process would then own the

host. The -d flag sets the home directory; -g assigns the primary group.

If you plan to run Prometheus for monitoring, create its account separately

at that point rather than here - it is not a YugabyteDB dependency.

Step 2 - Create two data directories

mkdir -p /yugabyte01/YUGABYTE

chown -R yugabytedb:yugabytedb /yugabyte01/YUGABYTE

su - yugabytedb

mkdir -p /yugabyte01/YUGABYTE/data1 /yugabyte01/YUGABYTE/data2

What this does: YugabyteDB spreads tablets across every directory listed in

--fs_data_dirs. Two directories backed by separate physical devices roughly

doubles available I/O throughput, because the tablet server writes to them

in parallel. On a single disk there is no benefit - use one directory.

Note the ownership command uses yugabytedb:yugabytedb. A typo in the group

name here causes a permission error at startup that is easy to misdiagnose.

Step 3 - Open the firewall ports on all three nodes

for p in 7000 7100 9000 9100 5433 9042 6379 11000 12000 13000 9300; do

firewall-cmd --zone=public --add-port=${p}/tcp --permanent

done

firewall-cmd --reload

firewall-cmd --list-ports

What this does: --permanent writes the rule to disk but does not apply it to

the running firewall. --reload activates the saved rules. Omitting the

reload is the single most common cause of a cluster where the masters never

find each other - the rules exist but are not live. The final command

confirms what is actually open.

F. INSTALL THE SOFTWARE

-----------------------

Download the tarball on each node, extract it, and run the post-install

script:

cd /opt/yugabyte

wget https://downloads.yugabyte.com/releases/2.11.0.1/yugabyte-2.11.0.1-b1-linux-x86_64.tar.gz

tar -zxvf yugabyte-2.11.0.1-b1-linux-x86_64.tar.gz

cd yugabyte-2.11.0.1

./bin/post_install.sh

What post_install.sh does: it patches the RPATH of the bundled binaries so

they resolve YugabyteDB's own shipped libraries instead of the system ones.

Skipping this step produces shared-library errors the moment you start

yb-master, and the error message does not point back to this cause.

If you host packages on an internal repository, substitute your own URL for

the download - everything after it is unchanged.

G. START THE MASTER PROCESSES

-----------------------------

Run these as the yugabytedb user. Start all three masters before starting

any tablet server.

Only --rpc_bind_addresses changes between nodes. --master_addresses is

identical everywhere and must list all three masters - this is how each

master discovers its peers and forms the Raft group.

Node 1 - ybnode1 (10.0.0.11)

cd /opt/yugabyte/yugabyte-2.11.0.1

./bin/yb-master \

--master_addresses 10.0.0.11:7100,10.0.0.12:7100,10.0.0.13:7100 \

--rpc_bind_addresses 10.0.0.11:7100 \

--fs_data_dirs "/yugabyte01/YUGABYTE/data1,/yugabyte01/YUGABYTE/data2" \

--placement_cloud lab \

--placement_region region1 \

--placement_zone zone1 \

--webserver_interface 0.0.0.0 \

--webserver_port 7000 \

--master_enable_metrics_snapshotter=true \

>& /opt/yugabyte/yb-master.out &

Node 2 - ybnode2 (10.0.0.12)

cd /opt/yugabyte/yugabyte-2.11.0.1

./bin/yb-master \

--master_addresses 10.0.0.11:7100,10.0.0.12:7100,10.0.0.13:7100 \

--rpc_bind_addresses 10.0.0.12:7100 \

--fs_data_dirs "/yugabyte01/YUGABYTE/data1,/yugabyte01/YUGABYTE/data2" \

--placement_cloud lab \

--placement_region region1 \

--placement_zone zone2 \

--webserver_interface 0.0.0.0 \

--webserver_port 7000 \

>& /opt/yugabyte/yb-master.out &

Node 3 - ybnode3 (10.0.0.13)

cd /opt/yugabyte/yugabyte-2.11.0.1

./bin/yb-master \

--master_addresses 10.0.0.11:7100,10.0.0.12:7100,10.0.0.13:7100 \

--rpc_bind_addresses 10.0.0.13:7100 \

--fs_data_dirs "/yugabyte01/YUGABYTE/data1,/yugabyte01/YUGABYTE/data2" \

--placement_cloud lab \

--placement_region region1 \

--placement_zone zone3 \

--webserver_interface 0.0.0.0 \

--webserver_port 7000 \

>& /opt/yugabyte/yb-master.out &

Flag by flag:

--master_addresses

The full membership list of the master Raft group. Must be byte-for-byte

identical on all three nodes. A mismatch - even a duplicated port such

as 10.0.0.11:7100:7100 - prevents quorum from forming, and the masters

will loop on election attempts without ever reporting a clear error.

--rpc_bind_addresses

The address this specific process listens on for inter-node traffic.

This is the one flag that differs per node. Bind to the real interface

IP, not 127.0.0.1, or the other nodes cannot reach it.

--fs_data_dirs

Comma-separated list of directories holding metadata and tablet data.

Must match what you created in section E and be owned by yugabytedb.

--placement_cloud / --placement_region / --placement_zone

Tells YugabyteDB where each node physically sits, so it can place the

three replicas of every tablet in three different failure domains. Keep

cloud and region identical for a single-site cluster, and give each node

a distinct zone. Setting all three nodes to the same zone means the

cluster believes they share a failure domain; setting them arbitrarily

(different clouds on different nodes) produces unbalanced placement and

replicas that cannot be satisfied.

--webserver_interface / --webserver_port

Binds the master admin UI. 0.0.0.0 makes it reachable from your

workstation rather than only from the node itself.

--master_enable_metrics_snapshotter

Periodically writes metrics into a system table. Optional; useful if you

intend to query historical metrics with SQL.

>& /opt/yugabyte/yb-master.out &

Redirects both stdout and stderr to a file and backgrounds the process.

This is fine for a lab. For anything longer-lived, use systemd units so

the processes survive logout and restart on boot.

Identify the master leader

Raft elects exactly one master as leader. Check the logs on each node:

cd /yugabyte01/YUGABYTE/data1/yb-data/master/logs

grep "This master" yb-master.INFO

Sample output - the role changes as the election completes:

I0207 06:30:45.855286 57707 sys_catalog.cc:384] T 0000...0000

P b36db44b23a4490b9514bccc1fab2e8e [sys.catalog]:

This master's current role is: FOLLOWER

I0207 06:52:20.829769 63103 sys_catalog.cc:384] T 0000...0000

P b36db44b23a4490b9514bccc1fab2e8e [sys.catalog]:

This master's current role is: LEADER

Exactly one node should report LEADER. If all three report FOLLOWER

indefinitely, they cannot reach each other - check firewall rules and the

--master_addresses list before anything else.

H. START THE TABLET SERVERS

---------------------------

Tablet servers hold the data and serve client traffic. Start them only after

the masters have elected a leader.

Node 1 - ybnode1 (10.0.0.11)

cd /opt/yugabyte/yugabyte-2.11.0.1

./bin/yb-tserver \

--tserver_master_addrs 10.0.0.11:7100,10.0.0.12:7100,10.0.0.13:7100 \

--rpc_bind_addresses 10.0.0.11:9100 \

--start_pgsql_proxy \

--pgsql_proxy_bind_address 10.0.0.11:5433 \

--cql_proxy_bind_address 10.0.0.11:9042 \

--fs_data_dirs "/yugabyte01/YUGABYTE/data1,/yugabyte01/YUGABYTE/data2" \

--placement_cloud lab \

--placement_region region1 \

--placement_zone zone1 \

>& /opt/yugabyte/yb-tserver.out &

Node 2 - ybnode2 (10.0.0.12)

cd /opt/yugabyte/yugabyte-2.11.0.1

./bin/yb-tserver \

--tserver_master_addrs 10.0.0.11:7100,10.0.0.12:7100,10.0.0.13:7100 \

--rpc_bind_addresses 10.0.0.12:9100 \

--start_pgsql_proxy \

--pgsql_proxy_bind_address 10.0.0.12:5433 \

--cql_proxy_bind_address 10.0.0.12:9042 \

--fs_data_dirs "/yugabyte01/YUGABYTE/data1,/yugabyte01/YUGABYTE/data2" \

--placement_cloud lab \

--placement_region region1 \

--placement_zone zone2 \

>& /opt/yugabyte/yb-tserver.out &

Node 3 - ybnode3 (10.0.0.13)

cd /opt/yugabyte/yugabyte-2.11.0.1

./bin/yb-tserver \

--tserver_master_addrs 10.0.0.11:7100,10.0.0.12:7100,10.0.0.13:7100 \

--rpc_bind_addresses 10.0.0.13:9100 \

--start_pgsql_proxy \

--pgsql_proxy_bind_address 10.0.0.13:5433 \

--cql_proxy_bind_address 10.0.0.13:9042 \

--fs_data_dirs "/yugabyte01/YUGABYTE/data1,/yugabyte01/YUGABYTE/data2" \

--placement_cloud lab \

--placement_region region1 \

--placement_zone zone3 \

>& /opt/yugabyte/yb-tserver.out &

Flag by flag:

--tserver_master_addrs

Where this tablet server finds the masters. Same list as

--master_addresses above. The tablet server heartbeats to whichever

master is currently leader and re-discovers it automatically after a

failover.

--rpc_bind_addresses

Inter-node data traffic for this tablet server. Port 9100, distinct from

the master's 7100. Differs per node.

--start_pgsql_proxy

Starts the YSQL (PostgreSQL-compatible) layer. Without this flag the

node serves YCQL only and psql cannot connect.

--pgsql_proxy_bind_address

Where YSQL listens. 5433 is the YugabyteDB default - deliberately not

5432, so a YugabyteDB node and a stock PostgreSQL instance can coexist

on the same host. Watch for typos: a doubled colon such as

10.0.0.12::5433 is silently invalid and the proxy will not start.

--cql_proxy_bind_address

Where YCQL listens, on the Cassandra default port 9042.

--placement_* flags

Same meaning as for the masters, and they must agree with the values you

gave the master on the same node. Mismatched placement between a node's

master and tablet server confuses replica placement.

I. VERIFY THE CLUSTER

---------------------

Step 1 - Confirm each tablet server found the master leader

cd /yugabyte01/YUGABYTE/data1/yb-data/tserver/logs

grep -i "Connected to a leader master server" yb-tserver.INFO

Expected output:

I0208 12:18:28.482936 112848 heartbeater.cc:305]

P 8e87587653304075a23a19c7e0b43f98:

Connected to a leader master server at 10.0.0.11:7100

Step 2 - Check cluster membership with yb-admin

./bin/yb-admin \

--master_addresses 10.0.0.11:7100,10.0.0.12:7100,10.0.0.13:7100 \

list_all_masters

./bin/yb-admin \

--master_addresses 10.0.0.11:7100,10.0.0.12:7100,10.0.0.13:7100 \

list_all_tablet_servers

What to look for: three masters, exactly one with role LEADER and two with

FOLLOWER; three tablet servers, all with status ALIVE. This is the

authoritative health check - the log greps above only tell you what one node

believes.

Step 3 - Connect over YSQL

./bin/ysqlsh -h 10.0.0.11 -p 5433 -U yugabyte

yugabyte=# \l

yugabyte=# SELECT version();

yugabyte=# CREATE TABLE t1 (id int PRIMARY KEY, name text);

yugabyte=# INSERT INTO t1 VALUES (1,'yugabyte');

yugabyte=# SELECT * FROM t1;

ysqlsh is YugabyteDB's build of psql, so the familiar backslash commands all

work. A successful insert and select confirms the full write path: client to

YSQL proxy, to tablet leader, replicated to two followers, acknowledged.

Step 4 - Open the web consoles

Master UI: http://10.0.0.11:7000

Tablet server UI: http://10.0.0.11:9000

The master UI is the most useful page in the product: it lists live tablet

servers, every table, and how replicas are distributed across your zones.

TROUBLESHOOTING QUICK REFERENCE

-------------------------------

Symptom Most likely cause

------------------------------ ----------------------------------------

All masters stay FOLLOWER firewall-cmd --reload not run, or

--master_addresses differs between nodes

Shared library errors on start post_install.sh was not run

"Too many open files" nofile limit not applied - the shell

predates the limits.conf change

psql cannot connect on 5433 --start_pgsql_proxy missing, or a typo

in --pgsql_proxy_bind_address

Permission denied on data dirs directories not owned by yugabytedb

Frequent leader elections clock skew - NTP not running

NEXT STEPS

----------

The cluster is functional but the processes are backgrounded from an

interactive shell. For anything beyond a lab, wrap yb-master and yb-tserver

in systemd units so they restart on boot, and point Prometheus at the

metrics endpoints on ports 11000, 12000 and 13000.