Redis Cluster

NoSQL > Redis Redis GCP

發布時間 : 2023-07-02 21:10

字數:2.8k 觀看數 :

前言
環境
設定 VM
1. 建立 6 台 Redis 機器
2. 安裝 Docker
建置 Redis Cluster
設定測試資料
縮容 Redis Cluster Node
1. 將 m3 的 Slot 重新分配給 m1、m2
2. 將 m3、s3 移出 Redis Cluster
擴充 Redis Cluster Node
1. 將 m3、s3 加入 Redis Cluster
2. m1、m2、m3 重新分配 SLOT
參考文件

前言

Redis Cluster 與 Redis Master-Slave 架構的差異

Redis Master-Slave
每個 Node 是包含所有資料，Slave 只是分擔讀取功能、failover 可快速替換 Master 的功能
而 Master 仍承受巨量寫入的壓力，且需要另建 Sentinel 來操作 failover

Redis Cluster
自帶 Sentinel 無需額外建立，且透過 Sharding 來將資料分散寫入到各個 Cluster Node 中，分散壓力
本篇透過建立 6 台 Redis 服務，組成 3 個 Master、3 個 Slave 架構的 Redis Cluster
並測試 faileover、節點縮容、節點擴充的流程

環境

Linux VM = ubuntu-2004-lts
Redis Docker映像檔 = redis:6.0.0

VPC、Firewall 建置參考w4560000 - Redis Master-Slave 設定 VPC & Firewall

機器	IP
redis-1	10.140.0.11
redis-2	10.140.0.12
redis-3	10.140.0.13
redis-4	10.140.0.14
redis-5	10.140.0.15
redis-6	10.140.0.16

設定 VM

建立 6 台 Redis 機器

for i in 1 2 3 4 5 6; do
  gcloud compute instances create redis-${i} \
    --async \
    --boot-disk-size 100GB \
    --can-ip-forward \
    --image-family ubuntu-2004-lts \
    --image-project ubuntu-os-cloud \
    --machine-type e2-standard-2 \
    --private-network-ip 10.240.0.1${i} \
    --scopes compute-rw,storage-ro,service-management,service-control,logging-write,monitoring \
    --subnet redis-vpc-subnet \
    --zone asia-east1-a \
    --tags redis-vpc
done

安裝 Docker

參考 w4560000 - Linux 安裝 Docker、Docker-Compose

6 台 VM 都先設定好 volumn 資料夾

mkdir redis && mkdir redisdata

建置 Redis Cluster

新增 redisTemplate.conf

cluster-enabled <yes/no>

是否啟動 redis cluster
ex: cluster-enabled yes
cluster-config-file

redis cluster 設定檔名，紀錄狀態設定 (使用者無法編輯)
ex: cluster-config-file nodes.conf
cluster-node-timeout

當 master 在該時間內，沒有回應其他 node 則會執行 failover，將它的 slave 升為新的 master
ex: cluster-node-timeout 15000
cluster-replica-validity-factor

若設為 0，則無論 slave 與 master 斷開連線多長時間，slave 都會嘗試在 failover 時嘗試成為 master
若設大於 0，則可斷開連線時間 = cluster-replica-validity-factor * cluster-node-timeout (秒)
若該 slave 超過可斷開連線時間，failover 發生時，則不會被推派為 master
若 cluster-replica-validity-factor = 10、cluster-node-timeout = 5000 毫秒 = 5 秒
則可斷開連線時間 = 50 秒，slave 超過 50 秒沒有與 master 連線，則喪失成為 master 的機會
ex: cluster-replica-validity-factor 10
cluster-announce-ip

當前 cluster-node 的 IP
ex: cluster-announce-ip 10.140.0.11
cluster-announce-port

當前 cluster-node 的 Port
ex: cluster-announce-port 6379
cluster-announce-bus-port

當前 cluster-node 用來與其他 cluster-node 同步資訊、檢測故障的 Port
ex: cluster-announce-bus-port 16379
repl-ping-replica-period

master 每隔幾秒會 ping slave，確認是否活著
ex: repl-ping-replica-period 10 (預設)
repl-timeout、repl-ping-replica-period 說明參考 w4560000 - Redis Master-Slave

在 cluster，repl-timeout 觸發時不會發生 failover，而是由 cluster-node-timeout 在控制 failover

redis-1 ~ redis-6
路徑 ~/redis/redisTemplate.conf

bind 0.0.0.0
protected-mode no
dir ./

# Cluster
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 15000
cluster-replica-validity-factor 10
cluster-announce-ip ${IP}
cluster-announce-port 6379
cluster-announce-bus-port 16379

# AOF
appendonly yes
appendfilename "appendonly.aof"
appendfsync everysec

# AOF rewrite
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 0
auto-aof-rewrite-min-size 1mb
aof-load-truncated no
aof-use-rdb-preamble yes

# REPLICATION
repl-ping-replica-period 10
repl-timeout 60

新增 redis.conf

redis-1 ~ redis-6
路徑 ~/redis/redis.conf

IP=$(hostname -i) envsubst < ./redis/redisTemplate.conf > ./redis/redis.conf

建置 Redis cluster

啟動 Redis 服務
redis-1 ~ redis-6

docker run --name redis \
-p 6379:6379 \
-p 16379:16379 \
-v /home/leozheng0629/redis:/usr/local/etc/redis \
-v /home/leozheng0629/redisdata:/data \
-d redis:6.0.0 \
redis-server /usr/local/etc/redis/redis.conf

建立 Cluster

docker exec -it redis redis-cli --cluster create 10.240.0.11:6379 10.240.0.12:6379 10.240.0.13:6379 10.240.0.14:6379 10.240.0.15:6379 10.240.0.16:6379 --cluster-replicas 1

# 輸出
>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 10.240.0.15:6379 to 10.240.0.11:6379
Adding replica 10.240.0.16:6379 to 10.240.0.12:6379
Adding replica 10.240.0.14:6379 to 10.240.0.13:6379
M: aefdddbe65c8105c47b545bab901578c7a92f2f9 10.240.0.11:6379
   slots:[0-5460] (5461 slots) master
M: 23808824f980269c05af462dbc3b82b5ab63995d 10.240.0.12:6379
   slots:[5461-10922] (5462 slots) master
M: bae34725dfd1fb24f4ea6c9464a5d2c7e56db445 10.240.0.13:6379
   slots:[10923-16383] (5461 slots) master
S: d3b65dce1ca8befabd8c929b7481970ac892e6ac 10.240.0.14:6379
   replicates bae34725dfd1fb24f4ea6c9464a5d2c7e56db445
S: b46f42b32e50918480923dcee3f13201f2b5fde4 10.240.0.15:6379
   replicates aefdddbe65c8105c47b545bab901578c7a92f2f9
S: 4436d2bf76ec3bc754acc7df8284e8fd87dd0f08 10.240.0.16:6379
   replicates 23808824f980269c05af462dbc3b82b5ab63995d
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
.....
>>> Performing Cluster Check (using node 10.240.0.11:6379)
M: aefdddbe65c8105c47b545bab901578c7a92f2f9 10.240.0.11:6379
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
M: bae34725dfd1fb24f4ea6c9464a5d2c7e56db445 10.240.0.13:6379
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
M: 23808824f980269c05af462dbc3b82b5ab63995d 10.240.0.12:6379
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
S: d3b65dce1ca8befabd8c929b7481970ac892e6ac 10.240.0.14:6379
   slots: (0 slots) slave
   replicates bae34725dfd1fb24f4ea6c9464a5d2c7e56db445
S: b46f42b32e50918480923dcee3f13201f2b5fde4 10.240.0.15:6379
   slots: (0 slots) slave
   replicates aefdddbe65c8105c47b545bab901578c7a92f2f9
S: 4436d2bf76ec3bc754acc7df8284e8fd87dd0f08 10.240.0.16:6379
   slots: (0 slots) slave
   replicates 23808824f980269c05af462dbc3b82b5ab63995d
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

備註 Port 除了 6379 外，cluster-announce-bus-port 設定的 16379 也要開通
cluster-announce-bus-port 是 cluster 之間用來做故障檢測、failover、設定更新的 Port
若 16379 沒開通，cluster create 時會一直卡在 Waiting for the cluster to join，無法順利建置

建置好後確認一下 cluster 狀態

docker exec -it redis redis-cli cluster nodes

# 輸出
bae34725dfd1fb24f4ea6c9464a5d2c7e56db445 10.240.0.13:6379@16379 master - 0 1688031261897 3 connected 10923-16383
23808824f980269c05af462dbc3b82b5ab63995d 10.240.0.12:6379@16379 master - 0 1688031258000 2 connected 5461-10922
d3b65dce1ca8befabd8c929b7481970ac892e6ac 10.240.0.14:6379@16379 slave bae34725dfd1fb24f4ea6c9464a5d2c7e56db445 0 1688031260000 4 connected
b46f42b32e50918480923dcee3f13201f2b5fde4 10.240.0.15:6379@16379 slave aefdddbe65c8105c47b545bab901578c7a92f2f9 0 1688031259890 5 connected
aefdddbe65c8105c47b545bab901578c7a92f2f9 10.240.0.11:6379@16379 myself,master - 0 1688031260000 1 connected 0-5460
4436d2bf76ec3bc754acc7df8284e8fd87dd0f08 10.240.0.16:6379@16379 slave 23808824f980269c05af462dbc3b82b5ab63995d 0 1688031260894 6 connected

docker exec -it redis redis-cli cluster info

# 輸出
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:1
cluster_stats_messages_ping_sent:603
cluster_stats_messages_pong_sent:652
cluster_stats_messages_sent:1255
cluster_stats_messages_ping_received:647
cluster_stats_messages_pong_received:603
cluster_stats_messages_meet_received:5
cluster_stats_messages_received:1255

Redis Clsuter 新增查詢 Key 備註

當 cluster 建立好後，要查詢或操作時，要透過 cluster 模式進入 (redis-cli -c)

docker exec -it redis redis-cli -c Set Key1 "1"

# 輸出
OK

# 若沒透過 cluster 模式 則會無法操作，因 Redis cluster 是透過計算 Key 的 Hash 值來判斷要存到哪個 redis node 中
# 若沒加上 -c，則只能查該台 redis 資料
docker exec -it redis redis-cli Get Key1

# 輸出
(error) MOVED 15360 10.240.0.13:6379

設定測試資料

先計算出 Key 的 Slot

docker exec -it redis redis-cli cluster keyslot Key1

# 輸出
(integer) 5291

docker exec -it redis redis-cli cluster keyslot Key2

# 輸出
(integer) 9416

docker exec -it redis redis-cli cluster keyslot Key3

# 輸出
(integer) 13545

確認Key可以分散到各個Node後，再來設定Key

docker exec -it redis redis-cli -c Set Key1 "1"
docker exec -it redis redis-cli -c Set Key2 "2"
docker exec -it redis redis-cli -c Set Key3 "3"

設定後再次確認Key分布在三個Node上

docker exec -it redis redis-cli --cluster check 10.240.0.11:6379

# 輸出
10.240.0.11:6379 (aefdddbe...) -> 1 keys | 5461 slots | 1 slaves.
10.240.0.13:6379 (bae34725...) -> 1 keys | 5461 slots | 1 slaves.
10.240.0.12:6379 (23808824...) -> 1 keys | 5462 slots | 1 slaves.
[OK] 3 keys in 3 masters.
...

目前 Reids Cluster 配置

master	slave	slot	Keys
10.240.0.11 (m1)	10.240.0.15 (s1)	0-5460 (共有5461)	Key1
10.240.0.12 (m2)	10.240.0.16 (s2)	5461-10922 (共有5462)	Key2
10.240.0.13 (m3)	10.240.0.14 (s3)	10923-16383 (共有5461)	Key3

查看 Cluster slot
docker exec -it redis redis-cli -c CLUSTER SLOTS

縮容 Redis Cluster Node

目標: 將 m3、s3 移出 Redis Cluster

將 m3 的 Slot 重新分配給 m1、m2

原本 m3 有 5461 個 Slot，先全部分配給 m1，再透過 redis-cli rebalance 自動平均 SLOT (這樣就不用自行計算要分配多少 SLOT 給各個 Node)

# redis-cli --cluster reshard <host>:<port> --cluster-from <node-id> --cluster-to <node-id> --cluster-slots <number of slots> --cluster-yes
# redis-cli --cluster reshard <要移除的Redis IP>:<要移除的Redis Port> --cluster-from <要移除的Redis ID> --cluster-to <要遷移的Redis IP> --cluster-slots <Slot數量> --cluster-yes

# 分給 m1
docker exec -it redis redis-cli --cluster reshard 10.240.0.13:6379 --cluster-from bae34725dfd1fb24f4ea6c9464a5d2c7e56db445 --cluster-to aefdddbe65c8105c47b545bab901578c7a92f2f9 --cluster-slots 5461 --cluster-yes

# 重新分配 SLOT
docker exec -it redis redis-cli --cluster rebalance 10.240.0.11:6379

分配好後再次確認目前SLOT

docker exec -it redis redis-cli --cluster check 10.240.0.11:6379

# 輸出
10.240.0.11:6379 (aefdddbe...) -> 2 keys | 8192 slots | 1 slaves.
10.240.0.13:6379 (bae34725...) -> 0 keys | 0 slots | 1 slaves.
10.240.0.12:6379 (23808824...) -> 1 keys | 8192 slots | 1 slaves.
...

m3 的 SLOT 已被分配給 m1、m2

將 m3、s3 移出 Redis Cluster

# redis-cli --cluster del-node <host>:<port> <node-id>
# redis-cli --cluster del-node <要移除的Redis IP>:<要移除的Redis Port> <要移除的Redis ID>

# 移除 m3
docker exec -it redis redis-cli --cluster del-node 10.240.0.13:6379 bae34725dfd1fb24f4ea6c9464a5d2c7e56db445

# 移除 s3
docker exec -it redis redis-cli --cluster del-node 10.240.0.14:6379 d3b65dce1ca8befabd8c929b7481970ac892e6ac

移除後確認 Cluster Node

docker exec -it redis redis-cli cluster nodes

# 輸出
aefdddbe65c8105c47b545bab901578c7a92f2f9 10.240.0.11:6379@16379 myself,master - 0 1690628923000 13 connected 2730-5460 10923-16383
b46f42b32e50918480923dcee3f13201f2b5fde4 10.240.0.15:6379@16379 slave 23808824f980269c05af462dbc3b82b5ab63995d 0 1690628921296 14 connected
23808824f980269c05af462dbc3b82b5ab63995d 10.240.0.12:6379@16379 master - 0 1690628924309 14 connected 0-2729 5461-10922
4436d2bf76ec3bc754acc7df8284e8fd87dd0f08 10.240.0.16:6379@16379 slave aefdddbe65c8105c47b545bab901578c7a92f2f9 0 1690628923304 13 connected

移除 m3、s3 後剩下 m1、s1、m2、s2，而原本在 m3 的 Key3 也被轉移給 m1 了

擴充 Redis Cluster Node

目標: 將 m3、s3 重新加入 Redis Cluster

將 m3、s3 加入 Redis Cluster

# redis-cli --cluster add-node <new-host>:<new-port> <old-host>:<old-host>
# redis-cli --cluster add-node <要加入的Redis IP>:<要加入的Redis Port> <已存在的Redis IP>:<已存在的Reids Port>

# 將 m3 加入 Redis Cluster
docker exec -it redis redis-cli --cluster add-node 10.240.0.13:6379 10.240.0.11:6379

# 將 s3 加入 Redis Cluster，並且為 m3 的 Slave
docker exec -it redis redis-cli --cluster add-node 10.240.0.14:6379 10.240.0.11:6379 --cluster-slave --cluster-master-id bae34725dfd1fb24f4ea6c9464a5d2c7e56db445

確認目前 Cluster 配置

docker exec -it redis redis-cli --cluster check 10.240.0.11:6379

# 輸出
10.240.0.11:6379 (aefdddbe...) -> 2 keys | 8192 slots | 1 slaves.
10.240.0.13:6379 (bae34725...) -> 0 keys | 0 slots | 1 slaves.
10.240.0.12:6379 (23808824...) -> 1 keys | 8192 slots | 1 slaves.
...

確認 m3、s3 已被加回 Cluster

m1、m2、m3 重新分配 SLOT

# 將目前 cluster Node 的 SLOT，分配給 m3
docker exec -it redis redis-cli --cluster reshard 10.240.0.11:6379 --cluster-from all --cluster-to bae34725dfd1fb24f4ea6c9464a5d2c7e56db445 --cluster-slots 5461 --cluster-yes

分配完後確認目前 Cluster 配置

docker exec -it redis redis-cli --cluster check 10.240.0.11:6379

# 輸出
10.240.0.11:6379 (aefdddbe...) -> 1 keys | 5461 slots | 1 slaves.
10.240.0.13:6379 (bae34725...) -> 1 keys | 5462 slots | 1 slaves.
10.240.0.12:6379 (23808824...) -> 1 keys | 5461 slots | 1 slaves.
...

重新分配過後的 SLOT 分布

docker exec -it redis redis-cli --cluster node

# 輸出
aefdddbe65c8105c47b545bab901578c7a92f2f9 10.240.0.11:6379@16379 myself,master - 0 1690631611000 26 connected 10923-16383
b46f42b32e50918480923dcee3f13201f2b5fde4 10.240.0.15:6379@16379 slave 23808824f980269c05af462dbc3b82b5ab63995d 0 1690631611006 27 connected
bae34725dfd1fb24f4ea6c9464a5d2c7e56db445 10.240.0.13:6379@16379 master - 0 1690631612010 28 connected 0-2729 2731-3640 5461-7281
23808824f980269c05af462dbc3b82b5ab63995d 10.240.0.12:6379@16379 master - 0 1690631610004 27 connected 2730 3641-5460 7282-10922
d3b65dce1ca8befabd8c929b7481970ac892e6ac 10.240.0.14:6379@16379 slave aefdddbe65c8105c47b545bab901578c7a92f2f9 0 1690631614019 26 connected
4436d2bf76ec3bc754acc7df8284e8fd87dd0f08 10.240.0.16:6379@16379 slave bae34725dfd1fb24f4ea6c9464a5d2c7e56db445 0 1690631613015 28 connected

參考文件

HackMD tienyulin - Redis (六) - 主從複製、哨兵與叢集模式#叢集模式
 CSDN 大话JAVA的那些事 - 二、redis.conf文件详解
 石頭的coding之路 - Redis Cluster 介紹
 TPIU - Docker + Redis Cluster 實戰
 ITpub博客壹頁書 - Redis主从复制网络闪断处理
 博客園一见 - redis的repl-ping-slave-period和repl-ping-replica-period
博客園一见 - Redis集群的主从切换研究
 Redis 官方文件 - Scaling with Redis Cluster
Baeldung - Redis Sentinel vs Clustering
微信公眾號 SH的全栈笔记 - 深度图解Redis Cluster原理

轉載請註明來源，若有任何錯誤或表達不清楚的地方，歡迎在下方評論區留言，也可以來信至 leozheng0621@gmail.com
如果文章對您有幫助，歡迎斗內(donate)，請我喝杯咖啡