Redis Cluster

NoSQL > Redis Redis GCP
發布時間 :
字數:2.8k 觀看數 : 評論:
  1. 前言
  2. 環境
  3. 設定 VM
    1. 建立 6 台 Redis 機器
    2. 安裝 Docker
  4. 建置 Redis Cluster
    1. 新增 redisTemplate.conf
    2. 新增 redis.conf
    3. 建置 Redis cluster
    4. Redis Clsuter 新增查詢 Key 備註
  5. 設定測試資料
  6. 縮容 Redis Cluster Node
    1. 將 m3 的 Slot 重新分配給 m1、m2
    2. 將 m3、s3 移出 Redis Cluster
  7. 擴充 Redis Cluster Node
    1. 將 m3、s3 加入 Redis Cluster
    2. m1、m2、m3 重新分配 SLOT
  8. 參考文件

前言

Redis Cluster 與 Redis Master-Slave 架構的差異

Redis Master-Slave
每個 Node 是包含所有資料，Slave 只是分擔讀取功能、failover 可快速替換 Master 的功能
而 Master 仍承受巨量寫入的壓力，且需要另建 Sentinel 來操作 failover

Redis Cluster
自帶 Sentinel 無需額外建立，且透過 Sharding 來將資料分散寫入到各個 Cluster Node 中，分散壓力
本篇透過建立 6 台 Redis 服務，組成 3 個 Master、3 個 Slave 架構的 Redis Cluster
並測試 faileover、節點縮容、節點擴充的流程

環境

Linux VM = ubuntu-2004-lts
Redis Docker映像檔 = redis:6.0.0

VPC、Firewall 建置參考w4560000 - Redis Master-Slave 設定 VPC & Firewall

機器 IP
redis-1 10.140.0.11
redis-2 10.140.0.12
redis-3 10.140.0.13
redis-4 10.140.0.14
redis-5 10.140.0.15
redis-6 10.140.0.16

設定 VM

建立 6 台 Redis 機器

for i in 1 2 3 4 5 6; do
  gcloud compute instances create redis-${i} \
    --async \
    --boot-disk-size 100GB \
    --can-ip-forward \
    --image-family ubuntu-2004-lts \
    --image-project ubuntu-os-cloud \
    --machine-type e2-standard-2 \
    --private-network-ip 10.240.0.1${i} \
    --scopes compute-rw,storage-ro,service-management,service-control,logging-write,monitoring \
    --subnet redis-vpc-subnet \
    --zone asia-east1-a \
    --tags redis-vpc
done

安裝 Docker

參考 w4560000 - Linux 安裝 Docker、Docker-Compose

6 台 VM 都先設定好 volumn 資料夾

mkdir redis && mkdir redisdata

建置 Redis Cluster

新增 redisTemplate.conf

  • cluster-enabled <yes/no>

    是否啟動 redis cluster
    ex: cluster-enabled yes

  • cluster-config-file

    redis cluster 設定檔名，紀錄狀態設定 (使用者無法編輯)
    ex: cluster-config-file nodes.conf

  • cluster-node-timeout

    當 master 在該時間內，沒有回應其他 node 則會執行 failover，將它的 slave 升為新的 master
    ex: cluster-node-timeout 15000

  • cluster-replica-validity-factor

    若設為 0，則無論 slave 與 master 斷開連線多長時間，slave 都會嘗試在 failover 時嘗試成為 master
    若設 大於 0，則可斷開連線時間 = cluster-replica-validity-factor * cluster-node-timeout (秒)
    若該 slave 超過可斷開連線時間，failover 發生時，則不會被推派為 master
    若 cluster-replica-validity-factor = 10、cluster-node-timeout = 5000 毫秒 = 5 秒
    則可斷開連線時間 = 50 秒，slave 超過 50 秒沒有與 master 連線，則喪失成為 master 的機會
    ex: cluster-replica-validity-factor 10

  • cluster-announce-ip

    當前 cluster-node 的 IP
    ex: cluster-announce-ip 10.140.0.11

  • cluster-announce-port

    當前 cluster-node 的 Port
    ex: cluster-announce-port 6379

  • cluster-announce-bus-port

    當前 cluster-node 用來與其他 cluster-node 同步資訊、檢測故障的 Port
    ex: cluster-announce-bus-port 16379

  • repl-ping-replica-period

    master 每隔幾秒會 ping slave，確認是否活著
    ex: repl-ping-replica-period 10 (預設)

  • repl-timeout、repl-ping-replica-period 說明參考 w4560000 - Redis Master-Slave

    在 cluster，repl-timeout 觸發時不會發生 failover，而是由 cluster-node-timeout 在控制 failover

redis-1 ~ redis-6
路徑 ~/redis/redisTemplate.conf

bind 0.0.0.0
protected-mode no
dir ./

# Cluster
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 15000
cluster-replica-validity-factor 10
cluster-announce-ip ${IP}
cluster-announce-port 6379
cluster-announce-bus-port 16379

# AOF
appendonly yes
appendfilename "appendonly.aof"
appendfsync everysec

# AOF rewrite
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 0
auto-aof-rewrite-min-size 1mb
aof-load-truncated no
aof-use-rdb-preamble yes

# REPLICATION
repl-ping-replica-period 10
repl-timeout 60

新增 redis.conf

redis-1 ~ redis-6
路徑 ~/redis/redis.conf

IP=$(hostname -i) envsubst < ./redis/redisTemplate.conf > ./redis/redis.conf

建置 Redis cluster

啟動 Redis 服務
redis-1 ~ redis-6

docker run --name redis \
-p 6379:6379 \
-p 16379:16379 \
-v /home/leozheng0629/redis:/usr/local/etc/redis \
-v /home/leozheng0629/redisdata:/data \
-d redis:6.0.0 \
redis-server /usr/local/etc/redis/redis.conf

建立 Cluster

docker exec -it redis redis-cli --cluster create 10.240.0.11:6379 10.240.0.12:6379 10.240.0.13:6379 10.240.0.14:6379 10.240.0.15:6379 10.240.0.16:6379 --cluster-replicas 1

# 輸出
>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 10.240.0.15:6379 to 10.240.0.11:6379
Adding replica 10.240.0.16:6379 to 10.240.0.12:6379
Adding replica 10.240.0.14:6379 to 10.240.0.13:6379
M: aefdddbe65c8105c47b545bab901578c7a92f2f9 10.240.0.11:6379
   slots:[0-5460] (5461 slots) master
M: 23808824f980269c05af462dbc3b82b5ab63995d 10.240.0.12:6379
   slots:[5461-10922] (5462 slots) master
M: bae34725dfd1fb24f4ea6c9464a5d2c7e56db445 10.240.0.13:6379
   slots:[10923-16383] (5461 slots) master
S: d3b65dce1ca8befabd8c929b7481970ac892e6ac 10.240.0.14:6379
   replicates bae34725dfd1fb24f4ea6c9464a5d2c7e56db445
S: b46f42b32e50918480923dcee3f13201f2b5fde4 10.240.0.15:6379
   replicates aefdddbe65c8105c47b545bab901578c7a92f2f9
S: 4436d2bf76ec3bc754acc7df8284e8fd87dd0f08 10.240.0.16:6379
   replicates 23808824f980269c05af462dbc3b82b5ab63995d
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
.....
>>> Performing Cluster Check (using node 10.240.0.11:6379)
M: aefdddbe65c8105c47b545bab901578c7a92f2f9 10.240.0.11:6379
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
M: bae34725dfd1fb24f4ea6c9464a5d2c7e56db445 10.240.0.13:6379
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
M: 23808824f980269c05af462dbc3b82b5ab63995d 10.240.0.12:6379
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
S: d3b65dce1ca8befabd8c929b7481970ac892e6ac 10.240.0.14:6379
   slots: (0 slots) slave
   replicates bae34725dfd1fb24f4ea6c9464a5d2c7e56db445
S: b46f42b32e50918480923dcee3f13201f2b5fde4 10.240.0.15:6379
   slots: (0 slots) slave
   replicates aefdddbe65c8105c47b545bab901578c7a92f2f9
S: 4436d2bf76ec3bc754acc7df8284e8fd87dd0f08 10.240.0.16:6379
   slots: (0 slots) slave
   replicates 23808824f980269c05af462dbc3b82b5ab63995d
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

備註 Port 除了 6379 外，cluster-announce-bus-port 設定的 16379 也要開通
cluster-announce-bus-port 是 cluster 之間用來做故障檢測、failover、設定更新的 Port
若 16379 沒開通，cluster create 時會一直卡在 Waiting for the cluster to join，無法順利建置

建置好後確認一下 cluster 狀態

docker exec -it redis redis-cli cluster nodes

# 輸出
bae34725dfd1fb24f4ea6c9464a5d2c7e56db445 10.240.0.13:6379@16379 master - 0 1688031261897 3 connected 10923-16383
23808824f980269c05af462dbc3b82b5ab63995d 10.240.0.12:6379@16379 master - 0 1688031258000 2 connected 5461-10922
d3b65dce1ca8befabd8c929b7481970ac892e6ac 10.240.0.14:6379@16379 slave bae34725dfd1fb24f4ea6c9464a5d2c7e56db445 0 1688031260000 4 connected
b46f42b32e50918480923dcee3f13201f2b5fde4 10.240.0.15:6379@16379 slave aefdddbe65c8105c47b545bab901578c7a92f2f9 0 1688031259890 5 connected
aefdddbe65c8105c47b545bab901578c7a92f2f9 10.240.0.11:6379@16379 myself,master - 0 1688031260000 1 connected 0-5460
4436d2bf76ec3bc754acc7df8284e8fd87dd0f08 10.240.0.16:6379@16379 slave 23808824f980269c05af462dbc3b82b5ab63995d 0 1688031260894 6 connected

docker exec -it redis redis-cli cluster info

# 輸出
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:1
cluster_stats_messages_ping_sent:603
cluster_stats_messages_pong_sent:652
cluster_stats_messages_sent:1255
cluster_stats_messages_ping_received:647
cluster_stats_messages_pong_received:603
cluster_stats_messages_meet_received:5
cluster_stats_messages_received:1255

Redis Clsuter 新增查詢 Key 備註

當 cluster 建立好後，要查詢或操作時，要透過 cluster 模式進入 (redis-cli -c)

docker exec -it redis redis-cli -c Set Key1 "1"

# 輸出
OK

# 若沒透過 cluster 模式 則會無法操作，因 Redis cluster 是透過計算 Key 的 Hash 值來判斷要存到哪個 redis node 中
# 若沒加上 -c，則只能查該台 redis 資料
docker exec -it redis redis-cli Get Key1

# 輸出
(error) MOVED 15360 10.240.0.13:6379

設定測試資料

先計算出 Key 的 Slot

docker exec -it redis redis-cli cluster keyslot Key1

# 輸出
(integer) 5291

docker exec -it redis redis-cli cluster keyslot Key2

# 輸出
(integer) 9416

docker exec -it redis redis-cli cluster keyslot Key3

# 輸出
(integer) 13545

確認Key可以分散到各個Node後，再來設定Key

docker exec -it redis redis-cli -c Set Key1 "1"
docker exec -it redis redis-cli -c Set Key2 "2"
docker exec -it redis redis-cli -c Set Key3 "3"

設定後再次確認Key分布在三個Node上

docker exec -it redis redis-cli --cluster check 10.240.0.11:6379

# 輸出
10.240.0.11:6379 (aefdddbe...) -> 1 keys | 5461 slots | 1 slaves.
10.240.0.13:6379 (bae34725...) -> 1 keys | 5461 slots | 1 slaves.
10.240.0.12:6379 (23808824...) -> 1 keys | 5462 slots | 1 slaves.
[OK] 3 keys in 3 masters.
...

目前 Reids Cluster 配置

master slave slot Keys
10.240.0.11 (m1) 10.240.0.15 (s1) 0-5460 (共有5461) Key1
10.240.0.12 (m2) 10.240.0.16 (s2) 5461-10922 (共有5462) Key2
10.240.0.13 (m3) 10.240.0.14 (s3) 10923-16383 (共有5461) Key3

查看 Cluster slot
docker exec -it redis redis-cli -c CLUSTER SLOTS

縮容 Redis Cluster Node

目標: 將 m3、s3 移出 Redis Cluster

將 m3 的 Slot 重新分配給 m1、m2

原本 m3 有 5461 個 Slot，先全部分配給 m1，再透過 redis-cli rebalance 自動平均 SLOT (這樣就不用自行計算要分配多少 SLOT 給各個 Node)

# redis-cli --cluster reshard <host>:<port> --cluster-from <node-id> --cluster-to <node-id> --cluster-slots <number of slots> --cluster-yes
# redis-cli --cluster reshard <要移除的Redis IP>:<要移除的Redis Port> --cluster-from <要移除的Redis ID> --cluster-to <要遷移的Redis IP> --cluster-slots <Slot數量> --cluster-yes

# 分給 m1
docker exec -it redis redis-cli --cluster reshard 10.240.0.13:6379 --cluster-from bae34725dfd1fb24f4ea6c9464a5d2c7e56db445 --cluster-to aefdddbe65c8105c47b545bab901578c7a92f2f9 --cluster-slots 5461 --cluster-yes

# 重新分配 SLOT
docker exec -it redis redis-cli --cluster rebalance 10.240.0.11:6379

分配好後再次確認目前SLOT

docker exec -it redis redis-cli --cluster check 10.240.0.11:6379

# 輸出
10.240.0.11:6379 (aefdddbe...) -> 2 keys | 8192 slots | 1 slaves.
10.240.0.13:6379 (bae34725...) -> 0 keys | 0 slots | 1 slaves.
10.240.0.12:6379 (23808824...) -> 1 keys | 8192 slots | 1 slaves.
...

m3 的 SLOT 已被分配給 m1、m2

將 m3、s3 移出 Redis Cluster

# redis-cli --cluster del-node <host>:<port> <node-id>
# redis-cli --cluster del-node <要移除的Redis IP>:<要移除的Redis Port> <要移除的Redis ID>

# 移除 m3
docker exec -it redis redis-cli --cluster del-node 10.240.0.13:6379 bae34725dfd1fb24f4ea6c9464a5d2c7e56db445

# 移除 s3
docker exec -it redis redis-cli --cluster del-node 10.240.0.14:6379 d3b65dce1ca8befabd8c929b7481970ac892e6ac

移除後確認 Cluster Node

docker exec -it redis redis-cli cluster nodes

# 輸出
aefdddbe65c8105c47b545bab901578c7a92f2f9 10.240.0.11:6379@16379 myself,master - 0 1690628923000 13 connected 2730-5460 10923-16383
b46f42b32e50918480923dcee3f13201f2b5fde4 10.240.0.15:6379@16379 slave 23808824f980269c05af462dbc3b82b5ab63995d 0 1690628921296 14 connected
23808824f980269c05af462dbc3b82b5ab63995d 10.240.0.12:6379@16379 master - 0 1690628924309 14 connected 0-2729 5461-10922
4436d2bf76ec3bc754acc7df8284e8fd87dd0f08 10.240.0.16:6379@16379 slave aefdddbe65c8105c47b545bab901578c7a92f2f9 0 1690628923304 13 connected

移除 m3、s3 後剩下 m1、s1、m2、s2，而原本在 m3 的 Key3 也被轉移給 m1 了

擴充 Redis Cluster Node

目標: 將 m3、s3 重新加入 Redis Cluster

將 m3、s3 加入 Redis Cluster

# redis-cli --cluster add-node <new-host>:<new-port> <old-host>:<old-host>
# redis-cli --cluster add-node <要加入的Redis IP>:<要加入的Redis Port> <已存在的Redis IP>:<已存在的Reids Port>

# 將 m3 加入 Redis Cluster
docker exec -it redis redis-cli --cluster add-node 10.240.0.13:6379 10.240.0.11:6379

# 將 s3 加入 Redis Cluster，並且為 m3 的 Slave
docker exec -it redis redis-cli --cluster add-node 10.240.0.14:6379 10.240.0.11:6379 --cluster-slave --cluster-master-id bae34725dfd1fb24f4ea6c9464a5d2c7e56db445

確認目前 Cluster 配置

docker exec -it redis redis-cli --cluster check 10.240.0.11:6379

# 輸出
10.240.0.11:6379 (aefdddbe...) -> 2 keys | 8192 slots | 1 slaves.
10.240.0.13:6379 (bae34725...) -> 0 keys | 0 slots | 1 slaves.
10.240.0.12:6379 (23808824...) -> 1 keys | 8192 slots | 1 slaves.
...

確認 m3、s3 已被加回 Cluster

m1、m2、m3 重新分配 SLOT

# 將目前 cluster Node 的 SLOT，分配給 m3
docker exec -it redis redis-cli --cluster reshard 10.240.0.11:6379 --cluster-from all --cluster-to bae34725dfd1fb24f4ea6c9464a5d2c7e56db445 --cluster-slots 5461 --cluster-yes

分配完後確認目前 Cluster 配置

docker exec -it redis redis-cli --cluster check 10.240.0.11:6379

# 輸出
10.240.0.11:6379 (aefdddbe...) -> 1 keys | 5461 slots | 1 slaves.
10.240.0.13:6379 (bae34725...) -> 1 keys | 5462 slots | 1 slaves.
10.240.0.12:6379 (23808824...) -> 1 keys | 5461 slots | 1 slaves.
...

重新分配過後的 SLOT 分布

docker exec -it redis redis-cli --cluster node

# 輸出
aefdddbe65c8105c47b545bab901578c7a92f2f9 10.240.0.11:6379@16379 myself,master - 0 1690631611000 26 connected 10923-16383
b46f42b32e50918480923dcee3f13201f2b5fde4 10.240.0.15:6379@16379 slave 23808824f980269c05af462dbc3b82b5ab63995d 0 1690631611006 27 connected
bae34725dfd1fb24f4ea6c9464a5d2c7e56db445 10.240.0.13:6379@16379 master - 0 1690631612010 28 connected 0-2729 2731-3640 5461-7281
23808824f980269c05af462dbc3b82b5ab63995d 10.240.0.12:6379@16379 master - 0 1690631610004 27 connected 2730 3641-5460 7282-10922
d3b65dce1ca8befabd8c929b7481970ac892e6ac 10.240.0.14:6379@16379 slave aefdddbe65c8105c47b545bab901578c7a92f2f9 0 1690631614019 26 connected
4436d2bf76ec3bc754acc7df8284e8fd87dd0f08 10.240.0.16:6379@16379 slave bae34725dfd1fb24f4ea6c9464a5d2c7e56db445 0 1690631613015 28 connected

參考文件

HackMD tienyulin - Redis (六) - 主從複製、哨兵與叢集模式#叢集模式
CSDN 大话JAVA的那些事 - 二、redis.conf文件详解
石頭的coding之路 - Redis Cluster 介紹
TPIU - Docker + Redis Cluster 實戰
ITpub博客 壹頁書 - Redis主从复制网络闪断处理
博客園 一见 - redis的repl-ping-slave-period和repl-ping-replica-period
博客園 一见 - Redis集群的主从切换研究
Redis 官方文件 - Scaling with Redis Cluster
Baeldung - Redis Sentinel vs Clustering
微信公眾號 SH的全栈笔记 - 深度图解Redis Cluster原理

轉載請註明來源，若有任何錯誤或表達不清楚的地方，歡迎在下方評論區留言，也可以來信至 leozheng0621@gmail.com
如果文章對您有幫助，歡迎斗內(donate)，請我喝杯咖啡

斗內💰

©2022-2022 LeoZheng

Built with Hexo and 3-hexo theme

×

歡迎斗內

github