.Net Core 連線 Redis Master/Slave 配合 Sentinel 測試故障轉移

Backend > .NetCore .NetCore Redis
發布時間 :
字數:1.5k 觀看數 : 評論:
  1. 前言
  2. 環境
  3. 方法一 詢問 Sentinel
  4. 方法二 由 StackExchange.Redis 自動識別 Master、Replica
  5. 實作方法二
  6. 參考文件

前言

上一篇 w4560000 - Redis Master-Slave Sentinel(哨兵模式) 建置完 Redis Master-Slave
並且搭配哨兵模式做故障轉移後，程式端方面也需要稍微做調整，因目前設定是 Master 有讀寫功能，Slave 只有讀的功能

爬文後發現有兩種方式可實現

環境

.Net 6
StackExchange.Redis Nuget 版本號: 2.6.116

機器 IP
redis-master 10.140.0.20
redis-slave-1 10.140.0.21
redis-slave-2 10.140.0.22

環境建置可參考 w4560000 - Redis Master-Slave Sentinel(哨兵模式)、w4560000 - Redis Master-Slave 環境

方法一 詢問 Sentinel

程式端主動跟 Sentinel 查詢目前 Master 的 Endpoint，以及被動跟 Sentinel 訂閱 +switch-master 的通知

剛故障轉移成功後，Sentinel 會發布消息 ex: +switch-master mymaster 10.240.0.21 6379 10.240.0.20 6379
參考 Redis 官方文件 - Sentinel Pub/Sub messages

需要自行向哨兵詢問目前 Master Endpoint，若 Redis 讀寫還需要細分到不同的 ConnectionMultiplexer 的話
Master、Slave 都需要動態詢問哨兵來調整 Endpoint
在專案啟動時，一開始就先向詢問哨兵目前 Master、Replica Endpoint 做一個初始化 Redis 連線的動作

using StackExchange.Redis;

namespace RedisTest
{
    internal class Program
    {
        private readonly object _lock = new object();
        private static IConnectionMultiplexer _masterConnectionMultiplexer;
        private static IConnectionMultiplexer _replicaConnectionMultiplexer;
        private static IConnectionMultiplexer _sentinelConnectionMultiplexer;

        private static void Main(string[] args)
        {
            InitSentinelConnection();
            ResetConnection();

            while (true)
            {
                try
                {
                    var value = _replicaConnectionMultiplexer.GetDatabase().StringGet("Key1");
                    Console.WriteLine($"{DateTime.Now:yyyy-MM-dd HH:mm:ss.fff} Key1 = {value}");
                    var newValue = Convert.ToInt32(value) + 1;
                    Console.WriteLine($"{DateTime.Now:yyyy-MM-dd HH:mm:ss.fff} Key1 預計更新為 {newValue}");
                    _masterConnectionMultiplexer.GetDatabase().StringSet("Key1", newValue.ToString());
                    Console.WriteLine($"更新後確認 Key1 = {_replicaConnectionMultiplexer.GetDatabase().StringGet("Key1")}\n");

                    Thread.Sleep(1000);
                }
                catch (Exception ex)
                {
                    Console.WriteLine(DateTime.Now.ToString("HH:mm:ss.fff") + $" Error {ex.Message}");
                }
            }

            Console.ReadLine();
        }

        private static void InitSentinelConnection()
        {
            var sentinelConfig = new ConfigurationOptions
            {
                EndPoints = {
                        { "34.80.222.88:26379" }
                    },
                AbortOnConnectFail = false,
                ServiceName = "mymaster",
                TieBreaker = string.Empty,
                CommandMap = CommandMap.Sentinel,
                AllowAdmin = true,
                Ssl = false,
                ConnectTimeout = 1000,
                SyncTimeout = 1000,
                ConnectRetry = 5,
            };

            _sentinelConnectionMultiplexer = ConnectionMultiplexer.SentinelConnect(sentinelConfig);
            ISubscriber subscriber = _sentinelConnectionMultiplexer.GetSubscriber();

            // 哨兵訂閱 switch-master 來切換 master connection
            subscriber.Subscribe("+switch-master", (channel, message) =>
            {
                Console.WriteLine($"Channel: {channel}, Message: {message}");
                ResetConnection();
            });

            Console.WriteLine("哨兵連線成功");
        }

        /// <summary>
        /// 重設 Connection
        /// </summary>
        private static void ResetConnection()
        {
            var endPoint = _sentinelConnectionMultiplexer.GetEndPoints().First();
            var server = _sentinelConnectionMultiplexer.GetServer(endPoint);

            var masterEndPoint = server.SentinelGetMasterAddressByName("mymaster");
            var masterConfiguration = new ConfigurationOptions()
            {
                AbortOnConnectFail = false,
                ConnectTimeout = 1000,
                SyncTimeout = 1000,
                ConnectRetry = 5,
            };
            masterConfiguration.EndPoints.Add(masterEndPoint);
            _masterConnectionMultiplexer = ConnectionMultiplexer.Connect(masterConfiguration);

            var replicaEndPoint = server.SentinelGetReplicaAddresses("mymaster").AsEnumerable();
            var replicaConfiguration = new ConfigurationOptions()
            {
                AbortOnConnectFail = false,
                ConnectTimeout = 1000,
                SyncTimeout = 1000,
                ConnectRetry = 5,
            };

            replicaEndPoint.ToList().ForEach(x => masterConfiguration.EndPoints.Add(x));
            _replicaConnectionMultiplexer = ConnectionMultiplexer.Connect(masterConfiguration);

            Console.WriteLine($"MasterEndPoint: {masterEndPoint}, ReplicaEndPoint: {string.Join(',', replicaEndPoint)}");
        }
    }
}

專案啟動 => 連線哨兵 => 初始化 Master、Replica 連線

當發生故障轉移，訂閱 +switch-master 來切換 Master、Replica 連線
備註: 仍需補上 Retry 機制

方法二的做法個人覺得較佳，由套件自動識別就不用那麼麻煩

方法二 由 StackExchange.Redis 自動識別 Master、Replica

透過 StackExchange.Redis 設定 ConfigurationOptions 的 Endpoint 指定 Master、Replica
Github 文件提到套件會自動識別 Master
StackExchange.Redis Github 文件 - Basic Usage

var configuration = new ConfigurationOptions()
{
    EndPoints = { "10.140.0.20:6379", "10.140.0.21:6379", "10.140.0.22:6379" },
    AbortOnConnectFail = false,
    ConnectTimeout = 10000,
    SyncTimeout = 3000,
    ConnectRetry = 5
};

因套件自動判斷了目前 Master，程式面要處理的事就較少
但仍有發現2個缺點

  1. 寫入會從 Master 寫入，但讀取仍然也只讀取 Master
  2. 當在故障轉移(failover 期間)，若程式面發送一個 Redis 操作命令過來，是會卡住等到 Timeout 才噴 Exception
    儘管在卡住期間 failover 已經完成，但仍然是向舊的 Master Endpoint 呼叫
    這點可以再透過 Retry 機制來處理，若 failover 完成，再次 Retry 後則可以自動對應到新的 Master

🔥🔥🔥 2023/08/19 補充
第一點可透過 CommandFlags 來指定要連線 Master 還是 Replica 來解決

實作方法二

用 Console 模擬，每秒更新、讀取 Key1 的值
ConfigurationOptions 的 Endpoint 設定 Master、Slave，當異常發生時，透過 Polly 套件來處理 Retry 機制

using Newtonsoft.Json;
using Polly;
using Polly.Retry;
using StackExchange.Redis;

namespace RedisTest
{
    public class RedisConnectionManager
    {
        private readonly object _lock = new object();
        private IConnectionMultiplexer _connectionMultiplexer;
        private readonly ConfigurationOptions _configurationOptions;
        private readonly RetryPolicy _retryPolicy;

        public RedisConnectionManager(ConfigurationOptions configurationOptions, RetryPolicy retryPolicy)
        {
            _configurationOptions = configurationOptions;
            _retryPolicy = retryPolicy;
        }

        public IConnectionMultiplexer GetConnection()
        {
            return _retryPolicy.Execute(() =>
            {
                if (_connectionMultiplexer == null || !_connectionMultiplexer.IsConnected)
                {
                    lock (_lock)
                    {
                        _connectionMultiplexer?.Close();
                        _connectionMultiplexer = ConnectionMultiplexer.Connect(_configurationOptions);
                    }
                }

                return _connectionMultiplexer;
            });
        }

        public T? Get<T>(string key)
        {
            return _retryPolicy.Execute(() =>
            {
                try
                {
                    var value = _connectionMultiplexer.GetDatabase().StringGet(key);

                    if (value.IsNullOrEmpty)
                        return default;

                    return JsonConvert.DeserializeObject<T>(value);
                }
                catch (Exception ex)
                {
                    Console.WriteLine($"Get 發生錯誤, Error:{ex.Message}");
                    throw ex;
                }
            });
        }

        public void Update(string key, string data)
        {
            _retryPolicy.Execute(() =>
            {
                try
                {
                    _connectionMultiplexer.GetDatabase().StringSet(key, data);
                    Console.WriteLine("已更新");
                }
                catch (Exception ex)
                {
                    Console.WriteLine($"Update 發生錯誤, Error:{ex.Message}");
                    throw ex;
                }
            });
        }
    }

    internal class Program
    {
        private static void Main(string[] args)
        {
            try
            {
                var retryPolicy = Policy.Handle<RedisConnectionException>()
                                        .Or<RedisTimeoutException>()
                                        .Retry(3, (exception, retryCount) =>
                                        {
                                            Console.WriteLine($"Redis connection failed. Retrying ({retryCount})...");
                                        });

                var configuration = new ConfigurationOptions()
                {
                    EndPoints = { "10.140.0.20:6379", "10.140.0.21:6379", "10.140.0.22:6379" },
                    AbortOnConnectFail = false,
                    ConnectTimeout = 10000,
                    SyncTimeout = 3000,
                    ConnectRetry = 5
                };

                var redisConnectionManager = new RedisConnectionManager(configuration, retryPolicy);
                var redisConnection = redisConnectionManager.GetConnection();

                while (true)
                {
                    Console.WriteLine($"是否已連接: {redisConnection.IsConnected}");
                    var value = redisConnectionManager.Get<string>("Key1");
                    var newValue = Convert.ToInt32(value) + 1;

                    Console.WriteLine($"{DateTime.Now:yyyy-MM-dd HH:mm:ss.fff} 預計更新為 {newValue}");
                    redisConnectionManager.Update("Key1", newValue.ToString());
                    Console.WriteLine($"更新後確認 {redisConnectionManager.Get<string>("Key1")}\n");
                    Thread.Sleep(1000);
                }
            }
            catch (Exception ex)
            {
                Console.WriteLine($"{DateTime.Now:yyyy-MM-dd HH:mm:ss.fff} {ex.Message}");
            }

            Console.ReadKey();
        }
    }
}

當異常發生Redis Sentinel 做故障轉移期間，程式端雖然該次操作會收到 Exception
但當故障轉移結束，套件自動判斷到新 Master 後，即可正常寫入

參考文件

Yowko’s Notes - 如何使用 StackExchange.Redis 配合 Sentinel 或是 Cluster 達到高可用性
軟體主廚的程式料理廚房 - [料理佳餚] C# 在 Redis 發生 Failover 時自動跟著執行 HA 切換
StackOverFlow - Redis failover with StackExchange / Sentinel from C#

轉載請註明來源，若有任何錯誤或表達不清楚的地方，歡迎在下方評論區留言，也可以來信至 leozheng0621@gmail.com
如果文章對您有幫助，歡迎斗內(donate)，請我喝杯咖啡

斗內💰

©2022-2022 LeoZheng

Built with Hexo and 3-hexo theme

×

歡迎斗內

github