Redis AOF 備份測試

NoSQL > Redis Redis
發布時間 :
字數:2.4k 觀看數 : 評論:
  1. 前言
  2. 環境
  3. AOF 檔備份策略
    1. Redis 服務建置、測試 AOF 檔案是否寫入
  4. 災難異常復原模擬
  5. AOF 檔毀損損
  6. AOF rewrite
    1. 手動執行
    2. 自動執行
    3. 測試 fileSize 超過 200 byte，AOF 是否會自動 rewrite (V)
    4. 測試只設定 auto-aof-rewrite-percentage 100
  7. 備註
  8. 參考文件

前言

透過 docker 建置 Redis，並測試 AOF 檔備份

環境

參考前一篇 w4560000 - Redis RDB 備份測試

AOF 檔備份策略

會將執行的命令寫到 AOF 檔
Redis 服務有一個 aof_buf 緩衝區，命令執行後會加到 aof_buf 中，再寫入 AOF 檔

  • appendonly <yes/no>

    是否開啟 AOF
    ex: appendonly yes/no

  • appendfilename

    AOF 檔名
    ex: appendfilename “appendonly.aof”

  • appendfsync <no/everysec/always>

    fsync 策略 (將 aof_buf 資料同步至 AOF 檔中)
    ex: appendfsync everysec

appendfsync 參數 說明
no 不主動呼叫 fsync，由作業系統控制，Linux 是每30秒1次，不太安全
因不確定同步時間，最多會遺失30秒內的操作資料
everysec 每秒同步，最多則遺失1秒內的操作資料
always 每次執行的命令都會直接同步，性能較差，但最安全

Redis 服務建置、測試 AOF 檔案是否寫入

新增 redis.conf (關閉 RDB、開啟 AOF)

vi ./redis/redis.conf

bind 127.0.0.1 172.18.0.2
protected-mode no
dir ./

# AOF
appendonly yes
appendfilename "appendonly.aof"
appendfsync everysec

建立 redis 容器

docker run --name redis1 \
--network=redis_network \
--ip 172.18.0.2 \
-p 6379:6379 \
-v /home/leozheng0411/redis:/usr/local/etc/redis \
-v /home/leozheng0411/redis1data:/data \
-d redis:6.0.0 \
redis-server /usr/local/etc/redis/redis.conf

測試一下服務是否正常

docker exec -it redis1 redis-cli ping

# 輸出
PONG

目前 redis.conf 是設定每秒會把操作命令同步到 AOF 檔中

更新資料

docker exec -it redis1 redis-cli SET Key1 1

# 輸出
OK

docker exec -it redis1 redis-cli SET Key1 2

# 輸出
OK

確認 appendonly.aof 檔案

cat appendonly.aof

# 輸出
*2
$6
SELECT
$1
0
*3
$3
SET
$4
Key1
$1
1
*3
$3
SET
$4
Key1
$1
2

可以發現 AOF 檔案中的內容是可閱讀的
todo 但實測發現 appendfsync 不管是設 no or everysec，仍然是每筆操作都馬上寫入 (待查)

災難異常復原模擬

# 強制關閉服務
docker rm -f redis1

# 重啟服務
docker run --name redis1 \
--network=redis_network \
--ip 172.18.0.2 \
-p 6379:6379 \
-v /home/leozheng0411/redis:/usr/local/etc/redis \
-v /home/leozheng0411/redis1data:/data \
-d redis:6.0.0 \
redis-server /usr/local/etc/redis/redis.conf

# 查看 Log 確認啟動時有載入 AOF 檔
docker logs redis1

# 輸出
1:C 16 Jun 2023 10:23:48.441 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 16 Jun 2023 10:23:48.441 # Redis version=6.0.0, bits=64, commit=00000000, modified=0, pid=1, just started
1:C 16 Jun 2023 10:23:48.441 # Configuration loaded
1:M 16 Jun 2023 10:23:48.445 * Running mode=standalone, port=6379.
1:M 16 Jun 2023 10:23:48.445 # Server initialized
1:M 16 Jun 2023 10:23:48.445 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
1:M 16 Jun 2023 10:23:48.445 * DB loaded from append only file: 0.000 seconds
1:M 16 Jun 2023 10:23:48.445 * Ready to accept connections

# 確認資料正常復原
docker exec -it redis1 redis-cli GET Key1

# 輸出
"2"

AOF 檔毀損損

# 安裝 Redis Tool
apt install redis-tools

# 檢查 AOF
redis-check-aof appendonly.aof

# 輸出
0x               0: Expected \r\n, got: 0a00
AOF analyzed: size=64, ok_up_to=0, diff=64
AOF is not valid. Use the --fix option to try fixing it.

修復 AOF 檔，會問你是否繼續
因為事實上並不是修復，而是把有問題的結尾之後全部刪除，若有問題的地方在第一行，也等於全部清掉，要小心🔥🔥🔥

redis-check-aof --fix appendonly.aof

# 輸出
0x               0: Expected \r\n, got: 0a00
AOF analyzed: size=64, ok_up_to=0, diff=64
This will shrink the AOF from 64 bytes, with 64 bytes, to 0 bytes
Continue? [y/N]: Y
Successfully truncated AOF

AOF rewrite

因 AOF 檔會不斷寫入操作命令，會越來越肥大，redis 有提供 rewrite 機制，幫 AOF 檔瘦身
假設有固定排程，每分鐘不斷寫入 SET Key1 123
此時 AOF 檔則會充斥著很多 SET Key1 123 的操作命令，而 rewrite 後，則會精簡到剩一筆 SET Key1 123，減少檔案大小

AOF rewrite 也有分手動執行以及自動執行

手動執行

redis-cli bgrewriteaof

依序執行

docker exec -it redis1 redis-cli SET Key1 1

# 輸出
OK

docker exec -it redis1 redis-cli SET Key1 2

# 輸出
OK

docker exec -it redis1 redis-cli SET Key1 3

# 輸出
OK

此時確認 appendonly.aof

cat appendonly.aof

# 輸出
*2
$6
SELECT
$1
0
*3
$3
SET
$4
Key1
$1
1
*3
$3
SET
$4
Key1
$1
2
*3
$3
SET
$4
Key1
$1
3

再手動執行 AOF write

docker exec -it redis1 redis-cli bgrewriteaof

# 輸出
Background append only file rewriting started

# 再次確認 appendonly.aof 檔案已被 rewrite
cat appendonly.aof

# 輸出
*2
$6
SELECT
$1
0
*3
$3
SET
$4
Key1
$1
3

自動執行

可調整 redis.conf 以下參數做調整

  • 當 aof rewrite 時 appendfsync 改成 No
    no-appendfsync-on-rewrite yes/no
no-appendfsync-on-rewrite 參數 說明
no 會阻塞 AOF 檔寫入，需等 rewrite 完成才能寫入
yes 開啟 appendfsync no

  • 當 AOF 檔案超過上次 rewrite 的 AOF 檔案大小的百分比 (設為 0 則停用自動寫入 AOF)

    auto-aof-rewrite-percentage 0~100
    ex: auto-aof-rewrite-percentage 100

  • 當 AOF 檔案多大時會 rewrite

    auto-aof-rewrite-min-size 200

  • 是否要讀取被截斷字尾的 AOF 檔

    aof-load-truncated yes/no

aof-load-truncated 參數 說明
no 服務器噴錯並且服務不啟動 (此時需修復 AOF 文件 redis-check-aof)
yes 載入截斷字尾的 AOF 檔，並且Redis伺服器會開始發出日誌以通知用戶該事件
  • 重寫 AOF 檔案時，Redis 能夠使用AOF文件中的RDB前導碼來加快重寫和恢復速度 (RDB、AOF 混合備份)

    aof-use-rdb-preamble yes/no

以此份 redis.conf 做自動 AOF rewrite 測試

當 AOF 檔超過上次 rewrite 的 100% 以及 當 AOF 檔 超過 200 byte 才 rewrite
auto-aof-rewrite-percentage 和 auto-aof-rewrite-min-size 兩個條件達成才會 rewrite

bind 172.18.0.2
protected-mode no
dir ./

# aof
appendonly yes
appendfilename "appendonly.aof"
appendfsync always

# aof rewrite
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 200
aof-load-truncated no
aof-use-rdb-preamble no

測試 fileSize 超過 200 byte，AOF 是否會自動 rewrite (V)

auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 200

docker exec -it redis1 redis-cli SET Key1 123
docker exec -it redis1 redis-cli SET Key1 123
docker exec -it redis1 redis-cli SET Key1 123
docker exec -it redis1 redis-cli SET Key1 123
docker exec -it redis1 redis-cli SET Key1 123

# 查看檔案大小
stat --format="%s" appendonly.aof

# 輸出
183

# 再次執行 SET Key 123
docker exec -it redis1 redis-cli SET Key1 123

# 查看檔案大小
stat --format="%s" appendonly.aof

# 輸出 (確認檔案大小是否有 rewrite)
55

# 查看 Redis Log
docker logs redis1

# 輸出
1:C 17 Jun 2023 09:01:22.275 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 17 Jun 2023 09:01:22.275 # Redis version=6.0.0, bits=64, commit=00000000, modified=0, pid=1, just started
1:C 17 Jun 2023 09:01:22.275 # Configuration loaded
1:M 17 Jun 2023 09:01:22.280 * Running mode=standalone, port=6379.
1:M 17 Jun 2023 09:01:22.281 # Server initialized
1:M 17 Jun 2023 09:01:22.281 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
1:M 17 Jun 2023 09:01:22.281 * Ready to accept connections
1:M 17 Jun 2023 09:06:28.442 * Starting automatic rewriting of AOF on 21400% growth
1:M 17 Jun 2023 09:06:28.443 * Background append only file rewriting started by pid 55
1:M 17 Jun 2023 09:06:28.469 * AOF rewrite child asks to stop sending diffs.
55:C 17 Jun 2023 09:06:28.469 * Parent agreed to stop sending diffs. Finalizing AOF...
55:C 17 Jun 2023 09:06:28.469 * Concatenating 0.00 MB of AOF diff received from parent.
55:C 17 Jun 2023 09:06:28.469 * SYNC append only file rewrite performed
55:C 17 Jun 2023 09:06:28.470 * AOF rewrite: 0 MB of memory used by copy-on-write
1:M 17 Jun 2023 09:06:28.543 * Background AOF rewrite terminated with success
1:M 17 Jun 2023 09:06:28.543 * Residual parent diff successfully flushed to the rewritten AOF (0.00 MB)
1:M 17 Jun 2023 09:06:28.543 * Background AOF rewrite finished successfully

測試只設定 auto-aof-rewrite-percentage 100

因 Redis 預設 auto-aof-rewrite-min-size 是 64mb，AOF 會在超過 64mb 時 rewrite

docker logs redis1

# 輸出
1:C 17 Jun 2023 09:19:18.145 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 17 Jun 2023 09:19:18.145 # Redis version=6.0.0, bits=64, commit=00000000, modified=0, pid=1, just started
1:C 17 Jun 2023 09:19:18.145 # Configuration loaded
1:M 17 Jun 2023 09:19:18.149 * Running mode=standalone, port=6379.
1:M 17 Jun 2023 09:19:18.149 # Server initialized
1:M 17 Jun 2023 09:19:18.149 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
1:M 17 Jun 2023 09:19:18.150 * Ready to accept connections
1:M 17 Jun 2023 09:56:21.308 * Starting automatic rewriting of AOF on 6711063300% growth
1:M 17 Jun 2023 09:56:21.316 * Background append only file rewriting started by pid 830
1:M 17 Jun 2023 09:56:22.394 * AOF rewrite child asks to stop sending diffs.
830:C 17 Jun 2023 09:56:22.394 * Parent agreed to stop sending diffs. Finalizing AOF...
830:C 17 Jun 2023 09:56:22.394 * Concatenating 0.05 MB of AOF diff received from parent.
830:C 17 Jun 2023 09:56:22.398 * SYNC append only file rewrite performed
830:C 17 Jun 2023 09:56:22.399 * AOF rewrite: 0 MB of memory used by copy-on-write
1:M 17 Jun 2023 09:56:22.419 * Background AOF rewrite terminated with success
1:M 17 Jun 2023 09:56:22.420 * Residual parent diff successfully flushed to the rewritten AOF (0.00 MB)
1:M 17 Jun 2023 09:56:22.420 * Background AOF rewrite finished successfully

備註

因 AOF rewrite 會造成線程阻塞，不能太頻繁的操作
可調整 auto-aof-rewrite-min-size 依照資料量來判斷設定大一點的數值來避免頻繁操作
ex: 3G or 5G

參考文件

稀土掘金 Jony_zhang - Redis 数据持久化(RDB/AOF/混合持久化)
知乎 程序员囧辉 - 面试必问的 Redis：RDB、AOF、混合持久化
CSDN 每天都要进步一点点 - redis持久化之aof

轉載請註明來源，若有任何錯誤或表達不清楚的地方，歡迎在下方評論區留言，也可以來信至 leozheng0621@gmail.com
如果文章對您有幫助，歡迎斗內(donate)，請我喝杯咖啡

斗內💰

©2022-2022 LeoZheng

Built with Hexo and 3-hexo theme

×

歡迎斗內

github