Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Support increment sync #744

Open
wants to merge 1 commit into
base: v4
Choose a base branch
from

Conversation

jjz921024
Copy link
Contributor

@jjz921024 jjz921024 commented Dec 29, 2023

for issue #722

通过sync_reader同步时,支持增量同步,避免每次都进行不必要的全量同步
以允许在shake短暂中断或重启后,可以从上次同步的offset处继续进行

  1. 在连接源redis实例时,通过info replication命令获取master_replid字段
  2. 修改保存aof的文件名,以接收到的起始offset作为文件名。在重启时可通过 文件名 + 文件大小 的方式得到上次同步的offset
  3. 若能拿到上次同步的offset 且 sync_rdb=false,尝试进行增量同步,替换 PSYNC ? -1

@Zheaoli
Copy link
Collaborator

Zheaoli commented Jan 2, 2024

这里我考虑的一点是将文件名作为 aof offset 的存放位置好像不太合适,我们这里需要一个 metadata 的抽象 @suxb201 WYDT

@suxb201
Copy link
Member

suxb201 commented Jan 2, 2024

@Zheaoli 是的,现在的 aof file 是我之前随便写的,看 issue 反馈来看貌似有 bug,但我没能复现出来。之前也考虑换个成熟的 disk queue,但是没发现好用的库。

@jjz921024
Copy link
Contributor Author

@Zheaoli 在这个pr中,原本也实现过在reader目录下增加一个保存元数据的文件。但后面考虑仅就增量同步这个功能,目前需要保存的信息只有repl_idrepl_offset,想着repl_id可以添加到目录上作为区分(目前目录上仅ip和端口作为区分应该不保险);repl_offset记录在aof文件名上(目前aof文件名仅是一个递增的序号,应该没有具体含义?),这种实现有点类似kafka记录msg的方式base_offset + size 能定位到具体某条消息

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants