Skip to content

distributed incremental subscription & Consumption based on MySQL database binlog

License

Notifications You must be signed in to change notification settings

jilieryuyi/wing-binlog-go

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

wing-binlog-go

wing-binlog-go is a mysql(mariadb) master-slave replication client, that uses replication protocol to read events from Mysql or MariaDB's binlog build on top of go-mysql, and send json data via Http, TCP、 Redis queue、kafka or subscribe. It allows you to receive events like insert, update and delete with the original data.

  • wing-binlog-go is a high performance Mysql/MariaDB middleware
  • wing-binlog-go is a light weight Mysql/MariaDB data watch system

wing-binlog-go can run on OSX, Linux and Windows system.

Installation:

  • On Xnix, run ./bin/build.sh
  • On Windows, run bin/build.bat

Project status The project is test with:

  • Mysql 5.5, 5.6 and 5.7
  • MariaDB 5.5, 10.0, 10.1 and 10.2
  • golang version 1.8+
  • consul v1.0.2

Important Mysql server settings

[mysqld]
expire_logs_days         = 10
max_binlog_size          = 100M
server-id		 = 1
log_bin			 = /var/log/mysql/mysql-bin.log
binlog-format            = row

The system variables server_id, log_bin and binlog-format are madantory, and row based replication is required for the project.

Features:

  • TCP protocol support
  • Http protocal support
  • Support client group, broadcast and filters

Use cases:

  • MySQL to NoSQL replication
  • MySQL to search engine replication, such as elasticsearch
  • Realtime analytics
  • Audit

Usage:

  • Help: ./wing-binlog-go -help
  • Start service: ./wing-binlog-go
  • Show version information: ./wing-binlog-go -version
  • Stop service: ./wing-binlog-go -stop
  • Reload service: ./wing-binlog-go -reload all|http|tcp

Known issues:

  1. The underlying package go-mysql does not support compressed binlog
  2. Schema changes after initial binlog position may cause incorrect json package

Special thanks:

Contributors

  • jilieryuyi
  • mia0x75

Help:

QQ group: 535218312 document