Skip to main content

MySQL to Kafka Sync

Overview​

This article briefly introduces how BladePipe synchronizes data from MySQL to Kafka, using the BladePipe Json Format for Kafka.

Features include:

  • Supports Multiple message formats
  • Supports DDL synchronization (with customizable DDL Topics)
  • Supports automatic Topic creation

Key Points​

Automatic topic creation​

The Current DataJob supports automatic creation of Kafka topics and allows customization of the number of partitions, as follows:

image.png

Batch writing of data.​

Support for merging identical operations on the same table into a single message, enabling batch writing of data and reducing network bandwidth usage to increase data processing efficiency.

image.png

DataJob position reset.​

BladePipe supports position reset for FullData and Incremental, allowing DataJobs to automatically resume from the last position after a restart by periodically recording the position.

For large tables with billions of records, this ability is essential, and the data initialization position reset feature minimizes the impact of such pauses on progress.

Example​

Install BladePipe​

Data Preparation​

  • Create Insert, Update, Delete loads, the ratio is 1:1:1 image.png

Add DataSource​

  • Log in to the BladePipe console, DataSource > Add DataSource , add MySQL and Kafka DataSource. image.png

Create DataJob​

  • DataJob > Create DataJob

  • Select the source and target DataSources.

  • Click on the Advanced of Kafka to make sure the Message Format is BladePipe Json Format.

  • Click Next Step. image.png

  • Select Incremental, then check Full Data option.

  • Check Synchronize DDL.

  • Click Next Step image.png

  • Select tables and columns, and click Next Step.

  • Finally,click Create DataJob.

DataJob Running​

  • Wait for automatic Schema Migration, Full Data, and Incremental to catch up. image.png

FAQ​

What other source data sources does it support?​

Currently, MySQL, Oracle, SQL Server, Postgres, and MongoDB are open to Kafka. If you have any other requirements, please provide us with feedback in the community.

Summary​

This article briefly introduces BladePipe support MySQL to Kafka data synchronization, which can help businesses quickly implement real-time data processing and analysis.