TiDB
| Function | Description |
|---|---|
Schema Migration | If the target schema does not exist, BladePipe will automatically generate and execute CREATE statements based on the source metadata and the mapping rule. |
Full Data Migration | Migrate data by sequentially scanning data in tables and writing it in batches to the target database. |
Incremental Data Sync | Sync of common DML like INSERT, UPDATE, DELETE is supported. |
Data Verification and Correction | Verify all existing data. Optionally, you can correct the inconsistent data based on verification results. Scheduled DataTasks are supported. |
Subscription Modification | Add, delete, or modify the subscribed tables with support for historical data migration. For more information, see Modify Subscription. |
Position Resetting | Reset positions by timestamp to consume again the incremental data that has not been collected as garbage by TiKV in a past period. |
Table Name Mapping | Support the mapping rules, namely, keeping the name the same as that in Source, converting the text to lowercase, converting the text to uppercase, truncating the name by "_digit" suffix. |
DDL Sync |
|
Metadata Retrieval | Retrieve the target metadata with filtering conditions from the source table. |
Advanced Functions
| Function | Description |
|---|---|
Removal of Target Data before Full Data Migration | Remove the existing data in the Target before running the Full Data Migration, applicable for DataJobs reruning and scheduled Full Data migrations. |
Recreating Target Table | Recreate target tables before running the Full Data Migration, applicable for DataJobs reruning and scheduled Full Data migrations. |
Incremental Data Write Conflict Resolution Rule | IGNORE: Ignore primary key conflicts (skip writing), REPLACE: Replace the entire row in case of primary key conflicts. |
Handling of Zero Value for Time | Allow setting zero value for time to different data types to prevent errors when writing to the Target. |
Scheduled Full Data Migration | For more information, see Create Scheduled Full Data DataJob. |
Custom Code | For more information, see Custom Code Processing, Debug Custom Code and Logging in Custom Code. |
Data Filtering Conditions | Support data filtering using WHERE conditions, with SQL-92 as the SQL language. For more information, see Data Filtering. |
Prerequisites
| Prerequisite | Description |
|---|---|
Permissions for Account | |
Connection to PD Nodes | Make sure that BladePipe Workers can communicate with PD nodes.
|
TiKV GC Frequency | Set GC cycle to 24 hours or more in TiDB Server.
|
TiKV Historical Data Caching | Adjust the size based on task needs.
|
Parameters
| Parameter | Description |
|---|---|
printDetailLog | Print received incremental data. It is used for determining if the source database has incremental data. |
pdHost | PD node address for DataJob requests. Format: [PD_IP]:[PD_PORT], multiple PD nodes separated by , |
cdcGrpcTimeout | Timeout for gRPC channel of PD nodes to DataJob, in ms. |
cdcStubTimeout | Timeout for each stub in gRPC channel, in ms. Auto-resubscribe the stub in case of time out. |
Tips: To modify the general parameters, see General Parameters and Functions.
Prerequisites
| Prerequisite | Description |
|---|---|
Permissions for Account | INSERT, UPDATE, DELETE, and DDL permissions. |
Port Preparation | Allow the migration and sync node (Worker) to connect to the TiDB port (e.g., port 4000). |
Parameters
| Parameter | Description |
|---|---|
keyConflictStrategy | Strategy for handling primary key conflicts during write in Incremental DataTask:
|
dstWholeReplace | Convert INSERT and UPDATE operations into full row replacement in the Target. |
writeStrategy | Strategy of writing data to the Target, including:
|
Tips: To modify the general parameters, see General Parameters and Functions.