Tunnel
| Function | Description |
|---|---|
Incremental Data Sync | Sync of common DML like INSERT, UPDATE, DELETE is supported. |
Subscription Modification | Add, delete, or modify the subscribed tables with support for historical data migration. For more information, see Modify Subscription. |
Table Name Mapping | Support the mapping rules, namely, keeping the name the same as that in Source, converting the text to lowercase, converting the text to uppercase, truncating the name by "_digit" suffix. |
Metadata Retrieval | Retrieve the target metadata with filtering conditions from the source table. |
Advanced Functions
| Function | Description |
|---|---|
Removal of Target Data before Full Data Migration | Remove the existing data in the Target before running the Full Data Migration, applicable for DataJobs reruning and scheduled Full Data migrations. |
Recreating Target Table | Recreate target tables before running the Full Data Migration, applicable for DataJobs reruning and scheduled Full Data migrations. |
Incremental Data Write Conflict Resolution Rule | IGNORE: Ignore primary key conflicts (skip writing), REPLACE: Replace the entire row in case of primary key conflicts. |
Handling of Zero Value for Time | Allow setting zero value for time to different data types to prevent errors when writing to the Target. |
Custom Code | For more information, see Custom Code Processing, Debug Custom Code and Logging in Custom Code. |
Data Filtering Conditions | Support data filtering using WHERE conditions, with SQL-92 as the SQL language. For more information, see Data Filtering. |
Prerequisites
| Prerequisite | Description |
|---|---|
Port Preparation | For CloudCanal (Docker version), modify the docker-compose.yml port mappings before installing/upgrading after extraction, and open relevant ECS security group ports for remote connection. |
Parameters
| Parameter | Description |
|---|---|
httpMaxContentSize | The maximum content size when Tunnel uses HTTP to transmit data. Currently it is used in Netty‘s HttpObjectAggregator. |
eventStoreSize | Size of the queue in the memory (it might be event count or memory size, in bytes). |
protocol | Protocol for data transfer in Tunnel. Support HTTP currently. |
uriPrefix | URI prefix for Tunnel data sources. |
contentCompressType | Data compression form for Tunnel data sources. Support DeInflater / GZIP currently. |
dbs | Definition of the data source schema. |
printDataInLog | Enable to print key data to logs for troubleshooting. Enabling this feature may impact performance and data security. |
schemaFormat | Type of Value Schema. |
printCustomCodeDebugLog | Print custom code DEBUG logs (Note that it will print data before and after processing, which may increase disk usage and impact sync performance). This parameter takes effect dynamically and DataJob restart is not required. |
Tips: To modify the general parameters, see General Parameters and Functions.
Prerequisites
| Prerequisite | Description |
|---|---|
Permissions for Account | |
Port Preparation | Allow the migration and sync node (Worker) to connect to the MySQL/MariaDB port (e.g., 3306). |
Parameters
| Parameter | Description |
|---|---|
keyConflictStrategy | Strategy for handling primary key conflicts during write in Incremental DataTask:
|
dstWholeReplace | Convert INSERT and UPDATE operations into full row replacement in the Target. |
deCycle | Enable filtering in bidirectional sync to filter DML/DDL with specific markers. |
specialSqlMode | Set a specific SQL mode when initializing the connection between databases. |
defaultGisSRID | Set the SRID for GIS data types. |
dstTimeZone | Target time zone, e.g., +08:00, Asia/Shanghai, America/New_York, etc. |
increParallelApplyStrategy | Parallel write strategy for relational databases in the Target:
|
Tips: To modify the general parameters, see General Parameters and Functions.