StarRocks
| Function | Description |
|---|---|
Schema Migration | If the target schema does not exist, BladePipe will automatically generate and execute CREATE statements based on the source metadata and the mapping rule. |
Full Data Migration | Migrate data by sequentially scanning data in tables and writing it in batches to the target database. |
Subscription Modification | Add, delete, or modify the subscribed tables with support for historical data migration. For more information, see Modify Subscription. |
Table Name Mapping | Support the mapping rules, namely, keeping the name the same as that in Source, converting the text to lowercase, converting the text to uppercase, truncating the name by "_digit" suffix. |
Advanced Functions
| Function | Description |
|---|---|
Removal of Target Data before Full Data Migration | Remove the existing data in the Target before running the Full Data Migration, applicable for DataJobs reruning and scheduled Full Data migrations. |
Recreating Target Table | Recreate target tables before running the Full Data Migration, applicable for DataJobs reruning and scheduled Full Data migrations. |
Incremental Data Write Conflict Resolution Rule | IGNORE: Ignore primary key conflicts (skip writing), REPLACE: Replace the entire row in case of primary key conflicts. |
Handling of Zero Value for Time | Allow setting zero value for time to different data types to prevent errors when writing to the Target. |
Custom Code | For more information, see Custom Code Processing, Debug Custom Code and Logging in Custom Code. |
Data Filtering Conditions | Support data filtering using WHERE conditions, with SQL-92 as the SQL language. For more information, see Data Filtering. |
Setting Target Primary Key | Change the primary key to another field to facilitate data aggregation and other operations. |
Prerequisites
| Prerequisite | Description |
|---|---|
Permissions for Account | SELECT permission |
Port Preparation | Allow the migration and sync node (Worker) to connect to the StarRocks FE QueryPort. |
Parameters
| Parameter | Description |
|---|---|
fullBatchSize | Number of records per batch written to the Target in full data migration, significantly affecting migration performance. |
fullPagingCount | Page size for scanning data in a source relational database. |
scanParallel | Number of threads for parallel data scanning in the source database. If the data source is a relational database, it refers to the number of tables scanned in parallel. |
minPoolSize | Minimum connection pool size for the data source. |
maxPoolSize | Maximum connection pool size for the data source. |
maxWait | Maximum wait time for data source connections (in milliseconds). |
soTimeoutSec | TCP socket timeout duration for data source connections. |
Tips: To modify the general parameters, see General Parameters and Functions.
Prerequisites
| Prerequisite | Description |
|---|---|
Permissions for Account | |
Port Preparation | Allow the migration and sync node (Worker) to connect to the MySQL/MariaDB port (e.g., 3306). |
Parameters
| Parameter | Description |
|---|---|
keyConflictStrategy | Strategy for handling primary key conflicts during write in Incremental DataTask:
|
dstWholeReplace | Convert INSERT and UPDATE operations into full row replacement in the Target. |
deCycle | Enable filtering in bidirectional sync to filter DML/DDL with specific markers. |
specialSqlMode | Set a specific SQL mode when initializing the connection between databases. |
defaultGisSRID | Set the SRID for GIS data types. |
dstTimeZone | Target time zone, e.g., +08:00, Asia/Shanghai, America/New_York, etc. |
increParallelApplyStrategy | Parallel write strategy for relational databases in the Target:
|
Tips: To modify the general parameters, see General Parameters and Functions.