Batch ETL

Applies to 1.0.0. Tasks → New/Edit → plugin BATCH_ETL.

Before you start

Use ADMIN to create, configure and save tasks. ADMIN or OPERATOR can execute published definitions; ADMIN reads detailed logs and input/output variables. Resources must be enabled and reachable from the execution node. See tasks for common timeout, retry and version settings, and the permission matrix.

Field reference

“Default” means the initial value of a new 1.0.0 form. An empty value means the field has not been filled in; required fields still need a value. Visibility follows the selected branch; conditional requirements apply only to that branch.

Tables and write settings

• Source database: Provide the source database connection for batch reads.
• Target database: Provide the target database connection for batch writes.
• Source schema (optional): Select the schema containing source tables.
• Target schema (optional): Select the schema containing target tables.
• Tables: List tables to synchronize without duplicates.
• Logical keys: Match target records and break incremental ties; every table must contain these columns.
• Extraction mode: Choose full or incremental extraction.
• Write mode: Choose append, insert, update or upsert behavior for target tables.
• Null policy: Choose whether source nulls overwrite or preserve existing target values.
• Target prefix (optional): Prefix added when deriving target table names.
• Target suffix (optional): Suffix added when deriving target table names.
• Marker column: Column used to track incremental progress in every table.
• Marker type: Type used to interpret and compare incremental markers.
• Initial marker value: Starting marker used when no checkpoint exists.
• Initial logical key values: Initial tie-break values in the same order as logical keys.
• Clear before import: Clear target tables before a full import.
• Clear mode: Choose DELETE or TRUNCATE for clearing tables.
• I understand: deletion and batch commits can leave partial results: Acknowledge that deletion followed by batch commits can leave partial results on failure.
• I understand: TRUNCATE may commit implicitly and cannot be fully rolled back: Acknowledge that TRUNCATE may commit implicitly and prevent complete rollback.
• I understand: automatic retries with append can duplicate data: Explicitly accept possible duplicate writes when retrying append operations.

Execution limits

• Fetch batch rows: 1–100000。
• Write batch rows: 1–100000。
• Query timeout (seconds): 1–86400。
• Maximum tables: 1–500。
• Maximum rows per table: 1–9223372036854775807。
• Maximum total rows: 1–9223372036854775807。

Behavior and boundaries

• At most 500 tables; table and logical-key lists cannot contain duplicates.
• Incremental mode forbids APPEND; every table must contain the configured watermark column and logical keys.
• Tables have independent commit boundaries; a failure does not imply rollback of tables already committed.

Typical configuration: Synchronize orders and order_items

Fill in the form below, replacing resource names with your enabled resources. This is a configuration guide, not importable JSON. Follow the tutorial linked below for data preparation and execution.

Prepare three rows each in source orders and order_items; create target copy_orders and copy_order_items with id primary keys.

Expected result: copy_orders has three rows totaling 600; copy_order_items has three rows with qty totaling 6. Keys and totals remain unchanged after a repeat run.

Follow the full tutorial. Save the definition before running it; inspect execution state and business results separately.

Configuration keys

Use these keys when inspecting imported configuration, API payloads or errors. [] denotes a collection item; the form does not require you to enter key names.

Related pages

Tasks and versions · Parameters and expressions · Scenario tutorials