Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.wirekite.io/llms.txt

Use this file to discover all available pages before exploring further.

​
Orchestrator

Wirekite has a main binary called the orchestrator. The actual binary is called wirekite. This is a top level binary which calls all the other binaries to execute the task of data movement. Note that other binaries can be called independently also for testing purposes.

​
Orchestrator Configuration File

Orchestrator takes a configuration file to understand what needs to be done. The configuration file has the following rules.
  1. The configuration variables can be represented using the format variable=value.
  2. Empty lines are allowed.
  3. Comments are allowed. To comment just start the line with #.
  4. Multilevel variables are allowed using the format parent.child=value.

​
Variables

The configuration file has the following main variables and allowed values.
​
home
string
home is the Wirekite installation directory.
​
source
string
source is the database type from which we are extracting. The possible values are
  1. mysql
  2. oracle
  3. postgres
  4. sqlserver
  5. spanner (CDC only)
​
target
string
target is the database type to which we are loading. The possible values are
  1. snowflake
  2. firebolt
  3. databricks
  4. bigquery
  5. spanner
  6. singlestore
  7. mysql
  8. postgres
  9. oracle
  10. sqlserver
  11. mongodb
​
log
string
log is the path to the orchestrator log file.
​
resume
boolean
default:"false"
When set to true, the orchestrator preserves the existing position.pkt file and resumes from the last saved position. When false (or absent), the orchestrator truncates the position file before starting. The Web Interface sets this automatically when resuming a migration.

​
Modes

The mode of operation is declared explicitly using the mode variable in the configuration file.
​
mode
string
required
The operation mode. Must be one of the values listed below.

​
Schema Modes

ModeDescription
schema-extractExtract the schema from the source database. Produces a .skt schema file and optionally an .okt objects file.
schema-apply-tablesApply CREATE TABLE statements to the target database using the generated SQL file.
schema-apply-objectsApply non-table objects (indexes, constraints, foreign keys) to the target database. Errors on individual statements are logged but do not stop execution.
Schema modes must run independently and cannot be combined with data or change modes. The schema loader generates SQL files for the target database. These SQL files are not automatically executed on the target database during extraction — this is done in separate apply steps so you can make granular changes to the schema properties on the target database.

​
Data and Change Modes

ModeDescription
dataOne-time bulk data extraction and loading. The orchestrator runs the extractor, mover, and loader in parallel.
changeContinuous CDC replication. Extracts ongoing changes (inserts, updates, deletes) from the source and applies them to the target.
data+changeCombined data migration followed by CDC replication. The orchestrator first completes the bulk data load, captures the source position automatically, and then starts continuous change replication from that position. Do not specify a starting position for the change source — the orchestrator captures this during the data phase.
In data and change modes, Wirekite automatically creates two metadata tables in the target database: wirekite_progress for tracking migration state and crash recovery, and wirekite_action for pause/stop/resume control. See Operations for details on monitoring and controlling running migrations.

​
Reset Modes

ModeDescription
reset-dropDrop tables from the target database using the specified drop file.
reset-truncTruncate tables on the target database.
reset-metadataReset Wirekite’s internal metadata (progress and action tables) without modifying target data.

​
Validation Mode

ModeDescription
validateCompare data between source and target databases to verify migration accuracy. See Data Validation for details.

​
Source and Target Child Variables

Both source and target variables have child variables that include the mode as part of the hierarchy. The format is source.<mode>.<variable>=value and target.<mode>.<variable>=value. For example, source.data.dsnFile is the connection string file for the data extractor and target.schema.schemaFile is the input schema file for the schema loader. The specific variables available depend on the source and target database types. For example if the source is mysql then the child variables can be any of the variables listed in mysql variables. If the target is snowflake then the child variables can be any of the variables listed in snowflake variables.

​
Mover Variables

The mover is an optional component that transfers files between the source and target. It is configured separately using the mover.<variable>=value format. The mover does not have a mode prefix since the same mover handles both data and change files.

​
Bringing It All Together

Below are example orchestrator configuration files for each mode.

​
Schema Mode Example

Schema mode extracts the database schema from the source and generates SQL files for the target. Schema extraction and application are separate modes. 
# Schema extraction configuration

# main
home=/opt/wirekite
source=mysql
target=snowflake
mode=schema-extract
log=/var/log/wirekite/schema-migration.log

# schema source
source.schema.tablesFile=/opt/wirekite/config/tables.txt
source.schema.outputDirectory=/opt/wirekite/output/schema
source.schema.dsnFile=/opt/wirekite/config/mysql-source.dsn
source.schema.logFile=/var/log/wirekite/schema-extractor.log

# schema target
target.schema.schemaFile=/opt/wirekite/output/schema/wirekite_schema.skt
target.schema.createTableFile=/opt/wirekite/output/sql/create-tables.sql
target.schema.createConstraintFile=/opt/wirekite/output/sql/create-constraints.sql
target.schema.createForeignKeyFile=/opt/wirekite/output/sql/create-foreign-keys.sql
target.schema.dropTableFile=/opt/wirekite/output/sql/drop-tables.sql
target.schema.logFile=/var/log/wirekite/schema-loader.log
target.schema.createMergeTables=true
To apply the extracted schema tables to the target, use mode=schema-apply-tables with the generated SQL file. Then use mode=schema-apply-objects to apply indexes, constraints, and foreign keys.

​
Data Mode Example

Data mode performs a one-time bulk data transfer from source to target. 
# Data migration configuration

# main
home=/opt/wirekite
source=oracle
target=snowflake
mode=data
log=/var/log/wirekite/data-migration.log

# data source
source.data.dsnFile=/opt/wirekite/config/oracle-source.dsn
source.data.tablesFile=/opt/wirekite/config/tables.txt
source.data.outputDirectory=/opt/wirekite/data/extract
source.data.logFile=/var/log/wirekite/data-extractor.log
source.data.maxThreads=8
source.data.maxRowsPerDump=50000

# data target
target.data.dsnFile=/opt/wirekite/config/snowflake-target.dsn
target.data.dataDirectory=/opt/wirekite/data/stage
target.data.doneDirectory=/opt/wirekite/data/done
target.data.logFile=/var/log/wirekite/data-loader.log
target.data.schemaFile=/opt/wirekite/output/schema/wirekite_schema.skt
target.data.dataFileExtension=dkt
target.data.maxThreads=4

# mover
mover.awsRegion=us-east-1
mover.awsBucket=wirekite-migration-bucket
mover.dataDirectory=/opt/wirekite/data/extract
mover.logFile=/var/log/wirekite/data-mover.log
mover.maxThreads=4
mover.gzipFiles=false
mover.removeFiles=false

​
Change Mode Example

Change mode captures ongoing changes (CDC) and applies them to the target. When running change mode independently you must specify the starting position for your source database type. 
# Change replication configuration

# main
home=/opt/wirekite
source=mysql
target=postgres
mode=change
log=/var/log/wirekite/change-replication.log

# change source
source.change.dsnFile=/opt/wirekite/config/mysql-source.dsn
source.change.tablesFile=/opt/wirekite/config/tables.txt
source.change.outputDirectory=/opt/wirekite/changes/extract
source.change.logFile=/var/log/wirekite/change-extractor.log
source.change.binlogFile=mysql-bin.000042
source.change.binlogPosition=154
source.change.exitWhenIdle=false

# change target
target.change.dsnFile=/opt/wirekite/config/postgres-target.dsn
target.change.dataDirectory=/opt/wirekite/changes/stage
target.change.workDirectory=/opt/wirekite/changes/work
target.change.logFile=/var/log/wirekite/change-loader.log
target.change.schemaFile=/opt/wirekite/output/schema/wirekite_schema.skt
target.change.changeFileExtension=ckt
target.change.doneDirectory=/opt/wirekite/changes/done
target.change.removeFiles=true
target.change.maxFilesPerBatch=30

​
Data + Change Mode Example

Data and change modes can be combined using mode=data+change. The orchestrator will first complete the bulk data load, capture the source position automatically, and then start continuous change replication. Do not specify a starting position for the change source — the orchestrator captures this automatically during the data phase. 
# Full migration configuration (data + change)

# main
home=/opt/wirekite
source=sqlserver
target=snowflake
mode=data+change
log=/var/log/wirekite/full-migration.log

# data source
source.data.dsnFile=/opt/wirekite/config/sqlserver-source.dsn
source.data.tablesFile=/opt/wirekite/config/tables.txt
source.data.outputDirectory=/opt/wirekite/data/extract
source.data.logFile=/var/log/wirekite/data-extractor.log
source.data.maxThreads=8
source.data.maxRowsPerDump=100000

# data target
target.data.dsnFile=/opt/wirekite/config/snowflake-target.dsn
target.data.dataDirectory=/opt/wirekite/data/stage
target.data.doneDirectory=/opt/wirekite/data/done
target.data.logFile=/var/log/wirekite/data-loader.log
target.data.schemaFile=/opt/wirekite/output/schema/wirekite_schema.skt
target.data.dataFileExtension=dkt
target.data.maxThreads=6

# change source
source.change.dsnFile=/opt/wirekite/config/sqlserver-source.dsn
source.change.tablesFile=/opt/wirekite/config/tables.txt
source.change.outputDirectory=/opt/wirekite/changes/extract
source.change.logFile=/var/log/wirekite/change-extractor.log
source.change.exitWhenIdle=false

# change target
target.change.dsnFile=/opt/wirekite/config/snowflake-target.dsn
target.change.dataDirectory=/opt/wirekite/changes/stage
target.change.workDirectory=/opt/wirekite/changes/work
target.change.logFile=/var/log/wirekite/change-loader.log
target.change.schemaFile=/opt/wirekite/output/schema/wirekite_schema.skt
target.change.changeFileExtension=ckt
target.change.doneDirectory=/opt/wirekite/changes/done
target.change.removeFiles=true
target.change.maxFilesPerBatch=30

# mover
mover.awsRegion=us-west-2
mover.awsBucket=wirekite-migration-bucket
mover.dataDirectory=/opt/wirekite/data/extract
mover.logFile=/var/log/wirekite/mover.log
mover.maxThreads=4
mover.gzipFiles=false
mover.removeFiles=false