Skip to main content

​
Overview

Wirekite supports MySQL 5.x and above (including MariaDB and SingleStore) as a target database for:
  • Schema Loading - Create target tables from Wirekite’s intermediate schema format
  • Data Loading - Bulk load extracted data using LOAD DATA INFILE
  • Change Loading (CDC) - Apply ongoing changes using merge table approach
All references to MySQL in this guide also apply to MariaDB and SingleStore setups. SingleStore uses the MySQL wire protocol, so the configuration and parameters are identical. See the SingleStore Notes section below for DSN and version differences.

​
Prerequisites

Before configuring MySQL as a Wirekite target, ensure the following requirements are met:

​
Database Configuration

  1. Version: MySQL 5.x or above (or MariaDB equivalent)
  2. User Permissions: Create a Wirekite user with:
    • CREATE TABLE privilege
    • FILE privilege for LOAD DATA operations
    • INSERT, UPDATE, DELETE on target tables
    • Read/write access to Wirekite internal tables (wirekite_progress and wirekite_action)

​
CLI Tools

Use the mysql command line tool or Wirekite cmdline tool to interact with the database.
For cloud-managed MySQL (RDS, Cloud SQL), you’ll need to use databaseRemote=true (the default) to use LOAD DATA LOCAL INFILE.

​
Schema Loader

The Schema Loader reads Wirekite’s intermediate schema format (.skt file) and generates MySQL-appropriate DDL statements for creating target tables.
If you need to add storage directives or other schema elements, use the CREATE TABLE output as a basis. Don’t change column names, column order, or datatypes to incompatible ones.

​
Required Parameters

​
string
required
Path to the Wirekite schema file (.skt) generated by the Schema Extractor. Must be an absolute path.
​
string
required
Output file for CREATE TABLE statements. Includes both base tables and merge tables for CDC operations.
​
string
required
Output file for non-columnar constraints including indexes. Column-level constraints (NOT NULL) are in CREATE TABLE.
​
string
required
Output file for FOREIGN KEY definitions. Consider applying these after initial data load for better performance.
​
string
required
Absolute path to the log file for Schema Loader operations.

​
Optional Parameters

​
string
default:"none"
Output file for DROP TABLE IF EXISTS statements. Set to “none” to skip generation.
​
string
default:"none"
Output file for recovery table creation SQL. Set to “none” to skip.
​
boolean
default:"true"
When true, generates merge tables ($wkm suffix) for CDC operations. Set to false if only doing data loads.
​
boolean
default:"false"
When true, only creates merge tables, skipping base table creation.

​
Data Loader

The Data Loader reads Wirekite’s intermediate data format (.dkt files) and loads records into MySQL tables using LOAD DATA INFILE.

​
Required Parameters

​
string
required
Path to a file containing the MySQL connection string.
Connection string format:
Example:
​
string
required
Directory containing data files (.dkt) to load.
​
string
required
Path to the Wirekite schema file used by Schema Loader. Required for table structure information.
​
string
required
Absolute path to the log file for Data Loader operations.

​
Optional Parameters

​
integer
default:"5"
Maximum number of parallel threads for loading tables. We recommend setting this to the number of CPUs on the host.
​
boolean
default:"false"
Set to true if data was extracted using hex encoding instead of base64.
​
boolean
default:"true"
When true, uses LOAD DATA LOCAL INFILE. When false, uses LOAD DATA INFILE (requires file access from server).
​
boolean
default:"true"
When true, removes data files from inputDirectory after loading.

​
Change Loader

The Change Loader applies ongoing data changes (INSERT, UPDATE, DELETE) to MySQL tables using a merge approach with shadow tables.
The Change Loader uses merge tables to stage changes, then applies INSERT-SELECT, UPDATE, and DELETE operations to transfer changes to base tables.

​
Required Parameters

​
string
required
Path to a file containing the MySQL connection string.
Connection string format:
​
string
required
Directory containing change files (.ckt) from the Change Extractor.
​
string
required
Working directory for intermediate CSV files during merge operations.
​
string
required
Path to the Wirekite schema file for table structure information.
​
string
required
Absolute path to the log file for Change Loader operations.

​
Optional Parameters

​
boolean
default:"true"
When true, uses LOAD DATA LOCAL INFILE for loading changes to merge tables.
​
integer
default:"60"
Maximum number of change files to process in a single batch before executing merge operations.
​
boolean
default:"true"
When true, removes change files from inputDirectory after fully processing.
The Change Loader should not start until the Data Loader has successfully completed the initial full load.

​
Orchestrator Configuration

When using the Wirekite Orchestrator, prefix target parameters with target.schema., target.data., or target.change.. Example orchestrator configuration for MySQL target:
For complete Orchestrator documentation, see the Execution Guide.

​
SingleStore Notes

SingleStore (formerly MemSQL) uses the MySQL wire protocol and is fully compatible with the MySQL loaders described above. The following differences apply when targeting SingleStore.

​
Version

SingleStore 7.x or above is required.

​
Connection String

The SingleStore DSN must include allowAllFiles=true in addition to multiStatements=true. The loc=Local parameter used by MySQL is not needed. Connection string format:
Example:
The allowAllFiles=true parameter is required for LOAD DATA operations to work correctly with SingleStore. It is automatically added if missing.

​
Orchestrator Configuration

When targeting SingleStore, use target=singlestore in the orchestrator configuration: