Skip to main content

​
Overview

Wirekite uses two metadata tables to track migration progress and enable operational control:
  • wirekite_progress - Tracks every file/operation processed, storing position checkpoints for crash recovery
  • wirekite_action - Single-row control table for pause/stop/resume operations
These tables are created automatically in the target database when a migration starts.

​
Metadata Tables

​
wirekite_progress

This table tracks the state of every extraction and loading operation, enabling crash recovery and progress monitoring. Schema: 
CREATE TABLE wirekite.wirekite_progress (
    source VARCHAR(256),
    target VARCHAR(256),
    operation VARCHAR(2) NOT NULL,
    transmission VARCHAR(1) NOT NULL,
    mark VARCHAR(64) NULL,
    table_name VARCHAR(256) NULL,
    start_time TIMESTAMP NOT NULL,
    finish_time TIMESTAMP NULL,
    restarts INT DEFAULT 0 NOT NULL,
    PRIMARY KEY (source, target)
);
The schema name varies by database: wirekite for PostgreSQL, MySQL, Oracle, SingleStore; public for Snowflake and Firebolt.
Column Reference:
​
source
string
Primary key component. Identifies the file or resource being processed (e.g., 0.ckt, schema.table.0.dkt, binlog name).
​
target
string
Primary key component. Identifies the destination or operation type.
​
operation
string
Two-letter operation code:
  • DE - Data Extract
  • DL - Data Load
  • CE - Change Extract
  • CL - Change Load
​
transmission
string
Transfer mode:
  • F - File-based processing
  • Q - Queue/Kafka-based processing
​
mark
string
Progress checkpoint. Meaning depends on operation type:
  • Data loaders: Row count successfully committed
  • Change extractors: LSN/binlog position/SCN (database-specific position marker)
  • Change loaders: Sequence number in .ckt file
​
table_name
string
Associated table name for multi-table operations.
​
start_time
timestamp
When this operation started.
​
finish_time
timestamp
NULL while in progress; set to completion time when done. This is the critical recovery indicator.
​
restarts
integer
Counter tracking how many times this operation was restarted after a crash.

​
wirekite_action

This single-row table controls the running state of loaders and extractors. Schema: 
CREATE TABLE wirekite.wirekite_action (
    action VARCHAR(5),
    PRIMARY KEY (action)
);

INSERT INTO wirekite.wirekite_action VALUES ('RUN');
Valid Values:
ValueBehavior
RUNNormal operation - processes data continuously
PAUSEPause operation - component sleeps for 5 seconds between checks
STOPStop operation - component exits gracefully

​
Pause, Stop, and Resume

Wirekite components poll the wirekite_action table during processing. This enables non-disruptive control without restarting processes.

​
Using the Web Interface

The easiest way to control migrations is through the Web Interface:
  1. Navigate to the migration details page
  2. Click Pause to pause replication
  3. Click Resume to continue
  4. Click Stop to gracefully shut down

​
Using SQL Commands

You can also control migrations directly via SQL on the target database: Pause Replication: 
UPDATE wirekite.wirekite_action SET action = 'PAUSE';
Resume Replication: 
UPDATE wirekite.wirekite_action SET action = 'RUN';
Stop Replication: 
UPDATE wirekite.wirekite_action SET action = 'STOP';
Always use STOP for graceful shutdown rather than killing the process. This ensures the current transaction completes and progress is recorded.

​
Monitoring Progress

​
Check Migration Status

Query wirekite_progress to see the state of all operations: 
SELECT source, operation, mark, start_time, finish_time, restarts
FROM wirekite.wirekite_progress
WHERE operation IN ('DE', 'DL', 'CE', 'CL')
ORDER BY start_time DESC;

​
Find Incomplete Operations

Identify operations that are still in progress or may be stuck: 
SELECT source, operation, table_name, start_time, restarts
FROM wirekite.wirekite_progress
WHERE finish_time IS NULL
ORDER BY start_time DESC;
Records with finish_time IS NULL and an old start_time may indicate a hung process.

​
Monitor Restart Counts

High values in the restarts column indicate repeated failures: 
SELECT source, operation, restarts, start_time
FROM wirekite.wirekite_progress
WHERE restarts > 0
ORDER BY restarts DESC;

​
Crash Recovery

Wirekite automatically recovers from crashes using the wirekite_progress table:
  1. On startup, components check for records with finish_time IS NULL
  2. If found, processing resumes from the position stored in mark
  3. The restarts counter is incremented
This ensures zero data loss - partially loaded files resume from the last committed position.

​
Resetting Migration State

Resetting migration state can cause duplicate data if not done correctly. Always truncate both the progress table AND the target tables together.
To reset a migration and start fresh: 1. Stop the migration: 
UPDATE wirekite.wirekite_action SET action = 'STOP';
2. Wait for processes to exit, then truncate progress on both source and target: 
TRUNCATE TABLE wirekite.wirekite_progress;
3. Truncate target tables that will be reloaded. 4. Reset action to RUN: 
UPDATE wirekite.wirekite_action SET action = 'RUN';
5. Restart the migration.

​
Safety Guidelines

Never Delete Progress Records

Deleting records from wirekite_progress can cause duplicate data in the target or missed changes during replication.

Monitor Restarts

High restart counts indicate repeated failures. Investigate the root cause before the issue compounds.

Check for Stuck Records

Records with NULL finish_time and old start_time may indicate hung processes that need investigation.

Coordinate Resets

When resetting, always truncate wirekite_progress AND target tables together to prevent duplicates.