Operations

Overview

Wirekite uses two metadata tables to track migration progress and enable operational control:

wirekite_progress - Tracks every file/operation processed, storing position checkpoints for crash recovery
wirekite_action - Single-row control table for pause/stop/resume operations

These tables are created automatically in the target database when a migration starts.

Metadata Tables

wirekite_progress

This table tracks the state of every extraction and loading operation, enabling crash recovery and progress monitoring. Schema:

CREATE TABLE wirekite.wirekite_progress (
    source VARCHAR(256),
    target VARCHAR(256),
    operation TEXT NOT NULL,
    transmission TEXT NOT NULL DEFAULT 'F',
    mark VARCHAR(64) NULL,
    table_name VARCHAR(256) NULL,
    start_time TIMESTAMP NOT NULL,
    finish_time TIMESTAMP NULL,
    restarts INT DEFAULT 0 NOT NULL,
    PRIMARY KEY (source, target)
);

The schema name varies by database: wirekite for PostgreSQL, MySQL, Oracle, SingleStore; public for Snowflake and Firebolt.

Column Reference:

string

Primary key component. Identifies the file or resource being processed (e.g., 0.ckt, schema.table.0.dkt, binlog name).

string

Primary key component. Identifies the destination or operation type.

string

Operation code:

D - Data Load
CE - Change Extract
C - Change Load

string

Transfer mode:

F - File-based processing

string

Progress checkpoint. Meaning depends on operation type:

Data loaders: Row count successfully committed
Change extractors: LSN/binlog position/SCN (database-specific position marker)
Change loaders: Sequence number in .ckt file

string

Associated table name for multi-table operations.

timestamp

When this operation started.

timestamp

NULL while in progress; set to completion time when done. This is the critical recovery indicator.

integer

Counter tracking how many times this operation was restarted after a crash.

wirekite_action

This single-row table controls the running state of loaders and extractors. Schema:

CREATE TABLE wirekite.wirekite_action (
    action VARCHAR(5),
    PRIMARY KEY (action)
);

INSERT INTO wirekite.wirekite_action VALUES ('RUN');

Valid Values:

Value	Behavior
`RUN`	Normal operation - processes data continuously
`PAUSE`	Pause operation - component sleeps for 5 seconds between checks
`STOP`	Stop operation - component exits gracefully

Pause, Stop, and Resume

Wirekite components poll the wirekite_action table during processing. This enables non-disruptive control without restarting processes.

Using the Web Interface

The easiest way to control migrations is through the Web Interface:

Navigate to the migration details page
Click Pause to pause replication
Click Resume to continue
Click Stop to gracefully shut down

Using SQL Commands

You can also control migrations directly via SQL on the target database: Pause Replication:

UPDATE wirekite.wirekite_action SET action = 'PAUSE';

Resume Replication:

UPDATE wirekite.wirekite_action SET action = 'RUN';

Stop Replication:

UPDATE wirekite.wirekite_action SET action = 'STOP';

Always use STOP for graceful shutdown rather than killing the process. This ensures the current transaction completes and progress is recorded.

Monitoring Progress

Check Migration Status

Query wirekite_progress to see the state of all operations:

SELECT source, operation, mark, start_time, finish_time, restarts
FROM wirekite.wirekite_progress
WHERE operation IN ('D', 'CE', 'C')
ORDER BY start_time DESC;

Find Incomplete Operations

Identify operations that are still in progress or may be stuck:

SELECT source, operation, table_name, start_time, restarts
FROM wirekite.wirekite_progress
WHERE finish_time IS NULL
ORDER BY start_time DESC;

Records with finish_time IS NULL and an old start_time may indicate a hung process.

Monitor Restart Counts

High values in the restarts column indicate repeated failures:

SELECT source, operation, restarts, start_time
FROM wirekite.wirekite_progress
WHERE restarts > 0
ORDER BY restarts DESC;

Crash Recovery

Wirekite automatically recovers from crashes using the wirekite_progress table:

On startup, components check for records with finish_time IS NULL
If found, processing resumes from the position stored in mark
The restarts counter is incremented

This ensures zero data loss - partially loaded files resume from the last committed position.

Resetting Migration State

Resetting migration state can cause duplicate data if not done correctly. Always truncate both the progress table AND the target tables together.

To reset a migration and start fresh: 1. Stop the migration:

UPDATE wirekite.wirekite_action SET action = 'STOP';

2. Wait for processes to exit, then truncate progress on both source and target:

TRUNCATE TABLE wirekite.wirekite_progress;

3. Truncate target tables that will be reloaded. 4. Reset action to RUN:

UPDATE wirekite.wirekite_action SET action = 'RUN';

5. Restart the migration.

Safety Guidelines

Never Delete Progress Records

Deleting records from wirekite_progress can cause duplicate data in the target or missed changes during replication.

Monitor Restarts

High restart counts indicate repeated failures. Investigate the root cause before the issue compounds.

Check for Stuck Records

Records with NULL finish_time and old start_time may indicate hung processes that need investigation.

Coordinate Resets

When resetting, always truncate wirekite_progress AND target tables together to prevent duplicates.

Introduction

Datatype Matrices

Source Guides

Data Movers

Target Guides

Queues

Running Wirekite

Tools

API/CLI

Overview

Metadata Tables

wirekite_progress

wirekite_action

Pause, Stop, and Resume

Using the Web Interface

Using SQL Commands

Monitoring Progress

Check Migration Status

Find Incomplete Operations

Monitor Restart Counts

Crash Recovery

Resetting Migration State

Safety Guidelines

Never Delete Progress Records

Monitor Restarts

Check for Stuck Records

Coordinate Resets

​Overview

​Metadata Tables

​wirekite_progress

​wirekite_action

​Pause, Stop, and Resume

​Using the Web Interface

​Using SQL Commands

​Monitoring Progress

​Check Migration Status

​Find Incomplete Operations

​Monitor Restart Counts

​Crash Recovery

​Resetting Migration State

​Safety Guidelines

Never Delete Progress Records

Monitor Restarts

Check for Stuck Records

Coordinate Resets

Overview

Metadata Tables

wirekite_progress

wirekite_action

Pause, Stop, and Resume

Using the Web Interface

Using SQL Commands

Monitoring Progress

Check Migration Status

Find Incomplete Operations

Monitor Restart Counts

Crash Recovery

Resetting Migration State

Safety Guidelines