Skip to main content

​
Overview

A Wirekite queue target is a first-class destination — alongside file/DB targets — that receives the change-data-capture stream produced by an extractor. A migration may have a file/DB target, a queue target, or both. When a queue is configured, every committed change is published as a message; the queue is the integration point for downstream consumers (your loader, a stream processor, a search index, etc.).
Queue targets are used during the replication phase only. Initial schema and bulk-data loads always go through the file/DB target. A migration whose only target is a queue is therefore a replication-only migration — it cannot be run in data or data-replication modes.

​
Supported backends

All four source databases (MySQL, PostgreSQL, Oracle, SQL Server) can publish to any of the three backends. The choice is per-queue-target, not per-deployment — a single Wirekite installation can have Kafka, Redpanda, and Pub/Sub queue targets in use at the same time.
Pick Kafka or Redpanda when you control the broker hardware, latency is dominant, or the downstream consumer is Kafka-native. Pick Pub/Sub when the deployment already runs in Google Cloud, or you don’t want to operate a broker cluster.

​
Configuration parameters

The same parameter set applies whether you create a queue target through the UX wizard, the API, or by hand-writing extractor configs for qa/fulltest. A queue target is identified by queueType; the rest of the parameters split into shared, kafka-only, and pubsub-only groups.

​
Shared parameters

​
string
required
Backend selector. One of kafka, redpanda, pubsub. Empty value defaults to kafka for backward compatibility with pre-Pub/Sub configs.
​
string
required
Topic / Pub/Sub topic id. A single topic carries every change record for the migration — there is one topic per queue target, not one per source table. Customer pre-creates the topic; Wirekite errors loudly at startup if it does not exist.
​
integer
default:"1"
Number of .ckt change-record lines packed into a single broker message. 1 means one line per message (highest fidelity, highest message count). Higher values reduce broker RPC volume but increase per-message size. See “Message compilation” below for the exact packing rules.

​
Kafka / Redpanda parameters

​
string
required
Comma-separated list of broker addresses (e.g. broker1.example.com:9092,broker2.example.com:9092). The first broker is dialed at startup as a connectivity check; subsequent metadata is discovered from the cluster.
​
string
default:""
Wire-format compression codec. One of "" (no compression), none, snappy, lz4, zstd. Unknown codecs are rejected at startup.
Authentication (SASL/TLS) is not supported in the current release. Plaintext brokers only. If you need SASL today, terminate it in front of the broker (e.g. with a sidecar proxy) until first-class auth lands.

​
Google Pub/Sub parameters

​
string
required
GCP project id that owns the topic. Used as the project segment in projects/<project>/topics/<topic>.
​
string
required
Absolute path to a service-account JSON key on the extractor host. The plaintext file is encrypted in place by the API at queue-target creation time, so the unencrypted contents do not need to remain on disk afterwards. Empty value falls back to Application Default Credentials.
The queueCompression field is rejected on Pub/Sub queue targets — gRPC handles transport compression transparently. Setting it surfaces a config typo rather than silently ignoring it.

​
Message compilation

The change extractor produces a stream of native Wirekite .ckt change record lines, each line representing one row event in the form:
OP is one of I (insert), U (update), D (delete). Every line is self-contained and carries the full row image as required by the downstream change loader.

​
Per-line vs batched messages

When queueBatchSize=1, every .ckt line becomes one queue message. When queueBatchSize=N (N > 1), the publisher accumulates lines in an internal buffer and emits one message holding N newline-terminated .ckt records once the buffer is full. The downstream consumer demultiplexes by splitting the message body on \n. Batching trade-offs: For high-volume migrations, queueBatchSize=100 is a typical sweet spot; beyond that, broker per-message limits start to bind on wide rows.

​
Ordering and partitioning

Change records for a given source row must arrive at the consumer in commit order, otherwise CDC semantics break (e.g. an UPDATE arriving before its preceding INSERT). Wirekite preserves per-table order: When batchSize > 1 and the backend is Pub/Sub, lines are accumulated in per-table batchers — a single Pub/Sub message never mixes rows from different tables, because every message has exactly one ordering key. Kafka does not have this constraint; Kafka batches may contain mixed-table rows because per-partition order is sufficient.

​
Flush boundaries

FlushAll is called at every source-side commit boundary — a SQL transaction in MySQL/Postgres/Oracle, a Log Sequence Number group in SQL Server. This guarantees:
  • A message never contains rows from two different transactions.
  • Counters never include a partial transaction.
  • Downstream consumers see committed state only.

​
Retries

Wirekite delegates retry policy to the underlying SDK on each backend. The Wirekite layer itself does not retry — it relies on the SDK’s internal retry budget and treats an exhausted-retries error as terminal. Once the SDK signals a permanent failure, Wirekite’s behavior is fail-fast:
  1. Increment the errors counter.
  2. Log a FATAL line naming the failure.
  3. Exit with code 1.
The orchestrator detects the non-zero exit, flips the migration to Replication Failed, and surfaces the failure in the UX. Recovery is through Resume from Failed, which rewinds to the last persisted progress-DB position and re-emits from there. This produces at-least-once delivery semantics — a small window of records may be re-published after a failure. Downstream consumers must therefore tolerate duplicates, or rely on broker-side deduplication (Pub/Sub’s enable-exactly-once-delivery subscription flag).
The fail-fast policy is a deliberate design choice. CDC streams are strongly ordered; silently dropping or reordering a record corrupts the downstream replica. Loud failure plus replay from the last commit boundary is the only safe recovery shape.

​
Stall watchdog (Pub/Sub only)

The Google Pub/Sub publisher has an additional safety net for the case where the SDK’s per-message future never resolves — neither succeeding nor failing. A watchdog ticks every 5 seconds and tracks whether the delivered counter is advancing. If it has been frozen for 30 seconds while there are unresolved publishes, the extractor logs a FATAL: QUEUE stall detected line and exits. The orchestrator then treats this the same way as any other publish failure (Replication Failed → Resume from Failed). Kafka does not need an equivalent watchdog — librdkafka’s message.timeout.ms already covers this failure mode.

​
Charts and operational signals

The Replication pane in the Wirekite UX shows a Queue subpane for every migration that has a queue target. It surfaces three series on one chart: In a healthy migration Delivered/min tracks Extracted/min closely; sustained divergence indicates broker back-pressure. Errors > 0 is always followed by extractor exit (fail-fast), so a non-zero point on the errors series is the marker for a failure incident.
The publishers run in async mode on the Go side (Async: true for Kafka, completion-worker pool for Pub/Sub), so submitted (Extracted) and delivered are tracked separately. On the SQL Server C extractor both counters are wired through librdkafka’s delivery-report callback (Kafka) or google-cloud-cpp’s per-publish future (Pub/Sub), producing the same semantics. The chart schema is identical across all four extractors.
The shutdown log line names the lifetime totals:
submitted and delivered should match exactly when a run completes cleanly. A non-zero errors always co-occurs with a FATAL line earlier in the log.

​
Error conditions

For backend-specific setup (broker bring-up, IAM, subscription flags), continue to the next pages.