Google Cloud Spanner

Overview

Wirekite supports Google Cloud Spanner as a target database for:

Schema Loading - Create target tables from Wirekite’s intermediate schema format
Data Loading - Bulk load extracted data using Spanner mutations
Change Loading (CDC) - Apply ongoing changes using shadow table merge approach

Spanner loaders respect Spanner’s mutation limits (80,000 mutations per commit). The Change Loader uses shadow tables with _wkm suffix for atomic change application.

Prerequisites

Before configuring Spanner as a Wirekite target, ensure the following requirements are met:

Google Cloud Configuration

Project Setup: Have a Google Cloud project with Spanner API enabled
Instance & Database: Create a Spanner instance and database
Authentication: Configure Application Default Credentials or service account
IAM Permissions: Ensure the service account has:
- spanner.databases.read
- spanner.databases.write
- spanner.sessions.create
Metadata Tables: Wirekite automatically creates wirekite_progress and wirekite_action tables in the target database for tracking migration state and operational control. See Operations for details.

Spanner Limitations

Spanner has specific limitations to be aware of:

Maximum STRING/BYTES length: 2,621,440 bytes (2.5MB)
Maximum mutations per commit: 80,000 (counting columns)
Change streams have maximum 7-day retention

Use Application Default Credentials for simplest authentication: run gcloud auth application-default login on the loader host.

Schema Loader

The Schema Loader reads Wirekite’s intermediate schema format (.skt file) and generates Spanner-appropriate DDL statements for creating target tables.

The Schema Loader can also generate Change Stream DDL for Spanner-to-Spanner replication scenarios.

Required Parameters

string

required

Path to the Wirekite schema file (.skt) generated by the Schema Extractor. Must be an absolute path.

string

required

Output file for CREATE TABLE statements. Includes both base tables and merge tables for CDC operations.

string

required

Output file for constraint definitions (indexes, unique constraints).

string

required

Output file for FOREIGN KEY constraints (interleaved tables in Spanner).

string

required

Absolute path to the log file for Schema Loader operations.

Optional Parameters

string

default:"none"

Output file for DROP TABLE statements. Set to “none” to skip generation.

string

default:"none"

Output file for recovery table creation DDL. Set to “none” to skip.

string

default:"none"

Output file for CREATE CHANGE STREAM statement. Used when Spanner is also a source.

string

default:"none"

Output file for DROP CHANGE STREAM statement. Set to “none” to skip.

string

default:"7d"

Change stream retention period. Maximum is “7d” (7 days) for Spanner.

boolean

default:"true"

When true, generates merge tables (_wkm suffix) for CDC operations. Set to false if only doing data loads.

Data Loader

The Data Loader reads Wirekite’s intermediate data format (.dkt files) and loads records into Spanner tables using batched mutations.

Required Parameters

string

required

Path to a file containing the Spanner connection string.

Connection string format:

projects/PROJECT_ID/instances/INSTANCE_ID/databases/DATABASE_ID

Example:

projects/my-project/instances/my-instance/databases/my-database

string

required

Directory containing data files (.dkt) to load.

string

required

Path to the Wirekite schema file used by Schema Loader. Required for table structure information.

string

required

Absolute path to the log file for Data Loader operations.

string

required

Absolute path to a GCP service-account JSON credentials file. Used to authenticate the Spanner client. If the loader host already has Application Default Credentials configured (e.g. via gcloud auth application-default login or a workload-identity-bound VM), point this at the ADC file path — the loader will not start without an explicit value.

Optional Parameters

integer

default:"5"

Maximum number of parallel threads for loading tables.

boolean

default:"false"

Set to true if data was extracted using hex encoding instead of base64.

The Data Loader automatically batches mutations to stay within Spanner’s 80,000 mutation limit per commit.

Change Loader

The Change Loader applies ongoing data changes (INSERT, UPDATE, DELETE) to Spanner tables using a merge approach with shadow tables.

Required Parameters

string

required

Path to a file containing the Spanner connection string.

Connection string format:

projects/PROJECT_ID/instances/INSTANCE_ID/databases/DATABASE_ID

string

required

Directory containing change files (.ckt) from the Change Extractor.

string

required

Working directory for temporary files during merge operations. Must be writable.

string

required

Path to the Wirekite schema file for table structure information.

string

required

Absolute path to the log file for Change Loader operations.

Optional Parameters

boolean

default:"true"

When true, loads changes locally.

integer

default:"30"

Maximum number of change files to process in a single batch.

boolean

default:"true"

When true, uses shadow table merge approach. When false, uses direct mutations.

The Change Loader should not start until the Data Loader has successfully completed the initial full load.

Orchestrator Configuration

When using the Wirekite Orchestrator, prefix target parameters with target.schema., target.data., or target.change.. Example orchestrator configuration for Spanner target:

# Main configuration
source=postgres
target=spanner

# Schema loading
target.schema.schemaFile=/opt/wirekite/output/schema/wirekite_schema.skt
target.schema.createTableFile=/opt/wirekite/output/schema/create_tables.sql
target.schema.createConstraintFile=/opt/wirekite/output/schema/constraints.sql
target.schema.createForeignKeyFile=/opt/wirekite/output/schema/foreign_keys.sql
target.schema.logFile=/var/log/wirekite/schema-loader.log

# Data loading
target.data.dsnFile=/opt/wirekite/config/spanner.dsn
target.data.inputDirectory=/opt/wirekite/output/data
target.data.schemaFile=/opt/wirekite/output/schema/wirekite_schema.skt
target.data.logFile=/var/log/wirekite/data-loader.log
target.data.gcpCredfile=/opt/wirekite/config/gcp-sa.json
target.data.maxThreads=8

# Change loading (CDC)
target.change.dsnFile=/opt/wirekite/config/spanner.dsn
target.change.inputDirectory=/opt/wirekite/output/changes
target.change.workDirectory=/opt/wirekite/work
target.change.schemaFile=/opt/wirekite/output/schema/wirekite_schema.skt
target.change.logFile=/var/log/wirekite/change-loader.log
target.change.maxFilesPerBatch=30

For complete Orchestrator documentation, see the Execution Guide.

Introduction

Datatype Matrices

Source Guides

Data Movers

Target Guides

Queues

Running Wirekite

Tools

API/CLI

Overview

Prerequisites

Google Cloud Configuration

Spanner Limitations

Schema Loader

Required Parameters

Optional Parameters

Data Loader

Required Parameters

Optional Parameters

Change Loader

Required Parameters

Optional Parameters

Orchestrator Configuration

​Overview

​Prerequisites

​Google Cloud Configuration

​Spanner Limitations

​Schema Loader

​Required Parameters

​Optional Parameters

​Data Loader

​Required Parameters

​Optional Parameters

​Change Loader

​Required Parameters

​Optional Parameters

​Orchestrator Configuration

Overview

Prerequisites

Google Cloud Configuration

Spanner Limitations

Schema Loader

Required Parameters

Optional Parameters

Data Loader

Required Parameters

Optional Parameters

Change Loader

Required Parameters

Optional Parameters

Orchestrator Configuration