Skip to main content

​
Overview

Wirekite supports Google Cloud Spanner as a target database for:
  • Schema Loading - Create target tables from Wirekite’s intermediate schema format
  • Data Loading - Bulk load extracted data using Spanner mutations
  • Change Loading (CDC) - Apply ongoing changes using shadow table merge approach
Spanner loaders respect Spanner’s mutation limits (80,000 mutations per commit). The Change Loader uses shadow tables with _wkm suffix for atomic change application.

​
Prerequisites

Before configuring Spanner as a Wirekite target, ensure the following requirements are met:

​
Google Cloud Configuration

  1. Project Setup: Have a Google Cloud project with Spanner API enabled
  2. Instance & Database: Create a Spanner instance and database
  3. Authentication: Configure Application Default Credentials or service account
  4. IAM Permissions: Ensure the service account has:
    • spanner.databases.read
    • spanner.databases.write
    • spanner.sessions.create

​
Spanner Limitations

Spanner has specific limitations to be aware of:
  • Maximum STRING/BYTES length: 2,621,440 bytes (2.5MB)
  • Maximum mutations per commit: 80,000 (counting columns)
  • Change streams have maximum 7-day retention
Use Application Default Credentials for simplest authentication: run gcloud auth application-default login on the loader host.

​
Schema Loader

The Schema Loader reads Wirekite’s intermediate schema format (.skt file) and generates Spanner-appropriate DDL statements for creating target tables.
The Schema Loader can also generate Change Stream DDL for Spanner-to-Spanner replication scenarios.

​
Required Parameters

​
schemaFile
string
required
Path to the Wirekite schema file (.skt) generated by the Schema Extractor. Must be an absolute path.
​
createTableFile
string
required
Output file for CREATE TABLE statements. Includes both base tables and merge tables for CDC operations.
​
createConstraintFile
string
required
Output file for constraint definitions (indexes, unique constraints).
​
createForeignKeyFile
string
required
Output file for FOREIGN KEY constraints (interleaved tables in Spanner).
​
logFile
string
required
Absolute path to the log file for Schema Loader operations.

​
Optional Parameters

​
dropTableFile
string
default:"none"
Output file for DROP TABLE statements. Set to “none” to skip generation.
​
createRecoveryTablesFile
string
default:"none"
Output file for recovery table creation DDL. Set to “none” to skip.
​
createChangeStreamFile
string
default:"none"
Output file for CREATE CHANGE STREAM statement. Used when Spanner is also a source.
​
dropChangeStreamFile
string
default:"none"
Output file for DROP CHANGE STREAM statement. Set to “none” to skip.
​
changeStreamRetention
string
default:"7d"
Change stream retention period. Maximum is “7d” (7 days) for Spanner.
​
createMergeTables
boolean
default:"true"
When true, generates merge tables (_wkm suffix) for CDC operations. Set to false if only doing data loads.

​
Data Loader

The Data Loader reads Wirekite’s intermediate data format (.dkt files) and loads records into Spanner tables using batched mutations.

​
Required Parameters

​
dsnFile
string
required
Path to a file containing the Spanner connection string.
Connection string format: 
projects/PROJECT_ID/instances/INSTANCE_ID/databases/DATABASE_ID
Example: 
projects/my-project/instances/my-instance/databases/my-database
​
inputDirectory
string
required
Directory containing data files (.dkt) to load.
​
schemaFile
string
required
Path to the Wirekite schema file used by Schema Loader. Required for table structure information.
​
logFile
string
required
Absolute path to the log file for Data Loader operations.

​
Optional Parameters

​
maxThreads
integer
default:"5"
Maximum number of parallel threads for loading tables.
​
hexEncoding
boolean
default:"false"
Set to true if data was extracted using hex encoding instead of base64.
The Data Loader automatically batches mutations to stay within Spanner’s 80,000 mutation limit per commit.

​
Change Loader

The Change Loader applies ongoing data changes (INSERT, UPDATE, DELETE) to Spanner tables using a merge approach with shadow tables.

​
Required Parameters

​
dsnFile
string
required
Path to a file containing the Spanner connection string.
Connection string format: 
projects/PROJECT_ID/instances/INSTANCE_ID/databases/DATABASE_ID
​
inputDirectory
string
required
Directory containing change files (.ckt) from the Change Extractor.
​
workDirectory
string
required
Working directory for temporary files during merge operations. Must be writable.
​
schemaFile
string
required
Path to the Wirekite schema file for table structure information.
​
logFile
string
required
Absolute path to the log file for Change Loader operations.

​
Optional Parameters

​
loadLocal
boolean
default:"true"
When true, loads changes locally.
​
maxFilesPerBatch
integer
default:"30"
Maximum number of change files to process in a single batch.
​
doMerge
boolean
default:"true"
When true, uses shadow table merge approach. When false, uses direct mutations.
The Change Loader should not start until the Data Loader has successfully completed the initial full load.

​
Orchestrator Configuration

When using the Wirekite Orchestrator, prefix target parameters with target.schema., target.data., or target.change.. Example orchestrator configuration for Spanner target: 
# Main configuration
source=postgres
target=spanner

# Schema loading
target.schema.schemaFile=/opt/wirekite/output/schema/wirekite_schema.skt
target.schema.createTableFile=/opt/wirekite/output/schema/create_tables.sql
target.schema.createConstraintFile=/opt/wirekite/output/schema/constraints.sql
target.schema.createForeignKeyFile=/opt/wirekite/output/schema/foreign_keys.sql
target.schema.logFile=/var/log/wirekite/schema-loader.log

# Data loading
target.data.dsnFile=/opt/wirekite/config/spanner.dsn
target.data.inputDirectory=/opt/wirekite/output/data
target.data.schemaFile=/opt/wirekite/output/schema/wirekite_schema.skt
target.data.logFile=/var/log/wirekite/data-loader.log
target.data.maxThreads=8

# Change loading (CDC)
target.change.dsnFile=/opt/wirekite/config/spanner.dsn
target.change.inputDirectory=/opt/wirekite/output/changes
target.change.workDirectory=/opt/wirekite/work
target.change.schemaFile=/opt/wirekite/output/schema/wirekite_schema.skt
target.change.logFile=/var/log/wirekite/change-loader.log
target.change.maxFilesPerBatch=30
For complete Orchestrator documentation, see the Execution Guide.