Skip to main content

​
Overview

Wirekite supports Google BigQuery as a target data warehouse for:
  • Schema Loading - Create target tables from Wirekite’s intermediate schema format
  • Data Loading - Bulk load extracted data via Google Cloud Storage staging
  • Change Loading (CDC) - Apply ongoing changes using MERGE operations
BigQuery loaders stage data through Google Cloud Storage (GCS) buckets before loading to BigQuery using COPY INTO commands.

​
Prerequisites

Before configuring BigQuery as a Wirekite target, ensure the following requirements are met:

​
Google Cloud Configuration

  1. Project Setup: Have a Google Cloud project with BigQuery API enabled
  2. Dataset: Create a BigQuery dataset in the desired location
  3. GCS Bucket: Create a Google Cloud Storage bucket for staging data
  4. Authentication: Configure Application Default Credentials or service account
  5. IAM Permissions: Ensure the service account has:
    • bigquery.tables.create, bigquery.tables.updateData
    • storage.objects.create, storage.objects.delete on the GCS bucket
  6. Metadata Tables: Wirekite automatically creates wirekite_progress and wirekite_action tables in the target dataset for tracking migration state and operational control. See Operations for details.

​
Storage Requirements

The GCS bucket must be accessible from both the loader host and BigQuery. Ensure the bucket is in the same region as your BigQuery dataset for best performance.
Use Application Default Credentials for simplest authentication: run gcloud auth application-default login on the loader host.

​
Schema Loader

The Schema Loader reads Wirekite’s intermediate schema format (.skt file) and generates BigQuery-appropriate DDL statements for creating target tables.

​
Required Parameters

​
string
required
Path to the Wirekite schema file (.skt) generated by the Schema Extractor. Must be an absolute path.
​
string
required
Output file for CREATE TABLE statements. Includes both base tables and merge tables for CDC operations.
​
string
required
Output file for constraint definitions (BigQuery has limited constraint support).
​
string
required
Output file for FOREIGN KEY constraints (informational only in BigQuery).
​
string
required
Absolute path to the log file for Schema Loader operations.

​
Optional Parameters

​
string
default:"none"
Output file for DROP TABLE IF EXISTS statements. Set to “none” to skip generation.
​
string
default:"none"
Output file for recovery table creation SQL. Set to “none” to skip.
​
boolean
default:"true"
When true, generates merge tables (_wkm suffix) for CDC operations. Set to false if only doing data loads.

​
Data Mover (GCS)

The Data Mover uploads extracted data files to Google Cloud Storage for subsequent loading into BigQuery.

​
Required Parameters

​
string
required
GCS bucket name (without gs:// prefix) for staging data files.
​
string
required
Local directory containing data files (.dkt) from the Data Extractor.
​
string
required
Absolute path to the log file for Data Mover operations.

​
Optional Parameters

​
string
Path to GCS service account credentials JSON file. Uses Application Default Credentials if not specified.
​
integer
default:"10"
Maximum number of parallel upload threads.
​
boolean
default:"false"
When true, compresses files with gzip before uploading. Changes extension to .dgz.
​
boolean
default:"true"
When true, deletes local files after successful upload to GCS.

​
Data Loader

The Data Loader reads data files from GCS and loads them into BigQuery tables using COPY INTO operations.

​
Required Parameters

​
string
required
Path to a file containing the BigQuery connection string.
Connection string format:
Example:
​
string
required
Path to the Wirekite schema file used by Schema Loader. Required for table structure information.
​
string
required
Absolute path to the log file for Data Loader operations.

​
Optional Parameters

​
string
Path to GCS service account credentials JSON file. Uses Application Default Credentials if not specified.
​
integer
default:"5"
Maximum number of parallel threads for loading tables.
​
boolean
default:"false"
Set to true if data was extracted using hex encoding instead of base64.
​
string
BigQuery dataset location (e.g., “US”, “EU”). Only needed if non-default.
The Data Loader creates temporary staging tables with auto-expiration for intermediate processing.

​
Change Loader

The Change Loader applies ongoing data changes (INSERT, UPDATE, DELETE) to BigQuery tables using MERGE operations with shadow tables.

​
Required Parameters

​
string
required
Path to a file containing the BigQuery connection string.
Connection string format:
​
string
required
Directory containing change files (.ckt) from the Change Extractor.
​
string
required
Working directory for temporary CSV files during merge operations. Must be writable.
​
string
required
Path to the Wirekite schema file for table structure information.
​
string
required
Absolute path to the log file for Change Loader operations.

​
Optional Parameters

​
string
Path to GCS service account credentials JSON file. Uses Application Default Credentials if not specified.
​
integer
default:"60"
Maximum number of change files to process in a single batch.
​
integer
Number of parallel threads for applying merge operations within each batch. Defaults to 2x the number of CPU cores on the host.
​
boolean
default:"false"
Set to true if change data was extracted using hex encoding.
​
string
BigQuery dataset location (e.g., “US”, “EU”). Only needed if non-default.
The Change Loader should not start until the Data Loader has successfully completed the initial full load.

​
Orchestrator Configuration

When using the Wirekite Orchestrator, prefix parameters with mover., target.schema., target.data., or target.change.. Example orchestrator configuration for BigQuery target:
For complete Orchestrator documentation, see the Execution Guide.