Skip to main content

​
Overview

Wirekite supports Snowflake as a target data warehouse for:
  • Schema Loading - Create target tables from Wirekite’s intermediate schema format
  • Data Loading - Bulk load extracted data via Snowflake internal stage
  • Change Loading (CDC) - Apply ongoing changes using MERGE operations
Snowflake loaders stage data through Snowflake’s internal stage using PUT commands, then use COPY INTO for high-performance bulk loading.

​
Prerequisites

Before configuring Snowflake as a Wirekite target, ensure the following requirements are met:

​
Snowflake Configuration

  1. User Setup: Create a properly configured Snowflake user with appropriate privileges
  2. Warehouse: Ensure a virtual warehouse is available for loading operations
  3. Database & Schema: Create the target database and schema
  4. CLI Tools: Either install the SnowSQL command line tool or use the Wirekite cmdline tool

​
Internal Tables

Ensure the Wirekite target metadata tables (wirekite_progress and wirekite_action) exist in your Snowflake database and are read-write accessible. Use the Wirekite cmdline tool to verify connectivity.
For connection string format details, see the Snowflake connection documentation.

​
Schema Loader

The Schema Loader reads Wirekite’s intermediate schema format (.skt file) and generates Snowflake-appropriate DDL statements for creating target tables.
The Schema Loader generates both base tables and merge tables (with $wkm suffix) for CDC operations.

​
Required Parameters

​
string
required
Path to the Wirekite schema file (.skt) generated by the Schema Extractor. Must be an absolute path.
​
string
required
Output file for CREATE TABLE statements. Includes both base tables and merge tables for CDC operations.
​
string
required
Output file for CHECK constraints. Snowflake doesn’t enforce many constraints, so you may choose not to load this file.
​
string
required
Output file for FOREIGN KEY constraints. You may elect not to load foreign keys if you don’t need them.
​
string
required
Absolute path to the log file for Schema Loader operations.

​
Optional Parameters

​
string
default:"none"
Output file for DROP TABLE IF EXISTS statements. Set to “none” to skip generation.
​
string
default:"none"
Output file for recovery table creation SQL. Set to “none” to skip.
​
boolean
default:"true"
When true, generates merge tables ($wkm suffix) for CDC operations. Set to false if only doing data loads without change capture.

​
Data Mover

The Data Mover uploads extracted data files to Snowflake’s internal stage using PUT commands for subsequent loading.

​
Required Parameters

​
string
required
Path to a file containing the Snowflake connection string.
Connection string format (Golang connector):
Example:
​
string
required
Directory where the Data Extractor wrote its files. These will be copied to the Snowflake staging area.
​
string
required
Absolute path to the log file for Data Mover operations.

​
Optional Parameters

​
integer
default:"10"
Maximum number of parallel threads for uploading to Snowflake stage.
​
boolean
default:"false"
When true, compresses files before uploading. Changes extension to .dgz.
​
boolean
default:"false"
When true, deletes local files after successful PUT to stage. Should typically be true in production to save disk space.

​
Data Loader

The Data Loader reads data files from Snowflake’s internal stage and loads them into target tables using COPY INTO commands.

​
Required Parameters

​
string
required
Path to a file containing the Snowflake connection string.
Connection string format (Golang connector):
​
string
required
Path to the Wirekite schema file used by Schema Loader. Required for table structure information.
​
string
required
Absolute path to the log file for Data Loader operations.

​
Optional Parameters

​
integer
default:"5"
Maximum number of parallel COPY threads. We recommend setting this to the number of CPUs on the host.
​
boolean
default:"false"
Set to true if data was extracted using hex encoding instead of base64.
​
string
default:"dkt"
File extension for data files to process (e.g., “dkt”, “dgz”).
For best performance, set maxThreads equal to the number of CPU cores available on the loader host.

​
Change Loader

The Change Loader applies ongoing data changes (INSERT, UPDATE, DELETE) to Snowflake tables using MERGE operations with shadow tables.
The Change Loader uses a merging approach that stages intermediate data to Snowflake’s internal stage before executing MERGE statements.

​
Required Parameters

​
string
required
Path to a file containing the Snowflake connection string.
Connection string format (Golang connector):
​
string
required
Directory where the Change Extractor wrote its files. These will be sourced for changes.
​
string
required
Working directory for intermediate files that are uploaded to Snowflake stage during merge operations.
​
string
required
Path to the Wirekite schema file for table structure information.
​
string
required
Absolute path to the log file for Change Loader operations.

​
Optional Parameters

​
integer
default:"60"
Maximum number of change files to process in a single batch before executing MERGE operations.
​
integer
Number of parallel threads for applying merge operations within each batch. Defaults to 2x the number of CPU cores on the host.
​
boolean
default:"false"
Set to true if change data was extracted using hex encoding.
​
boolean
default:"true"
When true, removes change files from inputDirectory after fully processing. Should typically be true in production to save disk space.
The Change Loader should not start until the Data Loader has successfully completed the initial full load.

​
Orchestrator Configuration

When using the Wirekite Orchestrator, prefix parameters with mover., target.schema., target.data., or target.change.. Example orchestrator configuration for Snowflake target:
For complete Orchestrator documentation, see the Execution Guide.