Skip to main content

​
Overview

Wirekite supports Firebolt as a target data warehouse for:
  • Schema Loading - Create target tables from Wirekite’s intermediate schema format
  • Data Loading - Bulk load extracted data via AWS S3 staging
  • Change Loading (CDC) - Apply ongoing changes using MERGE operations
Firebolt loaders stage data through AWS S3 buckets before loading using Firebolt’s COPY SQL command.

​
Prerequisites

Before configuring Firebolt as a Wirekite target, ensure the following requirements are met:

​
Firebolt Configuration

  1. User Setup: Create a properly configured Firebolt user
  2. Engine: Ensure a Firebolt engine is available for loading operations
  3. Database: Create the target database

​
AWS S3 Requirements

An AWS S3 bucket is required for staging data. The bucket must be:
  • In the same AWS Region as your Firebolt database
  • Accessible for read/write from the loader host
  • Accessible for read from Firebolt
Install the AWS CLI tool (aws) for managing the bucket. Run aws configure to set up shell-level authentication. AWS CLI install instructions

​
Internal Tables

Ensure the Wirekite target metadata tables (wirekite_progress and wirekite_action) exist in your Firebolt database. Use the Wirekite cmdline tool to verify connectivity.

​
Schema Loader

The Schema Loader reads Wirekite’s intermediate schema format (.skt file) and generates Firebolt-appropriate DDL statements for creating target tables.
Firebolt does not support FOREIGN KEYs or external CHECK constraints. The Schema Loader will generate empty files for these. Column-level constraints (NOT NULL) are included in CREATE TABLE.

​
Required Parameters

​
schemaFile
string
required
Path to the Wirekite schema file (.skt) generated by the Schema Extractor. Must be an absolute path.
​
createTableFile
string
required
Output file for CREATE TABLE statements. Includes both base tables and merge tables for CDC operations.
​
createConstraintFile
string
required
Output file for constraints (always empty for Firebolt). Required but the file will be empty.
​
createForeignKeyFile
string
required
Output file for FOREIGN KEY constraints (always empty for Firebolt). Required but the file will be empty.
​
logFile
string
required
Absolute path to the log file for Schema Loader operations.

​
Optional Parameters

​
dropTableFile
string
default:"none"
Output file for DROP TABLE IF EXISTS statements. Set to “none” to skip generation.
​
createRecoveryTablesFile
string
default:"none"
Output file for recovery table creation SQL. Set to “none” to skip.
​
createMergeTables
boolean
default:"true"
When true, generates merge tables for CDC operations. Set to false if only doing data loads without change capture.

​
Data Mover (AWS)

The Data Mover uploads extracted data files to an AWS S3 bucket for subsequent loading into Firebolt.

​
Required Parameters

​
awsBucket
string
required
AWS S3 bucket name (without s3:// prefix) for staging data files.
​
awsRegion
string
required
AWS Region where the S3 bucket resides (must match Firebolt region).
​
dataDirectory
string
required
Local directory containing data files (.dkt) from the Data Extractor.
​
logFile
string
required
Absolute path to the log file for Data Mover operations.

​
Optional Parameters

​
awsCredentials
string
AWS credentials in format: aws_access_key_id=KEY,aws_secret_access_key=SECRET. Only required if AWS access isn’t configured via IAM roles or environment.
​
maxThreads
integer
default:"10"
Maximum number of parallel upload threads.
​
gzipFiles
boolean
default:"false"
When true, compresses files with gzip before uploading. Changes extension to .dgz.
​
removeFiles
boolean
default:"true"
When true, deletes local files after successful upload to S3. Should be true in production to save disk space.

​
Data Loader

The Data Loader reads data files from AWS S3 and loads them into Firebolt tables using COPY commands.

​
Required Parameters

​
dsnFile
string
required
Path to a file containing the Firebolt connection string.
Connection string format: 
firebolt://<database>?account_name=<account_name>&account_id=<account_id>&client_secret=<client_secret>&engine=<engine_name>
Example: 
firebolt://mydb?account_name=myaccount&account_id=12345&client_secret=mysecret&engine=myengine
​
awsBucket
string
required
AWS S3 bucket name (without s3:// prefix) where data files were staged.
​
schemaFile
string
required
Path to the Wirekite schema file used by Schema Loader. Required for table structure information.
​
logFile
string
required
Absolute path to the log file for Data Loader operations.

​
Optional Parameters

​
awsRegion
string
default:"us-east-1"
AWS Region where the S3 bucket resides.
​
awsCredentials
string
AWS credentials in format: aws_access_key_id=KEY,aws_secret_access_key=SECRET. Only required if AWS access isn’t set up in Firebolt.
​
maxThreads
integer
default:"5"
Maximum number of parallel threads for loading data. We recommend setting this to the number of CPUs on the host.
​
hexEncoding
boolean
default:"false"
Set to true if data was extracted using hex format instead of base64.
​
countRows
boolean
default:"false"
When true, counts rows in temporary tables before inserting (for progress tracking).
​
removeFiles
boolean
default:"true"
When true, removes files from S3 after successful loading to Firebolt.

​
Change Loader

The Change Loader applies ongoing data changes (INSERT, UPDATE, DELETE) to Firebolt tables using MERGE operations with shadow tables.
The Change Loader uploads intermediate change data to S3 before executing MERGE. The same S3 bucket used for data loading can be used.

​
Required Parameters

​
dsnFile
string
required
Path to a file containing the Firebolt connection string.
Connection string format: 
firebolt://<database>?account_name=<account_name>&account_id=<account_id>&client_secret=<client_secret>&engine=<engine_name>
​
awsBucket
string
required
AWS S3 bucket name (without s3:// prefix) for staging change data.
​
awsRegion
string
required
AWS Region where the S3 bucket resides.
​
inputDirectory
string
required
Directory containing change files (.ckt) from the Change Extractor.
​
workDirectory
string
required
Working directory for intermediate CSV files before uploading to S3.
​
schemaFile
string
required
Path to the Wirekite schema file for table structure information.
​
logFile
string
required
Absolute path to the log file for Change Loader operations.

​
Optional Parameters

​
awsCredentials
string
AWS credentials in format: aws_access_key_id=KEY,aws_secret_access_key=SECRET. Only required if AWS access isn’t set up in Firebolt.
​
maxFilesPerBatch
integer
default:"30"
Maximum number of change files to process in a single batch before executing MERGE.
​
removeFiles
boolean
default:"true"
When true, removes change files from inputDirectory after fully processing.
The Change Loader should not start until the Data Loader has successfully completed the initial full load.

​
Orchestrator Configuration

When using the Wirekite Orchestrator, prefix parameters with mover., target.schema., target.data., or target.change.. Example orchestrator configuration for Firebolt target: 
# Main configuration
source=postgres
target=firebolt

# Data mover (AWS)
mover.awsBucket=my-firebolt-staging
mover.awsRegion=us-east-1
mover.dataDirectory=/opt/wirekite/output/data
mover.logFile=/var/log/wirekite/data-mover.log
mover.maxThreads=10
mover.removeFiles=true
mover.awsCredentials=aws_access_key_id=KEY,aws_secret_access_key=SECRET

# Schema loading
target.schema.schemaFile=/opt/wirekite/output/schema/wirekite_schema.skt
target.schema.createTableFile=/opt/wirekite/output/schema/create_tables.sql
target.schema.createConstraintFile=/opt/wirekite/output/schema/constraints.sql
target.schema.createForeignKeyFile=/opt/wirekite/output/schema/foreign_keys.sql
target.schema.logFile=/var/log/wirekite/schema-loader.log

# Data loading
target.data.dsnFile=/opt/wirekite/config/firebolt.dsn
target.data.awsBucket=my-firebolt-staging
target.data.awsRegion=us-east-1
target.data.schemaFile=/opt/wirekite/output/schema/wirekite_schema.skt
target.data.logFile=/var/log/wirekite/data-loader.log
target.data.maxThreads=8
target.data.awsCredentials=aws_access_key_id=KEY,aws_secret_access_key=SECRET

# Change loading (CDC)
target.change.dsnFile=/opt/wirekite/config/firebolt.dsn
target.change.awsBucket=my-firebolt-staging
target.change.awsRegion=us-east-1
target.change.inputDirectory=/opt/wirekite/output/changes
target.change.workDirectory=/opt/wirekite/work
target.change.schemaFile=/opt/wirekite/output/schema/wirekite_schema.skt
target.change.logFile=/var/log/wirekite/change-loader.log
target.change.maxFilesPerBatch=30
target.change.awsCredentials=aws_access_key_id=KEY,aws_secret_access_key=SECRET
For complete Orchestrator documentation, see the Execution Guide.