Firebolt

Overview

Wirekite supports Firebolt as a target data warehouse for:

Schema Loading - Create target tables from Wirekite’s intermediate schema format
Data Loading - Bulk load extracted data via AWS S3 staging
Change Loading (CDC) - Apply ongoing changes using MERGE operations

Firebolt loaders stage data through AWS S3 buckets before loading using Firebolt’s COPY SQL command.

Prerequisites

Before configuring Firebolt as a Wirekite target, ensure the following requirements are met:

Firebolt Configuration

User Setup: Create a properly configured Firebolt user
Engine: Ensure a Firebolt engine is available for loading operations
Database: Create the target database

AWS S3 Requirements

An AWS S3 bucket is required for staging data. The bucket must be:

In the same AWS Region as your Firebolt database
Accessible for read/write from the loader host
Accessible for read from Firebolt

Install the AWS CLI tool (aws) for managing the bucket. Run aws configure to set up shell-level authentication. AWS CLI install instructions

Internal Tables

Ensure the Wirekite target metadata tables (wirekite_progress and wirekite_action) exist in your Firebolt database. Use the Wirekite cmdline tool to verify connectivity.

Schema Loader

The Schema Loader reads Wirekite’s intermediate schema format (.skt file) and generates Firebolt-appropriate DDL statements for creating target tables.

Firebolt does not support FOREIGN KEYs or external CHECK constraints. The Schema Loader will generate empty files for these. Column-level constraints (NOT NULL) are included in CREATE TABLE.

Required Parameters

string

required

Path to the Wirekite schema file (.skt) generated by the Schema Extractor. Must be an absolute path.

string

required

Output file for CREATE TABLE statements. Includes both base tables and merge tables for CDC operations.

string

required

Output file for constraints (always empty for Firebolt). Required but the file will be empty.

string

required

Output file for FOREIGN KEY constraints (always empty for Firebolt). Required but the file will be empty.

string

required

Absolute path to the log file for Schema Loader operations.

Optional Parameters

string

default:"none"

Output file for DROP TABLE IF EXISTS statements. Set to “none” to skip generation.

string

default:"none"

Output file for recovery table creation SQL. Set to “none” to skip.

boolean

default:"true"

When true, generates merge tables for CDC operations. Set to false if only doing data loads without change capture.

Data Mover (AWS)

The Data Mover uploads extracted data files to an AWS S3 bucket for subsequent loading into Firebolt.

Required Parameters

string

required

AWS S3 bucket name (without s3:// prefix) for staging data files.

string

required

AWS Region where the S3 bucket resides (must match Firebolt region).

string

required

Local directory containing data files (.dkt) from the Data Extractor.

string

required

Absolute path to the log file for Data Mover operations.

Optional Parameters

string

AWS credentials in format: aws_access_key_id=KEY,aws_secret_access_key=SECRET. Only required if AWS access isn’t configured via IAM roles or environment.

integer

default:"10"

Maximum number of parallel upload threads.

boolean

default:"false"

When true, compresses files with gzip before uploading. Changes extension to .dgz.

boolean

default:"true"

When true, deletes local files after successful upload to S3. Should be true in production to save disk space.

Data Loader

The Data Loader reads data files from AWS S3 and loads them into Firebolt tables using COPY commands.

Required Parameters

string

required

Path to a file containing the Firebolt connection string.

Connection string format:

firebolt://<database>?account_name=<account_name>&account_id=<account_id>&client_secret=<client_secret>&engine=<engine_name>

Example:

firebolt://mydb?account_name=myaccount&account_id=12345&client_secret=mysecret&engine=myengine

string

required

AWS S3 bucket name (without s3:// prefix) where data files were staged.

string

required

Path to the Wirekite schema file used by Schema Loader. Required for table structure information.

string

required

Absolute path to the log file for Data Loader operations.

Optional Parameters

string

default:"us-east-1"

AWS Region where the S3 bucket resides.

string

AWS credentials in format: aws_access_key_id=KEY,aws_secret_access_key=SECRET. Only required if AWS access isn’t set up in Firebolt.

integer

default:"5"

Maximum number of parallel threads for loading data. We recommend setting this to the number of CPUs on the host.

boolean

default:"false"

Set to true if data was extracted using hex format instead of base64.

boolean

default:"false"

When true, counts rows in temporary tables before inserting (for progress tracking).

boolean

default:"true"

When true, removes files from S3 after successful loading to Firebolt.

Change Loader

The Change Loader applies ongoing data changes (INSERT, UPDATE, DELETE) to Firebolt tables using MERGE operations with shadow tables.

The Change Loader uploads intermediate change data to S3 before executing MERGE. The same S3 bucket used for data loading can be used.

Required Parameters

string

required

Path to a file containing the Firebolt connection string.

Connection string format:

firebolt://<database>?account_name=<account_name>&account_id=<account_id>&client_secret=<client_secret>&engine=<engine_name>

string

required

AWS S3 bucket name (without s3:// prefix) for staging change data.

string

required

AWS Region where the S3 bucket resides.

string

required

Directory containing change files (.ckt) from the Change Extractor.

string

required

Working directory for intermediate CSV files before uploading to S3.

string

required

Path to the Wirekite schema file for table structure information.

string

required

Absolute path to the log file for Change Loader operations.

Optional Parameters

string

AWS credentials in format: aws_access_key_id=KEY,aws_secret_access_key=SECRET. Only required if AWS access isn’t set up in Firebolt.

integer

default:"60"

Maximum number of change files to process in a single batch before executing MERGE.

integer

Number of parallel threads for applying merge operations within each batch. Defaults to 2x the number of CPU cores on the host.

boolean

default:"true"

When true, removes change files from inputDirectory after fully processing.

The Change Loader should not start until the Data Loader has successfully completed the initial full load.

Orchestrator Configuration

When using the Wirekite Orchestrator, prefix parameters with mover., target.schema., target.data., or target.change.. Example orchestrator configuration for Firebolt target:

# Main configuration
source=postgres
target=firebolt

# Data mover (AWS)
mover.awsBucket=my-firebolt-staging
mover.awsRegion=us-east-1
mover.dataDirectory=/opt/wirekite/output/data
mover.logFile=/var/log/wirekite/data-mover.log
mover.maxThreads=10
mover.removeFiles=true
mover.awsCredentials=aws_access_key_id=KEY,aws_secret_access_key=SECRET

# Schema loading
target.schema.schemaFile=/opt/wirekite/output/schema/wirekite_schema.skt
target.schema.createTableFile=/opt/wirekite/output/schema/create_tables.sql
target.schema.createConstraintFile=/opt/wirekite/output/schema/constraints.sql
target.schema.createForeignKeyFile=/opt/wirekite/output/schema/foreign_keys.sql
target.schema.logFile=/var/log/wirekite/schema-loader.log

# Data loading
target.data.dsnFile=/opt/wirekite/config/firebolt.dsn
target.data.awsBucket=my-firebolt-staging
target.data.awsRegion=us-east-1
target.data.schemaFile=/opt/wirekite/output/schema/wirekite_schema.skt
target.data.logFile=/var/log/wirekite/data-loader.log
target.data.maxThreads=8
target.data.awsCredentials=aws_access_key_id=KEY,aws_secret_access_key=SECRET

# Change loading (CDC)
target.change.dsnFile=/opt/wirekite/config/firebolt.dsn
target.change.awsBucket=my-firebolt-staging
target.change.awsRegion=us-east-1
target.change.inputDirectory=/opt/wirekite/output/changes
target.change.workDirectory=/opt/wirekite/work
target.change.schemaFile=/opt/wirekite/output/schema/wirekite_schema.skt
target.change.logFile=/var/log/wirekite/change-loader.log
target.change.maxFilesPerBatch=30
target.change.awsCredentials=aws_access_key_id=KEY,aws_secret_access_key=SECRET

For complete Orchestrator documentation, see the Execution Guide.

Introduction

Datatype Matrices

Source Guides

Data Movers

Target Guides

Queues

Running Wirekite

Tools

API/CLI

Overview

Prerequisites

Firebolt Configuration

AWS S3 Requirements

Internal Tables

Schema Loader

Required Parameters

Optional Parameters

Data Mover (AWS)

Required Parameters

Optional Parameters

Data Loader

Required Parameters

Optional Parameters

Change Loader

Required Parameters

Optional Parameters

Orchestrator Configuration

​Overview

​Prerequisites

​Firebolt Configuration

​AWS S3 Requirements

​Internal Tables

​Schema Loader

​Required Parameters

​Optional Parameters

​Data Mover (AWS)

​Required Parameters

​Optional Parameters

​Data Loader

​Required Parameters

​Optional Parameters

​Change Loader

​Required Parameters

​Optional Parameters

​Orchestrator Configuration

Overview

Prerequisites

Firebolt Configuration

AWS S3 Requirements

Internal Tables

Schema Loader

Required Parameters

Optional Parameters

Data Mover (AWS)

Required Parameters

Optional Parameters

Data Loader

Required Parameters

Optional Parameters

Change Loader

Required Parameters

Optional Parameters

Orchestrator Configuration