> ## Documentation Index
> Fetch the complete documentation index at: https://docs.wirekite.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Firebolt

> This guide explains how to configure Firebolt as a target data warehouse for Wirekite data loading and replication.

## Overview

Wirekite supports Firebolt as a target data warehouse for:

* **Schema Loading** - Create target tables from Wirekite's intermediate schema format
* **Data Loading** - Bulk load extracted data via AWS S3 staging
* **Change Loading (CDC)** - Apply ongoing changes using MERGE operations

<Note>
  Firebolt loaders stage data through AWS S3 buckets before loading using Firebolt's [COPY SQL](https://docs.firebolt.io/reference-sql/commands/data-management/copy-from) command.
</Note>

## Prerequisites

Before configuring Firebolt as a Wirekite target, ensure the following requirements are met:

### Firebolt Configuration

1. **User Setup**: Create a properly configured Firebolt user
2. **Engine**: Ensure a Firebolt engine is available for loading operations
3. **Database**: Create the target database

### AWS S3 Requirements

<Warning>
  An AWS S3 bucket is required for staging data. The bucket must be:

  * In the same AWS Region as your Firebolt database
  * Accessible for read/write from the loader host
  * Accessible for read from Firebolt
</Warning>

<Tip>
  Install the AWS CLI tool (`aws`) for managing the bucket. Run `aws configure` to set up shell-level authentication. [AWS CLI install instructions](https://docs.aws.amazon.com/cli/v1/userguide/install-linux.html)
</Tip>

### Internal Tables

Ensure the Wirekite target metadata tables (`wirekite_progress` and `wirekite_action`) exist in your Firebolt database. Use the Wirekite cmdline tool to verify connectivity.

***

## Schema Loader

The Schema Loader reads Wirekite's intermediate schema format (`.skt` file) and generates Firebolt-appropriate DDL statements for creating target tables.

<Note>
  Firebolt does not support FOREIGN KEYs or external CHECK constraints. The Schema Loader will generate empty files for these. Column-level constraints (NOT NULL) are included in CREATE TABLE.
</Note>

### Required Parameters

<ResponseField name="schemaFile" type="string" required>
  Path to the Wirekite schema file (`.skt`) generated by the Schema Extractor. Must be an absolute path.
</ResponseField>

<ResponseField name="createTableFile" type="string" required>
  Output file for CREATE TABLE statements. Includes both base tables and merge tables for CDC operations.
</ResponseField>

<ResponseField name="createConstraintFile" type="string" required>
  Output file for constraints (always empty for Firebolt). Required but the file will be empty.
</ResponseField>

<ResponseField name="createForeignKeyFile" type="string" required>
  Output file for FOREIGN KEY constraints (always empty for Firebolt). Required but the file will be empty.
</ResponseField>

<ResponseField name="logFile" type="string" required>
  Absolute path to the log file for Schema Loader operations.
</ResponseField>

### Optional Parameters

<ResponseField name="dropTableFile" type="string" default="none">
  Output file for DROP TABLE IF EXISTS statements. Set to "none" to skip generation.
</ResponseField>

<ResponseField name="createRecoveryTablesFile" type="string" default="none">
  Output file for recovery table creation SQL. Set to "none" to skip.
</ResponseField>

<ResponseField name="createMergeTables" type="boolean" default="true">
  When `true`, generates merge tables for CDC operations. Set to `false` if only doing data loads without change capture.
</ResponseField>

***

## Data Mover (AWS)

The Data Mover uploads extracted data files to an AWS S3 bucket for subsequent loading into Firebolt.

### Required Parameters

<ResponseField name="awsBucket" type="string" required>
  AWS S3 bucket name (without `s3://` prefix) for staging data files.
</ResponseField>

<ResponseField name="awsRegion" type="string" required>
  AWS Region where the S3 bucket resides (must match Firebolt region).
</ResponseField>

<ResponseField name="dataDirectory" type="string" required>
  Local directory containing data files (`.dkt`) from the Data Extractor.
</ResponseField>

<ResponseField name="logFile" type="string" required>
  Absolute path to the log file for Data Mover operations.
</ResponseField>

### Optional Parameters

<ResponseField name="awsCredentials" type="string">
  AWS credentials in format: `aws_access_key_id=KEY,aws_secret_access_key=SECRET`. Only required if AWS access isn't configured via IAM roles or environment.
</ResponseField>

<ResponseField name="maxThreads" type="integer" default="10">
  Maximum number of parallel upload threads.
</ResponseField>

<ResponseField name="gzipFiles" type="boolean" default="false">
  When `true`, compresses files with gzip before uploading. Changes extension to `.dgz`.
</ResponseField>

<ResponseField name="removeFiles" type="boolean" default="true">
  When `true`, deletes local files after successful upload to S3. Should be `true` in production to save disk space.
</ResponseField>

***

## Data Loader

The Data Loader reads data files from AWS S3 and loads them into Firebolt tables using COPY commands.

### Required Parameters

<ResponseField name="dsnFile" type="string" required>
  Path to a file containing the Firebolt connection string.
</ResponseField>

**Connection string format:**

```
firebolt://<database>?account_name=<account_name>&account_id=<account_id>&client_secret=<client_secret>&engine=<engine_name>
```

**Example:**

```
firebolt://mydb?account_name=myaccount&account_id=12345&client_secret=mysecret&engine=myengine
```

<ResponseField name="awsBucket" type="string" required>
  AWS S3 bucket name (without `s3://` prefix) where data files were staged.
</ResponseField>

<ResponseField name="schemaFile" type="string" required>
  Path to the Wirekite schema file used by Schema Loader. Required for table structure information.
</ResponseField>

<ResponseField name="logFile" type="string" required>
  Absolute path to the log file for Data Loader operations.
</ResponseField>

### Optional Parameters

<ResponseField name="awsRegion" type="string" default="us-east-1">
  AWS Region where the S3 bucket resides.
</ResponseField>

<ResponseField name="awsCredentials" type="string">
  AWS credentials in format: `aws_access_key_id=KEY,aws_secret_access_key=SECRET`. Only required if AWS access isn't set up in Firebolt.
</ResponseField>

<ResponseField name="maxThreads" type="integer" default="5">
  Maximum number of parallel threads for loading data. We recommend setting this to the number of CPUs on the host.
</ResponseField>

<ResponseField name="hexEncoding" type="boolean" default="false">
  Set to `true` if data was extracted using hex format instead of base64.
</ResponseField>

<ResponseField name="countRows" type="boolean" default="false">
  When `true`, counts rows in temporary tables before inserting (for progress tracking).
</ResponseField>

<ResponseField name="removeFiles" type="boolean" default="true">
  When `true`, removes files from S3 after successful loading to Firebolt.
</ResponseField>

***

## Change Loader

The Change Loader applies ongoing data changes (INSERT, UPDATE, DELETE) to Firebolt tables using MERGE operations with shadow tables.

<Note>
  The Change Loader uploads intermediate change data to S3 before executing MERGE. The same S3 bucket used for data loading can be used.
</Note>

### Required Parameters

<ResponseField name="dsnFile" type="string" required>
  Path to a file containing the Firebolt connection string.
</ResponseField>

**Connection string format:**

```
firebolt://<database>?account_name=<account_name>&account_id=<account_id>&client_secret=<client_secret>&engine=<engine_name>
```

<ResponseField name="awsBucket" type="string" required>
  AWS S3 bucket name (without `s3://` prefix) for staging change data.
</ResponseField>

<ResponseField name="awsRegion" type="string" required>
  AWS Region where the S3 bucket resides.
</ResponseField>

<ResponseField name="inputDirectory" type="string" required>
  Directory containing change files (`.ckt`) from the Change Extractor.
</ResponseField>

<ResponseField name="workDirectory" type="string" required>
  Working directory for intermediate CSV files before uploading to S3.
</ResponseField>

<ResponseField name="schemaFile" type="string" required>
  Path to the Wirekite schema file for table structure information.
</ResponseField>

<ResponseField name="logFile" type="string" required>
  Absolute path to the log file for Change Loader operations.
</ResponseField>

### Optional Parameters

<ResponseField name="awsCredentials" type="string">
  AWS credentials in format: `aws_access_key_id=KEY,aws_secret_access_key=SECRET`. Only required if AWS access isn't set up in Firebolt.
</ResponseField>

<ResponseField name="maxFilesPerBatch" type="integer" default="60">
  Maximum number of change files to process in a single batch before executing MERGE.
</ResponseField>

<ResponseField name="maxMergeThreads" type="integer">
  Number of parallel threads for applying merge operations within each batch. Defaults to 2x the number of CPU cores on the host.
</ResponseField>

<ResponseField name="removeFiles" type="boolean" default="true">
  When `true`, removes change files from `inputDirectory` after fully processing.
</ResponseField>

<Warning>
  The Change Loader should not start until the Data Loader has successfully completed the initial full load.
</Warning>

***

## Orchestrator Configuration

When using the Wirekite Orchestrator, prefix parameters with `mover.`, `target.schema.`, `target.data.`, or `target.change.`.

**Example orchestrator configuration for Firebolt target:**

```
# Main configuration
source=postgres
target=firebolt

# Data mover (AWS)
mover.awsBucket=my-firebolt-staging
mover.awsRegion=us-east-1
mover.dataDirectory=/opt/wirekite/output/data
mover.logFile=/var/log/wirekite/data-mover.log
mover.maxThreads=10
mover.removeFiles=true
mover.awsCredentials=aws_access_key_id=KEY,aws_secret_access_key=SECRET

# Schema loading
target.schema.schemaFile=/opt/wirekite/output/schema/wirekite_schema.skt
target.schema.createTableFile=/opt/wirekite/output/schema/create_tables.sql
target.schema.createConstraintFile=/opt/wirekite/output/schema/constraints.sql
target.schema.createForeignKeyFile=/opt/wirekite/output/schema/foreign_keys.sql
target.schema.logFile=/var/log/wirekite/schema-loader.log

# Data loading
target.data.dsnFile=/opt/wirekite/config/firebolt.dsn
target.data.awsBucket=my-firebolt-staging
target.data.awsRegion=us-east-1
target.data.schemaFile=/opt/wirekite/output/schema/wirekite_schema.skt
target.data.logFile=/var/log/wirekite/data-loader.log
target.data.maxThreads=8
target.data.awsCredentials=aws_access_key_id=KEY,aws_secret_access_key=SECRET

# Change loading (CDC)
target.change.dsnFile=/opt/wirekite/config/firebolt.dsn
target.change.awsBucket=my-firebolt-staging
target.change.awsRegion=us-east-1
target.change.inputDirectory=/opt/wirekite/output/changes
target.change.workDirectory=/opt/wirekite/work
target.change.schemaFile=/opt/wirekite/output/schema/wirekite_schema.skt
target.change.logFile=/var/log/wirekite/change-loader.log
target.change.maxFilesPerBatch=30
target.change.awsCredentials=aws_access_key_id=KEY,aws_secret_access_key=SECRET
```

For complete Orchestrator documentation, see the [Execution Guide](/run/execution).