Skip to main content

​
Overview

Wirekite supports MongoDB as a target database for:
  • Schema Loading - Create collections with optional JSON Schema validators
  • Data Loading - Bulk load extracted data using InsertMany operations
  • Change Loading (CDC) - Apply ongoing changes using BulkWrite operations
MongoDB loaders convert relational table data to BSON documents. Primary keys are mapped to MongoDB’s _id field. Composite primary keys are stored as nested BSON documents (e.g., {_id: {col1: val1, col2: val2}}).

​
Prerequisites

Before configuring MongoDB as a Wirekite target, ensure the following requirements are met:

​
Database Configuration

  1. Version: MongoDB 4.x or above
  2. User Permissions: The connection user must have:
    • readWrite role on the target database
    • dbAdmin role if using schema validation (schemaValidation=true)

​
Limitations

MongoDB does not support FOREIGN KEY or CHECK constraints. These are skipped during schema loading with a warning in the log.

​
Schema Loader

The Schema Loader reads Wirekite’s intermediate schema format (.skt file) and creates MongoDB collections. Optionally generates JSON Schema validators for type enforcement.
Collections are created with names in schema.tablename format (e.g., public.users). If the collection already exists, creation is skipped.

​
Required Parameters

​
dsnFile
string
required
Path to a file containing the MongoDB connection string.
Connection string format: 
mongodb://username:password@host:port/database
Example: 
mongodb://wirekite:secretpass@mongodb.example.com:27017/myapp
If no database name is specified in the URI, the default database test is used.
​
schemaFile
string
required
Path to the Wirekite schema file (.skt) generated by the Schema Extractor.
​
logFile
string
required
Absolute path to the log file for Schema Loader operations.

​
Optional Parameters

​
dropCollections
boolean
default:"false"
When true, drops all existing collections before creating them.
​
schemaValidation
boolean
default:"false"
When true, creates MongoDB JSON Schema validators for each collection. Validation action is set to warn (lenient) to avoid blocking documents during migration.

​
Data Loader

The Data Loader reads Wirekite’s intermediate data format (.dkt files) and loads documents into MongoDB collections using InsertMany with unordered batches for maximum throughput.
The Data Loader uses a 3-stage pipeline architecture (Scanner, Parsers, Writers) for high-performance parallel loading.

​
Required Parameters

​
dsnFile
string
required
Path to a file containing the MongoDB connection string.
​
inputDirectory
string
required
Directory containing data files (.dkt) to load.
​
schemaFile
string
required
Path to the Wirekite schema file for table structure information.
​
logFile
string
required
Absolute path to the log file for Data Loader operations.

​
Optional Parameters

​
maxThreads
integer
default:"10"
Maximum number of parallel threads for loading files. Each thread loads one file at a time.
​
hexEncoding
boolean
default:"false"
Set to true if data was extracted using hex encoding instead of base64.

​
Change Loader

The Change Loader applies ongoing data changes (INSERT, UPDATE, DELETE) to MongoDB collections using BulkWrite operations.
Updates use sparse $set operations, only modifying changed fields. Inserts and replaces use full document upserts.

​
Required Parameters

​
dsnFile
string
required
Path to a file containing the MongoDB connection string.
​
inputDirectory
string
required
Directory containing change files (.ckt) from the Change Extractor.
​
schemaFile
string
required
Path to the Wirekite schema file for table structure information.
​
logFile
string
required
Absolute path to the log file for Change Loader operations.

​
Optional Parameters

​
hexEncoding
boolean
default:"false"
Set to true if change data was extracted using hex encoding.
The Change Loader should not start until the Data Loader has successfully completed the initial full load.

​
Orchestrator Configuration

When using the Wirekite Orchestrator, prefix target parameters with target.schema., target.data., or target.change. depending on the operation. Example orchestrator configuration for MongoDB target: 
# Main configuration
source=postgres
target=mongodb

# Schema loading
target.schema.dsnFile=/opt/wirekite/config/mongodb.dsn
target.schema.schemaFile=/opt/wirekite/output/schema/wirekite_schema.skt
target.schema.logFile=/var/log/wirekite/schema-loader.log

# Data loading
target.data.dsnFile=/opt/wirekite/config/mongodb.dsn
target.data.inputDirectory=/opt/wirekite/output/data
target.data.schemaFile=/opt/wirekite/output/schema/wirekite_schema.skt
target.data.logFile=/var/log/wirekite/data-loader.log
target.data.maxThreads=8

# Change loading (CDC)
target.change.dsnFile=/opt/wirekite/config/mongodb.dsn
target.change.inputDirectory=/opt/wirekite/output/changes
target.change.schemaFile=/opt/wirekite/output/schema/wirekite_schema.skt
target.change.logFile=/var/log/wirekite/change-loader.log
For complete Orchestrator documentation, see the Execution Guide.