Skip to main content
The Snowflake Data Mover transfers files extracted by Wirekite Extractors directly to Snowflake’s internal stage using native PUT commands. This is the recommended mover for Snowflake targets as it leverages Snowflake’s optimized file transfer mechanism. The mover watches a local directory for extracted data files, uploads them to the user’s Snowflake stage (@~), and optionally compresses and removes local files after successful upload.

​
When to Use

Use the Snowflake Data Mover when:
  • Snowflake is your target database
  • You want to use Snowflake’s native PUT command for optimal performance
  • You prefer staging data in Snowflake’s internal stage rather than external cloud storage

​
Features

  • Native Snowflake PUT - Uses Snowflake’s optimized file transfer protocol
  • Multi-threaded uploads - Parallel uploads for better throughput
  • Optional gzip compression - Compress files before upload
  • Crash recovery - Safe restarts without re-uploading completed files
  • Automatic cleanup - Removes local files after successful upload
  • Coordinated completion - Waits for DATA.DONE signal before finalizing

​
Configuration

The Snowflake Data Mover uses a configuration file passed as a command-line argument: 
snowflake_data_mover <config_file>

​
Configuration Parameters

​
dsnFile
string
required
Path to a file containing the Snowflake connection string (DSN). The file should contain a single line with the full Snowflake connection URL.
​
dataDirectory
string
required
The full path on the local machine where the Wirekite Data Extractor writes its files.
​
logFile
string
required
The path to a file where the Snowflake Data Mover will write its logging output.
​
maxThreads
integer
Maximum number of parallel threads for uploading files. Default is 10.
​
gzipFiles
boolean
Whether to gzip files before uploading. When enabled, files are compressed locally and uploaded with SOURCE_COMPRESSION=GZIP. Default is false.
​
removeFiles
boolean
Whether to remove local files after successful upload. Default is false.

​
Example Configuration

dsnFile = /etc/wirekite/snowflake_dsn.txt
dataDirectory = /data/wirekite/extract
logFile = /var/log/wirekite/snowflake_mover.log
maxThreads = 15
gzipFiles = true
removeFiles = true

​
Snowflake DSN File Format

The DSN file should contain a single line with your Snowflake connection string: 
user:password@account/database/schema?warehouse=COMPUTE_WH
Or with additional parameters: 
user:[email protected]/database/schema?warehouse=COMPUTE_WH&role=DATA_LOADER

​
Orchestrator Integration

When using the Snowflake Data Mover with the Wirekite Orchestrator, add the mover configuration to your orchestrator config: 
moverType = snowflake
moverConfig = /path/to/snowflake_mover.cfg

​
How It Works

  1. The mover connects to Snowflake using the provided DSN
  2. Monitors dataDirectory for files with the specified extension
  3. For each file found:
    • Optionally compresses the file with gzip
    • Executes a PUT command to upload to @~/filename
    • Verifies the upload status is UPLOADED
    • Removes the local file (if configured)
  4. When DATA.DONE appears in the directory, uploads it and exits gracefully

​
PUT Command Details

The mover uses different PUT commands based on compression settings: Without compression: 
PUT file:///path/to/file.dkt @~/file.dkt SOURCE_COMPRESSION=NONE AUTO_COMPRESS=FALSE;
With compression: 
PUT file:///path/to/file.dgz @~/file.dgz SOURCE_COMPRESSION=GZIP;

​
Notes

  • Files are uploaded to the user’s default stage (@~)
  • The Snowflake user must have permissions to PUT files to their stage
  • For large migrations, increase maxThreads to improve throughput
  • Enable gzipFiles to reduce network transfer time
  • The Snowflake Loader should be configured to read from the same stage
  • This mover requires direct network connectivity to Snowflake from the extraction host