Skip to main content
The Snowflake Data Mover transfers files extracted by Wirekite Extractors directly to Snowflake’s internal stage using native PUT commands. This is the recommended mover for Snowflake targets as it leverages Snowflake’s optimized file transfer mechanism. The mover watches a local directory for extracted data files, uploads them to the user’s Snowflake stage (@~), and optionally compresses and removes local files after successful upload.

​
When to Use

Use the Snowflake Data Mover when:
  • Snowflake is your target database
  • You want to use Snowflake’s native PUT command for optimal performance
  • You prefer staging data in Snowflake’s internal stage rather than external cloud storage

​
Features

  • Native Snowflake PUT - Uses Snowflake’s optimized file transfer protocol
  • Multi-threaded uploads - Parallel uploads for better throughput
  • Optional gzip compression - Compress files before upload
  • Crash recovery - Safe restarts without re-uploading completed files
  • Automatic cleanup - Removes local files after successful upload
  • Coordinated completion - Waits for DATA.DONE signal before finalizing

​
Configuration

The Snowflake Data Mover uses a configuration file passed as a command-line argument:

​
Configuration Parameters

​
string
required
Path to a file containing the Snowflake connection string (DSN). The file should contain a single line with the full Snowflake connection URL.
​
string
required
The full path on the local machine where the Wirekite Data Extractor writes its files.
​
string
required
The path to a file where the Snowflake Data Mover will write its logging output.
​
integer
Maximum number of parallel threads for uploading files. Default is 10.
​
boolean
Whether to gzip files before uploading. When enabled, files are compressed locally and uploaded with SOURCE_COMPRESSION=GZIP. Default is false.
​
boolean
Whether to remove local files after successful upload. Default is false.

​
Example Configuration

​
Snowflake DSN File Format

The DSN file should contain a single line with your Snowflake connection string:
Or with additional parameters:

​
Orchestrator Integration

When using the Snowflake Data Mover with the Wirekite Orchestrator, add the mover configuration to your orchestrator config:

​
How It Works

  1. The mover connects to Snowflake using the provided DSN
  2. Monitors dataDirectory for files with the specified extension
  3. For each file found:
    • Optionally compresses the file with gzip
    • Executes a PUT command to upload to @~/filename
    • Verifies the upload status is UPLOADED
    • Removes the local file (if configured)
  4. When DATA.DONE appears in the directory, uploads it and exits gracefully

​
PUT Command Details

The mover uses different PUT commands based on compression settings: Without compression:
With compression:

​
Notes

  • Files are uploaded to the user’s default stage (@~)
  • The Snowflake user must have permissions to PUT files to their stage
  • For large migrations, increase maxThreads to improve throughput
  • Enable gzipFiles to reduce network transfer time
  • The Snowflake Loader should be configured to read from the same stage
  • This mover requires direct network connectivity to Snowflake from the extraction host