How to Move Large Amounts of Data Between Object Storage Buckets?

RAHOOOLDAV
RAHOOOLDAV
 Whether you're migrating 10 GB, 10 TB, or multiple petabytes, the correct migration strategy depends on:
  • Number of objects (10,000 vs 100 million)
  • Total data size
  • Source and destination providers
  • Whether both support the S3 API
  • Downtime requirements
  • Available bandwidth
  • Data integrity requirements
If you're using Synclyz.com's S3 Object Storage, because it exposes an S3-compatible API, nearly every enterprise S3 migration tool works without modification.

chart

AWS recommends AWS CLI, S3 Batch Operations, Cross-Region Replication, and AWS DataSync depending on bucket size and object count. For multi-terabyte migrations, DataSync or parallelized transfers are generally preferred.

Option 1 — rclone (Recommended)

For S3-compatible providers—including Synclyz.com's S3 Object Storage—rclone is typically the most flexible choice.

Advantages

  • Works with any S3-compatible provider
  • Resume interrupted transfers
  • Verify checksums
  • Multi-threaded
  • Parallel uploads/downloads
  • Copy bucket-to-bucket directly
  • Preserves metadata where supported

Example:

rclone sync synclyz:bucket1 aws:bucket2 \
    --progress \
    --transfers 128 \
    --checkers 128 \
    --fast-list

For same-provider or same-region S3 copies, rclone can use server-side copy, avoiding downloading data to your machine and dramatically increasing performance. Increasing --transfers and --checkers can significantly improve throughput depending on the provider.

Option 2 — s5cmd (Fastest CLI)

If your workload contains millions of objects, s5cmd is often substantially faster than the standard AWS CLI because it performs highly parallel operations.

Example:

s5cmd cp \
"s3://bucket1/*" \
"s3://bucket2/"

Best suited for:

  • Image hosting
  • CDN assets
  • Backup archives
  • Log storage

Community experience consistently recommends s5cmd for raw throughput, while rclone is preferred for more complex synchronization and verification workflows.

Option 3 — AWS DataSync

If both buckets are on AWS:

  • Automatic retries
  • Incremental sync
  • Scheduling
  • Verification
  • Cross-account transfers
  • Managed service

Ideal for:

  • Enterprise migration
  • Production workloads
  • Petabyte-scale datasets

AWS provides detailed guidance for configuring DataSync between S3 buckets, including cross-account transfers and IAM permissions.

Option 4 — AWS CLI

For small or medium migrations:

aws s3 sync \
s3://bucket1 \
s3://bucket2

Simple and reliable, but AWS notes that for buckets with millions of objects, you should consider S3 Batch Operations or parallelized approaches instead. AWS also recommends splitting transfers by prefixes and increasing concurrency to improve performance.

Option 5 — MinIO Client (mc mirror)

Excellent if you're running:

  • MinIO
  • Self-hosted object storage
  • S3-compatible services
  • Synclyz Object Storage

Example:

mc mirror \
source/bucket \
destination/bucket

Features:

  • Resume
  • Incremental sync
  • Verification
  • Delete sync
  • Metadata support

Enterprise Migration Strategy

For 100+ TB or tens of millions of objects:

  1. Inventory the source bucket.
  2. Copy historical data in parallel.
  3. Verify checksums and object counts.
  4. Run an incremental sync to capture changes.
  5. Freeze writes briefly (if needed).
  6. Perform a final delta sync.
  7. Switch applications to the new bucket.

This minimizes downtime and reduces risk.

Best Practices

Parallelism

Increase concurrency only as far as your provider and network can sustain.

Examples:

  • 32 transfers
  • 64 transfers
  • 128 transfers
  • 256 transfers (only after testing)

Multipart Upload

Enable multipart uploads for large objects (typically >100 MB). This improves throughput and resiliency against network interruptions.

Checksum Verification

Always verify copied data using checksums or provider-supported integrity checks before deleting the source.

Incremental Sync

Avoid copying the entire dataset again. Perform a final delta sync to transfer only changed objects.

Server-Side Copy

If both buckets are accessible within the same S3-compatible environment, server-side copy avoids downloading data through your own machine, reducing bandwidth usage and accelerating transfers.

Using Synclyz.com's S3 Object Storage

Because Synclyz.com's S3 Object Storage is S3-compatible, it can integrate with tools such as:

  • ✅ rclone
  • ✅ AWS CLI
  • ✅ MinIO Client
  • ✅ Cyberduck
  • ✅ s3cmd
  • ✅ s5cmd
  • ✅ boto3 (Python)
  • ✅ AWS SDKs
  • ✅ Go SDK
  • ✅ Java SDK
  • ✅ PHP SDK
  • ✅ Node.js SDK

A typical migration to or from Synclyz only requires:

  • Endpoint URL
  • Access Key ID
  • Secret Access Key
  • Bucket name
  • Region (if applicable)

Professional Recommendation

For most production environments:

  • Up to 5 TB: Use rclone with tuned parallelism.
  • 5–100 TB: Use rclone or s5cmd, then perform a delta sync before cutover.
  • AWS-to-AWS: Use AWS DataSync or native replication.
  • 100+ million objects: Partition the transfer by prefixes, run multiple workers in parallel, and validate integrity before switching production traffic.

This approach provides the best balance of speed, reliability, resumability, cost efficiency, and compatibility for modern S3-compatible object storage systems, including Synclyz.com's S3 Object Storage.

Post a Comment