Skip to main content

dd (Data Duplicator): POSIX Baseline I/O & Block Operations

Origin: Derived from IBM OS/360 JCL DD statement; part of GNU Coreutils and POSIX standard.
Applicability: Zero-dependency raw block copying, partition backups, filesystem zeroing, secure cryptographic wiping, and sanity baseline benchmarking.

While modern benchmarks rely on asynchronous engines like FIO, dd remains the universal tool installed on virtually every Unix-like operating system in existence. Understanding its synchronization flags, cache interactions, and block-sizing behavior is critical for storage sanity checks and disk manipulation.

1. The Core Architecture & Flag Matrix​

                      +-------------------+
                      |   Source (if=)    |
                      +---------+---------+
                                |
                   [iflag=direct / iflag=sync]
                                |
                                v
               +---------------------------------+
               |   dd Buffer (bs=Block Size)     |
               +----------------+----------------+
                                |
                   [oflag=direct / conv=fdatasync]
                                |
                                v
                      +---------+---------+
                      | Target Device/File|
                      +-------------------+

Critical Flags & Cache Traps​

  • bs=BYTES: Sets both input and output block sizes simultaneously. Too small (e.g., bs=512) causes millions of system calls and severe CPU bottlenecking.
  • oflag=direct: Uses O_DIRECT to write directly to storage, completely bypassing the host Page Cache.
  • conv=fdatasync: Flushes output data from volatile kernel buffers to non-volatile physical media before exiting and reporting elapsed time. Without this flag, dd reports the speed of writing into host RAM!
  • status=progress: Prints continuous transfer rates and bytes transferred to stderr.

2. Installation & Verification​

dd is pre-installed on every Linux, BSD, and macOS system as part of the core operating system.

# Check version and features
dd --version

# Display usage overview
dd --help

3. The 10 Essential dd Workload Snippets​

Snippet 1: Beginner Sequential Write (Buffered Baseline)​

Objective: Write 1 GiB of data with standard buffered I/O and display real-time progress.

dd if=/dev/zero of=test_1g.img bs=1M count=1024 status=progress

Snippet 2: Direct I/O Sequential Write (Bypassing Page Cache)​

Objective: Measure true hardware write throughput without RAM buffering by passing oflag=direct.

dd if=/dev/zero of=direct_write.img bs=1M count=2048 oflag=direct status=progress

Snippet 3: Synchronous Durability Flush (conv=fdatasync)​

Objective: Ensure data is physically committed to flash/disk platters before timing completes, preventing false reporting.

dd if=/dev/zero of=fdatasync_write.img bs=1M count=2048 conv=fdatasync status=progress

Snippet 4: Direct I/O Sequential Read Benchmark​

Objective: Read directly from physical media into /dev/null without populating or benefiting from the Linux Page Cache.

# First create the target file
dd if=/dev/zero of=read_target.img bs=1M count=2048 conv=fdatasync

# Benchmark direct read throughput
dd if=read_target.img of=/dev/null bs=1M iflag=direct status=progress

Snippet 5: Block Size Scaling Sweep (Sweet-Spot Analysis)​

Objective: Run a sequential sweep across 4K, 64K, 1M, and 4M block sizes to identify the controller's optimal transfer size.

for bs in 4K 64K 1M 4M; do
  echo "--- Testing Block Size: $bs ---"
  dd if=/dev/zero of=sweep_${bs}.dat bs=$bs count=1000 oflag=direct status=none conv=fdatasync 2>&1
  sync
done

Snippet 6: Raw Block Device Cloning / Mirroring​

Objective: Clone an entire block device sector-by-sector to another drive with alignment safety.

sudo dd if=/dev/nvme0n1 of=/dev/nvme1n1 bs=64K conv=noerror,sync status=progress

Note: conv=noerror,sync ensures that read errors do not abort the copy and missing blocks are padded with null bytes to preserve partition offsets.

Snippet 7: Zeroing Out a Block Device / Partition Table (Wipe)​

Objective: Overwrite the first 100 MiB of a drive to destroy GPT, MBR, and filesystem superblocks before re-provisioning.

sudo dd if=/dev/zero of=/dev/sdb bs=1M count=100 oflag=direct status=progress

Snippet 8: Cryptographic Random Data Preconditioning (Uncompressible)​

Objective: Write cryptographically secure pseudo-random data to prevent hardware compression controllers from inflating metrics.

dd if=/dev/urandom of=uncompressible_test.dat bs=1M count=1024 status=progress

Snippet 9: Storage Deletion & Sparse File Truncation (Zero-Seek)​

Objective: Instantly truncate or punch holes in an existing file to release space back to the filesystem without reading data.

# Create a 10 GiB sparse file in 1 millisecond
dd if=/dev/null of=sparse_image.img bs=1M seek=10240

# Check sparse vs allocated disk space
ls -lh sparse_image.img
du -sh sparse_image.img

# Truncate / zero out file instantly
dd if=/dev/null of=sparse_image.img count=0

Snippet 10: Real-Time Signal Profiling (USR1) & Pipe Viewer (pv)​

Objective: Monitor transfer speeds of long-running dd background processes or pipe streams.

# Terminal 1: Launch background dd copy
dd if=/dev/nvme0n1 of=/mnt/backup/nvme_backup.raw bs=4M oflag=direct &
DD_PID=$!

# Send USR1 signal to print current status without interrupting execution
kill -USR1 $DD_PID

# Alternative: Precise real-time bandwidth meter with pipe viewer (pv)
dd if=/dev/sdb bs=1M | pv -r -b -a | dd of=/dev/sdc bs=1M oflag=direct

4. Troubleshooting & Gotchas with dd​

  1. The In-Flight Cache Illusion:

    # DANGEROUS: Measures RAM speed, NOT disk speed!
    dd if=/dev/zero of=fast_test.img bs=1M count=1000
    # Shows 4.8 GB/s on a 500 MB/s SATA disk!

    Solution: Always use oflag=direct or conv=fdatasync.

  2. Accidental Drive Overwrites ("Disk Destroyer"): Swapping if (input file) and of (output file) will instantly destroy data on the target drive. Always verify device letters with lsblk -o NAME,SIZE,TYPE,MOUNTPOINT before executing dd.