Skip to main content

Enterprise Block Storage: Operations, Lifecycle & Benchmarking

Architectural Paradigm: Block storage exposes a linear, raw sequence of addressable sectors (LBAs) directly to the operating system, delegating all metadata, directory hierarchy, and consistency guarantees to the host filesystem or database engine.

1. SAN vs. NAS: Architectural Dissection​

+-----------------------------------+-----------------------------------+
| SAN (Storage Area Network)        | NAS (Network Attached Storage)    |
+-----------------------------------+-----------------------------------+
| Protocol: NVMe-oF, iSCSI, FC      | Protocol: NFSv4, SMB3, CephFS     |
| Transport: PCIe, Ethernet, Fibre  | Transport: IP / TCP / RDMA        |
| Data Unit: Fixed Sectors (512B/4K)| Data Unit: Hierarchical File Tree |
| Filesystem Location: Host Kernel  | Filesystem Location: Remote Server|
| Locking: Distributed lock manager | Locking: Server-managed (NLM/NFSv4|
| Latency Profile: 10 µs - 200 µs   | Latency Profile: 500 µs - 5 ms    |
| Best For: Databases, Hypervisors  | Best For: Shared home dirs, K8s RWX|
+-----------------------------------+-----------------------------------+

2. Production Block Device Lifecycle: Format, Filesystem & Mount​

Step 1: Discover and Inspect Block Topology​

# Display full block device tree, UUIDs, transport types, and rotational status
lsblk -o NAME,PATH,SIZE,FSTYPE,UUID,ROTA,DISC-GRAN,MODEL,TRAN

# Query low-level SCSI / SAS details
lsscsi -v

# Inspect hardware queue parameters and physical sector geometry
cat /sys/block/nvme0n1/queue/hw_sector_size
cat /sys/block/nvme0n1/queue/physical_block_size

Step 2: 1 MiB Partition Alignment with parted​

Partition misalignment causes every logical 4K I/O to cross physical sector boundaries, creating a severe Read-Modify-Write penalty.

# Create a GPT partition table aligned to 1 MiB boundary
sudo parted -s /dev/sdb mklabel gpt
sudo parted -s /dev/sdb mkpart primary 1MiB 100%

# Verify optimal alignment
sudo parted /dev/sdb align-check optimal 1

Step 3: Filesystem Creation & Parameter Tuning​

# Format XFS for high-concurrency database workloads
# -b size=4096: 4KB block size
# -m reflink=1: Copy-on-Write reflink support enabled
# -d agcount=16: 16 Allocation Groups to maximize concurrent CPU write threads
sudo mkfs.xfs -f -b size=4096 -m reflink=1 -d agcount=16 /dev/sdb1

# Alternatively, format ext4 with 4KB blocks and reserved block percentage set to 1%
sudo mkfs.ext4 -b 4096 -m 1 -O 64bit,has_journal,extents /dev/sdb1

Step 4: Production Persistent Mounting (/etc/fstab)​

# Query the unique filesystem UUID
FS_UUID=$(sudo blkid -s UUID -o value /dev/sdb1)

# Append to /etc/fstab with performance tuning flags
echo "UUID=${FS_UUID} /mnt/production_data xfs noatime,nodiratime,logbufs=8,logbsize=256k,allocsize=64M 0 2" | sudo tee -a /etc/fstab

# Mount and verify
sudo mkdir -p /mnt/production_data
sudo mount -a
mount | grep /mnt/production_data

[!TIP] Essential Production Mount Options:

  • noatime,nodiratime: Eliminates metadata write overhead when reading files.
  • logbufs=8,logbsize=256k: Buffers filesystem journal operations in memory before committing to disk.
  • allocsize=64M: Allocates speculative extents in 64 MB chunks to eliminate physical file fragmentation.

3. Direct Read/Write Operations & Raw Disk Testing​

# 1. Bypass OS Page Cache to test raw disk write speed using dd
sudo dd if=/dev/zero of=/dev/sdb1 bs=1M count=1024 oflag=direct status=progress

# 2. Issue a raw SCSI INQUIRY command directly to a block device
sudo sg_raw -r 36 /dev/sdb 12 00 00 00 24 00

# 3. Securely TRIM / deallocate the entire raw SSD namespace
sudo blkdiscard -v /dev/sdb

4. Enterprise I/O Benchmarking Tools​

1. FIO (Flexible I/O Tester): The Gold Standard​

FIO simulates virtually any I/O pattern. Below is an enterprise-grade job file testing mixed OLTP database concurrency:

# database-oltp.fio
[global]
ioengine=libaio
direct=1
runtime=60
time_based=1
group_reporting=1
filename=/dev/sdb1

[oltp-mixed-workload]
rw=randrw
rwmixread=70
bs=8k
iodepth=32
numjobs=4
rate_iops=50000
fio database-oltp.fio --output=fio_results.json --output-format=json

2. Oracle Vdbench: Multi-Threaded Validation & Data Integrity Verification​

Vdbench is Oracle’s multi-threaded storage validation tool, famed for its ability to detect silent data corruption by writing cryptographic checksum patterns and validating them on subsequent reads.

Create parameter file vdbench_script.par:

* Storage Definition (Physical raw block device)
sd=sd1,lun=/dev/sdb1,openflag=o_direct

* Workload Definition: 70% 8K random reads, 30% 8K random writes
wd=wd1,sd=sd1,rdpct=70,rhpct=0,seekpct=100,xfersize=8k

* Run Definition: Concurrency of 64 threads, 60 second duration
rd=rd1,wd=wd1,iorate=max,elapsed=60,warmup=10,interval=1,threads=64,data_validation=yes

Run the benchmark:

./vdbench -f vdbench_script.par -o /tmp/vdbench_results/
  • data_validation=yes: Embeds logical block address (LBA) numbers and timestamp signatures into every written block. If a controller returns stale data, Vdbench halts immediately with an integrity breach alert.

3. Filebench: Macro-Workload Modeling​

Filebench models macro application behaviors (fileservers, webservers, mail servers) rather than micro block patterns.

Create workload model fileserver.f:

define fileset name=bigfileset,path=/mnt/production_data,size=16k,entries=10000,dirwidth=20,prealloc=100

define process name=fileserver,instances=4 {
  thread name=worker,instances=8 {
    flowop createfile name=create1,fileset=bigfileset,fd=1
    flowop writewholefile name=wfile1,srcfd=1,fd=1
    flowop closefile name=close1,fd=1
    flowop readwholefile name=rfile1,fileset=bigfileset,fd=1
    flowop closefile name=close2,fd=1
    flowop deletefile name=dfile1,fileset=bigfileset
  }
}

run 60
filebench -f fileserver.f