06. Block Storage, LVM & RAID: Linux Device Mapper, mdadm & Parity Mathematics

Prerequisites: Module 01 (Storage Metrics & IOPS), Module 02 (Linux I/O Path & Block Layer).
Target Audience: Systems Engineers, Infrastructure Architects, Database SREs, and Linux Storage Administrators.

Block storage operates at the raw hardware interface, exposing unformatted arrays of fixed-size sectors with no filesystem semantics. This guide dissects the Linux Device Mapper (dm) subsystem, Logical Volume Management (LVM2), software RAID architectures (mdadm), parity calculations, and the mathematical trade-offs between rebuild times and data loss risks.

---

1. The Linux Storage Stack & The Device Mapper Subsystem

In modern Linux kernels, the path from physical disks to mounted filesystems traverses the Device Mapper and Multi-Queue Block Layer:

+-----------------------------------------------------------------------------------+
| Filesystem (Ext4, XFS) / Direct Database Engine (PostgreSQL, Oracle)              |
+-----------------------------------------|-----------------------------------------+
                                          ▼
+-----------------------------------------------------------------------------------+
| Logical Volume: /dev/mapper/vg_prod-lv_database  (Minor dev: dm-0)                |
| Linux Device Mapper Framework (dm-linear, dm-thin, dm-crypt, dm-cache)            |
+-----------------------------------------|-----------------------------------------+
                                          ▼
+-----------------------------------------------------------------------------------+
| Software RAID Layer: /dev/md0 (Linux Multiple Devices mdadm driver)               |
| Handles striping, mirroring, and parity checksum calculations                     |
+-----------------------------------------|-----------------------------------------+
                                          ▼
+-----------------------------------------------------------------------------------+
| Physical Block Devices: /dev/sda, /dev/sdb, /dev/nvme0n1                          |
+-----------------------------------------------------------------------------------+

Device Mapper Target Types

The Device Mapper is a modular kernel framework for constructing virtual block devices:

• dm-linear: Concatenates ranges of blocks from multiple underlying physical drives into a single contiguous linear address space.
• dm-striped: Stripes reads and writes across multiple physical disks in round-robin chunks (similar to RAID 0).
• dm-crypt: Transparent disk encryption using Linux Crypto API (LUKS).
• dm-thin: Implements dynamic thin-provisioning and copy-on-write snapshotting from a shared storage pool.
• dm-cache / dm-writecache: Uses small, fast NVMe SSDs as read/write caches in front of large, slower mechanical hard drives.

---

2. Logical Volume Manager (LVM2) Architecture

LVM abstracts physical storage media into flexible, resizable logical volumes:

Physical Disks:    [ /dev/sdb ] (500 GB)             [ /dev/sdc ] (500 GB)
                         │                                 │
                         ▼                                 ▼
Physical Volumes:  [ PV 1: 125,000 PEs ]             [ PV 2: 125,000 PEs ]
                         └─────────────────┬───────────────┘
                                           ▼
Volume Group (VG):               [ vg_production ]
                                 Total: 250,000 Physical Extents (4 MB each = 1,000 GB)
                                           │
                         ┌─────────────────┴─────────────────┐
                         ▼                                   ▼
Logical Volumes (LV): [ lv_database ] (600 GB)            [ lv_backups ] (400 GB)
                      (150,000 PEs)                       (100,000 PEs)

1. Physical Volumes (PV)

A raw block device (/dev/sdb, /dev/nvme0n1, or a RAID partition /dev/md0) initialized for LVM with pvcreate. LVM writes a metadata header at sector 0 describing the volume group membership.

2. Volume Groups (VG)

A unified storage pool formed by aggregating one or more Physical Volumes. The VG slices storage into uniform allocation chunks called Physical Extents (PE) (default $4\text{ MB}$).

3. Logical Volumes (LV)

Virtual block devices carved out of a Volume Group. Applications format filesystems directly on LVs (/dev/vg_production/lv_database):

• Linear LVs: Allocated sequentially from available PEs across one or more PVs.
• Striped LVs: PEs are striped across multiple PVs for higher bandwidth.
• Thinly Provisioned LVs: Virtual block devices whose physical extents are allocated dynamically on-demand from a thin-pool as data is written.

[!CAUTION] In an LVM Thin Pool, administrators can easily overcommit storage (e.g., creating 2 TB of virtual LVs on a 500 GB physical pool). If physical pool space reaches 100%, all write operations to all thin volumes block or fail with I/O errors, causing immediate database crashes.

---

3. RAID Architectures & The Parity Write Penalty

RAID (Redundant Array of Independent Disks) balances performance, fault tolerance, and capacity efficiency.

RAID 0 (Striping):          RAID 1 (Mirroring):         RAID 5 (Distributed Parity):
  Disk 0      Disk 1          Disk 0      Disk 1          Disk 0      Disk 1      Disk 2
┌────────┐  ┌────────┐      ┌────────┐  ┌────────┐      ┌────────┐  ┌────────┐  ┌────────┐
│ Chunk 0│  │ Chunk 1│      │ Chunk 0│  │ Chunk 0│      │ Chunk 0│  │ Chunk 1│  │Parity 0│
├────────┤  ├────────┤      ├────────┤  ├────────┤      ├────────┤  ├────────┤  ├────────┤
│ Chunk 2│  │ Chunk 3│      │ Chunk 1│  │ Chunk 1│      │ Chunk 2│  │Parity 1│  │ Chunk 3│
├────────┤  ├────────┤      ├────────┤  ├────────┤      ├────────┤  ├────────┤  ├────────┤
│ Chunk 4│  │ Chunk 5│      │ Chunk 2│  │ Chunk 2│      │Parity 2│  │ Chunk 4│  │ Chunk 5│
└────────┘  └────────┘      └────────┘  └────────┘      └────────┘  └────────┘  └────────┘
Capacity: N * S             Capacity: 1 * S             Capacity: (N - 1) * S
Faults: 0 Disks             Faults: 1 Disk              Faults: 1 Disk

Comprehensive RAID Comparison Matrix

RAID Level	Description	Capacity Efficiency	Read Performance	Write Performance	Fault Tolerance	The Write Penalty
RAID 0	Block Striping	$N \times S$ ($100\%$)	$N \times \text{Speed}$	$N \times \text{Speed}$	0 Disks (1 failure destroys array)	$1$ (None)
RAID 1	Mirroring	$1 \times S$ ($50\%$ for 2 disks)	$N \times \text{Speed}$	$1 \times \text{Speed}$	$N - 1$ Disks	$2$ (2 writes per host write)
RAID 5	Distributed Parity	$(N - 1) \times S$	$(N - 1) \times \text{Speed}$	Degraded by parity RMW	1 Disk	4 ($2\text{ reads} + 2\text{ writes}$)
RAID 6	Dual Distributed Parity	$(N - 2) \times S$	$(N - 2) \times \text{Speed}$	Severe parity overhead	2 Disks	6 ($3\text{ reads} + 3\text{ writes}$)
RAID 10	Striped Mirrors (1+0)	$\frac{N}{2} \times S$ ($50\%$)	$N \times \text{Speed}$	$\frac{N}{2} \times \text{Speed}$	1 per mirror group (up to $\frac{N}{2}$)	2 (2 writes, zero parity)

---

4. Governing Mathematical Formulations

1. The RAID 5 / RAID 6 Write Penalty Equation

In RAID 5, parity is calculated via XOR logic: $P = D_0 \oplus D_1 \oplus D_2$

When an application modifies a single data chunk $D_0 \to D_0'$, the controller cannot rewrite the entire stripe without reading all other disks. Instead, it computes new parity using differential XOR: $P_{\text{new}} = P_{\text{old}} \oplus D_{\text{old}} \oplus D_{\text{new}}$

This forces a Read-Modify-Write (RMW) sequence:

1. Read $D_{\text{old}}$
2. Read $P_{\text{old}}$
3. Write $D_{\text{new}}$
4. Write $P_{\text{new}}$

$\text{RAID 5 Write Penalty} = 4\text{ Physical Disk Operations per Host Write}$

For RAID 6 (Dual Parity $P$ and $Q$ across Galois fields $\text{GF}(2^8)$): $\text{RAID 6 Write Penalty} = 6\text{ Physical Disk Operations per Host Write (3 reads + 3 writes)}$

2. Sizing Backend Disk IOPS for RAID Arrays

To size a storage array for an application workload with a specific read/write ratio:

$\text{Backend Disk IOPS} = (\text{Host Read IOPS}) + (\text{Host Write IOPS} \times \text{Write Penalty})$

$\text{Disks Required} = \frac{\text{Backend Disk IOPS}}{\text{Max IOPS per Single Disk}}$

---

5. Hands-on Linux Lab: LVM & Software RAID Management

Step 1: Create a High-Performance RAID 10 Array with mdadm

# Create a 4-disk RAID 10 array with a 256 KB chunk size
sudo mdadm --create /dev/md0 \
    --level=10 \
    --raid-devices=4 \
    --chunk=256 \
    /dev/sdb /dev/sdc /dev/sdd /dev/sde

# Monitor initial sync and rebuild progress
cat /proc/mdstat

Step 2: Initialize LVM on the RAID Array

# 1. Create Physical Volume on the RAID device
sudo pvcreate /dev/md0

# 2. Create Volume Group
sudo vgcreate vg_storage /dev/md0

# 3. Create Logical Volume with 100% of available space
sudo lvcreate -l 100%FREE -n lv_database vg_storage

# 4. Format with XFS
sudo mkfs.xfs /dev/vg_storage/lv_database

Step 3: Online Volume Extension

LVM allows online, zero-downtime volume resizing:

# Extend logical volume by 100 GB
sudo lvextend -L +100G /dev/vg_storage/lv_database

# Expand filesystem online to consume the new space
# For XFS:
sudo xfs_growfs /mnt/database
# For Ext4:
# sudo resize2fs /dev/vg_storage/lv_database

---

6. Real-World Production Failure Scenarios

Failure Scenario 1: The RAID 5 Rebuild Death Spiral

Incident

A cloud storage pod with an 8-disk RAID 5 array of 16 TB enterprise mechanical drives experienced a single drive failure. The spare drive was slotted in and rebuild began. At 78% rebuild progress (14 hours into the resilver), a second drive threw an Unrecoverable Read Error (URE). The entire 112 TB array crashed permanently, resulting in catastrophic total data loss.

Root Cause Analysis

During a RAID 5 rebuild, every single remaining sector on all surviving drives must be read sequentially without error to recalculate the missing drive's data.

• Reading $7 \times 16\text{ TB} = 112\text{ TB} = 8.96 \times 10^{14}\text{ bits}$.
• With an industry standard SATA UBER of $10^{-14}$, the mathematical probability of encountering an uncorrectable bit error across 112 TB exceeds $99.9\%$!

Remediation

Never deploy RAID 5 on drives larger than 2 TB. For large-capacity drives, always use RAID 6 (dual parity), RAID 10, or Erasure Coding ($k+m$, $m \ge 2$).

---

Failure Scenario 2: The Classic RAID Write Hole

Incident

A database server suffered a hard power loss. Upon reboot, the software RAID 5 array appeared clean, but database checksum queries began throwing table corruption errors across files that were being written during the power failure.

Root Cause

The RAID Write Hole:

• Updating a block requires writing both data and parity chunks.
• Power dropped precisely after writing the data chunk, but before writing the parity chunk.
• The RAID array state became inconsistent: data and parity no longer matched, but the controller had no way of knowing which chunk was valid.

Solution

1. In mdadm, configure a write-intent bitmap:

   sudo mdadm --grow /dev/md0 --bitmap=internal

2. Or configure a dedicated journaling device for mdadm (--write-journal=/dev/nvme0n1).
3. Or use OpenZFS / Btrfs, which eliminate the write hole entirely through atomic Copy-on-Write Merkle trees.

---

7. Practical Engineering Exercises (With Solutions)

Exercise: Database Backend IOPS on RAID 5 vs. RAID 10

Problem: An OLTP database workload generates $15,000\text{ Read IOPS}$ and $5,000\text{ Write IOPS}$. You plan to deploy on an array of enterprise mechanical HDDs, each capable of delivering $150\text{ IOPS}$.

1. How many backend disk IOPS are generated under RAID 5?
2. How many backend disk IOPS are generated under RAID 10?
3. How many physical disks are required to sustain this workload for each RAID level?

Solution:

1. RAID 5 Calculation: $\text{Write Penalty} = 4$ $\text{Backend IOPS} = 15,000 + (5,000 \times 4) = 15,000 + 20,000 = 35,000\text{ Disk IOPS}$ $\text{Disks Required} = \frac{35,000}{150} \approx 234\text{ Disks}$
2. RAID 10 Calculation: $\text{Write Penalty} = 2$ $\text{Backend IOPS} = 15,000 + (5,000 \times 2) = 15,000 + 10,000 = 25,000\text{ Disk IOPS}$ $\text{Disks Required} = \frac{25,000}{150} \approx 167\text{ Disks}$

• Engineering Conclusion: Despite RAID 10 having a 50% capacity overhead compared to RAID 5's higher storage efficiency, RAID 10 requires 67 fewer physical drives to meet the transaction performance requirement because it avoids the 4x write penalty!

---

8. Summary Checklist & Key Takeaways

1. Beware the RAID 5 Write Penalty: Every host write triggers 4 disk I/O operations ($2\text{ reads} + 2\text{ writes}$), severely bottlenecking write-intensive workloads.
2. Never Use RAID 5 on Drives $> 2\text{ TB}$: High-capacity disk rebuilds will almost certainly fail due to Unrecoverable Bit Error Rates (UBER).
3. RAID 10 for Random Write Databases: Choose RAID 10 for databases; it offers a 2x write penalty with zero parity calculation overhead.
4. Monitor LVM Thin Pools Closely: Overcommitting thin pools without strict capacity alerting risks catastrophic total container crashes.
5. Mitigate the Write Hole: Always enable internal bitmaps or journaling on software RAID 5/6 arrays.