md (software RAID) and lvm (logical volume management)

md

Building a RAID array using mdadm – two primary steps:

  1. “mdadm –create” to build the array using available resources
  2. “mdadm –detail –scan” to build config string for /etc/mdadm/mdadm.conf

Simple examples:

RAID6 array

  • Set up partitions to be used (in this case the whole disk):

    # for x in /dev/sd{b,c,d,e,f}1 ; do fdisk $x ; done
    

  • Create the array (in this case, with one hot-spare):

    # mdadm --create /dev/md0 --level=6 --raid-devices=4 --spare-devices=1 /dev/sd{b,c,d,e,f}1
    

  • Configure the array for reboot (append to the end of /etc/mdadm/mdadm.conf):

    # mdadm --detail --scan
    ARRAY /dev/md/0 metadata=1.2 spares=1 name=debian6-vm:0 UUID=9b42abcd:309fabcd:6bfbabcd:298dabcd
    

Considerations when setting up the partitions might be that any replacement disks will need to support that same size partition. Unconfirmed but it sounds like it might be a reasonable concern: “Enter a value smaller than the free space value minus 2% or the disk size to make sure that when you will later install a new disk in replacement of a failed one, you will have at least the same capacity even if the number of cylinders is different.” (http://www.jerryweb.org/settings/raid/)

RAID1 array

Pretty much the same process – but for reference here’s the simpler RAID1 config without any spare:

# mdadm --create /dev/md1 --level=1 --raid-devices=2 /dev/sd{g,h}1

And here’s how the config looks:

ARRAY /dev/md1 metadata=1.2 name=debian6-vm:1 UUID=fa94abcd:317fabcd:3a61abcd:4932abcd

/proc/mdstat

The current state of the RAID arrays:

# cat /proc/mdstat 
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md1 : active raid1 sdh1[1] sdg1[0]
      10481285 blocks super 1.2 [2/2] [UU]
      
md0 : active raid6 sdb1[0] sdf1[4](S) sde1[3] sdd1[2] sdc1[1]
      16769024 blocks super 1.2 level 6, 512k chunk, algorithm 2 [4/4] [UUUU]
      
unused devices: <none>

replacing a failed disk

Some handy notes here:

lvm

Regardless of using RAID or plain disks, lvm allows much more flexibility in allocating that space. Not any use whatsoever on a smaller machine which is using a single large partition for everything (but even then, use of lvm could allow new disk to be added to extend a filesystem if space is running low).

For some very details documentation: http://tldp.org/HOWTO/LVM-HOWTO/

The steps:

  1. “pvcreate” to initialise partitions to be used by lvm
  2. “vgcreate” to create volume groups (these contain physical devices)
  3. “lvcreate” to define logical volumes (these are then used as disk partitions)

pvcreate

Partitions can’t be used until they’re initialised:

(this is run in testmode with -t to show what would happen)

# pvcreate -t -v /dev/md0
  Test mode: Metadata will NOT be updated.
    Test mode: Wiping internal cache
    Wiping internal VG cache

The physical volume (PV) will now be listed as available (along with those already allocated to a group). In this example all PVs are already allocated to groups:

# pvs
  PV         VG          Fmt  Attr PSize  PFree
  /dev/md0   brad_group1 lvm2 a-   15.99g 9.35g
  /dev/md1   brad_group1 lvm2 a-    9.99g 9.02g

vgcreate (and vgextend)

In order to use PVs they need to be allocated to a volume group (VG) within lvm.

When a group is created it starts with a number of initialised PVs:

# vgcreate brad_group1 /dev/md0

Later on more PVs can be added to a VG with vgextend:

# vgextend /dev/brad_group1 /dev/md1

Something determined at this stage is the “physical extent size” (defaulting to 4MB) – this is the volume chunk size of all PVs within the group. Space is allocated to logical volumes in whole physical extent chunks.

lvcreate (and lvextend)

Now that we have PVs and VGs it’s possible to create logical volumes (LVs) and use them.

There’s lots of options here, but here’s a simple example which uses extents, though it’s also possible to ask for a specific size:

# lvcreate -l 250 /dev/brad_group1

It’s also possible to ask for a specific LV name (if not, one is auto-generated).

This LV can now be formatted and mounted:

# mkfs.xfs /dev/brad_group1/lvol0
# mount /dev/brad_group1/lvol0 /mnt/test

If the size of an LV needs to be changed the lvextend program can be used:

# lvextend -l +1000 /dev/brad_group1/lvol0

Note: that this only changes the size of the LV, not the filesystem. Various filesystems have different approaches (see http://tldp.org/HOWTO/LVM-HOWTO/extendlv.html), xfs happens to be quite easy for growth but it doesn’t support shrinking:

# lvextend -l +1000 /dev/brad_group1/lvol0
# xfs_growfs /mnt/test

There’s some other stuff going on here as well, for instance it’s possible to set up a LV to stripe across PVs in the VG. This cane be seen using lvdisplay:

# lvcreate --extents 500 --stripes 2 /dev/brad_group1
# lvdisplay -m /dev/brad_group1/lvol2
  --- Logical volume ---
  LV Name                /dev/brad_group1/lvol2
  VG Name                brad_group1
  LV UUID                20t1dj-bwfQ-rkgf-wZFq-J2Oj-miI8-L4MKhh
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                1.95 GiB
  Current LE             500
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     512
  Block device           253:2
   
  --- Segments ---
  Logical extent 0 to 499:
    Type                striped
    Stripes             2
    Stripe size         64.00 KiB
    Stripe 0:
      Physical volume   /dev/md0
      Physical extents  1450 to 1699
    Stripe 1:
      Physical volume   /dev/md1
      Physical extents  0 to 249

displaying information

Various commands show info about the lvm configuration, examples follow.

pvs

# pvs
  PV         VG          Fmt  Attr PSize  PFree
  /dev/md0   brad_group1 lvm2 a-   15.99g 9.35g
  /dev/md1   brad_group1 lvm2 a-    9.99g 9.02g

pvdisplay

# pvdisplay 
  --- Physical volume ---
  PV Name               /dev/md0
  VG Name               brad_group1
  PV Size               15.99 GiB / not usable 4.00 MiB
  Allocatable           yes 
  PE Size               4.00 MiB
  Total PE              4093
  Free PE               2393
  Allocated PE          1700
  PV UUID               OFyENQ-lRD8-jPoW-aEr4-MY8j-WMqN-pZYy2H
   
  --- Physical volume ---
  PV Name               /dev/md1
  VG Name               brad_group1
  PV Size               10.00 GiB / not usable 3.63 MiB
  Allocatable           yes 
  PE Size               4.00 MiB
  Total PE              2558
  Free PE               2308
  Allocated PE          250
  PV UUID               1WsYxg-YnSN-VKme-E0iw-gX6Y-KEcI-VUckAb

vgs

# vgs
  VG          #PV #LV #SN Attr   VSize  VFree 
  brad_group1   2   3   0 wz--n- 25.98g 18.36g

vgdisplay

# vgdisplay 
  --- Volume group ---
  VG Name               brad_group1
  System ID             
  Format                lvm2
  Metadata Areas        2
  Metadata Sequence No  7
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                3
  Open LV               3
  Max PV                0
  Cur PV                2
  Act PV                2
  VG Size               25.98 GiB
  PE Size               4.00 MiB
  Total PE              6651
  Alloc PE / Size       1950 / 7.62 GiB
  Free  PE / Size       4701 / 18.36 GiB
  VG UUID               BA5Vkh-SrmB-JOr6-vAz0-48Eh-YI6o-m3Ghvr

lvs

# lvs
  LV    VG          Attr   LSize    Origin Snap%  Move Log Copy%  Convert
  lvol0 brad_group1 -wi-ao    4.69g                                      
  lvol1 brad_group1 -wi-ao 1000.00m                                      
  lvol2 brad_group1 -wi-ao    1.95g

lvdisplay

# lvdisplay 
  --- Logical volume ---
  LV Name                /dev/brad_group1/lvol0
  VG Name                brad_group1
  LV UUID                06M0bJ-RaDR-fVNz-weRo-pVAz-wTMl-17xr5G
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                4.69 GiB
  Current LE             1200
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     4096
  Block device           253:0
   
  --- Logical volume ---
  LV Name                /dev/brad_group1/lvol1
  VG Name                brad_group1
  LV UUID                b0a2x6-Qxkl-sAX1-uz2y-KVFK-qiJu-IBodH7
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                1000.00 MiB
  Current LE             250
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     4096
  Block device           253:1
   
  --- Logical volume ---
  LV Name                /dev/brad_group1/lvol2
  VG Name                brad_group1
  LV UUID                20t1dj-bwfQ-rkgf-wZFq-J2Oj-miI8-L4MKhh
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                1.95 GiB
  Current LE             500
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     512
  Block device           253:2

Leave a Reply

Your email address will not be published. Required fields are marked *