md
Building a RAID array using mdadm - two primary steps:
- “mdadm –create” to build the array using available resources
- “mdadm –detail –scan” to build config string for /etc/mdadm/mdadm.conf
Simple examples:
RAID6 array
- Set up partitions to be used (in this case the whole disk):
# for x in /dev/sd{b,c,d,e,f}1 ; do fdisk $x ; done
- Create the array (in this case, with one hot-spare):
# mdadm --create /dev/md0 --level=6 --raid-devices=4 --spare-devices=1 /dev/sd{b,c,d,e,f}1
- Configure the array for reboot (append to the end of /etc/mdadm/mdadm.conf):
# mdadm --detail --scan
ARRAY /dev/md/0 metadata=1.2 spares=1 name=debian6-vm:0 UUID=9b42abcd:309fabcd:6bfbabcd:298dabcd
Considerations when setting up the partitions might be that any replacement disks will need to support that same size partition. Unconfirmed but it sounds like it might be a reasonable concern: “Enter a value smaller than the free space value minus 2% or the disk size to make sure that when you will later install a new disk in replacement of a failed one, you will have at least the same capacity even if the number of cylinders is different.” (http://www.jerryweb.org/settings/raid/)
RAID1 array
Pretty much the same process - but for reference here’s the simpler RAID1 config without any spare:
# mdadm --create /dev/md1 --level=1 --raid-devices=2 /dev/sd{g,h}1
And here’s how the config looks:
ARRAY /dev/md1 metadata=1.2 name=debian6-vm:1 UUID=fa94abcd:317fabcd:3a61abcd:4932abcd
/proc/mdstat
The current state of the RAID arrays:
# cat /proc/mdstat
Personalities :
md1 : active raid1 sdh1 sdg1
10481285 blocks super 1.2
md0 : active raid6 sdb1 sdf1[4](S) sde1 sdd1 sdc1
16769024 blocks super 1.2 level 6, 512k chunk, algorithm 2
unused devices: <none>
replacing a failed disk
Some handy notes here:
- http://www.howtoforge.com/replacing_hard_disks_in_a_raid1_array
- http://consultancy.edvoncken.net/index.php/HOWTO_Replace_a_failing_disk_on_Linux_Software_RAID-5
lvm
Regardless of using RAID or plain disks, lvm allows much more flexibility in allocating that space. Not any use whatsoever on a smaller machine which is using a single large partition for everything (but even then, use of lvm could allow new disk to be added to extend a filesystem if space is running low).
For some very details documentation: http://tldp.org/HOWTO/LVM-HOWTO/
The steps:
- “pvcreate” to initialise partitions to be used by lvm
- “vgcreate” to create volume groups (these contain physical devices)
- “lvcreate” to define logical volumes (these are then used as disk partitions)
pvcreate
Partitions can’t be used until they’re initialised:
(this is run in testmode with -t to show what would happen)
# pvcreate -t -v /dev/md0
Test mode: Metadata will NOT be updated.
Test mode: Wiping internal cache
Wiping internal VG cache
The physical volume (PV) will now be listed as available (along with those already allocated to a group). In this example all PVs are already allocated to groups:
# pvs
PV VG Fmt Attr PSize PFree
/dev/md0 brad_group1 lvm2 a- 15.99g 9.35g
/dev/md1 brad_group1 lvm2 a- 9.99g 9.02g
vgcreate (and vgextend)
In order to use PVs they need to be allocated to a volume group (VG) within lvm.
When a group is created it starts with a number of initialised PVs:
# vgcreate brad_group1 /dev/md0
Later on more PVs can be added to a VG with vgextend:
# vgextend /dev/brad_group1 /dev/md1
Something determined at this stage is the “physical extent size” (defaulting to 4MB) - this is the volume chunk size of all PVs within the group. Space is allocated to logical volumes in whole physical extent chunks.
lvcreate (and lvextend)
Now that we have PVs and VGs it’s possible to create logical volumes (LVs) and use them.
There’s lots of options here, but here’s a simple example which uses extents, though it’s also possible to ask for a specific size:
# lvcreate -l 250 /dev/brad_group1
It’s also possible to ask for a specific LV name (if not, one is auto-generated).
This LV can now be formatted and mounted:
# mkfs.xfs /dev/brad_group1/lvol0
# mount /dev/brad_group1/lvol0 /mnt/test
If the size of an LV needs to be changed the lvextend program can be used:
# lvextend -l +1000 /dev/brad_group1/lvol0
Note: that this only changes the size of the LV, not the filesystem. Various filesystems have different approaches (see http://tldp.org/HOWTO/LVM-HOWTO/extendlv.html), xfs happens to be quite easy for growth but it doesn’t support shrinking:
# lvextend -l +1000 /dev/brad_group1/lvol0
# xfs_growfs /mnt/test
There’s some other stuff going on here as well, for instance it’s possible to set up a LV to stripe across PVs in the VG. This cane be seen using lvdisplay:
# lvcreate --extents 500 --stripes 2 /dev/brad_group1
# lvdisplay -m /dev/brad_group1/lvol2
--- Logical volume ---
LV Name /dev/brad_group1/lvol2
VG Name brad_group1
LV UUID 20t1dj-bwfQ-rkgf-wZFq-J2Oj-miI8-L4MKhh
LV Write Access read/write
LV Status available
# open 1
LV Size 1.95 GiB
Current LE 500
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 512
Block device 253:2
--- Segments ---
Logical extent 0 to 499:
Type striped
Stripes 2
Stripe size 64.00 KiB
Stripe 0:
Physical volume /dev/md0
Physical extents 1450 to 1699
Stripe 1:
Physical volume /dev/md1
Physical extents 0 to 249
displaying information
Various commands show info about the lvm configuration, examples follow.
pvs
# pvs
PV VG Fmt Attr PSize PFree
/dev/md0 brad_group1 lvm2 a- 15.99g 9.35g
/dev/md1 brad_group1 lvm2 a- 9.99g 9.02g
pvdisplay
# pvdisplay
--- Physical volume ---
PV Name /dev/md0
VG Name brad_group1
PV Size 15.99 GiB / not usable 4.00 MiB
Allocatable yes
PE Size 4.00 MiB
Total PE 4093
Free PE 2393
Allocated PE 1700
PV UUID OFyENQ-lRD8-jPoW-aEr4-MY8j-WMqN-pZYy2H
--- Physical volume ---
PV Name /dev/md1
VG Name brad_group1
PV Size 10.00 GiB / not usable 3.63 MiB
Allocatable yes
PE Size 4.00 MiB
Total PE 2558
Free PE 2308
Allocated PE 250
PV UUID 1WsYxg-YnSN-VKme-E0iw-gX6Y-KEcI-VUckAb
vgs
# vgs
VG #PV #LV #SN Attr VSize VFree
brad_group1 2 3 0 wz--n- 25.98g 18.36g
vgdisplay
# vgdisplay
--- Volume group ---
VG Name brad_group1
System ID
Format lvm2
Metadata Areas 2
Metadata Sequence No 7
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 3
Open LV 3
Max PV 0
Cur PV 2
Act PV 2
VG Size 25.98 GiB
PE Size 4.00 MiB
Total PE 6651
Alloc PE / Size 1950 / 7.62 GiB
Free PE / Size 4701 / 18.36 GiB
VG UUID BA5Vkh-SrmB-JOr6-vAz0-48Eh-YI6o-m3Ghvr
lvs
# lvs
LV VG Attr LSize Origin Snap% Move Log Copy% Convert
lvol0 brad_group1 -wi-ao 4.69g
lvol1 brad_group1 -wi-ao 1000.00m
lvol2 brad_group1 -wi-ao 1.95g
lvdisplay
# lvdisplay
--- Logical volume ---
LV Name /dev/brad_group1/lvol0
VG Name brad_group1
LV UUID 06M0bJ-RaDR-fVNz-weRo-pVAz-wTMl-17xr5G
LV Write Access read/write
LV Status available
# open 1
LV Size 4.69 GiB
Current LE 1200
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 4096
Block device 253:0
--- Logical volume ---
LV Name /dev/brad_group1/lvol1
VG Name brad_group1
LV UUID b0a2x6-Qxkl-sAX1-uz2y-KVFK-qiJu-IBodH7
LV Write Access read/write
LV Status available
# open 1
LV Size 1000.00 MiB
Current LE 250
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 4096
Block device 253:1
--- Logical volume ---
LV Name /dev/brad_group1/lvol2
VG Name brad_group1
LV UUID 20t1dj-bwfQ-rkgf-wZFq-J2Oj-miI8-L4MKhh
LV Write Access read/write
LV Status available
# open 1
LV Size 1.95 GiB
Current LE 500
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 512
Block device 253:2