md
Building a RAID array using mdadm - two primary steps:
- “mdadm –create” to build the array using available resources
- “mdadm –detail –scan” to build config string for /etc/mdadm/mdadm.conf
Simple examples:
RAID6 array
- Set up partitions to be used (in this case the whole disk):
# for x in /dev/sd{b,c,d,e,f}1 ; do fdisk $x ; done
<span id="line-18" class="anchor"></span>
- Create the array (in this case, with one hot-spare):
# mdadm --create /dev/md0 --level=6 --raid-devices=4 --spare-devices=1 /dev/sd{b,c,d,e,f}1
<span id="line-21" class="anchor"></span>
- Configure the array for reboot (append to the end of /etc/mdadm/mdadm.conf):
# mdadm --detail --scan
<span id="line-24" class="anchor"></span>ARRAY /dev/md/0 metadata=1.2 spares=1 name=debian6-vm:0 UUID=9b42abcd:309fabcd:6bfbabcd:298dabcd
<span id="line-25" class="anchor"></span>
Considerations when setting up the partitions might be that any replacement disks will need to support that same size partition. Unconfirmed but it sounds like it might be a reasonable concern: “Enter a value smaller than the free space value minus 2% or the disk size to make sure that when you will later install a new disk in replacement of a failed one, you will have at least the same capacity even if the number of cylinders is different.” (http://www.jerryweb.org/settings/raid/)
RAID1 array
Pretty much the same process - but for reference here’s the simpler RAID1 config without any spare:
# mdadm --create /dev/md1 --level=1 --raid-devices=2 /dev/sd{g,h}1
<span id="line-35" class="anchor"></span>
And here’s how the config looks:
ARRAY /dev/md1 metadata=1.2 name=debian6-vm:1 UUID=fa94abcd:317fabcd:3a61abcd:4932abcd
<span id="line-40" class="anchor"></span>
/proc/mdstat
The current state of the RAID arrays:
# cat /proc/mdstat
<span id="line-48" class="anchor"></span>Personalities :
<span id="line-49" class="anchor"></span>md1 : active raid1 sdh1 sdg1
<span id="line-50" class="anchor"></span> 10481285 blocks super 1.2
<span id="line-51" class="anchor"></span>
<span id="line-52" class="anchor"></span>md0 : active raid6 sdb1 sdf1[4](S) sde1 sdd1 sdc1
<span id="line-53" class="anchor"></span> 16769024 blocks super 1.2 level 6, 512k chunk, algorithm 2
<span id="line-54" class="anchor"></span>
<span id="line-55" class="anchor"></span>unused devices: <none>
<span id="line-56" class="anchor"></span>
replacing a failed disk
Some handy notes here:
- http://www.howtoforge.com/replacing_hard_disks_in_a_raid1_array
- http://consultancy.edvoncken.net/index.php/HOWTO_Replace_a_failing_disk_on_Linux_Software_RAID-5
lvm
Regardless of using RAID or plain disks, lvm allows much more flexibility in allocating that space. Not any use whatsoever on a smaller machine which is using a single large partition for everything (but even then, use of lvm could allow new disk to be added to extend a filesystem if space is running low).
For some very details documentation: http://tldp.org/HOWTO/LVM-HOWTO/
The steps:
- “pvcreate” to initialise partitions to be used by lvm
- “vgcreate” to create volume groups (these contain physical devices)
- “lvcreate” to define logical volumes (these are then used as disk partitions)
pvcreate
Partitions can’t be used until they’re initialised:
(this is run in testmode with -t to show what would happen)
# pvcreate -t -v /dev/md0
<span id="line-84" class="anchor"></span> Test mode: Metadata will NOT be updated.
<span id="line-85" class="anchor"></span> Test mode: Wiping internal cache
<span id="line-86" class="anchor"></span> Wiping internal VG cache
<span id="line-87" class="anchor"></span>
The physical volume (PV) will now be listed as available (along with those already allocated to a group). In this example all PVs are already allocated to groups:
# pvs
<span id="line-93" class="anchor"></span> PV VG Fmt Attr PSize PFree
<span id="line-94" class="anchor"></span> /dev/md0 brad_group1 lvm2 a- 15.99g 9.35g
<span id="line-95" class="anchor"></span> /dev/md1 brad_group1 lvm2 a- 9.99g 9.02g
<span id="line-96" class="anchor"></span>
vgcreate (and vgextend)
In order to use PVs they need to be allocated to a volume group (VG) within lvm.
When a group is created it starts with a number of initialised PVs:
# vgcreate brad_group1 /dev/md0
<span id="line-106" class="anchor"></span>
Later on more PVs can be added to a VG with vgextend:
# vgextend /dev/brad_group1 /dev/md1
<span id="line-112" class="anchor"></span>
Something determined at this stage is the “physical extent size” (defaulting to 4MB) - this is the volume chunk size of all PVs within the group. Space is allocated to logical volumes in whole physical extent chunks.
lvcreate (and lvextend)
Now that we have PVs and VGs it’s possible to create logical volumes (LVs) and use them.
There’s lots of options here, but here’s a simple example which uses extents, though it’s also possible to ask for a specific size:
# lvcreate -l 250 /dev/brad_group1
<span id="line-124" class="anchor"></span>
It’s also possible to ask for a specific LV name (if not, one is auto-generated).
This LV can now be formatted and mounted:
# mkfs.xfs /dev/brad_group1/lvol0
<span id="line-132" class="anchor"></span># mount /dev/brad_group1/lvol0 /mnt/test
<span id="line-133" class="anchor"></span>
If the size of an LV needs to be changed the lvextend program can be used:
# lvextend -l +1000 /dev/brad_group1/lvol0
<span id="line-138" class="anchor"></span>
Note: that this only changes the size of the LV, not the filesystem. Various filesystems have different approaches (see http://tldp.org/HOWTO/LVM-HOWTO/extendlv.html), xfs happens to be quite easy for growth but it doesn’t support shrinking:
# lvextend -l +1000 /dev/brad_group1/lvol0
<span id="line-144" class="anchor"></span># xfs_growfs /mnt/test
<span id="line-145" class="anchor"></span>
There’s some other stuff going on here as well, for instance it’s possible to set up a LV to stripe across PVs in the VG. This cane be seen using lvdisplay:
# lvcreate --extents 500 --stripes 2 /dev/brad_group1
<span id="line-150" class="anchor"></span># lvdisplay -m /dev/brad_group1/lvol2
<span id="line-151" class="anchor"></span> --- Logical volume ---
<span id="line-152" class="anchor"></span> LV Name /dev/brad_group1/lvol2
<span id="line-153" class="anchor"></span> VG Name brad_group1
<span id="line-154" class="anchor"></span> LV UUID 20t1dj-bwfQ-rkgf-wZFq-J2Oj-miI8-L4MKhh
<span id="line-155" class="anchor"></span> LV Write Access read/write
<span id="line-156" class="anchor"></span> LV Status available
<span id="line-157" class="anchor"></span> # open 1
<span id="line-158" class="anchor"></span> LV Size 1.95 GiB
<span id="line-159" class="anchor"></span> Current LE 500
<span id="line-160" class="anchor"></span> Segments 1
<span id="line-161" class="anchor"></span> Allocation inherit
<span id="line-162" class="anchor"></span> Read ahead sectors auto
<span id="line-163" class="anchor"></span> - currently set to 512
<span id="line-164" class="anchor"></span> Block device 253:2
<span id="line-165" class="anchor"></span>
<span id="line-166" class="anchor"></span> --- Segments ---
<span id="line-167" class="anchor"></span> Logical extent 0 to 499:
<span id="line-168" class="anchor"></span> Type striped
<span id="line-169" class="anchor"></span> Stripes 2
<span id="line-170" class="anchor"></span> Stripe size 64.00 KiB
<span id="line-171" class="anchor"></span> Stripe 0:
<span id="line-172" class="anchor"></span> Physical volume /dev/md0
<span id="line-173" class="anchor"></span> Physical extents 1450 to 1699
<span id="line-174" class="anchor"></span> Stripe 1:
<span id="line-175" class="anchor"></span> Physical volume /dev/md1
<span id="line-176" class="anchor"></span> Physical extents 0 to 249
<span id="line-177" class="anchor"></span>
displaying information
Various commands show info about the lvm configuration, examples follow.
pvs
# pvs
<span id="line-187" class="anchor"></span> PV VG Fmt Attr PSize PFree
<span id="line-188" class="anchor"></span> /dev/md0 brad_group1 lvm2 a- 15.99g 9.35g
<span id="line-189" class="anchor"></span> /dev/md1 brad_group1 lvm2 a- 9.99g 9.02g
<span id="line-190" class="anchor"></span>
pvdisplay
# pvdisplay
<span id="line-196" class="anchor"></span> --- Physical volume ---
<span id="line-197" class="anchor"></span> PV Name /dev/md0
<span id="line-198" class="anchor"></span> VG Name brad_group1
<span id="line-199" class="anchor"></span> PV Size 15.99 GiB / not usable 4.00 MiB
<span id="line-200" class="anchor"></span> Allocatable yes
<span id="line-201" class="anchor"></span> PE Size 4.00 MiB
<span id="line-202" class="anchor"></span> Total PE 4093
<span id="line-203" class="anchor"></span> Free PE 2393
<span id="line-204" class="anchor"></span> Allocated PE 1700
<span id="line-205" class="anchor"></span> PV UUID OFyENQ-lRD8-jPoW-aEr4-MY8j-WMqN-pZYy2H
<span id="line-206" class="anchor"></span>
<span id="line-207" class="anchor"></span> --- Physical volume ---
<span id="line-208" class="anchor"></span> PV Name /dev/md1
<span id="line-209" class="anchor"></span> VG Name brad_group1
<span id="line-210" class="anchor"></span> PV Size 10.00 GiB / not usable 3.63 MiB
<span id="line-211" class="anchor"></span> Allocatable yes
<span id="line-212" class="anchor"></span> PE Size 4.00 MiB
<span id="line-213" class="anchor"></span> Total PE 2558
<span id="line-214" class="anchor"></span> Free PE 2308
<span id="line-215" class="anchor"></span> Allocated PE 250
<span id="line-216" class="anchor"></span> PV UUID 1WsYxg-YnSN-VKme-E0iw-gX6Y-KEcI-VUckAb
<span id="line-217" class="anchor"></span>
vgs
# vgs
<span id="line-223" class="anchor"></span> VG #PV #LV #SN Attr VSize VFree
<span id="line-224" class="anchor"></span> brad_group1 2 3 0 wz--n- 25.98g 18.36g
<span id="line-225" class="anchor"></span>
vgdisplay
# vgdisplay
<span id="line-231" class="anchor"></span> --- Volume group ---
<span id="line-232" class="anchor"></span> VG Name brad_group1
<span id="line-233" class="anchor"></span> System ID
<span id="line-234" class="anchor"></span> Format lvm2
<span id="line-235" class="anchor"></span> Metadata Areas 2
<span id="line-236" class="anchor"></span> Metadata Sequence No 7
<span id="line-237" class="anchor"></span> VG Access read/write
<span id="line-238" class="anchor"></span> VG Status resizable
<span id="line-239" class="anchor"></span> MAX LV 0
<span id="line-240" class="anchor"></span> Cur LV 3
<span id="line-241" class="anchor"></span> Open LV 3
<span id="line-242" class="anchor"></span> Max PV 0
<span id="line-243" class="anchor"></span> Cur PV 2
<span id="line-244" class="anchor"></span> Act PV 2
<span id="line-245" class="anchor"></span> VG Size 25.98 GiB
<span id="line-246" class="anchor"></span> PE Size 4.00 MiB
<span id="line-247" class="anchor"></span> Total PE 6651
<span id="line-248" class="anchor"></span> Alloc PE / Size 1950 / 7.62 GiB
<span id="line-249" class="anchor"></span> Free PE / Size 4701 / 18.36 GiB
<span id="line-250" class="anchor"></span> VG UUID BA5Vkh-SrmB-JOr6-vAz0-48Eh-YI6o-m3Ghvr
<span id="line-251" class="anchor"></span>
lvs
# lvs
<span id="line-257" class="anchor"></span> LV VG Attr LSize Origin Snap% Move Log Copy% Convert
<span id="line-258" class="anchor"></span> lvol0 brad_group1 -wi-ao 4.69g
<span id="line-259" class="anchor"></span> lvol1 brad_group1 -wi-ao 1000.00m
<span id="line-260" class="anchor"></span> lvol2 brad_group1 -wi-ao 1.95g
<span id="line-261" class="anchor"></span>
lvdisplay
# lvdisplay
<span id="line-267" class="anchor"></span> --- Logical volume ---
<span id="line-268" class="anchor"></span> LV Name /dev/brad_group1/lvol0
<span id="line-269" class="anchor"></span> VG Name brad_group1
<span id="line-270" class="anchor"></span> LV UUID 06M0bJ-RaDR-fVNz-weRo-pVAz-wTMl-17xr5G
<span id="line-271" class="anchor"></span> LV Write Access read/write
<span id="line-272" class="anchor"></span> LV Status available
<span id="line-273" class="anchor"></span> # open 1
<span id="line-274" class="anchor"></span> LV Size 4.69 GiB
<span id="line-275" class="anchor"></span> Current LE 1200
<span id="line-276" class="anchor"></span> Segments 1
<span id="line-277" class="anchor"></span> Allocation inherit
<span id="line-278" class="anchor"></span> Read ahead sectors auto
<span id="line-279" class="anchor"></span> - currently set to 4096
<span id="line-280" class="anchor"></span> Block device 253:0
<span id="line-281" class="anchor"></span>
<span id="line-282" class="anchor"></span> --- Logical volume ---
<span id="line-283" class="anchor"></span> LV Name /dev/brad_group1/lvol1
<span id="line-284" class="anchor"></span> VG Name brad_group1
<span id="line-285" class="anchor"></span> LV UUID b0a2x6-Qxkl-sAX1-uz2y-KVFK-qiJu-IBodH7
<span id="line-286" class="anchor"></span> LV Write Access read/write
<span id="line-287" class="anchor"></span> LV Status available
<span id="line-288" class="anchor"></span> # open 1
<span id="line-289" class="anchor"></span> LV Size 1000.00 MiB
<span id="line-290" class="anchor"></span> Current LE 250
<span id="line-291" class="anchor"></span> Segments 1
<span id="line-292" class="anchor"></span> Allocation inherit
<span id="line-293" class="anchor"></span> Read ahead sectors auto
<span id="line-294" class="anchor"></span> - currently set to 4096
<span id="line-295" class="anchor"></span> Block device 253:1
<span id="line-296" class="anchor"></span>
<span id="line-297" class="anchor"></span> --- Logical volume ---
<span id="line-298" class="anchor"></span> LV Name /dev/brad_group1/lvol2
<span id="line-299" class="anchor"></span> VG Name brad_group1
<span id="line-300" class="anchor"></span> LV UUID 20t1dj-bwfQ-rkgf-wZFq-J2Oj-miI8-L4MKhh
<span id="line-301" class="anchor"></span> LV Write Access read/write
<span id="line-302" class="anchor"></span> LV Status available
<span id="line-303" class="anchor"></span> # open 1
<span id="line-304" class="anchor"></span> LV Size 1.95 GiB
<span id="line-305" class="anchor"></span> Current LE 500
<span id="line-306" class="anchor"></span> Segments 1
<span id="line-307" class="anchor"></span> Allocation inherit
<span id="line-308" class="anchor"></span> Read ahead sectors auto
<span id="line-309" class="anchor"></span> - currently set to 512
<span id="line-310" class="anchor"></span> Block device 253:2