Howto: Increase disk space in a mdadm raid

I currently have a Ubuntu Linux server running two mdadm RAID`s. One of the RAID sets is set up using 6 x 500 GB SATA drives. Now i have purchased 6 x 1500 GB SATA drives that will replace the old disks, but the challenge is to increase the RAID and filesystem without loosing any data or having downtime. (Note: not having downtime is possible since i use a system that supports hot swapping of drives)

In summary, this can be achieved by doing the following:
1) Replace all disks in the RAID (one by one)
2) Grow the RAID
3) Expand the filesystem

In this guide i will be working on /dev/md1

Now, let`s get to work!

Part one: Replace the disks

PS: If your system does not support hot swap, you have to turn of/restart your machine for each disk you are replacing.

Remove a disk in the RAID, then insert a new (bigger) drive.
Check dmesg (or similiar) to get the name of the last drive.

[14522870.380610] scsi 15:0:0:0: Direct-Access ATA WDC WD15EARS-00Z 80.0 PQ: 0 ANSI: 5
[14522870.381589] sd 15:0:0:0: [sdm] 2930277168 512-byte hardware sectors: (1.50 TB/1.36 TiB)
[14522870.381622] sd 15:0:0:0: [sdm] Write Protect is off
[14522870.381626] sd 15:0:0:0: [sdm] Mode Sense: 00 3a 00 00
[14522870.381673] sd 15:0:0:0: [sdm] Write cache: enabled, read cache: enabled, doesn’t support DPO or FUA
[14522870.381845] sd 15:0:0:0: [sdm] 2930277168 512-byte hardware sectors: (1.50 TB/1.36 TiB)
[14522870.381870] sd 15:0:0:0: [sdm] Write Protect is off
[14522870.381875] sd 15:0:0:0: [sdm] Mode Sense: 00 3a 00 00
[14522870.381918] sd 15:0:0:0: [sdm] Write cache: enabled, read cache: enabled, doesn’t support DPO or FUA
[14522870.381926] sdm: unknown partition table
[14522870.397752] sd 15:0:0:0: [sdm] Attached SCSI disk
[14522870.397878] sd 15:0:0:0: Attached scsi generic sg9 type 0

Now, tell mdadm to add your new drive to the RAID you removed a drive from by doing:

mdadm –manage /dev/md1 –add /dev/sdm

Mdadm will then start syncing data to your new drive, to get a ETA of when it`s done (and when you can replace the next drive) check the mdadm status.

root@bais:/home/samba/raid1/test# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid5 sdl[2] sdi[4] sdf[3] sde[1] sdd[0]
5860553728 blocks level 5, 128k chunk, algorithm 2 [5/5] [UUUUU]

md1 : active raid5 sdm[6] sdg[1] sdk[5] sdj[7](F) sdh[2] sdc[3] sda[0]
2441932480 blocks level 5, 64k chunk, algorithm 2 [6/5] [UUUU_U]
[==>………………] recovery = 14.2% (69439012/488386496) finish=155.8min speed=44805K/sec

unused devices:

So after around 155 minutes  the drive is active. (And the next one can be replaced)

Repeat this process for each disk in the RAID.

When you have changed all disks, run the command “mdadm –manage /dev/mdX –remove failed” to remove any devices listes as failed for the given RAID.

Part two: Increase the space available for the RAID

This is done by simply issuing the command:

mdadm –grow /dev/md1 –size=max

And the RAID size is increased, note that this has caused the RAID to start a resync (again):

root@bais:~# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid5 sdl[2] sdi[4] sdf[3] sde[1] sdd[0]
5860553728 blocks level 5, 128k chunk, algorithm 2 [5/5] [UUUUU]

md1 : active raid5 sdc[0] sdj[3] sdh[5] sdg[2] sdn[1] sdm[4]
7325692480 blocks level 5, 64k chunk, algorithm 2 [6/6] [UUUUUU]
[======>…………..]  resync = 34.6% (508002752/1465138496) finish=247.0min speed=64561K/sec

PS: note that the resync speed has increased by around 20MB/s after all the drives was replaced 🙂

You will now also notice that the RAID reports it`s new size:

root@bais:~# mdadm –detail /dev/md1
/dev/md1:
Version : 00.90
Creation Time : Sat Jun 13 01:55:27 2009
Raid Level : raid5
Array Size : 7325692480 (6986.32 GiB 7501.51 GB)
Used Dev Size : 1465138496 (1397.26 GiB 1500.30 GB)
Raid Devices : 6
Total Devices : 6
Preferred Minor : 1
Persistence : Superblock is persistent

Update Time : Fri Mar  5 08:03:47 2010
State : active, resyncing
Active Devices : 6
Working Devices : 6
Failed Devices : 0
Spare Devices : 0

Layout : left-symmetric
Chunk Size : 64K

Rebuild Status : 35% complete

UUID : ed415534:2925f54a:352a6ad4:582f9bd3 (local to host bais)
Events : 0.247

Number   Major   Minor   RaidDevice State
0       8       32        0      active sync   /dev/sdc
1       8      208        1      active sync   /dev/sdn
2       8       96        2      active sync   /dev/sdg
3       8      144        3      active sync   /dev/sdj
4       8      192        4      active sync   /dev/sdm
5       8      112        5      active sync   /dev/sdh

Part three: resize file system

Start off by unounting the file system in question and perform a file system check to make sure everything is a-ok

root@bais:/home/torhenning# umount /home/samba/raid1
root@bais:/home/torhenning# fsck /dev/md1
fsck 1.41.4 (27-Jan-2009)
e2fsck 1.41.4 (27-Jan-2009)
/dev/md1 has gone 188 days without being checked, check forced.
Pass 1: Checking inodes, blocks, and sizes

Be warned: The fsck CAN take quite a time to finish.

When it`s complete, you are ready for the last step, which is to resize the filesystem:

root@bais:/home/torhenning# resize2fs /dev/md1 6986G
resize2fs 1.41.4 (27-Jan-2009)
Resizing the filesystem on /dev/md1 to 1831337984 (4k) blocks.

And voila! Mount up the filesystem again and you are finished! 🙂

10 thoughts on “Howto: Increase disk space in a mdadm raid

  1. Hi,
    Just a thought because I’m about to go through this process: Assuming your machine supported another physical drive, would it make sense & would it be possible to add extra drives to the array and resynchronizing BEFORE removing any of the original drives? That way you’re never running on less than two disks so there’s less chance of something going wrong.

    Rob

  2. RAID5 will never use more than one disk as spare, if you want more spare disks you need to go towards RAID6. (A RAID5 can be converted without any problem). You could also consider using hot spare disks but they will not be taken into the RAID before a disk actually dies.

    But you can in theory add a new disk to the RAID5 and simply dont expand the partition, and then remove the extra disk again afterwards. Note that i have not tried this myself. (I have tried expanding and changing a RAID5 to 6)

  3. You just exposed the biggest flaw in RAID 5: If you entirely lose any of the disks (say for a broken disk or an intentional remove) you end up with a stripping raid in terms of security for a while.
    So let’s say that you have 6 disks, you remove the sixth, but disk 3 has 10mb of bad blocks that you never realized up until now, so you lost 10mb of data, simply put, what’s worse is that these 10mb of data most likely will corrupt 40mb more of other interleaved data.

    For a small array the likelyhood of that is quite small, but for a reasonably numbered array like 5, 6 or more it’s wiser/safer adopting a raid 6.

  4. I used Mdadm arrays on very different disks: IDE, SATA, 15k SCSI, even SSD.
    Now I can tell You, that sometimes:
    – mdadm is doing resync over and over and You have to manually check SMART values of every disk in the array (it was in case in case of SATA drives on AHCI mode, in some cases it isn’t enough since some problems with electronics cannot be discovered this way,
    – mdadm is kicking out bad drive (it was in case of IDE drives),
    – server can hung (it was in case both SATA and SCSI drives).

    You have been warned 😀

    But anyway it is better even than some HW RAID solutions (it was in case of Dell R610, server hung on SATA drive failure). Also, HW solutions mostly do not allow checking SMART values.

  5. OMG!! what U did?!
    These drives are ADVANCED format, DO NOT create Raid autodetect partitions with standard way! Use fresh “parted” or other program that supports Advanced Format Disks with 4kB sector. I see VERY LOW rebuild rate and I also went into this issue with 1TB WDC Green drives – these are NOT recommended for RAID arrays (didn’t know that).
    Please double check Your partition aligment..
    http://community.wdc.com/t5/Desktop-Portable-Drives/Problem-with-WD-Advanced-Format-drive-in-LINUX-WD15EARS/td-p/6395
    I’ve replaced Green series with Black servers, recreated partitions and now it’s fine.

  6. Today, tried exacly these steps (but I didn’t read them from there, just from Mdadm Wiki).
    We HAVE LOST 1TB of data, of course I did backup first.
    We have expaneded 3 x drive config in RAID5 to 4 x drive config in RAID5.
    Then I’ve discovered that /dev/md0 is UNMOUNTABE, FS is corrupted, even Testdisk was unable to find any partitions => all data lost.
    Be careful..

  7. Ouch! Shit can happen but i have not gotten any of that. Allthough i have had worse cases where SAN`s have died completely, so i know how it feel. (And a RAID5 which died..yay!)

    I am currently in the process of replacing all my RAID6 drives from 1.5TB to 3TB. AND reshaping the RAID from using 11 drives, down to 10. If i make it through the whole process then i will consider myself lucky 🙂

    PS: I do not use RAID partitions, i work directly on the devices:

    (mdadm-output after the resize)

    root@dom0:/home/***# mdadm –detail /dev/md0
    /dev/md0:
    Version : 0.90
    Creation Time : Sat Jan 24 14:16:52 2009
    Raid Level : raid6
    Array Size : 11721107456 (11178.12 GiB 12002.41 GB)
    Used Dev Size : 1465138432 (1397.26 GiB 1500.30 GB)
    Raid Devices : 10
    Total Devices : 11
    Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Fri Aug 17 21:14:34 2012
    State : active
    Active Devices : 10
    Working Devices : 11
    Failed Devices : 0
    Spare Devices : 1

    Layout : left-symmetric
    Chunk Size : 128K

    UUID : 7957ee15:f1537981:c6619b65:eace9c87 (local to host dom0)
    Events : 0.680094

    Number Major Minor RaidDevice State
    0 8 32 0 active sync /dev/sdc
    1 8 160 1 active sync /dev/sdk
    2 8 144 2 active sync /dev/sdj
    3 8 64 3 active sync /dev/sde
    4 8 48 4 active sync /dev/sdd
    5 8 128 5 active sync /dev/sdi
    6 8 80 6 active sync /dev/sdf
    7 8 176 7 active sync /dev/sdl
    8 8 192 8 active sync /dev/sdm
    9 8 96 9 active sync /dev/sdg

    10 8 16 – spare /dev/sdb

    • Performance?
      hdparm -t /dev/md0
      dd if=/dev/zero of=/mnt/array/somefile.tmp bs=1024k count=200000
      Rebuild rate and speed of RAID6 in this config?

      I’m asking because we’re going to bring back to life 2 old SATA arrays..
      http://toomeek.waw.pl/blog/?p=1833
      There is interface limit 320MB/s 🙁 this for HD video storage.
      Also max capacity limited to 750GB per disk.

      I see You have RAID6 (2 parity disks) and one spare which means You loose capacity of 3 disks, right?
      Right. 16500 – 12000 = 4500GB.

      So will assigning whole device avoid Advanced Format performance-related problems? If they are 1,5TB and 3TB then yes, they are Advanced Format.

  8. Ok I thik the RAID5 expand issue was caused by devices was added by following order: /dev/sdb1 /dev/sdc1 which means partitions.
    But one of them (1 of 4) was showed as /dev/sdd, not partition but whole device.
    I think it caused data corruption and parity regenate problem, but I still don’t know kow this happend..