RAID array recovery

[Big Dog]

Once upon a time at a company I shall not disclose...
I was asked to tidy up some files for prior to a hardware migration.
I created a script to remove all the files that failed the specified business rules.
Thinking I was smarter than the average bear but not smart enough to not make a mistake I thought I know, I will ship all the detritus to this storage over here that is scheduled for destruciton in 30 days instead of listening to the honchos and just shredding it. So I created a locked down folder that only I had access to and away I went...

75% of the data on the source was moved to my hidden location and deleted from the source.
All sorts... full cds, itunes collections, family albums etc.

So a a couple of days later I get a panicked call from one of the execs... omg all my files are gone... I didnt think our files counted in the clean up etc.
No worrries I will put them back in a folder labelled recovered files, you sort what you need onto a drive you can take home and I will delete it again in a week.


Sounds great right?

Well someone asked their wife to sort the files onto a drive and take them home. I guess he was too busy to do it himself...

To bad he forgot about the videos of him banging his secretary on his desk then eh?

[Akzle]

Does the controller still show that there is a RAID set, or does it show nothing at all?

a quick googluh shows raid five stripes parity. ie it's not all on one disk.

it also shows that fucking two disks fucks your array.

[Gremlin]

a quick googluh shows raid five stripes parity. ie it's not all on one disk.

it also shows that fucking two disks fucks your array.

p.dath is pointing more towards the controller. ie, we haven't actually been told that the disks are fucked. It's just the controller that is fucked, so the disks could actually be perfectly fine...

On a side note, I never really liked roaming profiles, it fucked up stuff like offline file sync blah blah and made login/logoff times long. We always prefer to make our clients aware that they should save on a network location, because that's the server and it's backed up etc etc rather than hiding stuff, re-directing folders and so on. It makes perfect sense now when you consider Apple loves saving everything into the profile...


There was the one time staff at a client wanted to share some movies and didn't have the space to put them somewhere for others, so thought the server share was the perfect place... until it ran out of space and the backup software scheduling fell off the earth because the hourly incrementals had grown by double digit GB, disk IO started complaining and generally their simple idea turned my life into a mess...

The files were immediately deleted, much to their disappointment

[pete376403]

If you have only lost the array configuration, and the drives themselves are fine then you may be able to tell the controller to re-read the configuration off the drives. Many controllers support this so that you can transport drives sets between servers with the same controller.

Does the controller still show that there is a RAID set, or does it show nothing at all?

That sounds a little bit modern. This is an eight year old Intel SSR212MC. The disks show as there, but unconfigured.
Also, I talked to Datalabs, they sound pretty sure they can get it all back at somewhere between $2k5 and $4K5. Also they will look at the disks and if they cant salvage them, there's no charge.
Over to management.

[wharekura]

Once upon a time at a company I shall not disclose...
...Well someone asked their wife to sort the files onto a drive and take them home. I guess he was too busy to do it himself...

To bad he forgot about the videos of him banging his secretary on his desk then eh?

Auckland City Council?

[wharekura]

...There was the one time staff at a client wanted to share some movies and didn't have the space to put them somewhere for others, so thought the server share was the perfect place...

In a prior company we had an admin that gave his server shares heaps of space and us plebs buga all - despite having sql server backups, log files etc. after asking for more space he refused. So being the labyrinth the network array of folders are, I played a joke. Told said admin in passing, that I implied that I no longer had to worry about shared space as there was heaps of space in a particular folder of his and walked off not mentioning which folder. I did not actually use any of his folders.

I am sure he would of spent time checking logs...

[Big Dog]

Auckland City Council?

I can neither confirm nor deny.
It wasn't but I could just be saying that.
It was but I could just be saying that.

2 of the above could be considered true.

Sent via tapatalk.

[Akzle]

The Tao of Backup


((just found it. it's lulz, this seemed the most appropriate thread to hijack)

[Big Dog]

The Tao of Backup


((just found it. it's lulz, this seemed the most appropriate thread to hijack)

That's pretty good.

Sent via tapatalk.

[ledaero]

As it's an Intel... I can't offer you any promises, but steps 1-3 won't hurt anything and may very well return to you a working array.

1) Go into the Intel Matrix Storage Manager in BIOS.

2) Mark all the drives in the RAID as "non-Raid." This warns you that all your data will be lost, but really is simply re-writing the partition table, which is ALREADY messed up anyhow...

3) Re-Create the RAID using the EXACT SAME settings as the original RAID that you are trying to recover. Disks must be in the same order as before, and the strip size must be the same as before. Note: Having the disks in the wrong order will not destroy your data, it will just mean that step (4) will not work... If (4) doesn't work, you will need to repeat steps 1-3 until you have the disks in the proper order...

4. Download and run TestDisk from http://www.cgsecurity.org/ according to steps on the web site. If your HDs are connected in correct order, TestDisk should find the partition(s) within a few seconds. If it doesn't - look again at the array configuration. Get it right and it will recover - there's nothing wrong with the disk.

5. (optionally - if you need to) have TestDisk write the fixed partition table to the array drive.


A Raid 5 Array

[pete376403]

That looks interesting, thanks. The disks are still installed in the original positions. No recovery attempts have been made, because I was able to get back almost everything that mattered from backups. Not actually sure what to do with the disks, as mentioned they hadn't failed, but they are near 8 years old so probably past their use-by date, and the whole volume was only 900gig (300G 16Krpm SAS drives).

Do organisations replace drives on a regular basis, as preventive maintenance, or just as they fail?

[jonbuoy]

That looks interesting, thanks. The disks are still installed in the original positions. No recovery attempts have been made, because I was able to get back almost everything that mattered from backups. Not actually sure what to do with the disks, as mentioned they hadn't failed, but they are near 8 years old so probably past their use-by date, and the whole volume was only 900gig (300G 16Krpm SAS drives).

Do organisations replace drives on a regular basis, as preventive maintenance, or just as they fail?

"If its not broken don´t fix it" or "Nothing prevents maintenance like preventive maintenance" Your call

[Akzle]

That looks interesting, thanks. The disks are still installed in the original positions. No recovery attempts have been made, because I was able to get back almost everything that mattered from backups. Not actually sure what to do with the disks, as mentioned they hadn't failed, but they are near 8 years old so probably past their use-by date, and the whole volume was only 900gig (300G 16Krpm SAS drives).

Do organisations replace drives on a regular basis, as preventive maintenance, or just as they fail?

ill take those scuzzies off your hands...

Typically run to failure. With sufficient redundancy, it is no problem. Find the company that can afford to bin perfectly good disks and ill show you one that doesnt need my custom.

[Gremlin]

Do organisations replace drives on a regular basis, as preventive maintenance, or just as they fail?

Try to replace whole servers on a cycle to prevent outages, but companies do try to run out as long as possible. Virtualised environments make migration much easier. Disks holding backups are regularly tested and any issues and they're replaced. Too important to risk issues there, especially when requesting massive amounts of data from them can put them under the most strain...