HDD Capacity size reporting wrong - often after RAID

[SecCon]

This small article deals with rather advanced methods of writing/erasing data on any storage device.
Use it with outmost care.

Below works best on a drive connected via SATA or even SAS, do not expect it to work with a USB dock. Maybe on newer, but it did not on mine.

I often end up with hdd's reporting the wrong sizes after picking them out of a raid to reuse them in something else.

Like now I have a WD Red, first of six, that I am replacing and refurbishing for other use. It should report 6TB but it is down to 1.3. This likely happens because the raid controller do not restore the original capacity and the drive still thinks it is part of a raid.

One might think that the old fdisk ( https://en.wikipedia.org/wiki/Fdisk ) might help with this in Windows, but in this scenario it is very crippled. Also missing in later versions of Windows, hence Linux.

Normal disk utilities from Seagate/WD, HGST or others, do not always address this properly. So I use linux dd or hdparm.

DD: https://linuxopsys.com/topics/linux-dd-command-with-examples
DD: https://www.maketecheasier.com/use-dd-command-linux/

A typical command to clear out the first few sectors could be, assuming disk to be /sdi ,

Code: Select all

$: dd if=/dev/zero of=/dev/sdi bs=1048576 count=1024

When that is not enough one can expand it to overwrite the entire disk, like

Code: Select all

$: sudo dd if=/dev/zero of=/dev/sdi

which will eventuall trigger "no space left on device" which is to be expected if you do not define the exact amount of space to overwrite.

hdparm
https://linux.die.net/man/8/hdparm
I have found that raid plays with hidden partitions. HPA mode.
https://superuser.com/questions/642637/harddrive-wipe-out-hidden-areas-like-hpa-and-dco-after-malware-infection/642765#642765

This actually helped me restore the HDD's capacity when everything else above failed.
It will give you the correct sector number and let you expand a crippled device restoring those, if HPA partitioning is used.

Like so
(assuming target disk is /sdc)

Code: Select all

$: hdparm -N /dev/sdc
/dev/sdc:
 max sectors   = 12631728/7814037168, HPA is enabled

That means that there are hidden partitions and that only 12631728 out of 7814037168 is actually visible to any OS.
In order to restore these you use the cmd:

Code: Select all

$: hdparm -N p7814037168 /dev/sdc

Fixed. On my SATA drives, by me, taking all the risks myself.

---


To be perfectly honest I thought HDSentinel would help with this, but from this article it describes no method to do that:
https://www.hdsentinel.com/hard_disk_case_capacity_is_not_correct.php
I find that very strange and a bit of a shortcoming.

[hdsentinel]

The information you wrote sounds interesting.

Can you please write FOR SURE a RAID controller / configuration where did you see any similar? I'd be happy to reproduce, investigate.
Personaly I examined countless of number of RAID controllers / configurations (IDE, SATA, SAS, NVMe, USB external etc...) in the years and never ever encountered that the RAID controller would use the HPA feature.

Generally RAID controllers use more or less sectors from the disk surface to store RAID metadata and WHILE the drive(s) used with this configuration, they appear with slightly smaller capacity. Usually RAID controllers do this during "initialize" stage or so: then they configure the drive(s) to work in RAID array, store different information related to the RAID configuration / volume and so.
This may result in a minimal capacity decrease (eg. 1 MByte or so) WHILE the drive is used with the actual RAID controller.

But as soon as we remove the drive from the array and connect to a completely different controller / HBA (eg. from a different manufacturer, just to make sure that it should not be confused by the mentioned RAID metadata) then we see the drive with the complete capacity and (for example by the Disk menu -> Surface test function of Hard Disk Sentinel) we can even inspect the above mentioned RAID metadata written to some sectors of the drive.


> Like now I have a WD Red, first of six, that I am replacing and refurbishing for other use.
> It should report 6TB but it is down to 1.3.
> This likely happens because the raid controller do not restore the original capacity and the drive still thinks it is part of a raid.

SURELY NOT.

I'd be really happy to see this situation, I wish that you should use Report menu -> Send test report to developer option when you see it.

But probably this situation was exactly same as described at
https://www.hdsentinel.com/hard_disk_case_capacity_is_not_correct.php
when the actual controller (where you connected the drive, independently from any previous RAID controller/configuration) could not manage the drive correctly. The numbers you wrote exactly suggests that: generally if the controller (due to compatibility issues) can't manage the complete capacity of the drive, then the reported capacity is the actual capacity MODULO 2 TB.
The above page shows 746 GB for the 3 TB (= 2794 GB) drive.

For a typical 6 TB drive (which is approx. 6000000000000 bytes), it could be reported as 1601953489920 bytes, roughly 1.45 TB but would be really nice to see this situation.


Fdisk creates logical drives on the actual physical drive. It is a completely different thing: it creates a partition on the available space (as reported by the drive). Of course Fdisk (or any similar which would create/manage a PARTITION/logical drive even under Linux) does not affect the drive capacity: they all read the capacity and make/manage a partition on those sectors, as reported by the drive.

> I have found that raid plays with hidden partitions. HPA mode.

HPA is a completely different situation, it is completely independent from RAID arrays/controllers.

HPA is generally used by manufacturers to make a "hidden" area to store eg. recovery/diagnostic partition. This area of the disk should remain "hidden" and yes, because of this, the disk capacity can be smaller. This is not rare, widely used technique.
But it is not really related / caused by disk (RAID) controllers - and while Hard Disk Sentinel detects and shows if HPA function is generally supported (most drives support it) it does not show the capacity "decrease" because it can be normal and expected.

I'd be more than happy to check what you saw, so if you would use Report menu -> Send test report to developer option, I'd be more than happy to examine for example
- when you see the above mentioned 6 TB drive with smaller capacity
- when you see any other drive with smaller capacity

[SecCon]

The described scenario happened with several SATA drives detached from a HPE 410 Raid controller in April last year. I pondered solutions and feedback in a Swedish tech forum for almost a month where the usual suspects where discussed and I finally managed to reset the configuration of the HDD's using hdparm.

You are probably right, since you develop the software and have lots of information about how these things work. I just take it from what I see in front of me and try to make things work.

There seems to be no established guide for what to do to drives when moving them from a raid controller and start using them as normal drives again, or a different raid controller - a thing I am doing as we speak. I want to ensure the capacity is "restored" and find no easy way to do that.

---

I was aiming to complement my text above with fdisk in Linux, that I have been using for a few days lately, but results are not as expected, yet. Have two WD60EFRX/WD60EFPX taken out from my servers LSI SAS3108 controller (now you may know who I am since I sent you a couple of developer reports abut that one) and trying to restore the capacity. I am putting them in a NAS instead, but want to verify capacity before...

Despite me deleting everything with fdisk yesterday, Windows Disk manager still reports wrong info. Via a USB dock, both are interpreted as 1493.03 GB. Both triggers the HDsentinel warning about capacity.



I checked both with the methods I used almost a year ago, but none the wiser. I will plug them back as SATA and continue to see what can be done. There are four more drives in my queue once these are done. The again, I might just put them in my NAS and it might be able to use them, running some scripts or whatever to restore capacity. I will try that and get back...

[SecCon]

And it seems that the NAS is taking it for what it is, ignoring what attempts I may have done to restore the drive in Linux and trying to confirm tat in Windows.



At least after a few minutes.

BTW the NAS is a 10 year old Asustor 606-T.

[hdsentinel]

Thanks, I see.

This is completely independent from the actual drive(s) and also completely independent from the HPE RAID controller (generally how/where the drives used).

> There seems to be no established guide for what to do to drives when moving them from a raid
> controller and start using them as normal drives again

Personally I recommend to use Disk menu -> Surface test -> Write test just to properly overwrite (clear) any possible previous data, including RAID metadata, just the make sure the drive should be empty before re-using. But this is usually not required.


> or a different raid controller - a thing I am doing as we speak. I want to ensure the capacity is "restored" and find no easy way to do that.

No need to manually restore any kind of capacity, this is automatic. As I tried to explain above, the RAID controller *never* (permanently) use any capacity: as soon as you remove from the RAID controller and connect the drive to a different controller, you can immediately access to the whole drive capacity.


> Despite me deleting everything with fdisk yesterday, Windows Disk manager still reports wrong info.
> Via a USB dock, both are interpreted as 1493.03 GB. Both triggers the HDsentinel warning about capacity.

Your USB dock is the problem. Probably it is too old to manage drives with 2 TB or larger.

The warning message of Hard Disk Sentinel shows that the current situation has a serious compatibility issue when the complete drive capacity can't be properly managed by the way the drive currently connected, managed.

The https://www.hdsentinel.com/hard_disk_case_capacity_is_not_correct.php

page explains it more specifically and suggests that you should change the configuration - which means connect the drive differently.

For example
- if the drive is connected to internal port (eg. old motherboard/chipset) then use with a (relatively )new USB dock/adapter or connect to newer system
- if the drive is connected to (old) USB dock/adapter, then connect to direct SATA port.

Very frequent issue with old USB docks/adapters as explained on the above page.

[hdsentinel]

There is no magic and no wonders of course.

You just (as suggested) connected the drive to a different system, without using the problematic dock - and the drive is immediately working as should.

[SecCon]

I tried to underline that the drives were in a USB dock when detecting in Windows, as a potential error cause.

It works with "healthy" drives above two TB, so it is not to old in that perspective, I have used it for larger than 2TB drives. Maybe newer docks would work better, I do not know of any docks that successfully "imitates" SATA connectors, but in any case I will send you the requested reports.

I have only one computer I can use to connect drives via SATA, a SuperMicro Server in my rack, that is currently not running Windows, but Ubuntu Server, only CLI/Shell access. Also the one where I did all of my fdisk in Linux attempts, as described in my previous post. I could switch that to a Desktop version and use QEMU or IPMI to get an interface and perhaps even install the Linux version of HDSentinel for reports.

Since I am replacing 12 drives, got 6 new 6TB SAS drives, and moving SATA drives from my SAS server to my SATA NAS, I will have more drives to experiment on.

As I tried to explain above, the RAID controller *never* (permanently) use any capacity: as soon as you remove from the RAID controller and connect the drive to a different controller, you can immediately access to the whole drive capacity.

If that always were the case we would not have this conversation.