Hard Disk Monitoring

Good morning,
somtimes I recommend to use HDSentinel in case of disastrous bugs, when LBA addr is limited to 32bit by BIOS, drivers or ext. enclosure bridge chips.
What is the source of the information shown as
- "Hard Disk Summary/Total Size"
- "ATA Information/Total Sectors"
- "ATA Information/Unformatted Capacity"
- "Disk Information/Capacity"
In case of an erronous handled 3TB drive the values are (in above order): 764428 MB / 732566646 / 2861588 MB / 3000 GB

I assume the last one comes from a database, the 2nd/3rd from the Identify Device, but wherefrom comes the wrong 764428 MB?
(due to cutoff of maxLBA address bit 33)
thanks in advance
Ernst

Yes, I can confirm that 4th is from the hard disk database, the 2nd, 3rd are from the ATA Identify Device, the 1st is the capacity provided by Windows when Hard Disk Sentinel detects its full addressable capacity.

So it would be worth a big WARNING !!! if these values does not correspond. Would be a nice feature of next versions HDS.
Undetected it destroys (after filling up the disk up to a critical value) data from the beginning, when reaching $MFT, all data lost

I agree that it may be really useful - and also an indicator that the hard disk can't be used properly due to limitations (for example if the user just bought the large capacity HDD).

I'm in the process of collecting reports about this situation exactly to check how the drive, OS responds on specific situations (to make sure detect all such situations), so if possible, please use Report -> Send test report to developer about the configuration as the "raw" information helps lots in the development of this function.
Thanks!

OK, next time stumbling upon such a problem I will instruct the user seeking for help to do so.
After a short meditation I found this is a little bit more complex.
Nominal sector count (MaxLBA) of a drive model can be retrieves from Database.
This might be influenced by logical sector size (normally 512, but meanwhile most of the external disk drives the USB3/SATA bridge controller emulates 4096 as logical size) and can be evaluated by the ID cmd returns. In this case the # of sectors shown by ID cmd is 1/8 of the nominal value relating to 512byte sector size.
This value might be reduced by
- DCO setting
- HPA setting
- DCO+HPA setting
or set to HPA size in offset mode
(see description at http://www.utica.edu/academic/institute ... 4A2671.pdf )
it would also be nice to show at the HDS reports the logical sector size, DCO and HPA size (if supported and set) and whether offset mode is active.
Additionally, another thing can influence sector count: Jumper - and this would also be worth to be adviced by a warning.
Traditionally, older drives <= 160GB may be jumpered to 2**28 to avoid confusion with old motherboard's & BIOS capabiltiy.
Advanced Format (4K physical sector size) drives may be jumpered +1, and I am uncertain whether the capacity is reduced by 1, or the maxLBA sector is mirrored to sector 0.

The mentioned exception of misadressing by adress limitation in my previous posts can only reduce the remaining size by 2**30 or 2**31 (yes, some silly BIOS Coder did so) or 2**32.
Old bridge controllers >2TiB HDDs not supporting can only be detected in a safe way
by reading maxLBA and MaxLBA&&(log.ANDed with)0x0000FFFFFFFF sector.
If their contents are equal, the misadressing occures.

Thanks!
Reports in such situations always helpful.

Yes, I completely agree that the different sector sizes can cause confusion, so I'd rather check the designed capacity vs the addressable capacity (sector count multiplied with sector size). This way there would be no problems in case of the USB bridge shows different sector size than the actual, so there would be no problems and this capacity issue could be detected even if the drive is connected to simple SATA connector or by using a such USB bridge / adapter.

I'm checking the possibilities, for example to check the total capacity differently (eg. by using READ NATIVE MAX ADDRESS command).
Just as I checked, this may be also blocked by various disk controller drivers, exactly to force the OS to use only the reported capacity and prevent any kind of access (even checking the size) of the protected area.

Anyway, thanks for the tip and increasing the attention about this field, this is surely an interesting area to check more and add notifications / information depending on the actual situation, especially in case of BIOS limitation. Personally I also saw such issue with some older USB bridges - they also do not use the highest bit(s) properly during addressing.

I'm in the process of collecting reports about this situation exactly to check how the drive, OS responds on specific situations (to make sure detect all such situations), so if possible, please use Report -> Send test report to developer about the configuration as the "raw" information helps lots in the development of this function.

The test for misadressing is very simple.
>2TB hdds must be initialized in GPT format, if sectorsize of 512. So the GPT header at sector 1 is mirrored to maxLBA-1(from ID data).
Even for controllers that does not support SMART, they support ID.
in case maxLBA >2**32, read sector maxLBA-1 and (maxLBA-1 with address bits 0-15 set to 0).
If both contents match, a misaddressing is ongoing or happened in the past, because both addresses refers(or referred in the past) to same physical sector.
Verification for content holds GPT header: Signature 0x4546492050415254 ='EFI PART' at sector+0x0, len 8
To determine the misaddressing occured in the past, modify last byte(this is not part of the internal CRC range and free/reserved for future use) and write it back to maxLBA-1,
read (maxLBA-1 with address bits 0-15 set to 0) again, if the change is noticeable there, then it' actual.
In both cases a warning would be desirable.

Fast implementation would be needful, because atm lot of people buys 3TB or 4TB and there are up to 5 new help requests a day at a german data rescue forum. Would be an excellent marketing approach... "protects your healthy hard drives >2TB from erroneous destruction!"

Thanks for the information !
I agree most points - just we need to consider what happen if the drive is not yet initialised, the partition record corrupted, contains non-Windows partitions etc...
In these situations we can't rely on the recorded information, for example the GPT structure saved.

Thanks for your attention and the tip, I'm sure such verification will be added to a later version - but only after checking different possibilities to prevent confusion or issues.

Last week I told André to send a tech report referring to logical 4K external disks.
He gots the answer

Dear André,

Thanks so much for your report and the information provided
(and of course for the suggestions on the forum).

According my experiences, this hard disk really has 4096 bytes/sector,
not the USB bridge emulates this value.
HDSentinel would report this if the drive would be connected to simple SATA connector.

Is it possible to have a report about the other drive you mentioned on the forum,
where the capacity is truncated? That would be really interesting to see.

Thanks so much for your help!

If you have any questions or I can assist, please let me know.

Have a nice day,

The HDD inside is a simple one 3,5" 3TB HDD with physical 4K sector technology("Advanced Format");
Within the case it shows 4K sectors and maxLBA=732566645
Shifted out of the case and internally connected it shows the typically 512byte sector layout with maxLBA=5860533168.
So the programmable USB3/SATA bridge controller of the external case is the only detectable culprit.

For this test case, when the maxLBA value shows only 1/8 of the typical count(or a little bit less in external RAID enclosures to reserve space for future metadata if this JBOD volume will get member of a RAID in the future).
it would be worth to show a comment

CAUTION! if you remove this HDD from its enclosure and connect it to internal SATA ports or another enclosure/docking station,
data will become inaccessible due to change of logical sector size (at the moment 4K, emulated by the external bridge controller)

At next chance I will tell other guys in trouble to send HDS tech reports concerning this cases to you:
- 3TB with correct adressing / 3TB with misadressing caused by 2**32 LBA limiting drivers
- 3TB with correct adressing / 3TB with misadressing caused by 2**32 LBA limiting BIOS
- 3TB with correct adressing / 3TB with misadressing caused by 2**32 LBA limiting bridge chip of ext.case/docking station

hdsentinel wrote:I agree most points - just we need to consider what happen if the drive is not yet initialised, the partition record corrupted, contains non-Windows partitions etc...
In these situations we can't rely on the recorded information, for example the GPT structure saved.

OK, here's another proposal how to determine misaddressing of disks >512GB by only reading 99 sectors, duration approx. 2s:
This will work regardless of drive initialization status or data formats.
Hope this could be realized soon. beta versions for tests are night and day acceptable.

> The HDD inside is a simple one 3,5" 3TB HDD with physical 4K sector technology("Advanced Format");
> Within the case it shows 4K sectors and maxLBA=732566645
> Shifted out of the case and internally connected it shows the typically 512byte sector layout with maxLBA=5860533168.
> So the programmable USB3/SATA bridge controller of the external case is the only detectable culprit.

This looks interesting because I also tested the corresponding model ( ST3000DM001 ) connected directly to SATA connector of various hard disk controllers and the drive also reported 4K (Advanced format) sectors there. This is why I wrote (in the answer for the report) that I'm not sure if the USB adapter caused this translation.

For example:

Total Sectors . . . . . . . . . . . . . . . . . : 732566646
Bytes Per Sector . . . . . . . . . . . . . . . . : 4096 [Advanced Format]

(when connected to Intel controller).


> For this test case, when the maxLBA value shows only 1/8 of the typical count(or a little bit less in external RAID enclosures to
> reserve space for future metadata if this JBOD volume will get member of a RAID in the future).
> it would be worth to show a comment
>

CAUTION! if you remove this HDD from its enclosure and connect it to internal SATA ports or another enclosure/docking station,
> data will become inaccessible due to change of logical sector size (at the moment 4K, emulated by the external bridge controller)

Excuse me but personally I never saw such situation. It would be nice if we can reproduce a such situation and of course if happens, a warning should be really good idea to show.


> At next chance I will tell other guys in trouble to send HDS tech reports concerning this cases to you:
> - 3TB with correct adressing / 3TB with misadressing caused by 2**32 LBA limiting drivers
> - 3TB with correct adressing / 3TB with misadressing caused by 2**32 LBA limiting BIOS
> - 3TB with correct adressing / 3TB with misadressing caused by 2**32 LBA limiting bridge chip of ext.case/docking station

Thanks !!
These reports would surely help to check the situation and possible damages which could be avoided - and/or we can be prepared for !
The best would be really to see the capacity truncation and/or a possible difference of the same drive (eg. two reports, one when used in the USB adapter and one when connected to direct SATA port).

> This will work regardless of drive initialization status or data formats.

Thanks for the tips!
I partially agree about the 1st (to check if mirrored GPT is saved) but it is still problem if the drive is not initialised, have foreign / Linux / other OS partition(s) and so (this is why we usually do not rely on stored information).

The 2nd idea (measure transfer speed) also sounds interesting, just we need to consider that the measured transfer speed may not be accurate, for example if the drive is used as a system drive as different software (and the OS) running may influence that the transfer speed measurement will be incorrect.

Thanks for the ideas and thoughts, I'm sure this is an interesting area and these HDDs may really cause incompatibility issues with older systems/adapters.

hdsentinel wrote:This looks interesting because I also tested the corresponding model ( ST3000DM001 ) connected directly to SATA connector of various hard disk controllers and the drive also reported 4K (Advanced format) sectors there. This is why I wrote (in the answer for the report) that I'm not sure if the USB adapter caused this translation.

That's interesting. Starting from December, 2011 a lot of incidents were reported to the subforum "Datenrettung"(data rescue) at the german board computerbase.de. After analyzing the faults, the summary was:
- Manufacturers(WD,ST) of exernal harddisk cases distributes models with 2,5 or 3TB capacity with a single WD or ST HDD inside
- Traders distibutes similar products under other labels like INTENSO,
- Traders of multibay-cases like ICYBOX or singlebay-cases like Enermax
All of them uses a programmable USB3.0(sometimes downgraded to USB2.0)/SATA bridge chip

Lots of problems were reported if the users tried to pull out the HDDs from the enclosure and connected them to the internal SATA, or the other way round, moved internal SATA devices to external cases. The symptoms all together were the same: partitioning and filesystem were not detected correctly, inaccessible user data, unmountable Truecrypt-Containers, and even all of the known rescue tools failed to recover from this situations.

Conclusion and solution for all this troubles was the fact, that the bridge chip changes sector size and number of sectors availiable, so the operating- and filesystem used this info to create partitioning information and NTFS partitions under calculation of 4K sectors.
Under access without the bridge chip this information, valid with 4K sectors only, caused incorrect LBA adressing in 512B sector mode.
Manual corrections of partitioning information LBA's and NTFS address pointers and metadata were successful and recovered access to the file system data.
The logical 4K sector mode also allows the usage of MBR partitioning schema above 2TiB and so even Windows XP users (where GPT support is unavailable) can make use of this >2TiB HDDs.
If you are familiar with German language, I gladly will post you a list of links concerning this error recovery threads.

So I would be really astonished if ST3000DM001 shows 732566646 4K sectors if directly connected to a SATA port.
ST's Product Manual Barracuda Gen 14 100686584 Rev. C October 2011
at http://www.seagate.com/staticfiles/supp ... 86584c.pdf (sorry, URL is off)
shows in Chapter 2.1, Specification summary, Table 1 for the ST3000DM001:
Guaranteed sectors: 5,860,533,168
Bytes per sector (4K physical emulated at 512-byte sectors): 4096

The information 4096 comes from ID data, and shows the physical size. There is another location in ID data where the information of logical sector size and the offset within the physical sector is told. This is used by the OS to calculate buffer length and structure of filesystems. (In MS Windows the circumvention of performance degradation when writing unaligned to the physical blocks is not yet implemented Booting from 4K logical sector media is also unsupported )

hdsentinel wrote:Excuse me but personally I never saw such situation. It would be nice if we can reproduce a such situation and of course if happens, a warning should be really good idea to show.

Maybe I can coerce the next patient seeking for help to produce both variants of tech report with the same HDD serial number to document this.

hdsentinel wrote:I partially agree about the 1st (to check if mirrored GPT is saved) but it is still problem if the drive is not initialised, have foreign / Linux / other OS partition(s) and so (this is why we usually do not rely on stored information).

Further explanation:
Uninitialised volumes possibly destroyment in the past is not of interest.
The external cases are specified as Linux/Mac/Windows capable only. Linux/Mac/Windows uses standard MBR for data storage medium identification.
The first check verifies the existence of
- a protective MBR with a GUID partition type=0xEE, always existing and requisite if >2TiB medium, or
- a MBR showing dynamic paritioning schema (partition type=0x42)
In the first case, maxLBA-1 holds a GPT Header (Mirror of sector 1), which is checked (and renewed, if incorrect) at each volume mount
in the second case maxLBA-1 holds a pointer to dynamic partitioning database (mirror of sector 6).
in both cases maxLBA-1 is really used by OS, and so it can be prooved if the same content exists at (maxlba-1)mod 2**x placed here by incorrect adressing (in the past or right now). If in the past, then addressing works now correct and two different sectors are read, if they have equal contents, this significant data pattern must have been placed there at the incorrect position under incorrect addressing.
If incorrect adressing takes place when probing, then the same (maxlba-1)mod 2**x sector is read twice and must be equal, so damage occured in the past (and now).
Better to detect this very common cases than nothing, and - there is no room for false positives.

hdsentinel wrote: The 2nd idea (measure transfer speed) also sounds interesting, just we need to consider that the measured transfer speed may not be accurate, for example if the drive is used as a system drive as different software (and the OS) running may influence that the transfer speed measurement will be incorrect.

Sorry, but that is no measurement of transfer speed. The time gap between simultaneous initiation and completion of two seeks to different physical location should be measured. This detects existence of a real seek delay, respectively the usage of HDD cache for the second seek request, if executed LBA is the same...
So no influence by transfer speed, rotational delay, NCQ. Only heavy I/O by other tasks to same locations that gets probed may cause a false-negative (undetected) result, if more than 50% of probes fails because data is already in cache for both requests and are completed <2ms.

hdsentinel wrote:

Next tech reports will follow soon from anywhere with description of the environment, the symptoms and a remark "by order of Ernst@at"

Thanks for the information !

This sounds really interesting. I did some further testing. I checked the drive with older USB adapter (released before drives >2 TB) and the LBA count and sector size is also reported the same way.

I'd be more than happy to see report(s) about some of such drives.

Technical background, for Your information:

Larger than 512B logical sector sizes
Reference: ATA8-ACS: http://www.t13.org/documents/UploadedDo ... A8-ACS.pdf
See page 119 /Table 22 — IDENTIFY DEVICE data (part 9 of 11)
Word 106 Bit 12 set to 1= Device Logical Sector Longer than 256 Words
Word 117-118: Words per Logical Sector
More documentation about larger physical/logical sector sizes and historical background: http://www.idema.org/?page_id=2172

Supplement 25.07.2012 09:17
Advanced Format Detection - SATA

• If an Advanced Format drive has 4,096 byte physical sectors it reports the following in IDENTIFY DEVICE data:
  1. word 106: 6003h
  2. word 209: 4001h (for adding one to each LBA) or 4000h (device adds nothing to each LBA) depending on jumper setting

Advanced Format Detection - SAS, USB Bulk Only Transport, USB Attached SCSI, and IEEE 1394

• Fields have been added to the READ CAPACITY (16) command in SBC-3 to report Advance Format device
  requirements. These fields are: Logical Blocks Per Physical Block Exponent (LSPS) and Lowest Aligned Logical
  Block Address (OSO). The physical sector size of an Advanced Format drive is specified as a 2^x value where x is the
  LSPS. For example, an Advanced Format drive with a physical sector size of 4,096 bytes would set the LSPS to 3.
  This would indicate that the physical sector size was 2³ sectors or eight 512-byte logical sectors per physical sector.
  The OSO is the first aligned logical sector address (LSA) on an Advanced Format drive.
  In case of LSA+1 jumper is present, this would be set to 7, otherwise 0

Thanks for the information!
The first part is well known, but the second part (supplement you added) is really interesting, especially the details related to the SCSI (SAS/USB) .
Thanks so much!!!

Good morning!

Now, after summer break, I had another case of >2TiB misadressing on the table.
This user sent 2 tech reports to you yesterday, a third will follow(I hope so) soon.

Case 1 shows the situation when ST33000651AS is detected only with 1.565.550.315 Sectors by Win (746,51 GiB capacity).
this HDD was attached to another PC with correct 48bit LBA adressing, initialized, partitioned, formatted and partially filled with data.
Lateron this HDD was moved to another PC with an outdated(2009) AMD AHCI driver. First destroyment has gone unnoticed when Windows volumemount noticed the absence of the GPT mirror information and rewrote it beyond maxLBA-2^32. At this time the volume was shown in Win as accessible with full capacity. After copying lot of GB's when reaching the 2TiB line, all further writes were redirected to LBA-2^32 causing overwrite of MBR, GPT info and all of $MFT. Win crashed, at restart the HDD was shown uninitialized. All data lost, nothing recoverable.

Case 2 is from the situation after AMD drivers update. Win shows full capacity, correct # of sectors, and is correct accessible in full sector range.
Note the #sectors is here 5.860.528.065 - this indicates the existence of a HPA size of 5.103 sectors. Gigabyte boards often creates a HPA of 2.113 or 5.103 size.

Case 3 will show same HDD within the external case it was initially shipped: a CNMemory Spaceloop 3,5" 3000GB USB2.0
there it is presented to windows with logical sectorsize of 4K and 732.564.000 Sectors.
strange behavior of this USB2.0 bridge-controller: 8*732.564.000=5.860.512.000 - other USB3 bridge-controllers shows full size 8*732.566.646=5.860.533.168

Thanks so much !

Yes, the first two reports are already received and examined and I see that the older driver caused the wrong capacity.
I can completely understand and agree that a warning should be immediately displayed to STOP any disk activity in this case.

I'll be so curious to see the 3rd report as well.
Some RAID controllers truncate the complete capacity of the drive to nearest 1GB (or 10GB) and make only this limited space available to the OS. Personally I did not see such for USB bridges, so I'm so curious to see that.

Thanks so much for your help!!

I got curious myself and still waiting to know what happened.

current situation: www is filled up more and more with messages of data loss on disks >2TB

After installing this early beta version: http://www.hdsentinel.com/beta4/hdsenti ... p41010.zip
the detection of misaddressing (caused by old drivers that doesn't support disks about 2TiB)
works fine.

Erroneous driver version is shown here:

HDSentinel reaction immediately after install:

HDSentinel Warning on HDD Status screen:

add to german language file following message text:

Die derzeit erkannte Kapazität (xxxx GB) der Platte ist falsch, sollte yyyy GB sein!
Die Verwendung der Platte kann schwere Beschädigung des Dateisystemes bis zum kompletten Datenverlust verursachen.
Um diese Platte ungefährdet verwenden zu können, ist ein anderer Festplattencontroller, eine neuere Version des Betriebssystemes und/oder der Treiber erforderlich.

This HDS beta version detects addressing malfunction caused by
Case 1: wrong Windows drivers.
Conditions:
disks(with 512e sectors emulated by HDD firmware) correctly initialized and formatted via

• - other system without this >2TiB addressing bug
  - preformatted external storage delivery
  - USB connection (USB adapter or external enclosure USB port)

are connected lateron to internal SATA ports or eSATA outlet
Symptoms:
due to the correct initialisation/partitioning Windows shows wrong information at administration/computer management(disk management: shown disk size and partition sizes looks normal (full size)
after write operations to disk maxLBA 2^32 and above

• - MBT/GPT information starting at LBA 0 gets destroyed, disk size is shown as size-2^32 and disk status looks uninitialized, or
  - partiton start beyond LBA 2^32 and beyond maxLBA-2^32 gets destroyed, partition status looks uninitialized or
  - strange file or directory malfunctions appears.

until now not detectable cases:
Case 2: wrong BIOS or external bridge-chip firmware.
Conditions:
all disk sizes above 2TiB, internal SATA or eSATA or USB connected.
Symptoms:
same as case 1
Detection:
singularity at distance of exactly 2^32 LBA's in seek test



under correct addressing the upper diagram shows an area of dots with increasing access time
under misaddressing the lower diagram shows a singularity: Access time is 0ms at seek distance of 0 and 2^32 sectors
possible detection method:
issue a series of read-Commands to the disk in pairs of LBA x / LBA x+2^32
x should be incremented in larger intervals to avoid positive false detections by device read-ahead cache matches.
if the second read of each pair shows in timestamp of i/o completion a difference between the first <2ms the singularity condition matches.
if a significant amount of probes shows this condition, misadressing takes place.

Case 3: old BIOS
Conditions:
all disk sizes above 500GB, internal SATA or eSATA or USB connected.
Symptoms:
i/O error when issuing commands to areas about limitation
Detection:
issue read command to maxLBA-1
if I/O error (wrong LBA) occures, addressing limitation detected

Case 4: HPA or DCO set
Conditions:
all disk sizes, internal SATA or eSATA or USB connected.
HPA or DCO set by BIOS or other service tools after partitioning and usage of disk
Symptoms:
ID command shows smaller maxLBA than expected for this disk model. No abnormality shown in Windows disk management screen
I/O errors when trying access to area hidden by HPA/DCO
Detection:
compare maxLBA with usual maxLBA per capacity table.

Thanks for your message and confirmation about the situation is improved !

Yes, Hard Disk Sentinel detects the situations which could be caused by mis-addressing and displays warning.
Also it goes further and checks if there are situations (for example if drive capacity limit jumper is used) which may limit accessing the total capacity.






I can confirm HPA / DCO limitation NOT checked and not reported. This is completely normal as in most cases, these limitations are set by manufacturers of desktop/notebook computers and offer original system image / possibility to restore system to default state. This is not really a problem, does not cause data loss or incompatibility - so these not reported.

ad case 4, HPA/DCO:

This is not really a problem, does not cause data loss or incompatibility - so these not reported.

I agree, this is of minor interest. But there are lot of mainboards in use where due to an uncorrected BIOS bug HDD capacity is shrinked by 1TB at unpredictible point of time, caused by configuration changes or BIOS checksum error(followed by BIOS configuration reset to default). This causes data loss and often destroyment of RAID configurations.
More often are unexpected issues when disks with a small HPA are used as a replacement disk of a RAID. Attach to the RAID is refused(because it's a little bit too small), and nobody knows why.

From my point of view Case 2 is of major interest, because lots of people atm buys 3TB disks and connect them to internal SATA Ports or put them in older external cases. No indication of malfunction is shown until the 2TiB level is crossed, in most cases followed by 100% data loss.
BIOS or controller firmware cuts down read/write LBA address to 32bit, but ID command passes through unmodified. So no capacity shrink is recognized by the system, even uninitialized disks are detected in full size, but commands targeting the >2TiB area are redirected to lba-2^32.
Are there any intentions to enhance this detection to this case in the near future?