Hard Drive SMART History missing past a certain date

[Rings]

I only just realized that my main NVMe is missing all drive health data before August 2023, as well as missing all SMART data before Jan 2024. I've had this NVMe installed since 2019 and had Hard Disk Sentinel installed since at least 2020 if not 2019.

I do have some backups of my full Windows partition which would include Hard Disk Sentinel's Program Files (x86) folder that should presumably fill the missing data, assuming that's where it would be stored, but I don't know which files they would be stored in (is it the files with .HDD extension?) nor how to merge them into the current .HDD files (can it be done via text editor?).

I really want to restore my drive's history to see its graphs since installation without also losing the data since then. How can I go about doing this? And how can I prevent this from happening in the future? I suspect this happened due to my system possibly crashing at one point (GPU failure)

[hdsentinel]

I'm afraid by default some of the details stored in the registry too, not only in the data files in the folder of the software.
So even if you have very old data files, it may be not really possible to edit/merge with the current status files (or may be not trivial, but this depends on some factors).

If you prefer, please send the following files in e-mail to info (at) hdsentinel (dot) com address (maybe compressed with ZIP/RAR/7Z)
- the current DISKDATA* file(s) (may be more than one) for the appropriate SSD from the folder
- previous DISKDATA* file(s) for the appropriate SSD you have

and use Report menu -> Send test report to developer option in the software about the drive.

So I can check the possibilities and try to merge them.


In order to try to avoid / minimise similar, maybe you can try to use the Configuration -> Advanced options -> Change folder to store statistics and settings button and specify an empty folder to be used as "data" folder in the future.

If this option used, then ONLY this folder used for storage (the registry will be not used) so then you can periodcially backup this folder (with all statistics) just to be sure.

[Rings]

Thank you for your response.

My concern now is that the drive that I want to try to restore SMART data for is not in the registry for some reason. Every other drive has their own folder in HD Sentinel's SMART folder in the registry, except for my boot drive. The boot drive does, however, have a key in the main HD Sentinel registry folder with a value of 3600(900/11645,11645).

My backup that I mentioned should include a full registry backup, as it's a backup of my entire Windows partition, but I'm just not sure what to do here as far as merging keys goes when I don't even have a folder in the SMART folder in the first place. I figure I can add the folder myself, but HD Sentinel clearly isn't getting its data for this particular drive from the registry at this point so I don't know what good that would do.

I used the Configuration -> Advanced options -> Change folder to store statistics and settings button last night and pointed it to HD Sentinel's Program Files (x86) folder, which basically added a reg.dat file which I understand is now what is being used instead of the registry. However, before I did that, I set the folder to an empty folder, and noticed that it recreated all the .HDD etc. files, leaving me with 240 additional files in this folder. I performed a hash check on them against what was already in the Program Files (x86) folder, and only the registry.reg file was modified, and reg.dat was newly created. Everything else matched. This is why I set the folder back to Hard Disk Sentinel's default Program Files (x86) installation folder.

As a side note, I was wondering what the default location is when you use the above mentioned button, or how to disable this setting if I ever want to go back to using the registry?

Anyway, I don't mind sending you these files, but I'd like to know why my boot drive still isn't included in the registry/reg.dat first? I'm unsure where my boot drive is getting its SMART data from when it seems to be absent from any of the registry locations, and only has a .HDD and .HDI file (no .DAT) while my other drives have .HDD, .HDI and .DAT files. I'm sure if the registry entry simply existed, I'd be able to figure out how to merge the data from my backup to the current readings myself, but as it stands I don't know where it's pulling the info from to begin with.



The drive in question is the WDS at the bottom. It's a WD Black Sn750 NVMe.

[hdsentinel]

The registry is used for IDE and SATA disk drives only.

For other drives, the design changed: for SCSI, SAS, NVMe, PCIe/M.2 SSDs, eMMC devices, tape drives, etc the registry is no longer used, so if you have the old and new files, then there is good chance that you can "combine" them.

Exactly to be easy to check and process, the *_scsi.hdd file is a simple text file with different sections storing the SMART values for each days.
So you may simply try to copy / paste the contents of the old file to the top of the new (latest) _scsi.hdd file. Just make sure to exclude any possible

[General]
LastDay=45000

or similar lines (if present in the OLD file) to prevent troubles, but you can simply copy/paste the rest of the old file to the new _scsi.hdd file related to the appropriate NVMe SSD.

[Rings]

Thank you for the heads up! So it seems that for whatever reason, I'm unable to simply merge the 2 files together without issue.

The old file starts with

Code: Select all

[44883]
SMART=FF_1200=00_00,FF_1201=00_0143,FF_1203=00_64,FF_1204=00_0A,FF_1205=00_00,FF_1220=00_0000000000000000000000000A8D4305,FF_1230=00_000000000000000000000000062606A6,FF_1240=00_000000009E2D8067,FF_1250=00_000000004E711AB0,FF_1260=00_0000000000000000000000000000159B,FF_1270=00_0000000000000000000000000000020D,FF_1280=00_000000000000000000000000000030F7,FF_1290=00_00000000000000000000000000000017,FF_12A0=00_00000000000000000000000000000000,FF_12B0=00_00000000000000000000000000000000,FF_12C0=00_00000004,FF_12C4=00_00000000
THRESHOLDS=

[General]
LastDay=45022

and ends with day 45022. Meanwhile, the current file starts with

Code: Select all

[45288]
SMART=FF_1200=00_00,FF_1201=00_0151,FF_1203=00_64,FF_1204=00_0A,FF_1205=00_00,FF_1220=00_00000000000000000000000015BDB1E7,FF_1230=00_0000000000000000000000000A6EAB0A,FF_1240=00_0000000142EA9B30,FF_1250=00_0000000092DB7A73,FF_1260=00_00000000000000000000000000002926,FF_1270=00_0000000000000000000000000000028A,FF_1280=00_00000000000000000000000000005307,FF_1290=00_0000000000000000000000000000002E,FF_12A0=00_00000000000000000000000000000000,FF_12B0=00_00000000000000000000000000000000,FF_12C0=00_0000000D,FF_12C4=00_00000000
THRESHOLDS=

[General]
LastDay=45449

and ends with day 45449. Following what you said, then, I added the old file's contents to the begging of the current file, making sure to remove the old file's [General] line with LastDay=45022, and move the current file's respective line with LastDay=45449 to the top such that it's after the first entry [44883]. The current file now has a gap where between the entries for [45022] and [45288].

However, unfortunately what Hard Disk Sentinel shows as far as health goes is still only as far back as August 17, 2023, same as before, but now I notice the SMART data graphs show more than before, starting at Nov 11, 2022 and ending at April 6, 2023 (the date this backup is from), with a gap between April 6, 2023 and Jan 1, 2024, and then from Jan 1, 2024 till present. So I checked with a copy of the current .HDD file I made before making any changes (of course restarting HD Sentinel before restoring it), the SMART data graphs go as far back as... December 28, 2023 (the graph is higher resolution so it shows every 3 days as opposed to every 12, hence Jan 1 and Dec 28 being the same thing here). Huh. So in either case, despite the Health showing as far back as August 17, 2023, the SMART value shows less.

Do you know why that may be? Is the SMART data stored in the same .HDD file? I would assume so, since restoring the old file's data to the .HDD file added more SMART data. But then why does only the SMART data stop at Dec 28, 2023, while the Health goes back to Aug 17, 2023? And why doesn't the health show further back when merging the old .HDD's contents into the current? Does it not like date gaps?

Edit: I'm so glad I'm a data hoarder: I found some more backups of my C drive that I never deleted, allowing me to restore my SMART data as far back as 2020, when I first started using the drive! So that's the full drive's SMART history recovered, minus an unfortunate gap between Aug 16, 2023 and Dec 28, 2023. Still, the Health doesn't go beyond August 16, 2023. Would love to get to the bottom of that! It's 100% still, but I want to learn how to restore missing Health data because my 2 Crucial SATA SSDs are deteriorating at a rate of 1% every couple weeks and I'd like to know when it started.

As well, how would I do the above for my SATA drives? Their .HDD files show invalid characters when edited with Notepad++.

[hdsentinel]

> The current file now has a gap where between the entries for [45022] and [45288].

Probably no data recorded for those days.
As you may see, each section represents one day - and probably no data recorded in any of the files you have.
Generally no problem if one or more days missing, as it is "normal": if Hard Disk Sentinel not started (or the PC not powered at all, the drive not connected at all) then it is expected that no data recorded for a disk drive on a specific date.


The Health % / Temperature history is completely different, it is independent from this file. That history is recorded differently in HDSentinel.STA file for all drives. So you can go back to SMART history independently from the Health history (and vice versa).


> As well, how would I do the above for my SATA drives? Their .HDD files show invalid characters when edited with Notepad++.

It is not possible at all. There are no invalid characters: as I wrote previously, for IDE/SATA drives, the registry is also required and their files are completely different (these are not text files). So I'm afraid for SATA drives you can't do any edits/merge or so.

[romanshein]

Dear HDSentinel

I have a few questions about your comments:
1) Is there a plan to unify log storage for SATA drives? From what I see those are getting reset each time I restore Windows from backup/move Windows drive to a different drive etc.
2) "Configuration -> Advanced options -> Change folder" seems to be gone for some time (at least since 6.1). Is it correct?
3) Drive temperatures are stored in a separate file (HDSentinel.sta). I also have the reg.dat file in my data directory. Reg.dat keeps records of HDD which was gone years ago, yet all temperature data got reset and HDSentinel.sta was completely reset since I moved the Windows partition a week ago, even though HDSentinel IS NOT located in the Windows drive.
Is there an HDSentinel documentation section describing how the log facility operates? In general, as a user, I find HDSentinel logging retention to be confusing. I wish I could understand it better to be able to preserve history across Windows restores, reinstalls and HDSentinel upgrades.

Feature suggestion:
It looks like HDSentinel is not able to handle the history between drive firmware upgrades. I have upgraded firmware for my Crucial SSD the drive descriptor changed to DISKDATA_CT2000MX500SSD1_2012E2963E25_M3CR033 (where M3CR033 is the firmware version). The drive and its connection path haven't changed at all, yet HDSentinel handles the drive as a completely new one. HDSentil is quite unique in its ability to read drive data respectively if the drive is connected directly via SATA or is behind a raid card. It should be feasible to identify the physical drive by its serial number or World Wide ID, not by it current Windows connection. I believe it will raise the HDSentinel value for the end users.

[hdsentinel]

> 1) Is there a plan to unify log storage for SATA drives?

No, there is no plan for that - as there would be no reason for that.

There are two different "official" easy ways to backup all settings, statistics easily:

a) You can use Configuration -> Update -> Backup configuration to save all settings, statistics, registration and activation details to a single file.
After installation of the software on the new installation, this file can be load back by Configuration -> Update -> Restore configuration.
This is the preferred way to keep existing statistics/logs for drives you want to use after re-install

b) You can use the Configuration -> Advanced options -> Change folder option to specify a folder to store all details.
Then you can simply backup/restore that folder. Then after a reinstall, specify the folder similarly AND THEN put back the saved files there to automatically use the previous settings, statistics etc.


> From what I see those are getting reset each time I restore Windows from backup/move Windows drive to a different drive etc.

If you simply copy/move files then yes, I'm afraid this is completely normal and expected.


> 2) "Configuration -> Advanced options -> Change folder" seems to be gone for some time (at least since 6.1). Is it correct?

Absolutely not. It is not "gone", it is available in all _installed_ versions of Hard Disk Sentinel Professional.
If you do not see this option, then you may use the Portable version: in that, yes, it is completely normal and expected that this option is not available, as the Portable version designed to store details in its own folder, NOT in any (other) fixed folder on the actual system (as then it would be not Portable)


> 3) Drive temperatures are stored in a separate file (HDSentinel.sta). I also have the reg.dat file in my
> data directory. Reg.dat keeps records of HDD which was gone years ago, yet all temperature data got
> reset and HDSentinel.sta was completely reset since I moved the Windows partition a week ago, even
> though HDSentinel IS NOT located in the Windows drive.

The location of Hard Disk Sentinel is not important. If you copy/move files, I'm afraid you may expect loss of previous statistics.
You'd need to use the above mentioned a) or b) solution if you prefer to backup/restore (and keep after re-installation).


> Is there an HDSentinel documentation section describing how the log facility operates?

No, I'm afraid no. There is no need for that, as generally the files you mentioned are not designed to be edited/processed by the user.


> In general, as a user, I find HDSentinel logging retention to be confusing.
> I wish I could understand it better to be able to preserve history across Windows restores, reinstalls and HDSentinel upgrades.

Absolutely not confusing. Please see above: there are two different solutions to keep all settings, statistics, etc...
I'm afraid it is normal and expected that manually checking / editing / copying / moving any file(s) can cause lost statistics.


> Feature suggestion:
> It looks like HDSentinel is not able to handle the history between drive firmware upgrades.

Not "not able" but yes, what you see is completely normal and expected behaviour for many reasons, designed exactly that way.

The reason is very simple: firmware update(s) can affect the operation of the drives and can sometimes can cause different set of S.M.A.R.T. attributes and/or even slightly different functionality/features. This is why the firmware version also used to identify the drive (in addition to the model and serial number as you can see).


> I have upgraded firmware for my Crucial SSD the drive descriptor changed to DISKDATA_CT2000MX500SSD1_2012E2963E25_M3CR033
> (where M3CR033 is the firmware version). The drive and its connection path haven't changed at all,
> yet HDSentinel handles the drive as a completely new one.

Yes, this is completely normal and expected behaviour. What you see as problem/limitation is the only good solution.


> HDSentil is quite unique in its ability to read drive data respectively if the drive is connected directly via SATA or is behind a raid card.

Thanks


> It should be feasible to identify the physical drive by its serial number or World Wide ID,
> not by it current Windows connection. I believe it will raise the HDSentinel value for the end users.

Of course Hard Disk Sentinel never identifies the drive by the "current Windows connection", exactly because it can change.
If uses the combination of the serial number (as you can see), the firmware version and model ID.
The World Wide ID sounds a good option too - but there are problems with this approach:
- it may be not always detected: you mentioned the RAID cards. Some RAID controllers do not allow to check/read the World Wide ID
- if only that would be used, then the firmware update would be "ignored". While you would love that (as you feel the current implementation has limitations) I can confirm that this IS the preferred way.