Deleting fingerprints in 5.2.0 & db size

Richard's Avatar

Richard

10 Oct, 2019 02:26 PM

Hendrik,

I decided to start a new thread on this rather than continuing older threads. This has to do with some testing results in 5.2.0 and Tools/Delete Fingerprints..

I created a test set of music files in root folder "K:/beaTest/Test2". 7730 items, copied from my larger music collection, all (or most) having fingerprints created months ago with an earlier beaTunes. I fired up 5.2.0 and created a new folder-based library for the test set of files. I "Got Info" on a few files and confirmed that the little fingerprint graphic was displayed on the songs. In the app mp3tag, I sampled some files to confirm that tag BEATUNESPRINT was present.

I closed, reopened, and then closed beaTunes again, and then looked at the h2.db size. The size was 219,260 KB. I made a backup copy of that file and the accompanying trace.db (1 KB).

Then I ran "Tools/Delete Fingerprints". It appeared to run properly. Then I closed beaTunes and took a look at the db file size. The size went down to 60,972 KB, a 72% reduction! Looked at another way, that's a db reduction of almost 20.5 KB per track, which makes me wonder what went away.

I opened and closed beaTunes again - still 60,972 KB. I opened beaTunes again. It still showed 7730 items. A cursory examination showed that the library listing appeared normal. I "Got Info" on the same files as before. Interestingly, they all still show the little fingerprint graphic. I expected that the fingerprint image would have gone away.

So, I thought you would be interested in this result. I actually tried some of this before (which is why this is "Test2") under less controlled circumstances and was seeing similar results. I wanted to be more deliberate with the new test and record all my steps along the way. I tried to upload the logs as I thought you might be interested to see them, but I got a "java.io.IOException: Error writing to server". If I can overcome the error, I'll upload the logs shortly.

(Update: I don't know how to resolve the Upload Logs error. Maybe the destination server is offline? I'll try again later.)

Richard

  1. Support Staff 1 Posted by hendrik on 11 Oct, 2019 06:51 AM

    hendrik's Avatar

    Hey Richard,

    Thanks for trying this out! Looks like you got some nice results. :-)

    Unfortunately, the upload logs mechanism in 5.2.0 is unfortunately broken, because of an annoying TLS incompatibility issue between Java11 and Cloudflare. It will be operational again in 5.2.1.

    You could manually zip the logs (see here for where to find them) and attach them to this discussion, but I don't think they will provide much insight. Sounds like things are working as intended.

    Cheers,

    -hendrik

  2. 2 Posted by Richard on 11 Oct, 2019 03:11 PM

    Richard's Avatar

    Nice results indeed. I'll try this on my regular library soon and report the results here for those who may be interested.

    I zipped the "logs" folder per the instructions and will attach it to this message. This is just in case you want to look at them. I was happy but surprised to see the db size go down so much.

    Maybe the fingerprint graphic still shows up in the "Get Info" display because the fingerprint is still in the file itself, even though it's no longer in the db? If so, that would make sense.

  3. Support Staff 3 Posted by hendrik on 11 Oct, 2019 03:46 PM

    hendrik's Avatar

    Maybe the fingerprint graphic still shows up in the "Get Info" display because the fingerprint is still in the file itself, even though it's no longer in the db? If so, that would make sense.

    I suspect the song object was still cached. When you restart beaTunes (that empties the cache for sure), do you still see the fingerprint icon?

  4. 4 Posted by Richard on 11 Oct, 2019 05:21 PM

    Richard's Avatar

    I still see the fingerprint icon.
    I opened beaTunes new just now. I "Got Info" on the first song displayed, and then clicked ">" to see the next songs. Of the 50-ish songs I looked at, only 1 did not have the icon. I pulled that one up in mp3tag (where if you display "Extended tags" you can see details on all of the tags in the file), and that particular song does not have the BEATUNESPRINT tag, but the others I looked at do have the tag. So that corresponds to beaTunes displaying the icon consistent with the presence or absence of the tag in the mp3 file itself, at least for the ones I looked at.
    Most of the songs in my test set do have the fingerprint tag. I guess at least a few of them don't.

        On Friday, October 11, 2019, 10:46:21 AM CDT, hendrik <[email blocked]> wrote:

    |

  5. Support Staff 5 Posted by hendrik on 12 Oct, 2019 09:52 AM

    hendrik's Avatar

    You know... when you open the Get Info dialog, beaTunes attempts to look up metadata. For that it uses the existing metadata. If nothing can be found, it automatically computes a fingerprint and uses it for lookup. The fingerprint is then also stored in the songs (i.e., database and file). Perhaps that's what you are seeing.

  6. 6 Posted by Richard on 12 Oct, 2019 03:13 PM

    Richard's Avatar

    So, it appears we have 4 cases to consider when a user does "Get Info" on an mp3 file in a folder-based library. (I'm putting my [retired] analyst hat on and constructing these use cases based on what you said below, plus observations on a sample of my files. I'm necessarily approaching this as a "black box" situation.) Is this what happens in each case?

    #1 precondition - Fingerprint in the metadata + fingerprint already stored in the db.#1 action - beaTunes sees fingerprint metadata. Fingerprint icon is displayed. beaTunes sees a fingerprint in db for the track. It does not check to see if the metadata fingerprint matches the db fingerprint. The end.

    #2 precondition - Fingerprint in the metadata + fingerprint NOT already stored in the db.#2 action - beaTunes sees fingerprint metadata. Fingerprint icon is displayed. beaTunes does not find a fingerprint in db for the track. beaTunes does not compute a "new" fingerprint [that's a guess on my part]. Instead, beaTunes inserts the metadata fingerprint value into the db. (The next time this track is handled by "Get Info", it becomes case #1 above as the fingerprint would now be stored in the db.) The end.

    #3 precondition - Fingerprint NOT in the metadata + fingerprint NOT already stored in the db.#3 action - beaTunes does not find a fingerprint in the metadata. It computes one and "uses it for lookup". (These are your words. I'm not sure what that means exactly.) It does not insert fingerprint metadata into the music file. It does not display an icon. It does not find a fingerprint in the db. beaTunes computes a fingerprint value and inserts it into the db (per what you said below). (The next time this track is handled by "Get Info", it becomes case #4 below as the fingerprint would now be stored in the db.) The end.

    #4 precondition - Fingerprint NOT in the metadata + fingerprint already stored in the db
    #4 action - beaTunes does not find a fingerprint in the metadata. It computes one and "uses it for lookup". (Again, your words.) It does not display an icon. It does not insert fingerprint tag into the file. It sees a fingerprint in the db. The end.

    I will say I don't think I have the actions described correctly because some of them don't make sense to me, notably that there actually is a db insert in cases #2 and #3.

    What I can say, based on observation of a handful of my own files, is that the icon is definitely not displayed in cases 3 & 4, and (using mp3tag v 2.98 as an inspection tool) I do not see evidence of fingerprint metadata being inserted into the file in cases 3 & 4. (It could be that I'm using mp3tag incorrectly or that mp3tag isn't a good tool for this purpose.)

    Yes, I did go to nerd school.

        On Saturday, October 12, 2019, 4:52:50 AM CDT, hendrik <[email blocked]> wrote:
     
       
    |

  7. 7 Posted by Richard on 16 Oct, 2019 04:01 PM

    Richard's Avatar

    I decided to take the plunge and Delete Fingerprints on my regular library. I'm pleased to say the results were consistent with my earlier tests, even better.

    The pre-Delete library db size was 11,221,720 KB, this after being reduced somewhat by the upgrade from v 5.14 to 5.2.0. The folder-based library itself has 374,388 items, which worked out to 30 KB of db per library item.

    The Delete itself took 10 minutes to run. I then shut down beaTunes, and it took 4 minutes to complete the shutdown. The immediate post-shutdown db size was reduced to 9,541,220 KB. Knowing how beaTunes allocates time at shutdown to do further db maintenance, I cycled through 27 more open/close iterations to arrive at a final db size of 2,943,188 KB, a 74% reduction. Remarkable!

    Hendrik, thanks again for implementing this option. beaTunes is now SO much more responsive. Honestly, with my library size beaTunes just wasn't usable on an everyday basis due to the sluggishness, even with the db residing on an SSD. Based on my experience so far, that is no longer the case. I'm looking forward to being able to take advantage of beaTunes' many features on a routine basis.

  8. Support Staff 8 Posted by hendrik on 17 Oct, 2019 01:49 PM

    hendrik's Avatar

    Thanks, Richard, for your feedback. And yes—I'm aware that too large a database file makes things a lot slower. And that's something that beaTunes sure as hell does not need. Rather the opposite!

Reply to this discussion

Internal reply

Formatting help / Preview (switch to plain text) No formatting (switch to Markdown)

Attaching KB article:

»

Attached Files

You can attach files up to 10MB

If you don't have an account yet, we need to confirm you're human and not a machine trying to post spam.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac