Removing metadata

Richard's Avatar

Richard

21 Feb, 2019 05:52 PM

Good morning Hendrik,

I've been working with the Key field lately, using Analysis to add Key to the tracks that don't already have it (which is quite a few as I purposely did not have Analysis populate that field for me earlier).

That led me to looking at the size of my h2 database which is pretty large (nearly 12 GB) which makes working with beaTunes difficult at times. My thinking is, if I could remove some of the metadata from my songs, that might reduce the size of the db. Specifically, I was thinking about removing Fingerprint because it looks like it takes up a lot of space relative to other metadata, and I'm not sure what benefit I get from having all my tracks fingerprinted. I;m also looking at removing Color, Instrumentation, Danceability and maybe Mood. So my first question is, can those metadata be removed? And would that shrink the size of the db file?

If the db can't be shrunk this way, I guess my only other option would be to blow the existing db away, resynchronize from scratch, and then reanalyze, checking off just the items I want included (and in the main, not ticking "create fingerprint"). Does that sound right? I kind of hate to do that because I know how long it will take to reprocess everything.

Second question, kind of unrelated: Virtually all of my tracks have the fingerprint, and when I did the Analysis, I ticked the box that says "Submit fingerprint to the online database". In the Tools menu, there is the opportunity to "Submit fingerprints to server". I assume, because I ticked the box at time of Analysis, that all the fingerprints were submitted at that time, yes? There is no indication that any given fingerprint has actually been submitted, is there? I just wondered about that.

Finally, (and this is a little off-topic), is v 5.1.12 going to be officially released any time soon? I have 5.1.12-SNAPSHOT that you sent me, which is working fine. The reason I am asking is that I noticed in the "Algorithm" mp3 tags for each song, it includes the beaTunes version that created that particular tag, including the characters "-SNAPSHOT". Solely in the interest of not writing the extra bytes to each mp3 file (and maybe to the db, too?), I was hoping a non-SNAPSHOT version of 5.1.12 might be ready soon.

Thanks as always. I'm sorry that I seem to be the main discussion creator in the forum the past few months.

  1. Support Staff 1 Posted by hendrik on 25 Feb, 2019 10:15 AM

    hendrik's Avatar

    Hey Richard,

    the fact that the database occupies X amount of space on your harddrive does not mean that it uses all of that space. Some space may just be empty. Every time beaTunes shuts down, it attempt to reclaim some of that space—but with a time limit. That means, beaTunes will only spent so and so many seconds on this “compression” procedure to not make the shutdown longer than necessary.

    The best way to claim this empty space (*if* there really is any) is to “Repair" the database. To do so:

    - Make sure you have *plenty* of space on the disk that the beaTunes database resides on.
    - Create a backup of the current database while beaTunes is NOT running.
    - Start beaTunes and select Tools -> Repair Database (NOT rebuild).
    - Restart beaTunes (you’ll be prompted for this).
    - Repair with a database of that size will take a while.

    I’d be curios to know:

    1. How many songs are in your library
    2. Database size before and after repair

    > I'm not sure what benefit I get from having all my tracks fingerprinted.

    The duplicates recognition can be based on acoustic properties rather than just textual metadata. Also, when looking up additional information the fingerprint may be used. For simply listening, there is no additional benefit.

    > I;m also looking at removing Color, Instrumentation, Danceability and maybe Mood.

    Removing any of these will hardly save any space.

    > If the db can't be shrunk this way, I guess my only other option would be to blow the existing db away, resynchronize from scratch, and then reanalyze, checking off just the items I want included (and in the main, not ticking "create fingerprint"). Does that sound right? I kind of hate to do that because I know how long it will take to reprocess everything.

    Any data that is stored in the audio files will be imported when you re-create the library. Fingerprints are always stored in the files for folder-based libraries and depending on your settings for iTunes-based libraries. So re-creating the library won’t do you any good.

    Regarding the submission of fingerprints: They probably have been submitted already. Note that submission is to benefit others.

    Regarding the release of 5.1.2: I hope that’s going to happen soon.

    Hope this helps!!

    -hendrik

    PS: Sorry for the late answer. Tip: You almost always get quicker answer, when you create a new discussion for every question, i.e. making questions shorter and more succinct.

  2. 2 Posted by Richard on 25 Feb, 2019 02:51 PM

    Richard's Avatar

    I was happy to help the community with the fingerprints as I have benefited from the online community many, many times over the years. It looks like the downside effect of that is messing me up in this particular regard right now, but I'll deal with it. Using another tool, maybe there's a way to remove a given mp3 tag (the fingerprint tag) from a population of mp3 files? Then, if I would recreate the library, there would be no pre-existing fingerprints to import. Oh well - not a priority at this time, and maybe never.

    Yes, sorry about the compound questions. The add-on questions were things I thought of while composing my post on the original topic.

    At this moment, the library (folder-based) is 367964 items, or 2653.9GB. With beaTunes not running, the h2.db is 11,351,248 KB.

    I'm running the db on a 5-year old SSD, so it's not all that big. I'm clearing some space now and will start the Repair shortly.

    Thanks again for your help, as always. It's my hope that these Q&A's are of benefit to the entire beaTunes user base, not just to me.

  3. 3 Posted by Richard on 26 Feb, 2019 01:23 AM

    Richard's Avatar

    Here is my follow-up. I did the steps you suggested.

    The Repair didn't take as long as I feared: 1 hour, 23 minutes.

    After Repair finished but before I closed beaTunes, the h2.db was 21,697,064 KB. Pretty big.

    After I closed beaTunes, the db shrunk to 19,738,220 KB. Still way bigger than before.

    Recalling what you said above, I did a series of opens and closes of beaTunes, recording the db size after each close. I waited sufficient time between each open and close for beaTunes to "settle down" to minimal CPU and I/O activity. (I had my eye on Process Hacker, a Task Manager-type app.) Here are my results (in KB):
    after 2nd close 19,317,204 - roughly 600,000 KB less
    after 3rd close 18,904,528
    after 4th close 18,491,764
    after 5th close 18,078,808 - tapering down to 400,000 KB less
    after 6th close 17,654,280
    after 7th close 17,232,444
    after 8th close 16,818,716
    after 9th close 16.397.620
    after 10th close 15,993,656
    after 11th close 14,061,520 - big drop from 10th close
    after 12th close 13,472,316 - 600,000 KB less
    after 13th close 13,313,216 - now tapering off
    after 14th close 13,192,568
    after 15th close 13,124,336 - tapering off some more
    after 16th close 13,056,516
    after 17th close 12,986,220
    after 18th close 12,910,068
    after 19th close 12,834,744
    after 20th close 12,761.096
    after 21st close 12,690,168
    after 22nd close 12,621,548
    after 23rd close 12,545,152
    after 24th close 12,459,652
    after 25th close 12,363,496
    after 26th close 12,265,864
    after 27th close 12,172,224
    after 28th close 12,069,864
    after 29th close 11,946,972
    after 30th close 11,823,588
    after 31st close 11,696,452
    after 32nd close 11,666,576 - tapering off even more, getting tedious
    after 33rd close 11,636,600
    after 34th close 11,606,780
    after 35th close 11,575,540
    after 36th close 11,540,784
    after 37th close 11,533,344
    after 38th close 11,533,344
    after 39th close 11,533,344
    after 40th close 11,533,344
    At that point, I concluded it wasn't going to get any smaller. Note that the final db size is roughly 180,000 KB larger than what I started with at the beginning of this process. I did not do anything else with beaTunes during this time, such as synchronize. I could restore my old db, but I figured I would just leave the current one in place as it is less than 2% larger than the old.

  4. Support Staff 4 Posted by hendrik on 26 Feb, 2019 10:20 AM

    hendrik's Avatar

    Hm... this very much sounds like a fail. Thanks for trying this though!

    You can remove all fingerprints from the database and audio files by using this simple beaTlet: https://github.com/beatunes/beaTlet-samples/blob/master/groovy/Remo... (see https://www.beatunes.com/en/beatlet-getting-started.html for installation instructions and general info).

    Once installed, restart beaTunes and you will see a new item in the Tools menu. Note that using this plugin will touch all of your audio files (if they contain fingerprints). This cannot be undone. As always when doing such a mass operation, I recommend keeping a backup of your audio and database files.

    I have tried this on a smaller library and it got a little smaller. I'm not sure though it's worth the effort. Effects may only be visible after several restarts, because (as you have seen) beaTunes always attempts to compact the database a little bit when it's shut down.

    But perhaps with 367,964 tracks every little bit counts. BTW, the script is easily adjustable to remove other items as well, if you're a little into programming.

    Thanks again for your help, as always. It's my hope that these Q&A's are of benefit to the entire beaTunes user base, not just to me.

    It definitely does.

  5. Support Staff 5 Posted by hendrik on 27 Feb, 2019 06:45 PM

    hendrik's Avatar

    Just FYI: 5.1.12 is out—https://blog.beatunes.com/2019/02/update-beatunes-5112.html

    Did the beaTlet help?

    -hendrik

  6. 6 Posted by Richard on 27 Feb, 2019 07:06 PM

    Richard's Avatar

    I very much appreciate the beaTlet. It's going to take some time for me to get prepared to run that. It's on my "to do" list but deferred atm.

    Meanwhile, I will install 5.1.12.

  7. Support Staff 7 Posted by hendrik on 01 Mar, 2019 03:55 PM

    hendrik's Avatar

    Alright. Let me know how it goes when you get to it.

Reply to this discussion

Internal reply

Formatting help / Preview (switch to plain text) No formatting (switch to Markdown)

Attaching KB article:

»

Attached Files

You can attach files up to 10MB

If you don't have an account yet, we need to confirm you're human and not a machine trying to post spam.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac