adding a second root folder to a library

Richard's Avatar

Richard

30 Aug, 2018 03:34 PM

Hello Hendrik,

I seem to be monopolizing the "Help" discussions. Sorry about that. I hope others will find some of these threads informative.

Today I am preparing to add "folder_b" to the existing library that contains only "folder_a" at the moment. I read the blog post on this and understand the edit I need to make to preferences2.xml.

Background: You may recall that my "folder_a" is kind of big, containing 87k tracks. It took awhile to get them all synchronized and analyzed, and we ran into a couple of problems and issues which you were so kind to help me through and address. "Folder_b" is bigger: 270k tracks.

I want to do this the best way possible because of the amount of computer time that will be involved, so here is my question: Should I (1) update preferences2.xml FIRST by adding a new <root> entry to the existing library, synchronize and THEN begin doing the analysis, knowing that 87,000 tracks of the newly-combined 357,000 have already been analyzed? OR (2) should I create a NEW library with just folder_b, do all the analysis (which will take many days of computer time), and THEN update preferences2.xml to merge the two libraries into one? Or (3) does it make any difference?

It seems to me that conceptually option (1) would be better. Why create a 2nd library only to merge it into another one later? But I'm concerned the analysis portion will be more time-consuming due to having to skip over the already-analyzed tracks. I WILL be using the option to not overwrite existing tags in every instance, so I won't actually be reanalyzing the original tracks.

Thanks for your help once again.

(System: Windows 8.1, folder-based library, beaTunes 5.1.9 "snapshot")

  1. Support Staff 1 Posted by hendrik on 30 Aug, 2018 03:58 PM

    hendrik's Avatar

    Frankly, adding 270k songs will be interesting. I assume you have set the heap memory to some fairly high value?

    Regarding your question:

    I agree, (1) is conceptionally the preferred method. The additional overhead should be minimal.

    That said, back up your library (the beaTunes database) before you try this, so that you can go back to a working starting point, in case something goes wrong! Create the backup, while beaTunes is not running.

  2. 2 Posted by Richard on 30 Aug, 2018 04:19 PM

    Richard's Avatar

    I will back up the library db. Thanks for the reminder.

    What heap memory value would you recommend? Right now my setting is -Xmx3072m. My PC has 16GB installed RAM, which was a lot when I got it 4-plus years ago. Now, not so much.

    beaTunes ultimately handled the 87k tunes folder OK. It took awhile, of course, to analyze it all. It analyzed somewhere around 1200 tracks per hour running all 4 threads I had available. I kept feeding it chunks of tracks at a time, which it handled fine. Ultimately, I just gave it the final 30k tracks or so all at once.

          From: hendrik <[email blocked]>
     To: [email blocked]
     Sent: Thursday, August 30, 2018 10:58 AM
     Subject: Re: adding a second root folder to a library [Questions #10073]
       
     #yiv2665034867 pre {width:92%;margin:10px 2%;padding:5px 2%;background:#efefef;border:1px solid #d6d6d6;}#yiv2665034867 blockquote {margin-left:0;padding-left:1em;border-left:5px solid #ccc;}

  3. Support Staff 3 Posted by hendrik on 31 Aug, 2018 09:11 AM

    hendrik's Avatar

    -Xmx3072m should be enough, as beaTunes makes an effort not to holed everything in memory. So that's what I would start with.

  4. 4 Posted by Richard on 05 Oct, 2018 03:55 PM

    Richard's Avatar

    I put this project on the shelf for the past 5 weeks or so but am picking it up again now. I did my backups. We'll see how it goes.

  5. 5 Posted by Richard on 06 Oct, 2018 03:08 PM

    Richard's Avatar

    Just to make sure I had enough resources, I upped the environment memory variable to -Xmx4096m. The synchronization of the newly added folder took about 3 hours. The db file went from 1.3GB to 3.2GB. I backed that up. Now I'm going to try taking the fingerprint (with "Submit to online db" checked) and estimating BPM (without using online resources and without replacing existing BPM). I'll be using 3 threads out of 4 available. I'll save other analyses for another day.

    It loaded 341438 tracks into the analysis queue, so we'll see what happens!

    Update, 3 hours in: Even though I selected not to replace existing BPM, and I've already taken the fingerprint on my original 87000 tracks, it still seems to be spending significant time on the tracks where the BPM is already calculated. I expected those to go much faster... essentially to skip over them. At least it doesn't appear to be modifying those files again on disc after processing is completed, which is good.

  6. 6 Posted by Richard on 07 Oct, 2018 03:24 PM

    Richard's Avatar

    24 hours in and it is averaging about 3500 records per hour. 73 hours to go.

    Hendrik, if I pause analysis and then close beaTunes, when I reopen beaTunes, will the unprocessed task queue still be there, ready to resume? Or does it clear the remaining task queue upon exit?

  7. Support Staff 7 Posted by hendrik on 07 Oct, 2018 03:32 PM

    hendrik's Avatar

    Hendrick, if I pause analysis and then close beaTunes, when I reopen beaTunes, will the unprocessed task queue still be there, ready to resume? Or does it clear the remaining task queue upon exit?

    It should not clear the queue.

  8. 8 Posted by Richard on 10 Oct, 2018 06:21 AM

    Richard's Avatar

    Status report: 86 hours later (now 1am on October 10) and the task queue just completed. After closing beaTunes, the h2.db file has now grown to 10,972,252 KB (!), an increase of almost 7.8 GB, a greater than 200% increase in size of an already pretty large file.

    I reopened beaTunes to do my next steps, discovered the new 5.1.10 release, and upgraded.

    Close and reopen again. Now I am resynchronizing with the library. It seemed like a good idea, and there are a few changes I wanted beaTunes to pick up. I was hoping this synchronization would go a lot quicker than the one I did before because it isn't adding a ton of new tracks. Just a handful, and picking up a few other tag changes I made with another app. However, it looks like it might be another long one. 12 minutes in and I only have about 3 mm on the progress bar. I'll let it run and see where I am in the morning.

  9. 9 Posted by Richard on 10 Oct, 2018 05:12 PM

    Richard's Avatar

    Day 4

    The synchronization took about 4 hours.

    For reasons unknown, beaTunes did not calculate a BPM value for about 17,200 tracks. Some weren't calculated because I excluded a few genres from the calculation. It appears that most of the rest that didn't get a BPM calculation don't have a genre. It's blank.

    I am attempting to run the BPM analysis again for those 17,200 tracks. It appears to be processing them (and assigning a value), but the process is VERY slow. beaTunes is estimating that it will take about 6 days to do the 17,200 tracks. Most of the time it doesn't appear to be doing anything (although it is using a lot of CPU), but every now and then a progress bar will flash up next to the top track in the list, it will do its thing, and then go away. The process repeats: wait quite awhile, process a few, and then wait some more on the next tracks. I'm assuming the problem isn't with the BPM analysis itself but rather dealing with the very large library db.

    I'll be honest. I love beaTunes and what it tries to do. But I think I've seen that it is almost completely unusable in its present state for a library of this size. At least my particular installation is. If there is something different I can do, I'd love to hear it. But an 11 GB library database, a 4 hour synchronization to pick up about 300 new, moved or updated tracks, extremely slow response simply scrolling around in the library or task queue... I can't deal with it in a live situation.

    Don't get me wrong: I'm happy I got the BPM values, and I've found some of the inspection features helpful, but I can't see using beaTunes on a regular basis as it is right now. For anyone with a reasonably sized music library, I think beaTunes is really innovative and is a great app.

    (My apologies for straying so far off the original topic of this thread.)

  10. Support Staff 10 Posted by hendrik on 11 Oct, 2018 06:37 AM

    hendrik's Avatar

    For reasons unknown, beaTunes did not calculate a BPM value for about 17,200 tracks. Some weren't calculated because I excluded a few genres from the calculation. It appears that most of the rest that didn't get a BPM calculation don't have a genre. It's blank.

    How did you start analysis (Analyze All, from the menu)?
    And what analysis settings did you use? Did you use online resources for this task?

    I am attempting to run the BPM analysis again for those 17,200 tracks. It appears to be processing them (and assigning a value), but the process is VERY slow. beaTunes is estimating that it will take about 6 days to do the 17,200 tracks. Most of the time it doesn't appear to be doing anything (although it is using a lot of CPU), but every now and then a progress bar will flash up next to the top track in the list, it will do its thing, and then go away. The process repeats: wait quite awhile, process a few, and then wait some more on the next tracks. I'm assuming the problem isn't with the BPM analysis itself but rather dealing with the very large library db.

    That, or lack of memory. What does the About dialog show? Does memory consumption consistently stay above 80-90%?

    I'll be honest. I love beaTunes and what it tries to do. But I think I've seen that it is almost completely unusable in its present state for a library of this size. At least my particular installation is. If there is something different I can do, I'd love to hear it. But an 11 GB library database, a 4 hour synchronization to pick up about 300 new, moved or updated tracks, extremely slow response simply scrolling around in the library or task queue... I can't deal with it in a live situation.

    I feel you. Once a software isn't really responsive anymore, it sucks to use it.

  11. 11 Posted by Richard on 11 Oct, 2018 06:54 AM

    Richard's Avatar

    Hendrik, thanks for the replies, as always.

    It's still chugging on the 17,200 files. It's down to 15294 atm. Memory usage in "About" seems to be cycling from a low of 16% to a high of 48% right now.

    My analysis that I started last week I did as Analyze All from the menu, as I recall. I hoped it would be smart enough not to analyze what it had already analyzed before. I selected Fingerprint and Send Fingerprint to Server. On BPM, I selected Do Not Overwrite Existing Value and I did Not use online resources. Based on my reading of comments elsewhere on your site, that seemed to be the better option.

    Now in analyzing the 17,200 that it didn't write a value for before, I didn't know how to do it other than sort the entire library by BPM, highlight those with no BPM (now at the top of the library display due to the sort), and Analyze Selected. So that's what I did. My BPM options were the same as before. I unchecked the Fingerprint item, figuring that had already been done the first time through. In both analysis runs, there are a few genres I excluded: Classical, Talk, Advertisement, and a few others that I knew would be a waste of time to analyze.

  12. Support Staff 12 Posted by hendrik on 11 Oct, 2018 07:03 AM

    hendrik's Avatar

    It's still chugging on the 17,200 files. It's down to 15294 atm. Memory usage in "About" seems to be cycling from a low of 16% to a high of 48% right now.

    So it's not the memory.

    Have you restarted beaTunes between the two runs? When you shut down beaTunes, it attempts to reduce the database file size, which may lead to better performance (not much though).

    Is the database on a fast disk? Like an SSD? A slow disk may be the bottleneck.

    My analysis that I started last week I did as Analyze All from the menu, as I recall. I hoped it would be smart enough not to analyze what it had already analyzed before.

    It is.

    On BPM, I selected Do Not Overwrite Existing Value and I did Not use online resources. Based on my reading of comments elsewhere on your site, that seemed to be the better option.

    It's more deterministic and you don't have to wait for a server to respond. On the downside, you have to estimate all files yourself, which may take a long time, if the files are stored on a slow disk or a disk connected via a slow interface, like USB 2.

    Now in analyzing the 17,200 that it didn't write a value for before, I didn't know how to do it other than sort the entire library by BPM, highlight those with no BPM (now at the top of the library display due to the sort), and Analyze Selected. So that's what I did.

    That's the correct way to do it.

    In both analysis runs, there are a few genres I excluded: Classical, Talk, Advertisement, and a few others that I knew would be a waste of time to analyze.

    You can only exclude genres for Analyze All. When you specifically pick files yourself, beaTunes assumes you know what you are doing and does not exclude anything.

  13. 13 Posted by Richard on 11 Oct, 2018 07:10 AM

    Richard's Avatar

    I did restart between the two runs. Several times as a matter of fact, and I took the 5.1.10 upgrade also between the two runs.

    The database is on my SSD C: drive. The music files are on an HDD spinning disk on the SATA bus in the PC chassis. No USB involved.

    OK on the excluded genres. That makes sense. I just put that info in for completeness.

  14. Support Staff 14 Posted by hendrik on 11 Oct, 2018 07:14 AM

    hendrik's Avatar

    OK. That's not a bad setup.

    I remark about the progress bars in the analysis panel: They are just there to give you an idea of what's going on. They are not very accurate.

    What's accurate though, is the logs. Would you mind uploading yours (via Help->Upload Logs, from within the app)? I'd love to see what's going on.

    Thanks!

  15. 15 Posted by Richard on 11 Oct, 2018 07:15 AM

    Richard's Avatar

    Done! And now I'm off to bed. 2:15AM here.

  16. Support Staff 16 Posted by hendrik on 11 Oct, 2018 07:24 AM

    hendrik's Avatar

    Thanks! Have a good night.

  17. Support Staff 17 Posted by hendrik on 11 Oct, 2018 05:37 PM

    hendrik's Avatar

    Hey Richard,

    thanks for your logs. They were quite enlightening to me.

    I have spent the day coding, trying to address most issues I was able to identify from your logs.

    Most of the time it doesn't appear to be doing anything (although it is using a lot of CPU), but every now and then a progress bar will flash up next to the top track in the list, it will do its thing, and then go away.

    It seems that you sorted by BPM, selected the tracks with no BPM and started the analysis for the selected songs. beaTunes automatically switched over to the analysis view and started analyzing. But it still had the sorted table view in the background. And because you kept on calculating new BPM values via the analysis, beaTunes constantly reloaded the complete table view in the background, to keep it properly sorted by BPM. So in essence, beaTunes was busy updating a table view you couldn't even see. The quick workaround to address this, is to actively select another, preferably empty and unsorted playlist. But obviously that's not very user-friendly. Therefore I've added some logic that prevents beaTunes from updating the table view while it's hidden. This should speed up your analysis significantly.

    The synchronization took about 4 hours.

    I have reviewed the synchronization code for folder-based libraries and found two flaws. beaTunes 5 has some logic that keeps track of inodes (an inode is basically a number for a file instead of a name). This allows keeping track of a file even when it's name has changed. Due to a bug, the inode information wasn't properly stored in the database. This is annoying, but does not cause 4h of synchronization time. What was worse, is that beaTunes checked for the existence of a file in its database individually, i.e. one database access for each file. I assume that's what beaTunes spent most of its time on during synchronization. I have rewritten the code, to load the relevant data from the database in one big chunk and then only compare each file with what's in memory. The tradeoff here is memory for speed. A quick back-of-the-envelope calculation tells me that it's worth it, even with 300k tracks. During testing I have measured a speed-up of x33, but this was with only 5,700 tracks. Note, that you may not see a speed-up at all during the first synchronization, because beaTunes will attempt to add those missing inodes to the database.

    Additionally I have stumbled over a couple of other little things that have to do with selecting tracks in a very large playlist and made some changes to speed things up.

    If you'd like to try all this out, please shut down beaTunes and install the latest snapshot from https://www.beatunes.com/download/beaTunes-5-1-11-SNAPSHOT-x64.exe

    Thanks—your feedback is very welcome!

    -hendrik

  18. 18 Posted by Richard on 11 Oct, 2018 05:45 PM

    Richard's Avatar

    Thank you very much Hendrik. I will download and install the snapshot.

    Your discovery of the constant resorting of the hidden playlist view was spot on. I went to the library tab, selected another sort (Artist), and that caused the BPM calculations in the task queue to go back to a reasonable (fast) pace.

  19. 19 Posted by Richard on 11 Oct, 2018 05:57 PM

    Richard's Avatar

    By the way, since the topic has shifted to optimization, I noticed this:

    When the task queue is processing, and I have the default view of seeing roughly 20 items in the pane that shows the tracks scrolling up as they are processed... If I shrink down the size of that pane so that only 3 lines of tracks were showing (by making the Messages pane bigger), the processing speeds up by a significant amount. In my earlier measurements, it was processing about 3300 tracks per hour. When I shrank down the Task Queue pane, the processing sped up to about 4400 tracks per hour.

    Now, while it is working through the latest group of 17,200 tracks (down to 12,700 atm), it again sped up visibly when I limited the display to 3 items instead of the upcoming 20.

    I thought you would be interested in that observation.

  20. 20 Posted by Richard on 11 Oct, 2018 11:27 PM

    Richard's Avatar

    5.1.11 is a huge improvement in the areas it addresses.

    My synchronization went from about 4 hours to 53 minutes. Let's call it a 75% reduction because my earlier 4 hour figure wasn't precise. It picked up 97 new tracks that I added and presumably logged a handful of tag changes I made with another app.

    I had a few more tracks to run the BPM analysis on, and it went much quicker also, even with the library window sorted on BPM but not visible. I didn't run a real good test on that because I didn't have enough unanalyzed tracks. I could have let it do some of the ones it had done before and overwritten the old values (presumably they would have been the same), but I didn't.

    Hendrik, thank you for the changes.

  21. Support Staff 21 Posted by hendrik on 12 Oct, 2018 07:26 AM

    hendrik's Avatar

    If I shrink down the size of that pane so that only 3 lines of tracks were showing (by making the Messages pane bigger), the processing speeds up by a significant amount.

    Thanks for sharing. I will look into this.

    My synchronization went from about 4 hours to 53 minutes. Let's call it a 75% reduction because my earlier 4 hour figure wasn't precise.

    Did you synchronize once or twice? Did the second time go even faster?

  22. 22 Posted by Richard on 12 Oct, 2018 01:13 PM

    Richard's Avatar

    I only did it once. I can do it again and let you know.

    I also see that the db file shrunk by a little bit, from 11,160,856 KB to 11,154,824 KB. The larger value is prior to upgrading to 5.1.11 "snapshot" and running synchronization once (as reported), with 97 new tracks. That may just be normal fluctuation or maybe it is due to the new version, idk. These sizes are with beaTunes not open.

  23. 23 Posted by Richard on 12 Oct, 2018 04:06 PM

    Richard's Avatar

    Much faster the 2nd time. I closed & reopened beaTunes, sorted the library on Artist (the BPM sort was still in effect from earlier), and synchronized. This time: 12 minutes, finding 10 new tracks, and removing a handful of others, reflecting my activity since the previous synchronization. (As you know, it doesn't tell you how many tracks were removed. I just know they aren't there anymore.)

  24. Support Staff 24 Posted by hendrik on 15 Oct, 2018 04:13 PM

    hendrik's Avatar

    Thanks for reporting the times! Very glad this is much faster. I'm in the process of making this a proper release.

    As you know, it doesn't tell you how many tracks were removed. I just know they aren't there anymore.

    Actually, it does:

    To see synchronization results, turn on the "Status Bar" (under "View"). Right after the synchronization results are shown for a few seconds.

Reply to this discussion

Internal reply

Formatting help / Preview (switch to plain text) No formatting (switch to Markdown)

Attaching KB article:

»

Attached Files

You can attach files up to 10MB

If you don't have an account yet, we need to confirm you're human and not a machine trying to post spam.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac