Request for smart sync with iTunes

phil's Avatar

phil

05 Apr, 2016 11:52 PM

For reasons I can't explain, iTunes regularly reverts the last played time and play count for a large chunk of my library back a few years. I had originally thought this was due to iTunes Match so I disabled it, but it happened again last week after I had to deinstall and reinstall iTunes.

As I have a number of playlists that are dependent on last played, and play count this is really annoying to me.

Since beaTunes keeps a copy of the metadata from the iTunes library, and syncs it from the library, I'd like the option for beaTunes to protect me from iTunes madness. Specifically what I'd like is for beaTunes to notice during a sync that the last played time has gone backwards, or that the play count has gone backwards, and then fix the iTunes library with the correct metadata from beaTunes. There may be other read-only fields in the same category but these are the two that affect me.

  1. Support Staff 1 Posted by hendrik on 06 Apr, 2016 11:46 AM

    hendrik's Avatar

    Hey Phil,

    I may actually have something for you before the end of the week.

    -hendrik

  2. 2 Posted by phil on 07 Apr, 2016 01:44 AM

    phil's Avatar

    Thanks. Really appreciate it.

  3. Support Staff 3 Posted by hendrik on 09 Apr, 2016 02:12 PM

    hendrik's Avatar

    Hey Phil,

    Sounds like your problem is somehow related to https://discussions.apple.com/thread/3495653: "iTunes Match vs. Play Counts"

    The latest beaTunes snapshot from https://www.beatunes.com/download/beaTunes-4-6-0-SNAPSHOT-x64.exe contains some logic that prevents beaTunes from importing backwards play/skip dates and counts. So if the iTunes XML file contains dates/counts that are lower than the ones already in beaTunes, they are ignored when synchronizing.

    Now, as you pointed out, that's not enough to restore the values.

    Please find attached a beaTlet that (once installed) offers a new option in the Tools menu. If you select this Reset dates & counts item, beaTunes will compare play dates from the internal database with what iTunes claims. If there is a discrepancy, beaTunes will attempt to write its values to iTunes.

    I have tested this beaTlet to a degree and in the process had to work on a few things in beaTunes itself, as some of the used functionality has never used by beaTunes (code for: it was broken). Before you ruin all dates/counts in your library, please make sure you test the beaTlet. You can do so, by commenting out the critical parts, by inserting // at the beginning of the lines in question (those that start with iTunesSong.set). When run in such a way, the beaTlet will still log to the log files (more or less) what it would do without actually doing it. If possible, run the complete script on a backed up library first, so that you can restore things. Please understand that I cannot take responsibility for any data loss.

    Let me know, if you encounter any issues.

    Cheers,

    -hendrik

  4. 4 Posted by phil on 10 Apr, 2016 04:57 AM

    phil's Avatar

    Hi Hendik,

     

    It looks like I must have run beaTunes and synchronised sometime after the time the library got reset because while it did find a few mismatches, they were all by 1 second. It’s probably worth ignoring 1 second changes as it’s just a rounding error.

     

    There is one problem with the beatlet in that it analyses every item in the iTunes library – including all the apps – and the log is throwing lots of errors when it tries to process them.

     

    2016-04-10 14:14:11,872 [ 94140] [WARN ][TunesUtilities] com.tagtraum.beatunes.library.itunes.ITunesAudioSong: Track with id -9096491182104802639 not found in iTunes library.

    2016-04-10 14:14:11,872 [ 94140] [WARN ][TunesUtilities] ResetDatesAndCounts.groovy: Track with id -9096491182104802639 not found in iTunes: ITunesAudioSong{song=Song@-9096491182104802639[name='Hitman GO',album='null',artist='null',genre='Games']}

     

    Other than that, it seems to work correctly.

     

    I’ll hang on to it, and use it the next time iTunes randomly changes my library. (Grumble, grumble about Apple software quality…).

     

     

    Thanks,
    Phil.

  5. Support Staff 5 Posted by hendrik on 10 Apr, 2016 05:58 AM

    hendrik's Avatar

    Hey Phil,

    It looks like I must have run beaTunes and synchronised sometime after the time the library got reset because while it did find a few mismatches, they were all by 1 second. It’s probably worth ignoring 1 second changes as it’s just a rounding error.

    I've adjusted the script to ignore differences of 2s and less.

    There is one problem with the beatlet in that it analyses every item in the iTunes library – including all the apps – and the log is throwing lots of errors when it tries to process them.

    Those are just warnings that can obviously be ignored. I've also changed the script to get rid of the second warning (for the same file).

    I’ll hang on to it, and use it the next time iTunes randomly changes my library. (Grumble, grumble about Apple software quality…).

    You are not alone with the feeling that Apple software quality is going downhill: http://sudophilosophical.com/2016/02/04/apples-declining-software-q...

    Cheers,

    -hendrik

  6. 6 Posted by phil on 07 May, 2016 02:02 AM

    phil's Avatar

    It’s happened again. I just reenabled iTunes Match on my library and iTunes blasted all the metadata again. Last played is missing, BPM is gone, ratings all messed up.

     

    No problems I though I’ll just run beaTunes and it will ignore the messed up metadata and then be able to restore it. Except that it didn’t ignore the messed up metadata, and synchronised all of it from my iTunes library. So both beaTunes and iTunes are in a bad state.

     

    The code you have for ignoring metadata that goes backwards (play count, last played, etc) and metatdata that is missing (BPM, etc), isn’t working.

     

     

    Phil.

  7. 7 Posted by phil on 07 May, 2016 03:26 AM

    phil's Avatar

    I’ve attached the XML library before I turned on iTunes Match, and again afterwards. Every single change in the after copy is Apple messing up my library. If possible I’d like beaTunes to protect me from Apple!

  8. Support Staff 8 Posted by hendrik on 08 May, 2016 10:54 AM

    hendrik's Avatar

    Hey Phil,

    that sucks. Can you please send me your beaTunes logs?

    Thanks,

    -hendrik

  9. 9 Posted by phil on 08 May, 2016 11:27 PM

    phil's Avatar

    Logs attached from after the “event”.

     

    I’ve restored both my iTunes library and beaTunes data back from before I turned on iTunes Match, so I haven’t permanently lost anything (yay for backups!).

     

    But you can appreciate that I’m extremely hesitant to turn iTunes Match back on without protection! :)

     

     

    Phil.

  10. Support Staff 10 Posted by hendrik on 09 May, 2016 07:11 PM

    hendrik's Avatar

    Hey Phil,

    I took a look at your logs and also the before/after XML files. I have to say, I'm at a loss. beaTunes 4.6 has code in it that should prevent it from reading values that are less than the current values for Play/Skip Count and Play Date. And when it encounters such a value, it should log this. The beaTunes logs you sent me, contain only 3 such messages and they are for movies with no play count whatsoever. So I cannot see that beaTunes actually imported the wrong values.

    But you wrote:

    Except that it didn’t ignore the messed up metadata, and synchronised all of it from my iTunes library. So both beaTunes and iTunes are in a bad state.

    I cannot find that in the logs. Did it really import them? I.e. did you synchronize beaTunes after you turned iTunes Match on? Or is it possible that you synchronized, then restored a backup (of the logs as well) and therefore I cannot see those messages in the logs?

    Thanks,

    -hendrik

  11. 11 Posted by phil on 09 May, 2016 08:52 PM

    phil's Avatar

    Hi Hendrik,

     

    Those of definitely the logs from beaTunes after the corruption.

     

    I turned Match on, and it messed everything up. At that point I thought “beaTunes can fix this!”, so I ran beaTunes and it definitely synchronised the bad data. Then I ran an inspection and it found a huge number of problems (where previously there were none) which I fixed.

     

    After that I realised there was no way back except restoring from backup.

     

    So I renamed the iTunes and beaTunes folders to “.busted” and restored from backup.

     

    What I sent you came from the “.busted” folders.

     

     

    Phil.

  12. 12 Posted by phil on 09 May, 2016 09:57 PM

    phil's Avatar

    It occurs to me that I have beaTunes set to automatically sync with iTunes. I never have to do it manually.

    Perhaps there is s different code path in this instance.

    --
    Sent from my iPhone

  13. Support Staff 13 Posted by hendrik on 10 May, 2016 09:43 AM

    hendrik's Avatar

    No, unfortunately that's still the same code path.

  14. 14 Posted by phil on 11 May, 2016 03:31 AM

    phil's Avatar

    If you can give me a build with more logging I'll do it again. I can always revert to a backup afterwards to fix it up.

    --
    Sent from my iPhone

  15. Support Staff 15 Posted by hendrik on 11 May, 2016 10:52 AM

    hendrik's Avatar

    Hey Phil,

    if you're willing to spend some time experimenting, I'm more than happy to provide special builds.
    Please download and install this dev snapshot: https://www.beatunes.com/download/beaTunes-4-6-2-SNAPSHOT-x64.exe It has some additional log statements in it.

    Assuming, iTunes Match is off and the library is in the proper state, please do the following:

    1. Start the snapshot and open the preferences.
    2. Turn off synchronization on Window activation (this is just to have a little more control)
    3. Go to File -> Synchronize... to start synchronization manually.
    4. In iTunes, turn on iTunes Match.
    5. Find a good example, i.e. a song, that now has a lower play count than before. Take note of title, album, artist, correct play count, and iTunes Match play count. I understand this may be tricky, as it's hard to know what has changed. "Beach Baby" by "First Class" may be a good candidate.
    6. In beaTunes, go to File -> Synchronize... (if for some reason, beatunes claims that nothing has changed, make a small change in iTunes, like creating a new playlist, then wait a couple of seconds to give iTunes a chance to export the XML file).
    7. Look for the example song in beaTunes. Verify that the play count is now correct/wrong in beaTunes.
    8. Shut down beaTunes.
    9. Attach the logs to this discussion—please don't forget the data for the sample song.

    It helps me a lot, if you don't do anything else with beaTunes as that will reduce the amount of garbage in the logs.

    Hopefully, we'll figure out what fails here.

    Thank you,

    -hendrik

  16. 16 Posted by phil on 15 May, 2016 02:03 AM

    phil's Avatar

    Hi Hendrik,

     

    There are a number of things to report, and I’m going to keep them in separate messages for clarity.

     

    First one is about synchronization.

     

    This is before doing anything with iTunes Match.

     

    In iTunes I created a smart playlist called BPM0, with the following settings:

    Media Kind is Music; BPM is 0

     

    There were 55 items (including a new album I’ve recently bought). Most were “checked” but 7 were “unchecked” as they are duplicates.

     

    I turned off Synchronization on Window Activation in Preferences / General. At this stage I can see the same items in the playlist in both iTunes / beaTunes.

     

    Then I selected all of the items in the playlist, right-click Analyse. This is where things didn’t work quite as expected.

     

    As the analysis was running the changes that were made (updating BPM, setting key, etc) were pushed immediately to iTunes on a track-by-track basis. I was watching the iTunes playlist shrink as the analysis was happening. As I’ve turned off automatic sync, this surprised me. I thought that I would have to manually start a sync.

     

    At the end of the analysis, the beaTunes window still shows all 55 tracks. The iTunes window shows 5 tracks (which are 5 of the 7 “unchecked” tracks, no idea why only 5 of them). I can confirm that 5 tracks still shown in iTunes do not have the BPM set, even though it was set to overwrite in beaTunes preferences.

     

    After doing a manual sync both apps show 5 items in the playlist.

     

    This is the analysis options panel:

     

     

    Running a second analysis of those 5 tracks again (with the settings above) does not push the changes to iTunes.

     

    Just in case you were wondering, I do not exclude unchecked tracks.

     

     

     

    Phil.

  17. Support Staff 17 Posted by hendrik on 15 May, 2016 09:59 AM

    hendrik's Avatar

    Hey Phil,

    except for those five track, this behavior is expected.

    To clarify: "Synchronization" is the process of beaTunes synchronizing with the state iTunes is in. I know, this could be communicated more clearly by the program.

    When using an iTunes-based library (and you do), any changes in beaTunes should automatically be propagated to iTunes. This is different from a folder-based library.

    There must be something about the 5 tracks that prevents proper analysis. Does the beaTunes Analysis view (vertical button on the far left window side) show any error messages, when you analyze those 5 tracks again? Perhaps DRM protection or missing files.

    Regarding checked and unchecked songs: See the explanation in the analysis preference panel? It says: "When analyzing all or newly added songs..." This refers to the menu items under Tools. When analyzing manually selected songs via the context menu's Analyze button, those settings are overruled, as the program assumes you know exactly what you are doing.

    Hope this clears things up a little.

    -hendrik

  18. 18 Posted by Ker Bob on 16 Nov, 2017 12:37 AM

    Ker Bob's Avatar

    One thought on synchronizing.

    Could you show:

    beaTunes->iTunes sync x%
    beaTunes<-iTunes sync x%
    or
    iTunes->beaTunes Sync x% left?

  19. Support Staff 19 Posted by hendrik on 16 Nov, 2017 08:25 AM

    hendrik's Avatar

    Hey Ker,

    When you synchronize manually, i.e. via the File menu, you should see a progress bar. Is that what you have in mind?

    Cheers,

    -hendrik

  20. 20 Posted by Ker Bob on 16 Nov, 2017 09:31 AM

    Ker Bob's Avatar

    Right.
    When it shows synchronization in the status bar, it would actually show
    which direction was syncing though.
    Basically what you are doing with the inspect feature showing how much is
    left.

    Part of the reason for this, is with the larger library I have been in the
    middle of having to shut down my laptop and was afraid to due to sync
    taking place. Especially when it would loop a few times on the progress
    bar.

  21. Support Staff 21 Posted by hendrik on 16 Nov, 2017 09:37 AM

    hendrik's Avatar

    When it shows synchronization in the status bar, it would actually show which direction was syncing though.

    The direction is always iTunes -> beaTunes.

    Basically what you are doing with the inspect feature showing how much is left.

    If beaTunes syncs without you having initiated it, you have "Sync on window activation" turned on. For a large collection like yours, that should be off. If, for some reason, it turns itself on again, without you doing anything, please let me know.

Reply to this discussion

Internal reply

Formatting help / Preview (switch to plain text) No formatting (switch to Markdown)

Attaching KB article:

»

Attached Files

You can attach files up to 10MB

If you don't have an account yet, we need to confirm you're human and not a machine trying to post spam.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac