Genre autobatch mode

info's Avatar

info

26 Apr, 2018 01:53 PM

Hi Hendrik!

I came accross this neat script that will try to defeat the random Genres that are filled in:
https://github.com/YetAnotherNerd/whatlastgenre

Good stuff:

  • It's based on several resources

  • It's based on a scoring system

  • Optional filters to get better individual results

Downside:

  • It's based around the Album, and demands your folder/file-structure to be so.

  • Since I dont use my collection this way, It doesnt work on a folder like 'Music Downloads', with several bought tracks. In the end I want to have my d/led material to have a Genre tagged thats closer to truth than the default tag, and then move those into my music-collection (which is based on Genre subfolders, like 2-step/Garage, Breakbeat, House > Deep, Funky, Tech, Disco. Drum & Bass, Techno, etc.

Conclusion:

  • The problem with Genres is that 9/10 times the tag is written with some vague indicator like 'Electronic' or just 'House'.
  • To manually review all the d/led material and tag them using beaTunes how it is now (selecting a suggestion/alternative from a dropdown-menu is time-robbing, like shown here: http://help.beatunes.com/discussions/questions/7992-genre-tagging )

How about implementing such autobatch-mode, keeping a score-system in check like Whatlastgenre does, to speed it up based on big data?
Ultimately this way, it would fill in just 1 Genre (that's most likely the right one) based on the several resources + score-system to come to an agreement.
If the user doesnt agree with the result afterwards, they can simply use the excisting Dropdown-menu on the Genre afterwards to redefine.

See Output example:
https://gist.github.com/YetAnotherNerd/3e59c4d02f1930d9e52e

  1. 1 Posted by info on 30 Apr, 2018 05:26 PM

    info's Avatar

    In addition, I think the MultimediaEval / Acoustibrainz Genre Task 2017 tried to do the same right?

    ...Even incl. your program and data =)

    What were the outcomes?
    Is it something u want to implement in beaTunes using the development script?

    Source:
    https://github.com/multimediaeval/2017-AcousticBrainz-Genre-Task
    https://multimediaeval.github.io/2017-AcousticBrainz-Genre-Task/
    http://ceur-ws.org/Vol-1984/Mediaeval_2017_paper_6.pdf

  2. Support Staff 2 Posted by hendrik on 01 May, 2018 07:06 AM

    hendrik's Avatar

    Hey there,

    when you let beaTunes tag you songs with the Import Metadata analysis task, it uses the central beaTunes database, which consists of many many user data submissions. So for each song, multiple user submissions are taken into account. What's presented to you is the dominant tempo.

    Are the result not satisfying? If not, in what way are they insufficient?

    The MultimediaEval task was not about finding agreement between different sources, but about automatic genre classification based on fixed features. A paper that is more in line with what you are asking is http://www.tagtraum.com/download/schreiber_msdgenre_ismir2015.pdf

    Cheers,

    -hendrik

  3. 3 Posted by info on 01 May, 2018 10:52 PM

    info's Avatar

    Aha!
    So beaTunes already does some sort of like Whatlastgenre does =)

    Though, I do indeed get mixed results that are puzzling me.
    Here's a 10+ pack of tracks that are in my eyes tagged wrong - genre-wise:
    https://mega.nz/#!9NBSyRYI!ItwaQKoVhB2bSoBt93T07vlTmRiZkyZlxUYUOsL_ylg

    Nomatter if I use/or not use the Force lookup via Fingerprint, I do get different results, though, both are far from what they should be.

    If u look them tunes up under Style of Discogs.com, they come close to the true genre.
    I'd be happy with the Style entered @ Discogs.com to be placed in the Genre-tag of my collection,
    since they are pretty accurate!

    beaTunes genre results:

    • 2 Brothers on the 4th Floor - Come Take My Hand (Extended Version) - Blues(?)
    • Dúné - Hardcore Vibes (South Bound Mix) - Electronica(?) - duh, but defo Hardcore or Happy Hardcore imho
    • Hi-Lo - Renegade Mastah (Original Mix) - Deep House(?)
    • Ice MC - It's A Rainy Day (Euro Club Mix) - Rock(?)
    • Inferno Bros - Slaves To the Rave (Stunned Guys Remix) - Breakbeat(?)
    • Kate Ryan - Only If I (Peter Luts Remix) - Dancehall(?)
    • Kym Sims - Too Blind To See It (Hurley's House Mix) - Electronica/Dance(?) - sure danceable, but true House as the tag also reveals
    • Noisecontrollers - Pillars Of Creation (Edit) - Techno(?)
    • The Chainsmokers - Closer (R3hab Remix) (feat. Halsey) - Rock - Pop - AC(?)
    • The Riders - Bang (Radio Edit) - Blues(?)
    • The Tribe Of Good - Loving You Baby (Weiss Remix) - No genre(?)

    compared to Discogs genre/style results:

    • 2 Brothers on the 4th Floor - Come Take My Hand (Extended Version) - Style:Euro House, Happy Hardcore, Jungle
    • Dúné - Hardcore Vibes (South Bound Mix) - Style: Hard Trance, Happy Hardcore
    • Hi-Lo - Renegade Mastah (Original Mix) - Style: House
    • Ice MC - It's A Rainy Day (Euro Club Mix) - Style: Euro House
    • Inferno Bros - Slaves To the Rave (Stunned Guys Remix) - Style: Hardcore, Gabber
    • Kate Ryan - Only If I (Peter Luts Remix) - Style: Trance
    • Kym Sims - Too Blind To See It (Hurley's House Mix) - Style: Deep House, Garage House, Downtempo, Breakbeat, House
    • Noisecontrollers - Pillars Of Creation (Edit) - Style: Hardstyle
    • The Chainsmokers - Closer (R3hab Remix) (feat. Halsey) - Style: Dance-pop
    • The Riders - Bang (Radio Edit) - Style: Italodance, Progressive Trance, Euro House
    • The Tribe Of Good - Loving You Baby (Weiss Remix) - Style: House
  4. 4 Posted by info on 08 May, 2018 09:16 AM

    info's Avatar

    @hendrik

    What's your take on this?

    • Don't u agree those tags by beaTunes are atleast very random compared to Discogs?
    • What is your take on looking up the MASTER release (if excists in db) for Styles?
    • Perhaps a good idea to implement?
  5. Support Staff 5 Posted by hendrik on 08 May, 2018 04:40 PM

    hendrik's Avatar

    Perhaps a good idea to implement?

    It's trivial to implement as a plugin. See https://github.com/beatunes/plugin-samples/tree/master/discogsgenre

    You can install it using the plugin pane in the beaTunes preferences (you may have to refresh available plugins).

    That said, perhaps I should take a closer look at Discogs genres for a future beaTunes update.

  6. 6 Posted by info on 08 May, 2018 05:00 PM

    info's Avatar

    Sweet!
    I'll try it out tonite and do some testings.

  7. Support Staff 7 Posted by hendrik on 09 May, 2018 08:43 AM

    hendrik's Avatar

    If you're into coding at all, please also take a look at the code!!

    -> https://github.com/beatunes/plugin-samples/tree/master/discogsgenre

  8. 8 Posted by info on 09 May, 2018 12:23 PM

    info's Avatar

    Not that much into coding I'm afraid, but check my result (based on the package I've sent u in this topic):
    https://youtu.be/HT0JlyhBkec

    Am I doing something wrong?
    Electronic is tagged on 1 track. That looks like a Genre tag.
    Though, where the neaty tags are is in the Style tag of Discogs.
    Which ofcoz, resembles a Genre, but just a level more detailed:
    https://gyazo.com/af80c81c52c8771aad89bcd6b874b263
    Those I want!

    If we look up that 1 track thats been tagged here, thru beaTunes, it ends up here:
    https://www.discogs.com/release/9415758

    So I think its just grabbing the Genre, instead of Style here.

    When I look up that track manually and click the Master release of it, I end up here:
    https://www.discogs.com/Noisecontrollers-Enc%C2%B2-The-Science-Of-H...
    With same description Genre/Style wise.

  9. 9 Posted by info on 09 May, 2018 12:58 PM

    info's Avatar

    Another tune, that's interesting, is the Love You Baby (Weiss Remix), see:
    https://gyazo.com/f6fe091fc351294abe6947018cb3b641

    When right-clicking a tune in beaTunes > Open in... Discogs it goes to the Album that's tagged in the file.
    Since this particular tune doesnt have one, it just searches based on Artist - Title, I assume (which is great!).

    Then we end up here:
    https://www.discogs.com/search/?type=release&artist=The+Tribe+O...

    If we click the 1st (and only) result, we have this overview:
    https://gyazo.com/cf05a3abe4dc6001f72c3ddd1227a56c

    Tricky part here is that there is no YEAR , but RELEASED instead.
    Once again, GENRE from Discogs is so wide (and not useful); Electronic here.
    Though, STYLE is right on spot: House.


    My idea to approach this code-wise (sorta lol):

    Search on Artist + Title tag of track @ Discogs

    Search up MASTER release;

    if excist, then
    grab YEAR + STYLE
    if > else;
    grab RELEASED + STYLE

    if MASTER release not excist, then
    grab YEAR + STYLE from ALBUM
    if YEAR not excist;
    grab RELEASED

    if ALBUM not excist, then
    just grab from whatever the 1st result is (like with the Loving You Baby (Weis Remix)
    either YEAR or RELEASED + STYLE wise

    As u can see I'm no coder, but I hope u get the idea =)

  10. Support Staff 10 Posted by hendrik on 09 May, 2018 05:07 PM

    hendrik's Avatar

    Discogs uses genre as exactly that and style as "sub-genre". For many applications genre is super useful and sufficient. Style is more meaningful when you are really into some genre. E.g. if all your music is Electronic anyway, the genre tag Electronic is not useful at all, only style has meaning for you. If you are generally into all kinds of music, Electronic is often sufficient when trying to distinguish from Pop, Rock, Swing, or Calypso.

    In essence, there are different use cases and needs.

    One could easily modify the plugin I wrote to prefer style over genre by replacing

                if (firstMatch.getGenre() != null && !firstMatch.getGenre().trim().isEmpty()) {
                    final String discogsGenre = firstMatch.getGenre().trim();
                    if (LOG.isDebugEnabled()) LOG.debug("Discogs genre for " + song + ": " + discogsGenre);
                    song.setGenre(discogsGenre);
                } else {
                    if (LOG.isDebugEnabled()) LOG.debug("Found no discogs genre for " + song);
                }
    

    with

                final DiscogsTrack discogsTrack = (DiscogsTrack) firstMatch;
                final List<String> styles = (List<String>)discogsTrack.getRelease().getDocument().get("styles");
                if (styles != null && !styles.isEmpty()) {
                    final String style = styles.get(0).trim();
                    if (LOG.isDebugEnabled()) LOG.debug("Discogs style for " + song + ": " + style);
                    song.setGenre(style);
                } else {
                    if (LOG.isDebugEnabled()) LOG.debug("Found no discogs style for " + song);
                }
    

    in the file DiscogsGenre.java and add the import statement

    import com.tagtraum.ubermusic.discogs.DiscogsTrack;
    

    to the top of the file.

  11. Support Staff 11 Posted by hendrik on 09 May, 2018 05:10 PM

    hendrik's Avatar

    Attached is a compiled version.
    To install, unzip the file and place the resulting jar file manually into your beaTunes plugin folder.

    See http://help.beatunes.com/kb/plugins/how-to-install-a-plugin for manual plugin installation.

  12. 12 Posted by info on 09 May, 2018 08:18 PM

    info's Avatar

    Hooray!
    Now it inserts the STYLE from DIscogs into that 1 track (and 1 track only) that first had ''Electronic'' as GENRE.
    Now showing ''Hardstyle''.

    Why doesnt it find the other tracks STYLE nor GENRE though?

  13. Support Staff 13 Posted by hendrik on 10 May, 2018 11:21 AM

    hendrik's Avatar

    Why doesnt it find the other tracks STYLE nor GENRE though?

    Perhaps it fails to look up the Discogs ID. Or there is no suitable master, ... who knows.
    I could take a look at the logs, if you ran the analysis on your 10 songs again and right afterwards uploaded them via Help -> Upload Logs.

  14. 14 Posted by info on 10 May, 2018 03:47 PM

    info's Avatar

    Just uploaded the LOG at 17:43/44.
    In this batch I didnt select any in the Analyze options;
    except ofcourse the Import Discogs Genre.

    I'm now going to do another batch and sent log at 17:46/47.
    This batch, same method, different tracks, that all get tagged.

  15. Support Staff 15 Posted by hendrik on 11 May, 2018 10:05 AM

    hendrik's Avatar

    I checked the logs and the problem appears to be the Discogs rate limit.
    Discogs allows only ~60 request per minute. beaTunes does attempt to throttle request, but it does not do so very well. This needs to be improved in beaTunes itself.

  16. Support Staff 16 Posted by hendrik on 11 May, 2018 10:11 AM

    hendrik's Avatar

    If you want to verify that results do improve for your ten test songs, wait a minute, then open the Get Info dialog for each one of them. For each one, wait until beaTunes has fetched data from the databases. This will cache Discogs responses.

    Then run the analysis again.

  17. 17 Posted by info on 11 May, 2018 11:09 AM

    info's Avatar

    Thanks.
    I did (I think) what u suggested, see screencapture here:
    https://youtu.be/2hLMN5kJ44M

    I've waited for each track, till it showed the Album Art (which is the indicator it fetched the data from Discogs, I assume)
    Though, not getting any better result.

    Upped the LOG at 13:09 / 11-5-2018 again.

  18. Support Staff 18 Posted by hendrik on 11 May, 2018 11:12 AM

    hendrik's Avatar

    I've waited for each track, till it showed the Album Art (which is the indicator it fetched the data from Discogs, I assume)

    No, it's not.

    Turn on the status bar (under the menu `View'). While beaTunes is still looking up data, it displays "Looking up reference data..." in the status bar.

  19. 19 Posted by info on 11 May, 2018 11:22 AM

    info's Avatar

    Log sent again at 13:16 / 11-5-2018.
    This time, with Status Bar set to On (see 2m42s)
    Re-did the same method:
    https://youtu.be/V8s0dbVpWPs

    Though not seeing Looking up reference data in a status bar?

  20. 20 Posted by info on 11 May, 2018 11:25 AM

    info's Avatar

    I see what the problem is status bar wise;
    I had to make the window screen/gui bigger size.
    Nevertheless, I'll create a new test video in a bit INCL. status bar.

  21. Support Staff 21 Posted by hendrik on 11 May, 2018 11:34 AM

    hendrik's Avatar

    However, looking at your logs, that's not the issue here. beaTunes does the lookup, but cannot find the corresponding Discogs release for some of your tracks. To avoid problematic correspondences, beaTunes makes an effort to check a couple of parameters, like track count and approximate length. If your track data does not match Discogs well enough, beaTunes will rather say it has not found the matching song than establish a wrong connection.

    Check for example your track "Slaves To the Rave (Stunned Guys Remix)" on album "Marc Acardipane: Best of 1989-1997" by "Inferno Bros"

    The Discogs master release (https://www.discogs.com/Marc-Acardipane-Best-Of-1989-1997/master/38... )for "Marc Acardipane: Best of 1989-1997" has a track "Slaves to the Rave", but it's a different remix. Also, "Inferno Bros" is spelled with a trailing dot: "Inferno Bros."
    Unfortunately, I cannot see what tracknumber your version of the track has. On the Discogs album it's 5/14 and the track length is 6:10. What's your track number and song length?

  22. 22 Posted by info on 11 May, 2018 11:34 AM

    info's Avatar

    v3 test incl. status bar now =):
    https://youtu.be/z2iCHfWHRWI

    Though, the results are the same. just that 1 tune.
    Though, it finds reference songs for each and any (?)

  23. 23 Posted by info on 11 May, 2018 11:42 AM

    info's Avatar

    See attachment.
    I think the data is incorrect, looking at the album art and the corresponding master-release u mentioned.

    If u ask me, the album is simply a different one.

    Mind you, I have used SongKong aswell on those tracks.
    I should re-rip them, keeping the tags intact.
    Then only do a Import Discogs Genre with beaTunes.
    You agree?

  24. Support Staff 24 Posted by hendrik on 11 May, 2018 11:45 AM

    hendrik's Avatar

    If u ask me, the album is simply a different one.

    That's what it looks like to me.

    I should re-rip them, keeping the tags intact.

    Re-rip from CD? Note, that In that case the tags don't come from the CD, but from CDDB or FreeDB.

    Then only do a Import Discogs Genre with beaTunes. You agree?

    That may work, but no guarantees.

  25. 25 Posted by info on 11 May, 2018 12:12 PM

    info's Avatar

    Yep, it seems that makes a difference, see results:
    https://gyazo.com/230f97a0e6195cf31232450373108e97

    Then again, it's now 2 tracks that get tagged (of the 10).

    This plugin stands or falls with the idea in mind of having proper tags set up in the Album, right?

    Would it be possible to go for just Artist + Title look up, and fetch whatever it finds (master release if excist, if not, just use whatever Discogs-db has as a result) to have more tracks tagged Style-wise?

  26. Support Staff 26 Posted by hendrik on 11 May, 2018 02:01 PM

    hendrik's Avatar

    This plugin stands or falls with the idea in mind of having proper tags set up in the Album, right?

    Yes—the matching has to happen somehow. The album name has to at least be similar—beaTunes uses the Levenshtein distance to measure this. Additionally the track duration or the track numbers have to be similar.

    Would it be possible to go for just Artist + Title look up, and fetch whatever it finds (master release if excist, if not, just use whatever Discogs-db has as a result) to have more tracks tagged Style-wise?

    That would require rewriting some of the Discogs code used also elsewhere in the app. Because it affects more than this plugin, I think that would be unwise. Also, Discogs data is oriented around the concept of an "album". Not using it as key will lead to faulty results when dealing with other aspects like track number and release date.

Reply to this discussion

Internal reply

Formatting help / Preview (switch to plain text) No formatting (switch to Markdown)

Attaching KB article:

»

Attached Files

You can attach files up to 10MB

If you don't have an account yet, we need to confirm you're human and not a machine trying to post spam.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac