Consolidate doesn't get full library

[CaticornStudios]

When I open and click on consolidate, it asks me where to save, I created a new folder, but it only moved 3372 files out of the 10k+ files i have in my library. I've ran the consolidaation a couple times over and still havent gotten any more files to move.

Steps to reproduce

Screenshots

Environment

• tuneup PRIME version: 2.1
• Engine PRIME version: 1.6.1.5
• Operating System: Win 10

[CaticornStudios]

Not sure if this matters, but I did import my collection from Serato. all files are on C: just in different folders.

[shayded-exe]

Ah that's probably it. Tuneup currently has issues dealing with libraries imported from other DJ software. This is a bug I need to fix. I'm currently on a bit of a mental health break right now, but I hope to be back to providing updates within a month.

[CaticornStudios]

Okay, cool. Sorry its been two months for me to reply, but I was just about to use the software again and remembered the problem, and was like Hmm wonder if that ever got looked into... No need to apologize homie! Mental health is extremely important. I struggle with mental health problems as well. Take it easy, and thank you for the work you are doing! It is greatly appreciated. Much love man, stay safe!

[CaticornStudios]

Hey, I see you added this to be worked on in 2.4, however before I saw that I tried running it again in 2.3 and I am getting the error "Dest already exists" I got that the first run, so I ran a second time, and created a new directory and it moved all but one file from the one directory(the one that was created when I noticed the bug originally) to the new directory, before throwing the error.

also, just a question, when consolidating, it says all existing files will be replaced, is that only if there are duplicates or does it mean EVERYTHING located in the destination directory will be replaced by the operation?

Thanks homie, I appreciate the work you're doing!

[shayded-exe]

It means that only files with the same name will be overwritten. I can update the language to specify this more clearly.

I fixed a bug (#65) in v2.2.0 that caused an error to happen if a track that was being consolidated was already in the target folder (source and dest were the same), now it just skips those tracks and keeps them in place.

As for the error you're receiving, could you please share your database file (Engine Library/Database2/m.db) along with which song it is you're getting the error for? I need to reproduce the error so I can test and fix it.

[CaticornStudios]

it wont let me upload my m.db file even as a zip, I will send it to your email provided on your profile.

The song that is left over is Snails & Antiserum - Wild (Far Too Loud Remix) located at C:\ENGINELIB I was consolidating to the new C:/ENGINELIB_A when I got the dest already exists error

[shayded-exe]

I took a look at your database and I don't see that track in there. Is that one of the ones you imported from Serato? If that's the case, then it isn't the one causing the error as Tuneup still doesn't look at any songs except the ones imported directly into Engine. Serato songs exist in a separate database file (either sm.db or stm.db, I'm not sure which).

As for the error, it looks like you actually have a lot of tracks with the same filename in multiple places.

For example, Adventure Club - Already Know.mp3 exists twice in your library at the following paths:

• ../../../../AA/Adventure Club/Adventure Club - Already Know.mp3
• ../../../../B/unrolling/Adventure Club - Already Know.mp3

Here's a CSV file with all the songs in your library that exist in multiple places:
duplicate-tracks.csv

I'm not sure how that would've happened, but Tuneup does not create new entries for tracks, it only updates them.

I also just discovered that when consolidating with move, it actually isn't overwriting existing files, hence the error. Note that even when I fix that, you'll still have these duplicate track entries, but at least they'll be pointing to the same file.

[CaticornStudios]

That's strange. All of my library, aside from maybe 300 songs were a direct import from serato. So every song in that duplicate files list is imported from serato. In that list, I noticed that it was showing ENGINELIB as a location for a bunch of songs, however, there is only the snails song in that directory now. Did it not update the database since it ran into the error even though it moved the files?

Also, on the topic of the duplicate files, I honestly just have that many duplicate files. That's one of my big reasons for wanting this consolidate lmao. My one friend just sends me a playlist to download, so I would just batch download without cross referencing what it already had, throw it in its own directory and set up a crate for him to play with to learn how to spin a little with music he knows.

I eventually wanna start converting my library to be 100% flac and having to search through ten directories to see if I have it or not is a pain.

Okay, so here's a scenario that I would do often with serato, and don't wanna mess things up in the future with consolidating: download songs to my newMusicDir, check quality and make sure it's all good. Import that folder into a new crate/playlist in Serato to keep new music grouped together. Move contents of newMusicDir to allMusicDir, update paths in serato.

Now when I do it it should be import into engine crate/playlist then run TUP and click consolidate, keep setting as move and select the current engine library location? Or should I be selecting a new location like I tried today?

[shayded-exe]

That's strange. All of my library, aside from maybe 300 songs were a direct import from serato. So every song in that duplicate files list is imported from serato.

Interesting, I guess they made their way into your main Engine DB somehow.

In that list, I noticed that it was showing ENGINELIB as a location for a bunch of songs, however, there is only the snails song in that directory now. Did it not update the database since it ran into the error even though it moved the files?

Yeah it looks like that's what happened. In the latest release (v2.3.1) I tweaked the code so it saves the changes to the DB after every file operation instead of doing it at the end for all tracks.

What happens when you search for missing files with relocate? I would expect that all those duplicate tracks pointing to ENGINELIB would show as missing. Do they show up highlighted red in Engine? If that's the case, I'll add a button to the relocate page to let you delete the missing tracks instead of relocating them. That way you could clean up most of the duplicates. Make sure that both copies of the duplicates have the same hot cues and beat grids though. You wouldn't want to delete the one with your cues.

Now when I do it it should be import into engine crate/playlist then run TUP and click consolidate, keep setting as move and select the current engine library location? Or should I be selecting a new location like I tried today?

You should just do it to the same folder. I personally keep all my songs in a single folder to keep it simple.
Maybe I could make it so the move/copy option remembers your last selection so you don't have to change it every time.

As for your original issue in this thread, I thought about it and having Tuneup consolidate the songs in your Engine Serato library doesn't seem necessary as Engine will just read the updated paths from Serato. It sounds like your transitioning to just managing everything with Engine, correct?

I've thought about expanding the smart playlist functionality to work with Serato/Rekordbox tracks as well. I haven't had anyone request it yet though and I imagine that most people who are using one of those DJ apps doesn't import songs directly into Engine as well; they would probably be using the smart playlist features those apps already have anyways.

[CaticornStudios]

I wonder if when engine updated to v2.x when it migrated the database, if it just made it all one thing now?

So, if it updates after each file operation, why are there plenty that are still listed as being in ENGINELIB when all but one was moved to ENGINELIB_A ?

As far as the relocate showing up in red, I'll have to check on that in the morning. I left my laptop charger at work and well 🤷🏽‍♂️ stupid me. However, I think there was a slight communication error. So the songs on the duplicate list you shared that have ENGINELIB as the directory, the duplicate path is not the path I was moving to. They were just never updated to ENGINELIB_A, so deleting the now missing tracks from the library could be potentially dangerous as you mentioned.

I like ending up with all my songs in the same folder(I know you wouldn't believe it by looking at the database) the main reason I had that work flow was to make imports into new crates as fast as possible with the serato file browser, I would just pull the new folder over to the crate browser it would import all the songs into a new crate and I would essentially be done aside from a quick consolidate and relocate. This saved time because it didn't have to search through 15k songs and cross reference why was already in the library then import what wasn't then me having to go into the whole collection and sort by import date and then create a new crate.But now since I have a prime 4 I haven't opened serato since the day I did the import to engine.

Now is what I'm thinking you're saying true. Since I imported my serato playlist to engine, any change I make in serato automatically reflects in engine?!

But yes, I wasn't planning on really ever using serato again(other than to practice using some pioneer gear that I just acquired to get a better feel for the "industry standard" 🙄) and want to manage everything with Engine.

I would enjoy the smart playlist working with my serato library, but I may be one of the few that would use it for the exact reason you said.

[CaticornStudios]

Okay, so all the tracks that got moved yesterday are showing red in Engine. after running relocate in TUP and directing it to ENGINELIB_A it was only able to relocate 1045/3377 missing files. Ive attached a directory listing of ENGINELIB_A
enginelib_a_contents.txt

[CaticornStudios]

I wanna tack on to that, after relocating the 1045 tracks, it doesn't update the database. I ran it again and it still produced 3377 missing tracks, even after loading up prime again they're all still listed as missing.

I do have a question about cues/beat grids though. Where does Engine store them? is it in the metadata, or is it a database function?

[shayded-exe]

Very odd. I got a message from someone else who's experiencing a similar issue, yet I haven't been able to replicate it.

Let's try something. Make a backup of your Engine Library folder. Then go into Engine and delete all your tracks except one of the ones that it relocates but fails, and one that it can't relocate.

Copy the files for those tracks into a separate folder. Let's call it ENGINELIB_TEST.

Run a relocate and see what happens. Send me that DB file as well.

Also see what happens if you try relocating the file manually via "select file".

Sorry this is such a pain.

I do have a question about cues/beat grids though. Where does Engine store them? is it in the metadata, or is it a database function?

They're stored in the Engine DB.

[CaticornStudios]

So, even more weirdness going on now. I went to find the Snails song in Engine that i mentioned earlier, but its apparently not in my engine database anymore. I even opened m.db to look for it, and nothing(More confusing is that it was moved in the first consolidation attempt into ENGINELIB because that's where I was initially trying to consolidate my library. So we now know why that specific song didn't move into ENGINELIB_A, but where did it go to in the DB and Collection? it was transferred from Serato)

When I run relocate, both files are found. Opened Engine, and both are found, showing white, able to be played.

Current version of m.db is attached in email again.

All old paths removed, and the have been updated to ENGINELIB_TEST.

Also, for more information to help you I am running Windows 10 Pro Version 10.0.19043 Build 19043
Processor AMD A6-5200 APU with Radeon(TM) HD Graphics, 2000 Mhz, 4 Core(s), 4 Logical Processor(s)

[CaticornStudios]

I'm going ahead and saving the 2 file db as a copy and revert back to the original m.db and re run relocate with the two files in ENGINELIB_TEST.

only found the Jauz song.
I'm going to zip and send you ENGINELIB_TEST to see if you can possibly replicate

[shayded-exe]

I'll be gone this weekend but I'll take a look at this when I get back!

[CaticornStudios]

Have you had a chance to look into this? Is there anything I could possibly help with on my end?

[shayded-exe]

Haven't looked at your DB file yet, but if the song came from Serato it normally wouldn't be in your m.db file. Look in sm.db or stm.db.