Store UIDs in a single centralized file to reduce filesystem clutter

[MikeSchulze]

Describe the project you are working on

This is a followup proposal about how Godot should store uid's

Describe the problem or limitation you are having in your project

I wrote this proposal based on the outcome of #4603 (comment)
With Godot 4.4-dev5, the UID files are created for each resource in the file system, which clutters the file system.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

Background

The main reason is to support users who deliberately or unknowingly destroy their projects by moving or renaming resources that are dependent on scenes.

dev5 state

Moreover, introducing UIDs files won't help when users improperly handle their project resources.
Here's a small example that reduces this solution to absurdity:
To hide the UID files in VS or Rider as suggested here, moving individual resources would result in the UID files remaining in their old location.
This completely breaks this solution, and worse, when Godot restarts, a new UID is created while the original UID remains as a corpse in the filesystem.
Additionally, UIDs are being created for all resources and scripts, regardless of whether they are referenced by a scene.

Case study

I use Rider and restructure my project by moving scenes and associated scripts to other directories and renaming them if necessary.
This has no effect within Rider, the scripts are still valid and no error message about invalid script references in the scenes is displayed.
To test the scene, a Godot instance is started to execute the scene, but this would end with an error on invalid script path reference.
This means that Godot must ALWAYS first check all references in scenes when starting.
To resolve missing or invalid references, Godot must scan the file system and redefine the references.
So far so correct?

I would therefore suggest a service that A is executed when Godot is started and B is scheduled as a task at regular intervals.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

Implementation Strategy

The service should maintain file consistency through similarity matching:

File Tracking

• Create hash index of all script/resource files
• Parse scenes to extract script references
• Store original paths and relationships in one meta file (+ save copy)

Change Detection

• Scan the filesystem periodically
• Compare file contents using Jaccard similarity:
  similarity = |A ∩ B| / |A ∪ B|
  where A,B are sets of code lines
• Threshold of 0.8 (80%) for match detection
• Prioritize paths with similar names/structure

Reference Management

• Update internal mapping on changes are detected
• Modify scene references automatically
• Keep backup of original paths
• Log all modifications

Error Handling

• Validate new references
• Enable rollback if needed
• Notify users of changes
• Provide manual override option

The service runs silently in editor background, only surfacing when user attention is required for ambiguous cases.

If this enhancement will not be used often, can it be worked around with a few lines of script?

--

Is there a reason why this should be core and not an add-on in the asset library?

handling uids are part of the core

[ydeltastar]

There are similar cache files in .godot folder like global_script_class_cache.cfg. I agree it could be a list in a single centralized file, something like uids.cfg or project.uids next to project.godot.

Periodical scans shouldn't be necessary. There are editor and OS filesystem-watching features to do it on demand.

[sockeye-d]

I think automatically repairing references is a good idea, but before the editor actually does it it should show a pop-up that displays which resources are being remapped where, in case it fails catastrophically

[RebelliousX]

Sorry to say this, but having a separate service for maintaining this is bad in my opinion. Many things can go wrong, and the idea of scanning the project periodically basically we are asking for trouble and stutter. Instead, I think a database file that can use fast query like SQL or an open source alternative would do just fine. More on that towards the end.

The whole implementation of UID is bad in my opinion, not the idea itself per se. Since they are useful to track changes of the locations of the files for example (prevent multiple duplicate resources? Copying tscn files using file explorer into other directories can lead to this problem and get spammed in the output log about it when the project opens).

Instead of polluting the project directory and include a reference to the UID file path in the resource file like tscn or others, Godot should have a single, fast and reliable database in the .godot directory that manages all of these shenanigans. Any time the file is moved or deleted, the database should be updated accordingly. If file can not be found, scan entire project directory and if not then recreate a new UID reference in the database.

This approach will:

• Organize everything into a single database hidden in the .godot directory.
• Eliminate the need to have irrelevant files to the user of Godot exposed to the project.
• Less clutter when dealing with source control / version control like Git, less unwanted spamming of changes.
• Less disk space, allocation of hundreds of thousands of these small files even if each one takes like 20 bytes of data, they will occupy 4kb minimum depending the cluster size on the disk drive. Larger drives can have larger cluster sizes and possibly 64kb. They add up.
• Database can be compressed and optimized, queried fast using SQL or anything similar, less seeking time for those who are still using HDDs
• Faster performance in general.

[Calinou]

Godot should have a single, fast and reliable database in the .godot directory

UIDs must be committed to version control, otherwise there's no guarantee that they will be consistent across different copies of the same project.

This is also where committing a centralized file can be problematic. It'll lead to merge conflicts much more often than separate files, as adding or removing files is an operation that doesn't lead to difficult-to-resolve merge conflicts.

There are ways to engineer file formats to cause merge conflicts less often (since Git deals with merge conflicts in terms of paragraphs separated by a line of whitespace), but I don't think we'll find a way to prevent 100% of merge conflicts.

[SeremTitus]

This is also where committing a centralized file can be problematic. It'll lead to merge conflicts much more often than separate files, as adding or removing files is an operation that doesn't lead to difficult-to-resolve merge conflicts.

I think this can be address by implementing a git hook that manages fully the centralized file (e.g project.uids), similar to pre-commit used in the engine's development to maintain code style.

On change detection, instead of going for full automated healing using Jaccard similarity algorithm as suggested by the proposal, a manual alternative would be better where a pop up shows files that are no longer mapped to there uids and file that are yet to be assigned to help the user resolve externally moved and renamed files. In this case new files are not automatically assigned uids if there is an existing uid with missing file the resolve popup first appears, can be further designed to also take file extension into consideration.

We can also have both centralized file (project.uids) and the current solution, leaving it as a stylistic preference on the developer.

[AThousandShips]

My two cents on this is that I don't think the issue of clutter is remotely serious enough to justify a highly complicated, potentially fragile point of failure, especially if ensuring it doesn't cause major frustration with VCS requires even more complex code to be added.

Separate .uid files are very limited in occurrence, and most files don't use them

Adding a central file to track this might also, depending on implementation and solutions, become a bottleneck if it's supposed to be highly up-to-date, with management of pushing UID entries to this file

Also, what happens if you commit only part of some new files? What if you add a menu system to your project and you're just working on that and you don't commit that to VCS? How are the UIDs handled then? That sounds like an additional point of friction with merge conflicts, unlike .uid files which you wouldn't include if you don't include their associated file

And in the end the reason to go with .uid files was to avoid having to handle moved files with various detection methods for out-of-editor changes, the main purpose of these files are to not be handled globally but locally and simply not caring about trying to figure out where file_x went after we restarted

[RebelliousX]

@Calinou The thing is, if a file resides in the project sub-directories, that the user has no control over editing that file, then that file should be considered metadata, extra fluff.

I personally set my git to ignore .uid files, since they can be recreated if needed, no?

@AThousandShips

And in the end the reason to go with .uid files was to avoid having to handle moved files with various detection methods for out-of-editor changes, the main purpose of these files are to not be handled globally but locally and simply not caring about trying to figure out where file_x went after we restarted

How does Git recognize if a file has moved to another directory instead of considering it being deleted and a new file created?
Monitoring file changes can be done by:

• creating hash for files
• get modified time and date
• only check during project starting or during changes in editor like saving a file or moving or renaming a file.

If there are mismatches (file path doesn't exist, modified date and time not the same on database) then mark that
entry in the database and see if other hashes match with different file path and fix the links from there.

If a file path is valid, but mismatching modified date and time during saving or project loading then recreate hash for those files.

This might be unrelated, but I thought of using a database similar to how Visual Studio uses a database .sdf for auto-complete and other things. While I certainly don't know how they implement that or how it works. That database can get corrupted and that is why Visual Studio allows you to refresh or reset that database by deleting the .vs folder in the same folder as the solution, it can be recreated when solution starts.

[Calinou]

I personally set my git to ignore .uid files, since they can be recreated if needed, no?

This will break any paths in scene/resource/script files that load paths by UID. When the UID is recreated, there's no guarantee it'll have the same value as before.

[RebelliousX]

This will break any paths in scene/resource/script files that load paths by UID. When the UID is recreated, there's no guarantee it'll have the same value as before.

Then, this is exactly why the current implementation is so bad that it can break things easily. Putting the UID inside scene/resource/script files is bad. What if I want to move the scene file externally (outside of editor), do I need to move the UID file too since it resides in the same directory?

At this point, it would be better to make a drastic changes and create new file types that hold both file types in one file. The scene, resource or script file and uid file in one file like .gdz instead of .gd, .tscnz instead of .tscn..etc.

[MikeSchulze]

Sorry to say this, but having a separate service for maintaining this is bad in my opinion.

I deliberately called it a service to keep the implementation open. Personally, I would prefer a database for storage anyway, as this solves several problems by itself.

UIDs must be committed to version control, otherwise there's no guarantee that they will be consistent across different copies of the same project.

I think this approach is terrible as already described and doesn't help in the slightest if the developers don't stick to it in a disciplined way

This is also where committing a centralized file can be problematic. It'll lead to merge conflicts much more often than separate files, as adding or removing files is an operation that doesn't lead to difficult-to-resolve merge conflicts.

There are ways to engineer file formats to cause merge conflicts less often (since Git deals with merge conflicts in terms of paragraphs separated by a line of whitespace), but I don't think we'll find a way to prevent 100% of merge conflicts.

For me, these are simply pretextual arguments, if you use a database and the information is tabulated with an unique id, which the uid is, there are no problems here.
Therefore, the merge conflicts argument is obsolete

My two cents on this is that I don't think the issue of clutter is remotely serious enough to justify a highly complicated, potentially fragile point of failure, especially if ensuring it doesn't cause major frustration with VCS requires even more complex code to be added.

Separate .uid files are very limited in occurrence, and most files don't use them

I don't know how you came to that conclusion, it's extremely nerdy to have all those uid files.
In my plugin I have almost 400 additional files ending with “uid” and I find that a huge burden.

And in the end the reason to go with .uid files was to avoid having to handle moved files with various detection methods for out-of-editor changes, the main purpose of these files are to not be handled globally but locally and simply not caring about trying to figure out where file_x went after we restarted

This statement is not correct. Godot already has a service/tool that runs in the background. It searches for “missing” uid files and restores them. According to the statement, Godot should also recognize the external move at runtime, so in the end it comes down to the same thing.

This will break any paths in scene/resource/script files that load paths by UID. When the UID is recreated, there's no guarantee it'll have the same value as before.

This is also not true, if the uid files are missing they are created again and only if the .godot folder is missing.
And by the way, there is a uid cache file!

[arkology]

This is why I was against this uids zerg rush and stay on dev build without uids.
You have to commit this files. And this complicates repository navigation in browser significantly (yes, there was metadata files before for images and so on, but not for scripts - and when you browse repository, 99% of time you are looking for scripts). Or in external editor, doesn't matter.
Unfortunately I don't see any way this could be done to make everyone happy.
And one of the "coolest" part of current implementation:

When the UID is recreated, there's no guarantee it'll have the same value as before.

[RebelliousX]

Honestly, just delete those uids, lol.

[univeous]

Because I don't want to see these uid files, even though dev6 has been released, I am still stuck at dev4 - I believe this change should be discussed more.

[RebelliousX]

Just imagine, I in mid of 2021 proposed here #2812 to add unique identifier for resource files. And my proposal was dismissed with adamant refusal. Now they implement this half baked implementation (not the idea itself) that almost everybody dislikes 🤣

[arkology]

Just imagine, I in mid of 2021 proposed here #2812 to add unique identifier for resource files. And my proposal was dismissed with adamant refusal. Now they implement this half baked implementation (not the idea itself) that almost everybody dislikes 🤣

Please note that this comment is not constructive and because of it core team might think that "everybody" is only trolling haters. Meanwhile there is a fairly large group of people who use Godot almost every day and very confused with "new" uid system. And also there are people that doesn't care about it at all or care just a little. Or just don't know how to use GitHub, or don't use dev builds.
Maybe first of all we should list all advantages and disadvantages of current uid system and only then suggest how it could be improved. Unfortunately, GIP issue tracker not really suitable for this.

[RebelliousX]

not trolling, just stating facts.

[dalexeev]

I would like to point out one thing, as I think there is some misunderstanding here. There is no point in storing UIDs in a single file, they have no value by themselves. With the current concept, UIDs should be stored next to the files, or better yet inside them (but unfortunately not all formats support this).

If we implemented a file system scan on each editor startup to check and fix broken dependencies, then we could probably do away with UIDs altogether. However, this is quite a complex task. I will try to describe a rough algorithm, but I probably missed a lot of details:

• A full scan of the project's file system is performed on each editor startup. To make this process fast, resources are not parsed immediately, but first the presence of matching entries in the cache is checked.
• For each resource file, a content hash is calculated (or the date of the last file modification is read first, if the "Trust file modification date" option is enabled; if the date does not match, the hash cannot be taken from the cache and must be calculated). Matching the path and content hash with the value in the cache allows us to get dependency information from the cache and avoid resource parsing. If the cache does not have such a path and hash, then parsing the resource is necessary to get information about its dependencies.
• Note that changing the file content does not necessarily mean a broken dependency. As long as the path and resource type match, we can assume that this is the same resource, just modified (the user replaced an image or corrected a few lines in the script while the editor was closed). Yes, this may be a disadvantage compared to UID, but we could optionally notify the user if we have reason to assume that this is a different resource located at the same path (change in its dependencies or significant content change).
• Build a dependency graph. If broken dependencies are detected, the editor tries to automatically fix them using both the dependency information stored in the resources (the type of the external resource and its path) and the cache information.
• If the automatic fix of dependencies fails, the user is notified. They can fix dependencies manually (using editor suggestions based on heuristics like file name similarity, etc.) or continue without fixing and fix dependencies later when opening the relevant resources.
• Write updated information to cache.

Thus, the editor never trusts the cache completely and takes information from it only when the path and the content hash match. Any external changes to the project file system by the user or VCS in the worst case can only lead to the cache being useless and all resources being parsed, as at the initial start of the editor.

For example, a user creates two resources through the Godot editor, a script res://script.gd and a scene res://scene.tscn that uses the script. Godot writes dependency data to scene files (external resource type and path) and cache (same plus content hash and last modification date).

res://scene.tscn                  --->  res://script.gd
PackedScene                             GDScript
abcdef0123456789abcdef0000000002        abcdef0123456789abcdef0000000001
2024-12-09 10:02:00                     2024-12-09 10:01:00

The user then closes the Godot editor, moves the scene and script to the res://folder/ folder, and restarts the editor. The editor finds the res://folder/scene.tscn and res://folder/script.gd resources, calculates their hashes and gets their modification dates, and can use the cache information to track the move and complete the dependency information. The editor can then fix broken dependencies automatically.

Note that if the resource supports relative dependency paths, then an identical content hash may not be sufficient to indicate identical dependencies.

res://folder/scene.tscn           -/->  res://script.gd
PackedScene                             GDScript
abcdef0123456789abcdef0000000002        abcdef0123456789abcdef0000000001
2024-12-09 10:02:00                     2024-12-09 10:01:00

res://folder/script.gd
GDScript
abcdef0123456789abcdef0000000001
2024-12-09 10:01:00

If the user deletes the cache before the second editor start, the dependency information will be incomplete since resources do not store hashes and last modification dates of dependencies, so automatic fixing will be impossible. But the editor will be able to offer candidate/s from resources of the appropriate type, ranking them by similarity of file name/path and possibly some other heuristics.

res://folder/scene.tscn           -/->  res://script.gd
PackedScene                             GDScript
abcdef0123456789abcdef0000000002
2024-12-09 10:02:00

res://folder/script.gd
GDScript
abcdef0123456789abcdef0000000001
2024-12-09 10:01:00

[RebelliousX]

@dalexeev step in the right direction, constructive post. It matches some of my points I mentioned earlier. Just to clarify, you said

Godot writes dependency data to scene files (external resource type and path) and cache (same plus content hash and last modification date).

Internally or externally? If internally and you mentioned not all file types support this, then creating new packed file format like I mentioned before would do it. I prefer not, since it can thrash VCS / git with unwanted nonsensical changes.

Also:

If the user deletes the cache before the second editor start, the dependency information will be incomplete since resources do not store hashes and last modification dates of dependencies, so automatic fixing will be impossible. But the editor will be able to offer candidate/s from resources of the appropriate type, ranking them by similarity of file name/path and possibly some other heuristics.

I am not sure if it is possible to resolve any issues missed by the automatic fixing with 100 percent accuracy since you mentioned heuristics. At that point:

• offer the user an option to choose from a list of files which have orphan dependencies (no matching modification date or hash)
• or delete orphaned UIDs and generate new ones from the files which have corrupted UIDs but no matching ones. Overwrite old corrupted UIDs. This will be the fastest compared to full project rescan.
• lastly, if all fails, a full project rescan to regenerate UIDs at startup, not preferable.

And, most who dislike current implementation is because of the extra UID files messing up project directories. I still think their place should be inside .godot folder. Or at least unified in a folder next to godot.project or even inside godot.project 🤷‍♂️

Ultimately, whether the UID files exist or not should not hinder the project from running without errors. That is, if UID files missing regenerate them silently, the user doesn't need to know anything about them.

[dalexeev]

Godot writes dependency data to scene files (external resource type and path) and cache (same plus content hash and last modification date).

Internally or externally? If internally and you mentioned not all file types support this, then creating new packed file format like I mentioned before would do it. I prefer not, since it can thrash VCS / git with unwanted nonsensical changes.

I meant ExtResources in resources/scenes, external dependencies:

[gd_scene load_steps=2 format=3 uid="uid://1uj1u8btp4b0"]

[ext_resource type="Script" path="res://node.gd" id="1_0l1yd"]

[node name="Node" type="Node"]
script = ExtResource("1_0l1yd")

Built-in resource types (tscn, scn, tres, res) support dependencies in this form. Other resource types support them via ResourceFormatLoader._get_dependencies() or often have no dependencies at all (images and other media files). GDScript files have many implicit dependencies via preload(), load(), etc. but we do not include them in the dependency graph since Godot does not support circular dependencies.

The cache can only be filled by the Godot editor and we cannot rely on it being always correct. Therefore, a full (i.e. for all files) scan would be required at each editor startup. The cache is needed to speed up startup by avoiding parsing unchanged resources.

The cache may contain additional dependency information that is not stored in the resources themselves, but can be obtained from the file itself: content hash, modification date, dependencies (obtained by parsing the file). There is no point in storing external information like UID in the cache if the connection can be broken after moving/renaming the file or clearing the cache.

I am not sure if it is possible to resolve any issues missed by the automatic fixing with 100 percent accuracy since you mentioned heuristics.

Yes, but I don't think the imperfection of the solution makes it bad. With a cache, most cases would be resolved automatically. In other cases, suggestions for manual fixes could be very good. Finally, if the user simply deletes the resource, no option will be able to find it and the editor will have to show the user a dialog to fix dependencies.

And, most who dislike current implementation is because of the extra UID files messing up project directories. I still think their place should be inside .godot folder.

Ultimately, whether the UID files exist or not should not hinder the project from running without errors. That is, if UID files missing regenerate them silently, the user doesn't need to know anything about them.

At a minimum, UIDs should be tracked by the VCS, there is no benefit in regenerating them, as they will be new every time. Storing UIDs in project.godot might improve the situation somewhat, but it would still be an incomplete solution, and would annoy users and create VCS conflicts more often.

[RebelliousX]

At a minimum, UIDs should be tracked by the VCS, there is no benefit in regenerating them, as they will be new every time. Storing UIDs in project.godot might improve the situation somewhat, but it would still be an incomplete solution, and would annoy users and create VCS conflicts more often.

Kind of agree, I don't mind adding them to source control, what I do mind (like others here) is their current existence next to the resource files. The user doesn't and won't physically open the .uid files, let alone edit them. They should be filtered in one location or better, instead of having many small 20 bytes files, just save them in a single JSON file. This offers better performance, reading one file vs >9000.

[MikeSchulze]

The discussion is progressing well, and I would like to present my detailed thoughts on implementing the proposed service.

General Service Requirements

The service should fulfill these core requirements:

• Complete resolution of all dependencies and UIDs if not already resolved
• Capability to handle partial dependency resolution
• Internal maintenance of a dependency tree, which can be leveraged for refactoring operations, enabling comprehensive project-wide renaming capabilities

Godot Integration Workflow

Startup Behavior

• The service performs a comprehensive dependency check at startup and creates missing dependencies
• For broken dependencies that cannot be automatically resolved, the existing fixing dialog will be displayed for manual correction

Runtime Operation

• Handle all file system events (move, rename, delete, additions) with automatic dependency updates
• External changes while Godot is running can be managed through:
  a) A context menu in the Godot filesystem inspector for manual dependency updates on selected paths
  b) A periodic filesystem monitoring task (e.g., 5-minute intervals) that tracks changes and updates dependencies accordingly
  • This task should be optionally enabled to address any performance concerns
  • Note: If many developers are changing dependencies externally, we should investigate why this workflow is prevalent

Dependency Tree Structure

Every resource is ultimately bound to a scene; resources without such bindings are considered orphaned as they are never utilized. This necessitates a clear dependency status system:

private enum State
{
    Valid,            // valid and up to date
    Invalid,          // broken but in use
    Processing,       // actual processing e.g. moving, renaming or updating
    Orphaned          // unused not referenced
}

Dependency Entry Structure

class ResourceDependency {
    State State;            // actual state
    UidMeta Uid;            // resource meta data
    DateTime Timestamp;     // last dependency modification date
    List<ResourceDependency> Dependencies = new List<ResourceDependency>(); // the list of dependencies
    Dependency? Parent;     // the parent
}

class UidMeta {
    UID id;                 // Godot uid
    int Type;              // resource type
    DateTime Timestamp;    // last file modification date
    string Location;       // full filepath to an existing file
    string? OldLocation;   // stores old file location if has changed
    MinHash FingerPrint;   // finger print to the file stored as MinHash
}

Storage Implementation

The dependency information should be stored in an indexed database system for several advantages:

• High-speed operations through index utilization
• Access to standard database operations (filtering, sorting, insertion, deletion, sub-selects)
• Efficient data management and retrieval

Next Steps

To advance this proposal effectively, we should organize a focused brainstorming session with a small group of developers (maximum 5 participants) to exchange ideas and refine the implementation details.

This approach provides a robust foundation for managing dependencies while maintaining system performance and user flexibility.

[Calinou]

Godot can and should define a protocol and perhaps even implement an api for external applications to adopt to make changes to the graph in a responsible manner.

Considering there are thousands of external programs that can make changes (including command like tools like cp or mv), I doubt any of them would bother implementing a Godot-specific protocol.

[RebelliousX]

@loteque lol, what? You want external programs to fix Godot's problem?

And now because of your comment, @Calinou posted an article on Godot's website which I didn't read at all except for the headlines. But I get the gist that this is a rejected proposal and no matter what anyone says nothing will change.

It is okay, I just added .uid files to gitignore and deleted all the .uid files. I am happy whether the unlikely event that this proposal gets implemented or not.

Feel free to downvote this comment or censor it like usual.

[caphindsight]

Honestly, I'm disappointed that the separate-file-UID design gets released.

I was perfectly happy with human-readable file path references. If I ever have to do a refactoring, I would either move the files inside Godot, or hack up a bash script. Run the tests before committing the new state and you'll be ok.

Instead now my cognitive load increases almost 2x, for each script file I have a useless to me file I can't get rid of.

Can this at least be made optional? E.g. I want to be able to disable the "track resources with uids instead of file paths" option in the project settings. Maybe even add it as a setting on project creation given how many people express the same viewpoint?

P.S. before someone compares them to import files that already exist: no. Import files define structured metadata about the import pipeline, they are part of my project. I want to think about them. With uid files, I wish they didn't exist at all.

[caphindsight]

It is okay, I just added .uid files to gitignore and deleted all the .uid files. I am happy whether the unlikely event that this proposal gets implemented or not.

I worry your solution won't work for long: it's only a matter of time before some resources stop using file path fallbacks for inter dependencies, and your projects start breaking, or at least open with annoying warnings.