Cosmos: Data migration

[smitpatel]

See #8443 (comment)

[MisinformedDNA]

You should edit your initial post and either copy/paste some relevant data or summarize. People shouldn't need to jump back and forth to understand the issue.

[paulroreilly]

I've been keeping a close eye on the CosmosDb provider for EFCore and the migrations strategy is the only concern I have now about moving my projects to Cosmos.

Just wanted to jump start this thread again to see if there has been progress on the topic - whether there will be tooling/code solution or a published best practice strategy to managing changes to the model.

In my use cases, I have very large object models which represent self contained transactions (many hundreds of properties and lists of lists of objects). Currently we are using EFCore with Azure SQL Server, performance is pretty good, but it is clear that Cosmos will be significantly faster and handle scale much better.

The main scenarios that I need to handle are:

1. Renaming an existing property.
2. Type changes to existing properties.

With those scenarios handled, I would be able to move our apps over - more complex scenarios like relationship changes happen less frequently, perhaps just a documented best practice strategy for handling those is all that is needed for this phase.

[denismaster]

Currently in my project we use CosmosDB stored procedures, which are written using JS, to create data migrations. Each migration receives a mode param ( 'up' | 'down' ). Each migrations are written manually.

We are also running this migrations manually, and some kind of a script to run it is under development.

We also have a document with an attribute _type that is equals to MigrationHistory. This document contains a list of applied migrations, similar to EFMigrationHistory in EF Core.

[PAHeartBeat]

Hi
I am not sure this is the right Issue to post my question or issue. So if I post in the wrong space/thread sorry...

I am facing a similar kind of issue with data migration with Cosmos + EF Core 6.0.

In my case, I have a few data in My Cosmos DB Collection. So when I add a new property to my data structure it throws me an error for the newly added property. The new property which is not yet in Cosmos Collection

Message: 
System.InvalidOperationException : Nullable object must have a value.

      Stack Trace: 
lambda_method284(Closure , QueryContext , JObject )
Enumerator.MoveNext()
List`1.ctor(IEnumerable`1 collection)
Enumerable.ToList[TSource](IEnumerable`1 source)

The data in the collection is created based on the data model below.

public class Person {
    public string id {get; set; }
    public string Name {get; set; }
    public string Mobile {get; set; }
 
   public Person() {
        id = string.Empty;
        Name = string.Empty;
        Mobile = string.Empty;
   }
}

Now I am changing the structure by adding one more property like

public class Person {
    public string id {get; set; }
    public string Name {get; set; }
    public string Mobile {get; set; }
    public bool IsMember {get; set; }

   public Person() {
        id = string.Empty;
        Name = string.Empty;
        Mobile = string.Empty;
        IsMember = false;
   }
}

After the change in the data structure for an entity when I call to get the Quearyable from DbSet of the entity and try to convert it to ToList I am facing that Nullable object error.

If I add the value of the field in all my documents in the collection it works fine without any error.

[roji]

@PAHeartBeat this issue is indeed about having the Cosmos provider modify all existing documents when a "schema" change occurs, i.e. a change is done in the CLR types compromising the model; in your case, the provider would automatically add JSON properties with default values (e.g. an empty string) when you add a property to a CLR type in the model.

However, this isn't currently implemented, so you have two options:

1. Perform the change yourself, but manually adding the missing JSON properties with default values (the same thing EF Core would do)
2. Make your model property non-required, so that it's valid for a document to be missing a value for it.

[roji]

Note: rather than having the Cosmos provider eagerly modify all existing documents on every schema change (which could be expensive), we could also think about applying schema changes lazily; in other words, when a schema change happens, we'd only apply the change when the document is next loaded (this probably requires having some sort of schema version in the Cosmos document).

Other options here might be to treat missing properties as null; this would at least allow the standard "add nullable property" scenario to work correctly.

[argium]

At a minimum, can you please include the name of the offending property in the exception? I'm migrating an existing dataset to EF Core and, when I hit this error, I lose a lot of time trying to figure out where the problem is. Sometimes it's null values, other times it's names or capitalisation. It's especially painful because I can't attach a breakpoint to understand what the current state is.

[roji]

See #21006 for having a default value when reading missing properties, specifically for the (very common) scenario of adding a new property with existing documents.