[BUG] - check_immutable_fields leads to a version mismatch error when performing an upgrade

[viniciusdc]

Describe the bug

In our latest release, we introduced a new function called check_immutable_fields, whose main goal is to assert that the user is not updating fields that would cause data destruction, such as changing the primary network storage mode from efs to ceph and vice versa.

The function relies on a storage snapshot of the nebari-config.yaml preserved in the nebari state .tf file under the first stage (e.g for an AWS deployment it will be at stages/01-terraform-state/aws/terraform.tfstate).
This allows us to compare the config's "previous" state with the current version present in your local host.

However, recently, when doing an upgrade from AWS, I noticed that the function raised an error of version mismatch due to inconsistencies with the preserved version on the state and the upgraded version on the host. When inspecting the code, this becomes clearer since we are loading the previous state yaml config into the nebari schema, which has an underlying version check, which results in errors since the installed version has changed.

/_nebari/stages/terraform_state/__init__.py:237 in deploy                                                                
                                                                                                
  234   def deploy(                                                                           
  235     self, stage_outputs: Dict[str, Dict[str, Any]], disable_prompt: bool = False      
  236   ):                                                                                    
❱ 237     self.check_immutable_fields()                                                     
  238                                                                                       
  239     # No need to run terraform init here as it's being called when running the        
  240     # terraform show command, inside check_immutable_fields                           
                                                                                                
/_nebari/stages/terraform_state/__init__.py:257 in check_immutable_fields                                                
                                                                                                
  254         yield                                                                     
  255                                                                                         
  256   def check_immutable_fields(self):                                                     
❱ 257     nebari_config_state = self.get_nebari_config_state()                              
  258     if not nebari_config_state:                                                       
  259       return                                                                        
  260                                                                                           
                                                                                                
/_nebari/stages/terraform_state/__init__.py:299 in get_nebari_config_state                                               
                                                                                                
  296       if resource["address"] == "terraform_data.nebari_config":                     
  297         from nebari.plugins import nebari_plugin_manager                          
  298                                                                                   
❱ 299         nebari_config_state = nebari_plugin_manager.config_schema(                
  300           **resource["values"]["input"]                                         
  301         )                                                                         
  302         break                                                                     
                                                                                                
/pydantic/main.py:164 in __init__                                                                                        
...

final expetion:

ValidationError: 1 validation error for ConfigSchema
nebari_version
  Assertion failed, nebari_version=2024.9.1 is not an accepted version, it must be equivalent to 2024.9.2.dev1+gc138a22.
Install a different version of nebari or run nebari upgrade to ensure your config file is compatible. [type=assertion_error, input_value='2024.9.1', input_type=str]
    For further information visit https://errors.pydantic.dev/2.4/v/assertion_error

Right now, there is a workaround for manually updating the data of the preserved configuration under the state file or removing the file entirely, but a better solution is required. I propose adding a circumvention parameter to the schema loading to avoid triggering this error when loading "state" configurations.

Expected behavior

Depllying after upgraded should not lead to mistmacth version errors that are obscure to the user to diagnose. Deploying after upgrading should not lead to mismatched version errors that are obscure to the user's ability to analyze.

OS and architecture in which you are running Nebari

Linux

How to Reproduce the problem?

• pip install nebari==2024.7.1
• Deploy nebari on 2024.7.1 (local)
• pip install nebari (latest)
• nebari upgrade -c ...
• nebari deploy -c ...

Command output

No response

Versions and dependencies used.

No response

Compute environment

AWS

Integrations

No response

Anything else?

No response

[marcelovilla]

I'm trying to come up with a potential solution for this. As a quick fix, we could override the nebari_version field on the fly to avoid issues, but that feels like a very hacky and not robust approach.

Another thing that comes to mind is to crate the instantiate the Pydantic model without any validation. Apparently, calling the model_construct method would be one way of accomplishing this. So instead of

nebari/src/_nebari/stages/terraform_state/__init__.py

Lines 299 to 301 in 98970ba

	nebari_config_state = nebari_plugin_manager.config_schema(
	**resource["values"]["input"]
	)

we would have something like:

 nebari_config_state = nebari_plugin_manager.config_schema.model_construct( 
     **resource["values"]["input"] 
 ) 

Lack of validation should not pose a problem as the input was validated before being stored in the Terraform state. However, I'm concerned about potential side effects of instantiating a model without validation that I might be overlooking (e.g., casting of specific types).

@Adam-D-Lewis @dcmcand @viniciusdc I'm curious if you have any thoughts on this

[Adam-D-Lewis]

After thinking about this, I think Nebari should support non-current versions as valid in the ConfigSchema class (e.g. during upgrades we support old and new nebari version).

A few possible options:

1. Subclass ConfigSchema with a class like CurrentConfigSchema that does enforce being on the latest version.
2. We could set a private variable and then check that value in the nebari_version field validator. e.g. _check_current which defaults to True, but can be overridden during instantiation (see pydantic docs).
3. We could add a check_current method and simply call it after instantiation in the locations where we would expect the version to be current.

[Adam-D-Lewis]

Thinking a bit more, the broader issue strikes me, we are instantiating an older config and the nebari config schema may or may not have changed since the older config was created. We could keep separate classes for each nebari config schema as it evolves, but I think in this case of checking for changes in immutable fields, we can simple compare the dict representations of the nebari config.

[marcelovilla]

we can simple compare the dict representations of the nebari config.

I think this is the way to go.

Keeping separate classes seems like a burden in terms of maintainability and while supporting two different versions might be possible, it wouldn't really solve the general underlying issue. This might not only happen with the version field but also with any other field that we update, like for example the provider field once we finally deprecate DigitalOcean.