object_store: Automatically fall back to use_azure_cli=True if no other credentials can be found

[daviewales]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

Currently, object_store automatically detects Azure credentials from environment variables and IMDS. object_store also supports retrieving credentials from Azure CLI, however, it does not automatically try this.
If you want to retrieve credentials from Azure CLI, you need to manually specify use_azure_cli=True.

Describe the solution you'd like

Ideally, object_store would try use_azure_cli=True if no environment or IMDS credentials can be found. This makes it possible to easily test code on your local machine (based on your Azure CLI credentials), then deploy the code unchanged to an Azure VM, and it will automatically use the environment or IMDS credentials, without requiring any code changes.

Describe alternatives you've considered

A workaround is to add AZURE_USE_AZURE_CLI=True to your local environment variables while testing. However, this is an extra step.

Additional context

Most Azure libraries fall back to the Azure CLI credential if environment or IMDS credentials are not found. For example, fsspec/adlfs, azure.identity.DefaultAzureCredential, polars, pandas.

I'm aware of #6470, but I'm wondering if simply adding a fallback to an already supported authentication method is more in scope than completely replicating DefaultAzureCredential.

Related to this downstream obstore issue: developmentseed/obstore#267 (comment)

[kylebarron]

Thanks for creating this issue. I don't know enough about Azure to argue for or against this.

I assume this would be considered a breaking change, and so it would need a resolution within the next few days (unless we want to wait a while).

[tustvold]

My 2 cents is I would not be expecting an application to be shelling out unless I told it to do so, especially given IIRC the azure binary doesn't have a very collision resistant name. I'm not familiar enough with Azure tooling to know if this is an unusual expectation.

That being said, applications wanting this behaviour by default could override the default options

[kylebarron]

I think that's a fair rationale for keeping the default here as false.

While I want the Python bindings to minimize the differences from object_store for best maintainability, I think this is a place where it might be reasonable to override the default on the application side for easy usability for Python end users. developmentseed/obstore#292