Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

update dbt-core to v1.7.3 #60

Merged
merged 4 commits into from
Jan 17, 2024
Merged

update dbt-core to v1.7.3 #60

merged 4 commits into from
Jan 17, 2024

Conversation

mwhitaker
Copy link
Owner

No description provided.

@mwhitaker mwhitaker mentioned this pull request Dec 11, 2023
@mwhitaker
Copy link
Owner Author

even though dbt-core is now at v1.7.3, some of the adapters can be a bit behind

ARG [email protected]
ARG [email protected]
ARG [email protected]
ARG [email protected]
ARG [email protected]
ARG [email protected]

@leo-schick
Copy link
Contributor

@mwhitaker is there a chance that you can add dbt-databricks as adapter? It is not recommended to use dbt-spark for databricks anymore.

@mwhitaker
Copy link
Owner Author

OK, so this is what I have now:

ARG [email protected]
ARG [email protected]
ARG [email protected]
ARG [email protected]
ARG [email protected]
ARG [email protected]
ARG [email protected]

Since I don't personally have access to databricks, is there a way for you to test?

@leo-schick
Copy link
Contributor

@mwhitaker I tested it in a private repository and it works there. Can you upgrade to [email protected]?

In addition, it would be great to add an additional substitution of environment variable DBT_HTTP_PATH to the already existing DBT_TOKEN. Then one can create a cluster on the fly, run it and drop it again. Otherwise one has always to use a predefined cluster.

as a user input
update ghcr instructions
@mwhitaker
Copy link
Owner Author

I updated the databricks adapter to v.1.7.3 and I added support for a new optional http_path input. See here

@leo-schick
Copy link
Contributor

the optional _http_path_ seems to works, but the parameter _token_ does not work. I think you write the databricks token to the wrong file. See here: https://github.com/mwhitaker/dbt-action/blob/master/entrypoint.sh#L30 Should it be not be written to ./profiles.yml instead of ./datab.yml?

entrypoint.sh Outdated
@@ -28,6 +28,10 @@ elif [ -n "${DBT_TOKEN}" ]
then
echo trying to use DBT_TOKEN/databricks
sed -i "s/_token_/${DBT_TOKEN}/g" ./datab.yml
elif [ -n "${INPUT_HTTP_PATH}" ]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems to be wrong: For Databricks, we need to use both, the token and the http path.

@leo-schick
Copy link
Contributor

I fixed these issues and others I found in my branch. Check out the commits in my branch

* fix optional parameter INPUT_HTTP_PATH

* fix databricks token profiles.yml file

* fix INPUT_HTTP_PATH escape slash in sed

* check if profiles.yml exist

---------

Co-authored-by: Leo Schick <[email protected]>
@mwhitaker
Copy link
Owner Author

Thank you Leo, I appreciate it. I basically only use this action with BigQuery so I don't have a way to test the other connectors.

I added those changes to this branch. Can you do one more check please?

@leo-schick
Copy link
Contributor

It works fine for me in my branch. Since I guess this is a fast forward merge, will work here, too. There aren't any differences between my and your branch.

@mwhitaker mwhitaker merged commit 551e391 into master Jan 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants