[Feature] Take --target-path / DBT_TARGET_PATH into account for dbt clean

[iwarp]

Is this a new bug in dbt-core?

• I believe this is a new bug in dbt-core
• I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

I'm planning to use different target directories for performance reasons when running in a container, using different target-dirs for dev, test, prod to cache the generation of the manifest.json.

When running dbt clean with a target-path this is ignored and only looks at the target folder.

Expected Behavior

Expect that it should clean the files in my provided target-path parameter.

Steps To Reproduce

Running the following command is correct and generates a target-dev folder
dbt parse --profiles-dir=../.dbt --target=dev -d --target-path=target-dev

When i run dbt clean it does not clean up the files in my target-dev but cleans up files in the target folder not target-dev.

dbt clean --profiles-dir=../.dbt --target=dev -d --target-path=target-dev

Relevant log output

Truncated logs

`
00:31:29  Running with dbt=1.9.2
00:21:53  Checking C:\git\dbt-88a3f9f78dc0\dbt_packages/*
00:21:53  Cleaned C:\git\dbt-88a3f9f78dc0\dbt_packages/*
00:21:53  Checking C:\git\dbt-88a3f9f78dc0\target/*
00:21:53  Cleaned C:\git\dbt-88a3f9f78dc0\target/*
`

Environment

- OS: Windows 11
- Python: 3.10.11
- dbt: 1.9.2

Which database adapter are you using with dbt?

bigquery

Additional Context

Related to PR #5402

[dbeatty10]

Thanks for reaching out @iwarp !

I see what you are saying: dbt clean doesn't take the --target-path CLI into account (nor the DBT_TARGET_PATH environment variable). Rather, it only takes clean-targets into consideration.

This behavior aligns with the description of dbt clean in our documentation:

dbt clean is a utility function that deletes the paths specified within the clean-targets list in the dbt_project.yml file.

I can see why you'd want it to include --target-path and/or DBT_TARGET_PATH though.

Since the current behavior matches our documentation, I'm going to convert this to a feature request for further consideration.

In the meantime, here are a couple workarounds:

Workaround for --target-path CLI flag

Instead of:

dbt clean --target-path SOME_CUSTOM_TARGET

Do this instead in macOS / Linux (or the equivalent commands in a Windows environment):

rm -rf SOME_CUSTOM_TARGET

Workaround for DBT_TARGET_PATH environment variable

dbt_project.yml

clean-targets:
  - '{{ env_var("DBT_TARGET_PATH", "target") }}'

[dbeatty10]

@iwarp I took a closer look at this. The relevant code is here:

dbt-core/core/dbt/config/project.py

Lines 414 to 420 in 94b6ae1

	global_flags = get_flags()
	flag_target_path = str(global_flags.TARGET_PATH) if global_flags.TARGET_PATH else None
	target_path: str = flag_or(flag_target_path, cfg.target_path, "target")
	log_path: str = str(global_flags.LOG_PATH)
	clean_targets: List[str] = value_or(cfg.clean_targets, [target_path])

Basically, if you have clean-targets: defined in your dbt_project.yml, then it will always take precedence. But if you leave it undefined, then you can override it with --target-path and/or DBT_TARGET_PATH. If neither of those are supplied, then it will clean the target-path directory defined in dbt_project.yml (using target as the fallback value).

Example

Here's an example after I made sure to not include clean-targets: in my dbt_projects.yml:

$ dbt parse --target-path my_target              
02:33:53  Running with dbt=1.9.1
02:33:53  Registered adapter: duckdb=1.9.1
02:33:53  Unable to do partial parsing because saved manifest not found. Starting full parse.
02:33:54  Performance info: /Users/dbeatty/copier-templates/duckdb-core-11347/my_target/perf_info.json

(dbt_1.9) $ dbt clean --target-path my_target
02:34:27  Running with dbt=1.9.1
02:34:27  Checking /Users/dbeatty/copier-templates/duckdb-core-11347/my_target/*
02:34:27  Cleaned /Users/dbeatty/copier-templates/duckdb-core-11347/my_target/*
02:34:27  Finished cleaning all paths.

Summary

The precedence order is essentially this:

1. cfg.clean_targets
2. flag_target_path
3. cfg.target_path
4. "target"

We could theoretically switch the precedence order so that it is something like this:

1. flag_target_path
2. cfg.clean_targets
3. cfg.target_path
4. "target"

But we'd need to further consider the pros/cons of such a change.