Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

restructure macros #6450

Merged
merged 4 commits into from
Aug 6, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
.DS_Store
.vscode
manifest.json
macros/dune/create_views_of_dependencies.sql
macros/dune/alter_table_locations.sql
dbt_macros/dune/create_views_of_dependencies.sql
dbt_macros/dune/alter_table_locations.sql
scripts/integration_test/test_diffs_tokens/*
.vs

Expand Down
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
2 changes: 1 addition & 1 deletion dbt_subprojects/daily_spellbook/dbt_project.yml
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ vars:
model-paths: ["models", "../../sources"]
test-paths: ["tests"]
seed-paths: ["seeds"]
macro-paths: ["macros", "../../macros"]
macro-paths: ["macros", "../../dbt_macros"]

target-path: "target" # directory which will store compiled SQL files
clean-targets: # directories to be removed by `dbt clean`
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ Adoption of `arrakis_` for macro names.

### Macro Usage Example

Below is an example of how to use the [`arrakis_compatible_v2_trades` macro](/macros/models/_sector/alm/arrakis_compatible_trades.sql) macro within our project. This macro is designed to standardize the trades data for projects compatible with Arrakis V2 on various blockchains.
Below is an example of how to use the [`arrakis_compatible_v2_trades` macro](/dbt_macros/models/_sector/alm/arrakis_compatible_trades.sql) macro within our project. This macro is designed to standardize the trades data for projects compatible with Arrakis V2 on various blockchains.

```sql
{{
Expand Down
2 changes: 1 addition & 1 deletion dbt_subprojects/daily_spellbook/models/contracts/readme.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@

# Contract Mapping

Table models are defined in [`macros/models/_sector/contracts`](/macros/models/_sector/contracts)
Table models are defined in [`macros/models/_sector/contracts`](../macros/models/_sector/contracts)

This repository contains all the source code for `contracts.contract_mapping` that you can use to join contract addresses on each chain to pull our mapped project names in the `contract_project` field.

Expand Down
2 changes: 1 addition & 1 deletion dbt_subprojects/dex/dbt_project.yml
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ vars:
model-paths: ["models", "../../sources"]
test-paths: ["tests"]
seed-paths: ["seeds"]
macro-paths: ["macros", "../../macros"]
macro-paths: ["macros", "../../dbt_macros"]

target-path: "target" # directory which will store compiled SQL files
clean-targets: # directories to be removed by `dbt clean`
Expand Down
2 changes: 1 addition & 1 deletion dbt_subprojects/dex/models/trades/readme.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ Adoption of the `base_` prefix for table aliases and `uniswap_` for macro names.

### Macro Usage Example

One of the key components in the dex.trades redesign is the utilization of dbt macros to standardize and simplify the process of data transformation across different blockchains and projects. Below is an example of how to use the [`uniswap_compatible_v2_trades` macro](/macros/models/_sector/dex/uniswap_compatible_trades.sql) macro within our project. This macro is designed to standardize the trades data for projects compatible with Uniswap V2 on various blockchains.
One of the key components in the dex.trades redesign is the utilization of dbt macros to standardize and simplify the process of data transformation across different blockchains and projects. Below is an example of how to use the [`uniswap_compatible_v2_trades` macro](/dbt_subprojects/dex/macros/models/_project/uniswap_compatible_trades.sql) macro within our project. This macro is designed to standardize the trades data for projects compatible with Uniswap V2 on various blockchains.

```sql
{{
Expand Down
2 changes: 1 addition & 1 deletion dbt_subprojects/hourly_spellbook/dbt_project.yml
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ vars:
model-paths: ["models", "../../sources"]
test-paths: ["tests"]
seed-paths: ["seeds"]
macro-paths: ["macros", "../../macros"]
macro-paths: ["macros", "../../dbt_macros"]

target-path: "target" # directory which will store compiled SQL files
clean-targets: # directories to be removed by `dbt clean`
Expand Down
2 changes: 1 addition & 1 deletion dbt_subprojects/nft/dbt_project.yml
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ model-paths: ["models", "../../sources"]
# ../tests/* should be added to a separate shared folder
test-paths: ["tests"]
seed-paths: ["seeds"]
macro-paths: ["macros", "../../macros"]
macro-paths: ["macros", "../../dbt_macros"]

target-path: "target" # directory which will store compiled SQL files
clean-targets: # directories to be removed by `dbt clean`
Expand Down
2 changes: 1 addition & 1 deletion dbt_subprojects/solana/dbt_project.yml
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ vars:
model-paths: ["models", "../../sources"]
test-paths: ["tests"]
seed-paths: ["seeds"]
macro-paths: ["macros", "../../macros"]
macro-paths: ["macros", "../../dbt_macros"]

target-path: "target" # directory which will store compiled SQL files
clean-targets: # directories to be removed by `dbt clean`
Expand Down
2 changes: 1 addition & 1 deletion dbt_subprojects/tokens/dbt_project.yml
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ vars:
model-paths: ["models", "../../sources"]
# test-paths: ["tests"]
# seed-paths: ["seeds"]
macro-paths: ["../../macros"]
macro-paths: ["macros", "../../dbt_macros"]

target-path: "target" # directory which will store compiled SQL files
clean-targets: # directories to be removed by `dbt clean`
Expand Down
23 changes: 12 additions & 11 deletions docs/macros/macro_overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,19 +4,20 @@ There are various use cases for macros in DBT, but the main focus for wizards in

## Directory Path for Macros

- Path: `spellbook/macros/models/`
- Within this path, there are two main subdirectories:
Path: `dbt_subprojects/<project>/macros/` <br>
Within this path, there are two main subdirectories, replicating the structure in `models/`:

1. **\_project**
1. **\project**
- For standalone project spell lineages.
2. **\_sector**
2. **\sector**
- For sector-level spell lineages, e.g., `dex.trades` lineage.

- **Note**: The ‘\_’ prefix in directory paths is used to keep them at the top & easily accessible, reserved for the most frequently used directories.
Some exceptions to the project specific path are listed in [other macro use cases](#other-macro-use-cases)

### My Multi Word Header
## Design Principles for Macros

Following [this](/macros/models/_sector/dex/uniswap_compatible_trades.sql) example, where uniswap compatible projects code lives, the key design principles include:
Following [this](/dbt_subprojects/dex/macros/models/_project/uniswap_compatible_trades.sql) example, where uniswap compatible projects code lives, the key design principles include:

- **Assign Arguments**
- Define arguments expected to be passed in via each model that calls the macro.
Expand All @@ -27,7 +28,7 @@ Following [this](/macros/models/_sector/dex/uniswap_compatible_trades.sql) examp
- **Multiple Macros in One File**
- It's common to have multiple macros within a single file, such as various versions of the uniswap contract code. Group similar macros together logically.

Within models, such as uniswap v2, call macro code with [this approach](/models/_sector/dex/trades/ethereum/platforms/uniswap_v2_ethereum_base_trades.sql).
Within models, such as uniswap v2, call macro code with [this approach](/dbt_subprojects/dex/target/compiled/dex/models/trades/ethereum/platforms/uniswap_v2_ethereum_base_trades.sql).

## When to Use a Macro

Expand All @@ -46,13 +47,13 @@ Within models, such as uniswap v2, call macro code with [this approach](/models/

## Other Macro Use Cases

1. **Generic Test Queries**
1. **Generic Test Queries** (`dbt_macros/generic-tests`)
- Seed tests within model schema files call a seed macro containing the test query.
2. **Universal Use Cases in Spellbook**
2. **Universal Use Cases in Spellbook** (`dbt_macros/shared`)
- incremental predicates, containing the incremental filter which can be controlled in one location and called across multiple models.
- incremental days forward, which can chunk data from the source into smaller time frames to help bypass performance limitations.
- Macros with lists for for-loops in models, like [`all_evm_chains`](/macros/public/all_evm_chains.sql).
3. **Dune Team Specific Cases**
- Macros with lists for for-loops in models, like [`all_evm_chains`](/dbt_macros/shared/all_evm_chains.sql).
3. **Dune Team Specific Cases** (`dbt_macros/dune`)
- Overriding dbt-trino core macros for Spellbook-specific scenarios.
- Backend database specific code in pre or post hooks for spell optimization.

Expand Down
2 changes: 1 addition & 1 deletion docs/models/model_config_block.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ Each model within Spellbook contains a config block with various properties. Dep

4. **incremental_predicates**
- Filters the target to the same date range as the source, for improved performance & less data in memory.
- Universal incremental macro can be found [here](/macros/incremental_predicate.sql).
- Universal incremental macro can be found [here](/dbt_macros/dune/incremental_predicate.sql).
- **Note**: This is a newer addition to Spellbook. Please add this property for new incremental spells.

## Optional Configs for Materialized as Table / Incremental
Expand Down
2 changes: 1 addition & 1 deletion docs/models/model_overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ Required for DBT to understand the models within the project. Key areas within s
- If is incremental checks, done via jinja syntax
- This needs to be applied on all sources which have events / transactions / any time-series data set
- If no, full refresh and/or initial historical load on incremental model and bypass incremental filter, but apply filter for earliest date of activity for particular model
- If yes, apply [incremental predicate macro](/macros/incremental_predicate.sql) filter [on the source](/macros/models/_sector/dex/uniswap_compatible_trades.sql#L29-L32), to match incremental predicate filter on target in the config block
- If yes, apply [incremental predicate macro](/dbt_macros/dune/incremental_predicate.sql) filter [on the source](/dbt_subprojects/dex/macros/models/_project/uniswap_compatible_trades.sql#L29-L32), to match incremental predicate filter on target in the config block
- Optional use cases:
- Lookup to existing spell, within itself, using the {{ this }} syntax

Expand Down
2 changes: 1 addition & 1 deletion docs/seeds/seed_overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ Seeds will be required in these sector-level spell additions to ensure proper le
3. Within the [model schema file](/models/_sector/dex/trades/arbitrum/_schema.yml#L20-L23), call the [generic seed test](/tests/generic/check_dex_base_trades_seed.sql) with parameters necessary:
- Seed file name.
- Filter(s) for project versions, if the spell is split into versions per project.
4. Ultimately, following the above steps, the test query built and executed against seed files lives in the generic seed macro [here](/macros/generic-tests/check_seed_macro.sql).
4. Ultimately, following the above steps, the test query built and executed against seed files lives in the generic seed macro [here](/dbt_macros/generic-tests/check_seed_macro.sql).

## How Do I Track Seed Tests Running During Development?

Expand Down
33 changes: 0 additions & 33 deletions macros/public/public_macros_schema.yml

This file was deleted.

2 changes: 1 addition & 1 deletion scripts/alter_table_locations.py
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ def get_rename_command(self, table_dict):
return rename_command

def generate_macro_file(self, tables_dict):
f = open("../macros/dune/alter_table_locations.sql", 'w')
f = open("../dbt_macros/dune/alter_table_locations.sql", 'w')
f.write("{% macro alter_table_locations() %}")
for table, table_dict in tables_dict.items():
# alter_command = self.get_alter_command(table_dict)
Expand Down
Loading