Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

clarifies mf timespine & creates mini guide #6836

Open
wants to merge 29 commits into
base: current
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 24 commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
674eb1e
add requirements
mirnawong1 Jan 28, 2025
7961004
add mini guide for mf timespine
mirnawong1 Jan 28, 2025
91247ad
add imgs
mirnawong1 Jan 28, 2025
d0768b0
fix links
mirnawong1 Jan 28, 2025
c278a82
Update mf-time-spine.md
mirnawong1 Jan 28, 2025
7145d14
remove dbt cloud
mirnawong1 Jan 28, 2025
15ec294
Merge branch 'clarify-timepsine' of github.com:dbt-labs/docs.getdbt.c…
mirnawong1 Jan 28, 2025
c879246
Merge branch 'current' into clarify-timepsine
mirnawong1 Jan 29, 2025
7f449bb
Update website/docs/guides/mf-time-spine.md
mirnawong1 Jan 30, 2025
64f114d
Update website/docs/guides/mf-time-spine.md
mirnawong1 Jan 30, 2025
5c67def
Merge branch 'current' into clarify-timepsine
mirnawong1 Jan 30, 2025
d027884
Merge branch 'current' into clarify-timepsine
mirnawong1 Feb 3, 2025
5c3af77
Merge branch 'current' into clarify-timepsine
mirnawong1 Feb 7, 2025
b57e449
remove unnecessary examples
mirnawong1 Feb 7, 2025
eab8830
package lock
mirnawong1 Feb 7, 2025
8ab5852
update timespine guide
mirnawong1 Feb 7, 2025
869efac
Merge branch 'current' into clarify-timepsine
mirnawong1 Feb 10, 2025
b3e28bb
Merge branch 'current' into clarify-timepsine
mirnawong1 Feb 11, 2025
a4d7706
Merge branch 'current' into clarify-timepsine
mirnawong1 Feb 27, 2025
2f51705
Merge branch 'current' into clarify-timepsine
mirnawong1 Feb 27, 2025
63e41d6
Update website/docs/guides/mf-time-spine.md
mirnawong1 Feb 27, 2025
98d2910
Update mf-time-spine.md
mirnawong1 Feb 27, 2025
5067891
Update website/docs/guides/mf-time-spine.md
mirnawong1 Feb 27, 2025
4fb4006
Update website/docs/guides/mf-time-spine.md
mirnawong1 Feb 27, 2025
eb62f06
Update website/docs/guides/mf-time-spine.md
mirnawong1 Feb 27, 2025
8ed072a
Update website/docs/guides/mf-time-spine.md
mirnawong1 Feb 27, 2025
eebc171
Update mf-time-spine.md
mirnawong1 Feb 27, 2025
fc3ed90
update mf tiemspine
mirnawong1 Feb 27, 2025
34c9eb9
Merge branch 'current' into clarify-timepsine
mirnawong1 Mar 3, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 28 additions & 2 deletions website/docs/docs/build/metricflow-time-spine.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,22 +10,41 @@ tags: [Metrics, Semantic Layer]

It's common in analytics engineering to have a date dimension or "time spine" table as a base table for different types of time-based joins and aggregations. The structure of this table is typically a base column of daily or hourly dates, with additional columns for other time grains, like fiscal quarters, defined based on the base column. You can join other tables to the time spine on the base column to calculate metrics like revenue at a point in time, or to aggregate to a specific time grain.

MetricFlow requires you to define at least one dbt model which provides a time-spine, and then specify (in YAML) the columns to be used for time-based joins. MetricFlow will join against the time-spine model for the following types of metrics and dimensions:
To use MetricFlow with time-based metrics and dimensions, you _must_ provide a time spine table. This table serves as the foundation for time-based joins and aggregations. You can either:

- Create a time spine SQL table from scratch (check out the [example time spine tables](#example-time-spine-tables) section for examples), or
- Use an existing table in your project, like a `dim_date` table

And once you have a time spine table, you need to configure it in YAML to tell MetricFlow how to use it.


:::tip
Check out our mini guide on [how to create a time spine table](/guides/mf-time-spine) to get started!
:::

## Prerequisites
MetricFlow requires you to define at least one dbt model which provides a time-spine, and then specify (in YAML) the columns to be used for time-based joins. This means you need to:

- Define at least one [time spine table](#example-time-spine-tables) at the whichever granularity needed for your metrics (either daily or hourly). You can optionally define additional tables for coarser grains (like monthly or yearly).
- [Configure each time spine table in a YAML file](#configuring-time-spine-in-yaml) to define how MetricFlow recognizes and uses its columns.

MetricFlow will then join against the time spine model for the following types of metrics and dimensions:

- [Cumulative metrics](/docs/build/cumulative)
- [Metric offsets](/docs/build/derived#derived-metric-offset)
- [Conversion metrics](/docs/build/conversion)
- [Slowly Changing Dimensions](/docs/build/dimensions#scd-type-ii)
- [Metrics](/docs/build/metrics-overview) with the `join_to_timespine` configuration set to true

To see the generated SQL for the metric and dimension types that use time spine joins, refer to the respective documentation or add the `compile=True` flag when querying the Semantic Layer to return the compiled SQL.
To see the generated SQL for the metric and dimension types that use time spine joins, refer to the respective documentation or add the `compile=true` flag when querying the Semantic Layer to return the compiled SQL.

## Configuring time spine in YAML

Time spine models are normal dbt models with extra configurations that tell dbt and MetricFlow how to use specific columns by defining their properties. Add the [`models` key](/reference/model-properties) for the time spine in your `models/` directory. If your project already includes a calendar table or date dimension, you can configure that table as a time spine. Otherwise, review the [example time-spine tables](#example-time-spine-tables) to create one. If the relevant model file doesn't exist, create it and add the configuration mentioned in the [next section](#creating-a-time-spine-table).

Some things to note when configuring time spine models:

- Make sure you already have a time spine SQL table defined in your project.
- Add the configurations under the `time_spine` key for that [model's properties](/reference/model-properties), just as you would add a description or tests.
- You only need to configure time-spine models that the Semantic Layer should recognize.
- At a minimum, define a time-spine table for a daily grain.
Expand Down Expand Up @@ -145,6 +164,13 @@ Note that if you're migrating from a `metricflow_time_spine.sql` file:

## Example time spine tables

The following examples show how to create time spine tables at different granularities:

- [Daily](#daily)
- [Daily (BigQuery)](#daily-bigquery)
- [Hourly](#hourly)


### Daily

<File name="metricflow_time_spine.sql">
Expand Down
247 changes: 247 additions & 0 deletions website/docs/guides/mf-time-spine.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,247 @@
---
title: "Quickstart with MetricFlow time spine"
id: "mf-time-spine"
level: 'Intermediate'
icon: 'guides'
tags: ['Quickstart']
hide_table_of_contents: true
recently_updated: true
---

<div style={{maxWidth: '900px'}}>

## Introduction

This guide explains how to configure a time spine using the [dbt Semantic Layer Jaffle shop example project](https://github.com/dbt-labs/jaffle-sl-template) as a reference.

### What is a time spine table?

A [time spine](/docs/build/metricflow-time-spine) is essential for time-based joins and aggregations in MetricFlow, the engine that powers the dbt Semantic Layer.

To use MetricFlow with time-based metrics and dimensions, you must provide a time spine table. This table serves as the foundation for time-based joins and aggregations. You can either:

- Create a time spine SQL table from scratch, or
- Use an existing table in your project, like a `dim_date` table

And once you have a time spine table, you need to configure it in YAML to tell MetricFlow how to use it. This guide will show you how to do both!

### Prerequisites
Before you start, make sure you have:

- A dbt project set up. If you don't have one, follow the [Semantic Layer quickstart guide](/guides/sl-snowflake-qs?step=1) or the [dbt Cloud quickstart guides)(/guides?tags=Quickstart) guide to help you get started.

## Add a time spine SQL model

Let's get started by assuming you're creating a time spine table from scratch. If you have a dbt project set up already and have your own time spine table (like a `dim_date` type model), you can skip this step and go to [Use an existing dim_date table](https://docs.getdbt.com//guides/mf-time-spine#using-an-existing-dim-date-table).

The time spine table is a dbt model that generates a series of dates (or timestamps) at a specific granularity. In this example, let's create a daily time spine table &mdash; `time_spine_daily.sql`.

1. Navigate to the `models/marts` directory in your dbt project.
2. Add a new SQL file named `time_spine_daily.sql` with the following content:

<File name='models/marts/time_spine_daily.sql'>

```sql
{{
config(
materialized = 'table',
)
}}

with

base_dates as (
{{
dbt.date_spine(
'day',
"DATE('2000-01-01')",
"DATE('2030-01-01')"
)
}}
),

final as (
select
cast(date_day as date) as date_day
from base_dates
)

select *
from final
where date_day > dateadd(year, -5, current_date()) -- Keep recent dates only
and date_day < dateadd(day, 30, current_date());
```
</File>

This generates a table of daily dates ranging from 5 years in the past to 30 days into the future.


## Add YAML configuration for the time spine

Now that you've created the SQL file, configure it in YAML so MetricFlow can recognize and use it.

1. Navigate to the `models/marts` directory.
2. Add a new YAML file named `_models.yml` with the following content:

<File name='models/marts/_models.yml'>

```yaml
models:
- name: time_spine_daily
description: A time spine with one row per day, ranging from 5 years in the past to 30 days into the future.
time_spine:
standard_granularity_column: date_day # The base column used for time joins
columns:
- name: date_day
description: The base date column for daily granularity
granularity: day
```
</File>

This time spine YAML file:
- Defines `date_day` as the base column for daily granularity.
- Configures `time_spine` properties so MetricFlow can use the table

### Using an existing dim_date table

This optional approach reuses an existing table, saving you the effort of creating a new one. However if you created a time spine table from scratch, you can skip this section.

If your project already includes a `dim_date` or similar table, you can configure it as a time spine:

1. Locate the existing table (`dim_date`).
2. Update `_models.yml` file to configure it as a time spine:

<File name='_models.yml'>

```yaml
models:
- name: dim_date
description: An existing date dimension table used as a time spine.
time_spine:
standard_granularity_column: date_day
columns:
- name: date_day
granularity: day
```
</File>

This time spine YAML file configures the `time_spine` property so MetricFlow can use the table.

## Run and test the time spine

For the time spine table you created, let's run it and validate the output.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@patkearns10 , i think maybe lines 133-143 dont make sense. i probably should just start with 144, query dbt sl query --metrics revenue --group-by metric_time -- does that make sense to you in terms of how to test the .sql and yaml file work?
Screenshot 2025-02-07 at 17 23 16

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

eh, I can imagine most people are building the time spine for the first time. And it will fail if they do not run that time spine model.
Why do you want to not include those lines?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think it's because when i ran dbt run, i didn't see anything that confirmed it for me -- maybe in just the compiled code?

Copy link
Contributor Author

@mirnawong1 mirnawong1 Feb 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❓ confirm this

oh i think i need to clarify that they need to preview the sql file (in a cli it'll be dbt show)

Screenshot 2025-02-27 at 11 04 25


1. Run the time spine model to create the table and validate the output:
```bash
dbt run --select time_spine_daily
```

2. Check that the table:
- Contains one row per day.
- Covers the date range you want (5 years back to 30 days forward)

3. (Optional) If you have [metrics](/docs/build/metrics-overview) already defined in your project, you can query the table/metrics using [Semantic Layer commands](/docs/build/metricflow-commands) to validate the time spine.

Let's say you have a `revenue` metric defined. You can query the table/metrics using the following command:

```bash
dbt sl query --metrics revenue --group-by metric_time
```

This will output results similar to the following in the dbt Cloud IDE:
<Lightbox src="/img/quickstarts/dbt-cloud/validate-mf-timespine-output.jpg" title="Validate the time spine output in dbt Cloud IDE" />

4. Double check that the results are correct and returning the expected data.

## Add additional granularities (optional)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i thought we could do this based on this example but when i tested it, it only returned date_day.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

did you get this working? I would be happy to hop on a call and co-dev-test together if you can find overlapping time!


To support multiple granularities (like hourly, monthly), create additional time spine tables and configure them in YAML.

1. Add a new SQL file named `time_spine_hourly.sql` with the following content:
<File name='models/marts/time_spine_hourly.sql'>

```sql
{{
config(
materialized = 'table',
)
}}

with hours as (

{{
dbt.date_spine(
'hour',
"to_date('01/01/2000','mm/dd/yyyy')",
"to_date('01/01/2025','mm/dd/yyyy')"
)
}}

),

final as (
select cast(date_hour as timestamp) as date_hour
from hours
)

select * from final
-- filter the time spine to a specific range
where date_hour > dateadd(year, -4, current_timestamp())
and date_hour < dateadd(day, 30, current_timestamp())

```
</File>

2. Then update `_models.yml` file and add the hourly time spine (below the daily time spine config):

<File name='_models.yml'>

```yaml
models:
- name: time_spine_daily
... rest of the daily time spine config ...

- name: time_spine_hourly
description: A time spine with one row per hour, ranging from four years ago to today.
time_spine:
standard_granularity_column: date_hour
columns:
- name: date_hour
granularity: hour
```
</File>

3. Run the model to create the table:
```bash
dbt run --select time_spine_hourly
```

4. Validate the output by querying the generated table:
```bash
dbt sl query --metrics revenue --group-by metric_time__hour
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also the ide doesn't support metric_time__hour -- is this right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

or should i scrap this section?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would think the IDE supports metric_time__hour, why do you say otherwise?

You could try
dbt sl list dimensions --metrics revenue

```

:::tip
Try creating a monthly time spine! Duplicate your daily time spine model, adjust it to generate one row per month, and update the YAML file to include `granularity: month`. Give it a try!
:::

## What's next

<ConfettiTrigger>

Congratulations 🎉! You've set up a time spine and are ready to bring the benefits of MetricFlow and the dbt Semantic Layer to your organization. You've learned:

- How to create a time spine table or use an existing table.
- How to configure a time spine in YAML.
- How to add additional granularities to your time spine.

Here are some additional resources to help you continue your journey:

- [MetricFlow time spine](/docs/build/metricflow-time-spine)
- [dbt Semantic Layer](/docs/use-dbt-semantic-layer/dbt-sl)
- [Build metrics](/docs/build/metrics-overview)
- [Quickstart with dbt Semantic Layer](/guides/sl-snowflake-qs?step=1)

</ConfettiTrigger>

</div>
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading