Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

try_cast_literal_to_type doesn't respect timestamp precision #5507

Closed
vrongmeal opened this issue Mar 7, 2023 · 1 comment · Fixed by #5517
Closed

try_cast_literal_to_type doesn't respect timestamp precision #5507

vrongmeal opened this issue Mar 7, 2023 · 1 comment · Fixed by #5517
Labels
bug Something isn't working

Comments

@vrongmeal
Copy link
Contributor

Describe the bug

In the function try_cast_literal_to_type: https://github.com/apache/arrow-datafusion/blob/928662bb12d915aef83abba1312392d25770f68f/datafusion/optimizer/src/unwrap_cast_in_comparison.rs#L295

Converting from Timestamp(Nanosecond, _) to Timestamp(Microsecond, _) doesn't actually return the correct value. It simply interchanges the inner integer.

To Reproduce

Running the following:

#[test]
fn test_repro() {
    let new_scalar = try_cast_literal_to_type(
        &ScalarValue::TimestampNanosecond(Some(123456), None),
        &DataType::Timestamp(TimeUnit::Microsecond, None)
    ).unwrap().unwrap();

    // This passes which is wrong
    assert_eq!(new_scalar, ScalarValue::TimestampMicrosecond(Some(123456), None));

    // This fails!
    // assert_eq!(new_scalar, ScalarValue::TimestampMicrosecond(Some(123), None));
}

Expected behavior

// This should be the correct result
assert_eq!(new_scalar, ScalarValue::TimestampMicrosecond(Some(123), None));

Additional context

@comphead
Copy link
Contributor

comphead commented Mar 8, 2023

❯ select extract(microsecond from 123456::timestamp);
+---------------------------------------------+
| datepart(Utf8("MICROSECOND"),Int64(123456)) |
+---------------------------------------------+
| 123.456                                     |
+---------------------------------------------+

The sql repr works better.

Thanks @vrongmeal for reporting the bug.

onpaws added a commit to splitgraph/socrata-to-seafowl that referenced this issue Apr 4, 2023
- See apache/datafusion#5507
- Apparently there is a precision issue when casting to timestamp.
  This breaks comparing timestamps, which we need to ingest Socrata
- Workaround: use to_timestamp_micros()
- Credit to @gruuya (thanks!)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants