You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Can we wrap SMB JobTest input in an exactly-once iterable as well so that any re-iteration is caught in JobTest? I believe this can be done by wrapping the test input here.
The text was updated successfully, but these errors were encountered:
SMB Transform asserts that iterable inputs to the
via
function are not re-iterated (https://github.com/spotify/scio/blob/v0.14.12/scio-smb/src/main/java/org/apache/beam/sdk/extensions/smb/SortedBucketSource.java#L322-L342). This is because for performance reasons, the iterable is materialized lazily, to avoid materializing the entire group in memory in case of very large key groups.Can we wrap SMB JobTest input in an exactly-once iterable as well so that any re-iteration is caught in JobTest? I believe this can be done by wrapping the test input here.
The text was updated successfully, but these errors were encountered: