Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add hypothesis strategy to generate complex schemas, use it to test schemas module #23361

Merged
merged 5 commits into from
Dec 15, 2022

Conversation

TheNeuralBit
Copy link
Member

@TheNeuralBit TheNeuralBit commented Sep 23, 2022

Fixes #23308

This PR adds a new module, apache_beam.typehints.testing.strategies, with hypothesis strategies for generating schema complex schema types. It also uses the strategy to exercise the conversion logic in apache_beam.typehints.schemas. This testing revealed a minor issue with schema ids in nested data, which is also patched in this PR.

GitHub Actions Tests Status (on master branch)

Build python source distribution and wheels
Python tests
Java tests
Go tests

See CI.md for more information about GitHub Actions CI.

@codecov
Copy link

codecov bot commented Sep 29, 2022

Codecov Report

Merging #23361 (32a31b2) into master (aa6cc16) will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##           master   #23361   +/-   ##
=======================================
  Coverage   73.34%   73.34%           
=======================================
  Files         719      721    +2     
  Lines       97169    97208   +39     
=======================================
+ Hits        71265    71297   +32     
- Misses      24555    24562    +7     
  Partials     1349     1349           
Flag Coverage Δ
python 82.99% <100.00%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
sdks/python/apache_beam/typehints/row_type.py 100.00% <100.00%> (ø)
sdks/python/apache_beam/typehints/schemas.py 88.97% <100.00%> (+0.02%) ⬆️
...s/python/apache_beam/typehints/testing/__init__.py 100.00% <100.00%> (ø)
...python/apache_beam/typehints/testing/strategies.py 100.00% <100.00%> (ø)
.../python/apache_beam/testing/test_stream_service.py 88.09% <0.00%> (-4.77%) ⬇️
...che_beam/runners/interactive/interactive_runner.py 90.50% <0.00%> (-1.27%) ⬇️
...eam/runners/portability/fn_api_runner/execution.py 92.49% <0.00%> (-0.64%) ⬇️
...hon/apache_beam/runners/worker/bundle_processor.py 93.42% <0.00%> (-0.13%) ⬇️
...ks/python/apache_beam/runners/worker/sdk_worker.py 89.40% <0.00%> (+0.16%) ⬆️
.../apache_beam/runners/interactive/dataproc/types.py 96.55% <0.00%> (+3.44%) ⬆️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@TheNeuralBit
Copy link
Member Author

Run Python PreCommit

@TheNeuralBit TheNeuralBit changed the title [WIP] Add hypothesis strategy to generate complex schemas, test schemas module Add hypothesis strategy to generate complex schemas, use it to test schemas module Sep 30, 2022
@TheNeuralBit
Copy link
Member Author

Run Python_PVR_Flink PreCommit

@TheNeuralBit
Copy link
Member Author

R: @pabloem

@github-actions
Copy link
Contributor

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control

@TheNeuralBit
Copy link
Member Author

R: @Abacn

@github-actions
Copy link
Contributor

This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the [email protected] list. Thank you for your contributions.

@github-actions github-actions bot added the stale label Dec 13, 2022
@Abacn
Copy link
Contributor

Abacn commented Dec 13, 2022

waiting on author

@TheNeuralBit
Copy link
Member Author

@Abacn does this need any action other than resolving the conflict?

Copy link
Contributor

@Abacn Abacn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@TheNeuralBit Sorry for the delay. Yes it is waiting for rebase, and note that since hypothesis is already a dependency, the change in setup.py is no longer needed.

@@ -98,6 +107,13 @@ def from_user_type(
fields = [(name, user_type.__annotations__[name])
for name in user_type._fields]

if _user_type_is_generated(user_type):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just curious about what is the purpose of the non-test code change?

@github-actions github-actions bot removed the stale label Dec 14, 2022
@pabloem
Copy link
Member

pabloem commented Dec 15, 2022

hopefully the rebase i did is enough to get ci to pass?

@pabloem
Copy link
Member

pabloem commented Dec 15, 2022

I'll just merge for now since Brian no longer works on this, and the change looks great : )

@pabloem pabloem merged commit 958a930 into apache:master Dec 15, 2022
@TheNeuralBit
Copy link
Member Author

Thanks @pabloem :)

lostluck pushed a commit to lostluck/beam that referenced this pull request Dec 22, 2022
…schemas` module (apache#23361)

* Add hypothesis

* Test schemas module with hypothesis-generated complex schemas

* Fix schema id handling for nested schemas

* removing duplicated dependency

Co-authored-by: Pablo <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Task]: Test more complex types in schemas_test and row_coder_test
3 participants