Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add Type Validation parameter for Pipeline Connections #8875

Merged
merged 38 commits into from
Mar 3, 2025

Conversation

sjrl
Copy link
Contributor

@sjrl sjrl commented Feb 19, 2025

Related Issues

Proposed Changes

This PR introduces a new connection_type_validation parameter for type compatibility checks and adds _types_are_compatible, and _strict_types_are_compatible functions to support it. The connection_type_validation parameter allows users to configure whether type validation is performed or not. If True then type validation is performed if False then all connections are allowed.

Additionally, I allow this option to be set at the pipeline level or at the individual connection level. I've updated the to_dict and from_dict methods of the pipeline to reflect this.

How did you test it?

  • Added unit tests
  • Updated existing tests

Notes for the reviewer

@mpangrazzi I'd appreciate your thoughts on the name of the variable and whether you think it makes sense to have the option available at both the pipeline level and the connection level or if it would make sense to only have it present in one spot.

Checklist

@github-actions github-actions bot added the type:documentation Improvements on the docs label Feb 19, 2025
@coveralls
Copy link
Collaborator

coveralls commented Feb 19, 2025

Pull Request Test Coverage Report for Build 13632268125

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

Details

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • 39 unchanged lines in 2 files lost coverage.
  • Overall coverage increased (+0.007%) to 89.978%

Files with Coverage Reduction New Missed Lines %
core/type_utils.py 2 95.45%
core/pipeline/base.py 37 90.85%
Totals Coverage Status
Change from base Build 13631205752: 0.007%
Covered Lines: 9571
Relevant Lines: 10637

💛 - Coveralls

@anakin87
Copy link
Member

I have marked the failing HF API test -depending on HF API instability- as xfail, so if you merge main, the errors should disappear.

@sjrl sjrl changed the title feat: feat: New Type Validation Mode for Pipeline Connections Feb 19, 2025
@sjrl sjrl changed the title feat: New Type Validation Mode for Pipeline Connections feat: Add Type Validation parameter for Pipeline Connections Feb 20, 2025
@sjrl sjrl marked this pull request as ready for review February 20, 2025 09:30
@sjrl sjrl requested review from a team as code owners February 20, 2025 09:30
@sjrl sjrl requested review from dfokina and mpangrazzi and removed request for a team February 20, 2025 09:30
@mpangrazzi
Copy link
Contributor

@sjrl in general, it looks reasonable to me!

As discussed, we may opt to have in first place the connection_type_validation at pipeline level, since it would be probably already enough useful. Then, according to feedbacks / needs, we'll extend it to the connection level. WDYT?

@sjrl
Copy link
Contributor Author

sjrl commented Mar 3, 2025

@mpangrazzi this is ready for another review!

Copy link
Contributor

@mpangrazzi mpangrazzi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

@sjrl sjrl merged commit 296e31c into main Mar 3, 2025
18 checks passed
@sjrl sjrl deleted the refactor-typing branch March 3, 2025 15:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
4 participants