Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Update test suite for base selector functionality #74

Merged
merged 13 commits into from
Feb 10, 2024

Conversation

kachayev
Copy link
Collaborator

@kachayev kachayev commented Feb 6, 2024

Notable changes:

  • use pytest parametrize for testing multiple scenarios
  • use static seed for random source/target assignment
  • fix test case for regression masks (should be nan)
  • fix API calls to masked input removal (routing parameters instead of estimator parameters)
  • avoid testing inequality when equality has a known form (== always a better check than !=)

@kachayev kachayev requested a review from YanisLalou February 6, 2024 14:30
@kachayev kachayev changed the title Update test suite for utils module [WIP] Update test suite for utils module Feb 6, 2024
@kachayev
Copy link
Collaborator Author

kachayev commented Feb 6, 2024

@YanisLalou I have a couple of questions regarding this suite,

  1. test_base_selector_remove_masked_transform has a comment regarding the essence of the check, but I'm not sure the code actually does what is described in there. Could you please double check?
  2. test_base_selector_domains only performs the assert of 'internal' state of the selector, which is something that is better to be avoided in tests. It's better to test the behavior rather than implementation details as the latter might change rather frequently while we only care about behavior staying consistent. What is the observed behavior that is verified in this test?

Copy link

codecov bot commented Feb 6, 2024

Codecov Report

Attention: 1 lines in your changes are missing coverage. Please review.

Comparison is base (8e8fc7b) 87.16% compared to head (434a05b) 87.29%.

Files Patch % Lines
skada/base.py 75.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main      #74      +/-   ##
==========================================
+ Coverage   87.16%   87.29%   +0.13%     
==========================================
  Files          38       38              
  Lines        2439     2433       -6     
==========================================
- Hits         2126     2124       -2     
+ Misses        313      309       -4     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@YanisLalou
Copy link
Collaborator

@YanisLalou I have a couple of questions regarding this suite,

  1. test_base_selector_remove_masked_transform has a comment regarding the essence of the check, but I'm not sure the code actually does what is described in there. Could you please double check?
  2. test_base_selector_domains only performs the assert of 'internal' state of the selector, which is something that is better to be avoided in tests. It's better to test the behavior rather than implementation details as the latter might change rather frequently while we only care about behavior staying consistent. What is the observed behavior that is verified in this test?
  1. For the test_base_selector_remove_masked_transform, we want to test if the _remove_masked behaves as expected for a pipeline with a predictor at the end (so with a .predict() method) and for a pipeline with an adapter at the end (so with a .adapt() method). By just doing a .fit() we can know if _remove_masked behaves as expected because otherwise a ValueError would be raised somewhere down the line in the adapt() and fit() methods respectively.
    But yes by looking back at the code, we could also add an assert to check if _remove_masked() outputs arrays with coherent shapes (like for test_base_selector_remove_masked_continuous)

  2. Tbh for test_base_selector_domains, the goal was to make codevoc happy. So basically testing if a selector works if we give to it a sample_domain or not. And also to check if the default assigned domains are correct or not. But yeah if its better to not performs asserts of 'internal' state of a selector, I don't really see where we could check that the domains are correctly assigned by default.

@kachayev
Copy link
Collaborator Author

kachayev commented Feb 7, 2024

Got it. I'll see what could be done

@kachayev kachayev changed the title [WIP] Update test suite for utils module [WIP] Update test suite for base selector functionality Feb 10, 2024
@kachayev
Copy link
Collaborator Author

Looking closely into domains_, it doesn't have any specific 'publicly visible' behavior because the functionality it was supposed to cover is just not finished. I created a separate issue for this #84.

@kachayev kachayev merged commit 78d42f3 into main Feb 10, 2024
7 checks passed
@kachayev kachayev deleted the fix-update-utils-test branch February 10, 2024 13:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants