Conditional MADE cannot estimate first dimension of input #85
+11
−7
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Hi, thanks for maintaining this package!
While working on a separate PR in the
sbipackage, we noticed that MADEs cannot correctly estimate the first dimension of the input whencontextis notNone. After a bit of digging, I think the reason is in the waycontextis used in MADEs. The output corresponding to the first dimension of the input is always independent of all the inputs (including itself), as enforced by the masking withoutput_layer=True.With the current implementation, even if
contextis provided, any input coming from the hidden layers is masked out in the final layer of MADE, and so the output for the first dimension is simply the bias term of the last layer, regardless ofcontext. I think the simplest fix is to introduce a dummy input variable, as done in this PR. This way the masking constraints are still satisfied between all the real dimensions of the input, and we can retain information fromcontextin the output layer.Happy to discuss further, and thanks again :)