Investigate porting some/all of Roslyn regex parsing checker tests

[danmoseley]

Look at dotnet/roslyn#23984 and see whether there is test material there that would make sense to pull into CoreFX (ie., not too duplicative, easy to port, not too long to run...)

cc @CyrusNajmabadi

[CyrusNajmabadi]

Note: my regex test were built by looking at the actual .net regex reference guide. I figured that i should at least be able to handle every case that it talked about.

Also, the regex test harness i built does a lot of interesting things. For example, it will try every substring from the left. Then every substring from the right. Then every mid substring. Then it will try having the original string but with each individual character removed.

This helped shake out a ton of edge cases and issues from parsing. I would recommend a harness that does something similar, just to ensure that you don't actually get any crashes, just actual validation exceptions.

[danmoseley]

@CyrusNajmabadi hmm, now I look at your PR, how do you see this working? Eg., in

src/Workspaces/CSharpTest/RegularExpressions/CSharpRegexParserTests_BasicTests.cs
src/Workspaces/CSharpTest/RegularExpressions/CSharpRegexParserTests_DotnetNegativeTests.cs
src/Workspaces/CSharpTest/RegularExpressions/CSharpRegexParserTests_ReferenceTests.cs

pull out all the regexes on lines like

    public void ReferenceTest25()
    {
        Test(@"@""(((?'Open'<)[^<>]*)+((?'Close-Open'>)[^<>]*)+)*""", @"<Tree>

And then create a ginormous test in CoreFX with these as input data that verifies that newing up a Regex on them does not throw an exception?

Longer term it would be interesting to find a way to create text that matches each, or nearly, so that we could add tests that verified the matching also - to protect against regressions. I believe I've seen some academic paper about a system that would create text matching expressions...

[CyrusNajmabadi]

@CyrusNajmabadi hmm, now I look at your PR, how do you see this working? Eg., in

I would take all the tests, and then update the call to "Test" to no longer take the "expected tree" parameter. I would then keep the core 'Test' method that does all the input manipulation to build up all the potential strings to test. The test would then say if it thought parsing should pass/fail. And it would verify that if it was expected to fail, it only failed with an ArgumentException of some sort.

[ViktorHofer]

@CyrusNajmabadi

// bug with .net regex parser. can happen with patterns like: a{2147483647,}

We fixed that some weeks ago. Can you please update your forked off RegexParser with the fix?

[ViktorHofer]

A lot of these tests are negative tests which need Assert.Throws around it. The current situation where we always throw ArgumentException isn't ideal in the source code. We might want to refactor the exception handling in RegexParser to understand what's going on internally. I will create a separate issue for that.

[CyrusNajmabadi]

We fixed that some weeks ago. Can you please update your forked off RegexParser with the fix?

My parser didn't have the problem :) It supports all those cases i reported as bugs :)