Performance inside Project.GetAllGlobs

[lifengl]

When I investigate the performance trace to load Roslyn.sln, I noticed that we spent quite some time in the Project.GetAllGlobs function (660 ms in the trace). I wonder whether we can create RegEx in a lazy way, to delay that overhead, until we start to use those globs to match file paths.

The reason is that this function (and the MsBuildGlob property) need access the msbuild model, and is not thread safe, so we have to pick up them inside a lock. We use them when we receive file system changes. But we do GetAllGlobs after every project changes, and may or may not receive file system changes after that, so if this slow operation can be delayed:

1, it may be delayed after the solution loading phase, (and at least, it will be done in a background process, instead of doing that within the performance critical path inside the project lock;

2, if the project is changed couple times, we may skip doing that for the middle state completely.

If you drill into the time spent inside this function, I am talking about the time spent primary under those two stacks (which will account more than 50% of the time).

Name	Inc %	Inc	Inc Ct
microsoft.build!Project.GetAllGlobs	100.0	660.239	1,191

• microsoft.build!Microsoft.Build.Evaluation.ItemSpec2[System.__Canon,System.__Canon]..ctor(class System.String,class Microsoft.Build.Evaluation.Expander2,class Microsoft.Build.Shared.IElementLocation,bool) | 49.0 | 323.558 | 459
• microsoft.build!Project.BuildGlobResultFromIncludeItem | 41.8 | 276.235 | 631
  |+ microsoft.build!Microsoft.Build.Globbing.CompositeGlob..ctor(class System.Collections.Generic.IEnumerable1) | 34.4 | 227.235 | 536 \|\|+ system.collections.immutable.ni!System.Collections.Immutable.ImmutableArray.ToImmutableArray[System.__Canon](System.Collections.Generic.IEnumerable1) | 34.4 | 227.235 | 536
  || + system.collections.immutable.ni!System.Collections.Immutable.ImmutableArray.CreateRangeSystem.__Canon | 34.4 | 227.235 | 536
  ||  + system.core.ni!System.Linq.Enumerable.ToArraySystem.__Canon | 34.4 | 227.235 | 536
  ||   + system.core.ni!System.Linq.Buffer1[System.__Canon]..ctor(System.Collections.Generic.IEnumerable1) | 34.4 | 227.235 | 536
  ||    + system.core.ni!System.Linq.Enumerable+WhereSelectEnumerableIterator2[System.__Canon,System.__Canon].MoveNext() | 34.4 | 227.178 | 535 \|\|    \|+ microsoft.build!Microsoft.Build.Evaluation.ItemSpec2+<>c[System.__Canon,System.__Canon].b__19_0(class Microsoft.Build.Evaluation.ItemFragment) | 34.3 | 226.178 | 534
  ||    ||+ microsoft.build!ItemFragment.get_MsBuildGlob | 34.3 | 226.178 | 534
  ||    || + mscorlib.ni!System.Lazy1[System.__Canon].get_Value() | 34.3 | 226.178 | 534 \|\|    \|\|  + mscorlib.ni!System.Lazy1[System.__Canon].LazyInitValue() | 34.3 | 226.178 | 534
  ||    ||   + mscorlib.ni!System.Lazy`1[System.__Canon].CreateValue() | 34.3 | 226.178 | 534
  ||    ||    + microsoft.build.ni!ItemFragment.CreateMsBuildGlob | 34.0 | 224.642 | 532

Name	Inc %	Inc
microsoft.build!Project.GetAllGlobs	100.0	660.239

• microsoft.build!Microsoft.Build.Evaluation.ItemSpec2[System.__Canon,System.__Canon]..ctor(class System.String,class Microsoft.Build.Evaluation.Expander2,class Microsoft.Build.Shared.IElementLocation,bool) | 49.0 | 323.558
  |+ microsoft.build!Microsoft.Build.Evaluation.ItemSpec`2[System.__Canon,System.__Canon].BuildItemFragments(class Microsoft.Build.Shared.IElementLocation,bool) | 48.4 | 319.640
  ||+ microsoft.build.ni!ItemFragment.CreateFileMatcher | 18.9 | 124.535

[lifengl]

Include @davkean, this stack is also in the critical path of loading solution today. I think this method can be much faster, if we push computation of creating matching structure (like RegEx or anything) to lazy (or reuse something cached during the project evaluation). If it still takes too long, we will consider to spin it off to a separated dataflow, so it won't be in the critical path, but that will make the dataflow more complex, and add other overhead to the other part of the product.

[cdmihai]

Closed by #2652