Regressions in System.Globalization.Tests.StringSearch

[performanceautofiler]

Run Information

Architecture	x64
OS	ubuntu 18.04
Baseline	c3843c592eb3dfdcb0a7849c6f4ba4e9fc2288a0
Compare	6410312a63f9bd552c7b9d777192368efc95535d
Diff	Diff

Regressions in System.Globalization.Tests.StringSearch

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
IsPrefix_FirstHalf - Duration of single invocation	158.82 ns	205.13 ns	1.29	0.00	True

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Globalization.Tests.StringSearch*'

Payloads

Baseline
Compare

Histogram

System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options: (en-US, IgnoreNonSpace, False))

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 205.12955646039745 > 166.6854562362098.
IsChangePoint: Marked as a change because one of 4/1/2022 9:47:46 AM, 4/5/2022 12:43:03 AM falls between 3/26/2022 9:07:40 PM and 4/5/2022 12:43:03 AM.
IsRegressionStdDev: Marked as regression because -821.1045760583897 (T) = (0 -205.31193459511871) / Math.Sqrt((0.036342227688011 / (40)) + (0.03480960980686019 / (15))) is less than -2.005745995316835 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (40) + (15) - 2, .025) and -0.29410378247662333 = (158.65183100090928 - 205.31193459511871) / 158.65183100090928 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

[dotnet-issue-labeler]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[EgorBo]

#67370 cc @tarekgh

[tarekgh]

@EgorBo

My change is just doing extra one array access HighCharTable[charB] at the end of the search. I am not sure if this is enough to show this delta regression. And why is this happening on Ubuntu only?

CC @adamsitnik

[EgorBo]

@tarekgh hard to say, it could be a data or loop alignment issue

[ghost]

Tagging subscribers to this area: @dotnet/area-system-globalization
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Run Information

Architecture	x64
OS	ubuntu 18.04
Baseline	c3843c592eb3dfdcb0a7849c6f4ba4e9fc2288a0
Compare	6410312a63f9bd552c7b9d777192368efc95535d
Diff	Diff

Regressions in System.Globalization.Tests.StringSearch

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
IsPrefix_FirstHalf - Duration of single invocation	158.82 ns	205.13 ns	1.29	0.00	True

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Globalization.Tests.StringSearch*'

Payloads

Baseline
Compare

Histogram

System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options: (en-US, IgnoreNonSpace, False))

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 205.12955646039745 > 166.6854562362098.
IsChangePoint: Marked as a change because one of 4/1/2022 9:47:46 AM, 4/5/2022 12:43:03 AM falls between 3/26/2022 9:07:40 PM and 4/5/2022 12:43:03 AM.
IsRegressionStdDev: Marked as regression because -821.1045760583897 (T) = (0 -205.31193459511871) / Math.Sqrt((0.036342227688011 / (40)) + (0.03480960980686019 / (15))) is less than -2.005745995316835 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (40) + (15) - 2, .025) and -0.29410378247662333 = (158.65183100090928 - 205.31193459511871) / 158.65183100090928 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Author:	performanceautofiler[bot]
Assignees:	-
Labels:	area-System.Globalization, untriaged, refs/heads/main, ubuntu 18.04, RunKind=micro, Regression, CoreClr, x64
Milestone:	-

[ghost]

Tagging subscribers to this area: @JulieLeeMSFT
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Run Information

Architecture	x64
OS	ubuntu 18.04
Baseline	c3843c592eb3dfdcb0a7849c6f4ba4e9fc2288a0
Compare	6410312a63f9bd552c7b9d777192368efc95535d
Diff	Diff

Regressions in System.Globalization.Tests.StringSearch

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
IsPrefix_FirstHalf - Duration of single invocation	158.82 ns	205.13 ns	1.29	0.00	True

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Globalization.Tests.StringSearch*'

Payloads

Baseline
Compare

Histogram

System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options: (en-US, IgnoreNonSpace, False))

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 205.12955646039745 > 166.6854562362098.
IsChangePoint: Marked as a change because one of 4/1/2022 9:47:46 AM, 4/5/2022 12:43:03 AM falls between 3/26/2022 9:07:40 PM and 4/5/2022 12:43:03 AM.
IsRegressionStdDev: Marked as regression because -821.1045760583897 (T) = (0 -205.31193459511871) / Math.Sqrt((0.036342227688011 / (40)) + (0.03480960980686019 / (15))) is less than -2.005745995316835 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (40) + (15) - 2, .025) and -0.29410378247662333 = (158.65183100090928 - 205.31193459511871) / 158.65183100090928 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Author:	performanceautofiler[bot]
Assignees:	-
Labels:	area-CodeGen-coreclr, untriaged, refs/heads/main, ubuntu 18.04, RunKind=micro, Regression, CoreClr, x64
Milestone:	-

[danmoseley]

Linking the existing issue for the regression last summer: #56012

[adamsitnik]

My change is just doing extra one array access HighCharTable[charB] at the end of the search. I am not sure if this is enough to show this delta regression. And why is this happening on Ubuntu only?

I agree that I would not expect such big difference from this change, unless the benchmark started executing the slow path. But the setup logic prevents from it:

https://github.com/dotnet/performance/blob/d7dac8a7ca12a28d099192f8a901cf8e30361384/src/benchmarks/micro/libraries/System.Globalization/StringSearch.cs#L58-L59

it's configurable to use high chars, but for this particular test case it's disabled (last param == false):

https://github.com/dotnet/performance/blob/d7dac8a7ca12a28d099192f8a901cf8e30361384/src/benchmarks/micro/libraries/System.Globalization/StringSearch.cs#L74

[adamsitnik]

And why is this happening on Ubuntu only?

That is a great question, as it seems to affect only Ubuntu x64. ARM64 is fine:

https://pvscmdupload.blob.core.windows.net/reports/allTestHistory%2frefs%2fheads%2fmain_arm64_ubuntu%2018.04%2fSystem.Globalization.Tests.StringSearch.IsSuffix_DifferentLastChar(Options%3a%20(en-US%2c%20IgnoreNonSpace%2c%20False)).html

@EgorBo is it possible that it's a code gen issue?

[EgorBo]

I'll run locally and see if there are any diffs. I wonder if we can collect diffs automatically in the perf infra for auto-file regressions 🙂

[EgorBo]

Regression was fixed by PGO update and #69878