[Perf] Windows/arm64: 3 Regressions on 12/25/2022 7:48:20 PM

[performanceautofiler]

Run Information

Architecture	arm64
OS	Windows 10.0.19041
Baseline	b631a55053524d3de23c6490109cb771268a8a21
Compare	bd6709248295deefa1956da3aeb9b0e086fbaca5
Diff	Diff

Regressions in System.Reflection.Attributes

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
IsDefinedClassHit - Duration of single invocation	410.93 ns	491.49 ns	1.20	0.13	False
IsDefinedClassHitInherit - Duration of single invocation	408.73 ns	488.92 ns	1.20	0.10	False
GetCustomAttributesMethodBaseMissInherit - Duration of single invocation	161.91 ns	180.28 ns	1.11	0.05	False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Reflection.Attributes*'

Payloads

Baseline
Compare

Histogram

System.Reflection.Attributes.IsDefinedClassHit

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 491.4917528075549 > 430.3844885740954.
IsChangePoint: Marked as a change because one of 9/21/2022 8:02:52 PM, 10/4/2022 3:13:24 PM, 10/6/2022 2:13:23 PM, 12/13/2022 12:26:27 AM, 12/23/2022 9:40:21 PM, 12/28/2022 6:20:56 PM falls between 12/19/2022 10:34:07 PM and 12/28/2022 6:20:56 PM.
IsRegressionStdDev: Marked as regression because -83.47694813865296 (T) = (0 -490.9371868467724) / Math.Sqrt((14.908189866741163 / (23)) + (2.2475612098635502 / (10))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (10) - 2, .025) and -0.1888719610300066 = (412.9437003640305 - 490.9371868467724) / 412.9437003640305 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Reflection.Attributes.IsDefinedClassHitInherit

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 488.9151559304329 > 429.19914881280266.
IsChangePoint: Marked as a change because one of 9/19/2022 9:36:09 PM, 12/12/2022 5:25:39 PM, 12/23/2022 9:40:21 PM, 12/28/2022 6:20:56 PM falls between 12/19/2022 10:34:07 PM and 12/28/2022 6:20:56 PM.
IsRegressionStdDev: Marked as regression because -148.1510778491285 (T) = (0 -489.46607041916593) / Math.Sqrt((1.4420303226458067 / (23)) + (2.273123674755585 / (10))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (10) - 2, .025) and -0.19474345702038962 = (409.6829888818656 - 489.46607041916593) / 409.6829888818656 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Reflection.Attributes.GetCustomAttributesMethodBaseMissInherit

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 180.28445930299875 > 170.74552220818478.
IsChangePoint: Marked as a change because one of 9/22/2022 11:37:30 PM, 10/6/2022 2:13:23 PM, 12/23/2022 9:40:21 PM, 12/28/2022 6:20:56 PM falls between 12/19/2022 10:34:07 PM and 12/28/2022 6:20:56 PM.
IsRegressionStdDev: Marked as regression because -18.59110952185248 (T) = (0 -180.36277681686016) / Math.Sqrt((1.089205575663926 / (23)) + (7.950481565639627 / (10))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (10) - 2, .025) and -0.10449156861985809 = (163.29936953908708 - 180.36277681686016) / 163.29936953908708 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

[dakersnar]

the only commit in the diff range is dotnet/runtime#79964. Is this an expected regression?

Edit: Whoops, tagged the wrong person. cc @jkotas

[jkotas]

I would not expect dotnet/runtime#79964 to introduce a regression here.

The trend shows that the benchmark is very noisy, and the regression is within the historic noise.