Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Perf] Linux/x64: 460 Improvements on 2/28/2023 11:10:36 AM #13752

Closed
performanceautofiler bot opened this issue Mar 8, 2023 · 28 comments
Closed

[Perf] Linux/x64: 460 Improvements on 2/28/2023 11:10:36 AM #13752

performanceautofiler bot opened this issue Mar 8, 2023 · 28 comments
Labels
arch-x64 branch-refs/heads/main kind-micro_mono mono-interpreter os-linux perf-improvement runtime-mono

Comments

@performanceautofiler
Copy link

performanceautofiler bot commented Mar 8, 2023 •
edited
Loading

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 967250cc86b4c613015066b47b882e7424fdb590
Compare 4862343c61f698da986901a182e94df24da1487f
Diff Diff

Improvements in System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
AndNotBenchmark - Duration of single invocation 37.30 ns 8.58 ns 0.23 0.18 False
BitwiseOrOperatorBenchmark - Duration of single invocation 36.01 ns 8.71 ns 0.24 0.24 False
ConditionalSelectBenchmark - Duration of single invocation 63.39 ns 10.86 ns 0.17 0.20 False
MultiplyBenchmark - Duration of single invocation 108.11 ns 10.78 ns 0.10 0.20 False
EqualityOperatorBenchmark - Duration of single invocation 25.40 ns 10.72 ns 0.42 0.25 False
EqualsStaticBenchmark - Duration of single invocation 109.00 ns 9.10 ns 0.08 0.18 False
SubtractionOperatorBenchmark - Duration of single invocation 104.68 ns 8.72 ns 0.08 0.19 False
XorBenchmark - Duration of single invocation 36.60 ns 10.32 ns 0.28 0.17 False
AddBenchmark - Duration of single invocation 107.68 ns 9.65 ns 0.09 0.17 False
OnesComplementBenchmark - Duration of single invocation 28.36 ns 6.66 ns 0.23 0.27 False
MultiplyOperatorBenchmark - Duration of single invocation 106.49 ns 9.03 ns 0.08 0.18 False
EqualsAllBenchmark - Duration of single invocation 28.25 ns 7.64 ns 0.27 0.23 False
EqualsBenchmark - Duration of single invocation 36.01 ns 8.54 ns 0.24 0.33 False
InequalityOperatorBenchmark - Duration of single invocation 25.19 ns 7.69 ns 0.31 0.25 False
BitwiseOrBenchmark - Duration of single invocation 35.67 ns 8.55 ns 0.24 0.20 False
AllBitsSetBenchmark - Duration of single invocation 15.90 ns 3.03 ns 0.19 0.29 False
BitwiseAndBenchmark - Duration of single invocation 35.99 ns 8.69 ns 0.24 0.23 False
ExclusiveOrOperatorBenchmark - Duration of single invocation 36.11 ns 9.02 ns 0.25 0.20 False
NegateBenchmark - Duration of single invocation 98.62 ns 6.67 ns 0.07 0.20 False
BitwiseAndOperatorBenchmark - Duration of single invocation 37.09 ns 8.53 ns 0.23 0.15 False
AddOperatorBenchmark - Duration of single invocation 105.69 ns 8.93 ns 0.08 0.17 False
SubtractBenchmark - Duration of single invocation 109.12 ns 8.67 ns 0.08 0.20 False
OnesComplementOperatorBenchmark - Duration of single invocation 28.35 ns 6.43 ns 0.23 0.20 False
UnaryNegateOperatorBenchmark - Duration of single invocation 96.88 ns 6.68 ns 0.07 0.17 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt32&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt32&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt32&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt32&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt32>.AndNotBenchmark

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.58353577782598 < 35.59073703285339.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 226.2258990694336 (T) = (0 -8.857394225645587) / Math.Sqrt((3.124270199842615 / (299)) + (0.3672580311462832 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7753627580855517 = (39.429767522781766 - 8.857394225645587) / 39.429767522781766 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt32&gt;.BitwiseOrOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.707389969290507 < 34.426167856606206.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 187.92202281024345 (T) = (0 -9.104401811134629) / Math.Sqrt((3.745451558043598 / (299)) + (0.5050355179868135 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7589657417450392 = (37.77223153691369 - 9.104401811134629) / 37.77223153691369 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt32&gt;.ConditionalSelectBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 10.861779548436942 < 59.996595867186414.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 255.10864841204864 (T) = (0 -11.20500338850064) / Math.Sqrt((8.53414346171704 / (299)) + (0.7447402548455812 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.8274888936890211 = (64.9523594631747 - 11.20500338850064) / 64.9523594631747 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt32&gt;.MultiplyBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 10.77644553303998 < 103.03553281260218.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 419.7272414870803 (T) = (0 -9.215487192162733) / Math.Sqrt((14.799822079375186 / (299)) + (0.41296268140017195 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.9166370223914813 = (110.54652144792166 - 9.215487192162733) / 110.54652144792166 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt32&gt;.EqualityOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 10.722335841331823 < 24.181447657656236.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 161.65121900528712 (T) = (0 -8.009429622631393) / Math.Sqrt((1.7749692916092672 / (299)) + (0.30689327679423534 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.6926305247720197 = (26.057986456497293 - 8.009429622631393) / 26.057986456497293 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt32&gt;.EqualsStaticBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 9.10493934738702 < 104.09005158677647.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 403.0685072494007 (T) = (0 -8.835562814983433) / Math.Sqrt((18.177270566593336 / (299)) + (0.31985531023530767 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.9222444606235339 = (113.63258342540229 - 8.835562814983433) / 113.63258342540229 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt32&gt;.SubtractionOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.718603526380258 < 99.41262073791847.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 439.56643746620597 (T) = (0 -9.069441117051515) / Math.Sqrt((11.235216820146894 / (299)) + (0.5560970758763666 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.915060817678026 = (106.7757055003486 - 9.069441117051515) / 106.7757055003486 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt32&gt;.XorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 10.323996715310098 < 34.29890872219455.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 205.95255161350244 (T) = (0 -9.036427307364477) / Math.Sqrt((3.157367169459682 / (299)) + (0.4140402784731462 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7603035464833292 = (37.69946186014803 - 9.036427307364477) / 37.69946186014803 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt32&gt;.AddBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 9.647242514443743 < 102.35295053348601.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 404.04936445730704 (T) = (0 -9.076566058831245) / Math.Sqrt((14.13115786754492 / (299)) + (0.7388283162910703 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.9178420132698004 = (110.47697758999351 - 9.076566058831245) / 110.47697758999351 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt32&gt;.OnesComplementBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 6.658339411561359 < 26.97920201809447.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 124.09757367051097 (T) = (0 -6.884374379531113) / Math.Sqrt((5.595302388095638 / (299)) + (0.6999673498008828 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7676864007668229 = (29.633970642507037 - 6.884374379531113) / 29.633970642507037 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt32&gt;.MultiplyOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 9.027379923689038 < 99.94304089885794.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 338.41814579731454 (T) = (0 -9.2301658663743) / Math.Sqrt((20.52872778474119 / (299)) + (0.7500306260113228 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.9142752096551091 = (107.67207279526939 - 9.2301658663743) / 107.67207279526939 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt32&gt;.EqualsAllBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 7.639164063437697 < 27.09795982699983.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 162.05251704229323 (T) = (0 -7.970839468076846) / Math.Sqrt((1.6402751066326755 / (299)) + (0.4976787137537339 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7204870382753047 = (28.516886726446977 - 7.970839468076846) / 28.516886726446977 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt32&gt;.EqualsBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.539668573097256 < 34.251225709921165.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 118.39186537905199 (T) = (0 -9.020584268714835) / Math.Sqrt((5.198631608700908 / (299)) + (1.9665798014937637 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7615720574849798 = (37.83358684205634 - 9.020584268714835) / 37.83358684205634 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt32&gt;.InequalityOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 7.6869629195296545 < 23.86736876947153.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 87.05376422455602 (T) = (0 -8.185250735646353) / Math.Sqrt((1.893198747786477 / (299)) + (1.6759760147040024 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.6854731933391951 = (26.024016275577967 - 8.185250735646353) / 26.024016275577967 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt32&gt;.BitwiseOrBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.54809008793295 < 34.23315815706993.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 179.46296817262703 (T) = (0 -9.061066044145647) / Math.Sqrt((3.569830692108138 / (299)) + (0.6477247219770629 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7605597512463033 = (37.842702266261135 - 9.061066044145647) / 37.842702266261135 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt32&gt;.AllBitsSetBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 3.0264182895778586 < 15.148708379169213.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 145.63734441171283 (T) = (0 -3.135402361299415) / Math.Sqrt((1.3133238788423784 / (299)) + (0.17444387440354725 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.807004257429968 = (16.245966463025347 - 3.135402361299415) / 16.245966463025347 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt32&gt;.BitwiseAndBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.692100132727578 < 34.42459462682457.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 117.98188295336976 (T) = (0 -9.156587422436035) / Math.Sqrt((7.894705952358686 / (299)) + (1.5532030997178567 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.758551137577054 = (37.92350616420149 - 9.156587422436035) / 37.92350616420149 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt32&gt;.ExclusiveOrOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 9.023611203617689 < 34.23114386695315.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 188.45797523171856 (T) = (0 -8.962151401476802) / Math.Sqrt((3.1435921111530107 / (299)) + (0.598352985058044 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7622443168786566 = (37.69479359575512 - 8.962151401476802) / 37.69479359575512 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt32&gt;.NegateBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 6.66722726171201 < 93.63882829734851.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 300.73225699241317 (T) = (0 -6.553632891869829) / Math.Sqrt((20.405707895177095 / (299)) + (1.4385521182936556 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.9351812363499789 = (101.10703325436971 - 6.553632891869829) / 101.10703325436971 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt32&gt;.BitwiseAndOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.52949331402228 < 34.09877614015595.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 121.31655464576914 (T) = (0 -8.963469137890808) / Math.Sqrt((5.408836101720952 / (299)) + (1.8223160143685717 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7634495909547987 = (37.892427132425794 - 8.963469137890808) / 37.892427132425794 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt32&gt;.AddOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.929266016110475 < 100.19141302368071.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 452.0602576958262 (T) = (0 -9.054845900676467) / Math.Sqrt((10.673303939109468 / (299)) + (0.5594098033941417 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.915911020287926 = (107.68171919413037 - 9.054845900676467) / 107.68171919413037 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt32&gt;.SubtractBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.674030721652276 < 102.31991387854251.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 373.01740721563624 (T) = (0 -9.079208570810376) / Math.Sqrt((17.676151684648445 / (299)) + (0.6613844994228198 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.9174570433535579 = (109.9937407100589 - 9.079208570810376) / 109.9937407100589 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt32&gt;.OnesComplementOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 6.426950884399135 < 27.132413126744094.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 177.42181977015935 (T) = (0 -6.855483111468721) / Math.Sqrt((2.0174344408913125 / (299)) + (0.43905439691918013 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.766502580858104 = (29.359995226768028 - 6.855483111468721) / 29.359995226768028 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt32&gt;.UnaryNegateOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 6.6769352142918486 < 92.07829050812019.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 403.6247789099512 (T) = (0 -6.675162236661231) / Math.Sqrt((14.14169017500092 / (299)) + (0.24017824821854689 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.932625221839644 = (99.07509039620065 - 6.675162236661231) / 99.07509039620065 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 8, 2023 •
edited
Loading

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 967250cc86b4c613015066b47b882e7424fdb590
Compare 4862343c61f698da986901a182e94df24da1487f
Diff Diff

Improvements in System.Memory.Span<Byte>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IndexOfAnyFourValues - Duration of single invocation 1.59 μs 472.82 ns 0.30 0.02 False
Reverse - Duration of single invocation 1.80 μs 543.76 ns 0.30 0.12 False
IndexOfAnyFiveValues - Duration of single invocation 2.41 μs 528.41 ns 0.22 0.05 False
IndexOfAnyFourValues - Duration of single invocation 164.04 ns 130.02 ns 0.79 0.12 False
IndexOfAnyThreeValues - Duration of single invocation 115.62 ns 108.94 ns 0.94 0.15 False
IndexOfAnyFiveValues - Duration of single invocation 222.02 ns 150.64 ns 0.68 0.14 False
IndexOfAnyTwoValues - Duration of single invocation 1.19 μs 337.30 ns 0.28 0.10 False
LastIndexOfValue - Duration of single invocation 689.49 ns 289.81 ns 0.42 0.04 False
IndexOfAnyTwoValues - Duration of single invocation 98.67 ns 84.39 ns 0.86 0.20 False
Reverse - Duration of single invocation 147.88 ns 71.29 ns 0.48 0.18 False
IndexOfValue - Duration of single invocation 702.10 ns 284.33 ns 0.40 0.03 False
SequenceCompareTo - Duration of single invocation 486.21 ns 379.22 ns 0.78 0.05 False
IndexOfAnyThreeValues - Duration of single invocation 1.42 μs 395.40 ns 0.28 0.01 False
LastIndexOfAnyValues - Duration of single invocation 1.19 μs 350.11 ns 0.29 0.17 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Memory.Span&lt;Byte&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Memory.Span&lt;Byte&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Memory.Span&lt;Byte&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Memory.Span&lt;Byte&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Memory.Span<Byte>.IndexOfAnyFourValues(Size: 512)

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 472.82377801087426 < 1.5128506707948532.
IsChangePoint: Marked as a change because one of 12/18/2022 12:43:31 PM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 419.68751526605695 (T) = (0 -498.6171524364111) / Math.Sqrt((1322.0495056994102 / (299)) + (110.68208756233075 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.6866282490442557 = (1591.1362492493072 - 498.6171524364111) / 1591.1362492493072 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Memory.Span&lt;Byte&gt;.Reverse(Size: 512)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 543.7571109648499 < 1.709586597641483.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 177.57717584253132 (T) = (0 -546.4388335937069) / Math.Sqrt((17191.45628983937 / (299)) + (44.0877680235475 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.712987182134885 = (1903.8830309331765 - 546.4388335937069) / 1903.8830309331765 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Memory.Span&lt;Byte&gt;.IndexOfAnyFiveValues(Size: 512)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 528.4064040779834 < 2.3274639580730767.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 334.3784490626496 (T) = (0 -559.0150116093484) / Math.Sqrt((9251.840905933514 / (299)) + (154.4009223722647 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7777522512660677 = (2515.2786239404513 - 559.0150116093484) / 2515.2786239404513 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Memory.Span&lt;Byte&gt;.IndexOfAnyFourValues(Size: 33)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 130.01537695115414 < 155.812600509118.
IsChangePoint: Marked as a change because one of 12/18/2022 12:43:31 PM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 24.550615045791627 (T) = (0 -132.86838160332138) / Math.Sqrt((23.095877074874437 / (299)) + (72.12636012615899 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.19001207583451335 = (164.03748455906037 - 132.86838160332138) / 164.03748455906037 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Memory.Span&lt;Byte&gt;.IndexOfAnyThreeValues(Size: 33)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 108.93652093247248 < 110.79938574485921.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 28.82411942477139 (T) = (0 -98.2011508505072) / Math.Sqrt((12.36580453310241 / (299)) + (19.191263918049437 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.16445973686958565 = (117.530124140983 - 98.2011508505072) / 117.530124140983 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Memory.Span&lt;Byte&gt;.IndexOfAnyFiveValues(Size: 33)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 150.64108949277303 < 211.20541206798129.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 95.47349882761006 (T) = (0 -143.88770810588414) / Math.Sqrt((32.31172093835501 / (299)) + (28.117865663691482 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.3580060384830195 = (224.12626400081547 - 143.88770810588414) / 224.12626400081547 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Memory.Span&lt;Byte&gt;.IndexOfAnyTwoValues(Size: 512)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 337.29599101176865 < 1.1250040293848438.
IsChangePoint: Marked as a change because one of 1/26/2023 10:17:57 AM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 134.20290420910615 (T) = (0 -348.8358516255098) / Math.Sqrt((11703.602988602266 / (299)) + (30.66416136483151 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7081923631471823 = (1195.4308509117466 - 348.8358516255098) / 1195.4308509117466 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Memory.Span&lt;Byte&gt;.LastIndexOfValue(Size: 512)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 289.8066229306045 < 653.6846676481896.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 288.2113991268702 (T) = (0 -303.59841587075266) / Math.Sqrt((243.36650156237982 / (299)) + (45.3317263503153 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.5586923183690191 = (687.9518043935162 - 303.59841587075266) / 687.9518043935162 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Memory.Span&lt;Byte&gt;.IndexOfAnyTwoValues(Size: 33)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 84.38947956903384 < 94.14712437289353.
IsChangePoint: Marked as a change because one of 1/26/2023 10:17:57 AM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 21.678572141851372 (T) = (0 -87.83356248977222) / Math.Sqrt((53.565117869371775 / (299)) + (19.732257851713495 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.16038307797126697 = (104.61147242904951 - 87.83356248977222) / 104.61147242904951 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Memory.Span&lt;Byte&gt;.Reverse(Size: 33)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 71.29381996064913 < 140.85152809163623.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 161.48843147918274 (T) = (0 -72.05910283084833) / Math.Sqrt((68.56486859439259 / (299)) + (2.8616399541461504 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.5469498297630659 = (159.05325185765452 - 72.05910283084833) / 159.05325185765452 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Memory.Span&lt;Byte&gt;.IndexOfValue(Size: 512)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 284.3257554226563 < 654.2249202575042.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 301.0040956116539 (T) = (0 -288.87076006021624) / Math.Sqrt((59.484107121677454 / (299)) + (74.58958760813678 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.5820322067991136 = (691.1316248746881 - 288.87076006021624) / 691.1316248746881 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Memory.Span&lt;Byte&gt;.SequenceCompareTo(Size: 512)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 379.22038014069255 < 460.6641889043532.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 60.02316670875525 (T) = (0 -394.46096011051765) / Math.Sqrt((144.6106897252097 / (299)) + (74.04811972098763 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.17921934805543766 = (480.5924203695296 - 394.46096011051765) / 480.5924203695296 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Memory.Span&lt;Byte&gt;.IndexOfAnyThreeValues(Size: 512)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 395.4023485617377 < 1.3493588040706705.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 542.7724649548321 (T) = (0 -410.80503069317194) / Math.Sqrt((555.2459327397161 / (299)) + (65.88023872561493 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7045871704071875 = (1390.613370649516 - 410.80503069317194) / 1390.613370649516 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Memory.Span&lt;Byte&gt;.LastIndexOfAnyValues(Size: 512)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 350.10544002690716 < 1.1317659936073718.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 161.79600550849725 (T) = (0 -361.1568247583528) / Math.Sqrt((6887.5671604319805 / (299)) + (30.225025702758913 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.6855305717918975 = (1148.4640234069254 - 361.1568247583528) / 1148.4640234069254 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 8, 2023 •
edited
Loading

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 967250cc86b4c613015066b47b882e7424fdb590
Compare 4862343c61f698da986901a182e94df24da1487f
Diff Diff

Improvements in System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Double>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
XorBenchmark - Duration of single invocation 40.09 ns 8.81 ns 0.22 0.21 False
AllBitsSetBenchmark - Duration of single invocation 15.90 ns 3.03 ns 0.19 0.27 False
BitwiseAndBenchmark - Duration of single invocation 35.97 ns 8.67 ns 0.24 0.20 False
OnesComplementOperatorBenchmark - Duration of single invocation 28.50 ns 6.26 ns 0.22 0.16 False
BitwiseOrBenchmark - Duration of single invocation 35.74 ns 8.77 ns 0.25 0.22 False
ConditionalSelectBenchmark - Duration of single invocation 62.73 ns 11.77 ns 0.19 0.22 False
AndNotBenchmark - Duration of single invocation 37.20 ns 7.92 ns 0.21 0.21 False
ExclusiveOrOperatorBenchmark - Duration of single invocation 36.59 ns 6.03 ns 0.16 0.21 False
BitwiseAndOperatorBenchmark - Duration of single invocation 36.07 ns 8.72 ns 0.24 0.19 False
OnesComplementBenchmark - Duration of single invocation 28.31 ns 6.66 ns 0.24 0.24 False
BitwiseOrOperatorBenchmark - Duration of single invocation 36.07 ns 10.69 ns 0.30 0.21 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Double&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Double&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Double&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Double&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Double>.XorBenchmark

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.806897345335422 < 34.17978481796835.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 170.74391660633833 (T) = (0 -8.998692215724784) / Math.Sqrt((3.7556334365567383 / (299)) + (0.7452167965234812 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7618211306977322 = (37.78123660627817 - 8.998692215724784) / 37.78123660627817 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Double&gt;.AllBitsSetBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 3.0266539370653383 < 15.235732683819817.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 152.8726362650302 (T) = (0 -3.228258737903862) / Math.Sqrt((0.8990337089038725 / (299)) + (0.19687590605840122 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.8006757212454002 = (16.196013642062976 - 3.228258737903862) / 16.196013642062976 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Double&gt;.BitwiseAndBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.66801276874526 < 34.531346905639296.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 110.86895193715416 (T) = (0 -9.376268015291075) / Math.Sqrt((4.126842420587568 / (299)) + (2.4440735105435105 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7520596711966487 = (37.81663136668528 - 9.376268015291075) / 37.81663136668528 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Double&gt;.OnesComplementOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 6.262165013611879 < 26.945414953135074.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 187.84585409586177 (T) = (0 -6.83555294828316) / Math.Sqrt((1.2187826777473705 / (299)) + (0.48135929527526733 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7668056881171704 = (29.312691605092592 - 6.83555294828316) / 29.312691605092592 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Double&gt;.BitwiseOrBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.76504226467602 < 34.24879526139016.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 109.01925815888613 (T) = (0 -9.131359162045587) / Math.Sqrt((6.077160325358599 / (299)) + (2.303537614876157 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7586728712841103 = (37.83809640728698 - 9.131359162045587) / 37.83809640728698 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Double&gt;.ConditionalSelectBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 11.766587785640782 < 59.81595511978617.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 203.2146231710083 (T) = (0 -11.391962237238506) / Math.Sqrt((7.892048466670911 / (299)) + (2.026424209405873 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.8246556125790727 = (64.96907260505151 - 11.391962237238506) / 64.96907260505151 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Double&gt;.AndNotBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 7.919605493933186 < 35.384742204206084.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 223.89951912733073 (T) = (0 -8.827620796856147) / Math.Sqrt((4.802992564378657 / (299)) + (0.12876126238173038 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.776683980331304 = (39.529724781735325 - 8.827620796856147) / 39.529724781735325 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Double&gt;.ExclusiveOrOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 6.0336511228401974 < 34.568232580997815.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 154.15241433064975 (T) = (0 -8.921644401671879) / Math.Sqrt((4.491675014169847 / (299)) + (0.9410171986355762 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7638466440031505 = (37.77902864861636 - 8.921644401671879) / 37.77902864861636 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Double&gt;.BitwiseAndOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.719263751619627 < 34.285045060082965.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 81.31300973918447 (T) = (0 -9.333919430951047) / Math.Sqrt((3.771975738287527 / (299)) + (5.160602772557532 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7529639931750305 = (37.78363952249413 - 9.333919430951047) / 37.78363952249413 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Double&gt;.OnesComplementBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 6.662484949264388 < 26.941520551281258.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 206.5637108373367 (T) = (0 -6.631806219918679) / Math.Sqrt((2.5347939334223324 / (299)) + (0.17500744708410892 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7748004333616343 = (29.448574519542905 - 6.631806219918679) / 29.448574519542905 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Double&gt;.BitwiseOrOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 10.690640443444998 < 34.32385553179602.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 215.2401912388601 (T) = (0 -9.009192117775301) / Math.Sqrt((3.2235408260182994 / (299)) + (0.33108827664402707 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7613230231176248 = (37.746381052141494 - 9.009192117775301) / 37.746381052141494 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 8, 2023 •
edited
Loading

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 967250cc86b4c613015066b47b882e7424fdb590
Compare 4862343c61f698da986901a182e94df24da1487f
Diff Diff

Improvements in System.Tests.Perf_Array

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IndexOfShort - Duration of single invocation 239.57 ns 184.78 ns 0.77 0.05 False
IndexOfChar - Duration of single invocation 243.09 ns 188.00 ns 0.77 0.05 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Tests.Perf_Array*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Tests.Perf_Array*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Tests.Perf_Array*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Tests.Perf_Array*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Array.IndexOfShort

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 184.78330510783715 < 229.10921095517463.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 72.10289963514262 (T) = (0 -190.35131296415645) / Math.Sqrt((40.88257342116494 / (299)) + (17.44774697218027 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.21257826946009184 = (241.73997945629347 - 190.35131296415645) / 241.73997945629347 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Array.IndexOfChar

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 188.00255270884216 < 231.39931242675107.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 92.10398502332805 (T) = (0 -190.48998816418143) / Math.Sqrt((19.59829333769243 / (299)) + (11.447726016075025 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.2118678996798842 = (241.69804539976238 - 190.48998816418143) / 241.69804539976238 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 967250cc86b4c613015066b47b882e7424fdb590
Compare 4862343c61f698da986901a182e94df24da1487f
Diff Diff

Improvements in System.IO.Tests.StringReaderReadLineTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ReadLineAsync - Duration of single invocation 73.66 μs 43.77 μs 0.59 0.03 False
ReadLine - Duration of single invocation 76.62 μs 47.86 μs 0.62 0.00 False
ReadLine - Duration of single invocation 71.76 μs 41.52 μs 0.58 0.03 False
ReadLine - Duration of single invocation 108.35 μs 89.53 μs 0.83 0.03 False
ReadLineAsync - Duration of single invocation 80.04 μs 53.03 μs 0.66 0.03 False
ReadLine - Duration of single invocation 76.81 μs 47.39 μs 0.62 0.04 False
ReadLineAsync - Duration of single invocation 136.06 μs 115.77 μs 0.85 0.01 False
ReadLineAsync - Duration of single invocation 80.23 μs 51.00 μs 0.64 0.01 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.IO.Tests.StringReaderReadLineTests*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.IO.Tests.StringReaderReadLineTests*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.IO.Tests.StringReaderReadLineTests*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.IO.Tests.StringReaderReadLineTests*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.IO.Tests.StringReaderReadLineTests.ReadLineAsync(LineLengthRange: [1025, 2048])

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 43.768465674053566 < 70.05333935717069.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 287.2008843461783 (T) = (0 -44824.835527562434) / Math.Sqrt((660805.9113643807 / (299)) + (381323.4036538038 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.3943041238961678 = (74005.51546743284 - 44824.835527562434) / 74005.51546743284 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StringReaderReadLineTests.ReadLine(LineLengthRange: [   0, 1024])

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 47.8595865752176 < 72.66966331198064.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 335.7750316995909 (T) = (0 -49323.80232895544) / Math.Sqrt((319272.6708249155 / (299)) + (269369.8335389049 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.3595196139773292 = (77010.63671169088 - 49323.80232895544) / 77010.63671169088 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StringReaderReadLineTests.ReadLine(LineLengthRange: [1025, 2048])

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 41.52078732656601 < 68.25413542067318.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 301.4045204243101 (T) = (0 -43148.23094197754) / Math.Sqrt((525226.4781480654 / (299)) + (353161.8961833849 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.4021206771763711 = (72168.79610119254 - 43148.23094197754) / 72168.79610119254 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StringReaderReadLineTests.ReadLine(LineLengthRange: [  33,  128])

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 89.52733649876136 < 102.99639054578884.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 63.265829808163154 (T) = (0 -89324.25348589169) / Math.Sqrt((5498484.298241637 / (299)) + (3970454.266467076 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.18511265868660143 = (109615.46333745091 - 89324.25348589169) / 109615.46333745091 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StringReaderReadLineTests.ReadLineAsync(LineLengthRange: [   0, 1024])

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 53.02922486627253 < 77.23320964935824.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 151.9723575917076 (T) = (0 -53885.93299077466) / Math.Sqrt((7895012.192850175 / (299)) + (361063.3198354477 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.3424057714320242 = (81944.04793989832 - 53885.93299077466) / 81944.04793989832 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StringReaderReadLineTests.ReadLine(LineLengthRange: [ 129, 1024])

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 47.38555542813455 < 72.84888371005721.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 135.24418694545193 (T) = (0 -48608.996797262414) / Math.Sqrt((11552673.427251415 / (299)) + (287538.2899719739 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.3705176528313674 = (77220.58770337606 - 48608.996797262414) / 77220.58770337606 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StringReaderReadLineTests.ReadLineAsync(LineLengthRange: [  33,  128])

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 115.7662239351852 < 128.10332073980678.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 105.2722050421348 (T) = (0 -114736.14115645183) / Math.Sqrt((2644445.9573399667 / (299)) + (1380303.8051354163 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.15207964178309624 = (135314.7616336646 - 114736.14115645183) / 135314.7616336646 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StringReaderReadLineTests.ReadLineAsync(LineLengthRange: [ 129, 1024])

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 51.00476639485923 < 76.23007013472834.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 141.52575627709504 (T) = (0 -52231.762589233535) / Math.Sqrt((9965504.39052515 / (299)) + (349860.5247042452 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.35364092400407426 = (80809.20424727318 - 52231.762589233535) / 80809.20424727318 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 8, 2023 •
edited
Loading

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 967250cc86b4c613015066b47b882e7424fdb590
Compare 4862343c61f698da986901a182e94df24da1487f
Diff Diff

Improvements in System.Text.Tests.Perf_Encoding

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
GetChars - Duration of single invocation 407.37 ns 364.30 ns 0.89 0.13 False
GetBytes - Duration of single invocation 3.88 μs 2.86 μs 0.74 0.06 False
GetString - Duration of single invocation 307.93 ns 283.84 ns 0.92 0.14 False
GetBytes - Duration of single invocation 2.39 μs 1.53 μs 0.64 0.07 False
GetString - Duration of single invocation 2.56 μs 1.19 μs 0.46 0.09 False
GetString - Duration of single invocation 3.31 μs 1.92 μs 0.58 0.05 False
GetChars - Duration of single invocation 3.32 μs 1.93 μs 0.58 0.05 False
GetChars - Duration of single invocation 317.23 ns 283.59 ns 0.89 0.13 False
GetChars - Duration of single invocation 2.61 μs 1.21 μs 0.46 0.07 False
GetString - Duration of single invocation 284.89 ns 242.32 ns 0.85 0.18 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Tests.Perf_Encoding*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Tests.Perf_Encoding*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Tests.Perf_Encoding*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Tests.Perf_Encoding*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Text.Tests.Perf_Encoding.GetChars(size: 16, encName: "utf-8")

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 364.2978420666263 < 389.91585624195915.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 1/17/2023 8:36:05 PM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 37.489748771682805 (T) = (0 -362.5269538212993) / Math.Sqrt((387.15379477589755 / (299)) + (69.92562012298568 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.14712417463863292 = (425.06416882867455 - 362.5269538212993) / 425.06416882867455 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Tests.Perf_Encoding.GetBytes(size: 512, encName: "utf-8")

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 2.864852442937397 < 3.6552232139250553.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 106.88009325545053 (T) = (0 -2857.1811735192964) / Math.Sqrt((14180.878767430222 / (299)) + (1652.2735359008623 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.25369820606707555 = (3828.452774396107 - 2857.1811735192964) / 3828.452774396107 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Tests.Perf_Encoding.GetString(size: 16, encName: "ascii")

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 283.8420839942973 < 297.22882458374994.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 1/18/2023 1:01:18 AM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 35.471642601054974 (T) = (0 -274.5575660922445) / Math.Sqrt((148.1019687120525 / (299)) + (67.7167383586744 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.152375739145925 = (323.91423744242223 - 274.5575660922445) / 323.91423744242223 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Tests.Perf_Encoding.GetBytes(size: 512, encName: "ascii")

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.525812782251995 < 2.2746272521164697.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 103.70906054898951 (T) = (0 -1431.5648454137065) / Math.Sqrt((9637.917775373904 / (299)) + (2746.707749838725 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.4082278405852825 = (2419.11489521503 - 1431.5648454137065) / 2419.11489521503 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Tests.Perf_Encoding.GetString(size: 512, encName: "ascii")

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.1869490975188124 < 2.4406349130550034.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 200.6455498936162 (T) = (0 -1184.3095786962253) / Math.Sqrt((14196.257899501326 / (299)) + (54.53905188584946 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.5416142473530999 = (2583.6526808644353 - 1184.3095786962253) / 2583.6526808644353 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Tests.Perf_Encoding.GetString(size: 512, encName: "utf-8")

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.9245502312909217 < 3.16183995466697.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 212.7013573423509 (T) = (0 -1924.3967089690357) / Math.Sqrt((11163.23000690671 / (299)) + (252.98608693936663 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.4194163773036445 = (3314.5900672012112 - 1924.3967089690357) / 3314.5900672012112 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Tests.Perf_Encoding.GetChars(size: 512, encName: "utf-8")

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.932026344369901 < 3.160555654703656.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 242.61408796320538 (T) = (0 -1925.9881398713417) / Math.Sqrt((5379.0514978242045 / (299)) + (692.9875992035102 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.418847396837038 = (3314.0833051233394 - 1925.9881398713417) / 3314.0833051233394 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Tests.Perf_Encoding.GetChars(size: 16, encName: "ascii")

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 283.5885092844899 < 315.78098382635324.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 1/18/2023 1:01:18 AM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 44.03444384375594 (T) = (0 -287.28322679850993) / Math.Sqrt((126.12746253086304 / (299)) + (44.72197321451217 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.15227535448566637 = (338.88743039222214 - 287.28322679850993) / 338.88743039222214 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Tests.Perf_Encoding.GetChars(size: 512, encName: "ascii")

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.2072818654493653 < 2.4621023446994497.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 306.72192879522976 (T) = (0 -1200.1853493135498) / Math.Sqrt((5561.968028442671 / (299)) + (102.7894758980523 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.5381557042013257 = (2598.6795987986625 - 1200.1853493135498) / 2598.6795987986625 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Tests.Perf_Encoding.GetString(size: 16, encName: "utf-8")

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 242.31980694086948 < 275.85599257886724.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 1/17/2023 8:36:05 PM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 43.60289359343476 (T) = (0 -246.24090531681975) / Math.Sqrt((234.60720474928382 / (299)) + (38.484139025228366 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.18315631848163438 = (301.4541348463419 - 246.24090531681975) / 301.4541348463419 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 8, 2023 •
edited
Loading

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 967250cc86b4c613015066b47b882e7424fdb590
Compare 4862343c61f698da986901a182e94df24da1487f
Diff Diff

Improvements in Microsoft.Extensions.Configuration.Xml.XmlConfigurationProviderBenchmarks

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Load - Duration of single invocation 897.61 μs 845.30 μs 0.94 0.02 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'Microsoft.Extensions.Configuration.Xml.XmlConfigurationProviderBenchmarks*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'Microsoft.Extensions.Configuration.Xml.XmlConfigurationProviderBenchmarks*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'Microsoft.Extensions.Configuration.Xml.XmlConfigurationProviderBenchmarks*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'Microsoft.Extensions.Configuration.Xml.XmlConfigurationProviderBenchmarks*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

Microsoft.Extensions.Configuration.Xml.XmlConfigurationProviderBenchmarks.Load(FileName: "deep.xml")

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 845.3030096846848 < 854.5057149225462.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 1/3/2023 9:28:36 AM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 63.30824560571884 (T) = (0 -838747.2984198726) / Math.Sqrt((145895434.5131798 / (299)) + (35613498.97313317 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.07769725081223537 = (909405.6145430815 - 838747.2984198726) / 909405.6145430815 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 967250cc86b4c613015066b47b882e7424fdb590
Compare 4862343c61f698da986901a182e94df24da1487f
Diff Diff

Improvements in System.Buffers.Text.Tests.Base64Tests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ConvertToBase64CharArray - Duration of single invocation 19.03 μs 15.09 μs 0.79 0.14 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Buffers.Text.Tests.Base64Tests*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Buffers.Text.Tests.Base64Tests*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Buffers.Text.Tests.Base64Tests*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Buffers.Text.Tests.Base64Tests*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Buffers.Text.Tests.Base64Tests.ConvertToBase64CharArray(NumberOfBytes: 1000)

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 15.090652013120659 < 18.103465693896997.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 61.144176726713724 (T) = (0 -13344.345201823242) / Math.Sqrt((1003158.3341420329 / (299)) + (389973.1850483143 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.330929972057995 = (19944.616623866958 - 13344.345201823242) / 19944.616623866958 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 8, 2023 •
edited
Loading

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 967250cc86b4c613015066b47b882e7424fdb590
Compare 4862343c61f698da986901a182e94df24da1487f
Diff Diff

Improvements in System.Text.Json.Tests.Perf_Reader

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ReadSingleSpanSequenceEmptyLoop - Duration of single invocation 67.86 μs 63.82 μs 0.94 0.03 False
ReadSpanEmptyLoop - Duration of single invocation 652.81 μs 614.88 μs 0.94 0.01 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Tests.Perf_Reader*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Tests.Perf_Reader*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Tests.Perf_Reader*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Tests.Perf_Reader*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Text.Json.Tests.Perf_Reader.ReadSingleSpanSequenceEmptyLoop(IsDataCompact: True, TestCase: Json4KB)

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 63.82204181848853 < 64.07800593516595.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 30.839729968590525 (T) = (0 -63098.13735353749) / Math.Sqrt((1252824.6623174835 / (299)) + (678454.0984793165 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.06253188256863461 = (67306.96882409665 - 63098.13735353749) / 67306.96882409665 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Reader.ReadSpanEmptyLoop(IsDataCompact: True, TestCase: Json40KB)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 614.8803746794871 < 616.5996393157599.
IsChangePoint: Marked as a change because one of 2/28/2023 11:10:36 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 26.319497068246065 (T) = (0 -607570.9456933544) / Math.Sqrt((89423312.09173852 / (299)) + (59096821.131006114 / (46))) is greater than 1.9669042814877102 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (46) - 2, .975) and 0.05169823188695457 = (640693.6759195485 - 607570.9456933544) / 640693.6759195485 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 8, 2023 •
edited
Loading

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 967250cc86b4c613015066b47b882e7424fdb590
Compare 4862343c61f698da986901a182e94df24da1487f
Diff Diff

Improvements in System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Byte>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
GreaterThanBenchmark - Duration of single invocation 252.66 ns 8.89 ns 0.04 0.18 False
LessThanBenchmark - Duration of single invocation 275.47 ns 8.81 ns 0.03 0.15 False
EqualsAllBenchmark - Duration of single invocation 28.21 ns 7.75 ns 0.27 0.27 False
AddBenchmark - Duration of single invocation 240.10 ns 8.87 ns 0.04 0.18 False
BitwiseOrBenchmark - Duration of single invocation 36.55 ns 8.51 ns 0.23 0.20 False
BitwiseOrOperatorBenchmark - Duration of single invocation 36.00 ns 8.91 ns 0.25 0.20 False
AddOperatorBenchmark - Duration of single invocation 237.50 ns 12.26 ns 0.05 0.17 False
AllBitsSetBenchmark - Duration of single invocation 15.66 ns 4.17 ns 0.27 0.38 False
AndNotBenchmark - Duration of single invocation 37.20 ns 8.81 ns 0.24 0.21 False
UnaryNegateOperatorBenchmark - Duration of single invocation 234.10 ns 7.37 ns 0.03 0.18 False
ExclusiveOrOperatorBenchmark - Duration of single invocation 37.24 ns 8.61 ns 0.23 0.20 False
SubtractBenchmark - Duration of single invocation 245.98 ns 8.72 ns 0.04 0.18 False
EqualsBenchmark - Duration of single invocation 40.18 ns 8.53 ns 0.21 0.30 False
BitwiseAndBenchmark - Duration of single invocation 36.11 ns 8.72 ns 0.24 0.19 False
EqualsStaticBenchmark - Duration of single invocation 278.51 ns 8.63 ns 0.03 0.15 False
NegateBenchmark - Duration of single invocation 242.66 ns 6.76 ns 0.03 0.18 False
OnesComplementBenchmark - Duration of single invocation 28.26 ns 6.69 ns 0.24 0.19 False
EqualityOperatorBenchmark - Duration of single invocation 25.20 ns 8.10 ns 0.32 0.32 False
XorBenchmark - Duration of single invocation 36.28 ns 9.87 ns 0.27 0.17 False
MultiplyBenchmark - Duration of single invocation 240.63 ns 9.34 ns 0.04 0.18 False
SubtractionOperatorBenchmark - Duration of single invocation 243.51 ns 8.72 ns 0.04 0.18 False
ConditionalSelectBenchmark - Duration of single invocation 61.68 ns 10.68 ns 0.17 0.28 False
InequalityOperatorBenchmark - Duration of single invocation 25.54 ns 7.55 ns 0.30 0.32 False
BitwiseAndOperatorBenchmark - Duration of single invocation 36.71 ns 8.86 ns 0.24 0.19 False
OnesComplementOperatorBenchmark - Duration of single invocation 28.52 ns 6.65 ns 0.23 0.24 False
MultiplyOperatorBenchmark - Duration of single invocation 237.78 ns 8.95 ns 0.04 0.20 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Byte>.GreaterThanBenchmark

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.886415686152201 < 241.73266349246072.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 313.0752037595204 (T) = (0 -9.095728131258081) / Math.Sqrt((187.59937688305044 / (299)) + (0.8497266788775084 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.9651010154745319 = (260.6301660330638 - 9.095728131258081) / 260.6301660330638 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;.LessThanBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.807498036318604 < 261.6399562962643.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 288.02861944431453 (T) = (0 -9.295127531487502) / Math.Sqrt((257.49431021624036 / (299)) + (0.6665706681339032 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.9666575737824884 = (278.7777791229135 - 9.295127531487502) / 278.7777791229135 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;.EqualsAllBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 7.753314607376503 < 26.829719173478843.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 144.09141448365685 (T) = (0 -7.856067608216452) / Math.Sqrt((3.110935215146839 / (299)) + (0.4535616143865513 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.722022513051804 = (28.26152468124339 - 7.856067608216452) / 28.26152468124339 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;.AddBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.8737525429273 < 228.61881493638745.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 370.6459769519876 (T) = (0 -9.012258394055086) / Math.Sqrt((120.89637992141766 / (299)) + (0.3346407165055087 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.9634779971418937 = (246.76243603257834 - 9.012258394055086) / 246.76243603257834 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;.BitwiseOrBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.505359320484214 < 34.56907413325542.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 190.77282836549364 (T) = (0 -8.917193302721142) / Math.Sqrt((3.372953475796385 / (299)) + (0.5417983032158556 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7636487696361376 = (37.72856730634798 - 8.917193302721142) / 37.72856730634798 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;.BitwiseOrOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.912097457048171 < 34.30073769371919.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 190.86135771938098 (T) = (0 -9.004043668442865) / Math.Sqrt((4.884121423891685 / (299)) + (0.30376236721663896 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7619372165291103 = (37.82213892136516 - 9.004043668442865) / 37.82213892136516 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;.AddOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 12.26090415128169 < 226.4366787468875.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 394.9905205236259 (T) = (0 -8.950314672111684) / Math.Sqrt((101.89630960347196 / (299)) + (0.5801219605517146 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.9632692367640022 = (243.67352822497273 - 8.950314672111684) / 243.67352822497273 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;.AllBitsSetBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 4.1728478092388075 < 15.076721657526612.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 90.89747336817135 (T) = (0 -3.2941478165059417) / Math.Sqrt((2.9959478120666234 / (299)) + (0.49202581204717194 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7979675472290649 = (16.305042934071857 - 3.2941478165059417) / 16.305042934071857 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;.AndNotBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.806923084564671 < 35.59211063606353.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 202.22784720175073 (T) = (0 -8.871440353042727) / Math.Sqrt((5.636531743954494 / (299)) + (0.20014033762726643 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7760516921065668 = (39.61378604059042 - 8.871440353042727) / 39.61378604059042 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;.UnaryNegateOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 7.365811430239982 < 222.62474652708687.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 487.0696000452276 (T) = (0 -6.745304759382319) / Math.Sqrt((67.0016093844243 / (299)) + (0.2597284096325878 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.9719107685165324 = (240.13845887355 - 6.745304759382319) / 240.13845887355 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;.ExclusiveOrOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.612731834112092 < 34.36334538173873.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 158.76075042307795 (T) = (0 -9.02833216037671) / Math.Sqrt((3.4343846838964054 / (299)) + (1.0034325050842297 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7611352785439606 = (37.7968421010144 - 9.02833216037671) / 37.7968421010144 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;.SubtractBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.724540523341263 < 233.65449803893696.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 305.7616563123134 (T) = (0 -8.97734726551989) / Math.Sqrt((184.2097169710781 / (299)) + (0.818303541592198 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.9644237069631378 = (252.34071622408965 - 8.97734726551989) / 252.34071622408965 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;.EqualsBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.528295397900635 < 35.55109312200229.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 155.81108362425778 (T) = (0 -9.072902433243712) / Math.Sqrt((5.08235990896144 / (299)) + (0.8803363357581743 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7644885712655765 = (38.524255412992495 - 9.072902433243712) / 38.524255412992495 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;.BitwiseAndBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.7208561465317 < 34.32131138458514.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 104.88992077569294 (T) = (0 -9.163384541964792) / Math.Sqrt((4.208465604114138 / (299)) + (2.834881809431666 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7574024484526367 = (37.77195805777038 - 9.163384541964792) / 37.77195805777038 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;.EqualsStaticBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.626177128700153 < 261.6846711237475.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 720.5969088096498 (T) = (0 -8.938979508759985) / Math.Sqrt((39.125916554948574 / (299)) + (0.32940952381455874 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.967670669704445 = (276.4975156317742 - 8.938979508759985) / 276.4975156317742 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;.NegateBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 6.758325684943339 < 224.47089377959648.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 408.1624166658928 (T) = (0 -6.780411163606341) / Math.Sqrt((94.46469060411597 / (299)) + (0.8545186951877198 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.9720637133944618 = (242.7098225095594 - 6.780411163606341) / 242.7098225095594 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;.OnesComplementBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 6.69083675964694 < 26.949731564920057.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 186.50377174041512 (T) = (0 -6.722665729983458) / Math.Sqrt((1.672123231311517 / (299)) + (0.4288424641957679 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7709318251775545 = (29.347881848686786 - 6.722665729983458) / 29.347881848686786 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;.EqualityOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.100031602859499 < 24.014620591633616.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 132.43256255499352 (T) = (0 -8.096004584200925) / Math.Sqrt((2.692919243936955 / (299)) + (0.4210547722492891 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.6867650755194065 = (25.846430111922324 - 8.096004584200925) / 25.846430111922324 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;.XorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 9.871744710106576 < 34.46387900636742.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 152.4635684029545 (T) = (0 -8.81877163668849) / Math.Sqrt((6.052494254544822 / (299)) + (0.7523847146556539 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7669902829820456 = (37.84722692920556 - 8.81877163668849) / 37.84722692920556 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;.MultiplyBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 9.342339110791393 < 228.6795012341388.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 387.9285458142737 (T) = (0 -9.522716938237116) / Math.Sqrt((106.35284182613537 / (299)) + (0.8606535212716879 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.9614097278383928 = (246.76470013888942 - 9.522716938237116) / 246.76470013888942 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;.SubtractionOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.724971364449042 < 231.23448336696043.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 442.9168899967426 (T) = (0 -8.850099765931244) / Math.Sqrt((85.75512211565669 / (299)) + (0.2950834908121508 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.9644050211463008 = (248.63337613730613 - 8.850099765931244) / 248.63337613730613 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;.ConditionalSelectBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 10.683561455388238 < 59.31447151155998.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 231.85704669425493 (T) = (0 -11.137565110038915) / Math.Sqrt((9.474311938816843 / (299)) + (1.0700484597793498 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.8292887213551499 = (65.24211638769131 - 11.137565110038915) / 65.24211638769131 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;.InequalityOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 7.554497530729926 < 24.42648441586822.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 104.85352286644634 (T) = (0 -8.111891708291722) / Math.Sqrt((2.678590364254196 / (299)) + (0.8991329482882859 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.6841790823378499 = (25.685099544196216 - 8.111891708291722) / 25.685099544196216 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;.BitwiseAndOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.863835117059901 < 34.30726876434844.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 91.06980805764863 (T) = (0 -9.287642164046536) / Math.Sqrt((3.0606349559857455 / (299)) + (4.093896153874164 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7536493148229039 = (37.70089844633415 - 9.287642164046536) / 37.70089844633415 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;.OnesComplementOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 6.648880250496778 < 27.05215370602452.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 176.23249146386266 (T) = (0 -6.752416006821936) / Math.Sqrt((2.6273880340320956 / (299)) + (0.3682177769207883 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7708964149687815 = (29.473200979818046 - 6.752416006821936) / 29.473200979818046 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;.MultiplyOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.949630021707135 < 225.91962611816012.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 373.57400232772636 (T) = (0 -9.454507361546623) / Math.Sqrt((112.77448879562974 / (299)) + (0.9204646049262613 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.9613729021716518 = (244.76359584560973 - 9.454507361546623) / 244.76359584560973 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 8, 2023 •
edited
Loading

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 967250cc86b4c613015066b47b882e7424fdb590
Compare 4862343c61f698da986901a182e94df24da1487f
Diff Diff

Improvements in System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt64>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
InequalityOperatorBenchmark - Duration of single invocation 25.15 ns 7.89 ns 0.31 0.30 False
AndNotBenchmark - Duration of single invocation 37.40 ns 8.64 ns 0.23 0.22 False
BitwiseAndOperatorBenchmark - Duration of single invocation 36.00 ns 8.69 ns 0.24 0.23 False
EqualsBenchmark - Duration of single invocation 36.59 ns 8.57 ns 0.23 0.31 False
AllBitsSetBenchmark - Duration of single invocation 16.14 ns 2.93 ns 0.18 0.33 False
ConditionalSelectBenchmark - Duration of single invocation 62.79 ns 12.11 ns 0.19 0.27 False
EqualityOperatorBenchmark - Duration of single invocation 25.44 ns 7.88 ns 0.31 0.26 False
OnesComplementOperatorBenchmark - Duration of single invocation 28.07 ns 6.65 ns 0.24 0.21 False
ExclusiveOrOperatorBenchmark - Duration of single invocation 35.94 ns 8.55 ns 0.24 0.19 False
OnesComplementBenchmark - Duration of single invocation 28.66 ns 7.45 ns 0.26 0.24 False
EqualsAllBenchmark - Duration of single invocation 27.96 ns 9.26 ns 0.33 0.27 False
BitwiseOrOperatorBenchmark - Duration of single invocation 35.91 ns 8.54 ns 0.24 0.18 False
EqualsStaticBenchmark - Duration of single invocation 85.62 ns 8.96 ns 0.10 0.11 False
BitwiseAndBenchmark - Duration of single invocation 35.98 ns 8.78 ns 0.24 0.19 False
BitwiseOrBenchmark - Duration of single invocation 35.82 ns 8.68 ns 0.24 0.19 False
XorBenchmark - Duration of single invocation 37.42 ns 12.74 ns 0.34 0.24 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt64&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt64&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt64&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt64&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt64>.InequalityOperatorBenchmark

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 7.891226920042877 < 23.90157603661138.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 93.82833765701544 (T) = (0 -8.067063834262617) / Math.Sqrt((4.144459477093348 / (299)) + (1.095606855285705 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.6915926414164016 = (26.15717041030304 - 8.067063834262617) / 26.15717041030304 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt64&gt;.AndNotBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.641377642859315 < 35.416769046515604.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 170.40557261760256 (T) = (0 -8.91467806871228) / Math.Sqrt((6.447331247288148 / (299)) + (0.5216927861912862 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7755158579942748 = (39.71183883663784 - 8.91467806871228) / 39.71183883663784 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt64&gt;.BitwiseAndOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.68632957443726 < 34.181438142196356.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 79.40341227679846 (T) = (0 -9.506841556447124) / Math.Sqrt((8.176503934091496 / (299)) + (4.735979027511237 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7493414660605257 = (37.927460146809565 - 9.506841556447124) / 37.927460146809565 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt64&gt;.EqualsBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.571799902548333 < 33.94145935363976.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 167.06261485069706 (T) = (0 -8.895210583217063) / Math.Sqrt((4.606904471859779 / (299)) + (0.6835198701324299 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7647237163721221 = (37.80751058311544 - 8.895210583217063) / 37.80751058311544 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt64&gt;.AllBitsSetBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 2.9255995327200566 < 15.812506307750644.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 128.46075290158777 (T) = (0 -3.2557156062089923) / Math.Sqrt((0.971810224812252 / (299)) + (0.3258220961933733 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7992595920551253 = (16.218536365149987 - 3.2557156062089923) / 16.218536365149987 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt64&gt;.ConditionalSelectBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 12.107729472038683 < 59.443092986940876.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 221.58598771112122 (T) = (0 -11.387834765268574) / Math.Sqrt((12.273834338585937 / (299)) + (0.8663727159939257 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.8259559956446949 = (65.43077888521043 - 11.387834765268574) / 65.43077888521043 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt64&gt;.EqualityOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 7.8787773949713555 < 23.925110987240434.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 185.15658645656015 (T) = (0 -8.051497582301172) / Math.Sqrt((1.4183126584310475 / (299)) + (0.21484287654180542 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.6893880876254629 = (25.921406300066963 - 8.051497582301172) / 25.921406300066963 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt64&gt;.OnesComplementOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 6.649080032769484 < 26.997666075532262.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 167.71981330340293 (T) = (0 -6.87580960084104) / Math.Sqrt((2.8197438093626395 / (299)) + (0.4131198396829586 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.767040992155803 = (29.515105101407293 - 6.87580960084104) / 29.515105101407293 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt64&gt;.ExclusiveOrOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.54855156419269 < 34.11918664889446.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 176.09774749419338 (T) = (0 -8.886743537896962) / Math.Sqrt((3.564922553109036 / (299)) + (0.7025353939608476 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.764608379152858 = (37.75301561676153 - 8.886743537896962) / 37.75301561676153 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt64&gt;.OnesComplementBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 7.453590181091289 < 26.94215095951999.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 141.61292352493302 (T) = (0 -7.016962654258896) / Math.Sqrt((2.2737704900023297 / (299)) + (0.8086483230461657 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7606979982781972 = (29.32262414760896 - 7.016962654258896) / 29.32262414760896 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt64&gt;.EqualsAllBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 9.264119730691709 < 26.493428160472135.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 142.695182942077 (T) = (0 -7.863830243266002) / Math.Sqrt((3.6084223218266174 / (299)) + (0.4325986242739616 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7257703149521747 = (28.676072183412817 - 7.863830243266002) / 28.676072183412817 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt64&gt;.BitwiseOrOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.536064235367352 < 34.59782851859598.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 255.4912724884806 (T) = (0 -8.77377744709378) / Math.Sqrt((2.751399565710791 / (299)) + (0.17066981800689615 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7673778753930005 = (37.716865762087444 - 8.77377744709378) / 37.716865762087444 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt64&gt;.EqualsStaticBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.9598664375019 < 80.42139545570075.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 464.8327903936993 (T) = (0 -9.01338761052483) / Math.Sqrt((7.437078842607591 / (299)) + (0.1411353356774145 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.8959462776704354 = (86.62244279908744 - 9.01338761052483) / 86.62244279908744 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt64&gt;.BitwiseAndBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.779583440034646 < 34.32052348764887.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 219.2072107103281 (T) = (0 -8.939658019606137) / Math.Sqrt((3.022048123892905 / (299)) + (0.33460931052680254 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7629403104565208 = (37.7105784489206 - 8.939658019606137) / 37.7105784489206 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt64&gt;.BitwiseOrBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.675098614192093 < 34.42759488440918.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 201.12395244993758 (T) = (0 -8.946061694735564) / Math.Sqrt((3.6226396942062666 / (299)) + (0.39586003519302365 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7631411563380508 = (37.769591189525535 - 8.946061694735564) / 37.769591189525535 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt64&gt;.XorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 12.743778071825067 < 34.61990206913103.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 163.1236745686641 (T) = (0 -9.040483644859426) / Math.Sqrt((4.257238021883466 / (299)) + (0.8001650742994614 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.76135754823298 = (37.88296498766028 - 9.040483644859426) / 37.88296498766028 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 8, 2023 •
edited
Loading

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 967250cc86b4c613015066b47b882e7424fdb590
Compare 4862343c61f698da986901a182e94df24da1487f
Diff Diff

Improvements in System.Text.Json.Serialization.Tests.ReadJson<IndexViewModel>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DeserializeFromUtf8Bytes - Duration of single invocation 606.66 μs 554.42 μs 0.91 0.01 False
DeserializeFromString - Duration of single invocation 654.08 μs 588.86 μs 0.90 0.01 False
DeserializeFromUtf8Bytes - Duration of single invocation 584.62 μs 542.13 μs 0.93 0.01 False
DeserializeFromStream - Duration of single invocation 653.38 μs 610.46 μs 0.93 0.01 False
DeserializeFromReader - Duration of single invocation 764.53 μs 666.90 μs 0.87 0.01 False
DeserializeFromStream - Duration of single invocation 642.28 μs 590.51 μs 0.92 0.01 False
DeserializeFromReader - Duration of single invocation 752.97 μs 693.77 μs 0.92 0.01 False
DeserializeFromString - Duration of single invocation 638.10 μs 574.82 μs 0.90 0.01 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;IndexViewModel&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;IndexViewModel&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;IndexViewModel&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;IndexViewModel&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.ReadJson<IndexViewModel>.DeserializeFromUtf8Bytes(Mode: SourceGen)

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 554.4157552083333 < 570.0262101422145.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 2/2/2023 5:46:53 AM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 65.78873897824286 (T) = (0 -554203.8828639194) / Math.Sqrt((206859884.20033526 / (299)) + (16434804.165366247 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.108056803976666 = (621344.3696132202 - 554203.8828639194) / 621344.3696132202 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;IndexViewModel&gt;.DeserializeFromString(Mode: SourceGen)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 588.855755837912 < 623.0440098951702.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 2/2/2023 8:18:56 AM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 81.99077277202394 (T) = (0 -590022.3749030374) / Math.Sqrt((208403799.4652715 / (299)) + (23621948.626546122 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.13209501423352935 = (679823.6956571606 - 590022.3749030374) / 679823.6956571606 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;IndexViewModel&gt;.DeserializeFromUtf8Bytes(Mode: Reflection)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 542.1259298273156 < 560.0568235081586.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 2/1/2023 11:20:33 PM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 64.29661847015262 (T) = (0 -542156.9319117729) / Math.Sqrt((163837362.57245943 / (299)) + (16769830.884932404 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.10137006507512965 = (603315.0141577465 - 542156.9319117729) / 603315.0141577465 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;IndexViewModel&gt;.DeserializeFromStream(Mode: SourceGen)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 610.4589967776585 < 615.231947172668.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 2/1/2023 11:20:33 PM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 55.67592588324488 (T) = (0 -608875.9630545186) / Math.Sqrt((282502737.550123 / (299)) + (23600847.504004907 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.09909413097228305 = (675848.5919418359 - 608875.9630545186) / 675848.5919418359 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;IndexViewModel&gt;.DeserializeFromReader(Mode: Reflection)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 666.9021277915632 < 712.5925982917942.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 2/2/2023 8:11:01 PM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 80.42957781601518 (T) = (0 -669011.927193274) / Math.Sqrt((174903838.31486726 / (299)) + (36737174.64444246 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.1232231870011671 = (763035.6064105502 - 669011.927193274) / 763035.6064105502 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;IndexViewModel&gt;.DeserializeFromStream(Mode: Reflection)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 590.5066934279313 < 605.4773094970589.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 2/3/2023 3:35:07 PM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 60.10556186263484 (T) = (0 -596478.0426898755) / Math.Sqrt((260691506.94816512 / (299)) + (15833784.968637917 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.09973795032862096 = (662560.4654862512 - 596478.0426898755) / 662560.4654862512 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;IndexViewModel&gt;.DeserializeFromReader(Mode: SourceGen)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 693.7713854166666 < 717.7344376136538.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 2/7/2023 5:28:13 PM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 81.02554129079931 (T) = (0 -683716.4521167912) / Math.Sqrt((216708236.8139028 / (299)) + (28630395.261932638 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.12039312960625959 = (777297.7623636996 - 683716.4521167912) / 777297.7623636996 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;IndexViewModel&gt;.DeserializeFromString(Mode: Reflection)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 574.8216774054983 < 610.542881427205.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 2/11/2023 12:14:55 PM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 86.89116944588379 (T) = (0 -576784.5627515749) / Math.Sqrt((165858105.01889023 / (299)) + (19906827.438682403 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.12967866946324907 = (662725.9869591569 - 576784.5627515749) / 662725.9869591569 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 967250cc86b4c613015066b47b882e7424fdb590
Compare 4862343c61f698da986901a182e94df24da1487f
Diff Diff

Improvements in System.Text.Json.Serialization.Tests.ReadJson<BinaryData>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DeserializeFromUtf8Bytes - Duration of single invocation 23.00 μs 17.81 μs 0.77 0.02 False
DeserializeFromStream - Duration of single invocation 32.60 μs 27.51 μs 0.84 0.02 False
DeserializeFromUtf8Bytes - Duration of single invocation 23.02 μs 17.93 μs 0.78 0.02 False
DeserializeFromReader - Duration of single invocation 33.66 μs 22.62 μs 0.67 0.02 False
DeserializeFromString - Duration of single invocation 31.45 μs 23.22 μs 0.74 0.02 False
DeserializeFromStream - Duration of single invocation 32.57 μs 27.41 μs 0.84 0.00 False
DeserializeFromString - Duration of single invocation 31.02 μs 23.23 μs 0.75 0.02 False
DeserializeFromReader - Duration of single invocation 33.65 μs 22.45 μs 0.67 0.03 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;BinaryData&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;BinaryData&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;BinaryData&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;BinaryData&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.ReadJson<BinaryData>.DeserializeFromUtf8Bytes(Mode: Reflection)

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 17.80623377129434 < 21.831679533388847.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 267.5771954163873 (T) = (0 -17726.191476649055) / Math.Sqrt((73321.36793367498 / (299)) + (5327.765235242719 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.22229941307150639 = (22793.079720639205 - 17726.191476649055) / 22793.079720639205 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;BinaryData&gt;.DeserializeFromStream(Mode: SourceGen)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 27.514270944779256 < 30.965936416391227.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 170.12358764133685 (T) = (0 -27625.29193714509) / Math.Sqrt((138347.73413917454 / (299)) + (21506.510137679343 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.15741098594994052 = (32786.20000557452 - 27625.29193714509) / 32786.20000557452 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;BinaryData&gt;.DeserializeFromUtf8Bytes(Mode: SourceGen)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 17.92887727099962 < 21.865718317795228.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 187.7090832438317 (T) = (0 -17875.45650660359) / Math.Sqrt((120324.09624626968 / (299)) + (16603.468393202034 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.2240051346825162 = (23035.534519020537 - 17875.45650660359) / 23035.534519020537 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;BinaryData&gt;.DeserializeFromReader(Mode: SourceGen)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 22.61902348209966 < 31.980898747231446.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 342.9982670285555 (T) = (0 -22750.682329118114) / Math.Sqrt((195709.45614187143 / (299)) + (14036.689912672735 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.3176239352377233 = (33340.38736696297 - 22750.682329118114) / 33340.38736696297 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;BinaryData&gt;.DeserializeFromString(Mode: Reflection)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 23.219511222222224 < 29.456005606892397.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 194.1599812572451 (T) = (0 -23440.215237732977) / Math.Sqrt((191819.7859920937 / (299)) + (39048.32329316167 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.2411806729719047 = (30890.377198925908 - 23440.215237732977) / 30890.377198925908 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;BinaryData&gt;.DeserializeFromStream(Mode: Reflection)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 27.411271500500376 < 30.96283319371955.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 198.66430281599577 (T) = (0 -27483.33279299569) / Math.Sqrt((98659.32589716956 / (299)) + (15282.559146519345 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.15613053377761713 = (32568.227543563087 - 27483.33279299569) / 32568.227543563087 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;BinaryData&gt;.DeserializeFromString(Mode: SourceGen)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 23.23004360915057 < 29.475044931031388.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 178.67251032205724 (T) = (0 -23529.60193758022) / Math.Sqrt((138835.923725073 / (299)) + (63029.76484233656 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.24394096872578677 = (31121.382014212493 - 23529.60193758022) / 31121.382014212493 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;BinaryData&gt;.DeserializeFromReader(Mode: Reflection)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 22.45096805925092 < 31.749678247500103.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 241.3469792211369 (T) = (0 -22702.15271310852) / Math.Sqrt((174255.5836292384 / (299)) + (60218.92957587582 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.314594440473916 = (33122.21851367192 - 22702.15271310852) / 33122.21851367192 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 8, 2023 •
edited
Loading

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 967250cc86b4c613015066b47b882e7424fdb590
Compare 4862343c61f698da986901a182e94df24da1487f
Diff Diff

Improvements in System.Runtime.Intrinsics.Tests.Perf_Vector128Of<SByte>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
NegateBenchmark - Duration of single invocation 235.97 ns 6.59 ns 0.03 0.18 False
SubtractionOperatorBenchmark - Duration of single invocation 243.13 ns 9.48 ns 0.04 0.18 False
AddOperatorBenchmark - Duration of single invocation 237.79 ns 8.64 ns 0.04 0.15 False
SubtractBenchmark - Duration of single invocation 246.85 ns 8.69 ns 0.04 0.16 False
ConditionalSelectBenchmark - Duration of single invocation 63.66 ns 10.65 ns 0.17 0.23 False
BitwiseOrBenchmark - Duration of single invocation 38.16 ns 10.41 ns 0.27 0.18 False
BitwiseOrOperatorBenchmark - Duration of single invocation 37.37 ns 9.21 ns 0.25 0.26 False
EqualsBenchmark - Duration of single invocation 35.64 ns 8.75 ns 0.25 0.26 False
XorBenchmark - Duration of single invocation 35.83 ns 8.64 ns 0.24 0.20 False
AllBitsSetBenchmark - Duration of single invocation 16.15 ns 3.21 ns 0.20 0.25 False
ExclusiveOrOperatorBenchmark - Duration of single invocation 36.04 ns 9.01 ns 0.25 0.20 False
MultiplyBenchmark - Duration of single invocation 240.91 ns 9.13 ns 0.04 0.21 False
MultiplyOperatorBenchmark - Duration of single invocation 238.03 ns 9.48 ns 0.04 0.17 False
AddBenchmark - Duration of single invocation 240.35 ns 8.51 ns 0.04 0.20 False
EqualsStaticBenchmark - Duration of single invocation 275.34 ns 8.76 ns 0.03 0.08 False
BitwiseAndOperatorBenchmark - Duration of single invocation 36.05 ns 9.01 ns 0.25 0.21 False
UnaryNegateOperatorBenchmark - Duration of single invocation 234.14 ns 6.49 ns 0.03 0.20 False
OnesComplementBenchmark - Duration of single invocation 28.45 ns 12.01 ns 0.42 0.21 False
OnesComplementOperatorBenchmark - Duration of single invocation 28.33 ns 6.65 ns 0.23 0.23 False
LessThanBenchmark - Duration of single invocation 274.82 ns 8.71 ns 0.03 0.10 False
InequalityOperatorBenchmark - Duration of single invocation 24.78 ns 7.87 ns 0.32 0.26 False
EqualsAllBenchmark - Duration of single invocation 29.46 ns 9.02 ns 0.31 0.26 False
BitwiseAndBenchmark - Duration of single invocation 36.06 ns 8.57 ns 0.24 0.20 False
EqualityOperatorBenchmark - Duration of single invocation 28.96 ns 7.85 ns 0.27 0.25 False
AndNotBenchmark - Duration of single invocation 37.18 ns 8.88 ns 0.24 0.23 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<SByte>.NegateBenchmark

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 6.591924315532935 < 224.39078147741034.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 449.8707407426031 (T) = (0 -6.894941373081491) / Math.Sqrt((77.05013078298379 / (299)) + (0.7661734910443324 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.9715528674241555 = (242.37737686561096 - 6.894941373081491) / 242.37737686561096 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;.SubtractionOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 9.475664303673431 < 233.03579615470701.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 445.41754110627454 (T) = (0 -8.982886197294578) / Math.Sqrt((84.99285430320755 / (299)) + (0.32015403403542647 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.9639660018660168 = (249.2891897228282 - 8.982886197294578) / 249.2891897228282 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;.AddOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.635840821654005 < 225.92188570770787.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 456.85843242081063 (T) = (0 -8.939955349841487) / Math.Sqrt((72.77063606538745 / (299)) + (0.9713671347804901 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.9633156083602108 = (243.69915787685844 - 8.939955349841487) / 243.69915787685844 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;.SubtractBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.689851099487843 < 233.91222470076073.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 322.8405300622228 (T) = (0 -8.974773648919824) / Math.Sqrt((168.69360692009622 / (299)) + (0.191810093719957 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.9644344620087635 = (252.34466159716865 - 8.974773648919824) / 252.34466159716865 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;.ConditionalSelectBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 10.649256890534986 < 60.01950825265551.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 242.75370989748288 (T) = (0 -11.405449207240991) / Math.Sqrt((7.463674327579791 / (299)) + (1.118781100058586 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.8245660925611761 = (65.01279811725234 - 11.405449207240991) / 65.01279811725234 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;.BitwiseOrBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 10.409633228782923 < 34.56449776657328.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 201.74603019184582 (T) = (0 -8.886123542066251) / Math.Sqrt((3.2295080152550315 / (299)) + (0.45646076505839445 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7647984549031073 = (37.780889315185235 - 8.886123542066251) / 37.780889315185235 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;.BitwiseOrOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 9.208472918505278 < 34.24078091410877.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 154.87687256020857 (T) = (0 -9.226107587405806) / Math.Sqrt((4.124219910087517 / (299)) + (0.950843066155058 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7558849988890762 = (37.79410337512828 - 9.226107587405806) / 37.79410337512828 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;.EqualsBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.747722961728721 < 33.93418841471176.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 198.56372204356353 (T) = (0 -8.903754979451499) / Math.Sqrt((3.163887752437145 / (299)) + (0.5266260387051407 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7669916615114273 = (38.21217316601808 - 8.903754979451499) / 38.21217316601808 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;.XorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.64377316235654 < 34.33512343379216.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 149.25589101176197 (T) = (0 -9.097404077425216) / Math.Sqrt((3.938113516549247 / (299)) + (1.1152585029053952 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7591269978591771 = (37.76846718631653 - 9.097404077425216) / 37.76846718631653 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;.AllBitsSetBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 3.209462567840714 < 15.226864139567622.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 124.54527315861166 (T) = (0 -3.219921945336531) / Math.Sqrt((1.4248631500329327 / (299)) + (0.2881604898794298 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.8014935737415437 = (16.220744114068015 - 3.219921945336531) / 16.220744114068015 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;.ExclusiveOrOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 9.008408541974465 < 34.22870768807167.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 178.94564874063403 (T) = (0 -8.870747565538503) / Math.Sqrt((6.420917644182899 / (299)) + (0.2218347659963394 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7655264010910714 = (37.83260719678708 - 8.870747565538503) / 37.83260719678708 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;.MultiplyBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 9.12942602021986 < 228.9871409467724.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 275.12742793118474 (T) = (0 -9.376982705858115) / Math.Sqrt((220.85416971397055 / (299)) + (0.680006267688995 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.962210608955509 = (248.13796800319489 - 9.376982705858115) / 248.13796800319489 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;.MultiplyOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 9.475764334135269 < 226.69112666173564.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 420.5594686452783 (T) = (0 -9.299977524745369) / Math.Sqrt((89.57941277960141 / (299)) + (0.5845468976243581 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.961920368377147 = (244.22446143528637 - 9.299977524745369) / 244.22446143528637 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;.AddBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.50619260820759 < 228.34377302212297.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 346.38150032892065 (T) = (0 -9.062125505879845) / Math.Sqrt((137.54966126524678 / (299)) + (0.7346785373455286 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.9634526637812365 = (247.95584147736946 - 9.062125505879845) / 247.95584147736946 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;.EqualsStaticBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.761850177256946 < 261.07451553470514.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 951.4393533593425 (T) = (0 -8.96738645243608) / Math.Sqrt((21.316702679860214 / (299)) + (0.3449467604607093 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.967481819675202 = (275.76532151762603 - 8.96738645243608) / 275.76532151762603 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;.BitwiseAndOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 9.013069759249719 < 34.20181761675132.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 99.74451881866001 (T) = (0 -9.392304507095993) / Math.Sqrt((7.486418048848096 / (299)) + (2.68379257927177 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7526986457849709 = (37.97918752571554 - 9.392304507095993) / 37.97918752571554 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;.UnaryNegateOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 6.4916711724283775 < 222.3862435899195.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 382.8510704222935 (T) = (0 -6.691276917886118) / Math.Sqrt((110.65242104502998 / (299)) + (0.2007489927836799 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.9722277906715748 = (240.93426773351854 - 6.691276917886118) / 240.93426773351854 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;.OnesComplementBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 12.007236719208237 < 26.979941623943642.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 131.49619853327937 (T) = (0 -6.9021170268683285) / Math.Sqrt((2.6582750434524214 / (299)) + (0.9617390526098911 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7654815678022513 = (29.431021528612227 - 6.9021170268683285) / 29.431021528612227 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;.OnesComplementOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 6.651564916349569 < 26.851872897741462.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 164.46071630727747 (T) = (0 -6.819892341313243) / Math.Sqrt((2.244200568589308 / (299)) + (0.5356042550922834 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7682705910499573 = (29.430413568195416 - 6.819892341313243) / 29.430413568195416 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;.LessThanBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.713067411706422 < 261.1298100696717.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 923.0068398421392 (T) = (0 -8.867910608890595) / Math.Sqrt((22.33476224146466 / (299)) + (0.45099234664830845 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.967968317094907 = (276.84810177365523 - 8.867910608890595) / 276.84810177365523 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;.InequalityOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 7.865829319429854 < 23.669753827167987.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 134.6233809894562 (T) = (0 -7.736705833554574) / Math.Sqrt((2.534585219637345 / (299)) + (0.4229728050653383 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.6969994861546764 = (25.53363931753603 - 7.736705833554574) / 25.53363931753603 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;.EqualsAllBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 9.018947747036686 < 26.54754100665515.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 112.59438538596342 (T) = (0 -8.0631390912437) / Math.Sqrt((3.5677597618224177 / (299)) + (0.957359929760039 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7150768829762858 = (28.299350279017926 - 8.0631390912437) / 28.299350279017926 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;.BitwiseAndBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.56659549075211 < 34.41588068700438.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 189.84559863139958 (T) = (0 -8.972019554136114) / Math.Sqrt((4.373492304420029 / (299)) + (0.4013267926794147 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7630662073294259 = (37.8672010142958 - 8.972019554136114) / 37.8672010142958 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;.EqualityOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 7.848688529148193 < 23.753460342867445.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 133.8278623141484 (T) = (0 -8.05080824294739) / Math.Sqrt((2.922896808406536 / (299)) + (0.3572726170854103 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.686645159001863 = (25.69230530252205 - 8.05080824294739) / 25.69230530252205 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;.AndNotBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.880675798312536 < 35.462271652300295.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 124.26234338651105 (T) = (0 -9.168187631596709) / Math.Sqrt((4.5200530545915285 / (299)) + (2.1056301269017004 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7683940369191644 = (39.58528316646497 - 9.168187631596709) / 39.58528316646497 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 8, 2023 •
edited
Loading

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 967250cc86b4c613015066b47b882e7424fdb590
Compare 4862343c61f698da986901a182e94df24da1487f
Diff Diff

Improvements in System.Memory.Span<Char>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IndexOfValue - Duration of single invocation 691.87 ns 494.40 ns 0.71 0.02 False
Reverse - Duration of single invocation 1.84 μs 944.25 ns 0.51 0.05 False
IndexOfAnyFiveValues - Duration of single invocation 2.68 μs 945.21 ns 0.35 0.03 False
LastIndexOfAnyValues - Duration of single invocation 1.08 μs 645.42 ns 0.60 0.03 False
IndexOfAnyThreeValues - Duration of single invocation 1.37 μs 710.77 ns 0.52 0.04 False
IndexOfAnyFourValues - Duration of single invocation 1.66 μs 852.50 ns 0.51 0.02 False
IndexOfAnyTwoValues - Duration of single invocation 1.01 μs 607.73 ns 0.60 0.04 False
IndexOfAnyFiveValues - Duration of single invocation 231.23 ns 175.26 ns 0.76 0.08 False
LastIndexOfValue - Duration of single invocation 691.76 ns 530.21 ns 0.77 0.01 False
Reverse - Duration of single invocation 159.52 ns 94.23 ns 0.59 0.13 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Memory.Span&lt;Char&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Memory.Span&lt;Char&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Memory.Span&lt;Char&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Memory.Span&lt;Char&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Memory.Span<Char>.IndexOfValue(Size: 512)

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 494.40475107677736 < 658.284543098203.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 129.51465458102794 (T) = (0 -510.51291286246203) / Math.Sqrt((78.28091546965685 / (299)) + (82.1812906927632 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.2645469084584719 = (694.1474836857565 - 510.51291286246203) / 694.1474836857565 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Memory.Span&lt;Char&gt;.Reverse(Size: 512)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 944.2516235649731 < 1.7474754771780847.
IsChangePoint: Marked as a change because one of 1/31/2023 6:07:33 PM, 2/1/2023 10:12:25 AM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 48.08639558795331 (T) = (0 -929.9725132080786) / Math.Sqrt((125345.40557152085 / (299)) + (39.42166703299626 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.5145049542616142 = (1915.5139097118704 - 929.9725132080786) / 1915.5139097118704 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Memory.Span&lt;Char&gt;.IndexOfAnyFiveValues(Size: 512)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 945.2060642951468 < 2.5446000625143506.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 356.22466831102827 (T) = (0 -1004.6505664098928) / Math.Sqrt((2218.5597130240117 / (299)) + (588.4078250283958 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.6128983011347621 = (2595.314278792774 - 1004.6505664098928) / 2595.314278792774 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Memory.Span&lt;Char&gt;.LastIndexOfAnyValues(Size: 512)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 645.4236272385601 < 0.9534101490966226.
IsChangePoint: Marked as a change because one of 12/18/2022 12:43:31 PM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 222.51406104547962 (T) = (0 -662.0428578474548) / Math.Sqrt((111.5533213216096 / (299)) + (96.63468444772046 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.34376282885248777 = (1008.8469336319224 - 662.0428578474548) / 1008.8469336319224 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Memory.Span&lt;Char&gt;.IndexOfAnyThreeValues(Size: 512)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 710.7719552619379 < 1.299972553080855.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 137.65387416130426 (T) = (0 -745.6539435781824) / Math.Sqrt((4427.043657824084 / (299)) + (242.7479667430864 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.45205322930140396 = (1360.814559829457 - 745.6539435781824) / 1360.814559829457 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Memory.Span&lt;Char&gt;.IndexOfAnyFourValues(Size: 512)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 852.5005677031829 < 1.5748317741325293.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 192.02401050642516 (T) = (0 -888.830224544576) / Math.Sqrt((2646.9638738423837 / (299)) + (338.52063811785 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.4639957572048079 = (1658.2522181343986 - 888.830224544576) / 1658.2522181343986 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Memory.Span&lt;Char&gt;.IndexOfAnyTwoValues(Size: 512)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 607.7257311455389 < 0.95637295444644.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 53.504136250258796 (T) = (0 -634.9435343065211) / Math.Sqrt((1915.9204577477485 / (299)) + (2077.1763138466927 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.37477829835014 = (1015.5494165205826 - 634.9435343065211) / 1015.5494165205826 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Memory.Span&lt;Char&gt;.IndexOfAnyFiveValues(Size: 33)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 175.2552860307532 < 220.24621775816485.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 63.48503254944896 (T) = (0 -182.86094693877328) / Math.Sqrt((33.49537989518332 / (299)) + (20.930708422370262 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.20583852807176584 = (230.2566334460731 - 182.86094693877328) / 230.2566334460731 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Memory.Span&lt;Char&gt;.LastIndexOfValue(Size: 512)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 530.2121856930975 < 657.1570518433584.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 56.23625893425619 (T) = (0 -550.1470788836331) / Math.Sqrt((33.909207739917534 / (299)) + (285.2575399226027 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.20266105896757983 = (689.9789419180818 - 550.1470788836331) / 689.9789419180818 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Memory.Span&lt;Char&gt;.Reverse(Size: 33)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 94.22756587404402 < 151.65032953179607.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 82.46054012375409 (T) = (0 -94.50225246502819) / Math.Sqrt((169.6049288710472 / (299)) + (7.069382946478296 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.4250226049034863 = (164.35820481110457 - 94.50225246502819) / 164.35820481110457 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 967250cc86b4c613015066b47b882e7424fdb590
Compare 4862343c61f698da986901a182e94df24da1487f
Diff Diff

Improvements in System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Single>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
BitwiseAndOperatorBenchmark - Duration of single invocation 35.89 ns 8.48 ns 0.24 0.22 False
OnesComplementOperatorBenchmark - Duration of single invocation 28.34 ns 6.66 ns 0.23 0.23 False
BitwiseOrOperatorBenchmark - Duration of single invocation 36.01 ns 8.44 ns 0.23 0.25 False
AllBitsSetBenchmark - Duration of single invocation 15.91 ns 3.07 ns 0.19 0.27 False
OnesComplementBenchmark - Duration of single invocation 28.41 ns 6.77 ns 0.24 0.26 False
BitwiseOrBenchmark - Duration of single invocation 36.27 ns 8.61 ns 0.24 0.19 False
BitwiseAndBenchmark - Duration of single invocation 36.79 ns 9.03 ns 0.25 0.24 False
XorBenchmark - Duration of single invocation 36.12 ns 11.82 ns 0.33 0.19 False
ConditionalSelectBenchmark - Duration of single invocation 63.08 ns 13.76 ns 0.22 0.23 False
ExclusiveOrOperatorBenchmark - Duration of single invocation 36.18 ns 9.76 ns 0.27 0.19 False
AndNotBenchmark - Duration of single invocation 37.27 ns 8.74 ns 0.23 0.17 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Single&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Single&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Single&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Single&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Single>.BitwiseAndOperatorBenchmark

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.476027612874503 < 34.16219059567296.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 171.27928052432787 (T) = (0 -8.983027372202093) / Math.Sqrt((5.901964577187283 / (299)) + (0.41517835075997983 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7632001155221934 = (37.935100314814555 - 8.983027372202093) / 37.935100314814555 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Single&gt;.OnesComplementOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 6.656150562994131 < 27.010543425005356.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 140.97648728351103 (T) = (0 -6.81781628148111) / Math.Sqrt((2.59761659479133 / (299)) + (0.8024136357190351 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7684525058205576 = (29.444569485159303 - 6.81781628148111) / 29.444569485159303 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Single&gt;.BitwiseOrOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.440728339791383 < 34.40114106049834.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 165.95849261920793 (T) = (0 -9.022554835933319) / Math.Sqrt((3.639681962779723 / (299)) + (0.8450154222674189 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7615609566000857 = (37.840089891656405 - 9.022554835933319) / 37.840089891656405 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Single&gt;.AllBitsSetBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 3.074826570552227 < 15.149665125632263.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 114.86276786037584 (T) = (0 -3.1758007379275854) / Math.Sqrt((0.7407698005033057 / (299)) + (0.4839126449387631 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.8034482768177988 = (16.15758278030284 - 3.1758007379275854) / 16.15758278030284 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Single&gt;.OnesComplementBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 6.769323512260965 < 27.07930742577824.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 150.35030597242698 (T) = (0 -6.8246440836468985) / Math.Sqrt((2.176822016159415 / (299)) + (0.7185644793871676 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7679617297228122 = (29.41171762526211 - 6.8246440836468985) / 29.41171762526211 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Single&gt;.BitwiseOrBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.611998614206442 < 34.333836576570704.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 184.26900611953374 (T) = (0 -8.931489977440478) / Math.Sqrt((3.4386604448749445 / (299)) + (0.6046188349485532 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7630555066201367 = (37.694439951055294 - 8.931489977440478) / 37.694439951055294 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Single&gt;.BitwiseAndBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 9.031782774554717 < 34.1827821635467.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 156.80694704852735 (T) = (0 -8.870154096828697) / Math.Sqrt((5.795315166177529 / (299)) + (0.7040978735472576 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7661929285648045 = (37.937920535851916 - 8.870154096828697) / 37.937920535851916 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Single&gt;.XorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 11.817608073519542 < 34.255590514083174.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 164.9936830534114 (T) = (0 -8.970383421525518) / Math.Sqrt((2.851624681805479 / (299)) + (0.9754031451186596 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7619712956720923 = (37.686141454469045 - 8.970383421525518) / 37.686141454469045 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Single&gt;.ConditionalSelectBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 13.760496588843205 < 59.81922490058548.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 262.7444954906595 (T) = (0 -11.361584256277263) / Math.Sqrt((7.9913583083550215 / (299)) + (0.7028194230809296 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.8252145110854125 = (65.00301785252513 - 11.361584256277263) / 65.00301785252513 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Single&gt;.ExclusiveOrOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 9.756563834859485 < 34.623155570188324.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 158.49203916277963 (T) = (0 -9.241245271628753) / Math.Sqrt((2.521678292550913 / (299)) + (1.1103570993781005 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7543462660723863 = (37.618989639912634 - 9.241245271628753) / 37.618989639912634 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Single&gt;.AndNotBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.735604385835753 < 35.456698880402364.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 214.9771989905959 (T) = (0 -8.753212946101476) / Math.Sqrt((5.155087516237896 / (299)) + (0.15246868990643295 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7785234928609355 = (39.522083218539095 - 8.753212946101476) / 39.522083218539095 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 8, 2023 •
edited
Loading

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 967250cc86b4c613015066b47b882e7424fdb590
Compare 4862343c61f698da986901a182e94df24da1487f
Diff Diff

Improvements in BenchmarksGame.ReverseComplement_1

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
RunBench - Duration of single invocation 4.14 ms 3.69 ms 0.89 0.07 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'BenchmarksGame.ReverseComplement_1*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'BenchmarksGame.ReverseComplement_1*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'BenchmarksGame.ReverseComplement_1*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'BenchmarksGame.ReverseComplement_1*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

BenchmarksGame.ReverseComplement_1.RunBench

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 3.6853996125 < 3.9307948217843633.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 11.101838437155703 (T) = (0 -3727715.1426980793) / Math.Sqrt((14934023906.936743 / (299)) + (62901330118.0972 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.09988237257128686 = (4141364.449607231 - 3727715.1426980793) / 4141364.449607231 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 967250cc86b4c613015066b47b882e7424fdb590
Compare 4862343c61f698da986901a182e94df24da1487f
Diff Diff

Improvements in System.Text.Json.Serialization.Tests.ReadJson<Dictionary<String, String>>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DeserializeFromReader - Duration of single invocation 339.23 μs 298.95 μs 0.88 0.02 False
DeserializeFromString - Duration of single invocation 291.14 μs 257.36 μs 0.88 0.01 False
DeserializeFromString - Duration of single invocation 283.78 μs 256.94 μs 0.91 0.01 False
DeserializeFromUtf8Bytes - Duration of single invocation 255.63 μs 230.33 μs 0.90 0.01 False
DeserializeFromReader - Duration of single invocation 337.37 μs 301.66 μs 0.89 0.01 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;Dictionary&lt;String, String&gt;&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;Dictionary&lt;String, String&gt;&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;Dictionary&lt;String, String&gt;&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;Dictionary&lt;String, String&gt;&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.ReadJson<Dictionary<String, String>>.DeserializeFromReader(Mode: SourceGen)

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 298.94936815130626 < 321.8008767285047.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 1/26/2023 10:17:57 AM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 43.30167948340022 (T) = (0 -301399.5010215779) / Math.Sqrt((95099652.87540607 / (299)) + (24843609.976510156 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.116759503696637 = (341242.84640823066 - 301399.5010215779) / 341242.84640823066 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;Dictionary&lt;String, String&gt;&gt;.DeserializeFromString(Mode: Reflection)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 257.3558549521858 < 275.58282468686775.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 1/17/2023 8:36:05 PM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 44.58339334449532 (T) = (0 -256678.8424752073) / Math.Sqrt((119935035.33626953 / (299)) + (11710841.941839345 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.12285853928431952 = (292631.06804434594 - 256678.8424752073) / 292631.06804434594 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;Dictionary&lt;String, String&gt;&gt;.DeserializeFromString(Mode: SourceGen)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 256.93817476478495 < 271.51790216892476.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 1/18/2023 9:06:44 AM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 45.506989297869 (T) = (0 -258200.7455934707) / Math.Sqrt((108329874.84488796 / (299)) + (9607125.005148225 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.11713744628529396 = (292458.5990277569 - 258200.7455934707) / 292458.5990277569 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;Dictionary&lt;String, String&gt;&gt;.DeserializeFromUtf8Bytes(Mode: Reflection)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 230.3333199142157 < 242.15663161058274.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 1/26/2023 2:33:00 AM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 37.328647575549056 (T) = (0 -229822.80691419562) / Math.Sqrt((111418548.7705761 / (299)) + (7839221.031024672 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.10657919333840696 = (257239.1477796052 - 229822.80691419562) / 257239.1477796052 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;Dictionary&lt;String, String&gt;&gt;.DeserializeFromReader(Mode: Reflection)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 301.6554925 < 320.4449230025769.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 1/25/2023 9:58:39 PM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 44.384927163758476 (T) = (0 -302470.3456712861) / Math.Sqrt((106407258.60808101 / (299)) + (17142593.370327324 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.11076903613897705 = (340148.23815622204 - 302470.3456712861) / 340148.23815622204 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 967250cc86b4c613015066b47b882e7424fdb590
Compare 4862343c61f698da986901a182e94df24da1487f
Diff Diff

Improvements in System.Memory.ReadOnlySpan

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IndexOfString - Duration of single invocation 1.38 μs 900.70 ns 0.65 0.02 False
IndexOfString - Duration of single invocation 360.85 ns 272.79 ns 0.76 0.05 False
IndexOfString - Duration of single invocation 1.38 μs 912.58 ns 0.66 0.02 False
IndexOfString - Duration of single invocation 545.13 ns 379.50 ns 0.70 0.02 False
IndexOfString - Duration of single invocation 190.05 ns 146.67 ns 0.77 0.11 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Memory.ReadOnlySpan*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Memory.ReadOnlySpan*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Memory.ReadOnlySpan*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Memory.ReadOnlySpan*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Memory.ReadOnlySpan.IndexOfString(input: "だだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだxだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだ", value: "x", comparisonType: Ordinal)

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 900.7006587434535 < 1.3152482470981681.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 151.35137982511102 (T) = (0 -945.1043652289956) / Math.Sqrt((490.9905409554288 / (299)) + (323.3293593696365 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.31855996081018445 = (1386.9222688362404 - 945.1043652289956) / 1386.9222688362404 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Memory.ReadOnlySpan.IndexOfString(input: "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAXAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", value: "x", comparisonType: OrdinalIgnoreCase)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 272.7885402693448 < 342.97556152847596.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 98.59194853966538 (T) = (0 -279.4215794669192) / Math.Sqrt((35.43923662891401 / (299)) + (29.84290709785411 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.2344669788941684 = (365.00264751909424 - 279.4215794669192) / 365.00264751909424 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Memory.ReadOnlySpan.IndexOfString(input: "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAXAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", value: "X", comparisonType: Ordinal)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 912.5798100346897 < 1.315103449751061.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 150.75510480233382 (T) = (0 -943.6412769385194) / Math.Sqrt((648.0857589502195 / (299)) + (307.86068269513 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.32051425941252726 = (1388.7580276852636 - 943.6412769385194) / 1388.7580276852636 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Memory.ReadOnlySpan.IndexOfString(input: "Hello WorldbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbareallyreallylongHello WorldbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbareallyreallylongHello Worldbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbareallyreallylong!xyz", value: "~", comparisonType: Ordinal)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 379.5031228077943 < 517.5614605826554.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 130.75717155801416 (T) = (0 -393.64805889253734) / Math.Sqrt((135.19416090900444 / (299)) + (44.0048834018755 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.28129863878534855 = (547.721320893628 - 393.64805889253734) / 547.721320893628 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Memory.ReadOnlySpan.IndexOfString(input: "Hello Worldbbbbbbbbbbbbbbcbbbbbbbbbbbbbbbbbbba!", value: "y", comparisonType: Ordinal)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 146.67304008960676 < 178.6430670367232.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 59.27349416269295 (T) = (0 -150.9780249919507) / Math.Sqrt((20.0802315929006 / (299)) + (16.871021932352125 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.203997587951232 = (189.67031092702373 - 150.9780249919507) / 189.67031092702373 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 8, 2023 •
edited
Loading

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 967250cc86b4c613015066b47b882e7424fdb590
Compare 4862343c61f698da986901a182e94df24da1487f
Diff Diff

Improvements in System.Tests.Perf_Guid

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
NotEqualsOperator - Duration of single invocation 24.66 ns 8.64 ns 0.35 0.24 False
EqualsOperator - Duration of single invocation 35.99 ns 7.81 ns 0.22 0.25 False
EqualsSame - Duration of single invocation 20.89 ns 8.89 ns 0.43 0.23 False
EqualsNotSame - Duration of single invocation 11.73 ns 8.00 ns 0.68 0.32 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Tests.Perf_Guid*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Tests.Perf_Guid*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Tests.Perf_Guid*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Tests.Perf_Guid*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Guid.NotEqualsOperator

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.641459451063062 < 23.54696013549441.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 100.33434067998685 (T) = (0 -8.789199823708408) / Math.Sqrt((2.625087851710822 / (299)) + (1.0141885879399766 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.6654411430983439 = (26.271012237144276 - 8.789199823708408) / 26.271012237144276 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Guid.EqualsOperator

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 7.814550781627221 < 22.97035442377263.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 107.63202909946574 (T) = (0 -8.073023978275685) / Math.Sqrt((2.208158547172943 / (299)) + (0.9028255071487861 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.6849596177170995 = (25.625362436953424 - 8.073023978275685) / 25.625362436953424 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Guid.EqualsSame

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.89026144587285 < 19.99873892520693.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 111.78030735340886 (T) = (0 -9.154444678385351) / Math.Sqrt((1.9268129101730085 / (299)) + (0.2982699134718239 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.5799977305482777 = (21.79617907859334 - 9.154444678385351) / 21.79617907859334 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Guid.EqualsNotSame

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.000846121585294 < 11.128692698017957.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 30.477164024623217 (T) = (0 -8.32788724208809) / Math.Sqrt((1.1217467316700587 / (299)) + (0.5243783688585729 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.3088419154120495 = (12.049178657951963 - 8.32788724208809) / 12.049178657951963 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 967250cc86b4c613015066b47b882e7424fdb590
Compare 4862343c61f698da986901a182e94df24da1487f
Diff Diff

Improvements in System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int64>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
InequalityOperatorBenchmark - Duration of single invocation 25.03 ns 7.85 ns 0.31 0.26 False
XorBenchmark - Duration of single invocation 36.10 ns 8.54 ns 0.24 0.18 False
AndNotBenchmark - Duration of single invocation 37.63 ns 8.68 ns 0.23 0.19 False
ConditionalSelectBenchmark - Duration of single invocation 62.86 ns 10.91 ns 0.17 0.22 False
BitwiseAndOperatorBenchmark - Duration of single invocation 36.51 ns 8.77 ns 0.24 0.21 False
AllBitsSetBenchmark - Duration of single invocation 16.07 ns 3.60 ns 0.22 0.32 False
EqualsStaticBenchmark - Duration of single invocation 84.36 ns 9.12 ns 0.11 0.18 False
ExclusiveOrOperatorBenchmark - Duration of single invocation 38.30 ns 9.50 ns 0.25 0.21 False
EqualityOperatorBenchmark - Duration of single invocation 25.72 ns 7.59 ns 0.30 0.29 False
BitwiseAndBenchmark - Duration of single invocation 36.09 ns 8.68 ns 0.24 0.22 False
BitwiseOrOperatorBenchmark - Duration of single invocation 36.08 ns 8.88 ns 0.25 0.21 False
BitwiseOrBenchmark - Duration of single invocation 37.17 ns 8.44 ns 0.23 0.23 False
OnesComplementOperatorBenchmark - Duration of single invocation 28.40 ns 6.60 ns 0.23 0.25 False
EqualsAllBenchmark - Duration of single invocation 28.09 ns 7.83 ns 0.28 0.22 False
EqualsBenchmark - Duration of single invocation 35.17 ns 8.68 ns 0.25 0.27 False
OnesComplementBenchmark - Duration of single invocation 28.26 ns 8.95 ns 0.32 0.23 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int64&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int64&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int64&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int64&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int64>.InequalityOperatorBenchmark

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 7.852237705792581 < 23.805801687320074.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 78.66959282913508 (T) = (0 -8.183116998529789) / Math.Sqrt((2.007541329828193 / (299)) + (2.0336453807819654 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.6824707601161458 = (25.771223467555338 - 8.183116998529789) / 25.771223467555338 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int64&gt;.XorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.539964797708649 < 34.26563333887335.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 136.1721154761469 (T) = (0 -9.299389301902472) / Math.Sqrt((3.268728093125888 / (299)) + (1.5425152277496954 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7538696508446493 = (37.782375614447105 - 9.299389301902472) / 37.782375614447105 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int64&gt;.AndNotBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.67879744535974 < 35.57963981074787.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 175.15092555619486 (T) = (0 -9.004381987327502) / Math.Sqrt((6.044781615556668 / (299)) + (0.47900656038192574 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7723127627449173 = (39.547152909759745 - 9.004381987327502) / 39.547152909759745 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int64&gt;.ConditionalSelectBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 10.912940177259546 < 59.73089411251661.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 249.65539946589737 (T) = (0 -11.340704992308314) / Math.Sqrt((7.941122099636922 / (299)) + (0.9160765511801382 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.8252988194849498 = (64.91487326458754 - 11.340704992308314) / 64.91487326458754 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int64&gt;.BitwiseAndOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.76711218002897 < 34.26356582712371.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 211.6348579817789 (T) = (0 -8.900045669125005) / Math.Sqrt((3.9674747539064765 / (299)) + (0.25486297575900874 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7647625281659297 = (37.83430250178356 - 8.900045669125005) / 37.83430250178356 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int64&gt;.AllBitsSetBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 3.6035709134480474 < 15.14950745011291.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 128.20263835139687 (T) = (0 -3.1173692031921627) / Math.Sqrt((1.4922930670187184 / (299)) + (0.26385047399199085 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.8089802649207138 = (16.319618503806648 - 3.1173692031921627) / 16.319618503806648 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int64&gt;.EqualsStaticBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 9.116612548891279 < 80.38754237577608.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 387.84149469860193 (T) = (0 -9.250584281399325) / Math.Sqrt((8.116898153331856 / (299)) + (0.5996408913949551 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.8933365532200481 = (86.72684561266249 - 9.250584281399325) / 86.72684561266249 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int64&gt;.ExclusiveOrOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 9.502220662807835 < 34.35192187728156.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 157.9792919187117 (T) = (0 -8.975065461534966) / Math.Sqrt((5.991535797072426 / (299)) + (0.6323916073674852 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7631115925263297 = (37.88731393507522 - 8.975065461534966) / 37.88731393507522 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int64&gt;.EqualityOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 7.589761108792566 < 24.257136036275806.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 137.46270618125766 (T) = (0 -8.063886947624729) / Math.Sqrt((1.972277372595416 / (299)) + (0.4890426299397578 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.6897021928767045 = (25.987573107213684 - 8.063886947624729) / 25.987573107213684 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int64&gt;.BitwiseAndBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.678284416924638 < 34.30756906345435.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 152.87773490881065 (T) = (0 -9.051240829009787) / Math.Sqrt((4.412118774571901 / (299)) + (0.9776825512729578 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7610494414089961 = (37.87913651418662 - 9.051240829009787) / 37.87913651418662 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int64&gt;.BitwiseOrOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.876357479833944 < 34.21870695084353.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 134.39849882239426 (T) = (0 -9.001987563138544) / Math.Sqrt((4.817603162246529 / (299)) + (1.4013317277906459 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7618816118877727 = (37.80467201422439 - 9.001987563138544) / 37.80467201422439 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int64&gt;.BitwiseOrBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.438458519454556 < 34.609641073110446.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 152.48772953044053 (T) = (0 -9.050270487608113) / Math.Sqrt((4.166842278084306 / (299)) + (1.0258192472361665 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7611242273687672 = (37.88693339604419 - 9.050270487608113) / 37.88693339604419 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int64&gt;.OnesComplementOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 6.604319350356632 < 26.97618685397867.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 166.2640416039228 (T) = (0 -6.747085870350611) / Math.Sqrt((4.3532287508404215 / (299)) + (0.2029673385850061 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.771989757640295 = (29.591152575095826 - 6.747085870350611) / 29.591152575095826 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int64&gt;.EqualsAllBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 7.82788670658453 < 26.625566763651236.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 192.07788534938717 (T) = (0 -8.1796454584899) / Math.Sqrt((1.4620707511772206 / (299)) + (0.29446050875741775 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7126558051422048 = (28.46636753019478 - 8.1796454584899) / 28.46636753019478 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int64&gt;.EqualsBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.676223241608426 < 33.60515584616541.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 197.21014272348245 (T) = (0 -8.884962719010792) / Math.Sqrt((4.1644248576355825 / (299)) + (0.34620750610716194 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7640915290372617 = (37.66275404503882 - 8.884962719010792) / 37.66275404503882 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int64&gt;.OnesComplementBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.952482253992297 < 26.998602011254302.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 128.9262828121672 (T) = (0 -6.9104964275254055) / Math.Sqrt((3.0502647956303277 / (299)) + (0.9606682133551238 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7655758330967061 = (29.47860077231783 - 6.9104964275254055) / 29.47860077231783 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 967250cc86b4c613015066b47b882e7424fdb590
Compare 4862343c61f698da986901a182e94df24da1487f
Diff Diff

Improvements in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Count - Duration of single invocation 1.97 ms 1.41 ms 0.71 0.02 False
Count - Duration of single invocation 2.56 ms 1.96 ms 0.77 0.03 False
Count - Duration of single invocation 3.90 ms 2.93 ms 0.75 0.01 False
Count - Duration of single invocation 6.70 ms 5.64 ms 0.84 0.02 False
Count - Duration of single invocation 3.58 ms 1.79 ms 0.50 0.07 False
Count - Duration of single invocation 3.46 ms 2.53 ms 0.73 0.03 False
Count - Duration of single invocation 1.96 ms 1.40 ms 0.72 0.03 False
Count - Duration of single invocation 14.54 ms 13.56 ms 0.93 0.02 False
Count - Duration of single invocation 3.58 ms 1.79 ms 0.50 0.08 False
Count - Duration of single invocation 14.37 ms 13.56 ms 0.94 0.01 False
Count - Duration of single invocation 3.59 ms 1.88 ms 0.52 0.07 False
Count - Duration of single invocation 4.68 ms 3.65 ms 0.78 0.00 False
Count - Duration of single invocation 9.92 ms 8.42 ms 0.85 0.03 False
Count - Duration of single invocation 2.05 ms 1.51 ms 0.74 0.02 False
Count - Duration of single invocation 13.71 ms 12.32 ms 0.90 0.02 False
Count - Duration of single invocation 3.79 ms 2.37 ms 0.63 0.03 False
Count - Duration of single invocation 3.48 ms 2.50 ms 0.72 0.01 False
Count - Duration of single invocation 3.73 ms 2.35 ms 0.63 0.03 False
Count - Duration of single invocation 2.05 ms 1.48 ms 0.72 0.03 False
Count - Duration of single invocation 10.12 ms 8.82 ms 0.87 0.05 False
Count - Duration of single invocation 3.73 ms 2.35 ms 0.63 0.02 False
Count - Duration of single invocation 5.53 ms 4.51 ms 0.82 0.01 False
Count - Duration of single invocation 6.97 ms 5.43 ms 0.78 0.01 False
Count - Duration of single invocation 3.44 ms 2.52 ms 0.73 0.01 False
Count - Duration of single invocation 3.80 ms 2.35 ms 0.62 0.03 False
Count - Duration of single invocation 5.13 ms 4.04 ms 0.79 0.01 False
Count - Duration of single invocation 3.64 ms 1.88 ms 0.52 0.07 False
Count - Duration of single invocation 2.03 ms 1.47 ms 0.72 0.01 False
Count - Duration of single invocation 3.96 ms 2.95 ms 0.74 0.02 False
Count - Duration of single invocation 4.56 ms 3.16 ms 0.69 0.01 False
Count - Duration of single invocation 3.79 ms 2.36 ms 0.62 0.02 False
Count - Duration of single invocation 4.56 ms 3.04 ms 0.67 0.01 False
Count - Duration of single invocation 3.78 ms 2.36 ms 0.63 0.03 False
Count - Duration of single invocation 3.63 ms 1.88 ms 0.52 0.06 False
Count - Duration of single invocation 3.58 ms 1.82 ms 0.51 0.08 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock|Street", Options: Compiled)

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.4087395314393942 < 1.8799742586691086.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 77.51734045637865 (T) = (0 -1477958.2604465531) / Math.Sqrt((231008066.13505882 / (299)) + (1988053562.5829618 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.2560724739712342 = (1986696.5648336613 - 1477958.2604465531) / 1986696.5648336613 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock|Street", Options: None)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.9555516473214287 < 2.402923109585174.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 91.89758193227344 (T) = (0 -2018717.9816776451) / Math.Sqrt((2949015523.193262 / (299)) + (1209620580.332394 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.21359723819657053 = (2567028.092638168 - 2018717.9816776451) / 2567028.092638168 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sher[a-z]+|Hol[a-z]+", Options: NonBacktracking)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 2.932246395475113 < 3.7072854058182614.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 189.32438997651266 (T) = (0 -2944732.946271209) / Math.Sqrt((2561719678.142884 / (299)) + (787086025.5803849 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.24441401065244828 = (3897283.68152245 - 2944732.946271209) / 3897283.68152245 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Holmes.{0,25}Watson|Watson.{0,25}Holmes", Options: None)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 5.642458407467532 < 6.399488916457528.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 76.97899499164579 (T) = (0 -5775348.240824399) / Math.Sqrt((15768754932.345877 / (299)) + (6561263124.858036 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.15601404439914165 = (6842943.5377425905 - 5775348.240824399) / 6842943.5377425905 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "aqj", Options: Compiled)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.785992029647436 < 3.4333156868538324.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 216.50110515845572 (T) = (0 -1850082.540987339) / Math.Sqrt((5881473337.429925 / (299)) + (2399582257.052878 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.4960032484703515 = (3670822.352271659 - 1850082.540987339) / 3670822.352271659 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Holmes.{0,25}Watson|Watson.{0,25}Holmes", Options: NonBacktracking)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 2.5268141855670105 < 3.3577184404205593.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 127.90189762275959 (T) = (0 -2585475.464249352) / Math.Sqrt((9623143307.67988 / (299)) + (933296693.4119487 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.2630109216675399 = (3508159.8089612788 - 2585475.464249352) / 3508159.8089612788 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock|Street", Options: NonBacktracking)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.3989158006566602 < 1.856154673911345.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 94.7296869898425 (T) = (0 -1449332.715055429) / Math.Sqrt((258697144.51002645 / (299)) + (1305354279.386428 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.2591383969142073 = (1956279.970535326 - 1449332.715055429) / 1956279.970535326 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "the\\s+\\w+", Options: Compiled)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 13.556435603174602 < 13.86031592552521.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 47.71724923738815 (T) = (0 -13416896.418368002) / Math.Sqrt((64119849945.14911 / (299)) + (36483551107.790955 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.10067249870644314 = (14918810.33224234 - 13416896.418368002) / 14918810.33224234 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "aqj", Options: None)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.7853827337962964 < 3.414053940363145.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 101.1660819638133 (T) = (0 -1862429.4123775128) / Math.Sqrt((5944002888.815278 / (299)) + (14001540700.688015 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.4919540929440235 = (3665868.352665048 - 1862429.4123775128) / 3665868.352665048 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "the\\s+\\w+", Options: None)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 13.561778214285715 < 13.772323904646493.
IsChangePoint: Marked as a change because one of 2/15/2023 6:58:43 PM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 53.25021074275123 (T) = (0 -13417432.908724759) / Math.Sqrt((64098808101.18054 / (299)) + (25795033868.02451 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.09880769563777339 = (14888534.71537384 - 13417432.908724759) / 14888534.71537384 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "aei", Options: Compiled)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.876213102678571 < 3.411961922075904.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 189.09905822107856 (T) = (0 -1955509.6855651683) / Math.Sqrt((5320902056.522639 / (299)) + (3007211545.896086 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.4665187727426404 = (3665564.1954235067 - 1955509.6855651683) / 3665564.1954235067 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock|Holmes", Options: None)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 3.6459659176470587 < 4.446853461073372.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 124.82543943928384 (T) = (0 -3742413.7486046515) / Math.Sqrt((5058622834.203989 / (299)) + (2305991638.6506085 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.21317744268908193 = (4756363.06284215 - 3742413.7486046515) / 4756363.06284215 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "the", Options: None)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.415977954907163 < 9.385772476372221.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 52.06400284798205 (T) = (0 -8464614.714174533) / Math.Sqrt((29857364504.876606 / (299)) + (39073954088.71917 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.158034028612125 = (10053392.894515298 - 8464614.714174533) / 10053392.894515298 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "[a-q][^u-z]{13}x", Options: Compiled)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.5106605931818184 < 1.951251217056025.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 123.77953980358 (T) = (0 -1531856.9593443554) / Math.Sqrt((717478533.2804744 / (299)) + (716304155.2179444 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.25338372590268454 = (2051732.6135121053 - 1531856.9593443554) / 2051732.6135121053 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "the", Options: NonBacktracking)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 12.317519746666669 < 12.907030024722221.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 43.08953179732376 (T) = (0 -12329286.149269167) / Math.Sqrt((56072982355.67306 / (299)) + (41293112628.35812 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.1024250038435575 = (13736218.368454015 - 12329286.149269167) / 13736218.368454015 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?i)Sherlock Holmes", Options: Compiled)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 2.3724282427672954 < 3.5955246921538464.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 153.80258315834214 (T) = (0 -2451656.641797642) / Math.Sqrt((12178022951.502064 / (299)) + (1529158418.8741279 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.34936954562512373 = (3768124.632519986 - 2451656.641797642) / 3768124.632519986 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock|Holmes", Options: NonBacktracking)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 2.4983929676767676 < 3.3301145308333333.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 213.83471002246083 (T) = (0 -2535188.045916279) / Math.Sqrt((1577910403.4141743 / (299)) + (703107780.1473202 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.2750673743748287 = (3497136.0872742757 - 2535188.045916279) / 3497136.0872742757 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?i)Sherlock", Options: NonBacktracking)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 2.346364483647799 < 3.549786416652486.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 166.1378481617326 (T) = (0 -2395028.8137654276) / Math.Sqrt((12716964905.178 / (299)) + (973526691.4938877 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.35552860314181045 = (3716268.5969326785 - 2395028.8137654276) / 3716268.5969326785 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "[a-q][^u-z]{13}x", Options: None)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.479417006439394 < 1.9501839726678376.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 105.15250585415583 (T) = (0 -1532624.3130568392) / Math.Sqrt((215533223.26672873 / (299)) + (1109826487.485321 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.25286613215857057 = (2051338.292941796 - 1532624.3130568392) / 2051338.292941796 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "the", Options: Compiled)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.819711130357142 < 9.45252907263889.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 74.24031575164405 (T) = (0 -8451403.668813668) / Math.Sqrt((45251344708.97477 / (299)) + (14805132771.504671 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.1594511447700082 = (10054625.16096246 - 8451403.668813668) / 10054625.16096246 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?i)Sherlock Holmes", Options: NonBacktracking)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 2.3519472529595014 < 3.557312796565656.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 191.79039885206225 (T) = (0 -2409300.936599503) / Math.Sqrt((5374236749.967982 / (299)) + (1356673035.8384764 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.3526693599965731 = (3721901.58739705 - 2409300.936599503) / 3721901.58739705 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Holmes.{0,25}Watson|Watson.{0,25}Holmes", Options: Compiled)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 4.514003880246913 < 5.252469962905982.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 109.88533734835082 (T) = (0 -4580367.819590807) / Math.Sqrt((6578113139.039182 / (299)) + (3046669596.0322165 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.1827000261659277 = (5604267.6694379905 - 4580367.819590807) / 5604267.6694379905 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock|Holmes|Watson", Options: None)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 5.432241453968254 < 6.563361210119047.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 118.70326141544973 (T) = (0 -5486871.405365597) / Math.Sqrt((16672198327.211212 / (299)) + (4571352024.173156 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.21111897434309337 = (6955258.431772576 - 5486871.405365597) / 6955258.431772576 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock|Holmes", Options: Compiled)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 2.5164172661764708 < 3.2816775856690956.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 136.29130721805592 (T) = (0 -2502386.004602609) / Math.Sqrt((8790074492.185379 / (299)) + (1066849107.158036 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.28218475865258097 = (3486114.3375910386 - 2502386.004602609) / 3486114.3375910386 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?i)Sherlock", Options: Compiled)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 2.345517510339123 < 3.574750288151848.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 118.52943401505567 (T) = (0 -2432523.018743008) / Math.Sqrt((5825153805.864684 / (299)) + (4772997501.418438 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.34898993677805995 = (3736536.739085277 - 2432523.018743008) / 3736536.739085277 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sher[a-z]+|Hol[a-z]+", Options: None)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 4.0443252718579235 < 4.835985636260932.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 128.32085811244625 (T) = (0 -4107813.5307625374) / Math.Sqrt((6946989153.702603 / (299)) + (2116305436.2405741 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.2051459550934328 = (5168009.846695061 - 4107813.5307625374) / 5168009.846695061 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "aei", Options: None)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.8805611051682694 < 3.480897484001009.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 308.66606594806683 (T) = (0 -1941188.7668897447) / Math.Sqrt((6281069855.438346 / (299)) + (487300794.6192068 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.471084863814233 = (3670132.7568132877 - 1941188.7668897447) / 3670132.7568132877 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "[a-q][^u-z]{13}x", Options: NonBacktracking)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.4701044694280074 < 1.9302281247355577.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 168.8625004544642 (T) = (0 -1512004.450328664) / Math.Sqrt((230261801.39026952 / (299)) + (399596970.4454804 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.25377141880777737 = (2026194.7725360354 - 1512004.450328664) / 2026194.7725360354 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sher[a-z]+|Hol[a-z]+", Options: Compiled)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 2.945756193574297 < 3.832277038346774.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 137.75482743053266 (T) = (0 -3027193.4037875216) / Math.Sqrt((4916235621.257085 / (299)) + (1790098072.113197 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.25151562341045736 = (4044430.9840919874 - 3027193.4037875216) / 4044430.9840919874 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock|Holmes|Watson", Options: Compiled)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 3.159298443290043 < 4.367663471718092.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 175.27142971962948 (T) = (0 -3206173.1562291738) / Math.Sqrt((3214361364.5706506 / (299)) + (2405423833.3344917 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.30079849446258167 = (4585478.050086368 - 3206173.1562291738) / 4585478.050086368 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?i)Sherlock", Options: None)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 2.359590539899353 < 3.5871214112673995.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 170.4720059021371 (T) = (0 -2428957.0397971864) / Math.Sqrt((6247082413.314845 / (299)) + (1781243794.340689 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.34986270157756216 = (3736067.820891165 - 2428957.0397971864) / 3736067.820891165 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock|Holmes|Watson", Options: NonBacktracking)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 3.0406996772486767 < 4.298930145292207.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 197.56575403627625 (T) = (0 -3112458.259494533) / Math.Sqrt((2589720294.114231 / (299)) + (1781441212.5937693 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.30223450721919465 = (4460607.885738876 - 3112458.259494533) / 4460607.885738876 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?i)Sherlock Holmes", Options: None)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 2.3617670615384623 < 3.5882140909343434.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 174.11068395563913 (T) = (0 -2445811.0890240404) / Math.Sqrt((11297575857.73104 / (299)) + (951709249.128556 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.3516192447466513 = (3772183.349378349 - 2445811.0890240404) / 3772183.349378349 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "aei", Options: NonBacktracking)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.8843752777777782 < 3.4669273344085987.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 222.67841016189544 (T) = (0 -1950108.7592741293) / Math.Sqrt((4343971627.934012 / (299)) + (2088755552.7542536 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.4671977110770109 = (3660098.31379684 - 1950108.7592741293) / 3660098.31379684 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "aqj", Options: NonBacktracking)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.822445748842593 < 3.4251018855153936.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 304.9450374129904 (T) = (0 -1848970.4516272047) / Math.Sqrt((5746297342.613908 / (299)) + (779751343.4953046 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.4967112219515667 = (3673776.432681103 - 1848970.4516272047) / 3673776.432681103 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 8, 2023 •
edited
Loading

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 967250cc86b4c613015066b47b882e7424fdb590
Compare 4862343c61f698da986901a182e94df24da1487f
Diff Diff

Improvements in System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
BitwiseAndBenchmark - Duration of single invocation 35.91 ns 8.54 ns 0.24 0.22 False
EqualityOperatorBenchmark - Duration of single invocation 25.39 ns 7.69 ns 0.30 0.26 False
AddOperatorBenchmark - Duration of single invocation 105.28 ns 8.65 ns 0.08 0.16 False
SubtractBenchmark - Duration of single invocation 107.79 ns 8.48 ns 0.08 0.19 False
SubtractionOperatorBenchmark - Duration of single invocation 104.57 ns 8.56 ns 0.08 0.17 False
EqualsStaticBenchmark - Duration of single invocation 110.38 ns 8.65 ns 0.08 0.17 False
UnaryNegateOperatorBenchmark - Duration of single invocation 97.43 ns 6.68 ns 0.07 0.21 False
EqualsAllBenchmark - Duration of single invocation 28.19 ns 8.14 ns 0.29 0.25 False
NegateBenchmark - Duration of single invocation 99.01 ns 6.86 ns 0.07 0.20 False
BitwiseOrOperatorBenchmark - Duration of single invocation 36.14 ns 10.27 ns 0.28 0.21 False
XorBenchmark - Duration of single invocation 36.36 ns 8.65 ns 0.24 0.22 False
AndNotBenchmark - Duration of single invocation 37.01 ns 8.65 ns 0.23 0.23 False
OnesComplementOperatorBenchmark - Duration of single invocation 28.36 ns 6.58 ns 0.23 0.27 False
AddBenchmark - Duration of single invocation 107.94 ns 8.81 ns 0.08 0.17 False
ConditionalSelectBenchmark - Duration of single invocation 64.65 ns 10.68 ns 0.17 0.29 False
BitwiseOrBenchmark - Duration of single invocation 36.50 ns 8.68 ns 0.24 0.16 False
BitwiseAndOperatorBenchmark - Duration of single invocation 36.09 ns 8.69 ns 0.24 0.23 False
AllBitsSetBenchmark - Duration of single invocation 16.00 ns 2.89 ns 0.18 0.30 False
MultiplyBenchmark - Duration of single invocation 107.60 ns 9.20 ns 0.09 0.15 False
InequalityOperatorBenchmark - Duration of single invocation 25.43 ns 7.82 ns 0.31 0.25 False
MultiplyOperatorBenchmark - Duration of single invocation 105.34 ns 9.04 ns 0.09 0.19 False
EqualsBenchmark - Duration of single invocation 35.08 ns 8.21 ns 0.23 0.28 False
OnesComplementBenchmark - Duration of single invocation 28.39 ns 6.65 ns 0.23 0.23 False
ExclusiveOrOperatorBenchmark - Duration of single invocation 35.94 ns 8.58 ns 0.24 0.20 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int32&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int32&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int32&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int32&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int32>.BitwiseAndBenchmark

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.544217525118523 < 34.18475736103156.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 164.61000018011669 (T) = (0 -8.902537399556515) / Math.Sqrt((4.8684433101897735 / (299)) + (0.6994858711473367 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7654893785350443 = (37.96219268851698 - 8.902537399556515) / 37.96219268851698 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int32&gt;.EqualityOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 7.686225812312663 < 23.993355541984585.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 114.26118697710108 (T) = (0 -8.107625252809918) / Math.Sqrt((4.26547965714939 / (299)) + (0.49361559339887306 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.6892437519785309 = (26.08998307976028 - 8.107625252809918) / 26.08998307976028 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int32&gt;.AddOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.647423320214644 < 99.8338570476466.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 497.84723192335497 (T) = (0 -9.007817791788463) / Math.Sqrt((9.029169852906522 / (299)) + (0.41722042333957754 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.9161432869042597 = (107.41916131990686 - 9.007817791788463) / 107.41916131990686 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int32&gt;.SubtractBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.484144391189714 < 102.26330311472496.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 406.1921918529565 (T) = (0 -9.235612263260458) / Math.Sqrt((13.150127339129634 / (299)) + (0.8042528267637792 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.9157591420442617 = (109.63340696402956 - 9.235612263260458) / 109.63340696402956 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int32&gt;.SubtractionOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.55653737016436 < 99.3374133913054.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 416.1505313393484 (T) = (0 -9.237493097329052) / Math.Sqrt((10.559125576390118 / (299)) + (0.9129808775530653 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.9133463124485108 = (106.60242348994295 - 9.237493097329052) / 106.60242348994295 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int32&gt;.EqualsStaticBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.649605229445363 < 103.88457920285202.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 382.4192449401239 (T) = (0 -8.945406983663505) / Math.Sqrt((20.954776938241388 / (299)) + (0.21984889970353266 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.9211913127636748 = (113.50787961787421 - 8.945406983663505) / 113.50787961787421 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int32&gt;.UnaryNegateOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 6.679792654262238 < 91.90146406673605.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 366.0794624724055 (T) = (0 -6.768902094881819) / Math.Sqrt((14.41058209293311 / (299)) + (0.7052533747367918 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.9314894077714209 = (98.80081130079886 - 6.768902094881819) / 98.80081130079886 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int32&gt;.EqualsAllBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.137677505647193 < 26.787375432671322.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 132.74598954529122 (T) = (0 -8.26431931302232) / Math.Sqrt((2.434244139932193 / (299)) + (0.721460285309317 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7111409589148107 = (28.61021514845033 - 8.26431931302232) / 28.61021514845033 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int32&gt;.NegateBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 6.864707374072291 < 93.67616172537525.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 416.4590596029692 (T) = (0 -6.744759018478372) / Math.Sqrt((12.206828437819937 / (299)) + (0.4913133682128606 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.9332541417347116 = (101.05134900911192 - 6.744759018478372) / 101.05134900911192 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int32&gt;.BitwiseOrOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 10.272506334481912 < 34.31209942990664.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 211.9914566952101 (T) = (0 -8.974818820438374) / Math.Sqrt((2.8405056188786086 / (299)) + (0.41809771818737335 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7621139785554008 = (37.72739047858892 - 8.974818820438374) / 37.72739047858892 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int32&gt;.XorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.650261478251798 < 34.28692731176777.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 150.78182399306593 (T) = (0 -8.864679930454189) / Math.Sqrt((6.850858579739785 / (299)) + (0.6574347444109225 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7656652810325101 = (37.82913590232447 - 8.864679930454189) / 37.82913590232447 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int32&gt;.AndNotBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.647311187323876 < 35.459406020909746.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 184.99342075632572 (T) = (0 -8.87918265862285) / Math.Sqrt((4.228635983959965 / (299)) + (0.6271474743756501 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7754899216314022 = (39.54914952212132 - 8.87918265862285) / 39.54914952212132 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int32&gt;.OnesComplementOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 6.583496243910236 < 26.967901634745242.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 138.5397936699101 (T) = (0 -6.8875546845657) / Math.Sqrt((1.792619874554565 / (299)) + (0.9645428928110852 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7661087164433084 = (29.447675774100677 - 6.8875546845657) / 29.447675774100677 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int32&gt;.AddBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.807561696855597 < 102.41358754986355.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 388.78805939209775 (T) = (0 -9.183914352674115) / Math.Sqrt((13.68260014484952 / (299)) + (1.0390829903137657 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.9168650798956754 = (110.46999673722408 - 9.183914352674115) / 110.46999673722408 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int32&gt;.ConditionalSelectBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 10.68013057591836 < 59.845873467371916.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 239.93368586078083 (T) = (0 -11.103619930587177) / Math.Sqrt((8.3848764673529 / (299)) + (1.0608609365111026 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.8293932975518322 = (65.08314017710165 - 11.103619930587177) / 65.08314017710165 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int32&gt;.BitwiseOrBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.678772555503542 < 34.23672949442318.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 99.28744527069281 (T) = (0 -9.171221701842272) / Math.Sqrt((3.210204505733592 / (299)) + (3.3909925014177467 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7570915435238564 = (37.75587657543323 - 9.171221701842272) / 37.75587657543323 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int32&gt;.BitwiseAndOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.68563046197723 < 34.29156663680118.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 114.80391381889892 (T) = (0 -8.901552225004474) / Math.Sqrt((3.7175345146232637 / (299)) + (2.404310606667441 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7648292416086426 = (37.85144159033171 - 8.901552225004474) / 37.85144159033171 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int32&gt;.AllBitsSetBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 2.8897930052593677 < 15.201615505343707.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 133.20951335680547 (T) = (0 -3.1470169704649273) / Math.Sqrt((1.4957799405579815 / (299)) + (0.21847370503695296 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.8061398988989318 = (16.233443357301468 - 3.1470169704649273) / 16.233443357301468 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int32&gt;.MultiplyBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 9.195236698428085 < 102.78294017968518.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 470.1178157255508 (T) = (0 -9.243175884912592) / Math.Sqrt((11.441262547619992 / (299)) + (0.3725377498025199 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.9161858408314187 = (110.28179458701173 - 9.243175884912592) / 110.28179458701173 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int32&gt;.InequalityOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 7.815686041204493 < 24.00060835141448.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 140.63700996140133 (T) = (0 -8.056874584301497) / Math.Sqrt((2.3989440336392494 / (299)) + (0.39490637232484044 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.6910848448632989 = (26.081189123713177 - 8.056874584301497) / 26.081189123713177 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int32&gt;.MultiplyOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 9.038312538421833 < 99.8533495402717.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 376.96886927284015 (T) = (0 -9.410425424795479) / Math.Sqrt((14.262975627521401 / (299)) + (0.9337303606811748 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.9123794528058552 = (107.39975640581629 - 9.410425424795479) / 107.39975640581629 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int32&gt;.EqualsBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.212046990298242 < 33.32143190673862.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 154.45038085453683 (T) = (0 -8.924860596346003) / Math.Sqrt((5.252154011313396 / (299)) + (0.8170165703967539 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7638853784210672 = (37.79884759640957 - 8.924860596346003) / 37.79884759640957 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int32&gt;.OnesComplementBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 6.653473146469778 < 26.884999079531916.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 136.35087550158755 (T) = (0 -6.894609323607597) / Math.Sqrt((2.492740936980576 / (299)) + (0.8952165669564727 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7659523231953949 = (29.45814040001588 - 6.894609323607597) / 29.45814040001588 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int32&gt;.ExclusiveOrOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.581755591362246 < 34.19418698751352.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 186.21005520902858 (T) = (0 -9.075643758001073) / Math.Sqrt((2.9367821241958696 / (299)) + (0.645761940712753 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7590015233958848 = (37.658510899674695 - 9.075643758001073) / 37.658510899674695 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 8, 2023 •
edited
Loading

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 967250cc86b4c613015066b47b882e7424fdb590
Compare 4862343c61f698da986901a182e94df24da1487f
Diff Diff

Improvements in System.Text.Json.Tests.Perf_Strings

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
WriteStringsUtf16 - Duration of single invocation 558.01 ms 493.04 ms 0.88 0.06 False
WriteStringsUtf16 - Duration of single invocation 556.47 ms 486.24 ms 0.87 0.04 False
WriteStringsUtf16 - Duration of single invocation 572.31 ms 503.47 ms 0.88 0.05 False
WriteStringsUtf16 - Duration of single invocation 565.72 ms 497.21 ms 0.88 0.03 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Tests.Perf_Strings*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Tests.Perf_Strings*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Tests.Perf_Strings*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Tests.Perf_Strings*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Text.Json.Tests.Perf_Strings.WriteStringsUtf16(Formatted: False, SkipValidation: False, Escaped: AllEscaped)

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 493.04175173333334 < 532.652025917857.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 51.15384300131127 (T) = (0 -493136662.6689112) / Math.Sqrt((114524330250606.7 / (299)) + (45466824466938.1 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.10757570005843949 = (552580944.6260078 - 493136662.6689112) / 552580944.6260078 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Strings.WriteStringsUtf16(Formatted: False, SkipValidation: True, Escaped: AllEscaped)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 486.2357154 < 533.7827079887501.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 55.024835862505995 (T) = (0 -490245533.30706495) / Math.Sqrt((120460527939089.27 / (299)) + (37693505977054.15 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.10968752014794915 = (550644346.1160202 - 490245533.30706495) / 550644346.1160202 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Strings.WriteStringsUtf16(Formatted: True, SkipValidation: False, Escaped: AllEscaped)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 503.4679013333333 < 539.7018643776785.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 50.635894736423865 (T) = (0 -504043776.91779286) / Math.Sqrt((97126860049917.14 / (299)) + (44840248568967.6 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.10201739171711047 = (561306836.3112496 - 504043776.91779286) / 561306836.3112496 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Strings.WriteStringsUtf16(Formatted: True, SkipValidation: True, Escaped: AllEscaped)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 497.21261219999997 < 538.335354132619.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 54.37871038150904 (T) = (0 -502570395.8563479) / Math.Sqrt((61106567931305.93 / (299)) + (40094000930376.13 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.10012463077507573 = (558488889.7328295 - 502570395.8563479) / 558488889.7328295 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 967250cc86b4c613015066b47b882e7424fdb590
Compare 4862343c61f698da986901a182e94df24da1487f
Diff Diff

Improvements in System.Text.Json.Tests.Perf_Get

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
GetString - Duration of single invocation 56.10 μs 46.64 μs 0.83 0.03 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Tests.Perf_Get*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Tests.Perf_Get*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Tests.Perf_Get*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Tests.Perf_Get*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Text.Json.Tests.Perf_Get.GetString

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 46.63967464964965 < 52.90294004596073.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 1/17/2023 8:36:05 PM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 56.10599075282762 (T) = (0 -47057.65978013319) / Math.Sqrt((5403502.635409398 / (299)) + (738596.6192330874 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.17975997280888528 = (57370.59667921915 - 47057.65978013319) / 57370.59667921915 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 967250cc86b4c613015066b47b882e7424fdb590
Compare 4862343c61f698da986901a182e94df24da1487f
Diff Diff

Improvements in System.IO.Tests.BinaryWriterExtendedTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
WriteAsciiCharArray - Duration of single invocation 32.84 μs 22.68 μs 0.69 0.12 False
WriteAsciiString - Duration of single invocation 33.30 μs 18.24 μs 0.55 0.09 False
WriteAsciiString - Duration of single invocation 13.45 ms 9.48 ms 0.71 0.10 False
WriteAsciiCharArray - Duration of single invocation 8.10 ms 4.47 ms 0.55 0.14 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.IO.Tests.BinaryWriterExtendedTests*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.IO.Tests.BinaryWriterExtendedTests*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.IO.Tests.BinaryWriterExtendedTests*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.IO.Tests.BinaryWriterExtendedTests*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.IO.Tests.BinaryWriterExtendedTests.WriteAsciiCharArray(StringLengthInChars: 8000)

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 22.675113672785553 < 31.30102476068646.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 83.15371908267537 (T) = (0 -18024.82306546901) / Math.Sqrt((3182208.7045022966 / (299)) + (1028594.735463996 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.45415698015135153 = (33021.99059075068 - 18024.82306546901) / 33021.99059075068 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.BinaryWriterExtendedTests.WriteAsciiString(StringLengthInChars: 8000)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 18.237368303571426 < 31.630143463944524.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 89.28882612580594 (T) = (0 -18041.676935141757) / Math.Sqrt((3420453.0926977284 / (299)) + (858013.8862673172 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.4602858135618779 = (33428.2058698678 - 18041.676935141757) / 33428.2058698678 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.BinaryWriterExtendedTests.WriteAsciiString(StringLengthInChars: 2000000)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 9.483192401785715 < 12.622178974889824.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 88.33603949500475 (T) = (0 -9385037.47610965) / Math.Sqrt((243622066827.4705 / (299)) + (48697295944.908104 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.28822723476291157 = (13185440.542928802 - 9385037.47610965) / 13185440.542928802 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.BinaryWriterExtendedTests.WriteAsciiCharArray(StringLengthInChars: 2000000)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 4.471366511160714 < 7.70664586904762.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 121.16738553684172 (T) = (0 -4343099.722197585) / Math.Sqrt((154826883965.89923 / (299)) + (21674761423.834557 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.46607355118321536 = (8134265.930864023 - 4343099.722197585) / 8134265.930864023 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 967250cc86b4c613015066b47b882e7424fdb590
Compare 4862343c61f698da986901a182e94df24da1487f
Diff Diff

Improvements in MicroBenchmarks.Serializers.Json_FromString<IndexViewModel>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SystemTextJson_SourceGen_ - Duration of single invocation 654.98 μs 593.08 μs 0.91 0.01 False
SystemTextJson_Reflection_ - Duration of single invocation 644.48 μs 574.59 μs 0.89 0.01 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'MicroBenchmarks.Serializers.Json_FromString&lt;IndexViewModel&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'MicroBenchmarks.Serializers.Json_FromString&lt;IndexViewModel&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'MicroBenchmarks.Serializers.Json_FromString&lt;IndexViewModel&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'MicroBenchmarks.Serializers.Json_FromString&lt;IndexViewModel&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Json_FromString<IndexViewModel>.SystemTextJson_SourceGen_

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 593.0779262308312 < 621.5299879925791.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 2/9/2023 4:10:34 PM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 79.91907536255783 (T) = (0 -590187.7716307731) / Math.Sqrt((212888885.25953725 / (299)) + (24241389.326191913 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.13046841545132687 = (678742.1896090269 - 590187.7716307731) / 678742.1896090269 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### MicroBenchmarks.Serializers.Json_FromString&lt;IndexViewModel&gt;.SystemTextJson_Reflection_

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 574.5855075387122 < 608.8678942606838.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 2/5/2023 11:24:06 PM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 84.28528081403353 (T) = (0 -577051.9163995776) / Math.Sqrt((168237466.5924709 / (299)) + (22391539.337440964 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.12959371914142226 = (662968.4655197659 - 577051.9163995776) / 662968.4655197659 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 8, 2023 •
edited
Loading

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 967250cc86b4c613015066b47b882e7424fdb590
Compare 4862343c61f698da986901a182e94df24da1487f
Diff Diff

Improvements in System.Text.RegularExpressions.Tests.Perf_Regex_Common

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
MatchesWord - Duration of single invocation 33.33 μs 23.00 μs 0.69 0.02 False
MatchesWord - Duration of single invocation 33.24 μs 23.70 μs 0.71 0.03 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Common*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Common*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Common*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Common*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchesWord(Options: Compiled)

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 22.998946948616815 < 31.45005757865995.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 182.11311208505293 (T) = (0 -23294.91957189462) / Math.Sqrt((297195.43787839386 / (299)) + (101606.49122957201 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.3051562308881338 = (33525.40615233503 - 23294.91957189462) / 33525.40615233503 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchesWord(Options: None)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 23.699454388353583 < 31.558837125579167.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 234.68108635890937 (T) = (0 -23244.12519036747) / Math.Sqrt((292722.5697370109 / (299)) + (45149.82634644812 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.30779431558671555 = (33579.795303283674 - 23244.12519036747) / 33579.795303283674 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 967250cc86b4c613015066b47b882e7424fdb590
Compare 4862343c61f698da986901a182e94df24da1487f
Diff Diff

Improvements in System.Tests.Perf_String

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Replace_Char - Duration of single invocation 105.49 ns 89.00 ns 0.84 0.17 False
IndexOfAny - Duration of single invocation 313.71 ns 241.12 ns 0.77 0.06 False
Replace_Char - Duration of single invocation 2.87 μs 242.55 ns 0.08 0.06 False
ToUpper - Duration of single invocation 991.53 ns 757.51 ns 0.76 0.07 False
Replace_Char - Duration of single invocation 8.66 μs 393.91 ns 0.05 0.06 False
Replace_Char - Duration of single invocation 98.82 μs 2.07 μs 0.02 0.08 False
ToLowerInvariant - Duration of single invocation 913.16 ns 723.32 ns 0.79 0.05 False
Replace_String - Duration of single invocation 6.08 μs 390.65 ns 0.06 0.07 False
ToLower - Duration of single invocation 996.98 ns 759.16 ns 0.76 0.05 False
Split - Duration of single invocation 476.89 ns 336.54 ns 0.71 0.06 False
ToUpperInvariant - Duration of single invocation 918.20 ns 701.53 ns 0.76 0.07 False
Split - Duration of single invocation 478.57 ns 332.52 ns 0.69 0.05 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Tests.Perf_String*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Tests.Perf_String*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Tests.Perf_String*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Tests.Perf_String*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_String.Replace_Char(text: "This is a very nice sentence", oldChar: 'z', newChar: 'y')

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 88.99560522720982 < 96.83078568753578.
IsChangePoint: Marked as a change because one of 12/18/2022 12:43:31 PM, 1/21/2023 8:50:04 PM, 1/23/2023 4:43:26 PM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 23.75791255811846 (T) = (0 -91.55264138870824) / Math.Sqrt((16.162335478815187 / (299)) + (8.35213194139773 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.11105341173517354 = (102.99003629387208 - 91.55264138870824) / 102.99003629387208 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_String.IndexOfAny

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 241.11870653365014 < 296.94072026030403.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 39.871203559617236 (T) = (0 -248.7905458678842) / Math.Sqrt((32.381972437274555 / (299)) + (126.87981850672413 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.21169414269493697 = (315.6015442006361 - 248.7905458678842) / 315.6015442006361 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_String.Replace_Char(text: "This is a very nice sentence", oldChar: 'i', newChar: 'I')

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 242.55216362561805 < 2.7016107900138753.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 745.5563443106103 (T) = (0 -239.1404639772778) / Math.Sqrt((3591.4951253968507 / (299)) + (25.426324800678596 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.9169832271893653 = (2880.6282860786296 - 239.1404639772778) / 2880.6282860786296 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_String.ToUpper(s: "This is a much longer piece of text that might benefit more from vectorization.")

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 757.5093477074236 < 938.3641616994579.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 102.05465623188046 (T) = (0 -767.2112523265936) / Math.Sqrt((683.3500445826494 / (299)) + (133.2066653740759 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.2313485796422533 = (998.1263704287662 - 767.2112523265936) / 998.1263704287662 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_String.Replace_Char(text: "yfesgj0sg1ijslnjsb3uofdz3tbzf6ysgblu3at20nfab2wei1kxfbvsbpzwhanjczcqa2psra3aacxb67qnwbnfp2tok6v0a58l", oldChar: 'b', newChar: '+')

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 393.91226001258946 < 8.266494426528487.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 735.1366249221453 (T) = (0 -384.67121572925527) / Math.Sqrt((39276.18178125043 / (299)) + (44.30940615318541 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.9564870240431272 = (8840.379387300838 - 384.67121572925527) / 8840.379387300838 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_String.Replace_Char(text: "yfesgj0sg1ijslnjsb3uofdz3tbzf6ysgblu3at20nfab2wei1kxfbvsbpzwhanjczcqa2psra3aacxb67qnwbnfp2tok6v0a58lzfdql1fehvs91yzkt9xam7ahjbhvpd9edll13ab46i74ktwwgkgbi792e5gkuuzevo5qm8qt83edag7zovoe686gmtw730kms2i5xgji4xcp25287q68fvhwszd3mszht2uh7bchlgkj5qnq1x9m4lg7vwn8cq5l756akua6oyx9k71bmxbysnmhvxvlxde4k9maumfgxd8gxhxx4mwpph2ttyox9zilt3ylv1q9s4bopfuoa8qlrzodg2q67sh85wx4slcd6w7ufnendaxai633ove2ktbaxdt2sz6y6mo42473xd274gz833p6hj3mu77c4m4od9e5s8btxleh0efqnu9zj9rwtbk5758lio35b3q426j5fwwq1qyknfedrsmqyfw1m38mkkotdf7n0vr6p3erhy8dkzntr9fwjrslxjgrbegih0n6bpb5bfuy55bu65ce9kejcfifxwpcs05umrsb8kvd64q2iwugbbi7vd35g5ho0rff9rhombgzzaniyq7bbjbqr88jyw4ccgnoyl31of3a5thv0vg08gnrqzxas800hewtw8tnwgw5pav81ntdpdd62689x3iqpc317y82b3e2trbpdzieoxldaz009tz37gqmh4bdp1bv9lnl5s58udb11z0h7i2sdl5nbyhjyfzxwzezmp4qx0i3eyvsd3fg8sryq9jhlvkonnfcvb4snl4mcbimdzg49tzdhqjmfxfcq3p1st6b9x2xyevo17evpqp4yc4f2rm0f26ivr3t2f5m0boc44vituxaovcqy1jrkcs6im2kdu3jvcexx2k76egve63aon5a6nbxss4rcke90npmqp35qluf571ms160y2nhaqef835wah41qru8tauu362v0r8konl8", oldChar: 'b', newChar: '+')

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 2.0677076180019522 < 94.92658471971414.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 741.9902881213392 (T) = (0 -2095.746874175629) / Math.Sqrt((5137066.517136889 / (299)) + (31891.347692716598 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.9793021926058585 = (101254.53552963457 - 2095.746874175629) / 101254.53552963457 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_String.ToLowerInvariant(s: "This is a much longer piece of text that might benefit more from vectorization.")

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 723.3248254147992 < 870.1930831321368.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 82.34091247249687 (T) = (0 -719.5814023547184) / Math.Sqrt((337.0602169363633 / (299)) + (230.80685901435746 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.21946978950101312 = (921.9135821721694 - 719.5814023547184) / 921.9135821721694 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_String.Replace_String(text: "This is a very nice sentence. This is another very nice sentence.", oldValue: "a", newValue: "b")

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 390.6491127950294 < 5.766598738024003.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 812.4222203605099 (T) = (0 -385.84328167135516) / Math.Sqrt((14663.647982104787 / (299)) + (65.87327273941527 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.9373221793219846 = (6155.977943992098 - 385.84328167135516) / 6155.977943992098 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_String.ToLower(s: "This is a much longer piece of text that might benefit more from vectorization.")

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 759.1624916692543 < 926.8725309763851.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 78.00470026213885 (T) = (0 -772.8084623900457) / Math.Sqrt((694.2437126897026 / (299)) + (228.84349608725444 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.2130137153444493 = (981.9846641015982 - 772.8084623900457) / 981.9846641015982 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_String.Split(s: "ABCDEFGHIJKLMNOPQRSTUVWXYZ", arr: [' '], options: RemoveEmptyEntries)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 336.5418050141538 < 457.21244821735.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 67.9901816409194 (T) = (0 -341.21488129559543) / Math.Sqrt((98.1118612653688 / (299)) + (191.50062169056136 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.2948279555337509 = (483.87465721767796 - 341.21488129559543) / 483.87465721767796 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_String.ToUpperInvariant(s: "This is a much longer piece of text that might benefit more from vectorization.")

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 701.5327302100161 < 887.5553785946215.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 90.72927963812738 (T) = (0 -717.7375928045554) / Math.Sqrt((623.9120519207546 / (299)) + (185.90629184149878 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.23706349059495982 = (940.756647449298 - 717.7375928045554) / 940.756647449298 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_String.Split(s: "ABCDEFGHIJKLMNOPQRSTUVWXYZ", arr: [' '], options: None)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 332.5239903276713 < 453.72001605498554.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 90.55574219468643 (T) = (0 -337.6427417558193) / Math.Sqrt((141.02152655345478 / (299)) + (96.70856113948321 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.2990015675296579 = (481.6597671494855 - 337.6427417558193) / 481.6597671494855 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 8, 2023 •
edited
Loading

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 967250cc86b4c613015066b47b882e7424fdb590
Compare 4862343c61f698da986901a182e94df24da1487f
Diff Diff

Improvements in System.IO.Tests.Perf_File

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
AppendAllText - Duration of single invocation 71.31 μs 54.32 μs 0.76 0.07 False
AppendAllTextAsync - Duration of single invocation 108.65 μs 87.53 μs 0.81 0.06 False
ReadAllLines - Duration of single invocation 118.83 μs 101.72 μs 0.86 0.01 False
WriteAllText - Duration of single invocation 112.03 μs 92.75 μs 0.83 0.04 False
WriteAllText - Duration of single invocation 764.67 μs 581.01 μs 0.76 0.06 False
WriteAllTextAsync - Duration of single invocation 1.56 ms 856.96 μs 0.55 0.60 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.IO.Tests.Perf_File*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.IO.Tests.Perf_File*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.IO.Tests.Perf_File*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.IO.Tests.Perf_File*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.IO.Tests.Perf_File.AppendAllText(size: 10000)

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 54.32497535714287 < 68.27388450693452.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 136.20214948997332 (T) = (0 -53299.77492676524) / Math.Sqrt((1247068.1682983118 / (299)) + (766536.8204956638 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.26777457016261347 = (72791.48299807358 - 53299.77492676524) / 72791.48299807358 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.Perf_File.AppendAllTextAsync(size: 10000)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 87.52719560714287 < 102.95847284845239.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 70.51358185964196 (T) = (0 -87784.61994468611) / Math.Sqrt((21361936.707650714 / (299)) + (1744962.999327281 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.20928274812558662 = (111018.97642499978 - 87784.61994468611) / 111018.97642499978 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.Perf_File.ReadAllLines

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 101.71852144031956 < 113.6803803602683.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 136.9449200250812 (T) = (0 -102722.34363939778) / Math.Sqrt((2200202.76380362 / (299)) + (537977.8654085092 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.15456057709734014 = (121501.71952795815 - 102722.34363939778) / 121501.71952795815 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.Perf_File.WriteAllText(size: 10000)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 92.74515585376162 < 106.39032412433781.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 85.70312536574212 (T) = (0 -93794.53601572891) / Math.Sqrt((2360453.1221577367 / (299)) + (1905725.793415271 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.16743544329093088 = (112657.37324499681 - 93794.53601572891) / 112657.37324499681 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.Perf_File.WriteAllText(size: 100000)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 581.0114313271606 < 727.9444944911858.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 143.40547561910105 (T) = (0 -575224.2148470761) / Math.Sqrt((299273036.8744041 / (299)) + (33341666.775431298 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.24587286498422212 = (762768.2232055491 - 575224.2148470761) / 762768.2232055491 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.Perf_File.WriteAllTextAsync(size: 100000)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 856.9629239783653 < 1.3365823674218749.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 13.516072675639172 (T) = (0 -919994.275208709) / Math.Sqrt((116437217661.61507 / (299)) + (7621572729.1957035 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.25652833894898075 = (1237430.185177665 - 919994.275208709) / 1237430.185177665 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 967250cc86b4c613015066b47b882e7424fdb590
Compare 4862343c61f698da986901a182e94df24da1487f
Diff Diff

Improvements in System.IO.Tests.StreamReaderReadLineTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ReadLineAsync - Duration of single invocation 240.41 μs 170.21 μs 0.71 0.02 False
ReadLine - Duration of single invocation 189.26 μs 115.16 μs 0.61 0.03 False
ReadLineAsync - Duration of single invocation 225.76 μs 149.04 μs 0.66 0.02 False
ReadLine - Duration of single invocation 219.12 μs 153.58 μs 0.70 0.02 False
ReadLine - Duration of single invocation 350.53 μs 309.61 μs 0.88 0.03 False
ReadLineAsync - Duration of single invocation 361.75 μs 289.93 μs 0.80 0.01 False
ReadLineAsync - Duration of single invocation 238.84 μs 167.93 μs 0.70 0.01 False
ReadLine - Duration of single invocation 187.89 μs 113.43 μs 0.60 0.03 False
ReadLine - Duration of single invocation 189.21 μs 111.51 μs 0.59 0.03 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.IO.Tests.StreamReaderReadLineTests*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.IO.Tests.StreamReaderReadLineTests*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.IO.Tests.StreamReaderReadLineTests*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.IO.Tests.StreamReaderReadLineTests*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.IO.Tests.StreamReaderReadLineTests.ReadLineAsync(LineLengthRange: [ 0, 1024])

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 170.21153629285118 < 230.1448400308893.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 226.56108560882802 (T) = (0 -170840.2531733709) / Math.Sqrt((15650554.824748686 / (299)) + (2607296.2807050394 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.3033549357610513 = (245232.8480357579 - 170840.2531733709) / 245232.8480357579 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StreamReaderReadLineTests.ReadLine(LineLengthRange: [ 129, 1024])

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 115.15821224547511 < 179.45214830478724.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 374.91851441255506 (T) = (0 -116196.44565381226) / Math.Sqrt((5039144.634521358 / (299)) + (1050550.9747473097 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.38982616430381534 = (190431.7079103148 - 116196.44565381226) / 190431.7079103148 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StreamReaderReadLineTests.ReadLineAsync(LineLengthRange: [1025, 2048])

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 149.03611321428573 < 214.5967916369167.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 360.45594088284935 (T) = (0 -150279.58234575414) / Math.Sqrt((8251079.493640332 / (299)) + (840595.8033200687 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.3384156958304848 = (227151.06963488145 - 150279.58234575414) / 227151.06963488145 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StreamReaderReadLineTests.ReadLine(LineLengthRange: [  33,  128])

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 153.57892888071896 < 208.04416449754407.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 232.4818497432502 (T) = (0 -155228.47289540424) / Math.Sqrt((12409717.623323217 / (299)) + (1661233.4934656338 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.2933769044730913 = (219676.47799518183 - 155228.47289540424) / 219676.47799518183 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StreamReaderReadLineTests.ReadLine(LineLengthRange: [   9,   32])

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 309.61420232371796 < 333.41755735855446.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 49.781021437983455 (T) = (0 -309120.9463674522) / Math.Sqrt((30285523.83317902 / (299)) + (32682841.92995134 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.12567457722124656 = (353553.6521230434 - 309120.9463674522) / 353553.6521230434 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StreamReaderReadLineTests.ReadLineAsync(LineLengthRange: [  33,  128])

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 289.93090316358024 < 336.86140661668077.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 146.76469734121403 (T) = (0 -287322.49751044495) / Math.Sqrt((25963331.050330173 / (299)) + (6641815.837685808 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.19613114590521713 = (357424.5923907474 - 287322.49751044495) / 357424.5923907474 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StreamReaderReadLineTests.ReadLineAsync(LineLengthRange: [ 129, 1024])

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 167.93481061613477 < 226.98900364308503.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 286.21583946231885 (T) = (0 -167334.69309984604) / Math.Sqrt((9876776.992800768 / (299)) + (1704743.0344473422 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.3104797636808082 = (242682.78766278832 - 167334.69309984604) / 242682.78766278832 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StreamReaderReadLineTests.ReadLine(LineLengthRange: [   0, 1024])

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 113.42549935300207 < 177.6567714361433.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 390.79349693100596 (T) = (0 -114867.22490107841) / Math.Sqrt((4835364.872634416 / (299)) + (884189.3173577375 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.38887937252019694 = (187961.622854687 - 114867.22490107841) / 187961.622854687 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StreamReaderReadLineTests.ReadLine(LineLengthRange: [1025, 2048])

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 111.50715583545917 < 179.4388150649789.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 390.6114904763286 (T) = (0 -113514.92870318817) / Math.Sqrt((5579275.463225116 / (299)) + (928531.0181964074 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.40278731832767345 = (190074.544943221 - 113514.92870318817) / 190074.544943221 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 8, 2023 •
edited
Loading

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 967250cc86b4c613015066b47b882e7424fdb590
Compare 4862343c61f698da986901a182e94df24da1487f
Diff Diff

Improvements in System.Text.Json.Tests.Utf8JsonReaderCommentsTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Utf8JsonReaderCommentParsing - Duration of single invocation 23.76 μs 16.21 μs 0.68 0.00 False
Utf8JsonReaderCommentParsing - Duration of single invocation 119.06 μs 39.68 μs 0.33 0.04 False
Utf8JsonReaderCommentParsing - Duration of single invocation 24.51 μs 17.06 μs 0.70 0.00 False
Utf8JsonReaderCommentParsing - Duration of single invocation 11.52 μs 3.31 μs 0.29 0.03 False
Utf8JsonReaderCommentParsing - Duration of single invocation 11.55 μs 3.33 μs 0.29 0.02 False
Utf8JsonReaderCommentParsing - Duration of single invocation 119.41 μs 40.05 μs 0.34 0.07 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Tests.Utf8JsonReaderCommentsTests*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Tests.Utf8JsonReaderCommentsTests*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Tests.Utf8JsonReaderCommentsTests*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Tests.Utf8JsonReaderCommentsTests*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Text.Json.Tests.Utf8JsonReaderCommentsTests.Utf8JsonReaderCommentParsing(CommentHandling: Skip, SegmentSize: 100, TestCase: LongSingleLine)

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 16.213363941805845 < 22.585379150436985.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 280.6687088475403 (T) = (0 -16193.225188274962) / Math.Sqrt((67953.89031895083 / (299)) + (21759.424486243188 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.31288720299679496 = (23567.055160230742 - 16193.225188274962) / 23567.055160230742 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Utf8JsonReaderCommentsTests.Utf8JsonReaderCommentParsing(CommentHandling: Skip, SegmentSize: 0, TestCase: LongMultiLine)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 39.684112933587144 < 113.11392536428572.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 165.7115346443813 (T) = (0 -40262.27837767489) / Math.Sqrt((81221168.42962687 / (299)) + (224517.50033362833 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.6839352203286468 = (127386.1593168968 - 40262.27837767489) / 127386.1593168968 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Utf8JsonReaderCommentsTests.Utf8JsonReaderCommentParsing(CommentHandling: Allow, SegmentSize: 100, TestCase: LongSingleLine)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 17.059605209471762 < 23.449939760218435.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 218.36202142913737 (T) = (0 -17295.55279035186) / Math.Sqrt((67200.15601118316 / (299)) + (41764.274079137605 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.29640306995396015 = (24581.62060090303 - 17295.55279035186) / 24581.62060090303 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Utf8JsonReaderCommentsTests.Utf8JsonReaderCommentParsing(CommentHandling: Skip, SegmentSize: 0, TestCase: LongSingleLine)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 3.3085259140015744 < 10.985937333754427.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 590.2128173772775 (T) = (0 -3417.5132316984805) / Math.Sqrt((29107.84261227345 / (299)) + (3351.3049855340637 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.6916282062814441 = (11082.444313365339 - 3417.5132316984805) / 11082.444313365339 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Utf8JsonReaderCommentsTests.Utf8JsonReaderCommentParsing(CommentHandling: Allow, SegmentSize: 0, TestCase: LongSingleLine)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 3.3349291139692756 < 10.945443237911121.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 241.00408812615782 (T) = (0 -3487.2235112711774) / Math.Sqrt((27047.60824619389 / (299)) + (43004.00221058067 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.6866583049741122 = (11129.139743062497 - 3487.2235112711774) / 11129.139743062497 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Utf8JsonReaderCommentsTests.Utf8JsonReaderCommentParsing(CommentHandling: Allow, SegmentSize: 0, TestCase: LongMultiLine)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 40.045203310067684 < 113.15798768953046.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 144.35662885278634 (T) = (0 -40646.59107458698) / Math.Sqrt((89984694.53432307 / (299)) + (3019379.6593735437 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.6821584454593868 = (127883.18737408271 - 40646.59107458698) / 127883.18737408271 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 967250cc86b4c613015066b47b882e7424fdb590
Compare 4862343c61f698da986901a182e94df24da1487f
Diff Diff

Improvements in System.Text.Json.Serialization.Tests.WriteJson<BinaryData>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SerializeToString - Duration of single invocation 23.67 μs 20.42 μs 0.86 0.05 False
SerializeObjectProperty - Duration of single invocation 31.52 μs 27.47 μs 0.87 0.01 False
SerializeToString - Duration of single invocation 26.18 μs 22.77 μs 0.87 0.02 False
SerializeObjectProperty - Duration of single invocation 30.89 μs 26.94 μs 0.87 0.02 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;BinaryData&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;BinaryData&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;BinaryData&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;BinaryData&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.WriteJson<BinaryData>.SerializeToString(Mode: SourceGen)

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 20.41811883420252 < 22.51228954650334.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 70.90406036802739 (T) = (0 -20382.771766708724) / Math.Sqrt((172605.07525375814 / (299)) + (89118.86193452199 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.14748819012518166 = (23909.07847916113 - 20382.771766708724) / 23909.07847916113 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.WriteJson&lt;BinaryData&gt;.SerializeObjectProperty(Mode: Reflection)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 27.473186101347395 < 29.536552174424045.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 1/5/2023 3:38:13 PM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 47.67617734783185 (T) = (0 -27674.758391190582) / Math.Sqrt((350465.6018330796 / (299)) + (238420.65108321005 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.11982551427153432 = (31442.354714799425 - 27674.758391190582) / 31442.354714799425 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.WriteJson&lt;BinaryData&gt;.SerializeToString(Mode: Reflection)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 22.769071174770925 < 24.97045617771752.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 144.3943322420242 (T) = (0 -22839.578015051608) / Math.Sqrt((93927.16130769643 / (299)) + (13020.03350313209 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.1332343361882286 = (26350.34931426575 - 22839.578015051608) / 26350.34931426575 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.WriteJson&lt;BinaryData&gt;.SerializeObjectProperty(Mode: SourceGen)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 26.940992717359908 < 29.041890767540423.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 1/5/2023 3:38:13 PM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 65.45988856003746 (T) = (0 -27103.12505423839) / Math.Sqrt((343538.76783271896 / (299)) + (128497.76462434238 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.1308128263118556 = (31182.15026026456 - 27103.12505423839) / 31182.15026026456 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 967250cc86b4c613015066b47b882e7424fdb590
Compare 4862343c61f698da986901a182e94df24da1487f
Diff Diff

Improvements in System.Text.Json.Serialization.Tests.ReadJson<HashSet<String>>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DeserializeFromString - Duration of single invocation 164.35 μs 145.57 μs 0.89 0.01 False
DeserializeFromUtf8Bytes - Duration of single invocation 147.65 μs 130.07 μs 0.88 0.01 False
DeserializeFromString - Duration of single invocation 166.15 μs 147.52 μs 0.89 0.01 False
DeserializeFromStream - Duration of single invocation 168.52 μs 155.99 μs 0.93 0.01 False
DeserializeFromReader - Duration of single invocation 189.54 μs 171.17 μs 0.90 0.01 False
DeserializeFromUtf8Bytes - Duration of single invocation 145.85 μs 131.39 μs 0.90 0.01 False
DeserializeFromReader - Duration of single invocation 187.07 μs 171.76 μs 0.92 0.01 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;HashSet&lt;String&gt;&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;HashSet&lt;String&gt;&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;HashSet&lt;String&gt;&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;HashSet&lt;String&gt;&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.ReadJson<HashSet<String>>.DeserializeFromString(Mode: SourceGen)

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 145.56900054238983 < 156.8942405848214.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 1/26/2023 10:17:57 AM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 44.12960425050877 (T) = (0 -148595.45572989376) / Math.Sqrt((29574265.758629203 / (299)) + (4016603.9946332695 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.1130958365253643 = (167543.98259642784 - 148595.45572989376) / 167543.98259642784 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;HashSet&lt;String&gt;&gt;.DeserializeFromUtf8Bytes(Mode: SourceGen)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 130.07103002754823 < 137.98536910292734.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 1/26/2023 10:17:57 AM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 39.43172727649357 (T) = (0 -132541.79759264062) / Math.Sqrt((29020661.578726668 / (299)) + (2953997.6844876655 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.10631954947821634 = (148310.0559212802 - 132541.79759264062) / 148310.0559212802 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;HashSet&lt;String&gt;&gt;.DeserializeFromString(Mode: Reflection)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 147.51772704439253 < 157.00006357443576.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 1/25/2023 8:05:29 AM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 46.30740422546474 (T) = (0 -148642.7527979799) / Math.Sqrt((29797566.331113566 / (299)) + (3125158.2611840167 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.11267781002689638 = (167518.35407439267 - 148642.7527979799) / 167518.35407439267 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;HashSet&lt;String&gt;&gt;.DeserializeFromStream(Mode: Reflection)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 155.9854577232143 < 157.91237576878328.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 1/18/2023 3:20:52 PM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 31.396567368360643 (T) = (0 -155534.54834362894) / Math.Sqrt((33159716.614534583 / (299)) + (4258238.847365456 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.083085418871321 = (169628.17643511915 - 155534.54834362894) / 169628.17643511915 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;HashSet&lt;String&gt;&gt;.DeserializeFromReader(Mode: SourceGen)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 171.17034667119563 < 180.70974304433435.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 1/26/2023 10:17:57 AM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 48.816235907781085 (T) = (0 -171897.5601810505) / Math.Sqrt((34480852.25064108 / (299)) + (3268878.1676311176 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.10881684872313231 = (192886.90538500357 - 171897.5601810505) / 192886.90538500357 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;HashSet&lt;String&gt;&gt;.DeserializeFromUtf8Bytes(Mode: Reflection)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 131.38520380755605 < 137.49893139551722.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 1/26/2023 10:17:57 AM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 38.443504798118674 (T) = (0 -132296.60205379908) / Math.Sqrt((30620505.384544544 / (299)) + (2708633.4002416427 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.10414245325907859 = (147675.93635292415 - 132296.60205379908) / 147675.93635292415 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;HashSet&lt;String&gt;&gt;.DeserializeFromReader(Mode: Reflection)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 171.7639082446808 < 178.39445147355912.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 1/26/2023 2:33:00 AM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 45.63994357479373 (T) = (0 -171022.49442969073) / Math.Sqrt((38603414.69998517 / (299)) + (4268423.765015002 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.11122943949921153 = (192425.9218637103 - 171022.49442969073) / 192425.9218637103 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 8, 2023 •
edited
Loading

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 967250cc86b4c613015066b47b882e7424fdb590
Compare 4862343c61f698da986901a182e94df24da1487f
Diff Diff

Improvements in System.Runtime.Intrinsics.Tests.Perf_Vector128Float

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ExclusiveOrOperatorBenchmark - Duration of single invocation 36.35 ns 8.64 ns 0.24 0.20 False
BitwiseAndOperatorBenchmark - Duration of single invocation 35.94 ns 8.71 ns 0.24 0.23 False
XorBenchmark - Duration of single invocation 36.54 ns 8.70 ns 0.24 0.21 False
ConditionalSelectBenchmark - Duration of single invocation 62.93 ns 12.49 ns 0.20 0.23 False
OnesComplementBenchmark - Duration of single invocation 28.41 ns 6.73 ns 0.24 0.23 False
BitwiseOrBenchmark - Duration of single invocation 36.08 ns 8.97 ns 0.25 0.19 False
BitwiseAndBenchmark - Duration of single invocation 36.52 ns 10.29 ns 0.28 0.20 False
AllBitsSetBenchmark - Duration of single invocation 16.07 ns 3.16 ns 0.20 0.27 False
BitwiseOrOperatorBenchmark - Duration of single invocation 36.51 ns 8.98 ns 0.25 0.21 False
OnesComplementOperatorBenchmark - Duration of single invocation 28.33 ns 6.62 ns 0.23 0.22 False
AndNotBenchmark - Duration of single invocation 37.83 ns 8.46 ns 0.22 0.22 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Float*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Float*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Float*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Float*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Runtime.Intrinsics.Tests.Perf_Vector128Float.ExclusiveOrOperatorBenchmark

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.636136177997257 < 34.29896838880994.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 183.4279097913406 (T) = (0 -8.852280124264503) / Math.Sqrt((3.8386345383903167 / (299)) + (0.5699235833127301 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7660234612461647 = (37.83405024884958 - 8.852280124264503) / 37.83405024884958 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Float.BitwiseAndOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.711384365568001 < 34.2541988978573.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 120.08316274004339 (T) = (0 -9.190494498952589) / Math.Sqrt((3.969210741000101 / (299)) + (2.0234415702356823 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7561567799422239 = (37.69017853674584 - 9.190494498952589) / 37.69017853674584 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Float.XorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.695574298631694 < 34.36928033784815.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 170.76919577937898 (T) = (0 -9.153693867016433) / Math.Sqrt((3.7580086453790322 / (299)) + (0.7236120163133409 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7572658056078624 = (37.71077202344478 - 9.153693867016433) / 37.71077202344478 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Float.ConditionalSelectBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 12.48619422318758 < 59.78028418575244.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 237.2305574749535 (T) = (0 -11.363952867117371) / Math.Sqrt((11.378558342332841 / (299)) + (0.6186152511948747 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.8253103091769559 = (65.0522238237215 - 11.363952867117371) / 65.0522238237215 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Float.OnesComplementBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 6.73013390940155 < 26.972705972317822.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 130.2180447455971 (T) = (0 -6.977202653196695) / Math.Sqrt((3.608893702502515 / (299)) + (0.8292621604469972 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7628719001953472 = (29.42376993255775 - 6.977202653196695) / 29.42376993255775 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Float.BitwiseOrBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.968022950719769 < 34.5480146231932.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 179.0142555137083 (T) = (0 -9.04419798364541) / Math.Sqrt((4.108375756544916 / (299)) + (0.5615777490441766 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7603306024235127 = (37.73614017934466 - 9.04419798364541) / 37.73614017934466 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Float.BitwiseAndBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 10.286663882215803 < 34.33140921207427.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 84.67751673665498 (T) = (0 -9.40110567985511) / Math.Sqrt((4.337093098383069 / (299)) + (4.604179864460264 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7512842365784046 = (37.798592057550444 - 9.40110567985511) / 37.798592057550444 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Float.AllBitsSetBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 3.1571835113455062 < 15.202899870013898.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 94.42233881959608 (T) = (0 -3.227441624107384) / Math.Sqrt((1.199626116234517 / (299)) + (0.7041749920558896 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.8012757519720995 = (16.24080431117927 - 3.227441624107384) / 16.24080431117927 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Float.BitwiseOrOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.977356719252136 < 34.33138180663902.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 151.2295379694876 (T) = (0 -9.069294134659888) / Math.Sqrt((5.049953488077905 / (299)) + (0.9119208859181059 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7605734611585916 = (37.87923501941953 - 9.069294134659888) / 37.87923501941953 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Float.OnesComplementOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 6.620148100549116 < 26.935199205159915.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 220.1761368121904 (T) = (0 -6.735524645346244) / Math.Sqrt((1.6700614341221562 / (299)) + (0.23296424203705127 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7704486892963511 = (29.342131067340393 - 6.735524645346244) / 29.342131067340393 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Float.AndNotBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.455013087525563 < 36.294442512721254.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 188.235324084166 (T) = (0 -8.963851755920183) / Math.Sqrt((4.066746722428191 / (299)) + (0.6047489730980269 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7735710792295347 = (39.587927749772675 - 8.963851755920183) / 39.587927749772675 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 967250cc86b4c613015066b47b882e7424fdb590
Compare 4862343c61f698da986901a182e94df24da1487f
Diff Diff

Improvements in System.Text.Perf_Utf8Encoding

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
GetString - Duration of single invocation 1.01 ms 589.48 μs 0.58 0.05 False
GetBytes - Duration of single invocation 1.15 ms 822.22 μs 0.72 0.04 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Perf_Utf8Encoding*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Perf_Utf8Encoding*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Perf_Utf8Encoding*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Perf_Utf8Encoding*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Text.Perf_Utf8Encoding.GetString(Input: EnglishAllAscii)

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 589.4826822089948 < 0.968330506951837.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 279.65458563239173 (T) = (0 -590752.8852460142) / Math.Sqrt((636331893.6615429 / (299)) + (5947328.347579025 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.4154875521939217 = (1010676.3123067077 - 590752.8852460142) / 1010676.3123067077 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Perf_Utf8Encoding.GetBytes(Input: EnglishAllAscii)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 822.224563870614 < 1.0874526217805631.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 107.13937644821247 (T) = (0 -825297.9919981252) / Math.Sqrt((968154531.183114 / (299)) + (235391589.3688171 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.2715582024533543 = (1132963.5322652904 - 825297.9919981252) / 1132963.5322652904 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 8, 2023 •
edited
Loading

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 967250cc86b4c613015066b47b882e7424fdb590
Compare 4862343c61f698da986901a182e94df24da1487f
Diff Diff

Improvements in System.Collections.Tests.Perf_BitArray

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
BitArrayNot - Duration of single invocation 760.06 ns 484.52 ns 0.64 0.04 False
BitArrayXor - Duration of single invocation 1.18 μs 649.58 ns 0.55 0.03 False
BitArrayCopyToBoolArray - Duration of single invocation 795.00 ns 712.16 ns 0.90 0.04 False
BitArrayBoolArrayCtor - Duration of single invocation 7.60 μs 744.64 ns 0.10 0.25 False
BitArrayAnd - Duration of single invocation 1.19 μs 645.24 ns 0.54 0.08 False
BitArrayOr - Duration of single invocation 1.18 μs 641.49 ns 0.54 0.05 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Collections.Tests.Perf_BitArray*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Collections.Tests.Perf_BitArray*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Collections.Tests.Perf_BitArray*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Collections.Tests.Perf_BitArray*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Collections.Tests.Perf_BitArray.BitArrayNot(Size: 512)

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 484.5168351086584 < 721.5722024146817.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 96.77253457281196 (T) = (0 -493.9783726922722) / Math.Sqrt((179.38362778792103 / (299)) + (366.4677030594215 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.36211792765956513 = (774.4039127480575 - 493.9783726922722) / 774.4039127480575 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.Tests.Perf_BitArray.BitArrayXor(Size: 512)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 649.5779922493753 < 1.1189864668016063.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 214.50621274503675 (T) = (0 -647.7833891316841) / Math.Sqrt((955.8041631114171 / (299)) + (124.09862490345041 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.44445432126755535 = (1166.0308304614891 - 647.7833891316841) / 1166.0308304614891 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.Tests.Perf_BitArray.BitArrayCopyToBoolArray(Size: 4)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 712.1648363514574 < 755.7003838555147.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 50.2745552192322 (T) = (0 -712.0438463708118) / Math.Sqrt((239.34849664487874 / (299)) + (73.33007549022642 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.09786643215394901 = (789.2887170476313 - 712.0438463708118) / 789.2887170476313 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.Tests.Perf_BitArray.BitArrayBoolArrayCtor(Size: 512)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 744.6414187400687 < 7.216284066051979.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 247.33708171024554 (T) = (0 -756.51029418337) / Math.Sqrt((230156.8296624276 / (299)) + (1547.7332148075145 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.9025616943266511 = (7763.992702413023 - 756.51029418337) / 7763.992702413023 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.Tests.Perf_BitArray.BitArrayAnd(Size: 512)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 645.2429917488104 < 1.1214908293699783.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 123.21198215717543 (T) = (0 -654.855646475217) / Math.Sqrt((1880.0999915551074 / (299)) + (529.0395676749272 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.4407425335417053 = (1170.9376910465464 - 654.855646475217) / 1170.9376910465464 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.Tests.Perf_BitArray.BitArrayOr(Size: 512)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 641.4929761202832 < 1.119439447129721.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 199.1939033109105 (T) = (0 -647.2430268367702) / Math.Sqrt((1339.7940049450417 / (299)) + (111.87154388536135 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.44633137284562824 = (1169.0079500500728 - 647.2430268367702) / 1169.0079500500728 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 967250cc86b4c613015066b47b882e7424fdb590
Compare 4862343c61f698da986901a182e94df24da1487f
Diff Diff

Improvements in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Count - Duration of single invocation 16.91 ms 11.06 ms 0.65 0.01 False
Count - Duration of single invocation 17.23 ms 11.59 ms 0.67 0.05 False
Count - Duration of single invocation 17.26 ms 11.43 ms 0.66 0.03 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas.Count(Pattern: "[\w\.+-]+@[\w\.-]+\.[\w\.-]+", Options: NonBacktracking)

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 11.055124178571429 < 16.06374044559524.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 208.41366270247065 (T) = (0 -11507395.111258974) / Math.Sqrt((10101689972.86914 / (299)) + (29817610264.43796 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.31887972431959116 = (16894806.280379187 - 11507395.111258974) / 16894806.280379187 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas.Count(Pattern: "[\\w\\.+-]+@[\\w\\.-]+\\.[\\w\\.-]+", Options: None)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 11.587887350340134 < 16.3914064908801.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 124.73961340118265 (T) = (0 -11908509.768653423) / Math.Sqrt((268509013048.49246 / (299)) + (49317663276.46487 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.3161176554364513 = (17413097.24299637 - 11908509.768653423) / 17413097.24299637 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas.Count(Pattern: "[\\w\\.+-]+@[\\w\\.-]+\\.[\\w\\.-]+", Options: Compiled)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 11.426492015873016 < 16.387670172465985.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 58.01991416170117 (T) = (0 -12040251.560469927) / Math.Sqrt((244402072081.55374 / (299)) + (361689701078.8 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.30777246565098504 = (17393488.35898709 - 12040251.560469927) / 17393488.35898709 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 967250cc86b4c613015066b47b882e7424fdb590
Compare 4862343c61f698da986901a182e94df24da1487f
Diff Diff

Improvements in System.Net.Primitives.Tests.IPAddressPerformanceTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
GetAddressBytes - Duration of single invocation 313.97 ns 78.85 ns 0.25 0.11 False
Ctor_Bytes - Duration of single invocation 365.21 ns 123.69 ns 0.34 0.09 False
Ctor_Span - Duration of single invocation 356.04 ns 111.32 ns 0.31 0.08 False
TryWriteBytes - Duration of single invocation 277.14 ns 41.70 ns 0.15 0.17 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Net.Primitives.Tests.IPAddressPerformanceTests*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Net.Primitives.Tests.IPAddressPerformanceTests*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Net.Primitives.Tests.IPAddressPerformanceTests*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Net.Primitives.Tests.IPAddressPerformanceTests*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Net.Primitives.Tests.IPAddressPerformanceTests.GetAddressBytes(address: 1020:3040:5060:7080:9010:1112:1314:1516)

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 78.84702907641442 < 297.6016349508163.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 315.5835645202538 (T) = (0 -77.80315110676197) / Math.Sqrt((165.09688938634602 / (299)) + (2.5337984627295085 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7594862721519648 = (323.4873610038619 - 77.80315110676197) / 323.4873610038619 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Net.Primitives.Tests.IPAddressPerformanceTests.Ctor_Bytes(address: [16, 32, 48, 64, 80, ...])

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 123.68619930135421 < 349.9507843535868.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 356.12852206290955 (T) = (0 -121.53241425412045) / Math.Sqrt((97.93885928176302 / (299)) + (8.21180945509038 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.6749821163852281 = (373.9253142087621 - 121.53241425412045) / 373.9253142087621 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Net.Primitives.Tests.IPAddressPerformanceTests.Ctor_Span(address: [16, 32, 48, 64, 80, ...])

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 111.31593811084524 < 339.96963233379785.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 396.948847349663 (T) = (0 -111.97927009446678) / Math.Sqrt((97.92601680271987 / (299)) + (3.514317645427924 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.6921512193657096 = (363.74764864666713 - 111.97927009446678) / 363.74764864666713 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Net.Primitives.Tests.IPAddressPerformanceTests.TryWriteBytes(address: 1020:3040:5060:7080:9010:1112:1314:1516)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 41.69587415303933 < 264.4648482345123.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 298.0300974794552 (T) = (0 -42.59442717849836) / Math.Sqrt((133.3849385416242 / (299)) + (10.263553158331021 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.8508259075247558 = (285.5350179895816 - 42.59442717849836) / 285.5350179895816 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 8, 2023 •
edited
Loading

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 967250cc86b4c613015066b47b882e7424fdb590
Compare 4862343c61f698da986901a182e94df24da1487f
Diff Diff

Improvements in System.IO.Tests.StreamReaderReadToEndTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ReadToEnd - Duration of single invocation 335.20 ms 205.28 ms 0.61 0.04 False
ReadToEnd - Duration of single invocation 336.93 ms 200.23 ms 0.59 0.06 False
ReadToEnd - Duration of single invocation 335.98 ms 200.87 ms 0.60 0.05 False
ReadToEndAsync - Duration of single invocation 375.12 ms 239.32 ms 0.64 0.05 False
ReadToEnd - Duration of single invocation 373.58 ms 202.31 ms 0.54 0.06 False
ReadToEndAsync - Duration of single invocation 371.99 ms 240.01 ms 0.65 0.05 False
ReadToEndAsync - Duration of single invocation 374.24 ms 240.60 ms 0.64 0.05 False
ReadToEnd - Duration of single invocation 336.55 ms 202.96 ms 0.60 0.07 False
ReadToEndAsync - Duration of single invocation 381.06 ms 238.52 ms 0.63 0.04 False
ReadToEndAsync - Duration of single invocation 373.51 ms 240.89 ms 0.64 0.06 False
ReadToEnd - Duration of single invocation 335.79 ms 199.72 ms 0.59 0.04 False
ReadToEndAsync - Duration of single invocation 373.09 ms 239.30 ms 0.64 0.04 False
ReadToEndAsync - Duration of single invocation 373.27 ms 237.56 ms 0.64 0.05 False
ReadToEndAsync - Duration of single invocation 371.69 ms 239.90 ms 0.65 0.04 False
ReadToEnd - Duration of single invocation 335.01 ms 200.47 ms 0.60 0.06 False
ReadToEnd - Duration of single invocation 334.33 ms 202.49 ms 0.61 0.06 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.IO.Tests.StreamReaderReadToEndTests*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.IO.Tests.StreamReaderReadToEndTests*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.IO.Tests.StreamReaderReadToEndTests*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.IO.Tests.StreamReaderReadToEndTests*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.IO.Tests.StreamReaderReadToEndTests.ReadToEnd(LineLengthRange: [ 33, 128])

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 205.28100133333334 < 319.04218378559517.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 173.5255555295636 (T) = (0 -202407178.81271142) / Math.Sqrt((134242533097320.78 / (299)) + (6917795559622.956 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.39829350191549545 = (336388553.9828175 - 202407178.81271142) / 336388553.9828175 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StreamReaderReadToEndTests.ReadToEnd(LineLengthRange: [ 129, 1024])

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 200.23375153846155 < 318.52837103571426.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 215.18426893372228 (T) = (0 -202063371.63866803) / Math.Sqrt((101352737143220.45 / (299)) + (2442799691291.004 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.3997088203082016 = (336608929.92366046 - 202063371.63866803) / 336608929.92366046 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StreamReaderReadToEndTests.ReadToEnd(LineLengthRange: [   9,   32])

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 200.87292083333335 < 318.75179247.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 244.51092795554166 (T) = (0 -202287049.82921058) / Math.Sqrt((79495173275828.1 / (299)) + (1615199230133.8691 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.3984276691549357 = (336263886.2479695 - 202287049.82921058) / 336263886.2479695 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StreamReaderReadToEndTests.ReadToEndAsync(LineLengthRange: [1025, 2048])

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 239.3199618666667 < 356.05808996166667.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 227.10670480084983 (T) = (0 -239885149.7964773) / Math.Sqrt((83448402173559.66 / (299)) + (3405369523476.5645 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.3595202047088123 = (374539761.53520334 - 239885149.7964773) / 374539761.53520334 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StreamReaderReadToEndTests.ReadToEnd(LineLengthRange: [   0,    0])

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 202.30810653333333 < 321.77615324115385.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 192.8713208935557 (T) = (0 -202342435.03560516) / Math.Sqrt((99167916391471.28 / (299)) + (7125620563486.902 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.3985468058263515 = (336422579.5053071 - 202342435.03560516) / 336422579.5053071 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StreamReaderReadToEndTests.ReadToEndAsync(LineLengthRange: [   0,    0])

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 240.01231664285714 < 354.99359514285715.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 309.7663058491486 (T) = (0 -239558698.63362175) / Math.Sqrt((30638467479194.98 / (299)) + (4022003769149.1626 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.3592734284835348 = (373886005.7990675 - 239558698.63362175) / 373886005.7990675 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StreamReaderReadToEndTests.ReadToEndAsync(LineLengthRange: [   0, 1024])

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 240.59624561538462 < 355.3049230696429.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 243.90075846243298 (T) = (0 -240037339.4304575) / Math.Sqrt((74041983157157.05 / (299)) + (2626639659083.974 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.35888881673435663 = (374408286.2628812 - 240037339.4304575) / 374408286.2628812 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StreamReaderReadToEndTests.ReadToEnd(LineLengthRange: [1025, 2048])

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 202.9591337142857 < 318.6343443160714.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 199.8893923193831 (T) = (0 -202020651.74221024) / Math.Sqrt((125454484896758.28 / (299)) + (1573009741317.7986 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.39975554113139766 = (336563959.495766 - 202020651.74221024) / 336563959.495766 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StreamReaderReadToEndTests.ReadToEndAsync(LineLengthRange: [  33,  128])

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 238.51898507142857 < 355.81718516166666.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 252.1793556831597 (T) = (0 -239766139.0462318) / Math.Sqrt((59177222528373.98 / (299)) + (4062717549931.8296 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.35932748984728974 = (374241340.53930503 - 239766139.0462318) / 374241340.53930503 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StreamReaderReadToEndTests.ReadToEndAsync(LineLengthRange: [   1,    8])

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 240.8923302142857 < 354.4736207352564.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 244.38679707640676 (T) = (0 -239772619.32076222) / Math.Sqrt((73315623645913.6 / (299)) + (2740673920792.707 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.359599918994619 = (374410663.6344219 - 239772619.32076222) / 374410663.6344219 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StreamReaderReadToEndTests.ReadToEnd(LineLengthRange: [   0, 1024])

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 199.72462546153847 < 318.9279786466667.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 262.9659499838264 (T) = (0 -201894327.87630352) / Math.Sqrt((65631062450050.6 / (299)) + (1864522205806.8218 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.3987082155205894 = (335767647.4011708 - 201894327.87630352) / 335767647.4011708 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StreamReaderReadToEndTests.ReadToEndAsync(LineLengthRange: [   9,   32])

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 239.3047452857143 < 355.05848082525637.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 359.6892172344352 (T) = (0 -239735454.2687787) / Math.Sqrt((29063989356889.598 / (299)) + (1950235700305.373 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.35846733174234224 = (373691732.51905876 - 239735454.2687787) / 373691732.51905876 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StreamReaderReadToEndTests.ReadToEndAsync(LineLengthRange: [ 129, 1024])

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 237.56054853846155 < 354.7011145872619.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 225.51857908951874 (T) = (0 -239828326.9514067) / Math.Sqrt((77180726240392.39 / (299)) + (4637776494185.508 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.3596685804658135 = (374537809.07060826 - 239828326.9514067) / 374537809.07060826 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StreamReaderReadToEndTests.ReadToEndAsync(LineLengthRange: [   1,    1])

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 239.89712564285713 < 354.7350753229762.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 309.98686571602894 (T) = (0 -239763793.202529) / Math.Sqrt((38053793068033.5 / (299)) + (2805530800404.785 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.35857640532827306 = (373799459.8175599 - 239763793.202529) / 373799459.8175599 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StreamReaderReadToEndTests.ReadToEnd(LineLengthRange: [   1,    8])

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 200.46747693333333 < 318.770073341923.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 229.8345264756889 (T) = (0 -201651110.59224525) / Math.Sqrt((93500015804067.94 / (299)) + (1516770730827.1545 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.40099699390163823 = (336644571.96251917 - 201651110.59224525) / 336644571.96251917 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StreamReaderReadToEndTests.ReadToEnd(LineLengthRange: [   1,    1])

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 202.48849133333334 < 319.2800204520238.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 218.91656724416697 (T) = (0 -201745843.08045745) / Math.Sqrt((101101716584920.75 / (299)) + (1787391510698.8848 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.3995872674336628 = (336011933.35480666 - 201745843.08045745) / 336011933.35480666 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 8, 2023 •
edited
Loading

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 967250cc86b4c613015066b47b882e7424fdb590
Compare 4862343c61f698da986901a182e94df24da1487f
Diff Diff

Improvements in System.Runtime.Intrinsics.Tests.Perf_Vector128Int

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
OnesComplementBenchmark - Duration of single invocation 28.18 ns 6.54 ns 0.23 0.22 False
AddBenchmark - Duration of single invocation 107.97 ns 10.11 ns 0.09 0.15 False
NegateBenchmark - Duration of single invocation 98.67 ns 6.48 ns 0.07 0.15 False
InequalityOperatorBenchmark - Duration of single invocation 33.78 ns 7.81 ns 0.23 0.29 False
SubtractionOperatorBenchmark - Duration of single invocation 105.78 ns 8.79 ns 0.08 0.17 False
AndNotBenchmark - Duration of single invocation 37.46 ns 8.80 ns 0.24 0.23 False
MultiplyBenchmark - Duration of single invocation 107.74 ns 9.10 ns 0.08 0.16 False
AddOperatorBenchmark - Duration of single invocation 104.90 ns 8.73 ns 0.08 0.15 False
EqualsStaticBenchmark - Duration of single invocation 113.21 ns 8.70 ns 0.08 0.18 False
EqualsBenchmark - Duration of single invocation 35.47 ns 8.49 ns 0.24 0.32 False
BitwiseAndBenchmark - Duration of single invocation 36.56 ns 10.15 ns 0.28 0.18 False
EqualityOperatorBenchmark - Duration of single invocation 25.36 ns 9.54 ns 0.38 0.27 False
BitwiseOrBenchmark - Duration of single invocation 36.69 ns 8.92 ns 0.24 0.22 False
OnesComplementOperatorBenchmark - Duration of single invocation 28.38 ns 6.59 ns 0.23 0.21 False
MultiplyOperatorBenchmark - Duration of single invocation 104.99 ns 9.18 ns 0.09 0.16 False
BitwiseOrOperatorBenchmark - Duration of single invocation 35.97 ns 7.08 ns 0.20 0.22 False
ConditionalSelectBenchmark - Duration of single invocation 64.33 ns 11.15 ns 0.17 0.27 False
EqualsAllBenchmark - Duration of single invocation 28.62 ns 7.74 ns 0.27 0.19 False
SubtractBenchmark - Duration of single invocation 107.51 ns 8.93 ns 0.08 0.17 False
ExclusiveOrOperatorBenchmark - Duration of single invocation 35.93 ns 9.04 ns 0.25 0.19 False
XorBenchmark - Duration of single invocation 36.09 ns 8.54 ns 0.24 0.25 False
AllBitsSetBenchmark - Duration of single invocation 15.79 ns 2.91 ns 0.18 0.28 False
UnaryNegateOperatorBenchmark - Duration of single invocation 97.00 ns 7.03 ns 0.07 0.19 False
BitwiseAndOperatorBenchmark - Duration of single invocation 35.99 ns 9.02 ns 0.25 0.23 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Int*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Int*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Int*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Int*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Runtime.Intrinsics.Tests.Perf_Vector128Int.OnesComplementBenchmark

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 6.538107242912199 < 26.960088072438612.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 125.08082633096106 (T) = (0 -6.942261191461414) / Math.Sqrt((2.6374331862131686 / (299)) + (1.0994102360281273 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7638010846112542 = (29.39158793356675 - 6.942261191461414) / 29.39158793356675 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Int.AddBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 10.114138122764476 < 102.24194597453845.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 434.5782456607078 (T) = (0 -9.071435460175753) / Math.Sqrt((12.559289204241121 / (299)) + (0.5797956513977824 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.9178133931859012 = (110.37607965364471 - 9.071435460175753) / 110.37607965364471 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Int.NegateBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 6.480261683088859 < 93.59335093682094.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 463.36085402712666 (T) = (0 -6.845297951580757) / Math.Sqrt((11.296156472041455 / (299)) + (0.15428778910242552 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.93204999450059 = (100.74021188475405 - 6.845297951580757) / 100.74021188475405 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Int.InequalityOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 7.8094498443974585 < 24.123760494375926.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 139.87925903157634 (T) = (0 -8.131604338462243) / Math.Sqrt((2.0422984780241045 / (299)) + (0.4524794881357719 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.6881616945121403 = (26.07634852857683 - 8.131604338462243) / 26.07634852857683 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Int.SubtractionOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.790934494158323 < 99.50467694299691.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 468.07733069164357 (T) = (0 -8.903051649542077) / Math.Sqrt((11.734800619970668 / (299)) + (0.2082678006205642 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.9165817901431393 = (106.72791546137283 - 8.903051649542077) / 106.72791546137283 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Int.AndNotBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.803436535390986 < 35.505349611723844.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 199.2125620447537 (T) = (0 -8.996794149210238) / Math.Sqrt((4.315943001084505 / (299)) + (0.42556588306211673 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.772397128464218 = (39.5284738215433 - 8.996794149210238) / 39.5284738215433 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Int.MultiplyBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 9.10490123389449 < 102.54271667970248.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 486.6500539460351 (T) = (0 -9.299352357104938) / Math.Sqrt((10.79432619219456 / (299)) + (0.3285847151349752 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.9157070613480404 = (110.32184315582339 - 9.299352357104938) / 110.32184315582339 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Int.AddOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.73125285610882 < 99.92988705394258.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 514.1090446246819 (T) = (0 -8.88718737191954) / Math.Sqrt((8.29132059467436 / (299)) + (0.4199583159243637 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.9171975034136521 = (107.3299446068251 - 8.88718737191954) / 107.3299446068251 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Int.EqualsStaticBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.700874522925906 < 103.71964355596184.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 451.5543078361051 (T) = (0 -9.047528240394438) / Math.Sqrt((14.050978277067218 / (299)) + (0.3030930231415399 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.9202409360198377 = (113.43573744351795 - 9.047528240394438) / 113.43573744351795 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Int.EqualsBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.48671662209191 < 34.11840744403464.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 144.2771726177806 (T) = (0 -9.001605611887102) / Math.Sqrt((5.004618073603593 / (299)) + (1.0872029101465341 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7619255857364501 = (37.810050440457104 - 9.001605611887102) / 37.810050440457104 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Int.BitwiseAndBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 10.148298443691846 < 34.67694501623763.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 75.96933406333964 (T) = (0 -9.70232967483359) / Math.Sqrt((5.1163698481495405 / (299)) + (5.603808256031338 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7430091203525742 = (37.75359533437347 - 9.70232967483359) / 37.75359533437347 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Int.EqualityOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 9.535467645688248 < 24.11838102822518.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 147.67106140354306 (T) = (0 -8.174448872431574) / Math.Sqrt((1.96334530215272 / (299)) + (0.3886862313534559 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.6875380511704186 = (26.16142190449553 - 8.174448872431574) / 26.16142190449553 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Int.BitwiseOrBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.915206733058652 < 34.334566374515575.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 177.19663322874322 (T) = (0 -9.056934267134224) / Math.Sqrt((4.175911537029901 / (299)) + (0.5818358568250739 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7605150125740149 = (37.81838003492117 - 9.056934267134224) / 37.81838003492117 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Int.OnesComplementOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 6.589025542060214 < 26.992854425950107.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 182.09009594223028 (T) = (0 -6.813907906108732) / Math.Sqrt((2.676885708830819 / (299)) + (0.3054259574302941 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.768614096634024 = (29.44824125837684 - 6.813907906108732) / 29.44824125837684 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Int.MultiplyOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 9.17935134836876 < 99.882257542629.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 561.402341477356 (T) = (0 -9.183055736646942) / Math.Sqrt((7.478153439000106 / (299)) + (0.25352454293386006 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.9142367165810209 = (107.07444223870247 - 9.183055736646942) / 107.07444223870247 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Int.BitwiseOrOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 7.079883126735476 < 34.17064543244381.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 154.08375459139705 (T) = (0 -9.309881276219825) / Math.Sqrt((2.8975454513835834 / (299)) + (1.1427971388080684 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7532098033086164 = (37.72386991474394 - 9.309881276219825) / 37.72386991474394 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Int.ConditionalSelectBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 11.146386447424812 < 60.7541660415444.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 222.35641319119432 (T) = (0 -11.545600261132474) / Math.Sqrt((8.450094131752119 / (299)) + (1.3820291122146329 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.8222160096944778 = (64.94173204961447 - 11.545600261132474) / 64.94173204961447 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Int.EqualsAllBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 7.736938187702681 < 26.826574384249103.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 178.87984571719338 (T) = (0 -8.014030042651278) / Math.Sqrt((2.721572980928129 / (299)) + (0.1924843901832332 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7194354176555632 = (28.563940521946584 - 8.014030042651278) / 28.563940521946584 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Int.SubtractBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.928145694310741 < 102.2512127824946.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 471.7572572674077 (T) = (0 -9.107146681455731) / Math.Sqrt((9.316291498707432 / (299)) + (0.6694998425430156 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.9169275113969737 = (109.6289136705116 - 9.107146681455731) / 109.6289136705116 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Int.ExclusiveOrOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 9.035197923094511 < 34.25051578856198.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 164.68717947170424 (T) = (0 -9.011256604213203) / Math.Sqrt((4.470377318837156 / (299)) + (0.7366590892267185 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7618046340860912 = (37.83136825369706 - 9.011256604213203) / 37.83136825369706 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Int.XorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.540521429507686 < 34.34082587766728.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 153.24128088401622 (T) = (0 -9.109172793366554) / Math.Sqrt((6.117772321846491 / (299)) + (0.707162702493048 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7601890292896661 = (37.98480430809592 - 9.109172793366554) / 37.98480430809592 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Int.AllBitsSetBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 2.9083634554771205 < 15.14035018184836.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 143.6688054608576 (T) = (0 -3.3338819042754357) / Math.Sqrt((0.5063583577520426 / (299)) + (0.29288671859438337 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7932315472318913 = (16.12374547298273 - 3.3338819042754357) / 16.12374547298273 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Int.UnaryNegateOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 7.0316593617053895 < 92.12833226620918.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 354.61360341366475 (T) = (0 -7.1271320464089865) / Math.Sqrt((13.477524680437265 / (299)) + (1.0274655907424424 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.9279162561899149 = (98.87294512874409 - 7.1271320464089865) / 98.87294512874409 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Int.BitwiseAndOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 9.021956391554713 < 34.22208132902343.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 183.7385974489271 (T) = (0 -8.92216734408555) / Math.Sqrt((5.8449560151202675 / (299)) + (0.2549724836887255 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7649479948616588 = (37.958269442689314 - 8.92216734408555) / 37.958269442689314 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 8, 2023 •
edited
Loading

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 967250cc86b4c613015066b47b882e7424fdb590
Compare 4862343c61f698da986901a182e94df24da1487f
Diff Diff

Improvements in System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int16>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
UnaryNegateOperatorBenchmark - Duration of single invocation 155.56 ns 5.80 ns 0.04 0.22 False
MultiplyOperatorBenchmark - Duration of single invocation 162.69 ns 10.61 ns 0.07 0.18 False
AndNotBenchmark - Duration of single invocation 37.15 ns 8.43 ns 0.23 0.21 False
SubtractBenchmark - Duration of single invocation 167.65 ns 8.67 ns 0.05 0.17 False
AddBenchmark - Duration of single invocation 166.39 ns 8.61 ns 0.05 0.14 False
OnesComplementBenchmark - Duration of single invocation 28.12 ns 6.66 ns 0.24 0.20 False
LessThanBenchmark - Duration of single invocation 179.55 ns 8.73 ns 0.05 0.17 False
AllBitsSetBenchmark - Duration of single invocation 15.89 ns 3.32 ns 0.21 0.27 False
BitwiseAndBenchmark - Duration of single invocation 35.92 ns 8.34 ns 0.23 0.20 False
AddOperatorBenchmark - Duration of single invocation 164.07 ns 8.66 ns 0.05 0.17 False
EqualsBenchmark - Duration of single invocation 45.68 ns 8.46 ns 0.19 0.30 False
ConditionalSelectBenchmark - Duration of single invocation 61.72 ns 10.98 ns 0.18 0.25 False
BitwiseAndOperatorBenchmark - Duration of single invocation 39.64 ns 9.26 ns 0.23 0.22 False
MultiplyBenchmark - Duration of single invocation 164.85 ns 7.21 ns 0.04 0.20 False
EqualityOperatorBenchmark - Duration of single invocation 25.68 ns 8.25 ns 0.32 0.24 False
BitwiseOrOperatorBenchmark - Duration of single invocation 35.98 ns 9.26 ns 0.26 0.20 False
EqualsAllBenchmark - Duration of single invocation 28.73 ns 7.40 ns 0.26 0.23 False
BitwiseOrBenchmark - Duration of single invocation 38.21 ns 8.73 ns 0.23 0.22 False
SubtractionOperatorBenchmark - Duration of single invocation 163.44 ns 9.22 ns 0.06 0.21 False
NegateBenchmark - Duration of single invocation 154.58 ns 6.91 ns 0.04 0.18 False
InequalityOperatorBenchmark - Duration of single invocation 25.16 ns 7.77 ns 0.31 0.28 False
ExclusiveOrOperatorBenchmark - Duration of single invocation 35.82 ns 8.79 ns 0.25 0.20 False
XorBenchmark - Duration of single invocation 36.00 ns 8.72 ns 0.24 0.18 False
EqualsStaticBenchmark - Duration of single invocation 167.34 ns 8.77 ns 0.05 0.16 False
OnesComplementOperatorBenchmark - Duration of single invocation 28.27 ns 5.20 ns 0.18 0.23 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int16>.UnaryNegateOperatorBenchmark

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 5.797492716776347 < 148.14219949809925.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 436.96500445952506 (T) = (0 -6.816724573277905) / Math.Sqrt((30.563046743148377 / (299)) + (0.7226297237113445 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.95648709105235 = (156.65982206519521 - 6.816724573277905) / 156.65982206519521 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;.MultiplyOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 10.612743677714855 < 154.5079124182743.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 460.73132336115 (T) = (0 -9.25147520184384) / Math.Sqrt((21.6283491977638 / (299)) + (1.9366627521380368 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.9437596636944106 = (164.49893100878177 - 9.25147520184384) / 164.49893100878177 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;.AndNotBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.429801851490371 < 35.31405535211145.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 195.05532719677998 (T) = (0 -8.735380140107125) / Math.Sqrt((5.215901565790839 / (299)) + (0.3652894547079859 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7800187182916921 = (39.70965198616361 - 8.735380140107125) / 39.70965198616361 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;.SubtractBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.674545128655756 < 158.42788932379324.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 597.2348930284695 (T) = (0 -8.974107561320357) / Math.Sqrt((18.316086956896346 / (299)) + (0.44166246712424956 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.9464951233938067 = (167.72503985705075 - 8.974107561320357) / 167.72503985705075 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;.AddBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.607516470872614 < 158.08924209228138.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 441.3406296940216 (T) = (0 -8.937807614598078) / Math.Sqrt((34.93673403329677 / (299)) + (0.5683082797522442 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.9466123743130137 = (167.41346893755497 - 8.937807614598078) / 167.41346893755497 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;.OnesComplementBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 6.657557677019017 < 26.929851226322295.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 165.47887442386678 (T) = (0 -6.8491602945100825) / Math.Sqrt((1.8559441353021895 / (299)) + (0.580019766319624 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7669236704337519 = (29.38591107581055 - 6.8491602945100825) / 29.38591107581055 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;.LessThanBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.73171192204182 < 159.9072391886915.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 545.1876212481893 (T) = (0 -8.922487869367648) / Math.Sqrt((23.396046277017216 / (299)) + (0.487283463733008 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.947887464000435 = (171.2157679189153 - 8.922487869367648) / 171.2157679189153 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;.AllBitsSetBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 3.317142846576915 < 15.121130058038998.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 122.68424878038495 (T) = (0 -3.2087742391624476) / Math.Sqrt((0.646205227528171 / (299)) + (0.42363695540678903 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.8016560127738565 = (16.177824616906268 - 3.2087742391624476) / 16.177824616906268 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;.BitwiseAndBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.34204426674775 < 34.26237491119005.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 116.34432893541451 (T) = (0 -9.148404637769834) / Math.Sqrt((3.747692732771386 / (299)) + (2.2619085549094176 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7579985271417012 = (37.803094872594365 - 9.148404637769834) / 37.803094872594365 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;.AddOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.663142204874518 < 155.82687252789603.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 402.24632976195596 (T) = (0 -9.118038992302814) / Math.Sqrt((40.870694354734994 / (299)) + (0.6388793777665475 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.9447575798057202 = (165.05502402385628 - 9.118038992302814) / 165.05502402385628 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;.EqualsBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.457481124227273 < 35.90730463982028.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 177.72026249024685 (T) = (0 -8.80762391290601) / Math.Sqrt((4.948050754388897 / (299)) + (0.5762651614644323 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7740058280770896 = (38.972792253733026 - 8.80762391290601) / 38.972792253733026 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;.ConditionalSelectBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 10.98316593464577 < 59.54160736229233.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 250.22916979632762 (T) = (0 -11.156907543405536) / Math.Sqrt((9.99307712391097 / (299)) + (0.6174146775771616 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.8287496648765683 = (65.14969757789963 - 11.156907543405536) / 65.14969757789963 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;.BitwiseAndOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 9.256553083438222 < 34.21307248604469.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 196.86741045052221 (T) = (0 -9.036201197307266) / Math.Sqrt((4.39071117390222 / (299)) + (0.32131555255011035 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7616836158201861 = (37.91682736546259 - 9.036201197307266) / 37.91682736546259 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;.MultiplyBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 7.211964032757322 < 156.67376334525207.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 478.92387175908146 (T) = (0 -8.97642350542312) / Math.Sqrt((27.96841015424331 / (299)) + (0.752537082990806 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.946407229458308 = (167.49317892494494 - 8.97642350542312) / 167.49317892494494 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;.EqualityOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.25392344159748 < 24.4485278368123.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 154.41179662105657 (T) = (0 -7.90248356872268) / Math.Sqrt((1.459736108823369 / (299)) + (0.4054275497114926 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.6942821839036125 = (25.848946815160964 - 7.90248356872268) / 25.848946815160964 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;.BitwiseOrOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 9.262019317448427 < 35.001849669738014.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 139.8438789661392 (T) = (0 -9.246862650563651) / Math.Sqrt((5.550770756321951 / (299)) + (1.0900804431878788 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7555261498370516 = (37.82352445629815 - 9.246862650563651) / 37.82352445629815 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;.EqualsAllBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 7.3987632230446 < 27.35503322698963.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 206.42581922945752 (T) = (0 -7.889105525047976) / Math.Sqrt((1.4896788020455372 / (299)) + (0.22679474985681586 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.721549508196082 = (28.332165886793987 - 7.889105525047976) / 28.332165886793987 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;.BitwiseOrBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.734976651483292 < 34.604220545351495.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 178.51098909463306 (T) = (0 -8.988212519813125) / Math.Sqrt((3.6040697059812943 / (299)) + (0.6612880859871718 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7624719641628986 = (37.840638424582906 - 8.988212519813125) / 37.840638424582906 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;.SubtractionOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 9.22134164314242 < 155.0490270141974.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 566.6662476767245 (T) = (0 -9.13031856552568) / Math.Sqrt((17.991200148505964 / (299)) + (0.7333035746985234 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.9447037454604352 = (165.1164014914043 - 9.13031856552568) / 165.1164014914043 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;.NegateBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 6.90664399302454 < 146.85352106446635.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 670.3492434951999 (T) = (0 -6.796335609030063) / Math.Sqrt((13.549438850652718 / (299)) + (0.27325381276003247 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.9570870608158925 = (158.37497356850864 - 6.796335609030063) / 158.37497356850864 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;.InequalityOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 7.769789287031393 < 24.041148296100733.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 115.33199910684247 (T) = (0 -8.092408846540566) / Math.Sqrt((1.9697924712343862 / (299)) + (0.8144817755136742 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.6878984421373177 = (25.92876787273534 - 8.092408846540566) / 25.92876787273534 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;.ExclusiveOrOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.78820160420893 < 34.28207949550812.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 160.13297843044452 (T) = (0 -9.108260666645865) / Math.Sqrt((3.536496299145432 / (299)) + (0.947066021412806 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7586826373863507 = (37.743909381391056 - 9.108260666645865) / 37.743909381391056 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;.XorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.717152354051183 < 34.19481458234288.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 185.0309006585702 (T) = (0 -8.9546143817855) / Math.Sqrt((3.4682901580059835 / (299)) + (0.5934278201063907 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7628166845918829 = (37.75398099304522 - 8.9546143817855) / 37.75398099304522 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;.EqualsStaticBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.769286493825884 < 159.07661508803596.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 418.4218665254776 (T) = (0 -9.049139327034275) / Math.Sqrt((40.35814615273128 / (299)) + (0.6846866571310771 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.947036826755587 = (170.85719704283136 - 9.049139327034275) / 170.85719704283136 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;.OnesComplementOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 5.204649801606719 < 26.894786760656785.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 147.48363958263246 (T) = (0 -6.870122557950027) / Math.Sqrt((1.8122031061852855 / (299)) + (0.804888411370737 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7657443909453361 = (29.327462363331804 - 6.870122557950027) / 29.327462363331804 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 8, 2023 •
edited
Loading

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 967250cc86b4c613015066b47b882e7424fdb590
Compare 4862343c61f698da986901a182e94df24da1487f
Diff Diff

Improvements in System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt16>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
BitwiseOrBenchmark - Duration of single invocation 36.39 ns 8.53 ns 0.23 0.20 False
BitwiseAndOperatorBenchmark - Duration of single invocation 36.62 ns 8.59 ns 0.23 0.20 False
EqualsStaticBenchmark - Duration of single invocation 163.06 ns 8.93 ns 0.05 0.19 False
LessThanOrEqualBenchmark - Duration of single invocation 162.93 ns 8.80 ns 0.05 0.16 False
OnesComplementOperatorBenchmark - Duration of single invocation 28.41 ns 6.52 ns 0.23 0.21 False
ExclusiveOrOperatorBenchmark - Duration of single invocation 36.22 ns 8.55 ns 0.24 0.20 False
AddBenchmark - Duration of single invocation 159.87 ns 8.78 ns 0.05 0.14 False
BitwiseAndBenchmark - Duration of single invocation 37.02 ns 8.60 ns 0.23 0.23 False
EqualsAllBenchmark - Duration of single invocation 27.80 ns 7.70 ns 0.28 0.19 False
MultiplyOperatorBenchmark - Duration of single invocation 158.44 ns 8.85 ns 0.06 0.17 False
SubtractionOperatorBenchmark - Duration of single invocation 159.80 ns 8.64 ns 0.05 0.17 False
EqualsBenchmark - Duration of single invocation 37.20 ns 8.56 ns 0.23 0.25 False
BitwiseOrOperatorBenchmark - Duration of single invocation 36.89 ns 8.71 ns 0.24 0.24 False
ConditionalSelectBenchmark - Duration of single invocation 62.87 ns 11.02 ns 0.18 0.25 False
AllBitsSetBenchmark - Duration of single invocation 15.80 ns 0.71 ns 0.05 0.34 False
XorBenchmark - Duration of single invocation 35.97 ns 9.75 ns 0.27 0.17 False
OnesComplementBenchmark - Duration of single invocation 28.41 ns 3.84 ns 0.14 0.20 False
MultiplyBenchmark - Duration of single invocation 161.70 ns 8.65 ns 0.05 0.17 False
UnaryNegateOperatorBenchmark - Duration of single invocation 150.98 ns 6.75 ns 0.04 0.20 False
AndNotBenchmark - Duration of single invocation 36.97 ns 8.63 ns 0.23 0.18 False
EqualityOperatorBenchmark - Duration of single invocation 25.21 ns 7.69 ns 0.31 0.27 False
AddOperatorBenchmark - Duration of single invocation 157.54 ns 8.88 ns 0.06 0.20 False
InequalityOperatorBenchmark - Duration of single invocation 25.12 ns 7.87 ns 0.31 0.25 False
NegateBenchmark - Duration of single invocation 153.00 ns 6.63 ns 0.04 0.21 False
SubtractBenchmark - Duration of single invocation 163.38 ns 8.66 ns 0.05 0.16 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt16>.BitwiseOrBenchmark

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.534665894357799 < 34.55726837221139.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 227.18983112655502 (T) = (0 -8.940314696648706) / Math.Sqrt((3.100864980122318 / (299)) + (0.2655639925831259 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.762835838474066 = (37.69673562449729 - 8.940314696648706) / 37.69673562449729 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;.BitwiseAndOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.585390318366551 < 34.5281039557567.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 149.86370291419362 (T) = (0 -8.8205712788103) / Math.Sqrt((3.957818607356038 / (299)) + (1.1360710669036431 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7666895430761335 = (37.80615491953115 - 8.8205712788103) / 37.80615491953115 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;.EqualsStaticBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.932172174924904 < 154.88318032556037.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 514.8017982546085 (T) = (0 -8.986078206914016) / Math.Sqrt((26.243258030336623 / (299)) + (0.3789536288139924 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.9466235083055111 = (168.35273210438115 - 8.986078206914016) / 168.35273210438115 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;.LessThanOrEqualBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.801399180248241 < 154.9278210663386.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 471.8516237090127 (T) = (0 -9.240900310969268) / Math.Sqrt((27.5210522178098 / (299)) + (1.0425637155311938 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.9452274525460636 = (168.71408653652324 - 9.240900310969268) / 168.71408653652324 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;.OnesComplementOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 6.520047458171311 < 27.01013782501076.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 214.85520313522832 (T) = (0 -6.628806066006304) / Math.Sqrt((2.040569111076962 / (299)) + (0.20917409605172596 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7748605262771784 = (29.44310900436455 - 6.628806066006304) / 29.44310900436455 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;.ExclusiveOrOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.554435354288147 < 34.326925735544904.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 198.80508849638832 (T) = (0 -8.852099849461329) / Math.Sqrt((3.1008110553541317 / (299)) + (0.5035093049354581 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7653143253312091 = (37.718961167758515 - 8.852099849461329) / 37.718961167758515 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;.AddBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.77974105474856 < 152.27328192918765.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 594.7155437236352 (T) = (0 -8.896400356397963) / Math.Sqrt((17.674377362953123 / (299)) + (0.4101681749456836 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.9456857962235301 = (163.79509847941642 - 8.896400356397963) / 163.79509847941642 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;.BitwiseAndBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.600760202631777 < 34.793131548540515.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 133.37369054399423 (T) = (0 -9.011600246977043) / Math.Sqrt((4.5119595970275475 / (299)) + (1.502875441564132 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7625194995412884 = (37.94669553740393 - 9.011600246977043) / 37.94669553740393 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;.EqualsAllBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 7.697212640225842 < 26.624047933986517.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 166.60659491402384 (T) = (0 -8.059992863480591) / Math.Sqrt((1.2152369261685179 / (299)) + (0.5126070113333899 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7166486712477764 = (28.445226987195984 - 8.059992863480591) / 28.445226987195984 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;.MultiplyOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.850650658262726 < 150.9704972032628.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 564.4541419689975 (T) = (0 -8.962919565224837) / Math.Sqrt((19.348814130497328 / (299)) + (0.39784529432992594 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.94455503090961 = (161.654334239287 - 8.962919565224837) / 161.654334239287 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;.SubtractionOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.635624859126711 < 151.95239189450518.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 574.9952693544049 (T) = (0 -8.75449275702955) / Math.Sqrt((19.789500067791927 / (299)) + (0.28702822724358923 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.9464082073785139 = (163.35510213031546 - 8.75449275702955) / 163.35510213031546 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;.EqualsBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.557445763195245 < 35.08381602043588.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 205.14192259289376 (T) = (0 -8.779534721354576) / Math.Sqrt((4.279635881944847 / (299)) + (0.33274047350243385 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7736307165292101 = (38.78412559664909 - 8.779534721354576) / 38.78412559664909 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;.BitwiseOrOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.710581021090604 < 34.86001207291668.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 157.79708112383148 (T) = (0 -9.284464043979552) / Math.Sqrt((4.285989928847612 / (299)) + (0.8678414216198121 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7547837349854458 = (37.86234996862259 - 9.284464043979552) / 37.86234996862259 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;.ConditionalSelectBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 11.015483062831843 < 59.513647775180225.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 209.47283209294142 (T) = (0 -11.4496033949698) / Math.Sqrt((13.45660879819404 / (299)) + (0.9727997493866715 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.8242390114572847 = (65.14303025888591 - 11.4496033949698) / 65.14303025888591 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;.AllBitsSetBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 0.7120643336255609 < 15.098560616096368.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 121.93397591384247 (T) = (0 -3.1864863808625463) / Math.Sqrt((1.220458533653701 / (299)) + (0.3512612086357253 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.8044372309192054 = (16.293931589535255 - 3.1864863808625463) / 16.293931589535255 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;.XorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 9.753639901535196 < 34.27961173525684.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 217.80522840863298 (T) = (0 -8.95175196280583) / Math.Sqrt((3.0968734646554745 / (299)) + (0.3312375571076126 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7624673021288102 = (37.68639872755634 - 8.95175196280583) / 37.68639872755634 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;.OnesComplementBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 3.8382777381316293 < 26.98784432784923.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 146.85366169685176 (T) = (0 -6.7979729422883715) / Math.Sqrt((1.8415754760798604 / (299)) + (0.8207628093433883 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.768529582668455 = (29.36864684765026 - 6.7979729422883715) / 29.36864684765026 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;.MultiplyBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.648971285344317 < 154.09750607868165.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 463.617816061096 (T) = (0 -8.968072938690806) / Math.Sqrt((30.06910978865156 / (299)) + (0.5906953185895137 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.9456175472162316 = (164.90747437135641 - 8.968072938690806) / 164.90747437135641 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;.UnaryNegateOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 6.750822129053401 < 143.19438278174715.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 412.96871824920527 (T) = (0 -6.862622814187031) / Math.Sqrt((35.29290035503373 / (299)) + (0.5647013399146253 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.955949538528224 = (155.79003226978784 - 6.862622814187031) / 155.79003226978784 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;.AndNotBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.62567530794508 < 35.405459118219746.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 166.9074635412185 (T) = (0 -9.065468253629973) / Math.Sqrt((3.876114408489869 / (299)) + (0.9510164318593113 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.7703580427753313 = (39.476532786910674 - 9.065468253629973) / 39.476532786910674 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;.EqualityOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 7.692761784810457 < 23.937981823989425.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 72.25668981137831 (T) = (0 -8.08965300193281) / Math.Sqrt((2.524861905903424 / (299)) + (2.507871393816212 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.6894908335733411 = (26.052863736773308 - 8.08965300193281) / 26.052863736773308 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;.AddOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.875303481777816 < 150.0100634036404.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 535.363394071064 (T) = (0 -8.855653774116755) / Math.Sqrt((19.499121404132108 / (299)) + (0.7315979018887369 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.9450014160223194 = (161.01603229840458 - 8.855653774116755) / 161.01603229840458 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;.InequalityOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 7.8728991852050285 < 23.90598076423835.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 154.50208853487774 (T) = (0 -8.038644662999904) / Math.Sqrt((1.5774199244268179 / (299)) + (0.378326172860391 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.6893100937413444 = (25.87353016968492 - 8.038644662999904) / 25.87353016968492 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;.NegateBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 6.627529191170205 < 145.38461174598396.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 454.5580303394452 (T) = (0 -6.943060840562624) / Math.Sqrt((25.364419982622312 / (299)) + (1.1484756393347306 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.9558327799797981 = (157.19940800862955 - 6.943060840562624) / 157.19940800862955 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;.SubtractBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 8.656340704578698 < 154.69397789499564.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 568.1377569101853 (T) = (0 -8.87508111047565) / Math.Sqrt((20.802370941822282 / (299)) + (0.32325330042580114 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.9465239814560359 = (165.96376005777645 - 8.87508111047565) / 165.96376005777645 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 8, 2023 •
edited
Loading

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 967250cc86b4c613015066b47b882e7424fdb590
Compare 4862343c61f698da986901a182e94df24da1487f
Diff Diff

Improvements in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_SliceSlice

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Count - Duration of single invocation 36.42 secs 31.26 secs 0.86 0.03 False
Count - Duration of single invocation 22.72 secs 17.11 secs 0.75 0.04 False
Count - Duration of single invocation 23.05 secs 19.56 secs 0.85 0.04 False
Count - Duration of single invocation 36.92 secs 30.95 secs 0.84 0.04 False
Count - Duration of single invocation 22.73 secs 17.90 secs 0.79 0.02 False
Count - Duration of single invocation 36.49 secs 30.44 secs 0.83 0.02 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_SliceSlice*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_SliceSlice*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_SliceSlice*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_SliceSlice*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_SliceSlice.Count(Options: IgnoreCase, NonBacktracking)

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 31.257986492 < 34.50734672745.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 75.64934282489045 (T) = (0 -31377499507.166676) / Math.Sqrt((2.461793330939714E+17 / (299)) + (1.85050214546129E+17 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.14262273611427617 = (36597074390.5204 - 31377499507.166676) / 36597074390.5204 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_SliceSlice.Count(Options: None)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 17.106158953 < 21.80475626045.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 128.94013777355104 (T) = (0 -17671056934.684395) / Math.Sqrt((1.2355324501332203E+17 / (299)) + (62522636036788136 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.23352499926465783 = (23054968417.405792 - 17671056934.684395) / 23054968417.405792 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_SliceSlice.Count(Options: NonBacktracking)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 19.559059511 < 21.98613652.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 62.15272263555381 (T) = (0 -18092768118.624115) / Math.Sqrt((1.095072588141605E+17 / (299)) + (3.155548896711343E+17 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.22423661630243388 = (23322534291.819115 - 18092768118.624115) / 23322534291.819115 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_SliceSlice.Count(Options: IgnoreCase, Compiled)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 30.954411300333334 < 34.5176400145.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 59.990620552996184 (T) = (0 -31324703054.998222) / Math.Sqrt((3.681567183125429E+17 / (299)) + (3.018392702615492E+17 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.14349977927766397 = (36572907160.00084 - 31324703054.998222) / 36572907160.00084 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_SliceSlice.Count(Options: Compiled)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 17.900534251666667 < 21.63425616834167.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 117.83212859210037 (T) = (0 -17716018068.80142) / Math.Sqrt((1.0826414019997408E+17 / (299)) + (78692725320449250 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.23085425039513022 = (23033369264.411327 - 17716018068.80142) / 23033369264.411327 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_SliceSlice.Count(Options: IgnoreCase)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 30.435010664333333 < 34.46722994964166.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 83.85443704370766 (T) = (0 -31107285857.099297) / Math.Sqrt((2.3976784708462317E+17 / (299)) + (1.5506716769069178E+17 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.1472170012090775 = (36477375723.01895 - 31107285857.099297) / 36477375723.01895 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 967250cc86b4c613015066b47b882e7424fdb590
Compare 4862343c61f698da986901a182e94df24da1487f
Diff Diff

Improvements in System.Perf_Convert

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ToBase64CharArray - Duration of single invocation 19.50 μs 13.38 μs 0.69 0.12 False
ToBase64String - Duration of single invocation 19.75 μs 13.62 μs 0.69 0.11 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Perf_Convert*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Perf_Convert*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Perf_Convert*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Perf_Convert*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Perf_Convert.ToBase64CharArray(binaryDataSize: 1024, formattingOptions: None)

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 13.384055485457656 < 18.531912510203135.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 125.01598308239548 (T) = (0 -13440.971067647717) / Math.Sqrt((732311.7972135006 / (299)) + (27422.756133992923 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.3387157445952471 = (20325.557364768785 - 13440.971067647717) / 20325.557364768785 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Perf_Convert.ToBase64String(formattingOptions: None)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 13.624289609098936 < 18.990998091763856.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 136.29245118864617 (T) = (0 -13644.84337497295) / Math.Sqrt((645542.4807600852 / (299)) + (18060.293190902834 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.33498770032401815 = (20518.18196689176 - 13644.84337497295) / 20518.18196689176 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 967250cc86b4c613015066b47b882e7424fdb590
Compare 4862343c61f698da986901a182e94df24da1487f
Diff Diff

Improvements in System.Globalization.Tests.StringSearch

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IndexOf_Word_NotFound - Duration of single invocation 1.14 μs 821.88 ns 0.72 0.02 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Globalization.Tests.StringSearch*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Globalization.Tests.StringSearch*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Globalization.Tests.StringSearch*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Globalization.Tests.StringSearch*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Globalization.Tests.StringSearch.IndexOf_Word_NotFound(Options: (en-US, OrdinalIgnoreCase, False))

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 821.882718752151 < 1.0594851588065322.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 129.53598027753875 (T) = (0 -845.0216652092315) / Math.Sqrt((640.1036766198185 / (299)) + (129.22932548946136 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.25317138053873145 = (1131.4800252550517 - 845.0216652092315) / 1131.4800252550517 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 8, 2023 •
edited
Loading

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 967250cc86b4c613015066b47b882e7424fdb590
Compare 4862343c61f698da986901a182e94df24da1487f
Diff Diff

Improvements in System.Text.Json.Reader.Tests.Perf_Base64

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ReadBase64EncodedByteArray_NoEscaping - Duration of single invocation 2.36 μs 1.84 μs 0.78 0.03 False
ReadBase64EncodedByteArray_HeavyEscaping - Duration of single invocation 2.35 μs 1.83 μs 0.78 0.06 False
ReadBase64EncodedByteArray_HeavyEscaping - Duration of single invocation 17.63 μs 12.18 μs 0.69 0.03 False
ReadBase64EncodedByteArray_NoEscaping - Duration of single invocation 17.64 μs 12.45 μs 0.71 0.05 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Reader.Tests.Perf_Base64*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Reader.Tests.Perf_Base64*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Reader.Tests.Perf_Base64*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Reader.Tests.Perf_Base64*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Text.Json.Reader.Tests.Perf_Base64.ReadBase64EncodedByteArray_NoEscaping(NumberOfBytes: 100)

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.8381488983931262 < 2.2329261175796895.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 138.97287789192703 (T) = (0 -1843.0251339342972) / Math.Sqrt((1884.3982771970434 / (299)) + (323.90478734202566 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.21500661507915508 = (2347.822503141398 - 1843.0251339342972) / 2347.822503141398 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Reader.Tests.Perf_Base64.ReadBase64EncodedByteArray_HeavyEscaping(NumberOfBytes: 100)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.8343455022667625 < 2.233442625826211.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 98.2347282187325 (T) = (0 -1851.27538147514) / Math.Sqrt((2562.3177240409273 / (299)) + (809.3747802201915 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.21227443843243923 = (2350.1527331309812 - 1851.27538147514) / 2350.1527331309812 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Reader.Tests.Perf_Base64.ReadBase64EncodedByteArray_HeavyEscaping(NumberOfBytes: 1000)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 12.175018769470404 < 16.78440851638145.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 274.77114044944636 (T) = (0 -12294.562636868739) / Math.Sqrt((61026.64659566284 / (299)) + (7110.294059666494 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.29642661512699175 = (17474.456682422475 - 12294.562636868739) / 17474.456682422475 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Reader.Tests.Perf_Base64.ReadBase64EncodedByteArray_NoEscaping(NumberOfBytes: 1000)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 12.449536512066368 < 16.745592988080173.
IsChangePoint: Marked as a change because one of 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 137.70606226629297 (T) = (0 -12312.07298096331) / Math.Sqrt((222462.60936897522 / (299)) + (32382.881679051115 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.29745539248767866 = (17524.96973047705 - 12312.07298096331) / 17524.96973047705 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 967250cc86b4c613015066b47b882e7424fdb590
Compare 4862343c61f698da986901a182e94df24da1487f
Diff Diff

Improvements in System.Text.Json.Serialization.Tests.ReadJson<ImmutableDictionary<String, String>>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DeserializeFromString - Duration of single invocation 466.33 μs 431.53 μs 0.93 0.01 False
DeserializeFromReader - Duration of single invocation 516.31 μs 482.21 μs 0.93 0.01 False
DeserializeFromString - Duration of single invocation 467.82 μs 433.03 μs 0.93 0.01 False
DeserializeFromReader - Duration of single invocation 515.11 μs 481.54 μs 0.93 0.01 False
DeserializeFromUtf8Bytes - Duration of single invocation 443.84 μs 413.90 μs 0.93 0.01 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;ImmutableDictionary&lt;String, String&gt;&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;ImmutableDictionary&lt;String, String&gt;&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;ImmutableDictionary&lt;String, String&gt;&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;ImmutableDictionary&lt;String, String&gt;&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.ReadJson<ImmutableDictionary<String, String>>.DeserializeFromString(Mode: SourceGen)

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 431.53024094387763 < 442.05583842842975.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 1/18/2023 3:20:52 PM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 37.935039603494886 (T) = (0 -436313.09987165255) / Math.Sqrt((106372727.55330403 / (299)) + (21686148.476690434 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.07286846110091487 = (470605.39046028897 - 436313.09987165255) / 470605.39046028897 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;ImmutableDictionary&lt;String, String&gt;&gt;.DeserializeFromReader(Mode: SourceGen)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 482.21019945276487 < 497.81798405494504.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 1/26/2023 2:33:00 AM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 39.98356709118265 (T) = (0 -483990.0627805959) / Math.Sqrt((104308199.37730998 / (299)) + (25042881.845114842 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.07198724426674372 = (521533.8472348669 - 483990.0627805959) / 521533.8472348669 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;ImmutableDictionary&lt;String, String&gt;&gt;.DeserializeFromString(Mode: Reflection)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 433.02896874999993 < 440.4942909909529.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 1/19/2023 10:17:02 AM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 43.59268031470125 (T) = (0 -434089.30538526335) / Math.Sqrt((116671989.17861344 / (299)) + (14502546.724573452 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.07744519109790338 = (470529.5568312731 - 434089.30538526335) / 470529.5568312731 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;ImmutableDictionary&lt;String, String&gt;&gt;.DeserializeFromReader(Mode: Reflection)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 481.5376478566859 < 497.2527023785769.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 1/26/2023 7:47:24 AM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 34.891967196998266 (T) = (0 -483800.44744853134) / Math.Sqrt((116856127.01591164 / (299)) + (33417388.86288281 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.07037592748449993 = (520425.90306358994 - 483800.44744853134) / 520425.90306358994 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;ImmutableDictionary&lt;String, String&gt;&gt;.DeserializeFromUtf8Bytes(Mode: Reflection)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 413.8953659620098 < 418.6413950446429.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 1/18/2023 11:30:09 AM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 25.942853877459513 (T) = (0 -417698.2441895948) / Math.Sqrt((124535193.4616825 / (299)) + (23915153.01544391 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.056377247467174124 = (442653.8498233851 - 417698.2441895948) / 442653.8498233851 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 967250cc86b4c613015066b47b882e7424fdb590
Compare 4862343c61f698da986901a182e94df24da1487f
Diff Diff

Improvements in System.IO.Tests.Perf_Directory

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
GetCurrentDirectory - Duration of single invocation 2.76 μs 2.15 μs 0.78 0.03 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.IO.Tests.Perf_Directory*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.IO.Tests.Perf_Directory*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.IO.Tests.Perf_Directory*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.IO.Tests.Perf_Directory*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.IO.Tests.Perf_Directory.GetCurrentDirectory

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 2.150756800734312 < 2.6290698460762547.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 167.448989405178 (T) = (0 -2143.9809191742984) / Math.Sqrt((1858.2075060692428 / (299)) + (321.71023931251415 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.22011821677344945 = (2749.110141160312 - 2143.9809191742984) / 2749.110141160312 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 967250cc86b4c613015066b47b882e7424fdb590
Compare 4862343c61f698da986901a182e94df24da1487f
Diff Diff

Improvements in MicroBenchmarks.Serializers.Json_FromStream<IndexViewModel>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SystemTextJson_SourceGen_ - Duration of single invocation 646.54 μs 608.86 μs 0.94 0.01 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'MicroBenchmarks.Serializers.Json_FromStream&lt;IndexViewModel&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'MicroBenchmarks.Serializers.Json_FromStream&lt;IndexViewModel&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'MicroBenchmarks.Serializers.Json_FromStream&lt;IndexViewModel&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'MicroBenchmarks.Serializers.Json_FromStream&lt;IndexViewModel&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Json_FromStream<IndexViewModel>.SystemTextJson_SourceGen_

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 608.8644554197228 < 614.6096507333551.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 2/1/2023 4:12:58 PM, 2/28/2023 7:53:18 AM, 3/7/2023 7:56:26 PM falls between 2/26/2023 10:10:21 PM and 3/7/2023 7:56:26 PM.
IsImprovementStdDev: Marked as improvement because 53.03493915179004 (T) = (0 -607778.5235082883) / Math.Sqrt((222508053.99799877 / (299)) + (30906691.222958993 / (47))) is greater than 1.9668840363001725 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (47) - 2, .975) and 0.09363870800193842 = (670569.8145696939 - 607778.5235082883) / 670569.8145696939 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@kotlarmilos
Copy link
Member

Looks like dotnet/runtime#81782.

/cc: @BrzVlad @SamMonoRT

This was referenced Mar 28, 2023
Closed
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
arch-x64 branch-refs/heads/main kind-micro_mono mono-interpreter os-linux perf-improvement runtime-mono
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@kotlarmilos