[Perf] Linux/arm64: 223 Improvements on 3/15/2023 4:14:49 PM #14322

performanceautofiler · 2023-03-21T17:14:41Z

Run Information

Name	Value
Architecture	arm64
OS	ubuntu 20.04
Queue	AmpereUbuntu
Baseline	367a360a9fedc8c3c59ac7f6b4b5d92790ace993
Compare	41cb2d30974d6dba121276f28c84b6ad362e150c
Diff	Diff
Configs	CompliationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Runtime.Intrinsics.Tests.Perf_Vector128Float

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector
BitwiseAndOperatorBenchmark - Duration of single invocation	12.90 ns	1.62 ns	0.13	0.08	False
BitwiseAndBenchmark - Duration of single invocation	12.93 ns	1.33 ns	0.10	0.10	False
BitwiseOrOperatorBenchmark - Duration of single invocation	12.86 ns	1.15 ns	0.09	0.09	False
OnesComplementOperatorBenchmark - Duration of single invocation	8.56 ns	1.44 ns	0.17	0.13	False
ExclusiveOrOperatorBenchmark - Duration of single invocation	13.06 ns	1.12 ns	0.09	0.12	False
OnesComplementBenchmark - Duration of single invocation	8.48 ns	0.90 ns	0.11	0.15	False
NegateBenchmark - Duration of single invocation	33.40 ns	1.11 ns	0.03	0.11	False
UnaryNegateOperatorBenchmark - Duration of single invocation	33.42 ns	0.96 ns	0.03	0.11	False
SubtractionOperatorBenchmark - Duration of single invocation	35.84 ns	1.11 ns	0.03	0.14	False
XorBenchmark - Duration of single invocation	13.06 ns	0.96 ns	0.07	0.12	False
BitwiseOrBenchmark - Duration of single invocation	12.99 ns	1.57 ns	0.12	0.08	False
SubtractBenchmark - Duration of single invocation	36.19 ns	1.12 ns	0.03	0.13	False
AddBenchmark - Duration of single invocation	34.99 ns	1.57 ns	0.04	0.13	False
AddOperatorBenchmark - Duration of single invocation	36.11 ns	1.58 ns	0.04	0.07	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
Dotnet SDK installed for dotnet commands
Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Float*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Float* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Float*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Float* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Runtime.Intrinsics.Tests.Perf_Vector128Float.BitwiseAndOperatorBenchmark

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.617769475627621 < 12.253450083794187.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 39.31465132315824 (T) = (0 -1.2176150139037702) / Math.Sqrt((20.290555672460997 / (299)) + (0.083180053193325 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.8968309228655454 = (11.802131488652565 - 1.2176150139037702) / 11.802131488652565 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Float.BitwiseAndBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.3258537031547384 < 12.304325084717913.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 36.025094766093666 (T) = (0 -1.3036707321188612) / Math.Sqrt((27.134916145561384 / (299)) + (0.09851947066082051 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.8955286712667304 = (12.478741755523393 - 1.3036707321188612) / 12.478741755523393 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Float.BitwiseOrOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.1506155781001461 < 12.24059656979235.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 38.07995052646937 (T) = (0 -1.2466865094284019) / Math.Sqrt((20.524289549897468 / (299)) + (0.1393872135480407 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.8940908997576423 = (11.77128789287738 - 1.2466865094284019) / 11.77128789287738 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Float.OnesComplementOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.44176082004035 < 8.064887321898004.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 42.76081243022171 (T) = (0 -1.0710101050262664) / Math.Sqrt((6.652980057295857 / (299)) + (0.05036792425582238 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.8633690714455972 = (7.838709114824012 - 1.0710101050262664) / 7.838709114824012 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Float.ExclusiveOrOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.1206933406799944 < 12.331283613781801.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 39.52603881854506 (T) = (0 -1.274950292271638) / Math.Sqrt((20.0553033338356 / (299)) + (0.07792424947442746 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.892290100315903 = (11.83689053662613 - 1.274950292271638) / 11.83689053662613 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Float.OnesComplementBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 0.901399594164782 < 8.044565700485176.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 42.620931820872315 (T) = (0 -1.0460306481334989) / Math.Sqrt((6.553020833298951 / (299)) + (0.06448269273687629 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.8667797874764492 = (7.851891453397741 - 1.0460306481334989) / 7.851891453397741 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Float.NegateBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.1062767489004899 < 31.79899931965374.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 35.275906688964305 (T) = (0 -0.9185265489445642) / Math.Sqrt((146.56395565211946 / (299)) + (0.04956573008387821 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9642394658042127 = (25.685481763658018 - 0.9185265489445642) / 25.685481763658018 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Float.UnaryNegateOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 0.9551590426654967 < 31.729260922671997.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 35.87247779937367 (T) = (0 -0.9977452308882901) / Math.Sqrt((141.74182786207243 / (299)) + (0.03374107678735779 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9612454919378003 = (25.745268893284415 - 0.9977452308882901) / 25.745268893284415 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Float.SubtractionOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.1120740189393923 < 34.223817173165166.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 35.381485852463044 (T) = (0 -1.329935024962443) / Math.Sqrt((224.9771230326025 / (299)) + (0.054017852525616694 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9585457308252185 = (32.081979767997026 - 1.329935024962443) / 32.081979767997026 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Float.XorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 0.9627061124992404 < 12.2633055727651.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 38.4593149806241 (T) = (0 -1.308320746391792) / Math.Sqrt((24.853891268705418 / (299)) + (0.05852881474311616 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.8962586154941483 = (12.611367706568394 - 1.308320746391792) / 12.611367706568394 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Float.BitwiseOrBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.5729453593686553 < 12.308153886765426.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 37.54836488783001 (T) = (0 -1.2982645056361608) / Math.Sqrt((24.76824534354623 / (299)) + (0.1421084385628657 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.8970324424412053 = (12.608481121782933 - 1.2982645056361608) / 12.608481121782933 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Float.SubtractBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.1190579000093681 < 34.346090733822145.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 34.56605311446832 (T) = (0 -1.2755875960330891) / Math.Sqrt((287.67508551202866 / (299)) + (0.048539974204522864 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9637907167869816 = (35.22819240935636 - 1.2755875960330891) / 35.22819240935636 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Float.AddBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.5705472903815823 < 34.17248871087112.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 34.52382773843423 (T) = (0 -1.3222742448657219) / Math.Sqrt((287.71599439813633 / (299)) + (0.054511464219030616 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.962479771860105 = (35.24163658961761 - 1.3222742448657219) / 35.24163658961761 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Float.AddOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.5782123814011528 < 34.28825277048132.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 35.39156568791755 (T) = (0 -1.2769663957921493) / Math.Sqrt((224.4475562440685 / (299)) + (0.07359533215179406 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9601245239402174 = (32.02385330466484 - 1.2769663957921493) / 32.02385330466484 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

The text was updated successfully, but these errors were encountered:

performanceautofiler · 2023-03-21T17:14:45Z

Run Information

Name	Value
Architecture	arm64
OS	ubuntu 20.04
Queue	AmpereUbuntu
Baseline	367a360a9fedc8c3c59ac7f6b4b5d92790ace993
Compare	41cb2d30974d6dba121276f28c84b6ad362e150c
Diff	Diff
Configs	CompliationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int32>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector
NegateBenchmark - Duration of single invocation	14.93 ns	0.94 ns	0.06	0.05	False
OnesComplementOperatorBenchmark - Duration of single invocation	8.48 ns	0.93 ns	0.11	0.13	False
BitwiseAndBenchmark - Duration of single invocation	12.87 ns	1.10 ns	0.09	0.05	False
UnaryNegateOperatorBenchmark - Duration of single invocation	14.81 ns	0.95 ns	0.06	0.04	False
SubtractBenchmark - Duration of single invocation	16.89 ns	1.27 ns	0.08	0.04	False
BitwiseOrBenchmark - Duration of single invocation	12.88 ns	1.30 ns	0.10	0.05	False
XorBenchmark - Duration of single invocation	12.87 ns	1.28 ns	0.10	0.04	False
AddBenchmark - Duration of single invocation	16.87 ns	1.16 ns	0.07	0.07	False
SubtractionOperatorBenchmark - Duration of single invocation	16.65 ns	1.15 ns	0.07	0.06	False
OnesComplementBenchmark - Duration of single invocation	8.33 ns	0.94 ns	0.11	0.13	False
BitwiseAndOperatorBenchmark - Duration of single invocation	12.84 ns	1.33 ns	0.10	0.06	False
BitwiseOrOperatorBenchmark - Duration of single invocation	12.77 ns	1.30 ns	0.10	0.05	False
ExclusiveOrOperatorBenchmark - Duration of single invocation	12.76 ns	1.28 ns	0.10	0.08	False
AddOperatorBenchmark - Duration of single invocation	16.94 ns	1.15 ns	0.07	0.05	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
Dotnet SDK installed for dotnet commands
Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int32&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int32&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int32&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int32&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int32>.NegateBenchmark

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 0.9416288770336846 < 14.071108910801911.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 234.28681586359212 (T) = (0 -0.9543782923866111) / Math.Sqrt((0.9233477919528406 / (299)) + (0.013991583440258583 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9385089086003062 = (15.52059445787239 - 0.9543782923866111) / 15.52059445787239 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int32&gt;.OnesComplementOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 0.9341621148261647 < 8.005090800578609.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 126.33624382227443 (T) = (0 -0.9265402212262822) / Math.Sqrt((0.2648134831847877 / (299)) + (0.06453460057916957 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9011585509866533 = (9.374004837800094 - 0.9265402212262822) / 9.374004837800094 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int32&gt;.BitwiseAndBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.09972439074918 < 12.197537088670865.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 219.02470287124152 (T) = (0 -1.2065154731439904) / Math.Sqrt((1.1796595783017503 / (299)) + (0.006213616586482908 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9224262260898755 = (15.553136225418617 - 1.2065154731439904) / 15.553136225418617 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int32&gt;.UnaryNegateOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 0.947097622216678 < 14.07347196160203.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 219.9096106655473 (T) = (0 -0.957972612691843) / Math.Sqrt((0.908118459841727 / (299)) + (0.02452017844258688 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9383709007379138 = (15.544160537182835 - 0.957972612691843) / 15.544160537182835 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int32&gt;.SubtractBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.2739733261263808 < 15.877241624636449.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 208.48034143688656 (T) = (0 -1.1706322918905347) / Math.Sqrt((1.3697550977276993 / (299)) + (0.06425035555397528 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9414460192905693 = (19.992360514303876 - 1.1706322918905347) / 19.992360514303876 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int32&gt;.BitwiseOrBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.3013871750507715 < 12.240756099338979.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 228.6505679429196 (T) = (0 -1.2573933877646308) / Math.Sqrt((0.8776553079857475 / (299)) + (0.016320913729208943 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.918510227743643 = (15.430076105858085 - 1.2573933877646308) / 15.430076105858085 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int32&gt;.XorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.279272925503383 < 12.239300023116632.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 212.63117360137474 (T) = (0 -1.1943655864944707) / Math.Sqrt((1.1136311499574083 / (299)) + (0.015204783008222991 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9232777632264174 = (15.567397885168546 - 1.1943655864944707) / 15.567397885168546 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int32&gt;.AddBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.1641260537778453 < 16.04599681547288.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 194.0691042787005 (T) = (0 -1.1237860171465008) / Math.Sqrt((1.430496075352588 / (299)) + (0.08395711476283108 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9437770038347318 = (19.988013691819603 - 1.1237860171465008) / 19.988013691819603 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int32&gt;.SubtractionOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.1502264673528333 < 15.829184074235362.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 142.17922804330456 (T) = (0 -1.0222112965676942) / Math.Sqrt((0.5350068433222737 / (299)) + (0.23543855848257939 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9443224100809982 = (18.359474575942972 - 1.0222112965676942) / 18.359474575942972 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int32&gt;.OnesComplementBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 0.9448897250194728 < 8.036727362847257.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 175.04278023298667 (T) = (0 -0.9743219092439119) / Math.Sqrt((0.2535181985386248 / (299)) + (0.026563641152101664 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.8964821929175273 = (9.412118906920709 - 0.9743219092439119) / 9.412118906920709 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int32&gt;.BitwiseAndOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.3327894820078108 < 12.257118367306935.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 121.83304909625004 (T) = (0 -1.0407955646144798) / Math.Sqrt((0.299741307435893 / (299)) + (0.19529223878077268 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9272395208369539 = (14.30440778547103 - 1.0407955646144798) / 14.30440778547103 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int32&gt;.BitwiseOrOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.2990099318687522 < 12.414246945384482.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 276.2025796489158 (T) = (0 -1.2943490081042102) / Math.Sqrt((0.2857355090376364 / (299)) + (0.022759362126365947 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9095385066794603 = (14.308286991436631 - 1.2943490081042102) / 14.308286991436631 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int32&gt;.ExclusiveOrOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.2812803017539287 < 12.204836616987258.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 152.1674410446451 (T) = (0 -1.1820106401643276) / Math.Sqrt((0.512761638016424 / (299)) + (0.10566770267436343 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9181142085743283 = (14.434868608887387 - 1.1820106401643276) / 14.434868608887387 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int32&gt;.AddOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.145141956136519 < 15.939909379979516.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 276.0599529479207 (T) = (0 -1.2599368700022475) / Math.Sqrt((0.8400156042896079 / (299)) + (0.0192666528744544 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9317297587860534 = (18.455140154753966 - 1.2599368700022475) / 18.455140154753966 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name	Value
Architecture	arm64
OS	ubuntu 20.04
Queue	AmpereUbuntu
Baseline	367a360a9fedc8c3c59ac7f6b4b5d92790ace993
Compare	41cb2d30974d6dba121276f28c84b6ad362e150c
Diff	Diff
Configs	CompliationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Numerics.Tests.Perf_VectorOf<UInt16>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector
ExclusiveOrOperatorBenchmark - Duration of single invocation	7.99 ns	1.18 ns	0.15	0.08	False
BitwiseOrOperatorBenchmark - Duration of single invocation	8.06 ns	1.33 ns	0.17	0.08	False
OnesComplementBenchmark - Duration of single invocation	6.32 ns	0.98 ns	0.16	0.09	False
AddOperatorBenchmark - Duration of single invocation	23.93 ns	1.33 ns	0.06	0.05	False
BitwiseAndOperatorBenchmark - Duration of single invocation	8.02 ns	1.19 ns	0.15	0.08	False
OnesComplementOperatorBenchmark - Duration of single invocation	6.31 ns	0.97 ns	0.15	0.09	False
NegateBenchmark - Duration of single invocation	23.97 ns	0.98 ns	0.04	0.04	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
Dotnet SDK installed for dotnet commands
Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;UInt16&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorOf&lt;UInt16&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;UInt16&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorOf&lt;UInt16&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_VectorOf<UInt16>.ExclusiveOrOperatorBenchmark

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.1798735184162026 < 7.77688909625881.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 104.4421734728669 (T) = (0 -1.2982655062663584) / Math.Sqrt((3.1746429868169113 / (299)) + (0.04633050306660674 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9023410535255341 = (13.293871715130617 - 1.2982655062663584) / 13.293871715130617 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt16&gt;.BitwiseOrOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.3325460541925453 < 7.64976806991265.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 85.76860109634868 (T) = (0 -1.3032613733006577) / Math.Sqrt((3.224677282270708 / (299)) + (0.15745073126521825 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9019371329253563 = (13.290059858321678 - 1.3032613733006577) / 13.290059858321678 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt16&gt;.OnesComplementBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 0.9835773213623604 < 6.023257854621862.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 123.93470849732047 (T) = (0 -0.9445779299532219) / Math.Sqrt((0.8924501907585101 / (299)) + (0.01410260579657632 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.8895547551674154 = (8.552454488964509 - 0.9445779299532219) / 8.552454488964509 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt16&gt;.AddOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.3337447312305053 < 22.60876522254819.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 416.28635922293984 (T) = (0 -1.3086378065851962) / Math.Sqrt((0.2504779440563861 / (299)) + (0.0402409899773243 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.946337748627495 = (24.386561747122393 - 1.3086378065851962) / 24.386561747122393 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt16&gt;.BitwiseAndOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.1917551254053824 < 7.655771115190063.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 89.74144036539975 (T) = (0 -1.2916063993885782) / Math.Sqrt((2.894855124830828 / (299)) + (0.14265450656921733 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.902147581284547 = (13.199534731425148 - 1.2916063993885782) / 13.199534731425148 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt16&gt;.OnesComplementOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 0.9694864588741979 < 6.045544199476805.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 114.43294686633617 (T) = (0 -0.9694973455032291) / Math.Sqrt((0.8726793561310946 / (299)) + (0.026703922808063236 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.8867677841421705 = (8.562027495076991 - 0.9694973455032291) / 8.562027495076991 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt16&gt;.NegateBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 0.9759151407062504 < 22.206634150785934.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 421.7840219786486 (T) = (0 -1.0515213339534435) / Math.Sqrt((0.2705982954060817 / (299)) + (0.04062456940463319 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9575466773272724 = (24.76888186254847 - 1.0515213339534435) / 24.76888186254847 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

performanceautofiler · 2023-03-21T17:14:46Z

Run Information

Name	Value
Architecture	arm64
OS	ubuntu 20.04
Queue	AmpereUbuntu
Baseline	367a360a9fedc8c3c59ac7f6b4b5d92790ace993
Compare	41cb2d30974d6dba121276f28c84b6ad362e150c
Diff	Diff
Configs	CompliationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Runtime.Intrinsics.Tests.Perf_Vector128Of<SByte>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector
ExclusiveOrOperatorBenchmark - Duration of single invocation	13.08 ns	1.17 ns	0.09	0.04	False
OnesComplementBenchmark - Duration of single invocation	9.04 ns	0.94 ns	0.10	0.12	False
SubtractionOperatorBenchmark - Duration of single invocation	50.22 ns	1.26 ns	0.03	0.05	False
OnesComplementOperatorBenchmark - Duration of single invocation	8.51 ns	0.91 ns	0.11	0.14	False
AddOperatorBenchmark - Duration of single invocation	49.60 ns	1.25 ns	0.03	0.05	False
BitwiseOrOperatorBenchmark - Duration of single invocation	12.80 ns	1.33 ns	0.10	0.07	False
BitwiseAndOperatorBenchmark - Duration of single invocation	12.87 ns	1.28 ns	0.10	0.06	False
SubtractBenchmark - Duration of single invocation	48.79 ns	1.26 ns	0.03	0.04	False
NegateBenchmark - Duration of single invocation	46.74 ns	0.93 ns	0.02	0.04	False
UnaryNegateOperatorBenchmark - Duration of single invocation	47.05 ns	0.07 ns	0.00	0.03	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
Dotnet SDK installed for dotnet commands
Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<SByte>.ExclusiveOrOperatorBenchmark

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.173444671311986 < 12.309392708542177.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 158.7568689119192 (T) = (0 -1.1963055168453112) / Math.Sqrt((0.28423407139631585 / (299)) + (0.10588394677146902 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9164562804553027 = (14.319514660886828 - 1.1963055168453112) / 14.319514660886828 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;.OnesComplementBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 0.9368609633292401 < 8.048560456936723.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 210.58421721114732 (T) = (0 -0.9621352926639114) / Math.Sqrt((0.2811857489111942 / (299)) + (0.011960315440310792 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.897627173795239 = (9.398346498117546 - 0.9621352926639114) / 9.398346498117546 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;.SubtractionOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.2639266774218731 < 47.10818545074614.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 396.78566042441656 (T) = (0 -1.150580017529076) / Math.Sqrt((4.123548621670754 / (299)) + (0.0694703888424495 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.97863986262176 = (53.86575924840205 - 1.150580017529076) / 53.86575924840205 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;.OnesComplementOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 0.9090640973242027 < 8.001329093806879.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 272.39827778404464 (T) = (0 -0.9286298590020596) / Math.Sqrt((0.23343493557258505 / (299)) + (0.0032360343635689883 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9009014666753907 = (9.370772985713321 - 0.9286298590020596) / 9.370772985713321 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;.AddOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.2472523532063353 < 47.1285456970046.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 430.92584915015397 (T) = (0 -1.2462717249728155) / Math.Sqrt((3.9944404991037907 / (299)) + (0.028513241260007996 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9768883791992492 = (53.92402963501058 - 1.2462717249728155) / 53.92402963501058 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;.BitwiseOrOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.3348778746141103 < 12.18953429703313.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 153.19932091374014 (T) = (0 -1.1663421698390692) / Math.Sqrt((0.5068881189463142 / (299)) + (0.10415494499285474 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9191026900964437 = (14.417564332232462 - 1.1663421698390692) / 14.417564332232462 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;.BitwiseAndOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.2808096453207618 < 12.244441382979456.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 288.01802162640615 (T) = (0 -1.2275229009634214) / Math.Sqrt((0.472602567983406 / (299)) + (0.009293300139788851 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.914852547413497 = (14.41643717663023 - 1.2275229009634214) / 14.41643717663023 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;.SubtractBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.2621416884765646 < 47.055703090194406.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 488.6810969030331 (T) = (0 -1.2168628158217432) / Math.Sqrt((3.3331085613388516 / (299)) + (0.014626631779938571 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9777376595594143 = (54.660147663689386 - 1.2168628158217432) / 54.660147663689386 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;.NegateBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 0.9330633888575299 < 44.408161466725986.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 498.25110091949864 (T) = (0 -0.906914993316807) / Math.Sqrt((2.0900804878020236 / (299)) + (0.04322369403269596 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9815639353508059 = (49.19243941555914 - 0.906914993316807) / 49.19243941555914 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;.UnaryNegateOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 0.07324173286830306 < 44.36210145258077.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 479.42362830236874 (T) = (0 -0.8920370961922374) / Math.Sqrt((2.112453108030004 / (299)) + (0.05559715976173488 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9818697727711522 = (49.201650091449345 - 0.8920370961922374) / 49.201650091449345 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name	Value
Architecture	arm64
OS	ubuntu 20.04
Queue	AmpereUbuntu
Baseline	367a360a9fedc8c3c59ac7f6b4b5d92790ace993
Compare	41cb2d30974d6dba121276f28c84b6ad362e150c
Diff	Diff
Configs	CompliationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt64>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector
BitwiseAndBenchmark - Duration of single invocation	12.85 ns	1.16 ns	0.09	0.06	False
BitwiseOrBenchmark - Duration of single invocation	13.03 ns	1.27 ns	0.10	0.06	False
SubtractionOperatorBenchmark - Duration of single invocation	14.20 ns	1.31 ns	0.09	0.08	False
AddBenchmark - Duration of single invocation	15.54 ns	1.23 ns	0.08	0.08	False
SubtractBenchmark - Duration of single invocation	14.30 ns	1.25 ns	0.09	0.11	False
UnaryNegateOperatorBenchmark - Duration of single invocation	12.71 ns	0.95 ns	0.08	0.10	False
XorBenchmark - Duration of single invocation	12.92 ns	1.26 ns	0.10	0.05	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
Dotnet SDK installed for dotnet commands
Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt64&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt64&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt64&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt64&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt64>.BitwiseAndBenchmark

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.1587163968662173 < 12.201293567278253.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 223.92890026992316 (T) = (0 -1.235510842643436) / Math.Sqrt((0.8995029164519027 / (299)) + (0.018443456011972167 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9200644182986614 = (15.45633141520938 - 1.235510842643436) / 15.45633141520938 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt64&gt;.BitwiseOrBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.2711111463780422 < 12.309091908634002.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 144.24880857511207 (T) = (0 -1.1299535409242536) / Math.Sqrt((0.8662675007053147 / (299)) + (0.12482210560225294 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9267832392808679 = (15.432990067108888 - 1.1299535409242536) / 15.432990067108888 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt64&gt;.SubtractionOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.3134667854287467 < 13.31676877522314.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 176.0435517285007 (T) = (0 -1.148527094251606) / Math.Sqrt((0.3933472687583221 / (299)) + (0.09754747869599664 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9263563451408169 = (15.595737289896162 - 1.148527094251606) / 15.595737289896162 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt64&gt;.AddBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.2324196611616394 < 13.366169522032623.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 205.1726815621852 (T) = (0 -1.2615605543485473) / Math.Sqrt((1.3512538359274158 / (299)) + (0.02746258184496087 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9267113130055119 = (17.213578330901566 - 1.2615605543485473) / 17.213578330901566 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt64&gt;.SubtractBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.254386260856568 < 13.188288370336863.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 163.1739341632099 (T) = (0 -1.1706433226419812) / Math.Sqrt((1.4966583083561205 / (299)) + (0.08682139283667359 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9325195401787674 = (17.347885976817846 - 1.1706433226419812) / 17.347885976817846 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt64&gt;.UnaryNegateOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 0.9530601433827347 < 11.662383716724792.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 249.50621390813373 (T) = (0 -0.8817732227709025) / Math.Sqrt((0.21219339718713556 / (299)) + (0.025383625579475214 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9287143326092995 = (12.3695723845595 - 0.8817732227709025) / 12.3695723845595 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt64&gt;.XorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.2606549358321926 < 12.28493769353937.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 233.7208590406593 (T) = (0 -1.2563139682472764) / Math.Sqrt((0.8730192346077302 / (299)) + (0.013610086632949119 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9185620867843426 = (15.42664735183483 - 1.2563139682472764) / 15.42664735183483 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

performanceautofiler · 2023-03-21T17:14:48Z

Run Information

Name	Value
Architecture	arm64
OS	ubuntu 20.04
Queue	AmpereUbuntu
Baseline	367a360a9fedc8c3c59ac7f6b4b5d92790ace993
Compare	41cb2d30974d6dba121276f28c84b6ad362e150c
Diff	Diff
Configs	CompliationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Numerics.Tests.Perf_VectorOf<UInt32>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector
BitwiseOrOperatorBenchmark - Duration of single invocation	8.00 ns	1.64 ns	0.20	0.07	False
ExclusiveOrOperatorBenchmark - Duration of single invocation	8.44 ns	1.51 ns	0.18	0.07	False
SubtractBenchmark - Duration of single invocation	11.83 ns	1.66 ns	0.14	0.05	False
OnesComplementOperatorBenchmark - Duration of single invocation	6.37 ns	0.92 ns	0.14	0.07	False
OnesComplementBenchmark - Duration of single invocation	6.37 ns	0.93 ns	0.15	0.08	False
ConditionalSelectBenchmark - Duration of single invocation	32.59 ns	1.86 ns	0.06	0.28	False
BitwiseAndBenchmark - Duration of single invocation	8.00 ns	1.19 ns	0.15	0.32	False
AddBenchmark - Duration of single invocation	11.68 ns	1.32 ns	0.11	0.04	False
UnaryNegateOperatorBenchmark - Duration of single invocation	11.78 ns	0.95 ns	0.08	0.05	False
XorBenchmark - Duration of single invocation	8.04 ns	1.50 ns	0.19	0.32	False
BitwiseOrBenchmark - Duration of single invocation	8.03 ns	1.57 ns	0.19	0.35	False
SubtractionOperatorBenchmark - Duration of single invocation	11.54 ns	1.66 ns	0.14	0.06	False
NegateBenchmark - Duration of single invocation	11.39 ns	0.90 ns	0.08	0.04	False
AndNotBenchmark - Duration of single invocation	15.88 ns	1.89 ns	0.12	0.05	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
Dotnet SDK installed for dotnet commands
Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;UInt32&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorOf&lt;UInt32&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;UInt32&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorOf&lt;UInt32&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_VectorOf<UInt32>.BitwiseOrOperatorBenchmark

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.6385361070614415 < 7.646563199029914.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 104.15783592527289 (T) = (0 -1.3128379487446873) / Math.Sqrt((3.222672084719032 / (299)) + (0.04622708253784584 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9016292272021454 = (13.345813104899394 - 1.3128379487446873) / 13.345813104899394 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt32&gt;.ExclusiveOrOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.5080525643330112 < 7.984776523350297.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 84.12510487830302 (T) = (0 -1.1843835295184861) / Math.Sqrt((2.9277955260062325 / (299)) + (0.1916992165547309 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9103567542348975 = (13.212189266573375 - 1.1843835295184861) / 13.212189266573375 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt32&gt;.SubtractBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.6616187227068364 < 11.179508976889906.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 132.68046378827142 (T) = (0 -1.2293636293879606) / Math.Sqrt((0.4248338012545146 / (299)) + (0.12685090341065525 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9085222616000044 = (13.43893772288559 - 1.2293636293879606) / 13.43893772288559 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt32&gt;.OnesComplementOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 0.9182019553026779 < 6.054116814386369.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 173.5338628907537 (T) = (0 -0.9140352473463982) / Math.Sqrt((0.539177718798212 / (299)) + (0.0008910245613456578 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.8909733847568542 = (8.383597393241653 - 0.9140352473463982) / 8.383597393241653 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt32&gt;.OnesComplementBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 0.9309519916582496 < 6.040559930310282.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 142.8855932795107 (T) = (0 -0.9515173615868273) / Math.Sqrt((0.5652388245237386 / (299)) + (0.014730798283030274 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.8865643917228341 = (8.388171721721738 - 0.9515173615868273) / 8.388171721721738 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt32&gt;.ConditionalSelectBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.8575540822090966 < 30.945950151373992.
IsChangePoint: Marked as a change because one of 1/5/2023 9:29:47 PM, 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 106.89896329596505 (T) = (0 -1.8822380774769982) / Math.Sqrt((71.15298880326866 / (299)) + (0.03287735960853804 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9652913160097357 = (54.22960081128286 - 1.8822380774769982) / 54.22960081128286 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt32&gt;.BitwiseAndBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.192725440575694 < 7.611382317301488.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 94.56248013064838 (T) = (0 -1.3197075880409617) / Math.Sqrt((5.315937565505413 / (299)) + (0.03842198283017412 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9100032541460653 = (14.663947851879616 - 1.3197075880409617) / 14.663947851879616 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt32&gt;.AddBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.3229726626864882 < 11.07522556031471.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 162.58271018202768 (T) = (0 -1.3259168160599988) / Math.Sqrt((0.6717853958892525 / (299)) + (0.061834694099783045 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9023717569427077 = (13.58128318750852 - 1.3259168160599988) / 13.58128318750852 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt32&gt;.UnaryNegateOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 0.9479066690520624 < 10.880265713354099.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 117.59409109209135 (T) = (0 -0.9022895008350947) / Math.Sqrt((0.2767242902246279 / (299)) + (0.14058080100531847 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9241334844609919 = (11.893119045004319 - 0.9022895008350947) / 11.893119045004319 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt32&gt;.XorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.4995654452926337 < 7.786116483224897.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 94.39108927943701 (T) = (0 -1.3296432941552208) / Math.Sqrt((5.232850201947199 / (299)) + (0.04663511787598047 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9096048344185825 = (14.70923014082685 - 1.3296432941552208) / 14.70923014082685 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt32&gt;.BitwiseOrBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.5658759051929805 < 7.638372750317343.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 94.48114878381745 (T) = (0 -1.2670306034494314) / Math.Sqrt((5.350147801791101 / (299)) + (0.039787690198359406 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9135919751092643 = (14.663344117072574 - 1.2670306034494314) / 14.663344117072574 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt32&gt;.SubtractionOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.6630603965964732 < 11.110920760747533.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 203.53728945886425 (T) = (0 -1.3163749695301652) / Math.Sqrt((0.22624593535474846 / (299)) + (0.037922574130694595 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.8921708461850335 = (12.207969022821501 - 1.3163749695301652) / 12.207969022821501 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt32&gt;.NegateBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 0.901419749239822 < 10.958323231124416.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 240.89003650243305 (T) = (0 -0.9706579131195495) / Math.Sqrt((0.2613742191049059 / (299)) + (0.025828425825992107 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9226327236994807 = (12.546104238556902 - 0.9706579131195495) / 12.546104238556902 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt32&gt;.AndNotBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.8883064350507766 < 14.744707482119834.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 134.8440412308228 (T) = (0 -1.4300802990580281) / Math.Sqrt((6.734447640513244 / (299)) + (0.05575910657662121 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9378603096936765 = (23.013959226515453 - 1.4300802990580281) / 23.013959226515453 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

performanceautofiler · 2023-03-21T17:14:49Z

Run Information

Name	Value
Architecture	arm64
OS	ubuntu 20.04
Queue	AmpereUbuntu
Baseline	367a360a9fedc8c3c59ac7f6b4b5d92790ace993
Compare	41cb2d30974d6dba121276f28c84b6ad362e150c
Diff	Diff
Configs	CompliationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Numerics.Tests.Perf_VectorOf<Int64>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector
UnaryNegateOperatorBenchmark - Duration of single invocation	5.80 ns	0.93 ns	0.16	0.05	False
AddBenchmark - Duration of single invocation	5.90 ns	1.56 ns	0.26	0.07	False
AndNotBenchmark - Duration of single invocation	15.29 ns	1.33 ns	0.09	0.03	False
SubtractBenchmark - Duration of single invocation	5.89 ns	1.67 ns	0.28	0.06	False
ConditionalSelectBenchmark - Duration of single invocation	32.48 ns	2.16 ns	0.07	0.29	False
BitwiseAndBenchmark - Duration of single invocation	8.03 ns	1.55 ns	0.19	0.35	False
BitwiseOrBenchmark - Duration of single invocation	8.54 ns	1.66 ns	0.19	0.30	False
XorBenchmark - Duration of single invocation	8.01 ns	1.51 ns	0.19	0.33	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
Dotnet SDK installed for dotnet commands
Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Int64&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorOf&lt;Int64&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Int64&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorOf&lt;Int64&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_VectorOf<Int64>.UnaryNegateOperatorBenchmark

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 0.9313519184036473 < 5.353032818203452.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 111.95014701567598 (T) = (0 -0.9163494077399046) / Math.Sqrt((1.194089589317154 / (299)) + (0.0009783418832063065 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.8860123165668888 = (8.039021235812948 - 0.9163494077399046) / 8.039021235812948 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int64&gt;.AddBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.5585740974857163 < 5.594516504239389.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 101.39007943335794 (T) = (0 -1.3020400269143129) / Math.Sqrt((1.6748635592797443 / (299)) + (0.018651968415969746 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.8638395904360737 = (9.562544876916032 - 1.3020400269143129) / 9.562544876916032 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int64&gt;.AndNotBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.3330337011411617 < 14.599913642599862.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 139.39405760902045 (T) = (0 -1.3798790520787148) / Math.Sqrt((6.65180341298193 / (299)) + (0.03894901779129084 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9404163509672181 = (23.158686560461057 - 1.3798790520787148) / 23.158686560461057 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int64&gt;.SubtractBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.6742005328148568 < 5.593481884906267.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 89.46418030337718 (T) = (0 -1.2486388721483979) / Math.Sqrt((1.9675312742208801 / (299)) + (0.042320492878919674 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.8713221081026218 = (9.703600624295266 - 1.2486388721483979) / 9.703600624295266 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int64&gt;.ConditionalSelectBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 2.161185981761698 < 30.890696453636004.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 108.47031946011556 (T) = (0 -1.9285649205594362) / Math.Sqrt((68.55603965706048 / (299)) + (0.05203972560335656 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9644140417276482 = (54.194547911270476 - 1.9285649205594362) / 54.194547911270476 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int64&gt;.BitwiseAndBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.5456497101355058 < 7.646712921933471.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 95.28905630610511 (T) = (0 -1.3206307199436607) / Math.Sqrt((5.020475672427381 / (299)) + (0.05049750474141172 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9099150734660966 = (14.659841227118504 - 1.3206307199436607) / 14.659841227118504 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int64&gt;.BitwiseOrBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.6568502398942957 < 7.783124790864319.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 93.6365489819974 (T) = (0 -1.3287228968069706) / Math.Sqrt((5.267643533011105 / (299)) + (0.05271093930799738 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9099198053796391 = (14.75044433914488 - 1.3287228968069706) / 14.75044433914488 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int64&gt;.XorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.5073828308075727 < 7.807766447349059.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 96.39248568333804 (T) = (0 -1.3173829113722002) / Math.Sqrt((5.305922861979466 / (299)) + (0.027280844400598504 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9103526899533292 = (14.695175021831261 - 1.3173829113722002) / 14.695175021831261 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name	Value
Architecture	arm64
OS	ubuntu 20.04
Queue	AmpereUbuntu
Baseline	367a360a9fedc8c3c59ac7f6b4b5d92790ace993
Compare	41cb2d30974d6dba121276f28c84b6ad362e150c
Diff	Diff
Configs	CompliationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Numerics.Tests.Perf_VectorConvert

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector
Widen_byte - Duration of single invocation	72.97 μs	49.52 μs	0.68	0.00	False
Convert_float_int - Duration of single invocation	25.63 μs	16.18 μs	0.63	0.01	False
Widen_ushort - Duration of single invocation	61.09 μs	34.57 μs	0.57	0.12	False
Convert_double_long - Duration of single invocation	18.30 μs	10.31 μs	0.56	0.01	False
Widen_int - Duration of single invocation	53.20 μs	26.33 μs	0.49	0.01	False
Widen_short - Duration of single invocation	61.99 μs	34.35 μs	0.55	0.01	False
Widen_uint - Duration of single invocation	52.55 μs	25.66 μs	0.49	0.01	False
Widen_float - Duration of single invocation	64.00 μs	39.81 μs	0.62	0.01	False
Widen_sbyte - Duration of single invocation	73.34 μs	49.36 μs	0.67	0.00	False
Convert_int_float - Duration of single invocation	26.37 μs	17.35 μs	0.66	0.01	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
Dotnet SDK installed for dotnet commands
Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorConvert*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorConvert* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorConvert*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorConvert* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_VectorConvert.Widen_byte

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 49.52043623817034 < 69.89227708290623.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 353.6600257860637 (T) = (0 -49753.4210876854) / Math.Sqrt((308090.9120438381 / (299)) + (59814.617315632175 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.31926800823992246 = (73088.11939195724 - 49753.4210876854) / 73088.11939195724 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorConvert.Convert_float_int

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 16.1830820239334 < 24.32349356558199.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 48.329423383447725 (T) = (0 -16248.500955520003) / Math.Sqrt((8533218.031631703 / (299)) + (7744.213761994776 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.3361012361332248 = (24474.365430179645 - 16248.500955520003) / 24474.365430179645 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorConvert.Widen_ushort

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 34.572562075300254 < 58.593057350260416.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 221.63871401257322 (T) = (0 -34357.134119733295) / Math.Sqrt((3219707.731936291 / (299)) + (6800.5583104447405 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.40513948013658946 = (57756.62188444148 - 34357.134119733295) / 57756.62188444148 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorConvert.Convert_double_long

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 10.31128394858273 < 17.39019736212141.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 123.22348503295406 (T) = (0 -10323.397524974886) / Math.Sqrt((2024236.0470262459 / (299)) + (748.5325672465368 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.4962558224452768 = (20493.333689903393 - 10323.397524974886) / 20493.333689903393 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorConvert.Widen_int

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 26.328222467732886 < 50.67598951538204.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 113.86236148043851 (T) = (0 -26341.22413770535) / Math.Sqrt((7908777.529777689 / (299)) + (2101.9052629361518 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.41333938248525864 = (44900.276840286555 - 26341.22413770535) / 44900.276840286555 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorConvert.Widen_short

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 34.34558462595097 < 58.82192137909079.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 88.89344258276505 (T) = (0 -34262.875025521746) / Math.Sqrt((11531923.59201999 / (299)) + (12834.114237819645 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.3395887953852269 = (51881.12313374173 - 34262.875025521746) / 51881.12313374173 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorConvert.Widen_uint

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 25.66310819581958 < 49.92212018785311.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 46.2260173551487 (T) = (0 -25790.67958151442) / Math.Sqrt((22952602.440262973 / (299)) + (9926.091186253987 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.3326114265072365 = (38644.173133710494 - 25790.67958151442) / 38644.173133710494 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorConvert.Widen_float

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 39.808359430100765 < 60.841812394125675.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 236.03579775275935 (T) = (0 -39495.08628148852) / Math.Sqrt((4506686.431566898 / (299)) + (25973.298791520047 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.4344000764286983 = (69828.66269165897 - 39495.08628148852) / 69828.66269165897 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorConvert.Widen_sbyte

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 49.355408188291136 < 69.79639641133721.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 420.96498058832816 (T) = (0 -49565.211075087325) / Math.Sqrt((332976.7808412712 / (299)) + (36184.12346166048 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.32189387924866664 = (73093.59045479441 - 49565.211075087325) / 73093.59045479441 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorConvert.Convert_int_float

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 17.34865931919643 < 25.0577746305194.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 38.66102005101807 (T) = (0 -17401.942552501154) / Math.Sqrt((14488201.415928358 / (299)) + (1409.1164654695367 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.32860586070853826 = (25919.11596199788 - 17401.942552501154) / 25919.11596199788 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

performanceautofiler · 2023-03-21T17:14:50Z

Run Information

Name	Value
Architecture	arm64
OS	ubuntu 20.04
Queue	AmpereUbuntu
Baseline	367a360a9fedc8c3c59ac7f6b4b5d92790ace993
Compare	41cb2d30974d6dba121276f28c84b6ad362e150c
Diff	Diff
Configs	CompliationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Numerics.Tests.Perf_VectorOf<UInt64>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector
ConditionalSelectBenchmark - Duration of single invocation	32.63 ns	1.84 ns	0.06	0.05	False
BitwiseOrBenchmark - Duration of single invocation	8.05 ns	1.26 ns	0.16	0.11	False
BitwiseOrOperatorBenchmark - Duration of single invocation	8.00 ns	1.15 ns	0.14	0.09	False
AddOperatorBenchmark - Duration of single invocation	5.93 ns	1.16 ns	0.19	0.09	False
BitwiseAndOperatorBenchmark - Duration of single invocation	7.98 ns	1.11 ns	0.14	0.12	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
Dotnet SDK installed for dotnet commands
Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;UInt64&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorOf&lt;UInt64&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;UInt64&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorOf&lt;UInt64&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_VectorOf<UInt64>.ConditionalSelectBenchmark

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.8418052602337913 < 30.976601646669835.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 355.76634318145506 (T) = (0 -1.9771946948198607) / Math.Sqrt((1.4930062394029497 / (299)) + (0.0718762725134915 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9446206644841042 = (35.7027522342216 - 1.9771946948198607) / 35.7027522342216 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt64&gt;.BitwiseOrBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.2617009080390402 < 7.645626600707342.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 112.9538457792101 (T) = (0 -1.4137884091391721) / Math.Sqrt((1.5480767255195238 / (299)) + (0.05585255010074061 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.879083153291716 = (11.692236835699042 - 1.4137884091391721) / 11.692236835699042 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt64&gt;.BitwiseOrOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.1451721330887195 < 7.784354490510566.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 129.40502235112152 (T) = (0 -1.3257898125695495) / Math.Sqrt((0.6835641938818079 / (299)) + (0.04927379657310447 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.8737063619751259 = (10.49767694797444 - 1.3257898125695495) / 10.49767694797444 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt64&gt;.AddOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.1558959798686828 < 5.529748218528944.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 67.97871507551058 (T) = (0 -1.249560627948651) / Math.Sqrt((1.1287219949879106 / (299)) + (0.1252859050243475 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.8493221617295866 = (8.2929290882587 - 1.249560627948651) / 8.2929290882587 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt64&gt;.BitwiseAndOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.1139744350794816 < 7.628498103688524.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 88.70467973856704 (T) = (0 -1.2910485275760288) / Math.Sqrt((0.7272114705721189 / (299)) + (0.1482351713882061 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.8764872772893445 = (10.452757410266772 - 1.2910485275760288) / 10.452757410266772 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name	Value
Architecture	arm64
OS	ubuntu 20.04
Queue	AmpereUbuntu
Baseline	367a360a9fedc8c3c59ac7f6b4b5d92790ace993
Compare	41cb2d30974d6dba121276f28c84b6ad362e150c
Diff	Diff
Configs	CompliationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Numerics.Tests.Perf_VectorOf<SByte>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector
OnesComplementBenchmark - Duration of single invocation	6.35 ns	1.03 ns	0.16	0.08	False
SubtractionOperatorBenchmark - Duration of single invocation	39.24 ns	1.29 ns	0.03	0.05	False
BitwiseAndOperatorBenchmark - Duration of single invocation	8.45 ns	1.10 ns	0.13	0.09	False
BitwiseOrOperatorBenchmark - Duration of single invocation	8.49 ns	1.29 ns	0.15	0.08	False
XorBenchmark - Duration of single invocation	8.47 ns	1.40 ns	0.17	0.35	False
SubtractBenchmark - Duration of single invocation	38.83 ns	1.13 ns	0.03	0.03	False
ExclusiveOrOperatorBenchmark - Duration of single invocation	8.43 ns	1.17 ns	0.14	0.07	False
BitwiseAndBenchmark - Duration of single invocation	8.05 ns	1.19 ns	0.15	0.33	False
AddOperatorBenchmark - Duration of single invocation	39.21 ns	1.33 ns	0.03	0.04	False
AddBenchmark - Duration of single invocation	39.12 ns	1.32 ns	0.03	0.04	False
OnesComplementOperatorBenchmark - Duration of single invocation	6.33 ns	0.94 ns	0.15	0.08	False
NegateBenchmark - Duration of single invocation	38.55 ns	0.93 ns	0.02	0.03	False
UnaryNegateOperatorBenchmark - Duration of single invocation	38.95 ns	0.89 ns	0.02	0.02	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
Dotnet SDK installed for dotnet commands
Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;SByte&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorOf&lt;SByte&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;SByte&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorOf&lt;SByte&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_VectorOf<SByte>.OnesComplementBenchmark

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.0283732043718445 < 6.026206853510613.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 117.60353077465301 (T) = (0 -0.9872595405233339) / Math.Sqrt((0.8644803837247229 / (299)) + (0.021788119968187056 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.8841123805311146 = (8.519111403340219 - 0.9872595405233339) / 8.519111403340219 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;SByte&gt;.SubtractionOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.2853659374229018 < 37.05486789974323.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 636.58041064114 (T) = (0 -1.3352608241662636) / Math.Sqrt((0.3509095014915064 / (299)) + (0.04490585262502498 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9665273699430877 = (39.891123640298574 - 1.3352608241662636) / 39.891123640298574 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;SByte&gt;.BitwiseAndOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.103207987109699 < 7.793206412601111.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 103.17380449581682 (T) = (0 -1.2963669206773396) / Math.Sqrt((3.1743106275026567 / (299)) + (0.05209716351152898 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9024469028287694 = (13.288834063380731 - 1.2963669206773396) / 13.288834063380731 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;SByte&gt;.BitwiseOrOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.2902932502396571 < 7.7994758593017455.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 106.30690362476896 (T) = (0 -1.3235850394593547) / Math.Sqrt((2.9811828658970363 / (299)) + (0.043211486173941306 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.8993290894639393 = (13.147641482643008 - 1.3235850394593547) / 13.147641482643008 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;SByte&gt;.XorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.3991220598506906 < 7.648990474978027.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 96.79822729160728 (T) = (0 -1.3031196769242979) / Math.Sqrt((5.379943993654416 / (299)) + (0.02668600690514596 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9120216643027957 = (14.811824599743016 - 1.3031196769242979) / 14.811824599743016 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;SByte&gt;.SubtractBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.1335686270165455 < 36.89375662822086.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 529.9428670283905 (T) = (0 -1.2644800233802993) / Math.Sqrt((1.075823282843646 / (299)) + (0.037986694132964856 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9693859487683565 = (41.303910214708836 - 1.2644800233802993) / 41.303910214708836 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;SByte&gt;.ExclusiveOrOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.1684069660375809 < 7.92166845539948.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 106.94824380055506 (T) = (0 -1.281571816139743) / Math.Sqrt((3.1141062336034815 / (299)) + (0.039246041573492395 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9035272920557168 = (13.284294008621616 - 1.281571816139743) / 13.284294008621616 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;SByte&gt;.BitwiseAndBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.1858048506762775 < 7.6469287727105995.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 90.7199448481634 (T) = (0 -1.4283729549227697) / Math.Sqrt((5.6436246570825706 / (299)) + (0.05367953833391225 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9037521640375813 = (14.840572160817187 - 1.4283729549227697) / 14.840572160817187 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;SByte&gt;.AddOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.3300336031883746 < 36.92865302281617.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 669.1679342038955 (T) = (0 -1.3039181773231532) / Math.Sqrt((0.28377961075961194 / (299)) + (0.04200546055807162 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9671093057827228 = (39.64398466962781 - 1.3039181773231532) / 39.64398466962781 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;SByte&gt;.AddBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.31672014896828 < 37.31679116744939.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 520.3963964891007 (T) = (0 -1.3681048735929027) / Math.Sqrt((0.9351029445941412 / (299)) + (0.049581453708227696 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9668574049117087 = (41.2793527467689 - 1.3681048735929027) / 41.2793527467689 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;SByte&gt;.OnesComplementOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 0.9350707311134192 < 6.142678617222838.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 94.69325004239896 (T) = (0 -1.0975196705142667) / Math.Sqrt((0.8450799117260384 / (299)) + (0.06081118025707267 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.8717331064600644 = (8.556531153321776 - 1.0975196705142667) / 8.556531153321776 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;SByte&gt;.NegateBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 0.9281019029137667 < 36.78655001686785.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 633.7589440167841 (T) = (0 -1.0225420187576937) / Math.Sqrt((0.5051192357336797 / (299)) + (0.0382551943445483 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9745416595409457 = (40.16530537024969 - 1.0225420187576937) / 40.16530537024969 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;SByte&gt;.UnaryNegateOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 0.8915965438825116 < 36.97576785501074.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 823.1138516107396 (T) = (0 -0.9737752668367141) / Math.Sqrt((0.2848393596421503 / (299)) + (0.022136643962940742 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9753017973823908 = (39.42696891402265 - 0.9737752668367141) / 39.42696891402265 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

performanceautofiler · 2023-03-21T17:14:51Z

Run Information

Name	Value
Architecture	arm64
OS	ubuntu 20.04
Queue	AmpereUbuntu
Baseline	367a360a9fedc8c3c59ac7f6b4b5d92790ace993
Compare	41cb2d30974d6dba121276f28c84b6ad362e150c
Diff	Diff
Configs	CompliationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Numerics.Tests.Perf_VectorOf<Byte>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector
ExclusiveOrOperatorBenchmark - Duration of single invocation	8.08 ns	1.11 ns	0.14	0.07	False
BitwiseOrOperatorBenchmark - Duration of single invocation	8.43 ns	1.14 ns	0.14	0.08	False
BitwiseAndOperatorBenchmark - Duration of single invocation	8.05 ns	1.31 ns	0.16	0.06	False
NegateBenchmark - Duration of single invocation	39.66 ns	1.05 ns	0.03	0.04	False
AddOperatorBenchmark - Duration of single invocation	38.96 ns	1.29 ns	0.03	0.03	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
Dotnet SDK installed for dotnet commands
Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Byte&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorOf&lt;Byte&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Byte&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorOf&lt;Byte&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_VectorOf<Byte>.ExclusiveOrOperatorBenchmark

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.1115551552185206 < 7.67221671504469.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 106.2910557030351 (T) = (0 -1.4071562191110651) / Math.Sqrt((2.9375702467222853 / (299)) + (0.04382472074442767 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.893202367653389 = (13.175912126442563 - 1.4071562191110651) / 13.175912126442563 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Byte&gt;.BitwiseOrOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.140437177528235 < 8.001285752595088.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 102.50860617679234 (T) = (0 -1.4004711071660703) / Math.Sqrt((3.2235363443089775 / (299)) + (0.048180720225075385 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.8946400195730337 = (13.29224912049838 - 1.4004711071660703) / 13.29224912049838 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Byte&gt;.BitwiseAndOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.31385715271296 < 7.929775281975281.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 103.11193822450605 (T) = (0 -1.3819172952829977) / Math.Sqrt((2.8522567904774276 / (299)) + (0.06314409977212104 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.894990027986311 = (13.159867284821793 - 1.3819172952829977) / 13.159867284821793 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Byte&gt;.NegateBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.0457519300991749 < 37.08829584888094.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 577.56554525009 (T) = (0 -1.0578932359317208) / Math.Sqrt((0.638650703818957 / (299)) + (0.04421363475452221 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9736825202909738 = (40.19736113138878 - 1.0578932359317208) / 40.19736113138878 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Byte&gt;.AddOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.2908849259514745 < 37.11175947437966.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 584.9847319352068 (T) = (0 -1.3752631919115605) / Math.Sqrt((0.23925302988438663 / (299)) + (0.06293535324744384 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9653760924357668 = (39.72004573314589 - 1.3752631919115605) / 39.72004573314589 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name	Value
Architecture	arm64
OS	ubuntu 20.04
Queue	AmpereUbuntu
Baseline	367a360a9fedc8c3c59ac7f6b4b5d92790ace993
Compare	41cb2d30974d6dba121276f28c84b6ad362e150c
Diff	Diff
Configs	CompliationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Numerics.Tests.Perf_VectorOf<Double>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector
BitwiseOrBenchmark - Duration of single invocation	8.02 ns	1.65 ns	0.21	0.32	False
BitwiseAndOperatorBenchmark - Duration of single invocation	8.01 ns	1.30 ns	0.16	0.10	False
SubtractBenchmark - Duration of single invocation	7.39 ns	1.09 ns	0.15	0.09	False
BitwiseOrOperatorBenchmark - Duration of single invocation	8.07 ns	1.49 ns	0.18	0.07	False
BitwiseAndBenchmark - Duration of single invocation	8.45 ns	1.13 ns	0.13	0.31	False
AndNotBenchmark - Duration of single invocation	15.29 ns	1.32 ns	0.09	0.05	False
UnaryNegateOperatorBenchmark - Duration of single invocation	6.89 ns	0.92 ns	0.13	0.03	False
XorBenchmark - Duration of single invocation	8.50 ns	1.29 ns	0.15	0.29	False
ConditionalSelectBenchmark - Duration of single invocation	32.57 ns	1.94 ns	0.06	0.34	False
NegateBenchmark - Duration of single invocation	7.00 ns	0.95 ns	0.14	0.05	False
OnesComplementOperatorBenchmark - Duration of single invocation	6.33 ns	1.36 ns	0.21	0.10	False
OnesComplementBenchmark - Duration of single invocation	6.32 ns	1.39 ns	0.22	0.06	False
SubtractionOperatorBenchmark - Duration of single invocation	7.14 ns	1.66 ns	0.23	0.05	False
ExclusiveOrOperatorBenchmark - Duration of single invocation	8.05 ns	1.08 ns	0.13	0.09	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
Dotnet SDK installed for dotnet commands
Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Double&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorOf&lt;Double&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Double&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorOf&lt;Double&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_VectorOf<Double>.BitwiseOrBenchmark

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.652933804400813 < 7.752520872333857.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 91.31660143023429 (T) = (0 -1.3728411751365992) / Math.Sqrt((5.284211621466104 / (299)) + (0.0663865724556792 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9067310038696352 = (14.719158906972048 - 1.3728411751365992) / 14.719158906972048 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Double&gt;.BitwiseAndOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.3004420042221483 < 7.828830175641695.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 112.81159277370458 (T) = (0 -1.2820776695026435) / Math.Sqrt((2.9257604558022394 / (299)) + (0.025537066029160126 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9030418544235037 = (13.223001140127343 - 1.2820776695026435) / 13.223001140127343 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Double&gt;.SubtractBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.0873448239414778 < 7.01144324954057.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 110.79250185847947 (T) = (0 -1.244879389895123) / Math.Sqrt((1.8464764616952547 / (299)) + (0.03406248071567537 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.8888143792607399 = (11.196406348393495 - 1.244879389895123) / 11.196406348393495 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Double&gt;.BitwiseOrOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.4866096877371773 < 7.7781410930996255.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 88.47553996549982 (T) = (0 -1.248650946874794) / Math.Sqrt((2.9670575605180587 / (299)) + (0.15083042364499166 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9055361588273326 = (13.218295290283871 - 1.248650946874794) / 13.218295290283871 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Double&gt;.BitwiseAndBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.1295372820124814 < 7.9203533244425675.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 98.42463414467258 (T) = (0 -1.2600248897676685) / Math.Sqrt((5.147932759875662 / (299)) + (0.02604080965590808 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9143212387728094 = (14.706385476635408 - 1.2600248897676685) / 14.706385476635408 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Double&gt;.AndNotBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.3157001065459457 < 14.633876568543553.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 126.78341974340707 (T) = (0 -1.2857487147730975) / Math.Sqrt((6.7980766059376725 / (299)) + (0.11882398788553422 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9441013121525904 = (23.00141137986808 - 1.2857487147730975) / 23.00141137986808 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Double&gt;.UnaryNegateOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 0.9169389804680026 < 6.63666385818933.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 110.80202546539584 (T) = (0 -0.904567006882766) / Math.Sqrt((0.5983691287142937 / (299)) + (0.06475720469676409 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9016281543412356 = (9.195385131031887 - 0.904567006882766) / 9.195385131031887 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Double&gt;.XorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.2949092251252063 < 7.964915574808199.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 94.18767692299551 (T) = (0 -1.2862940950441384) / Math.Sqrt((5.408181341437925 / (299)) + (0.040140298627470095 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.912567076203347 = (14.711781777259718 - 1.2862940950441384) / 14.711781777259718 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Double&gt;.ConditionalSelectBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.9429811212174708 < 30.90803550804634.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 108.26869912713346 (T) = (0 -1.9179902383408627) / Math.Sqrt((69.48442701488358 / (299)) + (0.05466281479521973 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9647756959127259 = (54.45076313186273 - 1.9179902383408627) / 54.45076313186273 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Double&gt;.NegateBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 0.945584434283537 < 6.659379679845138.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 143.04623868240807 (T) = (0 -0.9323050270624686) / Math.Sqrt((1.2658906036843323 / (299)) + (0.0005780378694714175 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9092661661867966 = (10.27516404720465 - 0.9323050270624686) / 10.27516404720465 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Double&gt;.OnesComplementOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.3556417893615564 < 6.046255468243147.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 113.14307262532282 (T) = (0 -1.0188827905749758) / Math.Sqrt((0.5819926899822633 / (299)) + (0.041743152390353845 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.8788251861676013 = (8.408371000133982 - 1.0188827905749758) / 8.408371000133982 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Double&gt;.OnesComplementBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.3851290101851246 < 6.031411036340767.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 144.0649442978185 (T) = (0 -0.9415537909023263) / Math.Sqrt((0.5879719699754381 / (299)) + (0.012918261399465789 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.8879819143210752 = (8.405372982368966 - 0.9415537909023263) / 8.405372982368966 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Double&gt;.SubtractionOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.6577042995740343 < 6.864029464111303.
IsChangePoint: Marked as a change because one of 3/6/2023 6:03:21 AM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 131.77419543858892 (T) = (0 -1.271660151809777) / Math.Sqrt((0.5423295633131835 / (299)) + (0.03461341857609527 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.8636568814336322 = (9.32691114286616 - 1.271660151809777) / 9.32691114286616 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Double&gt;.ExclusiveOrOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.0784535727348203 < 7.834605378974261.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 108.71293003080234 (T) = (0 -1.2573167522597917) / Math.Sqrt((3.204184500112256 / (299)) + (0.02912754101328939 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9056852843887495 = (13.33107717190434 - 1.2573167522597917) / 13.33107717190434 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

performanceautofiler · 2023-03-21T17:14:53Z

Run Information

Name	Value
Architecture	arm64
OS	ubuntu 20.04
Queue	AmpereUbuntu
Baseline	367a360a9fedc8c3c59ac7f6b4b5d92790ace993
Compare	41cb2d30974d6dba121276f28c84b6ad362e150c
Diff	Diff
Configs	CompliationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int64>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector
SubtractionOperatorBenchmark - Duration of single invocation	13.79 ns	1.14 ns	0.08	0.04	False
XorBenchmark - Duration of single invocation	12.75 ns	1.27 ns	0.10	0.04	False
BitwiseOrBenchmark - Duration of single invocation	13.06 ns	1.27 ns	0.10	0.05	False
AddBenchmark - Duration of single invocation	13.77 ns	0.04 ns	0.00	0.06	False
BitwiseAndBenchmark - Duration of single invocation	13.05 ns	1.22 ns	0.09	0.04	False
UnaryNegateOperatorBenchmark - Duration of single invocation	11.68 ns	0.93 ns	0.08	0.08	False
SubtractBenchmark - Duration of single invocation	14.15 ns	1.25 ns	0.09	0.05	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
Dotnet SDK installed for dotnet commands
Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int64&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int64&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int64&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int64&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int64>.SubtractionOperatorBenchmark

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.1369340310013962 < 13.081768762174752.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 266.3767614991747 (T) = (0 -1.26399196767858) / Math.Sqrt((0.5749940272965905 / (299)) + (0.018320518960737817 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.919539805912316 = (15.709531676012435 - 1.26399196767858) / 15.709531676012435 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int64&gt;.XorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.2729369117627085 < 12.245246839958432.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 139.45475302922563 (T) = (0 -1.0814354456712598) / Math.Sqrt((0.8153838418604522 / (299)) + (0.14220438371297905 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9300387941872891 = (15.457644463223057 - 1.0814354456712598) / 15.457644463223057 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int64&gt;.BitwiseOrBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.2688802773830445 < 12.367996789381435.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 200.9389013051645 (T) = (0 -1.238061258303211) / Math.Sqrt((1.1530566751659295 / (299)) + (0.022038559651118075 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9204374234365582 = (15.560849230617885 - 1.238061258303211) / 15.560849230617885 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int64&gt;.AddBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 0.042718170936777614 < 13.171617797457909.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 159.34626580636473 (T) = (0 -1.204980467276765) / Math.Sqrt((1.3700334891039867 / (299)) + (0.10043603503648649 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.930218979318274 = (17.26802582571456 - 1.204980467276765) / 17.26802582571456 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int64&gt;.BitwiseAndBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.2162205977398246 < 12.296795273066545.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 225.4211066610068 (T) = (0 -1.1848903760632683) / Math.Sqrt((1.1498766301933638 / (299)) + (0.004269625868412633 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9239910524745764 = (15.588827560951872 - 1.1848903760632683) / 15.588827560951872 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int64&gt;.UnaryNegateOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 0.9293507594214223 < 11.103037703283318.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 379.00124408583457 (T) = (0 -0.8972982257356296) / Math.Sqrt((0.25435838535864425 / (299)) + (0.0011624508178355308 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9274232138862044 = (12.36343290716574 - 0.8972982257356296) / 12.36343290716574 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int64&gt;.SubtractBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.253785351387602 < 13.337154940271882.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 187.38361761657583 (T) = (0 -1.1371446022584066) / Math.Sqrt((1.544359151698404 / (299)) + (0.04179414508800114 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9344621035110903 = (17.3509475155467 - 1.1371446022584066) / 17.3509475155467 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name	Value
Architecture	arm64
OS	ubuntu 20.04
Queue	AmpereUbuntu
Baseline	367a360a9fedc8c3c59ac7f6b4b5d92790ace993
Compare	41cb2d30974d6dba121276f28c84b6ad362e150c
Diff	Diff
Configs	CompliationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int16>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector
BitwiseOrBenchmark - Duration of single invocation	13.00 ns	1.29 ns	0.10	0.05	False
BitwiseAndOperatorBenchmark - Duration of single invocation	12.91 ns	1.14 ns	0.09	0.05	False
AddOperatorBenchmark - Duration of single invocation	34.53 ns	1.29 ns	0.04	0.04	False
BitwiseOrOperatorBenchmark - Duration of single invocation	12.93 ns	1.26 ns	0.10	0.09	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
Dotnet SDK installed for dotnet commands
Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int16>.BitwiseOrBenchmark

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.2904021270183899 < 12.276994766613896.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 226.62053305998663 (T) = (0 -1.232113712841319) / Math.Sqrt((0.9175410733342474 / (299)) + (0.016075237915977463 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9204892694647601 = (15.49619409288704 - 1.232113712841319) / 15.49619409288704 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;.BitwiseAndOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.1429013080280939 < 12.31163627470097.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 289.4402800693728 (T) = (0 -1.2487990568977958) / Math.Sqrt((0.270187588398123 / (299)) + (0.020139615011296352 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.912460718803935 = (14.265585001786954 - 1.2487990568977958) / 14.265585001786954 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;.AddOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.2938246843988344 < 32.86140493295487.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 720.6379002012734 (T) = (0 -1.226801226647607) / Math.Sqrt((0.42997517944522523 / (299)) + (0.015797730949934294 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9658321230470672 = (35.90510549828903 - 1.226801226647607) / 35.90510549828903 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;.BitwiseOrOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.2611961936688334 < 12.223026150161434.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 159.11088566872465 (T) = (0 -1.1904615367003961) / Math.Sqrt((0.2822195766818042 / (299)) + (0.10415591561929344 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9164217854276038 = (14.243682313519734 - 1.1904615367003961) / 14.243682313519734 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

performanceautofiler · 2023-03-21T17:14:55Z

Run Information

Name	Value
Architecture	arm64
OS	ubuntu 20.04
Queue	AmpereUbuntu
Baseline	367a360a9fedc8c3c59ac7f6b4b5d92790ace993
Compare	41cb2d30974d6dba121276f28c84b6ad362e150c
Diff	Diff
Configs	CompliationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Double>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector
AddBenchmark - Duration of single invocation	17.40 ns	1.12 ns	0.06	0.06	False
BitwiseOrBenchmark - Duration of single invocation	13.01 ns	1.17 ns	0.09	0.08	False
ExclusiveOrOperatorBenchmark - Duration of single invocation	12.88 ns	1.26 ns	0.10	0.05	False
OnesComplementBenchmark - Duration of single invocation	8.34 ns	0.93 ns	0.11	0.14	False
SubtractionOperatorBenchmark - Duration of single invocation	17.12 ns	1.32 ns	0.08	0.04	False
SubtractBenchmark - Duration of single invocation	17.49 ns	1.35 ns	0.08	0.05	False
NegateBenchmark - Duration of single invocation	15.16 ns	0.94 ns	0.06	0.06	False
XorBenchmark - Duration of single invocation	12.82 ns	1.12 ns	0.09	0.05	False
UnaryNegateOperatorBenchmark - Duration of single invocation	15.29 ns	0.96 ns	0.06	0.08	False
BitwiseAndBenchmark - Duration of single invocation	12.81 ns	1.17 ns	0.09	0.06	False
OnesComplementOperatorBenchmark - Duration of single invocation	8.45 ns	0.93 ns	0.11	0.12	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
Dotnet SDK installed for dotnet commands
Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Double&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Double&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Double&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Double&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Double>.AddBenchmark

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.1200124877394317 < 16.56141222450776.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 168.07394452078213 (T) = (0 -1.1493957237094032) / Math.Sqrt((0.8875237687155494 / (299)) + (0.17333910430496097 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9425714026445292 = (20.01434436218057 - 1.1493957237094032) / 20.01434436218057 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Double&gt;.BitwiseOrBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.1695920369384132 < 12.307451124792689.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 188.06553462603614 (T) = (0 -1.2486172562805615) / Math.Sqrt((1.3329866330070703 / (299)) + (0.024513265437498138 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.919939551731739 = (15.595931365470035 - 1.2486172562805615) / 15.595931365470035 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Double&gt;.ExclusiveOrOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.2589395129324406 < 12.279860762115984.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 161.10283763241338 (T) = (0 -1.18057512298163) / Math.Sqrt((0.3819416064066131 / (299)) + (0.09767729483052558 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9178516675270235 = (14.371260954931627 - 1.18057512298163) / 14.371260954931627 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Double&gt;.OnesComplementBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 0.9294132502959711 < 8.197581283505126.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 179.0313765344935 (T) = (0 -0.9634142393438627) / Math.Sqrt((0.18025140030694636 / (299)) + (0.028637179167091104 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.8969488411556377 = (9.348892823213198 - 0.9634142393438627) / 9.348892823213198 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Double&gt;.SubtractionOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.3220636086102158 < 16.54358440808771.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 280.58180295429435 (T) = (0 -1.2133991504055421) / Math.Sqrt((0.26280945711255943 / (299)) + (0.05454774570255405 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9353088829980652 = (18.75681247503043 - 1.2133991504055421) / 18.75681247503043 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Double&gt;.SubtractBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.3478840994965786 < 16.570102978718424.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 304.3739201680809 (T) = (0 -1.238365275002348) / Math.Sqrt((0.8233586414323434 / (299)) + (0.018973370343510116 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9381449552959811 = (20.02044103157665 - 1.238365275002348) / 20.02044103157665 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Double&gt;.NegateBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 0.936398666780825 < 14.518340344615215.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 454.32954888770684 (T) = (0 -0.9517649770443773) / Math.Sqrt((0.04766197312347465 / (299)) + (0.014711198076761816 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9371793635217829 = (15.150514709834228 - 0.9517649770443773) / 15.150514709834228 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Double&gt;.XorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.1212693815624042 < 12.203443086273666.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 239.58486359346813 (T) = (0 -1.2083108842835715) / Math.Sqrt((0.9854428412630434 / (299)) + (0.004607111619417198 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9219776962414132 = (15.486736818516354 - 1.2083108842835715) / 15.486736818516354 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Double&gt;.UnaryNegateOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 0.9562179837807054 < 14.548699836199662.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 452.74640315788696 (T) = (0 -0.9626899514025848) / Math.Sqrt((0.05646537620649747 / (299)) + (0.014349694429600702 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9365799975901047 = (15.179594998759846 - 0.9626899514025848) / 15.179594998759846 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Double&gt;.BitwiseAndBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.1728130222125803 < 12.235598609500656.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 175.80791050616622 (T) = (0 -1.1531163815858791) / Math.Sqrt((1.3229249033610928 / (299)) + (0.042062114627602454 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.926126210865874 = (15.609276241296737 - 1.1531163815858791) / 15.609276241296737 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Double&gt;.OnesComplementOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 0.9270409278999561 < 8.02359053144465.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 228.80940798177627 (T) = (0 -0.9585091885208976) / Math.Sqrt((0.19028902665754616 / (299)) + (0.012797203978102042 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.897565874097102 = (9.35732286552152 - 0.9585091885208976) / 9.35732286552152 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

performanceautofiler · 2023-03-21T17:14:56Z

Run Information

Name	Value
Architecture	arm64
OS	ubuntu 20.04
Queue	AmpereUbuntu
Baseline	367a360a9fedc8c3c59ac7f6b4b5d92790ace993
Compare	41cb2d30974d6dba121276f28c84b6ad362e150c
Diff	Diff
Configs	CompliationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt32>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector
UnaryNegateOperatorBenchmark - Duration of single invocation	15.23 ns	0.88 ns	0.06	0.04	False
BitwiseOrBenchmark - Duration of single invocation	12.99 ns	1.29 ns	0.10	0.06	False
BitwiseAndOperatorBenchmark - Duration of single invocation	13.02 ns	1.28 ns	0.10	0.05	False
OnesComplementBenchmark - Duration of single invocation	9.11 ns	0.95 ns	0.10	0.14	False
AddOperatorBenchmark - Duration of single invocation	16.85 ns	1.12 ns	0.07	0.08	False
AddBenchmark - Duration of single invocation	16.89 ns	1.26 ns	0.07	0.05	False
BitwiseAndBenchmark - Duration of single invocation	12.70 ns	1.27 ns	0.10	0.06	False
SubtractionOperatorBenchmark - Duration of single invocation	16.97 ns	0.04 ns	0.00	0.06	False
NegateBenchmark - Duration of single invocation	14.73 ns	0.95 ns	0.06	0.04	False
SubtractBenchmark - Duration of single invocation	17.39 ns	1.32 ns	0.08	0.07	False
OnesComplementOperatorBenchmark - Duration of single invocation	8.49 ns	0.94 ns	0.11	0.12	False
BitwiseOrOperatorBenchmark - Duration of single invocation	12.98 ns	1.28 ns	0.10	0.06	False
ExclusiveOrOperatorBenchmark - Duration of single invocation	12.92 ns	1.13 ns	0.09	0.07	False
XorBenchmark - Duration of single invocation	11.86 ns	1.24 ns	0.10	0.08	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
Dotnet SDK installed for dotnet commands
Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt32&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt32&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt32&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt32&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt32>.UnaryNegateOperatorBenchmark

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 0.8793761556699341 < 14.095173897334407.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 238.69195348803885 (T) = (0 -0.9376539837589198) / Math.Sqrt((0.8944832238028166 / (299)) + (0.01357469167066569 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9396858583399662 = (15.54617139449806 - 0.9376539837589198) / 15.54617139449806 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt32&gt;.BitwiseOrBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.2900035466290969 < 12.266363772056145.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 146.20667058261753 (T) = (0 -1.1722799201749934) / Math.Sqrt((1.1287514112385637 / (299)) + (0.10634570536908991 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9246581517507066 = (15.559479192707332 - 1.1722799201749934) / 15.559479192707332 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt32&gt;.BitwiseAndOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.2770993792103054 < 12.35268669936641.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 256.4783842055799 (T) = (0 -1.267511704913156) / Math.Sqrt((0.4875581037684611 / (299)) + (0.018157530091294127 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.912246879120529 = (14.444064122278744 - 1.267511704913156) / 14.444064122278744 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt32&gt;.OnesComplementBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 0.9499142833868041 < 8.383759685900745.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 148.74984193997562 (T) = (0 -0.963870144966561) / Math.Sqrt((0.5194201749588182 / (299)) + (0.02794195865855071 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.8984899955585492 = (9.49532167070789 - 0.963870144966561) / 9.49532167070789 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt32&gt;.AddOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.1212311234533465 < 15.889998154661336.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 270.582700803285 (T) = (0 -1.2451973161680718) / Math.Sqrt((0.9438924461099258 / (299)) + (0.01633888432111015 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9326770151310373 = (18.495872079821172 - 1.2451973161680718) / 18.495872079821172 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt32&gt;.AddBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.2554252493622349 < 15.921701793094291.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 258.25748234599337 (T) = (0 -1.2215008057854781) / Math.Sqrt((1.371553527632018 / (299)) + (0.012322399724967388 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9388414462668635 = (19.972689529504898 - 1.2215008057854781) / 19.972689529504898 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt32&gt;.BitwiseAndBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.272794837844132 < 12.15548195729308.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 126.66497674672217 (T) = (0 -1.1166342622241865) / Math.Sqrt((0.8937235803852885 / (299)) + (0.17622045973208436 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.927657659961076 = (15.43541806393569 - 1.1166342622241865) / 15.43541806393569 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt32&gt;.SubtractionOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 0.041329621012015635 < 15.914803669594741.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 153.19286401453382 (T) = (0 -1.1371580213393235) / Math.Sqrt((0.9464035022079365 / (299)) + (0.1742148954058686 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9385273589653409 = (18.49860364219879 - 1.1371580213393235) / 18.49860364219879 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt32&gt;.NegateBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 0.9465086770228996 < 14.03906126683715.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 334.56207317098466 (T) = (0 -0.9252984252736118) / Math.Sqrt((0.49341116098798016 / (299)) + (0.003998530346315483 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.939923493640022 = (15.402001237043143 - 0.9252984252736118) / 15.402001237043143 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt32&gt;.SubtractBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.324752486403264 < 16.031400349307297.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 198.80026165426816 (T) = (0 -1.1275749316164703) / Math.Sqrt((1.4324278222496212 / (299)) + (0.07606444475030291 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9436346019769508 = (20.004736436978213 - 1.1275749316164703) / 20.004736436978213 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt32&gt;.OnesComplementOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 0.9400158210704367 < 8.050826687790039.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 151.699624402165 (T) = (0 -0.98088674659326) / Math.Sqrt((0.5200493449270388 / (299)) + (0.025754932590132913 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.8969890685792097 = (9.522161707153456 - 0.98088674659326) / 9.522161707153456 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt32&gt;.BitwiseOrOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.2796226620546989 < 12.317119398280097.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 243.2742388557054 (T) = (0 -1.269401136939715) / Math.Sqrt((0.4933615499336345 / (299)) + (0.023104916247599833 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9121272154309411 = (14.445896339407529 - 1.269401136939715) / 14.445896339407529 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt32&gt;.ExclusiveOrOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.12978798549023 < 12.29537279562813.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 242.7854363390222 (T) = (0 -1.2247104750356863) / Math.Sqrt((0.5383419823078408 / (299)) + (0.020849398226847452 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9151329616937577 = (14.430932190850404 - 1.2247104750356863) / 14.430932190850404 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt32&gt;.XorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.2357628010412547 < 12.36282051650635.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 189.991017496703 (T) = (0 -1.2675150533798414) / Math.Sqrt((1.160519201854669 / (299)) + (0.03204528460411751 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9185567049183717 = (15.563160259042162 - 1.2675150533798414) / 15.563160259042162 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

performanceautofiler · 2023-03-21T17:14:57Z

Run Information

Name	Value
Architecture	arm64
OS	ubuntu 20.04
Queue	AmpereUbuntu
Baseline	367a360a9fedc8c3c59ac7f6b4b5d92790ace993
Compare	41cb2d30974d6dba121276f28c84b6ad362e150c
Diff	Diff
Configs	CompliationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Numerics.Tests.Perf_VectorOf<Single>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector
BitwiseAndBenchmark - Duration of single invocation	8.13 ns	1.12 ns	0.14	0.31	False
BitwiseOrBenchmark - Duration of single invocation	7.99 ns	1.13 ns	0.14	0.31	False
ConditionalSelectBenchmark - Duration of single invocation	32.63 ns	1.86 ns	0.06	0.33	False
AddBenchmark - Duration of single invocation	27.86 ns	1.31 ns	0.05	0.03	False
NegateBenchmark - Duration of single invocation	27.91 ns	0.92 ns	0.03	0.04	False
SubtractionOperatorBenchmark - Duration of single invocation	28.01 ns	1.29 ns	0.05	0.04	False
UnaryNegateOperatorBenchmark - Duration of single invocation	28.09 ns	0.89 ns	0.03	0.03	False
OnesComplementBenchmark - Duration of single invocation	6.32 ns	0.94 ns	0.15	0.11	False
SubtractBenchmark - Duration of single invocation	27.96 ns	1.75 ns	0.06	0.04	False
AndNotBenchmark - Duration of single invocation	15.56 ns	1.83 ns	0.12	0.04	False
OnesComplementOperatorBenchmark - Duration of single invocation	6.54 ns	0.88 ns	0.13	0.08	False
XorBenchmark - Duration of single invocation	8.16 ns	1.30 ns	0.16	0.32	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
Dotnet SDK installed for dotnet commands
Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Single&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorOf&lt;Single&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Single&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorOf&lt;Single&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_VectorOf<Single>.BitwiseAndBenchmark

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.1178049430946335 < 7.6684859567349735.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 97.2401920150497 (T) = (0 -1.2604194762715564) / Math.Sqrt((5.545186565383705 / (299)) + (0.01831263140299851 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9151874644428466 = (14.861240357826413 - 1.2604194762715564) / 14.861240357826413 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Single&gt;.BitwiseOrBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.1347313206839398 < 7.899409604518254.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 93.10857706764145 (T) = (0 -1.3367930525368477) / Math.Sqrt((5.560501196398837 / (299)) + (0.04126459976623201 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9096392261449644 = (14.79395312263975 - 1.3367930525368477) / 14.79395312263975 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Single&gt;.ConditionalSelectBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.8598840129324363 < 30.848664425503554.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 108.21206921999458 (T) = (0 -1.8677792879850716) / Math.Sqrt((70.51498824036221 / (299)) + (0.04372613465154495 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9658470818995826 = (54.6887174470237 - 1.8677792879850716) / 54.6887174470237 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Single&gt;.AddBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.3064148406116003 < 26.470634343616297.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 558.8726507630627 (T) = (0 -1.2467243184678596) / Math.Sqrt((0.2502835443719203 / (299)) + (0.022744276173600603 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9535867473177277 = (26.86138648808919 - 1.2467243184678596) / 26.86138648808919 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Single&gt;.NegateBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 0.9203907509775243 < 26.708221108761652.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 503.61728518297855 (T) = (0 -0.9714781847990905) / Math.Sqrt((0.3337335529237669 / (299)) + (0.02602121206791304 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9632872765437619 = (26.461621294783537 - 0.9714781847990905) / 26.461621294783537 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Single&gt;.SubtractionOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.2879238624735008 < 26.530831893566546.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 418.9830702401036 (T) = (0 -1.2894635354133972) / Math.Sqrt((0.892794577748643 / (299)) + (0.004976837957548047 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.948872996448935 = (25.220792259524803 - 1.2894635354133972) / 25.220792259524803 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Single&gt;.UnaryNegateOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 0.8919917806341611 < 26.69074192025785.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 375.70540693635803 (T) = (0 -0.9621607877413122) / Math.Sqrt((0.8486763676464816 / (299)) + (0.02522573050801341 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9621580650862684 = (25.425782004402087 - 0.9621607877413122) / 25.425782004402087 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Single&gt;.OnesComplementBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 0.941970816034508 < 6.013245851636338.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 125.92365783987917 (T) = (0 -0.9456451581317804) / Math.Sqrt((0.8816721627804722 / (299)) + (0.012312521729238754 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.8892086441310642 = (8.53536948541781 - 0.9456451581317804) / 8.53536948541781 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Single&gt;.SubtractBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.747813857430025 < 26.45944748056543.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 255.00892651533704 (T) = (0 -1.2751829881045595) / Math.Sqrt((0.2496145043837052 / (299)) + (0.1663767276825341 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9525518848886848 = (26.875313911056978 - 1.2751829881045595) / 26.875313911056978 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Single&gt;.AndNotBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.8334424616592198 < 14.76805585238073.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 140.04951130927947 (T) = (0 -1.3776179420924541) / Math.Sqrt((6.801149422095139 / (299)) + (0.025098047280930444 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9404594629339906 = (23.13747927005027 - 1.3776179420924541) / 23.13747927005027 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Single&gt;.OnesComplementOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 0.8824808718277121 < 6.158223963944893.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 118.78219384336478 (T) = (0 -0.9690958206515519) / Math.Sqrt((0.816136489245724 / (299)) + (0.024028427836958924 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.8865471380149194 = (8.541836703766803 - 0.9690958206515519) / 8.541836703766803 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Single&gt;.XorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.2986749333393406 < 7.748048050226016.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 94.78387727173671 (T) = (0 -1.320315099083504) / Math.Sqrt((5.333816378531622 / (299)) + (0.04514895114661068 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9110336071738284 = (14.840605054801115 - 1.320315099083504) / 14.840605054801115 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name	Value
Architecture	arm64
OS	ubuntu 20.04
Queue	AmpereUbuntu
Baseline	367a360a9fedc8c3c59ac7f6b4b5d92790ace993
Compare	41cb2d30974d6dba121276f28c84b6ad362e150c
Diff	Diff
Configs	CompliationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt16>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector
AddOperatorBenchmark - Duration of single invocation	33.64 ns	1.13 ns	0.03	0.04	False
BitwiseOrOperatorBenchmark - Duration of single invocation	12.98 ns	1.12 ns	0.09	0.04	False
BitwiseOrBenchmark - Duration of single invocation	12.88 ns	1.14 ns	0.09	0.04	False
BitwiseAndOperatorBenchmark - Duration of single invocation	13.01 ns	1.29 ns	0.10	0.06	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
Dotnet SDK installed for dotnet commands
Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt16>.AddOperatorBenchmark

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.133244767647153 < 32.09101888001242.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 710.0554168518454 (T) = (0 -1.2129393816972367) / Math.Sqrt((0.5088650830019894 / (299)) + (0.012215903440962309 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9661730375457827 = (35.85717704741821 - 1.2129393816972367) / 35.85717704741821 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;.BitwiseOrOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.1213167829208888 < 12.287780993860181.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 221.78251901037288 (T) = (0 -1.1747720331869018) / Math.Sqrt((0.3773274090666502 / (299)) + (0.04094930214470248 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9182177601597413 = (14.364635102701117 - 1.1747720331869018) / 14.364635102701117 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;.BitwiseOrBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.1355553834622072 < 12.275528712340183.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 198.41409826896015 (T) = (0 -1.2668435822684458) / Math.Sqrt((1.299528982779126 / (299)) + (0.01586344373329482 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9188573253308805 = (15.612544045834477 - 1.2668435822684458) / 15.612544045834477 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;.BitwiseAndOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.2880507895781617 < 12.242721527888406.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 231.47321924619183 (T) = (0 -1.264028584897505) / Math.Sqrt((0.4032842187055573 / (299)) + (0.033520228154539006 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.912102069337802 = (14.380640993191447 - 1.264028584897505) / 14.380640993191447 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

performanceautofiler · 2023-03-21T17:14:58Z

Run Information

Name	Value
Architecture	arm64
OS	ubuntu 20.04
Queue	AmpereUbuntu
Baseline	367a360a9fedc8c3c59ac7f6b4b5d92790ace993
Compare	41cb2d30974d6dba121276f28c84b6ad362e150c
Diff	Diff
Configs	CompliationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Runtime.Intrinsics.Tests.Perf_Vector128Int

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector
SubtractionOperatorBenchmark - Duration of single invocation	16.62 ns	1.15 ns	0.07	0.07	False
SubtractBenchmark - Duration of single invocation	16.54 ns	1.49 ns	0.09	0.08	False
UnaryNegateOperatorBenchmark - Duration of single invocation	14.85 ns	0.93 ns	0.06	0.12	False
XorBenchmark - Duration of single invocation	13.03 ns	1.56 ns	0.12	0.08	False
BitwiseAndBenchmark - Duration of single invocation	12.76 ns	1.17 ns	0.09	0.11	False
AddBenchmark - Duration of single invocation	16.96 ns	1.62 ns	0.10	0.15	False
BitwiseOrBenchmark - Duration of single invocation	12.79 ns	1.58 ns	0.12	0.07	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
Dotnet SDK installed for dotnet commands
Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Int*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Int* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Int*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Int* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Runtime.Intrinsics.Tests.Perf_Vector128Int.SubtractionOperatorBenchmark

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.1516886034411962 < 15.877153678024351.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 36.070782295817494 (T) = (0 -1.2723037611074444) / Math.Sqrt((40.93835555325876 / (299)) + (0.03906384465504619 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9135939848653847 = (14.724712846961777 - 1.2723037611074444) / 14.724712846961777 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Int.SubtractBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.493451513254826 < 15.772085386457132.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 36.483560801081545 (T) = (0 -1.3051437807069346) / Math.Sqrt((47.06960539308133 / (299)) + (0.1600039557148894 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9193542420557722 = (16.18366314579787 - 1.3051437807069346) / 16.18366314579787 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Int.UnaryNegateOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 0.9298757722870368 < 14.283431089196245.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 38.52513551474297 (T) = (0 -1.1011079115102471) / Math.Sqrt((25.067154083282624 / (299)) + (0.05624514337259411 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9116418182453205 = (12.461867024011422 - 1.1011079115102471) / 12.461867024011422 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Int.XorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.5643681179592197 < 12.36615301391515.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 36.868349611519534 (T) = (0 -1.325087309955045) / Math.Sqrt((26.85485120936962 / (299)) + (0.04852754677183782 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.8943215227231703 = (12.538856956501336 - 1.325087309955045) / 12.538856956501336 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Int.BitwiseAndBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.1655033391621699 < 12.235105771081033.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 35.060606246909856 (T) = (0 -1.1541607786834986) / Math.Sqrt((27.568838805712435 / (299)) + (0.2117032503443446 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9073629256911871 = (12.458951098088594 - 1.1541607786834986) / 12.458951098088594 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Int.AddBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.620214108573072 < 15.911298164567363.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 35.94900752872796 (T) = (0 -1.2590217519064133) / Math.Sqrt((49.907737170386525 / (299)) + (0.07422280758641346 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9219279243449696 = (16.126402959613024 - 1.2590217519064133) / 16.126402959613024 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Int.BitwiseOrBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.5775647164292275 < 12.210108389873465.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 36.48853640638721 (T) = (0 -1.3995816452838623) / Math.Sqrt((26.9623074055699 / (299)) + (0.05641327371539812 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.8884400919289606 = (12.545561120332147 - 1.3995816452838623) / 12.545561120332147 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

performanceautofiler · 2023-03-21T17:14:59Z

Run Information

Name	Value
Architecture	arm64
OS	ubuntu 20.04
Queue	AmpereUbuntu
Baseline	367a360a9fedc8c3c59ac7f6b4b5d92790ace993
Compare	41cb2d30974d6dba121276f28c84b6ad362e150c
Diff	Diff
Configs	CompliationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Numerics.Tests.Perf_VectorOf<Int32>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector
ConditionalSelectBenchmark - Duration of single invocation	32.66 ns	1.85 ns	0.06	0.27	False
BitwiseAndBenchmark - Duration of single invocation	8.39 ns	1.14 ns	0.14	0.33	False
AddBenchmark - Duration of single invocation	11.72 ns	1.11 ns	0.09	0.05	False
UnaryNegateOperatorBenchmark - Duration of single invocation	11.79 ns	0.85 ns	0.07	0.07	False
SubtractBenchmark - Duration of single invocation	11.56 ns	1.29 ns	0.11	0.05	False
SubtractionOperatorBenchmark - Duration of single invocation	11.66 ns	1.16 ns	0.10	0.08	False
BitwiseOrOperatorBenchmark - Duration of single invocation	7.99 ns	1.35 ns	0.17	0.10	False
ExclusiveOrOperatorBenchmark - Duration of single invocation	8.46 ns	1.14 ns	0.14	0.07	False
XorBenchmark - Duration of single invocation	8.02 ns	1.30 ns	0.16	0.34	False
AddOperatorBenchmark - Duration of single invocation	11.78 ns	1.29 ns	0.11	0.03	False
BitwiseOrBenchmark - Duration of single invocation	7.97 ns	1.28 ns	0.16	0.31	False
BitwiseAndOperatorBenchmark - Duration of single invocation	8.04 ns	1.30 ns	0.16	0.07	False
AndNotBenchmark - Duration of single invocation	15.28 ns	1.47 ns	0.10	0.07	False
OnesComplementBenchmark - Duration of single invocation	6.90 ns	0.98 ns	0.14	0.10	False
NegateBenchmark - Duration of single invocation	11.38 ns	0.99 ns	0.09	0.06	False
OnesComplementOperatorBenchmark - Duration of single invocation	6.90 ns	0.93 ns	0.13	0.07	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
Dotnet SDK installed for dotnet commands
Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Int32&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorOf&lt;Int32&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Int32&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorOf&lt;Int32&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_VectorOf<Int32>.ConditionalSelectBenchmark

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.8453865320062504 < 30.927627974030287.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 104.48030427221535 (T) = (0 -1.8281921081902088) / Math.Sqrt((72.39454337671634 / (299)) + (0.183041453662857 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9663360898118905 = (54.30718232001307 - 1.8281921081902088) / 54.30718232001307 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int32&gt;.BitwiseAndBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.13856774170697 < 7.9521949037725275.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 92.25241196181189 (T) = (0 -1.3191520294950163) / Math.Sqrt((5.549280472340844 / (299)) + (0.04985419582080582 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9108208421307383 = (14.79215616084779 - 1.3191520294950163) / 14.79215616084779 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int32&gt;.AddBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.112972886778522 < 11.129548271463996.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 185.41590408004905 (T) = (0 -1.3518897431144123) / Math.Sqrt((0.43187251719690967 / (299)) + (0.05069249951599639 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.899522479781098 = (13.454648762918943 - 1.3518897431144123) / 13.454648762918943 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int32&gt;.UnaryNegateOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 0.8511952576090949 < 10.946724682986044.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 163.8341301372507 (T) = (0 -1.0210042409240492) / Math.Sqrt((0.6089646252733794 / (299)) + (0.04411482706311396 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9148883000018827 = (11.996050377875601 - 1.0210042409240492) / 11.996050377875601 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int32&gt;.SubtractBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.2894191992392956 < 11.057972658229476.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 124.10689726187461 (T) = (0 -1.2439472812183061) / Math.Sqrt((0.6869567724797014 / (299)) + (0.1361447822327769 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9083185896990454 = (13.568151680203314 - 1.2439472812183061) / 13.568151680203314 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int32&gt;.SubtractionOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.1613653407841487 < 11.139659652232165.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 170.12603429615027 (T) = (0 -1.3369001416299087) / Math.Sqrt((0.5235792572938756 / (299)) + (0.04379200136254339 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.8916726076930236 = (12.341293491506038 - 1.3369001416299087) / 12.341293491506038 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int32&gt;.BitwiseOrOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.351830268520858 < 7.620350346672579.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 99.22375537223314 (T) = (0 -1.4270922665991803) / Math.Sqrt((3.3144115570703936 / (299)) + (0.06024364045112963 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.8930785286527847 = (13.34710651301141 - 1.4270922665991803) / 13.34710651301141 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int32&gt;.ExclusiveOrOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.1441062235698893 < 7.940626891357864.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 84.36855924061085 (T) = (0 -1.3303219797496206) / Math.Sqrt((2.949785415545679 / (299)) + (0.17948525106112875 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.8993178708912628 = (13.21308946807099 - 1.3303219797496206) / 13.21308946807099 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int32&gt;.XorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.299022407871748 < 7.632268357841124.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 91.9396993706102 (T) = (0 -1.3326996396751078) / Math.Sqrt((5.74327608418465 / (299)) + (0.045684352638511755 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9105012346094895 = (14.890704177427823 - 1.3326996396751078) / 14.890704177427823 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int32&gt;.AddOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.2923170526854315 < 11.036089521783731.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 209.79588819890398 (T) = (0 -1.3224831518304092) / Math.Sqrt((0.2052246988764993 / (299)) + (0.03618483305505245 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.8917506657959058 = (12.21700956919503 - 1.3224831518304092) / 12.21700956919503 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int32&gt;.BitwiseOrBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.2815683745856323 < 7.923340444903863.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 95.9061193644845 (T) = (0 -1.3263091680349142) / Math.Sqrt((5.186156581213592 / (299)) + (0.03668945317538796 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9096441959628304 = (14.67873793131553 - 1.3263091680349142) / 14.67873793131553 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int32&gt;.BitwiseAndOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.2976144833821257 < 7.655594120814523.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 104.49240542415751 (T) = (0 -1.4084262092286854) / Math.Sqrt((2.843750419016749 / (299)) + (0.05924970431779908 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.8935555163176797 = (13.231556587113348 - 1.4084262092286854) / 13.231556587113348 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int32&gt;.AndNotBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.4705365910913082 < 14.783194217357964.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 133.0221107984586 (T) = (0 -1.5477721423561013) / Math.Sqrt((6.669402056775815 / (299)) + (0.06620016829590927 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9326766613276577 = (22.990127537925524 - 1.5477721423561013) / 22.990127537925524 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int32&gt;.OnesComplementBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 0.975094232035454 < 6.274911250863299.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/6/2023 6:03:21 AM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 114.29350594642301 (T) = (0 -0.9630772408158427) / Math.Sqrt((0.8382217540381234 / (299)) + (0.027947746055674776 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.8867834932928134 = (8.506509066797678 - 0.9630772408158427) / 8.506509066797678 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int32&gt;.NegateBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 0.9931988977667091 < 10.86697856073458.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 200.11738015400468 (T) = (0 -0.9895743409439175) / Math.Sqrt((0.5287178275759997 / (299)) + (0.02925306147261986 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9217547773431942 = (12.647089590176325 - 0.9895743409439175) / 12.647089590176325 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int32&gt;.OnesComplementOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 0.9291581419774402 < 6.036196040404511.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 108.3169704345916 (T) = (0 -1.0169629948513874) / Math.Sqrt((0.7693777864238006 / (299)) + (0.039729493719656755 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.8804419334550924 = (8.506017404266084 - 1.0169629948513874) / 8.506017404266084 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

performanceautofiler · 2023-03-21T17:15:01Z

Run Information

Name	Value
Architecture	arm64
OS	ubuntu 20.04
Queue	AmpereUbuntu
Baseline	367a360a9fedc8c3c59ac7f6b4b5d92790ace993
Compare	41cb2d30974d6dba121276f28c84b6ad362e150c
Diff	Diff
Configs	CompliationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Byte>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector
OnesComplementOperatorBenchmark - Duration of single invocation	8.45 ns	0.93 ns	0.11	0.09	False
OnesComplementBenchmark - Duration of single invocation	8.38 ns	0.97 ns	0.12	0.14	False
ExclusiveOrOperatorBenchmark - Duration of single invocation	12.89 ns	1.24 ns	0.10	0.03	False
BitwiseOrOperatorBenchmark - Duration of single invocation	13.04 ns	1.30 ns	0.10	0.05	False
UnaryNegateOperatorBenchmark - Duration of single invocation	46.76 ns	0.85 ns	0.02	0.03	False
NegateBenchmark - Duration of single invocation	46.73 ns	0.91 ns	0.02	0.03	False
SubtractBenchmark - Duration of single invocation	49.34 ns	1.12 ns	0.02	0.04	False
SubtractionOperatorBenchmark - Duration of single invocation	49.53 ns	1.30 ns	0.03	0.05	False
AddOperatorBenchmark - Duration of single invocation	49.43 ns	1.33 ns	0.03	0.05	False
BitwiseAndOperatorBenchmark - Duration of single invocation	13.05 ns	1.20 ns	0.09	0.06	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
Dotnet SDK installed for dotnet commands
Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Byte>.OnesComplementOperatorBenchmark

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 0.9312186776599715 < 8.025090425469296.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 171.89343833447035 (T) = (0 -0.905478867980442) / Math.Sqrt((0.4854535579084364 / (299)) + (0.015788036779795258 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9047000801550327 = (9.501360226256885 - 0.905478867980442) / 9.501360226256885 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;.OnesComplementBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 0.9723567926561959 < 8.288088707034984.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 151.23667390053964 (T) = (0 -0.8912830000213735) / Math.Sqrt((0.4728630869595736 / (299)) + (0.0299961012745009 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9062829023194311 = (9.510356403260332 - 0.8912830000213735) / 9.510356403260332 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;.ExclusiveOrOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.2359841286777111 < 12.348198472457348.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 380.23267135633307 (T) = (0 -1.1812705430911108) / Math.Sqrt((0.29838723354736496 / (299)) + (0.0035467367903562175 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9175401388772411 = (14.325400588930659 - 1.1812705430911108) / 14.325400588930659 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;.BitwiseOrOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.2970959716042336 < 12.243194989084717.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 255.54861447008108 (T) = (0 -1.2368135073378943) / Math.Sqrt((0.28476158372318155 / (299)) + (0.029912130562405582 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9135260795896268 = (14.30273429802229 - 1.2368135073378943) / 14.30273429802229 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;.UnaryNegateOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 0.8537998647189048 < 44.448393421974856.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 530.2196686050235 (T) = (0 -0.8830961292608062) / Math.Sqrt((2.0237436815822734 / (299)) + (0.02777244905007125 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9820586376208087 = (49.22124143064155 - 0.8830961292608062) / 49.22124143064155 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;.NegateBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 0.9126951568754573 < 44.27582690926295.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 460.4795011166101 (T) = (0 -0.8753582901644046) / Math.Sqrt((2.632448705942362 / (299)) + (0.03876941274402723 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.982164169313402 = (49.078638699018256 - 0.8753582901644046) / 49.078638699018256 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;.SubtractBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.1204178544990275 < 46.95434348626227.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 489.66537585259596 (T) = (0 -1.2305063528678875) / Math.Sqrt((3.234203467352813 / (299)) + (0.019713222402919867 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9774935191271622 = (54.67342317176453 - 1.2305063528678875) / 54.67342317176453 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;.SubtractionOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.2983675951714984 < 47.05462201240572.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 414.07719494620216 (T) = (0 -1.2567706639995422) / Math.Sqrt((4.6290237810632915 / (299)) + (0.010888016384892781 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9766292494882252 = (53.77536606564447 - 1.2567706639995422) / 53.77536606564447 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;.AddOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.327156726851909 < 47.05150268203778.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 411.21391644854526 (T) = (0 -1.1781735529794686) / Math.Sqrt((4.035767570899494 / (299)) + (0.052357094240426026 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.978120923646383 = (53.84932772926197 - 1.1781735529794686) / 53.84932772926197 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;.BitwiseAndOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.1970316928453373 < 12.3123307686158.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 130.54650284049035 (T) = (0 -1.1049607942589736) / Math.Sqrt((0.2909990338964031 / (299)) + (0.16645449102708526 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.922745633364976 = (14.302891116552468 - 1.1049607942589736) / 14.302891116552468 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name	Value
Architecture	arm64
OS	ubuntu 20.04
Queue	AmpereUbuntu
Baseline	367a360a9fedc8c3c59ac7f6b4b5d92790ace993
Compare	41cb2d30974d6dba121276f28c84b6ad362e150c
Diff	Diff
Configs	CompliationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Single>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector
ExclusiveOrOperatorBenchmark - Duration of single invocation	12.98 ns	1.29 ns	0.10	0.05	False
SubtractBenchmark - Duration of single invocation	35.69 ns	1.17 ns	0.03	0.04	False
OnesComplementOperatorBenchmark - Duration of single invocation	8.44 ns	0.99 ns	0.12	0.13	False
NegateBenchmark - Duration of single invocation	33.45 ns	0.93 ns	0.03	0.03	False
XorBenchmark - Duration of single invocation	12.93 ns	1.29 ns	0.10	0.06	False
SubtractionOperatorBenchmark - Duration of single invocation	36.21 ns	1.26 ns	0.03	0.06	False
UnaryNegateOperatorBenchmark - Duration of single invocation	33.70 ns	0.93 ns	0.03	0.03	False
AddBenchmark - Duration of single invocation	35.76 ns	1.29 ns	0.04	0.03	False
OnesComplementBenchmark - Duration of single invocation	8.47 ns	0.91 ns	0.11	0.14	False
BitwiseOrBenchmark - Duration of single invocation	13.68 ns	1.29 ns	0.09	0.05	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
Dotnet SDK installed for dotnet commands
Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Single&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Single&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Single&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Single&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Single>.ExclusiveOrOperatorBenchmark

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.2897933379326538 < 12.381167122948415.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 265.3813371303545 (T) = (0 -1.2864088402401612) / Math.Sqrt((0.4733459307356428 / (299)) + (0.015621482021971762 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9108188976688184 = (14.424679742833563 - 1.2864088402401612) / 14.424679742833563 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Single&gt;.SubtractBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.1683712473042156 < 34.09676113076658.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 179.67144988705866 (T) = (0 -1.2099353471360925) / Math.Sqrt((15.558004482565448 / (299)) + (0.016459864055121027 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9715664398788102 = (42.553072565626636 - 1.2099353471360925) / 42.553072565626636 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Single&gt;.OnesComplementOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 0.994887536274642 < 7.951512791518901.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 196.74199361175755 (T) = (0 -0.9706094585701367) / Math.Sqrt((0.27078354979594704 / (299)) + (0.016698048235188338 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.8966834613330938 = (9.394521642845524 - 0.9706094585701367) / 9.394521642845524 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Single&gt;.NegateBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 0.9324355179039556 < 31.804020902944714.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 469.43969857639024 (T) = (0 -0.9404258239952049) / Math.Sqrt((0.31298291720343163 / (299)) + (0.059622025217238125 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9705513876820443 = (31.93446991122909 - 0.9404258239952049) / 31.93446991122909 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Single&gt;.XorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.2947847828799872 < 12.308680872143082.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 247.63133558134376 (T) = (0 -1.22867062445601) / Math.Sqrt((0.8276029614060484 / (299)) + (0.009380747380833991 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9203733730438628 = (15.430398993703822 - 1.22867062445601) / 15.430398993703822 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Single&gt;.SubtractionOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.256784795997252 < 34.487663383493604.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 274.89879952277516 (T) = (0 -1.1266298816324347) / Math.Sqrt((4.291645342081398 / (299)) + (0.08386980289791046 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9711353965135291 = (39.03153847793174 - 1.1266298816324347) / 39.03153847793174 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Single&gt;.UnaryNegateOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 0.9306420744766738 < 31.942216917456864.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 753.7282843230233 (T) = (0 -0.9168787714885318) / Math.Sqrt((0.4824040185748331 / (299)) + (0.0012703488911542924 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9712098785303164 = (31.846992116862594 - 0.9168787714885318) / 31.846992116862594 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Single&gt;.AddBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.286205566464036 < 34.1389134655876.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 179.40295080556612 (T) = (0 -1.2238900396520096) / Math.Sqrt((15.617705797686979 / (299)) + (0.014274223719560553 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9712267809719612 = (42.53573569434008 - 1.2238900396520096) / 42.53573569434008 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Single&gt;.OnesComplementBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 0.9141934195586917 < 8.036842905562757.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 276.2723661394373 (T) = (0 -0.9294469861961782) / Math.Sqrt((0.27319884469066275 / (299)) + (0.0005694910695132237 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9013729395851875 = (9.423853679578858 - 0.9294469861961782) / 9.423853679578858 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Single&gt;.BitwiseOrBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.2880944039694735 < 12.262058288194046.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 162.03906820966174 (T) = (0 -1.1526266250456538) / Math.Sqrt((0.831179690609081 / (299)) + (0.08977849220723855 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9253177303161554 = (15.43373855568548 - 1.1526266250456538) / 15.43373855568548 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

performanceautofiler · 2023-03-21T17:15:02Z

Run Information

Name	Value
Architecture	arm64
OS	ubuntu 20.04
Queue	AmpereUbuntu
Baseline	367a360a9fedc8c3c59ac7f6b4b5d92790ace993
Compare	41cb2d30974d6dba121276f28c84b6ad362e150c
Diff	Diff
Configs	CompliationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Numerics.Tests.Perf_VectorOf<Int16>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector
ConditionalSelectBenchmark - Duration of single invocation	32.49 ns	1.93 ns	0.06	0.38	False
BitwiseOrOperatorBenchmark - Duration of single invocation	8.08 ns	1.13 ns	0.14	0.07	False
BitwiseAndOperatorBenchmark - Duration of single invocation	8.37 ns	1.75 ns	0.21	0.10	False
BitwiseOrBenchmark - Duration of single invocation	8.48 ns	1.31 ns	0.16	0.30	False
BitwiseAndBenchmark - Duration of single invocation	8.03 ns	1.10 ns	0.14	0.33	False
ExclusiveOrOperatorBenchmark - Duration of single invocation	8.05 ns	1.33 ns	0.16	0.10	False
AddOperatorBenchmark - Duration of single invocation	23.91 ns	1.69 ns	0.07	0.06	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
Dotnet SDK installed for dotnet commands
Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Int16&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorOf&lt;Int16&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Int16&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorOf&lt;Int16&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_VectorOf<Int16>.ConditionalSelectBenchmark

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.9280825948546025 < 30.919115962737873.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 108.05639111756534 (T) = (0 -1.799031948540489) / Math.Sqrt((69.50408483185925 / (299)) + (0.06736269494470686 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9668775656811892 = (54.314605358543616 - 1.799031948540489) / 54.314605358543616 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int16&gt;.BitwiseOrOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.133041548332985 < 7.675312309786614.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 104.29851071090698 (T) = (0 -1.3218773210937496) / Math.Sqrt((3.197829960298923 / (299)) + (0.04536339566542541 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9006987842145204 = (13.311793925558785 - 1.3218773210937496) / 13.311793925558785 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int16&gt;.BitwiseAndOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.7495253626619354 < 7.935498259239968.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 102.52307077895318 (T) = (0 -1.3474600858848593) / Math.Sqrt((3.251673422750332 / (299)) + (0.04914409123525073 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.8987326725753523 = (13.305970643764603 - 1.3474600858848593) / 13.305970643764603 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int16&gt;.BitwiseOrBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.3143455505768469 < 7.981385611260291.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 86.65066682109878 (T) = (0 -1.2242533305311878) / Math.Sqrt((5.61546190283757 / (299)) + (0.10694455032919448 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9175508458117144 = (14.848585683916324 - 1.2242533305311878) / 14.848585683916324 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int16&gt;.BitwiseAndBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.0970592220922426 < 7.766622262048093.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 96.90620433170751 (T) = (0 -1.2973893075876874) / Math.Sqrt((5.383152978745122 / (299)) + (0.02426399963356311 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.9122086726952904 = (14.778103343678326 - 1.2973893075876874) / 14.778103343678326 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int16&gt;.ExclusiveOrOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.3256623789549378 < 7.6329565958490155.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 105.88187652725266 (T) = (0 -1.3271872940168827) / Math.Sqrt((2.9189581403793863 / (299)) + (0.04919835924299826 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.8991734851019011 = (13.16307813830732 - 1.3271872940168827) / 13.16307813830732 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int16&gt;.AddOperatorBenchmark

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.688034729066479 < 22.62609064907572.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/21/2023 3:33:39 AM falls between 3/11/2023 9:11:51 PM and 3/21/2023 3:33:39 AM.
IsImprovementStdDev: Marked as improvement because 445.62355940645585 (T) = (0 -1.2851912951969284) / Math.Sqrt((0.23783910180425288 / (299)) + (0.03404262033211679 / (18))) is greater than 1.9675235316906419 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (18) - 2, .975) and 0.947292349767593 = (24.38339196549375 - 1.2851912951969284) / 24.38339196549375 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

kotlarmilos · 2023-03-23T09:48:20Z

Should be dotnet/runtime#83337. @fanyang-mono @jandupej Well done!

/cc: @SamMonoRT

performanceautofiler bot added ampere arch-arm64 branch-refs/heads/main kind-micro_mono mono-aot os-linux perf-improvement runtime-mono untriaged labels Mar 21, 2023

kotlarmilos removed the untriaged label Mar 23, 2023

kotlarmilos closed this as completed Mar 28, 2023

kotlarmilos mentioned this issue Apr 4, 2023

.NET 8 Per-Preview Performance report on WASM, Mono AOT, and Interpreter dotnet/runtime#84302

Closed

[Perf] Linux/arm64: 223 Improvements on 3/15/2023 4:14:49 PM #14322

[Perf] Linux/arm64: 223 Improvements on 3/15/2023 4:14:49 PM #14322

Comments

performanceautofiler bot commented Mar 21, 2023 • edited Loading

Run Information

Improvements in System.Runtime.Intrinsics.Tests.Perf_Vector128Float

Repro

Payloads

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

Payloads

Histogram

System.Runtime.Intrinsics.Tests.Perf_Vector128Float.BitwiseAndOperatorBenchmark

Description of detection logic

Description of detection logic

Description of detection logic

Description of detection logic

Description of detection logic

Description of detection logic

Description of detection logic

Description of detection logic

Description of detection logic

Description of detection logic

Description of detection logic

Description of detection logic

Description of detection logic

Description of detection logic

Docs

performanceautofiler bot commented Mar 21, 2023 • edited Loading

Run Information

Improvements in System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int32>

Repro

Payloads

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

Payloads

Histogram

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int32>.NegateBenchmark

Description of detection logic

Description of detection logic

Description of detection logic

Description of detection logic

Description of detection logic

Description of detection logic

Description of detection logic

Description of detection logic

Description of detection logic

Description of detection logic

Description of detection logic

Description of detection logic

Description of detection logic

Description of detection logic

Docs

Run Information

Improvements in System.Numerics.Tests.Perf_VectorOf<UInt16>

Repro

Payloads

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

Payloads

Histogram

System.Numerics.Tests.Perf_VectorOf<UInt16>.ExclusiveOrOperatorBenchmark

Description of detection logic

Description of detection logic

Description of detection logic

Description of detection logic

Description of detection logic

Description of detection logic

Description of detection logic

Docs

performanceautofiler bot commented Mar 21, 2023 • edited Loading

Run Information

Improvements in System.Runtime.Intrinsics.Tests.Perf_Vector128Of<SByte>

Repro

Payloads

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

Payloads

Histogram

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<SByte>.ExclusiveOrOperatorBenchmark

Description of detection logic

Description of detection logic

Description of detection logic

Description of detection logic

performanceautofiler bot commented Mar 21, 2023 •

edited

Loading

performanceautofiler bot commented Mar 21, 2023 •

edited

Loading

performanceautofiler bot commented Mar 21, 2023 •

edited

Loading

performanceautofiler bot commented Mar 21, 2023 •

edited

Loading

performanceautofiler bot commented Mar 21, 2023 •

edited

Loading