Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update perf proj from arcade #38127

Merged
merged 7 commits into from
Jun 23, 2020
Merged

Update perf proj from arcade #38127

Changes from 3 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
9965784
Update perf proj from arcade
DrewScoggins Jun 18, 2020
2fcb7b9
Add timeout for pr and internal jobs
DrewScoggins Jun 19, 2020
b2bc9a2
Change to a variable that is set
DrewScoggins Jun 19, 2020
ec3b233
Escape variable with single quotes
DrewScoggins Jun 19, 2020
223d805
Change timeout from days to minutes
DrewScoggins Jun 19, 2020
e5ca09d
Fix typo
DrewScoggins Jun 19, 2020
1257340
Change to default
DrewScoggins Jun 22, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 40 additions & 2 deletions eng/common/performance/perfhelixpublish.proj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,21 +64,54 @@
<WorkItemCommand>$(WorkItemCommand) $(CliArguments)</WorkItemCommand>
</PropertyGroup>

<PropertyGroup Condition="$(HelixSourcePrefix) == 'pr'">
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you need to also escape $(HelixSourcePrefix) for the comparison to be correct:

Suggested change
<PropertyGroup Condition="$(HelixSourcePrefix) == 'pr'">
<PropertyGroup Condition="'$(HelixSourcePrefix)' == 'pr'">

<WorkItemTimeout>15</WorkItemTimeout>
</PropertyGroup>

<PropertyGroup Condition="$(HelixSourcePrefix) == 'official'">
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ditto.

<WorkItemTimeout>45</WorkItemTimeout>
</PropertyGroup>

<ItemGroup>
<HelixCorrelationPayload Include="$(CorrelationPayloadDirectory)">
<PayloadDirectory>%(Identity)</PayloadDirectory>
</HelixCorrelationPayload>
</ItemGroup>

<PropertyGroup>
<PartitionCount>5</PartitionCount>
<PartitionCount>30</PartitionCount>
</PropertyGroup>
<ItemGroup>
<Partition Include="$(BuildConfig).Partition0" Index="0" />
<Partition Include="$(BuildConfig).Partition1" Index="1" />
<Partition Include="$(BuildConfig).Partition2" Index="2" />
<Partition Include="$(BuildConfig).Partition3" Index="3" />
<Partition Include="$(BuildConfig).Partition4" Index="4" />
<Partition Include="$(BuildConfig).Partition5" Index="5" />
<Partition Include="$(BuildConfig).Partition6" Index="6" />
<Partition Include="$(BuildConfig).Partition7" Index="7" />
<Partition Include="$(BuildConfig).Partition8" Index="8" />
<Partition Include="$(BuildConfig).Partition9" Index="9" />
<Partition Include="$(BuildConfig).Partition10" Index="10" />
<Partition Include="$(BuildConfig).Partition11" Index="11" />
<Partition Include="$(BuildConfig).Partition12" Index="12" />
<Partition Include="$(BuildConfig).Partition13" Index="13" />
<Partition Include="$(BuildConfig).Partition14" Index="14" />
<Partition Include="$(BuildConfig).Partition15" Index="15" />
<Partition Include="$(BuildConfig).Partition16" Index="16" />
<Partition Include="$(BuildConfig).Partition17" Index="17" />
<Partition Include="$(BuildConfig).Partition18" Index="18" />
<Partition Include="$(BuildConfig).Partition19" Index="19" />
<Partition Include="$(BuildConfig).Partition20" Index="20" />
<Partition Include="$(BuildConfig).Partition21" Index="21" />
<Partition Include="$(BuildConfig).Partition22" Index="22" />
<Partition Include="$(BuildConfig).Partition23" Index="23" />
<Partition Include="$(BuildConfig).Partition24" Index="24" />
<Partition Include="$(BuildConfig).Partition25" Index="25" />
<Partition Include="$(BuildConfig).Partition26" Index="26" />
<Partition Include="$(BuildConfig).Partition27" Index="27" />
<Partition Include="$(BuildConfig).Partition28" Index="28" />
<Partition Include="$(BuildConfig).Partition29" Index="29" />
</ItemGroup>

<PropertyGroup Condition="'$(Compare)' == 'true'">
Expand All @@ -94,7 +127,7 @@
<PreCommands Condition="'$(Compare)' == 'true'">$(WorkItemCommand) --bdn-artifacts $(BaselineArtifactsDirectory) --bdn-arguments="--anyCategories $(BDNCategories) $(ExtraBenchmarkDotNetArguments) $(BaselineCoreRunArgument) --partition-count $(PartitionCount) --partition-index %(HelixWorkItem.Index)"</PreCommands>
<Command>$(WorkItemCommand) --bdn-artifacts $(ArtifactsDirectory) --bdn-arguments="--anyCategories $(BDNCategories) $(ExtraBenchmarkDotNetArguments) $(CoreRunArgument) --partition-count $(PartitionCount) --partition-index %(HelixWorkItem.Index)"</Command>
<PostCommands Condition="'$(Compare)' == 'true'">$(DotnetExe) run -f $(_Framework) -p $(ResultsComparer) --base $(BaselineArtifactsDirectory) --diff $(ArtifactsDirectory) --threshold 2$(Percent) --xml $(XMLResults);$(FinalCommand)</PostCommands>
<Timeout>4:00</Timeout>
<Timeout>$(WorkItemTimeout)</Timeout>
</HelixWorkItem>
</ItemGroup>

Expand Down Expand Up @@ -152,6 +185,11 @@
<PayloadDirectory>$(WorkItemDirectory)\ScenarioCorrelation</PayloadDirectory>
<Command>$(Python) %HELIX_CORRELATION_PAYLOAD%\performance\src\scenarios\crossgen2\test.py crossgen2 --single System.Private.CoreLib.dll --core-root %HELIX_CORRELATION_PAYLOAD%\Core_Root</Command>
</HelixWorkItem>
<HelixWorkItem Include="Crossgen2 Composite Framework R2R">
<PayloadDirectory>$(WorkItemDirectory)\ScenarioCorrelation</PayloadDirectory>
<Command>$(Python) %HELIX_CORRELATION_PAYLOAD%\performance\src\scenarios\crossgen2\test.py crossgen2 --composite %HELIX_CORRELATION_PAYLOAD%\performance\src\scenarios\crossgen2\framework-r2r.dll.rsp --core-root %HELIX_CORRELATION_PAYLOAD%\Core_Root</Command>
<Timeout>1:00</Timeout>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why 1 hour timeout? This leads to very long hangs blocking a machine for long periods of time.

I also see the other workitems we send have 4 hours timeouts. Can we modify this for the purpose of PR runs?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For crossgen2 composite workitem, 1 hour timeout also takes care of CI runs where we run 5 iterations, while we only have 1 iteration for PR runs and each iteration takes ~8min. For the rest of the workitems we will be able to shorten the timeout once the partition is enabled. @DrewScoggins Do you have an estimate of the timeout value after partition?

Copy link
Member

@safern safern Jun 19, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I opened: #38136 to follow up.

The reason why this concerns me is that if we enter a deadlock like: #38138 the machine is blocked for that period of time.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the micro benchmarks we should be able to set the timeout to about 10 minutes once we get this partition change in. A run take about 3-5 minutes, but I want to leave some wiggle room. These are of course Helix timeouts.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, Helix timeouts are the ones we care about reducing on PRs for these micro benchmarks. 10-15 mins sounds good to me.

</HelixWorkItem>

</ItemGroup>
</Project>