Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add Exception Replay smoke test #8388

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Add Exception Replay smoke test #8388

wants to merge 1 commit into from
+134 −34

Conversation

jpbempel
Copy link
Member

@jpbempel jpbempel commented Feb 14, 2025 •
edited by jira bot
Loading

What Does This Do

refactor smoke test to be able to add JVM options/properties
add smoke test for max captured frames

Motivation

Additional Notes

Contributor Checklist

Jira ticket: DEBUG-3390

@jpbempel jpbempel added comp: testing Testing tag: no release notes Changes to exclude from release notes comp: debugger Dynamic Instrumentation type:task labels Feb 14, 2025
@jpbempel jpbempel requested a review from a team as a code owner February 14, 2025 10:16
@jpbempel jpbempel requested review from ojung and removed request for a team February 14, 2025 10:16
@pr-commenter
Copy link

pr-commenter bot commented Feb 14, 2025 •
edited
Loading

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master jpbempel/er-smoke-tests
git_commit_date 1739537145 1739547762
git_commit_sha acf173a 63a2db7
release_version 1.47.0-SNAPSHOT~acf173a318 1.47.0-SNAPSHOT~63a2db7594
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1739550426 1739550426
ci_job_id 809092373 809092373
ci_pipeline_id 55950972 55950972
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-1ny-1exg-project-304-concurrent-0-6v49yndm 6.8.0-1021-aws #23~22.04.1-Ubuntu SMP Tue Dec 10 16:50:46 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux Linux runner-1ny-1exg-project-304-concurrent-0-6v49yndm 6.8.0-1021-aws #23~22.04.1-Ubuntu SMP Tue Dec 10 16:50:46 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
module Agent Agent
parent None None
variant iast iast

Summary

Found 0 performance improvements and 1 performance regressions! Performance is the same for 57 metrics, 5 unstable metrics.

scenario Δ mean execution_time candidate mean execution_time baseline mean execution_time
scenario:startup:insecure-bank:tracing:Remote Config worse
[+18.140µs; +49.178µs] or [+2.633%; +7.139%]		 722.540µs 688.881µs
Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.47.0-SNAPSHOT~63a2db7594, baseline=1.47.0-SNAPSHOT~acf173a318

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.044 s) : 0, 1043842
Total [baseline] (8.66 s) : 0, 8659926
Agent [candidate] (1.055 s) : 0, 1054691
Total [candidate] (8.694 s) : 0, 8693834
section iast
Agent [baseline] (1.17 s) : 0, 1169604
Total [baseline] (9.229 s) : 0, 9229010
Agent [candidate] (1.179 s) : 0, 1179445
Total [candidate] (9.244 s) : 0, 9243706
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.17 s) : 0, 1170161
Total [baseline] (9.223 s) : 0, 9223411
Agent [candidate] (1.178 s) : 0, 1177639
Total [candidate] (9.239 s) : 0, 9239217
section iast_TELEMETRY_OFF
Agent [baseline] (1.168 s) : 0, 1167896
Total [baseline] (9.217 s) : 0, 9217319
Agent [candidate] (1.183 s) : 0, 1182801
Total [candidate] (9.255 s) : 0, 9254528
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.044 s -
Agent iast 1.17 s 125.763 ms (12.0%)
Agent iast_HARDCODED_SECRET_DISABLED 1.17 s 126.319 ms (12.1%)
Agent iast_TELEMETRY_OFF 1.168 s 124.055 ms (11.9%)
Total tracing 8.66 s -
Total iast 9.229 s 569.085 ms (6.6%)
Total iast_HARDCODED_SECRET_DISABLED 9.223 s 563.486 ms (6.5%)
Total iast_TELEMETRY_OFF 9.217 s 557.393 ms (6.4%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.055 s -
Agent iast 1.179 s 124.754 ms (11.8%)
Agent iast_HARDCODED_SECRET_DISABLED 1.178 s 122.948 ms (11.7%)
Agent iast_TELEMETRY_OFF 1.183 s 128.111 ms (12.1%)
Total tracing 8.694 s -
Total iast 9.244 s 549.872 ms (6.3%)
Total iast_HARDCODED_SECRET_DISABLED 9.239 s 545.383 ms (6.3%)
Total iast_TELEMETRY_OFF 9.255 s 560.694 ms (6.4%) 
gantt
    title insecure-bank - break down per module: candidate=1.47.0-SNAPSHOT~63a2db7594, baseline=1.47.0-SNAPSHOT~acf173a318

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (719.255 ms) : 0, 719255
BytebuddyAgent [candidate] (726.762 ms) : 0, 726762
GlobalTracer [baseline] (240.437 ms) : 0, 240437
GlobalTracer [candidate] (242.483 ms) : 0, 242483
AppSec [baseline] (55.378 ms) : 0, 55378
AppSec [candidate] (55.719 ms) : 0, 55719
Remote Config [baseline] (688.881 µs) : 0, 689
Remote Config [candidate] (722.54 µs) : 0, 723
Telemetry [baseline] (12.913 ms) : 0, 12913
Telemetry [candidate] (13.681 ms) : 0, 13681
section iast
BytebuddyAgent [baseline] (835.379 ms) : 0, 835379
BytebuddyAgent [candidate] (842.288 ms) : 0, 842288
GlobalTracer [baseline] (230.126 ms) : 0, 230126
GlobalTracer [candidate] (232.547 ms) : 0, 232547
IAST [baseline] (22.688 ms) : 0, 22688
IAST [candidate] (22.867 ms) : 0, 22867
AppSec [baseline] (57.042 ms) : 0, 57042
AppSec [candidate] (57.077 ms) : 0, 57077
Remote Config [baseline] (624.864 µs) : 0, 625
Remote Config [candidate] (607.147 µs) : 0, 607
Telemetry [baseline] (8.619 ms) : 0, 8619
Telemetry [candidate] (8.652 ms) : 0, 8652
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (835.135 ms) : 0, 835135
BytebuddyAgent [candidate] (841.045 ms) : 0, 841045
GlobalTracer [baseline] (230.334 ms) : 0, 230334
GlobalTracer [candidate] (231.978 ms) : 0, 231978
IAST [baseline] (22.751 ms) : 0, 22751
IAST [candidate] (22.859 ms) : 0, 22859
AppSec [baseline] (57.401 ms) : 0, 57401
AppSec [candidate] (56.991 ms) : 0, 56991
Remote Config [baseline] (616.45 µs) : 0, 616
Remote Config [candidate] (612.907 µs) : 0, 613
Telemetry [baseline] (8.688 ms) : 0, 8688
Telemetry [candidate] (8.745 ms) : 0, 8745
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (833.527 ms) : 0, 833527
BytebuddyAgent [candidate] (845.051 ms) : 0, 845051
GlobalTracer [baseline] (230.549 ms) : 0, 230549
GlobalTracer [candidate] (233.247 ms) : 0, 233247
IAST [baseline] (23.775 ms) : 0, 23775
IAST [candidate] (27.423 ms) : 0, 27423
AppSec [baseline] (55.654 ms) : 0, 55654
AppSec [candidate] (52.375 ms) : 0, 52375
Remote Config [baseline] (611.522 µs) : 0, 612
Remote Config [candidate] (607.768 µs) : 0, 608
Telemetry [baseline] (8.592 ms) : 0, 8592
Telemetry [candidate] (8.653 ms) : 0, 8653
Loading
Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.47.0-SNAPSHOT~63a2db7594, baseline=1.47.0-SNAPSHOT~acf173a318

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.042 s) : 0, 1042385
Total [baseline] (10.448 s) : 0, 10447737
Agent [candidate] (1.04 s) : 0, 1040165
Total [candidate] (10.451 s) : 0, 10450544
section appsec
Agent [baseline] (1.19 s) : 0, 1190075
Total [baseline] (10.759 s) : 0, 10759438
Agent [candidate] (1.19 s) : 0, 1189663
Total [candidate] (10.799 s) : 0, 10798646
section iast
Agent [baseline] (1.169 s) : 0, 1168634
Total [baseline] (10.988 s) : 0, 10987531
Agent [candidate] (1.183 s) : 0, 1182761
Total [candidate] (11.063 s) : 0, 11062874
section profiling
Agent [baseline] (1.261 s) : 0, 1260580
Total [baseline] (10.889 s) : 0, 10888670
Agent [candidate] (1.272 s) : 0, 1272480
Total [candidate] (10.951 s) : 0, 10950739
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.042 s -
Agent appsec 1.19 s 147.69 ms (14.2%)
Agent iast 1.169 s 126.249 ms (12.1%)
Agent profiling 1.261 s 218.195 ms (20.9%)
Total tracing 10.448 s -
Total appsec 10.759 s 311.701 ms (3.0%)
Total iast 10.988 s 539.793 ms (5.2%)
Total profiling 10.889 s 440.933 ms (4.2%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.04 s -
Agent appsec 1.19 s 149.499 ms (14.4%)
Agent iast 1.183 s 142.596 ms (13.7%)
Agent profiling 1.272 s 232.315 ms (22.3%)
Total tracing 10.451 s -
Total appsec 10.799 s 348.102 ms (3.3%)
Total iast 11.063 s 612.33 ms (5.9%)
Total profiling 10.951 s 500.195 ms (4.8%) 
gantt
    title petclinic - break down per module: candidate=1.47.0-SNAPSHOT~63a2db7594, baseline=1.47.0-SNAPSHOT~acf173a318

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (718.653 ms) : 0, 718653
BytebuddyAgent [candidate] (717.784 ms) : 0, 717784
GlobalTracer [baseline] (239.927 ms) : 0, 239927
GlobalTracer [candidate] (241.109 ms) : 0, 241109
AppSec [baseline] (55.671 ms) : 0, 55671
AppSec [candidate] (55.46 ms) : 0, 55460
Remote Config [baseline] (696.773 µs) : 0, 697
Remote Config [candidate] (717.512 µs) : 0, 718
Telemetry [baseline] (12.249 ms) : 0, 12249
Telemetry [candidate] (9.947 ms) : 0, 9947
section appsec
BytebuddyAgent [baseline] (739.42 ms) : 0, 739420
BytebuddyAgent [candidate] (739.175 ms) : 0, 739175
GlobalTracer [baseline] (237.969 ms) : 0, 237969
GlobalTracer [candidate] (238.606 ms) : 0, 238606
IAST [baseline] (21.467 ms) : 0, 21467
IAST [candidate] (21.697 ms) : 0, 21697
AppSec [baseline] (177.679 ms) : 0, 177679
AppSec [candidate] (176.844 ms) : 0, 176844
Remote Config [baseline] (668.09 µs) : 0, 668
Remote Config [candidate] (657.396 µs) : 0, 657
Telemetry [baseline] (8.266 ms) : 0, 8266
Telemetry [candidate] (8.289 ms) : 0, 8289
section iast
BytebuddyAgent [baseline] (834.283 ms) : 0, 834283
BytebuddyAgent [candidate] (844.109 ms) : 0, 844109
GlobalTracer [baseline] (229.855 ms) : 0, 229855
GlobalTracer [candidate] (232.743 ms) : 0, 232743
IAST [baseline] (22.931 ms) : 0, 22931
IAST [candidate] (23.28 ms) : 0, 23280
AppSec [baseline] (57.092 ms) : 0, 57092
AppSec [candidate] (57.9 ms) : 0, 57900
Remote Config [baseline] (626.937 µs) : 0, 627
Remote Config [candidate] (631.42 µs) : 0, 631
Telemetry [baseline] (8.643 ms) : 0, 8643
Telemetry [candidate] (8.77 ms) : 0, 8770
section profiling
BytebuddyAgent [baseline] (707.81 ms) : 0, 707810
BytebuddyAgent [candidate] (713.176 ms) : 0, 713176
GlobalTracer [baseline] (349.863 ms) : 0, 349863
GlobalTracer [candidate] (355.649 ms) : 0, 355649
AppSec [baseline] (55.324 ms) : 0, 55324
AppSec [candidate] (54.848 ms) : 0, 54848
Remote Config [baseline] (672.832 µs) : 0, 673
Remote Config [candidate] (705.395 µs) : 0, 705
Telemetry [baseline] (8.875 ms) : 0, 8875
Telemetry [candidate] (8.963 ms) : 0, 8963
ProfilingAgent [baseline] (95.753 ms) : 0, 95753
ProfilingAgent [candidate] (96.603 ms) : 0, 96603
Profiling [baseline] (95.777 ms) : 0, 95777
Profiling [candidate] (96.628 ms) : 0, 96628
Loading

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
end_time 2025-02-14T15:56:59 2025-02-14T16:04:03
git_branch master jpbempel/er-smoke-tests
git_commit_date 1739537145 1739547762
git_commit_sha acf173a 63a2db7
release_version 1.47.0-SNAPSHOT~acf173a318 1.47.0-SNAPSHOT~63a2db7594
start_time 2025-02-14T15:56:45 2025-02-14T16:03:49
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1739549402 1739549402
ci_job_id 809092374 809092374
ci_pipeline_id 55950972 55950972
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-1ny-1exg-project-304-concurrent-1-vz586cqu 6.8.0-1021-aws #23~22.04.1-Ubuntu SMP Tue Dec 10 16:50:46 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux Linux runner-1ny-1exg-project-304-concurrent-1-vz586cqu 6.8.0-1021-aws #23~22.04.1-Ubuntu SMP Tue Dec 10 16:50:46 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 17 unstable metrics.

Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.47.0-SNAPSHOT~63a2db7594, baseline=1.47.0-SNAPSHOT~acf173a318
    dateFormat X
    axisFormat %s
section baseline
no_agent (383.631 µs) : 364, 404
.   : milestone, 384,
iast (506.434 µs) : 485, 528
.   : milestone, 506,
iast_FULL (744.191 µs) : 722, 766
.   : milestone, 744,
iast_GLOBAL (562.693 µs) : 539, 586
.   : milestone, 563,
iast_HARDCODED_SECRET_DISABLED (506.192 µs) : 484, 528
.   : milestone, 506,
iast_INACTIVE (454.696 µs) : 434, 476
.   : milestone, 455,
iast_TELEMETRY_OFF (497.232 µs) : 474, 520
.   : milestone, 497,
tracing (458.406 µs) : 436, 480
.   : milestone, 458,
section candidate
no_agent (377.434 µs) : 357, 398
.   : milestone, 377,
iast (503.224 µs) : 482, 525
.   : milestone, 503,
iast_FULL (739.238 µs) : 718, 761
.   : milestone, 739,
iast_GLOBAL (560.499 µs) : 537, 584
.   : milestone, 560,
iast_HARDCODED_SECRET_DISABLED (511.808 µs) : 490, 534
.   : milestone, 512,
iast_INACTIVE (464.511 µs) : 444, 485
.   : milestone, 465,
iast_TELEMETRY_OFF (499.49 µs) : 476, 523
.   : milestone, 499,
tracing (449.051 µs) : 428, 470
.   : milestone, 449,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 383.631 µs [363.76 µs, 403.501 µs] -
iast 506.434 µs [484.757 µs, 528.11 µs] 122.803 µs (32.0%)
iast_FULL 744.191 µs [722.335 µs, 766.047 µs] 360.56 µs (94.0%)
iast_GLOBAL 562.693 µs [538.983 µs, 586.403 µs] 179.062 µs (46.7%)
iast_HARDCODED_SECRET_DISABLED 506.192 µs [484.034 µs, 528.35 µs] 122.561 µs (31.9%)
iast_INACTIVE 454.696 µs [433.827 µs, 475.565 µs] 71.065 µs (18.5%)
iast_TELEMETRY_OFF 497.232 µs [474.131 µs, 520.333 µs] 113.601 µs (29.6%)
tracing 458.406 µs [436.329 µs, 480.483 µs] 74.775 µs (19.5%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 377.434 µs [356.83 µs, 398.039 µs] -
iast 503.224 µs [481.756 µs, 524.692 µs] 125.79 µs (33.3%)
iast_FULL 739.238 µs [717.601 µs, 760.875 µs] 361.803 µs (95.9%)
iast_GLOBAL 560.499 µs [537.277 µs, 583.72 µs] 183.064 µs (48.5%)
iast_HARDCODED_SECRET_DISABLED 511.808 µs [489.957 µs, 533.659 µs] 134.374 µs (35.6%)
iast_INACTIVE 464.511 µs [443.643 µs, 485.378 µs] 87.076 µs (23.1%)
iast_TELEMETRY_OFF 499.49 µs [476.307 µs, 522.673 µs] 122.055 µs (32.3%)
tracing 449.051 µs [428.367 µs, 469.735 µs] 71.617 µs (19.0%)
Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.47.0-SNAPSHOT~63a2db7594, baseline=1.47.0-SNAPSHOT~acf173a318
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.354 ms) : 1334, 1374
.   : milestone, 1354,
appsec (1.746 ms) : 1722, 1769
.   : milestone, 1746,
appsec_no_iast (1.761 ms) : 1738, 1784
.   : milestone, 1761,
iast (1.506 ms) : 1482, 1529
.   : milestone, 1506,
profiling (1.521 ms) : 1496, 1546
.   : milestone, 1521,
tracing (1.481 ms) : 1456, 1506
.   : milestone, 1481,
section candidate
no_agent (1.362 ms) : 1342, 1382
.   : milestone, 1362,
appsec (1.76 ms) : 1736, 1784
.   : milestone, 1760,
appsec_no_iast (1.761 ms) : 1737, 1785
.   : milestone, 1761,
iast (1.51 ms) : 1485, 1535
.   : milestone, 1510,
profiling (1.52 ms) : 1496, 1543
.   : milestone, 1520,
tracing (1.496 ms) : 1471, 1521
.   : milestone, 1496,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.354 ms [1.334 ms, 1.374 ms] -
appsec 1.746 ms [1.722 ms, 1.769 ms] 391.864 µs (28.9%)
appsec_no_iast 1.761 ms [1.738 ms, 1.784 ms] 407.219 µs (30.1%)
iast 1.506 ms [1.482 ms, 1.529 ms] 151.738 µs (11.2%)
profiling 1.521 ms [1.496 ms, 1.546 ms] 166.953 µs (12.3%)
tracing 1.481 ms [1.456 ms, 1.506 ms] 127.03 µs (9.4%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.362 ms [1.342 ms, 1.382 ms] -
appsec 1.76 ms [1.736 ms, 1.784 ms] 397.418 µs (29.2%)
appsec_no_iast 1.761 ms [1.737 ms, 1.785 ms] 399.022 µs (29.3%)
iast 1.51 ms [1.485 ms, 1.535 ms] 147.804 µs (10.9%)
profiling 1.52 ms [1.496 ms, 1.543 ms] 157.364 µs (11.6%)
tracing 1.496 ms [1.471 ms, 1.521 ms] 133.521 µs (9.8%)

Dacapo

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master jpbempel/er-smoke-tests
git_commit_date 1739537145 1739547762
git_commit_sha acf173a 63a2db7
release_version 1.47.0-SNAPSHOT~acf173a318 1.47.0-SNAPSHOT~63a2db7594
See matching parameters
Baseline Candidate
application biojava biojava
ci_job_date 1739549931 1739549931
ci_job_id 809092375 809092375
ci_pipeline_id 55950972 55950972
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-hzy35tlz-project-304-concurrent-0-bcrz7mlu 6.8.0-1021-aws #23~22.04.1-Ubuntu SMP Tue Dec 10 16:50:46 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux Linux runner-hzy35tlz-project-304-concurrent-0-bcrz7mlu 6.8.0-1021-aws #23~22.04.1-Ubuntu SMP Tue Dec 10 16:50:46 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
variant appsec appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics.

Execution time for biojava 
gantt
    title biojava - execution time [CI 0.99] : candidate=1.47.0-SNAPSHOT~63a2db7594, baseline=1.47.0-SNAPSHOT~acf173a318
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.592 s) : 15592000, 15592000
.   : milestone, 15592000,
appsec (15.025 s) : 15025000, 15025000
.   : milestone, 15025000,
iast (18.995 s) : 18995000, 18995000
.   : milestone, 18995000,
iast_GLOBAL (18.03 s) : 18030000, 18030000
.   : milestone, 18030000,
profiling (15.143 s) : 15143000, 15143000
.   : milestone, 15143000,
tracing (14.932 s) : 14932000, 14932000
.   : milestone, 14932000,
section candidate
no_agent (15.474 s) : 15474000, 15474000
.   : milestone, 15474000,
appsec (15.014 s) : 15014000, 15014000
.   : milestone, 15014000,
iast (18.669 s) : 18669000, 18669000
.   : milestone, 18669000,
iast_GLOBAL (18.231 s) : 18231000, 18231000
.   : milestone, 18231000,
profiling (14.915 s) : 14915000, 14915000
.   : milestone, 14915000,
tracing (14.813 s) : 14813000, 14813000
.   : milestone, 14813000,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.592 s [15.592 s, 15.592 s] -
appsec 15.025 s [15.025 s, 15.025 s] -567.0 ms (-3.6%)
iast 18.995 s [18.995 s, 18.995 s] 3.403 s (21.8%)
iast_GLOBAL 18.03 s [18.03 s, 18.03 s] 2.438 s (15.6%)
profiling 15.143 s [15.143 s, 15.143 s] -449.0 ms (-2.9%)
tracing 14.932 s [14.932 s, 14.932 s] -660.0 ms (-4.2%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.474 s [15.474 s, 15.474 s] -
appsec 15.014 s [15.014 s, 15.014 s] -460.0 ms (-3.0%)
iast 18.669 s [18.669 s, 18.669 s] 3.195 s (20.6%)
iast_GLOBAL 18.231 s [18.231 s, 18.231 s] 2.757 s (17.8%)
profiling 14.915 s [14.915 s, 14.915 s] -559.0 ms (-3.6%)
tracing 14.813 s [14.813 s, 14.813 s] -661.0 ms (-4.3%)
Execution time for tomcat 
gantt
    title tomcat - execution time [CI 0.99] : candidate=1.47.0-SNAPSHOT~63a2db7594, baseline=1.47.0-SNAPSHOT~acf173a318
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.469 ms) : 1458, 1481
.   : milestone, 1469,
appsec (2.353 ms) : 2309, 2396
.   : milestone, 2353,
iast (2.112 ms) : 2057, 2166
.   : milestone, 2112,
iast_GLOBAL (2.163 ms) : 2108, 2218
.   : milestone, 2163,
profiling (1.966 ms) : 1923, 2010
.   : milestone, 1966,
tracing (1.934 ms) : 1892, 1976
.   : milestone, 1934,
section candidate
no_agent (1.472 ms) : 1460, 1483
.   : milestone, 1472,
appsec (2.359 ms) : 2315, 2402
.   : milestone, 2359,
iast (2.097 ms) : 2043, 2152
.   : milestone, 2097,
iast_GLOBAL (2.151 ms) : 2096, 2207
.   : milestone, 2151,
profiling (1.963 ms) : 1919, 2007
.   : milestone, 1963,
tracing (1.933 ms) : 1891, 1975
.   : milestone, 1933,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.469 ms [1.458 ms, 1.481 ms] -
appsec 2.353 ms [2.309 ms, 2.396 ms] 883.147 µs (60.1%)
iast 2.112 ms [2.057 ms, 2.166 ms] 642.155 µs (43.7%)
iast_GLOBAL 2.163 ms [2.108 ms, 2.218 ms] 693.211 µs (47.2%)
profiling 1.966 ms [1.923 ms, 2.01 ms] 496.876 µs (33.8%)
tracing 1.934 ms [1.892 ms, 1.976 ms] 464.812 µs (31.6%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.472 ms [1.46 ms, 1.483 ms] -
appsec 2.359 ms [2.315 ms, 2.402 ms] 887.025 µs (60.3%)
iast 2.097 ms [2.043 ms, 2.152 ms] 625.739 µs (42.5%)
iast_GLOBAL 2.151 ms [2.096 ms, 2.207 ms] 679.687 µs (46.2%)
profiling 1.963 ms [1.919 ms, 2.007 ms] 491.346 µs (33.4%)
tracing 1.933 ms [1.891 ms, 1.975 ms] 461.276 µs (31.3%)

@jpbempel jpbempel force-pushed the jpbempel/er-smoke-tests branch from 481c167 to 0abcd89 Compare February 14, 2025 12:23
@jpbempel
Add Exception Replay smoke test
63a2db7 
refactor smoke test to be able to add JVM options/properties
ad smoke test for max captured frames
@jpbempel jpbempel force-pushed the jpbempel/er-smoke-tests branch from 0abcd89 to 63a2db7 Compare February 14, 2025 15:43
@jpbempel jpbempel requested a review from evanchooly February 14, 2025 16:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ojung ojung Awaiting requested review from ojung ojung is a code owner automatically assigned from DataDog/debugger-java

@evanchooly evanchooly Awaiting requested review from evanchooly

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
comp: debugger Dynamic Instrumentation comp: testing Testing tag: no release notes Changes to exclude from release notes type:task
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@jpbempel