Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Gary/impl llm obs api #8390

Draft
wants to merge 2 commits into
base: gary/add-llm-obs-api
Choose a base branch
from
Draft

Gary/impl llm obs api #8390

wants to merge 2 commits into from

Conversation

gary-huang
Copy link
Contributor

What Does This Do

Motivation

Additional Notes

Contributor Checklist

Jira ticket: [PROJ-IDENT]

@gary-huang gary-huang force-pushed the gary/impl-llm-obs-api branch from 3bcbe81 to c437b9c Compare February 19, 2025 14:47
@pr-commenter
Copy link

pr-commenter bot commented Feb 19, 2025 •
edited
Loading

Benchmarks

Startup

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
end_time 2025-02-20T23:22:53 2025-02-20T23:29:55
git_branch master gary/impl-llm-obs-api
git_commit_date 1740074363 1740093193
git_commit_sha 121f38f 7064f57
release_version 1.47.0-SNAPSHOT~121f38f42d 1.47.0-SNAPSHOT~7064f57e6a
start_time 2025-02-20T23:22:39 2025-02-20T23:29:41
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1740094552 1740094552
ci_job_id 817071417 817071417
ci_pipeline_id 56506246 56506246
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-nnfmazrx-project-304-concurrent-0-wj8eavmf 6.8.0-1021-aws #23~22.04.1-Ubuntu SMP Tue Dec 10 16:50:46 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux Linux runner-nnfmazrx-project-304-concurrent-0-wj8eavmf 6.8.0-1021-aws #23~22.04.1-Ubuntu SMP Tue Dec 10 16:50:46 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 16 unstable metrics.

Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.47.0-SNAPSHOT~7064f57e6a, baseline=1.47.0-SNAPSHOT~121f38f42d
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.327 ms) : 1306, 1347
.   : milestone, 1327,
appsec (1.724 ms) : 1700, 1747
.   : milestone, 1724,
appsec_no_iast (1.754 ms) : 1731, 1776
.   : milestone, 1754,
iast (1.513 ms) : 1489, 1537
.   : milestone, 1513,
profiling (1.532 ms) : 1506, 1557
.   : milestone, 1532,
tracing (1.478 ms) : 1453, 1503
.   : milestone, 1478,
section candidate
no_agent (1.341 ms) : 1322, 1360
.   : milestone, 1341,
appsec (1.738 ms) : 1714, 1762
.   : milestone, 1738,
appsec_no_iast (1.755 ms) : 1733, 1778
.   : milestone, 1755,
iast (1.499 ms) : 1475, 1522
.   : milestone, 1499,
profiling (1.555 ms) : 1531, 1579
.   : milestone, 1555,
tracing (1.485 ms) : 1460, 1509
.   : milestone, 1485,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.327 ms [1.306 ms, 1.347 ms] -
appsec 1.724 ms [1.7 ms, 1.747 ms] 397.017 µs (29.9%)
appsec_no_iast 1.754 ms [1.731 ms, 1.776 ms] 427.216 µs (32.2%)
iast 1.513 ms [1.489 ms, 1.537 ms] 186.341 µs (14.0%)
profiling 1.532 ms [1.506 ms, 1.557 ms] 205.066 µs (15.5%)
tracing 1.478 ms [1.453 ms, 1.503 ms] 151.777 µs (11.4%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.341 ms [1.322 ms, 1.36 ms] -
appsec 1.738 ms [1.714 ms, 1.762 ms] 396.433 µs (29.6%)
appsec_no_iast 1.755 ms [1.733 ms, 1.778 ms] 414.281 µs (30.9%)
iast 1.499 ms [1.475 ms, 1.522 ms] 157.507 µs (11.7%)
profiling 1.555 ms [1.531 ms, 1.579 ms] 213.821 µs (15.9%)
tracing 1.485 ms [1.46 ms, 1.509 ms] 143.425 µs (10.7%)
Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.47.0-SNAPSHOT~7064f57e6a, baseline=1.47.0-SNAPSHOT~121f38f42d
    dateFormat X
    axisFormat %s
section baseline
no_agent (383.14 µs) : 363, 403
.   : milestone, 383,
iast (506.06 µs) : 484, 528
.   : milestone, 506,
iast_FULL (723.372 µs) : 701, 745
.   : milestone, 723,
iast_GLOBAL (563.745 µs) : 541, 586
.   : milestone, 564,
iast_HARDCODED_SECRET_DISABLED (505.197 µs) : 484, 527
.   : milestone, 505,
iast_INACTIVE (462.288 µs) : 441, 484
.   : milestone, 462,
iast_TELEMETRY_OFF (496.657 µs) : 474, 520
.   : milestone, 497,
tracing (456.845 µs) : 436, 478
.   : milestone, 457,
section candidate
no_agent (378.052 µs) : 356, 400
.   : milestone, 378,
iast (508.909 µs) : 487, 531
.   : milestone, 509,
iast_FULL (727.968 µs) : 706, 750
.   : milestone, 728,
iast_GLOBAL (560.204 µs) : 538, 582
.   : milestone, 560,
iast_HARDCODED_SECRET_DISABLED (507.373 µs) : 485, 529
.   : milestone, 507,
iast_INACTIVE (460.994 µs) : 440, 482
.   : milestone, 461,
iast_TELEMETRY_OFF (496.723 µs) : 473, 520
.   : milestone, 497,
tracing (453.638 µs) : 433, 474
.   : milestone, 454,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 383.14 µs [363.239 µs, 403.041 µs] -
iast 506.06 µs [484.216 µs, 527.904 µs] 122.92 µs (32.1%)
iast_FULL 723.372 µs [701.404 µs, 745.341 µs] 340.233 µs (88.8%)
iast_GLOBAL 563.745 µs [541.146 µs, 586.344 µs] 180.605 µs (47.1%)
iast_HARDCODED_SECRET_DISABLED 505.197 µs [483.743 µs, 526.652 µs] 122.057 µs (31.9%)
iast_INACTIVE 462.288 µs [440.67 µs, 483.905 µs] 79.148 µs (20.7%)
iast_TELEMETRY_OFF 496.657 µs [473.535 µs, 519.78 µs] 113.518 µs (29.6%)
tracing 456.845 µs [435.919 µs, 477.771 µs] 73.705 µs (19.2%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 378.052 µs [356.025 µs, 400.078 µs] -
iast 508.909 µs [487.008 µs, 530.81 µs] 130.857 µs (34.6%)
iast_FULL 727.968 µs [705.995 µs, 749.941 µs] 349.916 µs (92.6%)
iast_GLOBAL 560.204 µs [538.151 µs, 582.257 µs] 182.152 µs (48.2%)
iast_HARDCODED_SECRET_DISABLED 507.373 µs [485.286 µs, 529.461 µs] 129.322 µs (34.2%)
iast_INACTIVE 460.994 µs [439.73 µs, 482.257 µs] 82.942 µs (21.9%)
iast_TELEMETRY_OFF 496.723 µs [473.385 µs, 520.061 µs] 118.671 µs (31.4%)
tracing 453.638 µs [432.929 µs, 474.347 µs] 75.586 µs (20.0%)

Dacapo

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master gary/impl-llm-obs-api
git_commit_date 1740074363 1740093193
git_commit_sha 121f38f 7064f57
release_version 1.47.0-SNAPSHOT~121f38f42d 1.47.0-SNAPSHOT~7064f57e6a
See matching parameters
Baseline Candidate
application biojava biojava
ci_job_date 1740095243 1740095243
ci_job_id 817071418 817071418
ci_pipeline_id 56506246 56506246
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-a8u8q-x-project-304-concurrent-1-xodznnf0 6.8.0-1021-aws #23~22.04.1-Ubuntu SMP Tue Dec 10 16:50:46 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux Linux runner-a8u8q-x-project-304-concurrent-1-xodznnf0 6.8.0-1021-aws #23~22.04.1-Ubuntu SMP Tue Dec 10 16:50:46 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
variant appsec appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics.

Execution time for tomcat 
gantt
    title tomcat - execution time [CI 0.99] : candidate=1.47.0-SNAPSHOT~7064f57e6a, baseline=1.47.0-SNAPSHOT~121f38f42d
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.467 ms) : 1456, 1479
.   : milestone, 1467,
appsec (2.353 ms) : 2309, 2396
.   : milestone, 2353,
iast (2.114 ms) : 2059, 2169
.   : milestone, 2114,
iast_GLOBAL (2.148 ms) : 2093, 2204
.   : milestone, 2148,
profiling (1.982 ms) : 1937, 2027
.   : milestone, 1982,
tracing (1.938 ms) : 1896, 1981
.   : milestone, 1938,
section candidate
no_agent (1.462 ms) : 1451, 1473
.   : milestone, 1462,
appsec (2.357 ms) : 2314, 2401
.   : milestone, 2357,
iast (2.093 ms) : 2038, 2148
.   : milestone, 2093,
iast_GLOBAL (2.145 ms) : 2090, 2201
.   : milestone, 2145,
profiling (1.95 ms) : 1907, 1993
.   : milestone, 1950,
tracing (1.935 ms) : 1893, 1977
.   : milestone, 1935,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.467 ms [1.456 ms, 1.479 ms] -
appsec 2.353 ms [2.309 ms, 2.396 ms] 885.748 µs (60.4%)
iast 2.114 ms [2.059 ms, 2.169 ms] 646.468 µs (44.1%)
iast_GLOBAL 2.148 ms [2.093 ms, 2.204 ms] 681.233 µs (46.4%)
profiling 1.982 ms [1.937 ms, 2.027 ms] 514.948 µs (35.1%)
tracing 1.938 ms [1.896 ms, 1.981 ms] 471.073 µs (32.1%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.462 ms [1.451 ms, 1.473 ms] -
appsec 2.357 ms [2.314 ms, 2.401 ms] 895.049 µs (61.2%)
iast 2.093 ms [2.038 ms, 2.148 ms] 630.92 µs (43.1%)
iast_GLOBAL 2.145 ms [2.09 ms, 2.201 ms] 682.937 µs (46.7%)
profiling 1.95 ms [1.907 ms, 1.993 ms] 487.606 µs (33.3%)
tracing 1.935 ms [1.893 ms, 1.977 ms] 472.9 µs (32.3%)
Execution time for biojava 
gantt
    title biojava - execution time [CI 0.99] : candidate=1.47.0-SNAPSHOT~7064f57e6a, baseline=1.47.0-SNAPSHOT~121f38f42d
    dateFormat X
    axisFormat %s
section baseline
no_agent (14.706 s) : 14706000, 14706000
.   : milestone, 14706000,
appsec (15.004 s) : 15004000, 15004000
.   : milestone, 15004000,
iast (18.77 s) : 18770000, 18770000
.   : milestone, 18770000,
iast_GLOBAL (18.074 s) : 18074000, 18074000
.   : milestone, 18074000,
profiling (15.623 s) : 15623000, 15623000
.   : milestone, 15623000,
tracing (14.921 s) : 14921000, 14921000
.   : milestone, 14921000,
section candidate
no_agent (15.16 s) : 15160000, 15160000
.   : milestone, 15160000,
appsec (15.044 s) : 15044000, 15044000
.   : milestone, 15044000,
iast (19.188 s) : 19188000, 19188000
.   : milestone, 19188000,
iast_GLOBAL (17.788 s) : 17788000, 17788000
.   : milestone, 17788000,
profiling (15.278 s) : 15278000, 15278000
.   : milestone, 15278000,
tracing (14.761 s) : 14761000, 14761000
.   : milestone, 14761000,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 14.706 s [14.706 s, 14.706 s] -
appsec 15.004 s [15.004 s, 15.004 s] 298.0 ms (2.0%)
iast 18.77 s [18.77 s, 18.77 s] 4.064 s (27.6%)
iast_GLOBAL 18.074 s [18.074 s, 18.074 s] 3.368 s (22.9%)
profiling 15.623 s [15.623 s, 15.623 s] 917.0 ms (6.2%)
tracing 14.921 s [14.921 s, 14.921 s] 215.0 ms (1.5%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.16 s [15.16 s, 15.16 s] -
appsec 15.044 s [15.044 s, 15.044 s] -116.0 ms (-0.8%)
iast 19.188 s [19.188 s, 19.188 s] 4.028 s (26.6%)
iast_GLOBAL 17.788 s [17.788 s, 17.788 s] 2.628 s (17.3%)
profiling 15.278 s [15.278 s, 15.278 s] 118.0 ms (0.8%)
tracing 14.761 s [14.761 s, 14.761 s] -399.0 ms (-2.6%)

@gary-huang gary-huang force-pushed the gary/impl-llm-obs-api branch 2 times, most recently from 8247b74 to 871d358 Compare February 20, 2025 06:56
@gary-huang gary-huang force-pushed the gary/impl-llm-obs-api branch from 871d358 to 180ccf5 Compare February 20, 2025 21:17
@github-actions GitHub Actions
Copy link
Contributor

Hi! 👋 Looks like you updated a Git Submodule.
If this was not intentional please make sure to:

@gary-huang gary-huang changed the base branch from master to gary/add-llm-obs-api February 20, 2025 21:18
@gary-huang gary-huang force-pushed the gary/impl-llm-obs-api branch 4 times, most recently from 944020d to 94136c5 Compare February 20, 2025 22:47
@gary-huang gary-huang force-pushed the gary/impl-llm-obs-api branch from 94136c5 to 7064f57 Compare February 20, 2025 23:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@gary-huang