Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🎨 Healtcheck diagnostics sensor is now optional #6327

Merged

Conversation

pcrespov
Copy link
Member

@pcrespov pcrespov commented Sep 9, 2024

What do these changes do?

Logs from production revealed that the webserver was unnecessarily restarting due to delays in the Directorv2 service. These restarts did not resolve the issue and instead caused system degradation, leading to lost requests.

Based on an analysis of the production logs, we've determined that the health check based on diagnostics is no longer effective in addressing the problem. Therefore, this pull request introduces a new environment variable, DIAGNOSTICS_HEALTHCHECK_ENABLED, which allows toggling this health check functionality. By default, this variable is set to False, effectively disabling the health check unless explicitly enabled.

Related issue/s

How to test

make devenv
cd services/web/server
make install-dev
pytest -vv tests/unit/isolated/test_diagnostics_healthcheck.py

Dev-ops checklist ⚠️

@pcrespov pcrespov self-assigned this Sep 9, 2024
@pcrespov pcrespov added the t:maintenance Some planned maintenance work label Sep 9, 2024
@pcrespov pcrespov added this to the Eisbock milestone Sep 9, 2024
Copy link

codecov bot commented Sep 9, 2024

Codecov Report

Attention: Patch coverage is 93.75000% with 1 line in your changes missing coverage. Please review.

Project coverage is 88.2%. Comparing base (cafbf96) to head (35c8f63).
Report is 516 commits behind head on master.

Files with missing lines Patch % Lines
...core_service_webserver/diagnostics/_healthcheck.py 83.3% 1 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff            @@
##           master   #6327      +/-   ##
=========================================
+ Coverage    84.5%   88.2%    +3.6%     
=========================================
  Files          10    1499    +1489     
  Lines         214   62005   +61791     
  Branches       25    2065    +2040     
=========================================
+ Hits          181   54725   +54544     
- Misses         23    6964    +6941     
- Partials       10     316     +306     
Flag Coverage Δ
integrationtests 64.6% <56.2%> (?)
unittests 86.2% <93.7%> (+1.6%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
...rc/simcore_service_webserver/diagnostics/plugin.py 100.0% <100.0%> (ø)
.../simcore_service_webserver/diagnostics/settings.py 100.0% <100.0%> (ø)
...er/src/simcore_service_webserver/rest/_handlers.py 100.0% <100.0%> (ø)
.../src/simcore_service_webserver/rest/healthcheck.py 94.5% <100.0%> (ø)
...core_service_webserver/diagnostics/_healthcheck.py 94.0% <83.3%> (ø)

... and 1443 files with indirect coverage changes

@pcrespov pcrespov changed the title Mai/optional healtcheck sensor 🎨 Healtcheck diagnostics sensor is now optional Sep 9, 2024
@pcrespov pcrespov marked this pull request as ready for review September 9, 2024 08:30
Copy link
Member

@sanderegg sanderegg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

very nice thanks!

Copy link

sonarqubecloud bot commented Sep 9, 2024

@pcrespov pcrespov merged commit b74bae4 into ITISFoundation:master Sep 9, 2024
57 checks passed
@pcrespov pcrespov deleted the mai/optional-healtcheck-sensor branch September 9, 2024 12:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
t:maintenance Some planned maintenance work
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Webserver: remove/review automatic unhealthyness for delayed calls
3 participants