HPCC-27029 Start/stop ELK by default via docker scripts #15718

rpastrana · 2022-02-01T15:57:51Z

Starts Elastic Stack instance by default via startall.sh
Stops Elastic Stack instance by default via stopall.sh
Performs dependancy check on 3rd party charts on start
Documents legacy option related to ELK

Signed-off-by: Rodrigo Pastrana [email protected]

Type of change:

This change is a bug fix (non-breaking change which fixes an issue).
This change is a new feature (non-breaking change which adds functionality).
This change improves the code (refactor or other change that does not change the functionality)
This change fixes warnings (the fix does not alter the functionality or the generated code)
This change is a breaking change (fix or feature that will cause existing behavior to change).
This change alters the query API (existing queries will have to be recompiled)

Checklist:

Smoketest:

Send notifications about my Pull Request position in Smoketest queue.
Test my draft Pull Request.

Testing:

github-actions · 2022-02-01T15:58:22Z

https://track.hpccsystems.com/browse/HPCC-27029
Jira updated

rpastrana · 2022-02-01T16:31:33Z

@richardkchapman this version will perform a helm dependency check on the elastic4hpcclogs charts on every start.
Which does have a (negligible?) added overhead.
The other option I considered was keeping the -c option in place for elastic4hpcclogs.
The users would be explicitly prompted to use the -c if the dependencies are absent, and would only need to do it once.

richardkchapman · 2022-02-02T16:08:41Z

dockerfiles/startall.sh

@@ -87,7 +88,7 @@ while [ "$#" -gt 0 ]; do
         echo "    -c                 Update chart dependencies"
         echo "    -p <location>      Use local persistent data"
         echo "    -pv <yamlfile>     Override dataplane definitions for local persistent data"
-         echo "    -e                 Deploy light-weight Elastic Stack for component log processing"
+         echo "    -e                 Ignored - light-weight Elastic Stack deployed by default"


Would it be sensible to repurpose -e to suppress the deployment of elk?

It would definitely be useful, I was concerned about overwriting the option w/ the complete opposite behavior, but I doubt anybody would be affected by that change. I'll repurpose to suppress.

Same question on the uninstall?

The stopall attempts to uninstall the elastic cluster regardless, no need to include the option.
That's based on precedent behavior encouraged by Jake/Gavin in other PRs related to stopall.

richardkchapman · 2022-02-03T09:54:51Z

I got the following warnings (errors) when I ran startall without params:

Deploying myelastic4hpcclogs - light-weight Elastic Stack:
NAME VERSION REPOSITORY STATUS
filebeat 7.16.3 https://helm.elastic.co wrong version
elasticsearch 7.16.3 https://helm.elastic.co wrong version
kibana 7.16.3 https://helm.elastic.co wrong version

W0203 09:53:25.825499 93370 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
W0203 09:53:25.831727 93370 warnings.go:70] rbac.authorization.k8s.io/v1beta1 ClusterRole is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 ClusterRole
W0203 09:53:25.833380 93370 warnings.go:70] rbac.authorization.k8s.io/v1beta1 ClusterRoleBinding is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 ClusterRoleBinding
W0203 09:53:25.863538 93370 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
W0203 09:53:25.876382 93370 warnings.go:70] rbac.authorization.k8s.io/v1beta1 ClusterRole is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 ClusterRole
W0203 09:53:25.880616 93370 warnings.go:70] rbac.authorization.k8s.io/v1beta1 ClusterRoleBinding is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 ClusterRoleBinding

Issue reported to elastic: elastic/helm-charts#1568

richardkchapman · 2022-02-03T10:03:46Z

System started ok, was able to connect a browser to kibana at localhost:5601. But no idea what to do with it - should we prepopulate a standard dashboard, have it already pointed at our elasticsearch?

richardkchapman · 2022-02-03T10:09:53Z

I'm not convinced that the default behaviour of "leave the pvc after running stopall.sh" is desirable. Default behaviour of startall/stopall in all other respects is "start from clean, and leave no trace".

rpastrana · 2022-02-03T13:59:14Z

I'm not convinced that the default behaviour of "leave the pvc after running stopall.sh" is desirable. Default behaviour of startall/stopall in all other respects is "start from clean, and leave no trace".

Fair enough... we can add logic to ensure the Elastic PVC is deleted upon stopall...
I should be able to track, or at least deduce the Elastic PVC name, if we feel removing all PVCs is too intrusive.
Have we considered assigning a namespace in the start-all?

rpastrana · 2022-02-03T14:12:40Z

filebeat 7.16.3 https://helm.elastic.co wrong version
elasticsearch 7.16.3 https://helm.elastic.co wrong version
kibana 7.16.3 https://helm.elastic.co wrong version

The "wrong version" status means Helm found the target charts locally, but they were apparently outdated (not surprised since we recently bumped the ELK charts). We hard-coded a dependency update in startall and it should have pulled the latest charts, Out of curiosity does helm list report the following?
$ helm list
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
...
myelastic4hpcclogs default 1 2022-02-03 09:08:56.5748994 -0500 EST deployed elastic4hpcclogs-1.2.2 7.16.3

The k8s warnings have to be addressed, I've reported a few of them on the Elastic bug tracker but your output shows some I haven't seen before. Do you mind sharing the Helm/k8s/docker version you're on? I'll use that info to report issue.

Pre-existing issue: elastic/helm-charts#1055

rpastrana · 2022-02-03T15:27:15Z

System started ok, was able to connect a browser to kibana at localhost:5601. But no idea what to do with it - should we prepopulate a standard dashboard, have it already pointed at our elasticsearch?

Absolutely, I'm targeting the discover view with the hpcc log centric fields laid out by default as the defaultroute in Kibana

https://track.hpccsystems.com/browse/HPCC-27127

richardkchapman · 2022-02-03T16:54:02Z

We could (optionally) use a namespace, but I suspect a lot of developers have got used to there not being one and may have scripts that would fail. Would be better to deduce the pvc name I think

richardkchapman · 2022-02-04T17:43:37Z

dockerfiles/startall.sh

-  if [[ -z "${DEP_UPDATE_ARG}" ]]; then
-    dependency_check "managed/logging/elastic"
-  fi
+  dependency_check "managed/logging/elastic"


As discussed offline, this change is wrong and should be reverted

rpastrana · 2022-02-04T20:23:40Z

@richardkchapman I've reversed the dep-check change.
As mentioned offline, it appears the helm install --dependency-update doesn't work exactly like the helm dependency update command.
It seems the former doesn't update outdated charts:

Helm dependency update doc:
https://helm.sh/docs/helm/helm_dependency_update/#synopsis

Helm install options doc:
https://helm.sh/docs/helm/helm_install/#options

https://track.hpccsystems.com/browse/HPCC-27135

richardkchapman · 2022-02-08T16:40:12Z

@rpastrana Looks reasonable - can you squash

- Starts Elastic Stack instance by default via startall.sh - Stops Elastic Stack instance by default via stopall.sh - Reverses -e option to supress deployment of elk - Adds logic to delete elastic PVC/PV - Defines option to suppress deletion of elastic4hpcclogs Signed-off-by: Rodrigo Pastrana <[email protected]>

rpastrana · 2022-02-08T16:42:53Z

@richardkchapman squashed

HPCCSmoketest · 2022-02-08T17:06:39Z

Automated Smoketest: ✅
OS: centos 7.6.1810 (Linux 3.10.0-957.1.3.el7.x86_64)
Host: ip-10-20-0-168.ca-central-1.compute.internal
GCC: gcc (GCC) 7.3.1 20180303 (Red Hat 7.3.1-5)
Git: 2.9.5, CMake: 3.22.1, cUrl: 7.67.0, node.js: v16.13.1, npm: 8.1.2
Sha: e20e64c
Containerized:True
Build: success
Milestone:Install hpccsystems-platform-community_8.6.0-closedown0.el7.x86_64.rpm
Time stats:

Prep time	Build time	Package time	Install time	Start time	Test time	Stop time	Summary
14 sec (00:00:14)	400 sec (00:06:40)	104 sec (00:01:44)	21 sec (00:00:21)	0 sec (00:00:00)	0 sec (00:00:00)	0 sec (00:00:00)	539 sec (00:08:59)

richardkchapman requested changes Feb 2, 2022

View reviewed changes

rpastrana force-pushed the HPCC-27029-start-elk-bydefault branch 2 times, most recently from 418aaa8 to cf49b40 Compare February 2, 2022 17:51

richardkchapman requested changes Feb 4, 2022

View reviewed changes

rpastrana force-pushed the HPCC-27029-start-elk-bydefault branch 2 times, most recently from c62eab7 to 6d237ea Compare February 4, 2022 20:18

rpastrana requested a review from richardkchapman February 7, 2022 15:17

richardkchapman approved these changes Feb 8, 2022

View reviewed changes

rpastrana force-pushed the HPCC-27029-start-elk-bydefault branch from 5fbbe1a to e20e64c Compare February 8, 2022 16:42

richardkchapman merged commit 172fb9c into hpcc-systems:candidate-8.6.x Feb 9, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HPCC-27029 Start/stop ELK by default via docker scripts #15718

HPCC-27029 Start/stop ELK by default via docker scripts #15718

rpastrana commented Feb 1, 2022 •

edited

Loading

github-actions bot commented Feb 1, 2022

rpastrana commented Feb 1, 2022

richardkchapman Feb 2, 2022

rpastrana Feb 2, 2022

richardkchapman Feb 2, 2022

rpastrana Feb 2, 2022

richardkchapman commented Feb 3, 2022 •

edited by rpastrana

Loading

richardkchapman commented Feb 3, 2022

richardkchapman commented Feb 3, 2022

rpastrana commented Feb 3, 2022

rpastrana commented Feb 3, 2022 •

edited

Loading

rpastrana commented Feb 3, 2022 •

edited

Loading

richardkchapman commented Feb 3, 2022

richardkchapman Feb 4, 2022

rpastrana commented Feb 4, 2022 •

edited

Loading

richardkchapman commented Feb 8, 2022

rpastrana commented Feb 8, 2022

HPCCSmoketest commented Feb 8, 2022

HPCC-27029 Start/stop ELK by default via docker scripts #15718

HPCC-27029 Start/stop ELK by default via docker scripts #15718

Conversation

rpastrana commented Feb 1, 2022 • edited Loading

Type of change:

Checklist:

Smoketest:

Testing:

github-actions bot commented Feb 1, 2022

rpastrana commented Feb 1, 2022

richardkchapman Feb 2, 2022

Choose a reason for hiding this comment

rpastrana Feb 2, 2022

Choose a reason for hiding this comment

richardkchapman Feb 2, 2022

Choose a reason for hiding this comment

rpastrana Feb 2, 2022

Choose a reason for hiding this comment

richardkchapman commented Feb 3, 2022 • edited by rpastrana Loading

richardkchapman commented Feb 3, 2022

richardkchapman commented Feb 3, 2022

rpastrana commented Feb 3, 2022

rpastrana commented Feb 3, 2022 • edited Loading

rpastrana commented Feb 3, 2022 • edited Loading

richardkchapman commented Feb 3, 2022

richardkchapman Feb 4, 2022

Choose a reason for hiding this comment

rpastrana commented Feb 4, 2022 • edited Loading

richardkchapman commented Feb 8, 2022

rpastrana commented Feb 8, 2022

HPCCSmoketest commented Feb 8, 2022

rpastrana commented Feb 1, 2022 •

edited

Loading

richardkchapman commented Feb 3, 2022 •

edited by rpastrana

Loading

rpastrana commented Feb 3, 2022 •

edited

Loading

rpastrana commented Feb 3, 2022 •

edited

Loading

rpastrana commented Feb 4, 2022 •

edited

Loading