Executors keep running if scheduler is shutdown

[frankscholten]

No description provided.

[mwl]

By shutdown you mean gracefully shutdown in a situation where you want to get rid of the whole ElasticSearch cluster?

[philwinder]

Correct. Like when you issue a destroy command from marathon. The scheduler is killed gracefully, but it doesn't pass that message on to the executors.

[frankscholten]

Asked on the service-dev channel

[philwinder]

I can confirm that DCOS uninstall package DOES shut down the executors correctly.

[philwinder]

This is how they shut down the executors:

This is the code that does it in the CLI: https://github.com/mesosphere/dcos-cli/blob/master/dcos/mesos.py#L194
I haven’t tested this but that maps to curl -X POST http://<dcos-url>/master/shutdown -d "frameworkId=<framework-id>" with curl.
Also note that /master/shutdown has been renamed to /master/teardown in recent versions of Mesos.

[philwinder]

Ok, confirmed by Mesosphere.
To kill the executors and framework state in Mesos, call /master/shutdown.
To kill the scheduler, destroy the app in Marathon.

The DCOS CLI does both of these things, so from a user perspective, it appears to cleanly shutdown the framework.

Unfortunately, since this is the responsibility of the entity that starts the framework (e.g. the user or the DCOS CLI) this is out of our control.

I recommend that we close this issue as a wontfix. Starting and stopping is largely a Mesos/Marathon issue. If it doesn't do this in a way that the user likes, that is a Mesos/Marathon problem, not an ES problem.