Fix Health Checks section in installation readme

docs/installation.md

@@ -3,11 +3,11 @@
 Trino Gateway is distributed as an executable JAR file. The [release
 notes](release-notes.md) contain links to download specific versions.
 
-Every Trino Gateway release includes a [Docker container](docker.md) and a 
+Every Trino Gateway release includes a [Docker container](docker.md) and a
 [Helm chart](installation.md#helm) as alternative deployment methods.
 
 Follow the [development instructions](development.md) to
-build the JAR file and the Docker image instructions  or use the 
+build the JAR file and the Docker image instructions  or use the
 `TrinoGatewayRunner` class for local testing.
 The [quickstart guide](quickstart.md) contains instructions for running the
 application locally.
@@ -34,7 +34,7 @@ performed with Linux and MacOS.
 ### Processor architecture
 
 No specific processor architecture is required, as long as a suitable Java
-distribution is installed.  
+distribution is installed.
 
 ### Backend database
 
@@ -64,11 +64,11 @@ of individual clusters.
 
 ### Trino configuration
 
-From a users perspective Trino Gateway acts as a transparent proxy for one 
-or more Trino clusters. The following Trino configuration tips should be 
+From a users perspective Trino Gateway acts as a transparent proxy for one
+or more Trino clusters. The following Trino configuration tips should be
 taken into account for all clusters behind the Trino Gateway.
 
-If all client and server communication is routed through Trino Gateway, 
+If all client and server communication is routed through Trino Gateway,
 then process forwarded HTTP headers must be enabled:
 
 ```properties
@@ -79,13 +79,13 @@ Without this setting, first requests go from the user to Trino Gateway and then
 to Trino correctly. However, the URL for subsequent next URIs for more results
 in a query provided by Trino is then using the local URL of the Trino cluster,
 and not the URL of the Trino Gateway. This circumvents the Trino Gateway for all
-these requests. In scenarios, where the local URL of the Trino cluster is private 
+these requests. In scenarios, where the local URL of the Trino cluster is private
 to the Trino cluster on the network level, these following calls do not work
 at all for users.
 
-This setting is also required for Trino to authenticate in the case TLS is 
-terminated at the Trino Gateway. Normally it refuses to authenticate plain HTTP 
-requests, but if `http-server.process-forwarded=true` it authenticates over 
+This setting is also required for Trino to authenticate in the case TLS is
+terminated at the Trino Gateway. Normally it refuses to authenticate plain HTTP
+requests, but if `http-server.process-forwarded=true` it authenticates over
 HTTP if the request includes `X-Forwarded-Proto: HTTPS`.
 
 To prevent Trino Gateway from sending `X-Forwarded-*` headers, add the following configuration:
@@ -117,7 +117,7 @@ You can manually set an environment variable on the command line.
 export DB_PASSWORD=my-super-secret-pwd
 ```
 
-To use this variable in the configuration file, you reference it with the 
+To use this variable in the configuration file, you reference it with the
 syntax `${ENV:VARIABLE}`. For example:
 
 ```yaml
@@ -133,10 +133,10 @@ Find more information in the [routing rules documentation](routing-rules.md).
 
 ### Configure logging <a name="logging">
 
-To configure the logging level for various classes, specify the path to the 
+To configure the logging level for various classes, specify the path to the
 `log.properties` file by setting `log.levels-file` in `serverConfig`.
 
-For additional configurations, use the `log.*` properties from the 
+For additional configurations, use the `log.*` properties from the
 [Trino logging properties documentation](https://trino.io/docs/current/admin/properties-logging.html) and specify
 the properties in `serverConfig`.
 
@@ -164,13 +164,13 @@ extraWhitelistPaths:
 
 ### Configure additional v1/statement-like paths
 
-The Trino client protocol specifies that queries are initiated by a POST to `v1/statement`. 
-The Trino Gateway incorporates this into its routing logic by extracting and recording the 
+The Trino client protocol specifies that queries are initiated by a POST to `v1/statement`.
+The Trino Gateway incorporates this into its routing logic by extracting and recording the
 query id from responses to such requests. If you use an experimental or commercial build of
-Trino that supports additional endpoints, you can cause Trino Gateway to treat them 
+Trino that supports additional endpoints, you can cause Trino Gateway to treat them
 equivalently to `/v1/statement` by adding them under the `additionalStatementPaths`
 configuration node. They must be absolute, and no path can be a prefix to any other path.
-The standard `/v1/statement` path is always included and does not need to be configured. 
+The standard `/v1/statement` path is always included and does not need to be configured.
 For example:
 
 ```yaml
@@ -181,9 +181,9 @@ additionalStatementPaths:
 
 ## Configure behind a load balancer
 
-A possible deployment of Trino Gateway is to run multiple instances of Trino 
-Gateway behind another generic load balancer, such as a load balancer from 
-your cloud hosting provider. In this deployment you must configure the 
+A possible deployment of Trino Gateway is to run multiple instances of Trino
+Gateway behind another generic load balancer, such as a load balancer from
+your cloud hosting provider. In this deployment you must configure the
 `serverConfig` to include enabling process forwarded HTTP headers:
 
 ```yaml
@@ -204,10 +204,10 @@ java -XX:MinRAMPercentage=50 -XX:MaxRAMPercentage=80 \
 ### Helm <a name="helm"></a>
 
 Helm manages the deployment of Kubernetes applications by templating Kubernetes
-resources with a set of Helm charts. The Trino Gateway Helm chart is 
+resources with a set of Helm charts. The Trino Gateway Helm chart is
 available in the [Trino Helm chart project](https://github.com/trinodb/charts).
 
-Configure the charts repository as a Helm chart repository with the 
+Configure the charts repository as a Helm chart repository with the
 following command:
 
 ```shell
@@ -216,12 +216,12 @@ helm repo add trino https://trinodb.github.io/charts/
 The Trino Gateway chart consists of the following components:
 
 * A `config` node for general configuration
-* `dataStoreSecret`, `backendStateSecret` and `authenticationSecret` for 
-  providing sensitive configurations through Kubernetes secrets, 
+* `dataStoreSecret`, `backendStateSecret` and `authenticationSecret` for
+  providing sensitive configurations through Kubernetes secrets,
 * Standard Helm options such as `replicaCount`, `resources` and `ingress`.
 
 The default `values.yaml` found in the `helm/trino-gateway` folder includes
-basic configuration options as an example. For a simple deployment, proceed with 
+basic configuration options as an example. For a simple deployment, proceed with
 the following steps:
 
 Create a yaml file containing the configuration for your `datastore`:
@@ -251,18 +251,18 @@ backendStateSecret:
 ```
 
 When a Secret is created with the `--from-file` option, the filename is used as
-the key. Finally, you can deploy Trino Gateway with the chart from the root 
+the key. Finally, you can deploy Trino Gateway with the chart from the root
 of this repository:
 
 ```shell
-helm install tg --values values-override.yaml helm/trino-gateway 
+helm install tg --values values-override.yaml helm/trino-gateway
 ```
 
 Secrets for `authenticationSecret` and `backendState` can be provisioned
-similarly. Alternatively,  you can directly define the `config.backEndState` 
-node in `values-override.yaml` and leave `backendStateSecret` undefined. 
+similarly. Alternatively,  you can directly define the `config.backEndState`
+node in `values-override.yaml` and leave `backendStateSecret` undefined.
 However, a [Secret](https://kubernetes.io/docs/concepts/configuration/secret/)
-is recommended to protect the database credentials required for this 
+is recommended to protect the database credentials required for this
 configuration.
 
 By default, the Trino Gateway process is started with the following command:
@@ -305,22 +305,30 @@ volumeMounts:
 ```
 
 Ensure that the `mountPath` matches the `rulesConfigPath` specified in your
-configuration. Note that the `subPath` is not strictly necessary, and if it 
-is not specified the file is mounted at `mountPath/<configMap key>`. 
+configuration. Note that the `subPath` is not strictly necessary, and if it
+is not specified the file is mounted at `mountPath/<configMap key>`.
 Kubernetes updates the mounted file when the ConfigMap is updated.
 
-Standard Helm options such as `replicaCount`, `image`, `imagePullSecrets`, 
-`service`, `ingress` and `resources` are supported. These are defined in 
-`helm/values.yaml`. 
+Standard Helm options such as `replicaCount`, `image`, `imagePullSecrets`,
+`service`, `ingress` and `resources` are supported. These are defined in
+`helm/values.yaml`.
 
-More detail about the chart are available in the [values 
+More detail about the chart are available in the [values
 reference documentation](https://github.com/trinodb/charts/blob/main/charts/gateway/README.md)
 
 ### Health Checks
 
-Trino Gateway checks the health of each backend and **deactivates it if 
-unhealthy**. A backend that fails a health check must be manually reset to 
-active. Automatic recovery is not supported.
+The Trino Gateway periodically performs health checks and maintains
+an in-memory TrinoStatus for each backend. If a backend fails a health check,
+it is marked as UNHEALTHY, and the Trino Gateway stops routing requests to it.
+
+It is important to distinguish TrinoStatus from the active/inactive
+state of a backend. The active/inactive state indicates whether a backend is
+manually turned on or off, whereas TrinoStatus is programmatically determined
+by the health check process. Health checks are only performed on backends
+that are marked as active.
+
+See [TrinoStatus](routing-rules.md#trinostatus) for more details on what each Trino status means.
 
 The type of health check is configured by setting
 
@@ -334,7 +342,7 @@ to one of the following values.
 #### INFO_API (default)
 
 By default Trino Gateway uses the `v1/info` REST endpoint. A successful check is
-defined as a 200 response with `starting: false`. Connection timeout parameters 
+defined as a 200 response with `starting: false`. Connection timeout parameters
 can be defined through the `monitor` node, for example
 
 ```yaml
@@ -349,11 +357,11 @@ All timeout parameters are optional.
 
 #### JDBC
 
-This uses a JDBC connection to query `system.runtime` tables for cluster 
+This uses a JDBC connection to query `system.runtime` tables for cluster
 information. It is required for the query count based routing strategy. This is
 recommended over `UI_API` since it does not restrict the Web UI authentication
 method of backend clusters. Configure a username and password by adding
-`backendState` to your configuration. The username and password must be valid 
+`backendState` to your configuration. The username and password must be valid
 across all backends.
 
 ```yaml
@@ -374,7 +382,7 @@ Other timeout parameters are not applicable to the JDBC connection.
 #### UI_API
 
 This pulls cluster information from the `ui/api/stats` REST endpoint. This is
-supported for legacy reasons and may be deprecated in the future. It is only 
+supported for legacy reasons and may be deprecated in the future. It is only
 supported for backend clusters with `web-ui.authentication.type=FORM`. Set
 a username and password using `backendState` as with the `JDBC` option.
 

docs/routing-rules.md

@@ -150,7 +150,7 @@ Trino clusters.  The three possible states of these cluster are updated with
 every healthcheck:
 
 - `PENDING`: A Trino cluster shows this state when it is still starting up. It
-  is treated as unhealthy by `RoutingManager`, and therefore requests are
+  is treated as unhealthy by `RoutingManager`, and therefore requests will
   not be routed to these clusters.
 - `HEALTHY`: A Trino cluster shows this state when healthchecks report
   the cluster as healthy and ready. `RoutingManager` only routes requests to