ERROR (MainThread) [homeassistant.components.websocket_api.http.connection]

[Misak]

The problem

I am experiencing a issue, where the web interface continuously displays the message "Loading data" without fully loading the dashboard. This problem is not limited to a specific device, as multiple devices on the network are affected. The issue occurs both when accessing Home Assistant through an nginx redirect and directly via IP and port.

In shared log, the KeyError is related to the "shell_command" configuration, but it has randomly changed to other components, such as "shopping_list, input_number, climate..."

Restarting the Home Assistant service temporarily resolves the issue, although it sometimes requires multiple restarts. However, the problem resurfaces as soon as I refresh the web page, causing it to get stuck on the perpetual "Loading data" state.

Below is debug error, that appear when I try to access from any device.

I have no idea what I could try so any help would be much appreciated.

What version of Home Assistant Core has the issue?

2023.6.3

What was the last working version of Home Assistant Core?

unknown

What type of installation are you running?

Home Assistant OS

Integration causing the issue

unknown

Link to integration documentation on our website

No response

Diagnostics information

No response

Example YAML snippet

No response

Anything in the logs that might be useful for us?

2023-06-29 08:29:04.507 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [139825427603792] Error handling message: Unknown error (unknown_error) User from 10.101.0.254 (Mozilla/5.0 (Linux; Android 12; M2007J3SG Build/SKQ1.211006.001; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/114.0.5735.130 Mobile Safari/537.36 Home Assistant/2023.6.0-10170 (Android 12; M2007J3SG))
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/websocket_api/decorators.py", line 26, in _handle_async_response
    await func(hass, connection, msg)
  File "/usr/src/homeassistant/homeassistant/components/websocket_api/commands.py", line 382, in handle_get_services
    payload = await _async_get_all_descriptions_json(hass)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/websocket_api/commands.py", line 363, in _async_get_all_descriptions_json
    descriptions = await async_get_all_descriptions(hass)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 614, in async_get_all_descriptions
    domain_yaml = loaded[domain]
                  ~~~~~~^^^^^^^^
KeyError: 'shell_command'
2023-06-29 08:29:04.510 DEBUG (MainThread) [homeassistant.components.websocket_api.http.connection] [139825427603792] Sending {"id":9,"type":"result","success":false,"error":{"code":"unknown_error","message":"Unknown error"}}
2023-06-29 08:29:04.560 DEBUG (MainThread) [zeroconf] IPv6 scope_id 0 associated to the receiving interface
2023-06-29 08:29:04.560 DEBUG (MainThread) [zeroconf] Ignoring duplicate message with no unicast questions received from '::ffff:10.0.11.170':5353 [socket 14 (('::', 5353, 0, 0))] (169 ytes) as [b'\x00\x00\x00\x00\x00\x03\x00\x00\x00\x03\x00\x00\x07Android\x05local\x00\x00\xff\x00\x01\xc0\x0c\x00\xff\x00\x01,{"nm":"Mi 10T Pro","as":"[8194]","ip":"170"}\x0b_mi-connect\x04_udp\xc0\x14\x00\xff\x00\x01\xc0\x0c\x00\x01\x00\x01\x00\x00\x00x\x00\x04\n\x00\x0b\xaa\xc0\x0c\x00\x1c\x00\x01\x00\x00\x00x\x00\x10\xfe\x80\x00\x00\x00\x00\x00\x00\xcc\x8a9\x8d\xcf\xd7\xb1m\xc0%\x00!\x00\x01\x00\x00\x00x\x00\x08\x00\x00\x00\x00\xddZ\xc0\x0c']
2023-06-29 08:29:04.560 DEBUG (MainThread) [zeroconf] IPv6 scope_id 2 associated to the receiving interface
2023-06-29 08:29:04.561 DEBUG (MainThread) [zeroconf] Ignoring duplicate message with no unicast questions received from 'fe80::cc8a:398d:cfd7:b16d':5353 [socket 14 (('::', 5353, 0, 0))] (169 bytes) as [b'\x00\x00\x00\x00\x00\x03\x00\x00\x00\x03\x00\x00\x07Android\x05local\x00\x00\xff\x00\x01\xc0\x0c\x00\xff\x00\x01,{"nm":"Mi 10T Pro","as":"[8194]","ip":"170"}\x0b_mi-connect\x04_udp\xc0\x14\x00\xff\x00\x01\xc0\x0c\x00\x01\x00\x01\x00\x00\x00x\x00\x04\n\x00\x0b\xaa\xc0\x0c\x00\x1c\x00\x01\x00\x00\x00x\x00\x10\xfe\x80\x00\x00\x00\x00\x00\x00\xcc\x8a9\x8d\xcf\xd7\xb1m\xc0%\x00!\x00\x01\x00\x00\x00x\x00\x08\x00\x00\x00\x00\xddZ\xc0\x0c']
2023-06-29 08:29:04.641 DEBUG (MainThread) [httpcore] http11.send_request_headers.started request=<Request [b'GET']>
2023-06-29 08:29:04.642 DEBUG (MainThread) [httpcore] http11.send_request_headers.complete
2023-06-29 08:29:04.642 DEBUG (MainThread) [httpcore] http11.send_request_body.started request=<Request [b'GET']>
2023-06-29 08:29:04.642 DEBUG (MainThread) [httpcore] http11.send_request_body.complete
2023-06-29 08:29:04.642 DEBUG (MainThread) [httpcore] http11.receive_response_headers.started request=<Request [b'GET']>
2023-06-29 08:29:04.646 DEBUG (MainThread) [httpcore] http11.receive_response_headers.complete return_value=(b'HTTP/1.1', 200, b'OK', [(b'Cache-Control', b'public, max-age=2678400'), (b'Content-Type', b'image/jpeg'), (b'Etag', b'"176d0a446b8e47b8-4086"'), (b'Last-Modified', b'Thu, 29 Jun 2023 05:32:05 GMT'), (b'Content-Length', b'16518'), (b'Accept-Ranges', b'bytes'), (b'Date', b'Thu, 29 Jun 2023 06:29:04 GMT'), (b'Server', b'Python/3.11 aiohttp/3.8.4')])
2023-06-29 08:29:04.647 DEBUG (MainThread) [httpcore] http11.receive_response_body.started request=<Request [b'GET']>
2023-06-29 08:29:04.647 DEBUG (MainThread) [httpcore] http11.receive_response_body.complete
2023-06-29 08:29:04.647 DEBUG (MainThread) [httpcore] http11.response_closed.started
2023-06-29 08:29:04.649 DEBUG (MainThread) [httpcore] http11.response_closed.complete

Additional information

No response

[Wesley-Vos]

This probably is due to an external service requesting service descriptions over the API. One service which can do that is the remote home-assistant integration, do you use that?

I have the same problem, as soon as my remote instance of HA sends a request to my 'main' HA for the service descriptions, the API call generates an error and afterwards, HA freezes.

This issue is (I think) due to this change #93131. This PR is merged in 2023.6.x which does also align with my observation that everything works fine on versions < 2023.6.0. @bdraco can you confirm if this is related to the change you implemented?

[Misak]

Yes, I use remote home-assistant integration, and I have two hass instances connected together. I can try downgrading to a version earlier than 2023.6.0, but I'm fairly certain that's around the time when the problems started. Do you have any suggestions on what we can do to resolve this?

[Wesley-Vos]

I added the PR in which it changed and notified the owner of the PR if this could be related. Hope that he might have a thought. I will try to debug the issue as soon as I have time and come with code changes to fix it but can't guarantee that that will be in the upcoming days. I really want to upgrade to 2023.7 because of the new interesting features but this bug avoids that and that is a pity...

[bdraco]

Traveling today but will take a look at this one when I can

@Wesley-Vos
Please post the backtrace you see.

[bdraco]

Looking at the code, the only way I can see this happening is if fetching the integration returned an exception.

We currently don't log that.

We could refactor this to log the exception and exclude that integration from the load

The real question is why would the integration be failing to load if there are services for it registered

[bdraco]

shell_command is a special case since it has a yaml file but its empty because it defines the services dynamiclly

#95911 will take care of that case as well as the integrations failing to load case.

I can't replicate the issue from the UI, but I could see how it could happen if you were calling the apis manually.