Add RecallObservations for retrieval of prompt extensions

[enyst]

• This change is worth documenting at https://docs.all-hands.dev/
• Include this change in the Release Notes. If checked, you must provide an end-user friendly description for your change below

Make microagents available in the event stream as recall observations (accessible from the UI, LLM)

---

• define and create RecallObservations, maybe RecallActions
  • final attributes
• memory component, on_event, initialization
• conversation_memory - move context/prompt manager; handle RecallObs, event->message
  • RecallObs should be before the user message in the messages to the LLM?
• fix agent controller flow
• agent config options

Other expected outcomes:

• no dependency from Runtime or Memory to agent or prompt manager (ref comment)
  • Runtime prompt manager
  • Runtime plugins
  • Memory
• no access from agent to the stream, direct or indirect

---

Cc: @xingyaoww
I'd love your opinion about this idea. It's basically an alternative to 6526, I was curious to see roughly how it looks like, so we can maybe see if it makes sense.

This PR proposes to refactor prompt extensions into recalled observations:

• take the retrieval of information out of the PromptManager, into a Memory component
• PromptManager remains responsible with loading templates and rendering them (just a 'view manager')
• Memory subscribes to the stream
  • on user messages, it may retrieve extensions and add them to the stream as RecallObservations
  • on the first user message, it may retrieve repo and runtime info
  • on recall actions (not yet in use, source=agent), it may retrieve e.g. library docs
• some logic is separated from the agent (like 6526)
• since the information is in the stream, session restore / refresh / runtime reconnect / etc will not lose it.

Not ready for review, just some food for discussion.

Cc: @csmith49

---

Link of any specific issues this addresses
Fix #6535

[csmith49]

This looks clean so far. I'm a fan of moving what we can to the event stream: much better visibility than modifying messages in-place, and I think leaning into the pub-sub approach is a good way to add extra functionality without having to tip-toe around the agent/controller/system control flow.

[xingyaoww]

I like this idea better than what I did in #6526! Happy to close that PR in favor of this one

[enyst]

This is starting to work:

{
  "messages": [
    {
      "content": [
        {
          "type": "text",
          "text": "You are OpenHands agent, a helpful AI assistant that can interact with a computer to solve tasks.\n<IMPORTANT>\n* If user provides a path, you should NOT assume it's relative to the current working directory. Instead, you should explore the file system to find the file before working on it.\n* When configuring git credentials, use \"openhands\" as the user.name and \"[email protected]\" as the user.email by default, unless explicitly instructed otherwise.\n* You MUST NOT include comments in the code unless they are necessary to describe non-obvious behavior.\n* If the user asks you to edit a file, you should edit the file directly, do NOT create a new file with the updated content unless the user explicitly instructs you to do so.\n* When you are doing global search-and-replace, consider using `sed` instead of running file editor multiple times.\n* Only use GITHUB_TOKEN and other credentials in ways that the user has asked for and would expect. Do NOT make potentially dangerous changes (e.g. pushing to main, deleting a repository) unless explicitly asked to do so.\n* Use APIs to work with GitHub or other platforms, unless the user asks otherwise or your task requires browsing.\n* If you've made repeated attempts to solve a problem, but the tests won't pass or the user says it's still broken, reflect on 5-7 different possible sources of the problem. Assess the likelihood of these options, and proceed with fixing the most likely one.\n</IMPORTANT>",
          "cache_control": {
            "type": "ephemeral"
          }
        }
      ],
      "role": "system"
    },
    {
      "content": [
        {
          "type": "text",
          "text": "implement hello world in py"
        }
      ],
      "role": "user"
    },
    {
      "content": [
        {
          "type": "text",
          "text": "<REPOSITORY_INFO>\nAt the user's request, repository enyst/playground has been cloned to directory playground.\n</REPOSITORY_INFO>\n\n<REPOSITORY_INSTRUCTIONS>\nThis repository contains the code for OpenHands, an automated AI software engineer. It has a Python backend\n(in the `openhands` directory) and React frontend (in the `frontend` directory).\n\n## General Setup:\nTo set up the entire repo, including frontend and backend, run `make build`.\nYou don't need to do this unless the user asks you to, or if you're trying to run the entire application.\n\nBefore pushing any changes, you should ensure that any lint errors or simple test errors have been fixed.\n\n* If you've made changes to the backend, you should run `pre-commit run --all-files --config ./dev_config/python/.pre-commit-config.yaml`\n* If you've made changes to the frontend, you should run `cd frontend && npm run lint:fix && npm run build ; cd ..`\n\nIf either command fails, it may have automatically fixed some issues. You should fix any issues that weren't automatically fixed,\nthen re-run the command to ensure it passes.\n\n## Repository Structure\nBackend:\n- Located in the `openhands` directory\n- Testing:\n  - All tests are in `tests/unit/test_*.py`\n  - To test new code, run `poetry run pytest tests/unit/test_xxx.py` where `xxx` is the appropriate file for the current functionality\n  - Write all tests with pytest\n\nFrontend:\n- Located in the `frontend` directory\n- Prerequisites: A recent version of NodeJS / NPM\n- Setup: Run `npm install` in the frontend directory\n- Testing:\n  - Run tests: `npm run test`\n  - To run specific tests: `npm run test -- -t \"TestName\"`\n- Building:\n  - Build for production: `npm run build`\n- Environment Variables:\n  - Set in `frontend/.env` or as environment variables\n  - Available variables: VITE_BACKEND_HOST, VITE_USE_TLS, VITE_INSECURE_SKIP_VERIFY, VITE_FRONTEND_PORT\n- Internationalization:\n  - Generate i18n declaration file: `npm run make-i18n`\n\n## IMPORTANT: Microagents\n\nTo understand the public microagents available in this repository, please read [microagents/README.md](microagents/README.md). These microagents are shared across all OpenHands users and provide reusable knowledge and common workflows.\n\nThe repo-specific microagents are in .openhands/microagents and they are \"private\" to this repo. To understand the tasks we have to perform now, read the contents of the private microagents whose names include 'task'. You can use the `str_replace_editor` tool with the `view` command and the absolute path to the file to view their content.\n</REPOSITORY_INSTRUCTIONS>\n\n<RUNTIME_INFORMATION>\nThe user has access to the following hosts for accessing a web application,\neach of which has a corresponding port:\n* http://localhost:53628 (port 53628)\n* http://localhost:55216 (port 55216)\n\nWhen starting a web server, use the corresponding ports. You should also\nset any options to allow iframes and CORS requests, and allow the server to\nbe accessed from any host (e.g. 0.0.0.0).\n</RUNTIME_INFORMATION>"
        }
      ],
      "role": "user"
    },
    {
      "content": [
        {
          "type": "text",
          "text": "I'll help you implement a \"Hello World\" program in Python. Let's first explore the repository structure to understand where we should create this file."
        }
      ],
      "role": "assistant",