Implement colored output in source code blocks

[CeleritasCelery]

This change reworks the output highlighting logic to only highlight the code inside source blocks. It can also highlight each block with a different language. This solves the issue seen in #21 where syntax highlighting outside the source block was leading to messy output.

This change also adds some basic font locking for things like code fences and SEARCH/REPLACE markers. I couldn't care less what the actual faces used are, so if you prefer something else let me know.

Closes #9

how it works:

most of the code here is to reliably find the start/end of the source blocks. aider-fontify-blocks is run as part of comint output functions, so we only get small chunks of the output at a time. The aider--block-* variables keep track of this state between invocations. We search for the start of new blocks and then highlight them incrementaly with aider--fontify-block.

The major mode is determined by first looking for a language id if there is a code fence. aider-language-name-map is used to map these id's to Emacs Lisp major modes. If that doesn't work, we look at the file being edited and match it against auto-mode-alist.

[CeleritasCelery]

cc @katafrakt to test

[tninja]

Thanks for the change. It must take lot of time. But I am still not sure if it capture output from different LLM. For example, the o3-mini model output format is different from claude-3.5-sonnet.

And I also not sure if we need to manually support all kinds of different language. Personally I don't think I can support them, or ask user to config them. So I personally prefer the current solution to copy over current language major mode way, it use much less code and there is no need to support other language. agree current solution is not perfect, so I didn't close that issue.

Actually, I wish there is a way to be able to support the aider colored output, hopefully it is possible in future when there is better comint mode or term. With the better than dumb term, we don't have to do a lot of work on this. (I know vterm can do that but I still feel it require more time to install, and not sure it will be maintained forever).

[CeleritasCelery]

But I am still not sure if it capture output from different LLM. For example, the o3-mini model output format is different from claude-3.5-sonnet.

I had not thought of that. I will test it with a handful of different models to see what the different output modes are. Right now it supports three different kinds of blocks:

1

<<<<<<< SEARCH
foo
=======
bar
>>>>>>> REPLACE

2

```lang
foo
```

3

```lang
<<<<<<< SEARCH
foo
=======
bar
>>>>>>> REPLACE
```

Personally I don't think I can support them [...] I personally prefer the current solution to copy over current language major mode way

That is understandable. Given our discussion in #65 I recognized that I would probably need to fork this repo to add the features I was looking for. You have different priorities for this project.

Actually, I wish there is a way to be able to support the aider colored output

You can now. Aider recently added a --no-fancy-input option that disables the input handling. That means we don't need to deal with cursor-position requests or resize requests. We only need to handle the output. I actually implemented this first, and it worked. However I didn't like the result for a few reasons:

1. Aider will redraw sections of the output to ensure consistent fontification. This results in "jumpy" output.
2. The colors used are just basic colors picked by pigments, and don't match the user theme or color pallet. This makes it hard to recognize the code at a glance because the user is used to looking at code using their theme.
3. Aider is highlighting everything inside the SEARCH/REPLACE block as one unit. This means multi-line constructs (like strings) can bleed over into the replace block and lead to incorrect highlighting.

Due to those issues, I decided to try the Emacs font-lock way similar to how org-babel and chatgpt-shell work. This resulted in much nicer experience overall that just works out of the box without any configuration.

[MatthewZMD]

And I also not sure if we need to manually support all kinds of different language. Personally I don't think I can support them

I personally think for something like aider that's at a lower end of the LLM toolchain is that, it needs to manually support and adjust to all kinds of different langauges and different LLM models. It's part of the "job". It'll not be easy work, and it shouldn't be your work alone, we as a community can help altogether.

Forking the repo will dilute the already limited resources even further, albeit that's the essence of open-source.

[tninja]

@CeleritasCelery I tried --no-fancy-input option. Aider still think it is a dumb terminal and cannot render color. Let me know if there is a way to let the aider output color in comint buffer, that would be super helpful. If that works, we don't even need any code for this. Less code = Less maintainance effort, and I hope we don't have to maintain a lot of code, to redo the color rendering which has been done by aider.

[CeleritasCelery]

You need to change the terminal setting to something that is not dumb.

[MatthewZMD]

I tried to set the terminal to xterm-256color but the buffer is complaining WARNING: your terminal doesn't support cursor position requests (CPR). and it's making lots of new lines for some reason.

[CeleritasCelery]

@MatthewZMD you need both term != dumb and --no-fancy-input. Also you have to handle the escape sequences ESC[A (move cursor up) and ESC[K (delete line) to deal with the repeated input. That is what I was talking about with aider redrawing input.

[MatthewZMD]

Gotcha. It isn't very ideal, I like your approach better.

[MatthewZMD]

to redo the color rendering which has been done by aider.

FWIW, The colors used are just basic colors picked by pigments, and don't match the user theme or color pallet. This makes it hard to recognize the code at a glance because the user is used to looking at code using their theme.

[MatthewZMD]

The upstream --multiline support feels somewhat like a third-class citizen, as I noted in #51. I prefer @CeleritasCelery’s solution in #65, while @tninja’s #86 effectively solves the issue within its own buffer—though that functions as a separate feature.

I don’t think we should blindly follow every upstream’s approach just for the sake of consistency. Their architectural choices were constrained by a pure CLI-based design. Given that we’re developing Emacs extensions, I don’t see why we should be bound by those same limitations.

[tninja]

@CeleritasCelery @MatthewZMD I set comint-terminfo-terminal to "xterm-256color", and start aider with --no-fancy-input. Simply ask a question /ask write a helloworld program in python. The output messed up and generated >500 lines, lots of them are duplicated.

Do you know anything wrong with this config, and how to make it work? Thanks in advance. A screenshot would also be helpful.

[MatthewZMD]

What you are experiencing is a prime example of making your life harder by trying to conform to upstream Aider’s approach. We could spend hours tweaking the output, but ultimately::

1. Aider will redraw sections of the output to ensure consistent fontification. This results in "jumpy" output.
2. The colors used are just basic colors picked by pigments, and don't match the user theme or color pallet. This makes it hard to recognize the code at a glance because the user is used to looking at code using their theme.
3. Aider is highlighting everything inside the SEARCH/REPLACE block as one unit. This means multi-line constructs (like strings) can bleed over into the replace block and lead to incorrect highlighting.

These issues stem from upstream design choices that we have little control over. What we can control is making our lives easier—because we’re using Emacs. We should feel liberated to tweak anything we can.

[MatthewZMD]

keep the aider comint session behavior as same as aider itself is the current plan.

It seems that the author of aider.el has made it clear that the priority is to be consistent with the upstream for the sake of consistency, even if it means sacrificing user experience. While I respect that as a design choice, I firmly believe that Emacs should embrace its own strengths rather than inherit the limitations of a CLI-based design.

If @CeleritasCelery decides to fork this package, I will fully support the effort—or I will take the initiative to fork it myself.

This isn’t about splitting the community but about offering a version of aider.el that prioritizes usability and leverages Emacs’ full capabilities. I still hope we can find common ground, but if the core philosophy remains incompatible, then forking may be the best path forward.

[MatthewZMD]

@CeleritasCelery I've forked aider.el into aidermacs and I already applied your two PRs. We should not trouble the aider.el author any further as the vision clearly do not align. My aidermacs fork prioritizes embracing the Emacs integration, and I have already begin fixing some of the existing aider.el bugs. Feel free to contribute to my fork and let's Make Aider Great Again ;)

[MatthewZMD]

@CeleritasCelery Additionally, I am chatting with manateelazycat about potentially re-creating cursor / aider from scratch for maximum compatibility within Emacs. Let me know if you're interested.

[MatthewZMD]

Within two days, comparing two aider.el, Aidermacs already achieved:

1. Better Support for Multiline Input

   • =aider= is primarily designed as a command-line program, where multiline input is restricted by terminal limitations.
   • Terminal-based tools require special syntax or manual formatting to handle multiline input, which can be cumbersome and unintuitive.
   • =aidermacs= eliminates these restrictions by handling multiline prompts natively within Emacs, allowing you to compose complex AI requests just like any other text input.
   • Whether you're pasting blocks of code or refining AI-generated responses, multiline interactions in =aidermacs= feel natural and seamless.

2. Smarter Syntax Highlighting

   • AI-generated code appears with proper syntax highlighting in major languages.
   • Ensures clarity and readability without additional configuration.

3. Flexible Terminal Backend Support

   • =aidermacs= supports multiple terminal backends (comint and vterm) for better compatibility and performance
   • Easy configuration to choose your preferred terminal emulation
   • Extensible architecture for adding new backends

4. Enhanced File Management from Emacs

   • List files currently in chat with =M-x aidermacs-list-added-files=
   • Drop specific files from chat with =M-x aidermacs-drop-file=
   • View output history with =M-x aidermacs-show-output-history=
   • and more

Feel free to try, to anyone who come across this PR: https://github.com/MatthewZMD/aidermacs