Chat mode inference stops mid-sentence

[Belluxx]

I noticed that in the chat mode the inference often stops mid-sentence and requires the user to press enter to continue.

However this introduces a new line at the end of the context that makes LLaMA terminate what it was saying and give back the control to the user with the reverse prompt User:

Does someone know why this happens and if it would be beneficial to make the Enter keypress just continue inference instead of also adding a \n when both -i and --reverse-prompt are passed? Obviously the \n append is skipped only if the termination was not due to a reverse prompt match.

Note: this happens also after very few tokens (5-10) sometimes.

[x02Sylvie]

I noticed it happening myself

it kinda occurs like this
Miko: The game developement is [interferene stops, i need to press enter]
hobby for some of people.

[vicfic18]

I noticed this too, but for me after pressing enter it prints out the introduction of some random book.

[Belluxx]

Yes, same for me, sometimes it gives User: and others it exits the chat completely by starting to write a book

[UltraPhil]

+1 here. Sometimes I wait it out and it restarts, but if I see the CPU fall to zero, then it won't buldge until I press enter.

[keldenl]

same here

[x4080]

is there solution for this ? I tried to use --mlock and -c 2048, but still stopped

[DannyDaemonic]

There are two possible fixes: -n -1 and -n -1 --ignore-eos.

Some models will use the eos to return control to you when they are done answering. So depending which model you're using, one might work better than the other. If in doubt, try -n -1 first.

[x4080]

@DannyDaemonic , I just tried that adding -c 2048 with vicuna 13b havent gotten stopped yet, I'll try your suggestion if I found another stop. Thank you very much

[DannyDaemonic]

-c 2048 increases the context, which is just another way of saying how far back the model remembers.

When the context fills up, it has to cut your old history in half and reprocess it all (to make room to remember new stuff). This can also be a slow process, so it's possible the pause you were seeing was simply the system "thinking" and it wasn't actually waiting for you to hit enter.

Or it could be a combination.

-n tells it how many tokens to generate before handing control back to you. So -n 5 would make it stop every 3 or 4 words. The default this number is now -1, so it probably isn't the cause unless you're manually setting -n (or --n_predict) to something else.

[x4080]

@DannyDaemonic maybe I was wrong, I meant -n 2048 ? Not sure sorry, I was only this morning playing with this 😄
so probably I should set -n -1 instead of 2048 ?

[x4080]

Ok, thanks all for your help, the key is like @DannyDaemonic said : -n -1
And I found that it needs huge memory, in my m2 it shows swap 307MB (bad thing) - when using 13B 4bit quantized vicuna I think after long session of chat - the stop like is because it doing the swaping I guess (using with --mlock), I will try without --mlock if it still doing swaping

[DannyDaemonic]

The long pause you get after a long conversation is usually due to the context memory being trimmed and re-evaluated so it has space to continue the conversation. --mlock won't change how often this happens, but it could make such re-evaluations faster.

Context length isn't usually limited by your computer's RAM, but rather by how much context the model has been trained to handle.

[x4080]

@DannyDaemonic is it possible I experienced memory leak then? It happened when lot of back and forth chatting going on, then the memory swapping happening

[DannyDaemonic]

I'm guessing the "swapping" you are experiencing is the context swapping. It can be a very slow process. It happens once the context fills up, so this will happen after a long conversation.

[x4080]

@DannyDaemonic, I see. Today I try to replicate again, without using --mlock, and it wont swap, and the speed different is almost un-noticeable, I think for now I better not use --mlock and with -n -1
M2 pro 16gb

[staviq]

This is still happening. I'm using the server, and in the browser console it tells you the reason for stopping, and when I'm getting incomplete sentences or incomplete code I asked for, the debug console says the reason for stopping was eos.

When this happens, almost certainly the next couple of sentences get progressively more meaningless, including repeating my own questions, attempts at talking to itself, or endless stream of a single character, usually \n or #, and if I continue, it starts to produce more and more garbage, completely ignoring my input, and replying to absolutely random pieces of previous conversation.

It's almost as if when the context gets full and it does it's context "compression" thing, it butchers the question/reply scheme and the model sees messed up things in the context.

[pugzly]

Maybe some breaking changes might have been introduced recently?

I clearly remember about a month or two ago I was able to have long conversations with large WizardLM models (in interactive/chat mode), but this morning, after long break, I downloaded and compiled latest llama.cpp, and re-quantized my model, and I can only get 1-2 responses from it before it freeze up and then it would start generating random gibberish or talking to itself, after I hit enter.

I tried prompting with and without "-n" parameter, tried different models, but I'm yet to find combination which would work.

[staviq]

I just ran couple of tests on a "single run" mode from the command line, giving it a prompt of a conversation to continue, with -n -1 and the ban eos flag (can't remember it exactly, I'm on my phone right now), just to see what happens.

It continued to generate for a fairly long time, way past the context limit, but at some point it REALLY tried to stop, by saying things like "the end", and generating some more and then saying "for real the end", and then doing some more to the point where it just started to sound like an a SMS "ok, bye I'm going to sklep now, BYE, BYYYYEEEE" and after that gave me a copious amount of emoji followed by continuous stream of non printable characters, aka it got completely off the rails.

(But it was genuinely hilarious, like it wanted to let me know it had absolutely enough, like in the "blink twice if you need help" kind of way)

Any prompt, and several different model, all get to the same point eventually, during repeated tests. Sometimes sooner, sometimes it got quite far.

Edit: Wait, now that I think about it, did an AI just imply that I'm a creep ? Did I just get friendzined by na AI ?

[pugzly]

With a bit more testing I think I found one of the recent commit introduced some changes, which are likely the cause of my issues #2304
Copy of llama.cpp prior to that commit works as it used to, but the latest one either is not working with older models or I need to prompt it differently? But don't see anything mentioned in the readme, so no idea yet.

[mattiashallberg]

Using the following parameters with build 916a9ac: -c 512 -b 1024 -n 1024 --keep 1024 --repeat_penalty 1.1 --color -i -p "my question..."
I do not get the issue with any annoying stop, but after a few questions and answers, completely random output appears.
Wondering if it has anything to do with the "--keep" parameter.

https://github.com/ggerganov/llama.cpp/blob/master/examples/main/README.md

[hiqsociety]

@mattiashallberg having same issues. how to make it "continue" response without stopping?

[mattiashallberg]

I used version master-fc8ef54, not sure if that was the whole reason it works for me though...