[fix] lm-sys/FastChat/issues/2295

[vaxilicaihouxian]

Why are these changes needed?

First,I use python -m fastchat.serve.cli --model-path /my/mac/path/llm_models/chatglm2-6b --load-8bit --device mps
to run chatglm2-6b.But I got this error: Trying to convert BFloat16 to the MPS backend but it does not have support for that dtype.

Related issue number (if applicable)

#2295

Checks

• I've run format.sh to lint the changes in this PR.
• I've included any doc changes needed.
• I've made sure the relevant tests are passing (if applicable).

[merrymercy]

It seems your modification is not related to the picture you posted in your issue.
In your issue, the picture shows "int32 vs. int64". However, in your code, you changed float and half.
Moreover, in the two if branches you added. You added ".half()" in one "if" branch and one "else" branch. Why is this?

[vaxilicaihouxian]

It seems your modification is not related to the picture you posted in your issue. In your issue, the picture shows "int32 vs. int64". However, in your code, you changed float and half. Moreover, in the two if branches you added. You added ".half()" in one "if" branch and one "else" branch. Why is this?

On mac os (usually device:mps) did not support bfloat. Maybe the reason is not correct.But it works with python -m fastchat.serve.cli --model-path /my/mac/path/llm_models/chatglm2-6b --load-8bit --device mps on my macbook(m2).
If I didn't change these two lines it will show the error Trying to convert BFloat16 to the MPS backend but it does not have support for that dtype..
BTW,I'm not very good at this.Just an experience.:)

[merrymercy]

In the two if branches you added. You added ".half()" in one "if" branch and one "else" branch. Why is this?

[vaxilicaihouxian]

In the two if branches you added. You added ".half()" in one "if" branch and one "else" branch. Why is this?

Oh,that's my mistake.I add half() to these two if under mps device on my local code base.Sorry about it.Now,I fix it.

[merrymercy]

Could you try something like this instead of if/else?
tensor = tmp_state_dict[name].to(torch_dtype).to(device).data

If it works, apply the same change to L178.

[vaxilicaihouxian]

I just fix it by inline to and remove mps condition.It works for chatglm2-6b: