RuntimeError: "LayerNormKernelImpl" not implemented for 'Half'

[888yyh]

i can only use gpu，and ran the code below

import torch
from transformers import AutoTokenizer, AutoModel
from modeling_chatglm import ChatGLMForConditionalGeneration

tokenizer = AutoTokenizer.from_pretrained(
"./model", trust_remote_code=True)

model = ChatGLMForConditionalGeneration.from_pretrained(
"./model").half()

while True:
a = input("请输入您的问题：（输入q以退出）")
if a.strip() == 'q':

    exit()
response, history = model.chat(tokenizer, "问题：" + a.strip() + '\n答案：', max_length=256, history=[])
print("回答：", response)

errors:RuntimeError: "LayerNormKernelImpl" not implemented for 'Half',however i use model = ChatGLMForConditionalGeneration.from_pretrained(
"./model",torch_type=torch.float32),error "RuntimeError: mixed dtype (CPU): expect parameter to have scalar type of bFloat "
if i use ChatGLMForConditionalGeneration.from_pretrained(
"./model",torch_type=torch.Bfloat) errors"RuntimeError: mixed dtype (CPU): expect parameter to have scalar type of Float"
how can i run the code correctly?

[Charon-HN]

i can only use gpu，and ran the code below

import torch from transformers import AutoTokenizer, AutoModel from modeling_chatglm import ChatGLMForConditionalGeneration

tokenizer = AutoTokenizer.from_pretrained( "./model", trust_remote_code=True)

model = ChatGLMForConditionalGeneration.from_pretrained( "./model").half()

while True: a = input("请输入您的问题：（输入q以退出）") if a.strip() == 'q':

    exit()
response, history = model.chat(tokenizer, "问题：" + a.strip() + '\n答案：', max_length=256, history=[])
print("回答：", response)

errors:RuntimeError: "LayerNormKernelImpl" not implemented for 'Half',however i use model = ChatGLMForConditionalGeneration.from_pretrained( "./model",torch_type=torch.float32),error "RuntimeError: mixed dtype (CPU): expect parameter to have scalar type of bFloat " if i use ChatGLMForConditionalGeneration.from_pretrained( "./model",torch_type=torch.Bfloat) errors"RuntimeError: mixed dtype (CPU): expect parameter to have scalar type of Float" how can i run the code correctly?

What a coincidence! I encoutered the same problem as you. I found a Blog that might be useful to you.

To summary, the reason for this problem should be that the model is not placed on the GPU. Because the model needs to be half() and CPU can't do it, only GPU can.

So you can try to add .cuda(), as shown below

model = ChatGLMForConditionalGeneration.from_pretrained(
"./model").half().cuda()