Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Missing weights for depth estimation. #10

Closed
JiuTongBro opened this issue Dec 5, 2023 · 5 comments
Closed

Missing weights for depth estimation. #10

JiuTongBro opened this issue Dec 5, 2023 · 5 comments

Comments

@JiuTongBro
Copy link

Hi, thanks for your impressive work firstly.

However, when I try to add an instance to the scene, the code threw an error:
FileNotFoundError: [Errno 2] No such file or directory: '.cache/dpt/omnidata_dpt_depth_v2.ckpt'

I suppose it is because the weights for depth estimation is missed. I tried to download it manually and put it under the utils/cache/dpt/ folder, but it still doesn't work. Do you know how to solve it?

This is the full logs about the error, it happened after the Wonder3D reconstruction.

Traceback (most recent call last):
  File "/home/zhonghongliang/diffusion_pro/gsedit_pro/gsed_env/lib/python3.8/site-packages/mediapy/__init__.py", line 1739, in write_video
    writer.add_image(image)
  File "/home/zhonghongliang/diffusion_pro/gsedit_pro/gsed_env/lib/python3.8/site-packages/mediapy/__init__.py", line 1648, in add_image
    if self._proc.stdin.write(data) != len(data):
BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "train_from_mesh.py", line 261, in <module>
    cams, images, camera_extent = render_multiview_images_from_mesh(args.mesh,
  File "/home/zhonghongliang/diffusion_pro/gsedit_pro/GaussianEditor/threestudio/utils/render.py", line 150, in render_multiview_images_from_mesh
    write_video(save_path, frames, fps=2)
  File "/home/zhonghongliang/diffusion_pro/gsedit_pro/gsed_env/lib/python3.8/site-packages/mediapy/__init__.py", line 1739, in write_video
    writer.add_image(image)
  File "/home/zhonghongliang/diffusion_pro/gsedit_pro/gsed_env/lib/python3.8/site-packages/mediapy/__init__.py", line 1610, in __exit__
    self.close()
  File "/home/zhonghongliang/diffusion_pro/gsedit_pro/gsed_env/lib/python3.8/site-packages/mediapy/__init__.py", line 1661, in close
    raise RuntimeError(f"Error writing '{self.path}': {s}")
RuntimeError: Error writing '/home/zhonghongliang/diffusion_pro/gsedit_pro/GaussianEditor/tmp_add/mesh_rendering.mp4': /home/zhonghongliang/diffusion_pro/gsedit_pro/gsed_env/bin/ffmpeg: error while loading shared libraries: libopenh264.so.5: cannot open shared object file: No such file or directory

/home/zhonghongliang/diffusion_pro/gsedit_pro/gsed_env/lib/python3.8/site-packages/timm/models/_factory.py:117: UserWarning: Mapping deprecated model name vit_base_resnet50_384 to current vit_base_r50_s16_384.orig_in21k_ft_in1k.
  model = create_fn(
Traceback (most recent call last):
  File "/home/zhonghongliang/diffusion_pro/gsedit_pro/GaussianEditor/extern/viser/src/viser/infra/_infra.py", line 449, in wrapped
    inner()
  File "/home/zhonghongliang/diffusion_pro/gsedit_pro/GaussianEditor/extern/viser/src/viser/infra/_infra.py", line 257, in <lambda>
    lambda: self._handle_incoming_message(client_id, message)
  File "/home/zhonghongliang/diffusion_pro/gsedit_pro/GaussianEditor/extern/viser/src/viser/infra/_infra.py", line 94, in _handle_incoming_message
    cb(client_id, message)
  File "/home/zhonghongliang/diffusion_pro/gsedit_pro/GaussianEditor/extern/viser/src/viser/_gui_api.py", line 157, in _handle_gui_updates
    cb(GuiEvent(client, client_id, handle))
  File "webui.py", line 511, in _
    self.add(self.camera)
  File "/home/zhonghongliang/diffusion_pro/gsedit_pro/gsed_env/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "webui.py", line 1371, in add
    depth_estimator = DPT(get_device(), mode="depth")
  File "/home/zhonghongliang/diffusion_pro/gsedit_pro/GaussianEditor/threestudio/utils/dpt.py", line 1023, in __init__
    checkpoint = torch.load(path, map_location="cpu")
  File "/home/zhonghongliang/diffusion_pro/gsedit_pro/gsed_env/lib/python3.8/site-packages/torch/serialization.py", line 986, in load
    with _open_file_like(f, 'rb') as opened_file:
  File "/home/zhonghongliang/diffusion_pro/gsedit_pro/gsed_env/lib/python3.8/site-packages/torch/serialization.py", line 435, in _open_file_like
    return _open_file(name_or_buffer, mode)
  File "/home/zhonghongliang/diffusion_pro/gsedit_pro/gsed_env/lib/python3.8/site-packages/torch/serialization.py", line 416, in __init__
    super().__init__(open(name, mode))
FileNotFoundError: [Errno 2] No such file or directory: '.cache/dpt/omnidata_dpt_depth_v2.ckpt'
@buaacyw
Copy link
Owner

buaacyw commented Dec 7, 2023

Thanks for your interest.
It supposed to be automatically downloaded from online.
I'm not sure why the download failed in your case.

@heheyas
Copy link
Collaborator

heheyas commented Dec 7, 2023

Hi, JiuTongBro

You can try downloading the checkpoint file directly using the following commands:
If necessary:

mkdir -p .cache/dpt
pip install gdown

Download pre-trained models

gdown '1Jrh-bRnJEjyMCS7f-WsaFlccfPjJPPHI&confirm=t' -O ./cache/dpt

Reference

@JiuTongBro
Copy link
Author

Hi, JiuTongBro

You can try downloading the checkpoint file directly using the following commands: If necessary:

mkdir -p .cache/dpt
pip install gdown

Download pre-trained models

gdown '1Jrh-bRnJEjyMCS7f-WsaFlccfPjJPPHI&confirm=t' -O ./cache/dpt

Reference

Thanks, it works, but then the pipeline broke again when reconstructing Gaussian from the mesh. Here is the full log:

Traceback (most recent call last):
  File "/home/zhonghongliang/diffusion_pro/gsedit_pro/gsed_env/lib/python3.8/site-packages/mediapy/__init__.py", line 1739, in write_video
    writer.add_image(image)
  File "/home/zhonghongliang/diffusion_pro/gsedit_pro/gsed_env/lib/python3.8/site-packages/mediapy/__init__.py", line 1648, in add_image
    if self._proc.stdin.write(data) != len(data):
BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "train_from_mesh.py", line 261, in <module>
    cams, images, camera_extent = render_multiview_images_from_mesh(args.mesh,
  File "/home/zhonghongliang/diffusion_pro/gsedit_pro/GaussianEditor/threestudio/utils/render.py", line 150, in render_multiview_images_from_mesh
    write_video(save_path, frames, fps=2)
  File "/home/zhonghongliang/diffusion_pro/gsedit_pro/gsed_env/lib/python3.8/site-packages/mediapy/__init__.py", line 1739, in write_video
    writer.add_image(image)
  File "/home/zhonghongliang/diffusion_pro/gsedit_pro/gsed_env/lib/python3.8/site-packages/mediapy/__init__.py", line 1610, in __exit__
    self.close()
  File "/home/zhonghongliang/diffusion_pro/gsedit_pro/gsed_env/lib/python3.8/site-packages/mediapy/__init__.py", line 1661, in close
    raise RuntimeError(f"Error writing '{self.path}': {s}")
RuntimeError: Error writing '/home/zhonghongliang/diffusion_pro/gsedit_pro/GaussianEditor/tmp_add/mesh_rendering.mp4': /home/zhonghongliang/diffusion_pro/gsedit_pro/gsed_env/bin/ffmpeg: error while loading shared libraries: libopenh264.so.5: cannot open shared object file: No such file or directory

/home/zhonghongliang/diffusion_pro/gsedit_pro/gsed_env/lib/python3.8/site-packages/timm/models/_factory.py:117: UserWarning: Mapping deprecated model name vit_base_resnet50_384 to current vit_base_r50_s16_384.orig_in21k_ft_in1k.
  model = create_fn(
/home/zhonghongliang/diffusion_pro/gsedit_pro/gsed_env/lib/python3.8/site-packages/torchvision/transforms/functional.py:1603: UserWarning: The default value of the antialias parameter of all the resizing transforms (Resize(), RandomResizedCrop(), etc.) will change from None to True in v0.17, in order to be consistent across the PIL and Tensor backends. To suppress this warning, directly pass antialias=True (recommended, future default), antialias=None (current default, which means False for Tensors and True for PIL), or antialias=False (only works on Tensors - PIL will still use antialiasing). This also applies if you are using the inference transforms from the models weights: update the call to weights.transforms(antialias=True).
  warnings.warn(
depth valid percent:  tensor(0.9923, device='cuda:0')
Traceback (most recent call last):
  File "/home/zhonghongliang/diffusion_pro/gsedit_pro/GaussianEditor/extern/viser/src/viser/infra/_infra.py", line 449, in wrapped
    inner()
  File "/home/zhonghongliang/diffusion_pro/gsedit_pro/GaussianEditor/extern/viser/src/viser/infra/_infra.py", line 257, in <lambda>
    lambda: self._handle_incoming_message(client_id, message)
  File "/home/zhonghongliang/diffusion_pro/gsedit_pro/GaussianEditor/extern/viser/src/viser/infra/_infra.py", line 94, in _handle_incoming_message
    cb(client_id, message)
  File "/home/zhonghongliang/diffusion_pro/gsedit_pro/GaussianEditor/extern/viser/src/viser/_gui_api.py", line 157, in _handle_gui_updates
    cb(GuiEvent(client, client_id, handle))
  File "webui.py", line 511, in _
    self.add(self.camera)
  File "/home/zhonghongliang/diffusion_pro/gsedit_pro/gsed_env/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "webui.py", line 1436, in add
    new_object_gaussian.load_ply(gs_path)
  File "/home/zhonghongliang/diffusion_pro/gsedit_pro/GaussianEditor/gaussiansplatting/scene/vanilla_gaussian_model.py", line 293, in load_ply
    plydata = PlyData.read(path)
  File "/home/zhonghongliang/diffusion_pro/gsedit_pro/gsed_env/lib/python3.8/site-packages/plyfile.py", line 158, in read
    (must_close, stream) = _open_stream(stream, 'read')
  File "/home/zhonghongliang/diffusion_pro/gsedit_pro/gsed_env/lib/python3.8/site-packages/plyfile.py", line 1345, in _open_stream
    return (True, open(stream, read_or_write[0] + 'b'))
FileNotFoundError: [Errno 2] No such file or directory: '/home/zhonghongliang/diffusion_pro/gsedit_pro/GaussianEditor/tmp_add/inpaint_gs.obj'

I checked the tmp_add folder, there are four items: multiview_pred_images (a folder), inpaint_mesh.obj, inpainted.png, and removed_bg.png. I suppose it is caused by a failure in train_from_mesh.py, which convert the 3D mesh reconstructed by Wonder3D to Gaussians. Could you please check the code and fix this potential bug? Thanks.

@heheyas
Copy link
Collaborator

heheyas commented Dec 8, 2023

This error is related to ffmpeg (used inside mediapy), you can try reinstall ffmepg.

Reference

@JiuTongBro
Copy link
Author

This error is related to ffmpeg (used inside mediapy), you can try reinstall ffmepg.

Reference

Yes, I have fixed it and successfully run the whole pipeline. Thanks!

@heheyas heheyas mentioned this issue Dec 20, 2023
Closed
@Nieschlafen Nieschlafen mentioned this issue Mar 4, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@heheyas @buaacyw @JiuTongBro