Scribd Logo
Upload

Welcome to Scribd!

  • 5 views

    GSplat

    Uploaded by

    sidchicharito

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd

    GSplat

    Uploaded by

    sidchicharito
    5 views3 pages

    Copyright

    © © All Rights Reserved

    Available Formats

    PDF, TXT or read online from Scribd

    Did you find this document useful?

    Is this content inappropriate?

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd
    Download as pdf or txt
    5 views3 pages

    GSplat

    Uploaded by

    sidchicharito

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd
    Download as pdf or txt
    of 3

    TheViewPro <> Preimage

    3DSMax + Gaussian Splatting workflow exploration and experiments


    Oct 24-Nov 03, 2023

    Pipeline
    Given a set of equirectangular images, depth maps and camera poses, we implemented a
    preprocessing script which does the following
    ● Filter out noisy depths to an extent by masking depth maps near the high gradient areas.
    ● Project depth and image to world frame to create a colored pointcloud.
    ● Subsample the pointcloud to be of reasonable size.
    ● Extract 6 cubemap faces for each equirectangular image, and create corresponding
    camera poses.
    ● Change the camera convention to blender format save in the format directly readable by
    gaussian splatting.

    Problems
    ● Reflective/Refractive surfaces / Mirror
    With limited number of images available, gaussian splatting is unable to learn the view
    dependent color of highly reflective (mirror like) or refractive (glass) surface, and it
    hallucinates additional geometry in the form of floaters to explain those views.
    ● High frequency geometry
    The given depth maps are not perfect for regions containing sharp changes in depth. For
    eg. edges, plants, etc. It creates a lot of noise in the pointcloud and gives a reason to
    gaussian splatting to hallucinate something there.

    Recommendations
    We have observed that gsplat training is close to perfect with 500 images, whereas it starts
    degrading by reducing the number of images. Somewhere around 100 images could be a good
    sweet spot.
    Using equirectangular 360 images adds the flexibility to substantially reduce the number of
    images required. We've shared the result which is trained with only 14 equirectangular images,
    it looks overall decent with some minor issues near the problematic areas (plants, TV, glossy
    Fridge, Mirror, Glass door).
    We think using around 20-25 equirectangular images, and keeping more frames near reflective
    and other problematic surfaces should produce better results.
    Also, if 3ds max can render uncertainty / probability in depth maps, it would be helpful in
    creating a super clean pointcloud.

    Screenshots
    Left image has been taken near the input camera position, and the right one is a slightly shifted
    location. It shows overfitting and hallucinations.
    Overfitting of Glass door Slightly shifted camera location

    Near the mirror Slightly shifted


    Glossy Fridge Hallucinations visible from different view

    Capturing more frames near these type of problematic surface should solve this problem.

    Codebase
    We have shared two script
    ● create_pcld.py
    This script assumes images and depth maps are in folder "360s", and their name is
    "xx.jpg". This script will save a clean pointcloud "points3d.ply".
    ● gsplat_cubemap.py
    This script will convert 360 images into cubemaps and "transforms_train.json", and
    "transforms_test.json" required to run gaussian splatting.

    Copy over the output of both script to gaussian splatting data directory to train on this data.

    You might also like