I just made the commit, have at it =)

It was crashing my debugger a lot in the beginning before i realized the bilateral filter was not optional ..

I split the mesh generation and video generation into two sections .. So it's easy and quick to generate videos once you have the mesh.
The UI is a bit messy like this, not sure how to make the layout, suggestions welcome.

Initially the video generation required the rgb and depthmap and the mesh, but was able to get rid of it by writing the required value into the ply comments. It already stored original width, height, and fov like this. So now it only needs the mesh to generate videos.

I optimized the read_ply function a bit, got me a 20% increase in speed for (re-)reading the mesh.

As far as I understand it projects in the camera perspective, but segments the mesh into 'layers', whose edges are inpainted by some margin. There are still holes, but the overall freedom of movement before they become visible is increased compared to the method previously discussed / the blender addon.

You can import this mesh into blender, and use the 'Color Attribute' Node to plug into the Shader 'color' to get the mesh rendering in color, and set the camera to origin and pointing up to align them. Might also need to set the FOV to match.

It also occurred to me that this might also be a source for filling the gaps in the stereo images, but the current method is near realtime while this method requires a dozen minutes or so for a single frame ..

Yes, the 3D format is ASCII PLY, list of vertices with rgba color and list of faces. Converting to Wavefront OBJ should be easy, but is equally bulky due to the ASCII nature. Is there such a thing as .ply.zip, or .zply, I might consider implementing something similar as the files are quite large.

Results are quite impressive, I've generated some very pleasing slow long panning shots of that spaceship image that could go straight into a movie..