View source for Slicer3:GPURayCaster

[http://www.slicer.org/slicerWiki/index.php/Main_Page Return to Slicer Main Page]<br>
[http://www.slicer.org/slicerWiki/index.php/Documentation-3.6#Modules Return to Slicer 3.6 Documentation]
__NOTOC__

= GPU Ray Caster =



This project involves moving the GPU Ray Caster from VTKEdge into VTK, then making this volume mapper available from Slicer. 


==Overview of GPU Ray Casting==

VTK supports several volume rendering techniques for both regular, rectilinear grids (vtkImageData), and tetrahedral meshes (represented by vtkUnstructuredGrid). Some of these volume mappers primarily utilize the CPU (relying on the GPU only for the final display of the resulting image), while other mappers make use of the resources available on the GPU such as 2D and 3D texture memory and texture mapping functionality. The new vtkKWEGPURayCastMapper in VTKEdge uses the latest advancements available on recent GPUs including fragment programs with conditional and loop operations, multi-texturing and frame buffer objects in order to deliver significantly improved performance over the CPU-based ray casting, while still maintaining high rendering quality.

The basic ray casting concept is quite simple. For each pixel in the final image a ray is traced from the camera through the pixel and into the scene. As the ray passes through a volume in the scene the scalar data is sampled along the ray and those samples are processed and combined to form a final RGBA result for the pixel. The implementation of the ray casting algorithm within an object-oriented visualization system, such as VTK, that utilizes object-order rendering (polygonal projection) for other data objects in the scene is a bit more complex, requiring the ray casting process to consider the current state of the frame buffer in order to intermix the volume data with other geometric data in the scene. In our GPU implementation of ray casting, the volume data is stored on the GPU in 3D texture memory. Ray casting is initiated by rendering a polygonal representation of the outer surface of the volume. A fragment program is executed at each pixel to traverse the data and determine a resulting value that is combined with the existing pixel value in the frame buffer computed during the opaque geometry phase of rendering.

For more detailed information, please see the July 2008 edition of The Source, which can be found here: http://kitware.com/products/archive/kitware_quarterly0708.pdf.

==Feature list==
*Integration of the VTK GPU ray caster into the [http://www.slicer.org/slicerWiki/index.php/Modules:VolumeRendering-Documentation-3.5 Volume Rendering module].
*High speed performance
*Flexibility to support a wide range of graphic card performances
*Interactive Framerate
{|
|[[Image:VtkGPURayCastingInteractiveFramerate.png|thumb|VTK GPU RayCasting runs in a degraded mode to support 20fps when interacting with the view]]
|}
*2 modes of rendering: Composite and Maximum Intensity Projection
{|
|[[Image:VtkGPURayCasting-Composite.png|thumb|VTK GPU RayCasting in Composite mode]]
|[[Image:VtkGPURayCasting-MIP.png|thumb|VTK GPU RayCasting in Maximum Intensity Projection mode]]
|}
*Support scalar mapping: Opacity and Color
{|
|[[Image:VtkGPURayCasting-ColorMapping.png|thumb|VTK GPU RayCasting with Custom Scalar mapping]]
|}
*Clipping
{|
|[[Image:VtkGPURayCasting-Clipping.png|thumb|VTK GPU RayCasting with clipping]]
|}
*Frame/Z-buffer aware (do not hide other objects in the scene)
{|
|[[Image:VtkGPURayCasting-FullyIntegrated.png|thumb|VTK GPU RayCasting within a scene]]
|}

==Integration into VTK==

The GPU Ray Caster was initially funded by the following SBIR projects:

"Visualization of Massive Multivariate AMR Data"
National Science Foundation
Contract Number OII-0548729 

"Rapid, Hardware Accelerated, Large Data Visualization"
Department of Defense / Army Research Laboratory
Contract Number W911NF-06-C-0179

The output of these projects was a new toolkit under GPL terms called VTKEdge. As part of this project, the GPU Ray Caster will be moved out of the VTKEdge toolkit into the BSD licensed Visualization Toolkit (VTK). In the process, the names of the classes will be changed to follow VTK conventions. 

This relocation is currently in process and is expected to be complete by January 11th. The new classes will be located in the VolumeRendering kit of VTK.


==Integration into Slicer==

The plan is to review the current Volume Renderer module and work with Yanling Liu <vrnova@gmail.com> and
Alex Yarmarkovich <alexy@bwh.harvard.edu> to integrate the new mappers to the current Volume Rendering GUI, so that they can be selected at run-time.


==Result March 2010==

The GPU Ray Caster has been moved into VTK 5.6 and Slicer, by shifting its official revision of VTK to 5.6, has support for the VTK ray caster (vtkGPUVolumeRayCast*).<br>
In the <i>Volume Rendering</i> module, an entry has been added: VTK GPU Ray Casting.
{|
|[[Image:VtkGPUVolumeRayCastMapper.png|thumb|600px|VTK GPU Ray Casting]]
|}

==Open Issues==

*When camera is inside the volume, the GPU mapper creates images that are different from the CPU mapper
**views are clipped. See below for an example of clipping. The camera is inside the volume. This does not happen in the VTK CPU mapper. 
{|
|[[image:Endoscopy.png|thumb|400px|Clipping problem when the camera is inside the volume]]
|}
*rasterplanes are oriented differently and are sparser
**partial mail trail:
<blockquote><p>On Tue, Mar 23, 2010 at 6:17 PM, Steve Pieper <pieper@bwh.harvard.edu> wrote: </p><p>Hi Lisa - Notice how the artifacts are oriented differently: in the CPU version the contours are aligned with view space while in the GPU version they appear to be in image space (and are more visible). Also, the noisy parts of the image (say in the upper right) are more smooth in the CPU version while in the GPU version they are sharper. As far as I know the same options were used for both mappers - these images were generated with the latest slicer3 with the VTK cvs head only picking a different option from the Technique option menu. These issues are noticable when the voxel size is large relative to the screen pixel size - this situations comes up in particular for the virtual endoscopy applications....    -Steve</p></blockquote>
<blockquote><p>On Wed, Mar 24, 2010 at 9:29 AM, Lisa Avila <lisa.avila@kitware.com> wrote:</p><p>Hi Steve - You will see a difference in artifacts because of where the rays originate. In the CPU ray casting method, rays start at the camera position then we figure out how many steps (using the sample distance) to take before we are inside the volume. In the GPU ray cast mapper, we start the rays at the bounding box of the volume. So this will give you a slightly different pattern on your volume. The reason you are seeing these lines is that you are not sampling finely enough. In general you can use the nyquist rule to sample at 1/2 your voxel spacing - but this only works if your transfer function is just a direct mapping of scalar value to color / opacity (that is, the scalar value with some simple shift / scale yields the opacity and color) But you've probably introduced higher frequencies in your function by mapping some values to a dark color / low opacity, then some other not too distant scalar value to a bright color / high opacity. Having said that - if you are truly using the same sample distance in both mappers (can you verify this?) I don't see why the apparent spacing for the GPU mapper looks bigger than for the CPU mapper. I'll have to consult with Francois. Also - we did attempt to alleviate this issue in the GPU mapper by jittering the starting point of the ray. This should essentially add noise to the artifact (instead of lining up along obviously noticeable contour lines, the effect will be a more subtle stippling) However, we never did get that working correctly. We should probably revisit that at some point. Lisa</p></blockquote>
{|
|[[image:VTK-GPU.png|thumb|400px|VTK GPU rendering]]
|[[image:VTK-CPU.png|thumb|400px|CPU rendering]]
|}
*Not axis-aligned Volume have odd clipping even when clipping is OFF
** See [http://na-mic.org/Mantis/view.php?id=800 Mantis bug]
{|
|[[Image:VtkGPURayCasting-AutoClipping.png|thumb|400px|VTK GPU RayCasting of non-axis-aligned volume]]
|[[Image:VtkGPURayCasting-AutoClipping-CPU.png|thumb|400px|VTK GPU RayCasting of non-axis-aligned volume]]
|}
*ROI clipping is not handled properly for non axis-aligned volumes. (backface culling turned off?)
** See [http://na-mic.org/Mantis/view.php?id=798 Mantis bug]
*Integration with dual view layout has not happened
{|
|[[Image:VtkGPURayCasting-DualView.png|thumb|400px|VTK GPU RayCasting doesn't support Dual View]]
|}