Documentation/Labs/Improving Markups

Overview

The goal of this page is to support improvements to the Markups Module for fiducial placement. This page will document improvements suggested by users and developers and coordinate efforts to address them.

History

Markups, also known as annotations (and almost the same as VTK widgets) proved to be very difficult to design well. First-generation VTK widgets were simple but too monolithic, their behavior and appearance was not customizable enough. Second-generation VTK widgets took a completely new approach, separated "widget" from "representation" allowing much more customization, but also resulting in very complex widgets, with very poor performance in some cases (mainly because many classes are used to implement simple widgets and there is a complex and costly synchronization of states between them). Widgets in Slicer are similarly problematic. First attempt resulted in "Annotations" module, which was very complex to use and develop and also very unstable and so most widgets had to be disabled and only ruler (line) and ROI widgets remained. "Markups" module was the second attempt. The project ran out of time and only markup fiducials were implemented. Implementation is complicated, which makes adding new features very difficult, it has serious issues (sometimes points cannot be clicked or hidden points can be clicked), and it becomes unusably slowe after placing 100-200 points. There have been additional attempts to rework widgets, but so far they have all failed. We don't give up, so here we are again.

Design considerations

Data synchronization between multiple classes (VTK and MRML nodes, instances for each viewer, widget and representation) has proven to be one of the fundamental issues. It would be nice to create generic VTK-only widgets and use them in Slicer, but this additional layer of abstraction would make implementation more complicated and degrade performance. Therefore, should be Slicer-specific, directly using information stored in MRML nodes.

Separation between "widget" (behavior) and "representation" (appearance) in second-generation VTK widgets failed. Lots of data and state information had to be manually synchronized between many classes. When widget was present in multiple views then it made this manual synchronization extremely slow and complicated. Therefore, we need to split classes in a way that minimizes need for manual synchronization. Separation of "data" (shared globally in the whole scene, such as widget point positions) must be separated from "display" (properties that control how a widget should appear in each view) works very well for nodes and should be used for widgets as well. Data class should add thin wrapper over VTK data object (e.g., vtkPointSet), which would allow the same data object used in multiple renderers and alleviate any need for synchronizing point lists. Some state information can be shared (e.g., selected point) by storing it in the data node; while other state information (e.g., mouse is hovered over a point) could be specific to a view by storing it in the view's displayable manager class.

Performance of markups module is very slow because each clickable point is implemented by a handle widget. Each handle widget requires instantiation of several classes per views (widget, representation, actors, pickers, etc.), many view observations, and trigger various expensive synchronization mechanisms when anything is changed. Instead, a single actor should be able to handle display, picking, visualization of points (using very efficient vtkGlyph3D filter).

Labels have been problematic. In current markups module, labels are rendered as polygons in 3D. This makes their appearance very low quality (anti-aliasing is not available) and they often very hard to see because obscured by 3D objects. We could render the text in a texture and display as a semi-transparent rectangle, but image quality may still be an issue (texture resolution would need to match screen size and resolution) and occlusion by other 3D objects could still make it difficult to see some labels. If labels were rendered as 2D text actors then they could be displayed in high quality but they would always appear as a separate layer over 3D objects (they could appear very disconnected from the 3D glyphs). Maybe adding semi-transparent callout lines between labels and glyphs would help. It would also allow laying out labels so that they never overlap each other but still close to the 3D glyph.

Persistence: it should be possible to save all annotations in the scene (without storage node), to avoid having lots of tiny files (with just a few numbers in each). On the other hand, it would be nice to be able to save annotations in a standard file format and not just in MRML scene. It would be nice to be able to save group of markups to a single file, but unfortunately storing multiple nodes in a single file is problematic. We could add importers/exporters for standard annotation file formats (DICOM, AIM, ...).

Class roles & responsibilities (https://github.com/Kitware/vtk-js/issues/682)

• WidgetState (MVC: model, VTK: source) - similar to vtkMRMLNode
  • Stores point positions, curve interpolation type, etc.
  • One instance shared between all renderers.
  • Maybe it could store some state information, such as selected or highlighted point, so that a point that is being selected or dragged can be highlighted in all views. API to update state (such as translate, scale...)
  • Slicer: Stored in the MRML data node
• WidgetRepresentation (MVC: view, VTK: mapper) - similar to vtkWidgetRepresentation
  • Stores actor, mapper, and display properties.
  • There must be a separate instance for each renderer.
  • There are different views, for example one for 3D and another one for 2D views.
  • API to check if it is picked at a certain location (but no event handling, so it would be almost the state of the rendering)
  • Slicer: stored in displayable manager pipeline
• Widget (MVC: controller) - similar to vtkAbstractWidget
  • listen to events from interactor (mouse move, click, etc.)
  • Probably we need a controller instance for each renderer, but it might be shared between renderers?
  • checks if currently interacting with the representation (using representation API), and update model/state accordingly.
  • Slicer: stored in displayable manager pipeline

Planned features

Widget types:

• point set
• line (arrow, ruler)
• ROI (may be 2D only: ellipse, circle, rectangle)
• bidimensional
• angle
• curve (open and closed, with different interpolations)
• plane
• slice intersection (show slice intersections and allow shift/rotate them), crosshair
• linear transforms (2D: similar to vtkAffineWidget; 3D: current transform widget)
• ROI-based image window width/level

Proposed Improvements:

• Restricting placement and motion to other nodes (model surface, volume rendered image surface, etc.)
• One click 3D angle, length measurements from landmarks (latter is already available)
• Read a template for fiducial labels to be populated from
• Allow adding fiducials with undefined position
• Fine control for fiducial placement/modifications by keyboard strokes
• Improved scaling of 3D glyphs and 2D points. For example, fixed in screen size, fixed relative to voxel size of the data set, etc.
• Projection display in slice views (similar to model projections, markup point projection)
• Select entire markup or just a point
• Delete using keyboard shortcut or right-click
• Right-click menu (delete, jump to slice, fit slice views to main axes)
• Highlight/display info when hovering over
• Compute metrics (length, surface, diameter, etc.), allow them to be exported into table or other standard annotation file formats
• Undo modifications
• Sensible behavior when non-linear transform is applied

References

• https://discourse.slicer.org/t/a-users-proposal-for-markups-module-enhancements/3362
• https://discourse.slicer.org/t/markups-module-enhancements/4335
• https://issues.slicer.org/view.php?id=3849
• https://issues.slicer.org/view.php?id=3713
• https://na-mic.org/wiki/Projects:ARRA:SlicerAM:vtkWidgets
• https://github.com/Kitware/vtk-js/issues/682
• https://en.wikibooks.org/wiki/Online_OsiriX_Documentation/Navigational_shortcuts
• https://github.com/lassoan/Slicer/commit/cbd43fd2b8eb552e99f5dd29c0ed29b0d1af7e26
• https://www.vtk.org/Wiki/VTK_Widget_Examples