Difference between revisions of "Documentation/Labs/Improving Markups"

From Slicer Wiki
Jump to: navigation, search
Line 8: Line 8:
 
== Design considerations ==
 
== Design considerations ==
  
−
Data synchronization between multiple classes (VTK and MRML nodes, instances for each viewer, widget and representation) has proven to be one of the fundamental issues. It would be nice to create generic VTK-only widgets and use them in Slicer, but this additional layer of abstraction would make implementation more complicated and degrade performance. Therefore, should be Slicer-specific, directly using information stored in MRML nodes.
+
Data synchronization between multiple classes (VTK and MRML nodes, instances for each viewer, widget and representation) has proven to be one of the fundamental issues. It would have been nice to create generic VTK-only widgets and use them in Slicer, but the additional layer of abstraction would have made implementation more complicated and degraded performance. Therefore, we do not use VTK widgets and implement Slicer-specific classes, which directly use information stored in MRML nodes.
  
−
Separation between "widget" (behavior) and "representation" (appearance) in second-generation VTK widgets failed. Lots of data and state information had to be manually synchronized between many classes. When widget was present in multiple views then it made this manual synchronization extremely slow and complicated. Therefore, we need to split classes in a way that minimizes need for manual synchronization. Separation of "data" (shared globally in the whole scene, such as widget point positions) must be separated from "display" (properties that control how a widget should appear in each view) works very well for nodes and should be used for widgets as well. Data class should add thin wrapper over VTK data object (e.g., vtkPointSet), which would allow the same data object used in multiple renderers and alleviate any need for synchronizing point lists. Some state information can be shared (e.g., selected point) by storing it in the data node; while other state information (e.g., mouse is hovered over a point) could be specific to a view by storing it in the view's [[Documentation/Nightly/Developers/DisplayableManagers|displayable manager]] class.
+
Separation between "widget" (behavior) and "representation" (appearance) is a good concept, but in second-generation VTK widgets, widgets and representations are too much separated (they don't access each other's state but communicate with state transitions and events). This scheme may be useful for client-server visualization, but it is inefficient and too complicated to use within a simple application. Therefore, we need to split classes in a way that minimizes need for manual synchronization. Separation of "data" (shared globally in the whole scene, such as widget point positions) must be separated from "display" (properties that control how a widget should appear in each view) works very well for displayable MRML nodes and chosen to be used for widgets as well. Data adds thin wrapper over VTK data object (e.g., vtkPointSet), which allows the same data object used in multiple renderers and alleviate any need for synchronizing point lists. State information (even low-level detail, such as markup that the user is currently hovered over) is stored in markup data and display nodes.
  
−
Performance of markups module is very slow because each clickable point is implemented by a handle widget. Each handle widget requires instantiation of several classes per views (widget, representation, actors, pickers, etc.), many view observations, and trigger various expensive synchronization mechanisms when anything is changed. Instead, a single actor should be able to handle display, picking, visualization of points (using very efficient vtkGlyph3D filter).
+
Performance of markups module was very slow because each clickable point was implemented by a handle widget. Each handle widget requires instantiation of several classes per views (widget, representation, actors, pickers, etc.), many view observations, and trigger various expensive synchronization mechanisms when anything is changed. Instead, a single glyph actor is created to display all markups.
  
−
Labels have been problematic. In current markups module, labels are rendered as polygons in 3D. This makes their appearance very low quality (anti-aliasing is not available) and they often very hard to see because obscured by 3D objects. We could render the text in a texture and display as a semi-transparent rectangle, but image quality may still be an issue (texture resolution would need to match screen size and resolution) and occlusion by other 3D objects could still make it difficult to see some labels. If labels were rendered as 2D text actors then they could be displayed in high quality but they would always appear as a separate layer over 3D objects (they could appear very disconnected from the 3D glyphs). Maybe adding semi-transparent callout lines between labels and glyphs would help. It would also allow laying out labels so that they never overlap each other but still close to the 3D glyph.
+
Point picking (determining what is under the mouse cursor position) and decision about what to do with is performed in VTK using pickers and a picking manager. Picking manager introduced to allow prioritization of events between multiple widgets. However, this solution was quite complicated (it hijacked events in each picker and ran it through the picking manager to decide if the widget may process the event) and due to its distributed nature, difficult to control. In Slicer widgets, each displayable manager can be queried if it can process a particular event (CanProcessEvent), returning a "distance" value. The displayable manager with the closest distance gets the right to process an event. Within a displayable manager, various flexible strategies can be implemented to decide which particular widget may process the event (taking into account the currently active widget, the interaction state of the application, etc.). This centralized solution allows reducing the number of picking operations, for example the interactor style can do a quick approximate picking using the Z buffer, then displayable managers may decide to use a more accurate picker (e.g., vtkCellPicker) and use the result in all the widgets that they manage.
  
−
Persistence: it should be possible to save all annotations in the scene (without storage node), to avoid having lots of tiny files (with just a few numbers in each). On the other hand, it would be nice to be able to save annotations in a standard file format and not just in MRML scene. It would be nice to be able to save group of markups to a single file, but unfortunately storing multiple nodes in a single file is problematic. We could add importers/exporters for standard annotation file formats (DICOM, AIM, ...).
+
In the old markups module, labels were rendered as polygons in 3D. This made their appearance very low quality (since anti-aliasing is problematic) and they often very hard to see because they are partially occluded by various 3D objects. We considered rendering the text in a texture and display as a semi-transparent rectangle by a 3D actor, but image quality still would have been an issue (texture resolution would need to match screen size and resolution) and occlusion by other 3D objects could still would have made it difficult to see some labels. Therefore, we decided to render labels as 2D text actors using existing VTK mappers. 2D actors can be displayed in high quality but they appear as a separate layer over 3D objects (they could appear very disconnected from the 3D glyphs), which is especially confusing when the markup point is occluded by other objects. Using VTK's fast point visibility checks (based on Z buffer) and label placement mapper features, the labels can be rendered acceptably well (and with the improvement of the mapper in VTK, label display will improve in Slicer, too).
 +
 
 +
Persistence: it should be possible to save all annotations in the scene (without storage node), to avoid having lots of tiny files (with just a few numbers in each). On the other hand, it would be nice to be able to save annotations in a standard file format and not just in MRML scene. It would be nice to be able to save group of markups to a single file, but unfortunately storing multiple nodes in a single file is problematic. We could add importers/exporters for standard annotation file formats (DICOM, AIM, ...). For now, all markups still use markup fiducial's storage node
  
 
Class roles & responsibilities (https://github.com/Kitware/vtk-js/issues/682)
 
Class roles & responsibilities (https://github.com/Kitware/vtk-js/issues/682)
  
−
* WidgetState (MVC: model, VTK: source) - similar to vtkMRMLNode
+
* MRML data and display nodes (MVC: model, VTK: source) - vtkMRMLMarkupsNode & vtkMRMLMarkupsDisplayNode
 
** Stores point positions, curve interpolation type, etc.
 
** Stores point positions, curve interpolation type, etc.
 
** One instance shared between all renderers.
 
** One instance shared between all renderers.
−
** Maybe it could store some state information, such as selected or highlighted point, so that a point that is being selected or dragged can be highlighted in all views. API to update state (such as translate, scale...)
+
* MRML display node (MVC: model, VTK: source) - vtkMRMLMarkupsNode
−
** Slicer: Stored in the MRML data node
+
** Stores point positions, curve interpolation type, etc.
−
* WidgetRepresentation (MVC: view, VTK: mapper) - similar to vtkWidgetRepresentation
+
** One instance shared between all renderers.
−
** Stores actor, mapper, and display properties.
+
** One markups may appear in various views in different ways, by defining various display nodes.
−
** There must be a separate instance for each renderer.
+
* Widget (MVC: controller) - vtkSlicerAbstractWidget (similar to vtkAbstractWidget)
−
** There are different views, for example one for 3D and another one for 2D views.  
+
** It can decide if it can process a user interface event (mouse move, click, etc.)
−
** API to check if it is picked at a certain location (but no event handling, so it would be almost the state of the rendering)
+
** It can process a user interface event by translating it to a widget event and then executing that (add/delete point, jump to a point, etc.).
−
** Slicer: stored in displayable manager pipeline
+
** There is one widget per display node in each renderer.
−
* Widget (MVC: controller) - similar to vtkAbstractWidget
+
** Maintains a very simple internal state (idle, places a point, being over some component of a widget)
−
** listen to events from interactor (mouse move, click, etc.)
+
** Keeps a reference to a widget representation.
−
** Probably we need a controller instance for each renderer, but it might be shared between renderers?
+
* WidgetRepresentation (MVC: view, VTK: actor) - vtkSlicerAbstractWidgetRepresentation (similar to vtkWidgetRepresentation)
−
** checks if currently interacting with the representation (using representation API), and update model/state accordingly.
+
** VTK actor that can be added to a renderer.
−
** Slicer: stored in displayable manager pipeline
+
** It can tell if an event position is near some component of the widget.
 +
** Stores VTK rendering pipeline (actors, mappers, filters).
 +
** There is one widget representation per widget.
 +
** There are different representation for 2D and 3D views.
  
−
== Planned features ==
+
== Features ==
  
 
Widget types:
 
Widget types:
−
* point set
+
* point set: DONE
−
* line (arrow, ruler)
+
* angle: DONE
 +
* curve (open and closed, with different interpolations): DONE
 +
* line (arrow, ruler): partially done (it is still just a simple line, no length or arrow tip is displayed)
 
* ROI (may be 2D only: ellipse, circle, rectangle)
 
* ROI (may be 2D only: ellipse, circle, rectangle)
 
* bidimensional
 
* bidimensional
−
* angle
 
−
* curve (open and closed, with different interpolations)
 
 
* plane
 
* plane
 
* slice intersection (show slice intersections and allow shift/rotate them), crosshair
 
* slice intersection (show slice intersections and allow shift/rotate them), crosshair
Line 51: Line 56:
 
* ROI-based image window width/level
 
* ROI-based image window width/level
  
−
Proposed Improvements:
+
Functions:
−
* Restricting placement and motion to other nodes (model surface, volume rendered image surface, etc.)
+
* Highlight/display info when hovering over: DONE
 +
* Delete using keyboard shortcut or right-click: DONE (hover over a point and click DEL key)
 +
* Select entire markup or just a point: DONE
 +
* Restricting placement and motion to other nodes (model surface, volume rendered image surface, etc.): partially done - point picking and movement in 3D view is restricted to visible surfaces (work well for volume rendered surfaces, too)
 
* One click 3D angle, length measurements from landmarks (latter is already available)
 
* One click 3D angle, length measurements from landmarks (latter is already available)
−
* Read a template for fiducial labels to be populated from
+
* Allow adding fiducials with undefined position to allow creating a template with only fiducial labels
−
* Allow adding fiducials with undefined position  
 
 
* Fine control for fiducial placement/modifications by keyboard strokes
 
* Fine control for fiducial placement/modifications by keyboard strokes
 
* Improved scaling of 3D glyphs and 2D points. For example, fixed in screen size, fixed relative to voxel size of the data set, etc.
 
* Improved scaling of 3D glyphs and 2D points. For example, fixed in screen size, fixed relative to voxel size of the data set, etc.
 
* Projection display in slice views (similar to model projections, markup point projection)
 
* Projection display in slice views (similar to model projections, markup point projection)
−
* Select entire markup or just a point
 
−
* Delete using keyboard shortcut or right-click
 
 
* Right-click menu (delete, jump to slice, fit slice views to main axes)
 
* Right-click menu (delete, jump to slice, fit slice views to main axes)
−
* Highlight/display info when hovering over
 
 
* Compute metrics (length, surface, diameter, etc.), allow them to be exported into table or other standard annotation file formats
 
* Compute metrics (length, surface, diameter, etc.), allow them to be exported into table or other standard annotation file formats
 
* Undo modifications
 
* Undo modifications
 
* Sensible behavior when non-linear transform is applied
 
* Sensible behavior when non-linear transform is applied
 +
 +
== Status ==
 +
 +
Since there are many small issues and ideas, we do not yet track the items using the Slicer bugtracker but use this spreadsheet:
 +
https://1drv.ms/x/s!Arm_AFxB9yqHtvxAgdAfdLS_GHsOxw
  
 
== References ==
 
== References ==

Revision as of 19:18, 4 March 2019

Home < Documentation < Labs < Improving Markups

Overview

The goal of this page is to support improvements to the Markups Module for fiducial placement. This page will document improvements suggested by users and developers and coordinate efforts to address them.

History

Markups, also known as annotations (and almost the same as VTK widgets) proved to be very difficult to design well. First-generation VTK widgets were simple but too monolithic, their behavior and appearance was not customizable enough. Second-generation VTK widgets took a completely new approach, separated "widget" from "representation" allowing much more customization, but also resulting in very complex widgets, with very poor performance in some cases (mainly because many classes are used to implement simple widgets and there is a complex and costly synchronization of states between them). Widgets in Slicer are similarly problematic. First attempt resulted in "Annotations" module, which was very complex to use and develop and also very unstable and so most widgets had to be disabled and only ruler (line) and ROI widgets remained. "Markups" module was the second attempt. The project ran out of time and only markup fiducials were implemented. Implementation is complicated, which makes adding new features very difficult, it has serious issues (sometimes points cannot be clicked or hidden points can be clicked), and it becomes unusably slowe after placing 100-200 points. There have been additional attempts to rework widgets, but so far they have all failed. We don't give up, so here we are again.

Design considerations

Data synchronization between multiple classes (VTK and MRML nodes, instances for each viewer, widget and representation) has proven to be one of the fundamental issues. It would have been nice to create generic VTK-only widgets and use them in Slicer, but the additional layer of abstraction would have made implementation more complicated and degraded performance. Therefore, we do not use VTK widgets and implement Slicer-specific classes, which directly use information stored in MRML nodes.

Separation between "widget" (behavior) and "representation" (appearance) is a good concept, but in second-generation VTK widgets, widgets and representations are too much separated (they don't access each other's state but communicate with state transitions and events). This scheme may be useful for client-server visualization, but it is inefficient and too complicated to use within a simple application. Therefore, we need to split classes in a way that minimizes need for manual synchronization. Separation of "data" (shared globally in the whole scene, such as widget point positions) must be separated from "display" (properties that control how a widget should appear in each view) works very well for displayable MRML nodes and chosen to be used for widgets as well. Data adds thin wrapper over VTK data object (e.g., vtkPointSet), which allows the same data object used in multiple renderers and alleviate any need for synchronizing point lists. State information (even low-level detail, such as markup that the user is currently hovered over) is stored in markup data and display nodes.

Performance of markups module was very slow because each clickable point was implemented by a handle widget. Each handle widget requires instantiation of several classes per views (widget, representation, actors, pickers, etc.), many view observations, and trigger various expensive synchronization mechanisms when anything is changed. Instead, a single glyph actor is created to display all markups.

Point picking (determining what is under the mouse cursor position) and decision about what to do with is performed in VTK using pickers and a picking manager. Picking manager introduced to allow prioritization of events between multiple widgets. However, this solution was quite complicated (it hijacked events in each picker and ran it through the picking manager to decide if the widget may process the event) and due to its distributed nature, difficult to control. In Slicer widgets, each displayable manager can be queried if it can process a particular event (CanProcessEvent), returning a "distance" value. The displayable manager with the closest distance gets the right to process an event. Within a displayable manager, various flexible strategies can be implemented to decide which particular widget may process the event (taking into account the currently active widget, the interaction state of the application, etc.). This centralized solution allows reducing the number of picking operations, for example the interactor style can do a quick approximate picking using the Z buffer, then displayable managers may decide to use a more accurate picker (e.g., vtkCellPicker) and use the result in all the widgets that they manage.

In the old markups module, labels were rendered as polygons in 3D. This made their appearance very low quality (since anti-aliasing is problematic) and they often very hard to see because they are partially occluded by various 3D objects. We considered rendering the text in a texture and display as a semi-transparent rectangle by a 3D actor, but image quality still would have been an issue (texture resolution would need to match screen size and resolution) and occlusion by other 3D objects could still would have made it difficult to see some labels. Therefore, we decided to render labels as 2D text actors using existing VTK mappers. 2D actors can be displayed in high quality but they appear as a separate layer over 3D objects (they could appear very disconnected from the 3D glyphs), which is especially confusing when the markup point is occluded by other objects. Using VTK's fast point visibility checks (based on Z buffer) and label placement mapper features, the labels can be rendered acceptably well (and with the improvement of the mapper in VTK, label display will improve in Slicer, too).

Persistence: it should be possible to save all annotations in the scene (without storage node), to avoid having lots of tiny files (with just a few numbers in each). On the other hand, it would be nice to be able to save annotations in a standard file format and not just in MRML scene. It would be nice to be able to save group of markups to a single file, but unfortunately storing multiple nodes in a single file is problematic. We could add importers/exporters for standard annotation file formats (DICOM, AIM, ...). For now, all markups still use markup fiducial's storage node

Class roles & responsibilities (https://github.com/Kitware/vtk-js/issues/682)

  • MRML data and display nodes (MVC: model, VTK: source) - vtkMRMLMarkupsNode & vtkMRMLMarkupsDisplayNode
    • Stores point positions, curve interpolation type, etc.
    • One instance shared between all renderers.
  • MRML display node (MVC: model, VTK: source) - vtkMRMLMarkupsNode
    • Stores point positions, curve interpolation type, etc.
    • One instance shared between all renderers.
    • One markups may appear in various views in different ways, by defining various display nodes.
  • Widget (MVC: controller) - vtkSlicerAbstractWidget (similar to vtkAbstractWidget)
    • It can decide if it can process a user interface event (mouse move, click, etc.)
    • It can process a user interface event by translating it to a widget event and then executing that (add/delete point, jump to a point, etc.).
    • There is one widget per display node in each renderer.
    • Maintains a very simple internal state (idle, places a point, being over some component of a widget)
    • Keeps a reference to a widget representation.
  • WidgetRepresentation (MVC: view, VTK: actor) - vtkSlicerAbstractWidgetRepresentation (similar to vtkWidgetRepresentation)
    • VTK actor that can be added to a renderer.
    • It can tell if an event position is near some component of the widget.
    • Stores VTK rendering pipeline (actors, mappers, filters).
    • There is one widget representation per widget.
    • There are different representation for 2D and 3D views.

Features

Widget types:

  • point set: DONE
  • angle: DONE
  • curve (open and closed, with different interpolations): DONE
  • line (arrow, ruler): partially done (it is still just a simple line, no length or arrow tip is displayed)
  • ROI (may be 2D only: ellipse, circle, rectangle)
  • bidimensional
  • plane
  • slice intersection (show slice intersections and allow shift/rotate them), crosshair
  • linear transforms (2D: similar to vtkAffineWidget; 3D: current transform widget)
  • ROI-based image window width/level

Functions:

  • Highlight/display info when hovering over: DONE
  • Delete using keyboard shortcut or right-click: DONE (hover over a point and click DEL key)
  • Select entire markup or just a point: DONE
  • Restricting placement and motion to other nodes (model surface, volume rendered image surface, etc.): partially done - point picking and movement in 3D view is restricted to visible surfaces (work well for volume rendered surfaces, too)
  • One click 3D angle, length measurements from landmarks (latter is already available)
  • Allow adding fiducials with undefined position to allow creating a template with only fiducial labels
  • Fine control for fiducial placement/modifications by keyboard strokes
  • Improved scaling of 3D glyphs and 2D points. For example, fixed in screen size, fixed relative to voxel size of the data set, etc.
  • Projection display in slice views (similar to model projections, markup point projection)
  • Right-click menu (delete, jump to slice, fit slice views to main axes)
  • Compute metrics (length, surface, diameter, etc.), allow them to be exported into table or other standard annotation file formats
  • Undo modifications
  • Sensible behavior when non-linear transform is applied

Status

Since there are many small issues and ideas, we do not yet track the items using the Slicer bugtracker but use this spreadsheet: https://1drv.ms/x/s!Arm_AFxB9yqHtvxAgdAfdLS_GHsOxw

References

Retrieved from "https://www.slicer.org/w/index.php?title=Documentation/Labs/Improving_Markups&oldid=60820"