Slicer3:Coordinate System Manager

Basic idea

The idea for the Slicer Coordinate System Manager (SCSM) is to make it much easier for developers and users to describe the relationship between coordinate spaces for the data they need to manipulate. We believe that the framework we are developing will be application- and toolkit-independent, flexible enought to grow with future needs, and intuitive enough to solve some of the most irritating problems with regard to transforms and coordinate systems.

Although it's being done as part of the Slicer3 effort, we believe the Coordinate Space Manager will be useful, and usable, in a variety of other applications.

We are also actively working to extend the concept of the Coordinate Space Manager to describe other aspects of rendering, visualization, and data description in Slicer3.

Motivation

One of the most difficult elements of computer graphics, robotics, or medical scene description is the process of relating the coordinate systems of multiple objects or data sets. For even the most experienced programmers or researchers, managing coordinate systems is confusing and error-prone.

Hierarchies

Scene description file formats such as VRML and Slicer's MRML offer a framework for describing objects and the coordinate systems they live in. The hierarchy-based transform model of VRML comes from the CAD world, where nested coordinate systems have proven effective for describing mechanical assemblies.

However, medical data (and many other kinds of data, too) is often not best described by a transform hierarchy. A complex multi-modality medical data set might be arranged hierarchically, but the structure might be oriented around, say, anatomical hierarchy, or a functional model, or a blood flow model, or a radiation exposure model, or a variety of other classifications. One application might prefer one kind of hierarchical description, while a second might require another. A user might even want to switch between different hierarchies. In any of these cases, a hierarchy based strictly on geometric transforms significantly limits the richness of scene description.

Naming and transform description

The current state of the art of transform specification also has its shortcomings for today's applications. To relate the coordinate systems of two different spaces, two mechanisms are required: a way to specify the coordinate spaces (naming), and a way to describe the transform (transform description).

Let's begin with naming. In a single hierarchy model like VRML or MRML, naming is generally implicit: a transform relates its parent to its child. In a programming model in such systems as ITK or VTK or Slicer, transforms relate a fixed number of coordinate systems together. These coordinate systems are given standard names like "IJK", "Index", "RAS", or "LPS", and geometric transform describe how to move between them.

The problem with this naming model is its rigidity: it forces programs and people to use conventions that may or may not be appropriate for their application. Is the RAS space appropriate for cardiology, or pancreatic surgery? Maybe, maybe not. But shouldn't the developer or user get to decide? The standard set of coordinate spaces might also not include new spaces required by emerging applications. For example, DTMRI depends on the known orientation of the MRI magnet's gradient coils with respect to the patient. The description of this relationship is beyond the abilities of a fixed set of coordinate systems.

Finally, the standard coordinate system set is usually domain specific, and therefore limits the ability to move from software from one area (e.g., medicine) to another where the same tools might be useful (astronomy). Galaxies don't have noses to orient the RAS coordinate system: thus, an application that depends on an RAS-based coordinate system is unnecessarily inflexible in accommodating new data.

This problem of inflexibility due to assumptions about application domain is a major factor as we move from naming to transform description. CAD transforms, and the transforms in Slicer, are traditionally affine transforms described in matrix form. This model is reasonably compact and easily interpreted. However, it is too simple for many tasks in medicine (ones that require deformation fields are a good example), and it is too general for other applications (the matrix form loses the information about the specification of the transform, such as rotation and scale). Other application areas might require exotic or esoteric transforms that a matrix just can't provide: astronomy, for instance, has transforms that require the time of measurement as a parameter.

Shoehorning a fixed transform specification model into all applications is a recipe for short-lived code. More broadly, there is no way to anticipate the kinds of transforms and transform descriptions that might be useful for a particular application in the future. In order to produce a coordinate space management system that can grow to include new applications in the future, then, the coordinate space naming mechanism must be flexible, and the transform specification mechanism must be extensible (or, ideally, completely separate from the naming system.

Slicer3 Coordinate Space Manager advantages

The Slicer3 Coordinate Space Manager is designed to provide both flexible naming and independent transform specification:

• Users specify coordinate space names. New names can be added at any time. Names are represented hierarchically, but the hierarchy is purely for organizational purposes (much like directories or folders in a file system). A coordinate space can be referred to by several different names for convenience.

• The transforms that relate coordinate spaces are specified in a completely separate way from the naming mechanism. An application using the coordinate space naming system could choose to use ITK transforms, or VTK transforms, or astrophysics-specific transforms, or its own transforms.

In addition, a general transform composition mechanism makes it easy to ask the following question: "I know where I am in this space. Where am I in that space?" In traditional systems, this question requires the user or programmer to manually compose the transform path between two coordinate spaces, passing through any intermediate ones. This process, as mentioned above, is one of the most confusing and error-prone parts of computer graphics and related fields.

The Slicer Coordinate Space Manager automatically composes the transforms that relate different coordinate spaces: the programmer or user just has to specify the two coordinate space name endpoints.

How SCSM works, by example

SCSM names coordinate spaces like you'd name files in folders. Let's say you're working with an CT scan. If the scan is in DICOM format, it provides two coordinate space definitions: an "index" coordinate system defined by the voxels/samples in the file, and an "LPS" coordinate system defined the patient's orientation and the physical size of the region scanned. Information in the DICOM file relates the two spaces: you can use this data to form a transform that converts a location and orientation in one space to the other.

Similarly, you can have an MRI scan contained in a DICOM file. This scan also has "index" and "LPS" coordinate spaces, but in addition, it contains information about the gradient coil orientation with respect to the patient when the data was acquired. This gradient coil orientation can thus be related to the other two spaces.

These two scans, the CT and MRI for a single patient, are represented in two sets of uncorrelated coordinate spaces: since the scans happened at two different times in two different scanners, we don't know the relationship between the sets of coordinate spaces. However, we can perform complete operations using each scan individually, since we have transforms to associate the different spaces for each scan.

In SCSM, the coordinate spaces for each scan is represented by the root of a tree, much like "/" is the root of a directory tree in POSIX-like file systems. The CT coordinate space tree has two nodes, named "index" and "LPS". The MRI tree has three: "index", "LPS", and "gradient_coil". These named nodes are hangers or reference points for the coordinate spaces we're interested in; the names themselves are arbitrary.

Each tree also contains "transform links" that tie together different spaces. A transform link defines a transform relationship between two named nodes in the tree. A transform link contains several pieces of information: the names of the two related nodes (a "from node" and a "to node"), a reference to a transform object that transforms a point from the "from node" space into the "to node" space, a flag that specifies whether the transform (and therefore the link) is invertible, and perhaps a name for the transform link so that it can be manipulated later.

In the example above, the CT scan would define an invertible transform link between index and LPS space. The MRI scan would define two invertible transform links: one from index to LPS, the other from gradient_coil to LPS.

Now, let's run an registration algorithm on the CT and MRI scan data. The output of the registration process is a transform that relates an MRI coordinate space (perhaps the LPS space) and a CT coordinate space (let's say the CT LPS space). SCSM lets us relate these two spaces using the registration-derived transform. We create a new tree that refers to the patient. We then "attach" the CT coordinate space hierarchy to the name "CT", and the MRI coordinate space hierarchy to the name "MRI". We then create an invertible transform link between the two spaces, linking "/MRI/LPS" to "/CT/LPS" using the registration-based transform.

Now, in addition to being able to transform between the two LPS coordinate spaces, we actually have enough information to go between any two coordinate spaces, since all the relationships have been defined. Using a traditional transform system, you'd be in charge of concatenating the transforms between the two spaces you're interested in to find the appropriate composite transform. SCSM takes care of this step for you: using a graph traversal algorithm, it finds the shortest path between two named coordinate spaces and produces the composite transform.

SCSM allows the relationships between potentially large hierarchies of coordinate spaces to be described easily. In this example above, multiple patients could be registered together, or registered to a common atlas coordinate system. Data from multiple scans of a patient could be associated. To add each new set of coordinate spaces, often only a single new transform link is required to allow transforms from any to any space.

Modularity

Using the example above, let's look at what's actually happening. Given the named spaces and the transform links, SCSM traverses the graph from the beginning space to the end accumulating transforms. It actually doesn't need to know anything about the transforms; it just knows to accumulate them in order from start to finish. If there's no path, the SCSM reports that. If there is a path, the manager reports the list of transforms. SCSM doesn't have to know how to transform anything.

Similarly, when the transform is specified, the transform definition isn't dependent on SCSM at all, really. When the description of the transform network is specified, SCSM just hands the description of the transform to the client code, and asks for an opaque object back. When SCSM traverses the transform graph, it accumulates these opaque objects, and hands them back as a list to the client. It's up to the client to interpret the results (say, concatenate the list into a composite transform).

Clients of the SCSM, then, can specify their own transform definitions and actual transform code to get whatever transform behavior they need. SCSM takes care of managing and traversing the network, which is a much smaller piece of code, much easier to agree on, and much less likely to change over time.

Example

<?xml version="1.0"?>
<mrml:MRML
    xmlns:mrml="http://www.slicer.org/schemas/2006/03/mrml/core#"
    xmlns:cspace="http://www.slicer.org/schemas/2006/03/mrml/cspace#"
>
  <mrml:CoordinateSpace id="cspace1">
    <cspace:CoordinateSpace path="ct">
        <cspace:CoordinateSpace path="ijk" />
        <cspace:CoordinateSpace path="ras" />
    </cspace:CoordinateSpace>

    <cspace:CoordinateSpace path="mri">
        <cspace:CoordinateSpace path="gradient_coil" />
        <cspace:CoordinateSpace path="ras" />
        <cspace:CoordinateSpace path="ijk" />
    </cspace:CoordinateSpace>

    <cspace:TransformLink
          cspace:from="ct/ijk"
          cspace:to="mri/ras"
          cspace:transform="ct_to_mri"
          cspace:invertible="true" />

     <vtk:Matrix4x4 path="ct_to_mri">
        1 0 0 0 0 1 0 0 0 0 1 0 0 0 0 1
     </vtk:Matrix4x4>

     <vtk:Matrix4x4 path="another_transform"
         ref="http://www.slicer.org/file.mrml#transforms/t1" />

  </mrml:CoordinateSpace>
<mrml:MRML>

Extensions

We are investigating extending the naming mechanism of SCSM to describe other elements of "scenes", including geometry, data volumes, rendering styles, and semantic information.

Implementation status

Michael Halle has implemented a prototype SCSM based on VTK and written in Python. He will work with Luis on a C++ implementation. Luis has already mocked up a skeleton in the NA-MIC sandbox.