Slicer3:XCEDE use cases

From Slicer Wiki
Jump to: navigation, search
Home < Slicer3:XCEDE use cases

Slicer3 use cases for XCEDE

Slicer use-case #1: Managing image processing project in a Neuroscience Lab

Overview

  • Step0: Investigator creates a new project on the hosted repository
  • Step1: Investigator uploads data from a study and associates it with the project.
  • Step2: Later, the investigator queries for all “Acquisitions” uploaded as part of the study.
  • Step3: The investigator then assigns each acquisition to a researcher for a segmentation task.
  • Step4: A researcher queries for all acquisition segmentations assigned to them.
  • Step5: A researcher downloads some subset of those assignments in an XCEDE catalog, opens the catalog with Slicer and creates a label map.
  • Step6: When finished, a researcher uploads completed work or work in progress and tags the work appropriately.

Notes

For example, we’ll use the public OASIS brains dataset on XNAT, and let “composite ID” for a visit be OAS1_0006_MR1, or OAS1_0009_MR1, etc.

  • Each visit (such as OAS1_0006_MR1) may contain multiple acquisitions, assessments, and derived data.
  • Each element's ID is made unique by the back end, so that each visit, acquisition, subject, study, etc. has a locally unique ID.
  • CompositeIDs are made globally unique by appending the uri of the hosted repository.
  • Each element can also have a user-assigned label without a uniquenss requirement (like scan001), which may be a more comfortable way for the user to refer to it .

There are different ways in which an ID might be generated for an element.

  • user may supply a preferred placeholder ID to which the back end applies a uniqueness check. The requested ID is used if unique.
  • no placeholder ID is supplied and the system just generates a unique ID.
  • system-generated ID is returned to user, who can accept or reject that ID.

Assuming also that PUT and POST are swapped on the spreadsheet Jeff sent last week. Below, we use the following:

  • PUT = CREATE NEW
  • POST = UPDATE EXISTING

Stepping thru details


Step0. An investigator creates a new project in the hosted repository: 

(not sure how to do this on command line -- should discuss this.)

Probably done through web browser interface, but we should allow programmatic control at some point.



Step1. An investigator uploads all tagged data from a study (using an upload script, or a webGUI or something outside of Slicer) and associates it with the new project – for instance: 

curl –X PUT -T localFile –u user:passwd http://central.xnat.org/CENTRAL_OASIS_CS/OAS1_0009_MR1/uploadFileName 
or
curl -X PUT -T localFile -u user:passwd http://central.xnat.org?project=CENTRAL_OASIS_CS&visit=OAS1_0009_MRI
or 
upload file plus XML file that associates the file, project and visit.

Response: Script looks for return value, parses XCEDE for the ID of each uploaded image, or parses error XML and posts message if upload failed.

Note: data could also come from web browser GUI or through DICOM transfer.


Step2. Investigator queries for all visits uploaded as part of the study, tagged with him as the investigator and with a particular protocol, which have not yet been assigned for segmentation. Maybe this is done via web form, maybe with a script or query tool inside Slicer. 

curl –X GET http://central.xnat.org/visits?project=CENTRAL_OASIS_CS&investigator=Marek%20Kubicki&protocol=fBIRN&technician=NULL


Response: Receives back XCEDE XML describing list of uris for appropriate visits and return code; If return code is good, parse XML to get list of uris and display them using some tool or another. Gets a list like:

http://central.xnat.org/visit/OAS1_0006_MR1

http://central.xnat.org/visit/OAS1_0009_MR1

http://central.xnat.org/visit/OAS1_0054_MR1

…

http://central.xnat.org/visit/OAS1_0450_MR1



Step3. Investigator updates the metadata with some segmentation assignments. 

curl –X POST http://central.xnat.org/visit/OAS1_0006_MR1?technician=Tech1&status=assigned 
curl –X POST http://central.xnat.org/visit/OAS1_0009_MR1?technician=Tech2&status=assigned 
curl –X POST http://central.xnat.org/visit/OAS1_0054_MR1?technician=Tech2&status=assigned
…
curl -X POST http://central.xnat.org/visit/OAS1_0450_MR1?technician=Tech2&status=assigned

Response: Probably only need an integer back that flags success/failure.



Step4. Technician “Tech2” queries for all of their assignments from this particular investigator. 

curl –X GET http://central.xnat.org/visits?technician=Tech2&investigator=Marek%20Kubicki&status=assigned

Response: Receives back XCEDE XML describing list of uris for visits containing data they need to segment. If return code is good, parse XML to get list of uris and display them using some tool or another.



Step5. Technician downloads an XCEDE catalog describing a single visit 

curl –X GET http://central.xnat.org/visit/OAS1_0009_MR1?view=XCAT
or
curl -X GET http://central.xnat.org/visit/OAS1_0009_MR1?view=XCEDE

Or, technician downloads all of their assigned visits: 

curl –X GET http://central.xnat.org/visits?project=CENTRAL_OASIS_CS&technician=Tech2&view=XCAT

QUESTION: how do we form the commandline for multiple visits? QUESTION: are multiple XCEDE catalogs returned, or one composite catalog?

Technician opens the catalog in Slicer and works on the segmentation.

Response: If return code is not good, post error.



Step6. At the day’s end, he wants to upload his work in progress with updated status

QUESTION: doing a ‘post’ will presumably overwrite the original data (if it were present in the DB). Back end will not do any automatic versioning: decisions about whether to overwrite, or create versions are made project-by-project on the client side.

Post a file that already has a uri on the DB, retrieved from the XCAT: 

curl –X POST {uri}?status=in%20progress

Put a newly created file: 

curl –X PUT -T updated-aseg.mgz http://central.xnat.org/visit/OAS1_0009_MR1/{path}/aseg.mgz?  {all tags from the visit?}  &status=completed

QUESTION: Provenance-- how to represent. Could associate provenance information in these ways:

  • add XCEDE Provenance XML into the data header and perform a single PUT operation,
  • PUT the data file, then POST an update to metadata using the ID returned from the first PUT.
  • PUT the data file, and then PUT a separate Provenance XML file that references the ID returned from the first PUT.

Response: Get XCEDE back. If data was POSTED, we get an integer or ID back that indicates success/failure; if data was PUT, then we get back XCEDE XML with new ID.


Slicer use-case #2: Query Atlas Module

Contents:

This is a record of the way we're using the xcede2.0 catalog in order to load in a query atlas scene that contains:

  • One FreeSurfer "lh" model and one FreeSurfer "rh" model at most
  • the FreeSurfer structural brain.mgz image volume
  • the FreeSurfer example functional example_func.nii
  • some FIPS-generated Statistical volumes
  • the FreeSurfer aparc+aseg.mgz label map volume
  • the FreeSurfer lh.aparc.annot annotation file
  • the matrix to align the example_func + statistics to the brain.mgz: anat2exf.register.dat

Assumptions:

  • we are assuming a single lh or rh model (or one of each) is in the catalog. (Reason: Slicer will load the lh.aparc.annot file as a scalar overlay onto a model; so the overlay needs to be associated with a model automatically during load. Since the catalog attribute list contains no references to other datasets, the only way to associate the lh.aparc.annot to the lh.pial surface is by matching on the "lh." and "rh." in the model and annotation names.)
  • we are detecting whether a matrix with the name "anat2efx" is in the catalog (we should probably check the uri instead) -- and if so, we are wrapping the list of all FIPS-generated statistical volumes and the example_func(noted above) in a derived transform in the load routine. We recognize files with the string "brain.mgz" "example_func" and "anat2exf" in them, in order to do some automatic calculation of the Slicer transform that does the Slicer ras2ras registration. Then we stuff volumes with "example_func" or "stat" in their name into the derived transform.
    • the tests to detect these files are weak -- need a better test: now, a structural volume with the string 'stat' embedded in it will be stuffed into the transform by mistake.
    • no registration transform is computed if volumes matching the brain.mgz, example_func and an anat2exf.register.dat file are not detected.

New value inventions for attributes:

  • Freesurfer:matrix-1 (this gives us a clue about file format).
  • Freesurfer:overlay-1 (for files that need to be associated with a model in order to load in Slicer)
  • Freesurfer:label-1 (for mgz files that are label maps) This gives us a hint about what kind of color map to use to display the image data. The default colormap for volume data is greyscale.

Validation:

Slicer use-case #3: Qdec Module

Description needed...

Prototype XCAT Files

<XCEDE>
<catalog ID="ID0">

<catalogList>

<catalog ID="ID1">
<entryList>

<entry ID="ID2" name="anat2exf" description="matrix that registers anatomical information to functional datasets" format="FreeSurfer:matrix-1" content="anat2exf" cachePath="" modelID="ID2" uri="./anat2exf.register.dat"/>

<entry ID="ID3" name="lh.pial" description="pial surface of left hemisphere" format="FreeSurfer:surface-1" content="lh.pial" cachePath="" uri="./lh.pial"/>

<entry ID="ID4" name="rh.pial" description="pial surface of right hemisphere" format="FreeSurfer:surface-1" content="rh.pial" cachePath="" uri="./rh.pial"/>

<entry ID="ID5" name="brain" description="extracted brain mri" format="FreeSurfer:mgz-1" content="brain" cachePath="" uri="./brain.mgz"/>

<entry ID="ID6" name="exf" description="example functional dataset" format="nifti:nii-1" content="example_func" cachePath="" uri="./example_func.nii"/>

<entry ID="ID7" name="zstat7" description="7th zstatisic contrast E1+E3+E5>Fix" format="nifti:nii-1" content="zstat7" cachePath="" uri="./zstat7.nii"/>

<entry ID="ID8" name="zstat14" description="14th zstatisic contrast P1+P3+P5>Fix" format="nifti:nii-1" content="zstat14" cachePath="" uri="./zstat14.nii"/>

<entry ID="ID9" name="zstat17" description="17th zstatisic contrast Learn>Fix" format="nifti:nii-1" content="zstat17" cachePath="" uri="./zstat17.nii"/>

<entry ID="ID10" name="aparc+aseg" description="parcellation and segmentation label map" labelmap="true" format="FreeSurfer:mgz-1" content="aparc+aseg" cachePath="" uri="./aparc+aseg.mgz"/>

<entry ID="ID11" name="lh.aparc.annot" description="annotations for surface of left hemisphere" format="FreeSurfer:overlay-1" content="lh.aparc.annot" cachePath="" uri="./lh.aparc.annot"/>

<entry ID="ID12" name="rh.aparc.annot" description="annotations for surface of right hemisphere" format="FreeSurfer:overlay-1" content="rh.aparc.annot" cachePath="" uri="./rh.aparc.annot"/>

</entryList>
</catalog>

</catalogList>

</catalog>
</XCEDE>

XCEDE REST Interface

A straw man REST interface to XCEDE. The implementation is using Grails (Groovy on Rails). Grails is a rapid application development environment patterned after Rails (Ruby on Rails). Grails, however, is completely Java-based utilizing industry standard components (Spring, Hibernate, Groovy). In addition, Grails supports many of the common AJAX frameworks and can be compiled and deployed as a .war file.

REST Interaction

REST support four basic operations akin to CRUD, Create, Read, Update and Delete. These verbs operate on resources (nouns) exposed as URIs, with CRUD relating to HTTP operations:

HTTP CRUD
POST Create, Update, Delete
GET Read
PUT Create, Update
DELETE Delete

A server has been set up on slicerl using code in the NAMICSandbox (see next section for details). Two interfaces are available, a human-friendly web site and a machine-friendly REST interface. The command line program curl can interact with the REST interface, and is useful for learning and debugging.

Retrieve a list of available Projects 

7: curl -X GET http://slicerl.bwh.harvard.edu:8080/XCEDE/rest/v1/project/
<?xml version="1.0" encoding="UTF-8"?><list>
  <project id="1">
    <createdDateTime>2008-06-18 21:36:50.291</createdDateTime>
    <description>Neurological implications of working at BWH</description>
    <name>BWH-0001</name>
    <subjectGroups>
      <subjectGroup id="1"/>
    </subjectGroups>
  </project>
  <project id="2">
    <createdDateTime>2008-06-18 21:36:50.332</createdDateTime>
    <description>Bar</description>
    <name>Foo</name>
    <subjectGroups>
      <subjectGroup id="2"/>
    </subjectGroups>
  </project>
</list>

Update an existing Project (Note how Project 2 has been changed) 

8: curl -X GET http://slicerl.bwh.harvard.edu:8080/XCEDE/rest/v1/project/2 > Project2.xml
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   242    0   242    0     0    802      0 --:--:-- --:--:-- --:--:--     0

9: cat Project2.xml 
<?xml version="1.0" encoding="UTF-8"?><project id="2">
  <createdDateTime>2008-06-18 21:36:50.332</createdDateTime>
  <description>Bar</description>
  <name>Foo</name>
  <subjectGroups>
    <subjectGroup id="2"/>
  </subjectGroups>

11: perl -e 's/Bar/Garf/g' -pi Project2.xml

12: curl -X PUT -H 'Content-Type: application/xml' -d @Project2.xml http://slicerl.bwh.harvard.edu:8080/XCEDE/rest/v1/project/2
<?xml version="1.0" encoding="UTF-8"?><project id="2">
  <createdDateTime>2008-06-18 21:36:50.332</createdDateTime>
  <description>Garf</description>
  <name>Foo</name>
  <subjectGroups>
    <null/>
  </subjectGroups>
</project>

Create a new project (Note using POST creates a new project, while PUT updates an existing project, this command created a project with id=3) 

15:  curl -X POST -H 'Content-Type: application/xml' -d @Project2.xml http://slicerl.bwh.harvard.edu:8080/XCEDE/rest/v1/project/
<?xml version="1.0" encoding="UTF-8"?><project id="3">
  <createdDateTime>2008-06-18 21:43:04.790 EDT</createdDateTime>
  <description>Garf</description>
  <name>Foo</name>
  <subjectGroups>
    <null/>
  </subjectGroups>
</project>

Delete a project 

17: curl -X DELETE http://slicerl.bwh.harvard.edu:8080/XCEDE/rest/v1/project/3
Grails/XCEDE development

So, you're ready to kick the tires? Great! The first step is to check out the code from Subversion (make sure you do the second step!). 

svn co http://www.na-mic.org/svn/NAMICSandBox/trunk/XCEDE

Resources that come with the Grails binary installation are omitted from Subversion and must be generated. 

cd XCEDE
grails upgrade

and answer "yes" to both questions.

The main code is in XCEDE/grails-app/controllers/. Grails uses an MVC model and "coding by convention". Thus, if code is put in the correct place, it works without configuration. This is handy if you know what you are doing, frustrating if not. I would suggest going through "Getting Started with Grails" as a first tutorial.

Retrieved from "https://www.slicer.org/w/index.php?title=Slicer3:XCEDE_use_cases&oldid=5440"