Slicer3:UIDesign:WorkingProblems:SlicerInformatics:Draft1

Discussion at the NA-MIC 2009 AHM in Salt Lake City

Planning XNAT and Slicer integration next steps

Rough timeline for the next several months

Discussion points and action items

Deleting:

• resources (Use http delete),
• tags on resources, and
• tags from whole repository

Kevin will email wendy the API for the last two of these.

QA: detecting whether uploaded data is valid. WUSTL is thinking about QA; right now no provision. Future, maybe look at checksum on server side... Maybe there's a short-term soln: API provision for client to query for remote file size, and check this against local file size, the request file be deleted on remote host if no match...

Server ID: is there a way to validate a server as XND? We want to expose capability to 'add a new server', and a way to query the resulting servername for its identity, like

 http://xnd.slicer.org:8000/identify

Kevin will look into this and update Wendy.

Security: no security yet. A first pass will come soon, probably http basic.

Tag/value length limits: There are no intentional limits on tag/value lengths. Long names and values may slow search time.

Special character restrictions: excaping all characters as for xml/html should be sufficient.

Note: WUSTL implemented a file browser metaphor for looking at hierarchy of resources, found that intuitive for users.

Note: XND current version is challenged by enormous datasets (like large collections of DICOM). Fix to this coming in version 0.8, with "collection support".

Note: Current XND version 0.7 is compatible with Slicer's existing implementation. Slicer will have to be modified to take advantage of Version 0.8's collection support: collections may not be supported by XND FileServer until a month or two after XND 0.8 is released. Misha will let Wendy know when draft API is available and advise.

Format tag: In the event that URIs don't contain an explicit extension (like those for XNAT-E, which also encode ticket info) we need some kind of tag on the dataset that Slicer can use to know how to open/load it. Current Slicer implementation automatically generates a "SlicerDataType" tag that it uses for that purpose. Better if we change that logic to use a "Format" tag that Slicer- and non-Slicer-people use. But we need to encourage the value assignments to be as consistent as possible. Can we create an XND default tag with a pre-populated set of values that takes a good stab at this? If so, Slicer can be modified to use "Format" instead of (or in addition to) "SlicerDataType".

Annotations, Thumbnails, and other derived data: We discussed the best way to accommodate annotations, and other data (thumbnails) associated with a particular dataset or collection of datasets. We would want to be able to:

• create these 'documents or datasets',
• upload them and associate them with a resource,
• query for them, and
• download them separately or bundled with data.

FetchMI release into nightly builds: This will happen soon after bug fixing and testing. Wendy will update XNAT team when module is released.

Cache Manager, Remote Data I/O ToDo List for Slicer 3.4 code freeze

• Fix Raise and rendering update behavior on CacheAndDataIOManager. When any data transfer takes place, it should raise, render itself, and then continue. After each data transfer, it should update.
• Disable Asynchronous IO option for now since that sometimes causes problems.
• Make sure "always redownload" is selected by default.
• Upload and download of MRML Scene files don't generate a DataTransfer entry -- not sure why. Only data files. Find and fix.
• Remove "Publish to XNAT" option from the File menu
• Make sure all path constructions are created with join, and not std::string "+"

FetchMI ToDo List for Slicer3.4 code freeze

• Logic & MRML
  • trim off last ampersand in query
  • make sure all query, delete, DL, UL commands exist in logic (so can be called from commandline)
  • modularize architecture for multi-services (set appropriate handler in node)
    • make tcl procs into c++ logic (done)
    • move requestDL, UL into logic (done)
    • make parser base class and derive XNDParser HIDParser, move parsing methods there (done)
    • make xml writer base class and derive XNDWriter HIDWriter, move xml writing methods there (done)
    • make xml query base class and derive XNDClientCaller HIDClientCaller, move query generation there.
  • efficiency: create a 'NewTagList' and add only new tags to DB instead of all tags at each upload.
  • escape all special characters in new tags & values before posting.
  • decode all escaped special characters in new tags
  • during a scene upload, keep track of all files uploaded so far. If something fails before MRML scene goes up, post message about what resource failed, then delete all component resources from the remote host (prevent partial upload footprint).
  • when user selects to delete a scene, query if they want to delete scenefile + all data it references. if no, query whether they just want to delete scene file alone. If opt to delete all, then download the scene, get out all refs, delete scene and refs from remote host.
  • after a scene close event, remove all tags from scene's UserTagTable
  • AllValuesForAllTagsOnServer -->ServerMetadata. (done)
  • Node --> ServerAndUserMetadata.
  • Node --> CurrentTagDefinitions.
  • Node --> CurrentTagSelections.
  • If scene is renamed, ResourceUpload widget doesn't update.
  • Restore after MID-UPLoad/Download crapout -- need to reset write or read state on all selected resources; make sure filenames point to cache files or to final/original destination. (DONE)
  • prompt user to save data locally prior to upload.
  • prior to DL; should we prompt user first to close existing scene?
• GUI
  • improve readability of current keyword/value pair in tagging panel (done)
  • put up a 'wait' message (or write text in status bar)
    • figure out what's overriding the status text when server is selected & query is made (still hmmm)
  • implement show tags for individual dataset
  • when a scene is closed, remove all tags from the empty scene's tag table.
  • when a tag is deleted from datasets, raise the tagViewer with updated info
• Overall
  • find and fix debug leaks: appears to be part of the TagTableCollection (done).
  • want to be able to download any kind of Volume or Model or MRML file. Problem: inferring what kind of node to create from the uri of a remote resource. Currently, when a filename is passed in on the command line, loader.tcl parses out the filename and sends it to the AddDataWidget's logic; this figures out what kind of nodes to create for the data. When a filename is loaded through AddVolumes, AddModel, AddTransform, etc., the node is created directly. Probably need to build logic like the AddDataWidget's logic, that infers the node type from the filename -- if the uri contains a valid filename.
  • when server metadata is refreshed, tag selection state changes on ResourceUpload widget. decouple.
  • debugging assist: beef up datatransfer tools so we can interrogate each one.
• Test
  • test: save locally, then upload.
  • test: download, local save, then upload.
  • test: download, remove a dataset, then upload.
• DebugNotes:
  • Slightly confusing lack of parallelism in logic: In a storage node, the first FileName is repeated as the first entry in FileNameList, but the first URI is *not* repeated in URIList. To get a uri that corresponds to a filename, need to do a little index jigger. Principle code changes for this are in vtkDataIOManagerLogic in Base/Logic. I'm not familiar enough with the ArchetypeStorageNode to be confident in making a change -- Maybe Nicole can address?
  • Added a modification to set up FileNameList initially when a file series is read -- (previously only getting set up in the write.) Seems ok now -- will wait for Nicole to test. Principle changes occur in vtkMRMLVolumeArchetypeStorageNode.cxx/h
• NEW
  • Fix: We are requiring a SlicerDataType tag on all storable nodes that can get uploaded to a remote server, and downloaded from a remote server. Why necessary? when datasets are encapsulated by mrml scene description and a scene is downloaded, we can use the scene context to decide what kind of storable node to create to hold the data about to be downloaded. When individual datasets come down (we want to be able to accommodate this), Slicer needs to create a storage node of a particular type, but has no way to know which (and cannot reliably infer this from the filename or extension.)
  • CHALLENGE: We are trying to do all tagging of storabel nodes from an external informatics module by looking for node created events on the scene, testing to see what the node is, and then tagging its storage node with an appropriate value for SlicerDataType. However, it's possible to miss nodes entirely. If they don't get tagged, they don't get uploaded by logic, since we won't be able to reliably download them later. First, FreeSurfer models can have scalar overlays; when these nodes are created and added to a model display node, no events are available to tell us how to tag this from inside an external informatics module. Second, fiducialList nodes are created without storage nodes if no fiducail points are immediately added, so no events triggered when user adds fiducials to a list after some time.
    • Approach: MRMLScene gets a new event MetadataAddedEvent
    • StorableNode gets new member SlicerDataType and methods to set it and invoke an event, and methods to get it. We do this in SlicerBase, instead of in an external informatics module. Values for these types should be unambiguous and semantically meaningful to someone looking at their markup..
    • Currently Supported SlicerDataTypes: MRML, ScalarVolume, LabelMap, VTKModel, FreeSurferModel, FreeSurferModelWithOverlay, DTIVolume, DWIVolume, UnstructuredGrid, FiducialList, ColorTable.
    • For now, we'll set SlicerDataType on the nodes from FetchMILogic for everyone we can; but for FreeSurferScalarOverlays, we'll set those in the nodes themselves to ensure they get set. We tried to catch them all with NodeAddedEvents posted by the scene, but not all storage nodes are created when the storable node is created, and if they are created later, they are usually added under no notify conditions.
    • NEED A CONSISTENT WAY TO DO THIS that hopefully doesn't require touching all these node classes...
  • Once all storable nodes in the scene are tagged, FetchMILogic can post them to a remote host, handling any differences in the save data pipeline for differnt types of nodes.
    • For instance: Save Data pipeline calls a storageNode's WriteData method first, which writes data to disk. If a uri is specified for the data, the call to WriteData writes to cache, and a superclass method called 'StageWriteData' writes from cache to the remote host using the informatics tools. FreeSurfer storable nodes have no writers! So there's no way to get the to write to cache;
    • Approach: vtkMRMLFreeSurferModelStorageNode and vtkMRMLFreeSuferModelOverlayStorageNode get an optional CopyData method which acts like write data, but does a disk copy from source to cache, and then an upload from cache to remote host. Not so great, but we'll test to see if it works.