AssetStore Import Updates by BryonLewis · Pull Request #1507 · Kitware/dive

resolves #1505

  • Update the AssetStore import so it will now support possible annotation file types (json/csv).

Importing Logic:

Below is the logic as each file is imported (GirderItem) in this case:

  1. The current logic when importing will attempt to see if the item is a image-sequence a video or a large image
  2. If it is a video file it will place it in a folder with the same name as the file but minus the extension.
  3. If it is an image sequence it will label the parent as dataset type of image-sequence
  4. If there are annotation files it will first to check and see if parent is of image-sequence and do nothing if it is (the post process will automatically import the file)
  5. If it isn't of a Dataset type it will check the parent folder and attempt to find a folder with the same name as the annotation file minus the extension. If found it will then move the item into that folder.
  6. If it can't find anything (this means it was processed before the video file) it will mark it as AnnotationFileFutureProcess so that once everything is imported it can move these files around.

Once all Items are imported a secondary process begins to kick off transcoding jobs. Injected into that is a new process to manage all dangling AnnotationFileFutureProcess files to place them in the proper folder.

  1. There is a global check for all items that have AnnotationFileFutureProcessMarker set to True
  2. For each annotation file it will check if the parent-folder is a image-sequence or large-image and will just remove the the AnnotationFileFutureProcessMarker from the metadata on the item.
  3. If not it will then look for a folder with the same base_name (annotation filename without extension) inside the parent folder and place the annotation file inside that folder.
  4. It loops through all folders that were imported in the system to do this check.

Finally it kicks off process to import all video files as datasets by running the quick check to see if files need to be transcoded. During this process it calles the /postprocess endpoint that will check and look for CSV/JSON files in the same folder as the DIVEDataset and will import it as annotations if it exists.

Instructions for Testing are found in the ./samples/scripts/assetStoreImport/README.md file

Discussion of Technical Implementation:

DIVE-AssetStoreImport-TechnicalDetails_small.mp4

Testing Process:

DIVE-AssetStoreImport-Demo_small.mp4