Pdf and text documents are supported as standard radiology reports to be included in the dataset alongside radiology images.

Radiology reports are prepared by radiologists to present the findings in the radiological imaging data like x-Ray, CT, MRI. Ragiologists spend years training skills to read complex radiological images. They use acquired skills and specialized knowledge base to produce detailed information that can be deduced from the imagery captured by magnetic resonance imaging.

Radiology reports are used by physicians to diagnose the pathological condition. Based on the diagnosis a treatment is prescribed by the doctor. For the purposes of machine learning, radiology reports are useful for annotators to identify the pathological condition in the images, create segmentation of the pathology in the images, and classify the pathologies.

Supported Folder Structure

Lets assume "samplenifti" is the top level folder in the file system. "samplenifti" contains folder "study1", "study2".

Folder "study1" has three other folder "series1", "series2", "report"

Folder "series1" has a nifti file. Folder "report" has radiology report file "report.pdf"

|- samplenifti/  

       |- study1/    

       |        |- series1/              

       |        |        |- <nifti file>    

       |        |- series2/              

       |        |        |- <nifti file>

       |        |- report/

       |        |        |- report.pdf

       |- study2/    

       |        |- series1/

       |        |        |- <nifti file>

       |        |- series2/

       |        |        |- <nifti file>

       |        |- report/

       |        |        |- report.txt

Every labeling job provides statistics for each annotator. Statistics include total number of labels for each class, number of accepted labels, number of rejected labels. Aggregated sum for all annotators is also available.

Statistics

The line charts show labeling time in seconds spent on each day, review time in seconds, number of labels added per day, number of reviews added per day.

Label Browser

Label browser allows search within the labels. Users can filter labels by annotator email, class name, review status. The free form search will search accross all  

CT and MRI dicom images can be viewed using multiple window settings in order to see different features of the image. For example, there would be different window settings to look at Abdominal / soft tissues, Bone, Brain, Liver, Lung, Subdural brain.

The Window Center (0028,1050), Window Width (0028,1051) and VOI LUT Function (0028,1056) Attributes are used only for Images with Photometric Interpretation (0028,0004) values of MONOCHROME1 and MONOCHROME2. They have no meaning for other Images. If any VOI LUT Table is included by an Image, a Window Width and Window Center or the VOI LUT Table, but not both, may be applied to the Image for display. Inclusion of both indicates that multiple alternative views may be presented.

Dicom file format is the most commonly used file format for storing radiology image data. All radiology equipment like CT and MRI machines, manufactured by many companies, generate dicom images that are sent to a pacs server. Automatic analysis by various AI algorithms happens on the PAC server. The images and the results of automatic analysis are then pushed to individual radiologist terminals. Radiologists use automatic analysis done by AI algorithm to speed up their workflow.

On the development side, ML engineers use images from PAC server to create data-labeling pipelines. Human annotators provide segmentation and classification labeled data on top of dicom images. The quality of data labeling tools plays a critical role in

Here is a comparison of best dicom image data labeling tools for data science teams. The three tools that support native dicom images are TrainingData.io, Labelbox and V7 darwin.

TrainingData.io

TrainingData.io provides dicom viewer similar to many industrial dicom viewers used by radiologists in their daily practice. Here are some unique features supported by this labeling tool:

1. Window level presets include support for Liver, Lungs, Bone, Subdural brain, Abdominal soft tissue. User can drag a slider for the window center and a slider for window width to set custom values for each.
2. Mutli-planar view allows radiologist to view axial plane, sagital plane, coronal plane in one view. It can be configured as 2x1, 1x2, or 2x2 windows.
3. Superpixel segmentation with brush and eraser is the fastest and most accurate tool for creating segmentation labels.
4. Classification is supported for DICOM data files.
5. Tags for each individual image
6. Cloud storage supported includes Amazon-S3 and OTC cloud from Deutch Telekom

Labelbox

LabelBox does not provide support for dicom format directly. Our guess -- this is a hidden feature only available to select customers. Users need to convert dicom files to png or jpeg format.

V7 Darwin

In the samples that were made available, as shown below, the auto annotation feature looks impressive.

There is no support for Window level presets, multi-planar views, classification for dicom.

TrainingData.io provides fastest way to annotate video data.

Video Timeline

Today we are announcing general availability of the feature Video Timeline. This feature allows annotators to play & pause the video, view annotations, propagate annotations, interpolate annotations in an interface as shown in the image below.

Video Timeline has controls for play, pause, left, right navigation as shown below.

Video time line shows every frame in the video as a separate line with it's timestamp. The red marker is the current frame.

Video timeline shows each annotation as a horizontal bar. Each end of the horizontal bar can be dragged to increase or decrease the duration of an annotation.  

Key frame is a frame on which annotator manually modified annotations. Key frame is shown as a diamond on the annotation bar.

Power Features

Here is a short video showing features of Video Timeline.

TrainingData.io provides geometric interpolation of objects in its annotation tools.

Draw a label as shown below

Navigate to a different frame

Propagate label in frame #54

Edit label in frame #54

Interpolate in frame #54

Video

All autonomous vehicles (AV) use a collection of hardware sensors to identify the physical environment surrounding them. The hardware sensors include camera or a collection of cameras strategically placed around the body of the vehicle to capture 2D vision data, and some form of RADAR placed on top of the vehicle to capture 3D position data. There are a few vendors like Tesla who believe vision data is enough for a vehicle to identify its environment. Other vendors use LIDAR sensors to capture 3D position data for the objects surrounding the vehicle. Fusion of 2D vision data and 3D position data gives an AV system precise understanding of its surrounding environment.

Developing precise understanding of its surrounding environment is the first component of an AV system. Image below shows all the important components of an AV system.

Sensor Fusion

Computer vision is a branch of computer science that uses camera or a combination of cameras to process 2D visual data. This allows computers to identify cars, trucks, cyclists, pedestrians, roads, lanes marking, traffic signals, building, horizon. Camera data is 2D in nature, and it does not provide distance of an object. Although focal length and aperture of a camera sensor can be used to approximate the depth of an object, it will not be precise because there is intrinsic loss of information when a 3D scene is captured by a camera sensor onto a 2D plane.

Radar technology has been in use in places like air traffic management to locate flying objects. Radar can be used to estimate location and speed of an object. It cannot be used to classify an object as a car, person, traffic signal, or a building because it has low precision. Lidar is a hardware that uses laser technology to estimate the position and speed of objects in the surrounding. Lidar is able to generate point cloud of upto 2 million points per second. Due to higher accuracy Lidar can be used to measure shape and contour of an object.

While RGB data from camera is lacking depth information, point cloud data generated by Lidar lacks texture and color information present in the RGB data. For example in a point cloud data, contour of a pedestrian 20 feet away might be a blob of points that can be identified as multiple different objects as shown in rendering of point cloud below. On the other hand a shadow ridden low quality partial visual information gives a hint that the object is a person as shown in image from a camera below.

When fusion of visual data and point cloud data is performed, the result is a perception model of the surrounding environment that retains both the visual features and precise 3D positions. In addition of accuracy, it helps to provide redundancy in case of sensor failure.

Fusion of camera sensor data and Lidar point cloud data involves 2D-to-3D and 3D-to-2D projection mapping.

3D-to-2D Projection

Hardware Setup

We start with the most comprehensive open source dataset made available by Motional: nuScenes dataset. It includes six cameras three in front and three in back. The capture frequency is 12 Hz. The pixel resolution is 1600x900. The image encoding is one byte per pixel as jpeg. The camera data is generated at1.7MB/s per camera footage. One Lidar is placed on top of the car. The capture frequency for lidar is 20 Hz. It has 32 channels (beams). Its vertical field of view is -30 degrees to +10 degrees. Its range is 100 meters. Its accuracy is 2 cm. It can collect upto 1.4 million points per second. It’s output format is compressed .pcd. Lidar’s output data rate is 26.7MB/s (20byte*1400000).

Important Links

Dataset Page: https://www.nuscenes.org/overview

Paper URL: https://arxiv.org/pdf/1903.11027.pdf

Devkit URL: https://github.com/nutonomy/nuscenes-devkit

Understanding Reference Frames & Coordinate Systems

In order to synchronize the sensors one has to define a world (global) coordinate system. Every sensor instrument has it’s own reference frame & coordinate system.

• Lidar has it’s own reference frame & coordinate system L1,
• Each camera has it’s own reference frame & coordinate system C1, C2, C3, C4, C5, C6.
• IMU has it’s own reference frame & coordinate system I1.
• For the purposes of this discussion here, ego vehicle reference frame is the same as lidar reference frame.

Define world reference frame & coordinate system

World reference frame (W1) is global reference frame. For example one can select lidar’s first frame as center (0, 0, 0) of the world coordinate system. Subsequently every frame from lidar will be converted back to world coordinate system. Camera matrices M1, M2, M3, M4, M5, M6 will be formulated to convert from each camera coordinate system C1, C2, C3, C4, C5, C6 back to world coordinate system W1.

Convert 3D Point Cloud Data to World Coordinate System

Each frame in lidar reference frame (L1) will be converted back to world coordinate system by multiplication with ego frame translation & rotation matrices.

Convert from World Coordinate System to Camera Coordinate System

Next step is to convert data from world reference frame to camera reference frame by multiplication with camera rotation & translation matrices.

Convert from 3D Camera Coordinate System to 2D Camera Frame

Once the data is in camera reference frame it needs to be projected from 3D camera reference frame to 2D camera sensor plane. This is achieved by multiplication with camera intrinsic matrix.

Result: Accurate Annotations

As shown in the video below the fusion of lidar point cloud data, and camera data allows annotators to utilize both visual information & depth information to create more accurate annotations.

10X Speedup in labeling: Interpolation of annotation between frames

One of the the most challenging task in development of autonomous vehicle systems is to manage humongous volumes of data used for training the neural networks. As the classification and detection accuracy improves the amount of new training data needed to further improve the performance grows exponentially. To increase speed and reduce cost of annotating new training data, annotation tool can provide automation. One example of automation is interpolation of annotations between frames in LIDAR point cloud tool.

The sensor data being generated has high degree of accuracy. The lidar point cloud data is accurate to plus-or-minus 2 cms. The camera data is recorded at 1600 x 900 pixel resolution. High accuracy level allows annotation tool to provide semi-automatic techniques to reduce the manual effort required in data labeling. As an example consider annotation of 10 consecutive frames of point cloud data. Each lidar frame is accompanied by six camera frames. Human annotators use the annotation tool to fit a truck inside a cuboid in frame 1 and frame 10. Based on position of the cuboid in frame #1, and frame #10, the annotation tool can automatically interpolate position of the cuboid from frame 2 to frame 9. This significantly reduces the amount of work required from the human labellers. Such semi-automatic techniques can boost productivity, increase speed, and reduce the cost of building AI.

TrainingData.io is one-stop shop for your radiology training data requirements. We are very excited to invite you to meet us at the AI Pavilion. We will be showcasing new products at the Virtual-RSNA 2020. Please use the calendar below to book an appointment with us.

3D Segmentation of MRI / CT with GPU

Model Assisted Annotation using NVIDIA Clara

Superpixel Segmentation with Brush & Eraser

Organ Segmentation using NVIDIA Clara DeepGrow

TrainingData.io's annotation tool has easily accessible controls to change brightness of image, contrast of image and opacity of the annotations. As shown in the image below the top bar has slider controls to change brightness, contrast and opacity.

The following video shows how to modify opacity of the annotations:

Machine learning model development can consume a lot of resources. Model development involves continuous back-and-forth between the labeling team and the model developement team. The labeling team and the model development team work on an active learning data pipeline.

Active Learning Data Pipeline

In one labeling cycle the labeling team performs following tasks:

• ingests new training data in labeling tools,
• distributes new labeling task to labellers,
• does quality analysis (QA) on new labels coming from labellers,
• prepares labels to be ready for the model development team

In one development cycle the model development team performs following tasks:

• injests the labels generated by the labeling team,
• it converts the labels to the format accepted by training infrastructure,
• retrains the network,
• feed new datasets to the network,
• discover edge cases that are failing,
• prepare dataset with edge cases to be labeled in next labeling cycle.

Critical Step: Discover Edge Cases

When a machine-learning-model-under-development is used to generate predictions for new datasets, the humans-in-the-loop (radiologists) have a crucial role to play. Based on their expertise humans-in-the-loop choose the edge-cases where machine-learning-model-under-development performed poorly. They correct the predictions made by the machine learning model. These edge cases need to be augmented and then used in next training cycle.

TrainingData.io has Automated Active-Learning-Data-Pipeline for Radiology using NVIDIA Clara

NVIDIA Clara allows machine learning engineers to bring their own models (BYOM) and import them in Clara. Clara also makes the inferencing possible through a web-interface.

Privacy Preserving On-Premises Training & Inferencing

The complete solution has the following parts:

• Project management, dataset management, ML model management in the cloud.
• Docker container with NVIDIA Clara train-sdk for training & inferencing.
• Docker container with annotation tool where edge case is identified and corrected by the radiologist.

Swagger API documentation

Postman API documentation

We are very excited to announce the general availability of a 3D point cloud annotation tool for datasets generated by LIDARs. It is well known that the autonomous driving industry is philosophically divided into two camps. One camp believes computer vision (camera sensor) to be sufficient for building the perception model of the real world. The other camp believes computer vision can never be sufficient alone and, needs fusion with a radar-like system called LIDAR for building reliable perception models of real-world objects.

TrainingData.io has been providing, to its customers, world-class semi-automatic workflow for annotation of datasets consisting of images and videos generated by camera systems. Now, in addition to computer vision datasets, data-scientists can easily annotate datasets generated by LIDAR systems.

The features include the following:

• Import point cloud files generated by LIDAR equipment from a wide variety of manufacturers.
• Define custom classes, attributes, ontology, questionnaire for every annotation job.
• Navigation of 3D space using a keyboard.
• Ability to draw cuboid annotation around for every object in the scene
• Tracking each annotation through all the frames in the dataset.
• Edit cuboid includes translation, scaling, and rotation.

In March 2020, to help data scientists working on COVID-19 diagnostic tools, TrainingData.io provided a free collaborative workspace preloaded with the open-source dataset including chest X-ray and chest CT images. We would like to thank our users for their contributions. We learned a lot from our users. Now, TrainingData.io has launched a drag-and-drop user experience to bring the power of AI to every data-scientist, hospital and, clinic around the world fighting COVID-19 infection.

In the fight against COVID-19, the medical staff has four different kinds of diagnostic data available to them: a) RT-PCR test results, b) Antibodies test results, c) Chest XRay imaging and, d) Chest CT imaging. RT-PCR test results and Antibodies test results are being used as the first step for the detection of COVID-19. Chest XRay and Chest CT imaging are being used to observe the progression of the disease through the lungs of a patient diagnosed with COVID-19.

Lung damage in asymptomatic cases: researchers have studied and published clinical patterns of asymptomatic infections in Nature medicine. They have found that asymptomatic cases of COVID-19 show long term damage caused to lungs. In one such study, it was found that Chest CT exams of asymptomatic patients showed "striped shadows", and in some cases "ground-glass opacities" which is a sign of lung inflammation.

Due to the sheer size of the population affected by this virus, the ability to study lung damage in asymptomatic cases depends on the easy availability of software tools to automatically detect and visualize COVID-19 infection in Chest CT exams.

COVID-19 Ground Glass Opacities (GGOs) & consolidations in Chest CT data

When a COVID-19 patient has the virus progressing through their body, there is a build-up of fluid in the tiny air sacs in the lungs called alveoli [2]. The presence of this fluid causes inflammation of the lungs. The growth in inflammation of the lungs can be observed in XRay and CT imaging. The inflammation of the lungs shows up in the form of ground-glass opacities (GGOs) that are followed by ground glass consolidations.

The medical staff has to use some criteria to make decisions about putting a patient on an oxygen-therapy or ventilator system or putting a recovering-patient off the ventilator system. Visualizing GGOs/consolidation patterns in CT imaging plays an important role in helping medical staff to make proper decisions.

Segmentation of GGOs/consolidations for COVID-19 on TrainingData.io

Starting with a chest CT dataset from the patients diagnosed with COVID-19 in RT-PCR tests, data scientists need to create segmentation masks of Lung, and segmentation masks of ground-glass-opacities (COVID-19 infection). TrainingData.io provides a privacy-preserving parallel-distributed framework for outsourcing annotation work to multiple radiologists. All slices in a CT exam can be accurately annotated at the pixel level and visualized in 3D annotation client provided by TrainingData.io.

Semi-automatic AI-assisted segmentation of GGOs/consolitations

TrainingData.io provides a segmentation ML model that generates ground-glass-opacities/consolidations for an input chest CT exam. This model can be used to seed the initial segmentation. Radiologists can later fix the results of automatic segmentation.  

From annotated chest CT dataset to ML (segmentation) model in a few clicks

Why would a regional hospital or clinic need an ML model for the segmentation of GGOs/consolidations trained using its dataset? The answer to this question lies in the science behind the mutation of viruses. COVID-19 is caused by a virus called SARS-CoV-2. This virus has been found to mutate at a fast rate with a large proportion of the genetic diversity being found in all the affected countries. Different mutations of the virus may affect the local population in different ways.

Once the CT dataset and the segmentation masks are ready, a data scientist has to re-train the existing machine learning model to better suit the local demographics. This can be achieved with few clicks in TrainingData.io web-application.

Note: GGOs/consolidations generated on TrainingData.io are only for research purposes, not meant for clinical diagnosis.

[1] https://www.nature.com/articles/s41591-020-0965-6.pdf

[2] https://www.webmd.com/lung/what-does-covid-do-to-your-lungs

When annotator is creating annotations, annotator might want to add a new class definition. Admin can allow annotator to add class under Project Settings --> Appearance --> Allow annotator to

Annotator can add new class or edit existing labels by selecting + Add Class inside the annotation tool in the right annotation panel.

This video shows how to invoke Label Editor to edit labels or add a new class

The annotation tool enables user to import PNG masks as pre-labels. This is fastest way to implement model assisted annotation without exposing your model's intellectual property.

The input dataset should contain image files and one Png mask file per label as seen in the  dataset below (import-prelabel-mask.csv)

File Naming Convention

If name of the image file to be annotated is image-filename.jpg, for the labels to be imported correctly, it is required that PNG masks should be named with a suffix <image-filename>_mask_<label-name>.png

Steps for pre-loading labels

1. First create a dataset with dicom (or jpeg) files and png mask files as shown in the attached zip file. Name the PNG mask files with suffix <dicom-filename>_mask_<label name>.png. In the attached example, there are 2 dicom files and 4 png mask files. Dicom file 00012_2f44c2d62d278ae9.dcm has 2 labels – 00012_2f44c2d62d278ae9_mask_FemurBone.png and 00012_2f44c2d62d278ae9_mask_TibiaBone.png, dicom file 00018_7ec96b8a49737895.dcm has 2 labels -- 00018_7ec96b8a49737895_mask_FemurBone.png and 00018_7ec96b8a49737895_mask_TibiaBone.png
2. Second, create a labeling instruction with labels and colors (exact same color hex code as those in PNG masks). In the attached example labeling instruction is "KneeSegmentation"
3. Third, create a labeling job with the dataset created in step 1, and the labeling instruction created in step 2.