Brain Tumor Segmentation in MRI Images Using UNet based 3D CNN
M.S. Abirami1*, M. Uma2, R.Gurumoorthy3, Shobhit Narayan4,Jassim Hameed5
1*Assistant Professor, Department of Software Engineering, SRM Institute of Science and Technology, Tamil Nadu, India. E-mail: [email protected]
2Assistant Professor, Department of Software Engineering, SRM Institute of Science and Technology, Tamil Nadu, India.
3Assistant Professor, Department of Computer & Information Science, Annamalai University, Tamil Nadu, India.
4Student, Department of Software Engineering, SRM Institute of Science and Technology, Tamil Nadu, India.
5Student, Department of Software Engineering, SRM Institute of Science and Technology, Tamil Nadu, India.
ABSTRACT
Deep Learning is quickly reaching into a wide variety of fields. It has found useful applications in Medical Image Analysis for various time and effort consuming tasks like fractionalization of brain tumor cells in Magnetic Resonance Imaging (MRI) scans. Large amount of time and effort is required for the segmentation process by doctors and radiologists due to the high quantity of data produced by scan centers. Fatal brain tumors like Glioblastoma need to be detected and diagnosed as quickly as possible. Therefore, Automatic methods of segmentation are required for faster and accurate detection, and analysis of fatal brain tumors. Several advances in image segmentation have been proposed the practice of two-dimensional Convolutional Neural Networks (CNN), but recent publications featured CNN applications using 3D kernels on 3D MRI images. Here we present an exploratory analysis of CNN based methods using 3D filters for segmentation of HighGrade Glioma (HGG) and Lower Grade Glioma (LGG) tumors in MRI images. Framework parameters like image resolution, batch size, and normalization techniques were optimized to gain better accuracy. The BraTS 2019 dataset was used for training provided by the University of Pennsylvania and MICCAI. The model was successfully trained on 250 patients having HGG for 100 epochs. A loss graph has shown the decrease of the training and validation loss over time. Dice coefficient (DC) parameter has been taken to evaluate the resemblance of the prediction with ground truth. This model has achieved mean dice score of 0.84 and 0.65 for WholeTumor and Tumor Core respectively.
KEYWORDS
Segmentation, UNet, CNN, BraTS, Brain Tumor, Glioblastoma.
Introduction
Glioma is a brain tumor that is linked with three types of glial cells in the brain. In case of brain tumors, 80% are Gliomas. If proper treatment isn’t administered, any type of Glioma may cause long lasting brain damage or death. Gliomas can be classified according to its cancerous intensity.
The tumor could be of two types in patients, High Grade Glioma (HGG) and Low Grade Glioma (LGG). Patients with LGGhave a better survival than those with HGG. If these types of tumors
are not diagnosed early may lead the end progress towards the death of the patient. They are detected using Magnetic Resonance Imaging (MRI) scanners which yields elaborate scanning of the brain of the patient. MRI is a non-invasive technique of detecting cancerous tumors using strong magnets and radio waves. The analysis and segmentation of MRI images is a key process in brain tumor detection, growth rate prediction, and diagnostic planning, thus a precise evaluation is important.
MRI images of each patient have different modalities like T1, T1ce, FLAIR, etc. Tissues in the body are characterized by different relaxation times into T1 and T2. The different modalities of MRI differ in brightness and contrast properties determined by T1 and T2. Traditionally these have been manually segmented by professionals but this task is clearly time consuming and requires years of knowledge, experience and skill.
Therefore, a programmed method for segmentation is required to amplify the operations of identifying and therapy of brain tumors. If the tumors are detected early in stage, this can help chart a course for better diagnosis.
Deep learning has recently found effective applications in medical image processing. Compared to other methods, Convolutional Neural Networks (CNN) showed high performance for biomedical image processing tasks particularly. The CNNcan extract features effectively when combined with other operations such as max pooling and transposed convolutions. In this research work, we train and explore a U-Net based Fully Connected Neural Network (FCNN) on the BraTS 2019 dataset. The images are cropped to a size of 64x64x64, and then instance normalization and bias field correction are performed to clean the data. The data contains four modalities which are T1, T2, T1ce and FLAIR. For each patient an additional ground truth is provided which contain masked MRI images manually segmented by trusted Radiologists.
Background and Related Work
Lachinov D.,Vasiliev E. [1] proposed anautomated deep cascaded approach for tumor segmentation in brain. The implementation of this method is based on neural networking. This modifies the 3D UNet architecture to process multi-modal MRI input. This methodology is applied on the BraTS 2018 dataset and attained a dice score of 0.878/0.786 for whole tumor and tumor core respectively.
Yi Ding, Chang Li. [2] presented a methodology for a Deep Residual Dilate Network with Middle Supervision (RDM-Net). The main aim of this methodology is to raise the depth of the neural network from the first generation of the model. The disappearing gradient problem can be solved by the use of ResNet. This problem is the main barrier during the training of deep feed- forward network.
Cicek O., Abdulkadir A. [3] proposed a method volumetric segmentation that finds from sparsely annotated volumetric images. This model replaces the previous UNet architecture from m Ronneberger by changing all the 2D functions with their 3D functions. This methodology is executed on a 3D structure, the Xenopus kidney and obtainedhigh quality results.
Xiaomei Zhao, Yihong Wu. [4]proposed a model which combines fully convolutional neural networks(FCNNs) and Conditional Random Fields (CRFs). This methodology segments brain images much faster by training them slice-by-slice compared to image patch-wise training. The 2D image patches and image slices are used, and three segmentation models are trained in various views. Then they are joinedto segment the brain tumor. They are evaluated on multimodal BraTS datasets.
Kai Hu, Qinghai Gan. [5] suggested a model based on multi-cascaded convolutional neural network (MCCNN) and fully connected conditional random fields (CRFs). This methodology uses image patches from threespecial views axial, coronal, sagittal for trainingthree segmentation models and merge those models to attain the final outcome. It has been evaluated on three datasets.
J.Ker, L.Wang. [6] proposed a review on how deep learning algorithms can be applied in medical image analysis. With large amount of medical data, the relationships between them can be discovered by automated algorithms with less time effort. They have covered research areas and applications on medical image classification, detection segmentation etc.
S.Pereira, A.Pinto. [7] presented an automated process based on Convolutional Neural Network (CNN) using small 3x3 kernels. This has helped to design a deeper architecture by preventing overfitting. Normalization and data augmentation are used which are effective in the segmentation of tumor. This methodology has been evaluated on BraTS dataset.
B.H.Henze, A.Jakab. [8] proposed a review of multimodal Brain Tumor Image Segmentation Benchmark (BRATS). Twenty algorithms were applied to this dataset of 65 multi-contrast MRI scans. This work found that various algorithms were working best for different sub-regions and there was no single algorithm which could segment all sub regions simultaneously.
Mohammad Havaei, Axel Davy [9] proposed CNN architecture using local features and global contextual features simultaneously. A 2-phase training procedure was used which helped to solve difficultiesrelated to tumor labels imbalances. Results were evaluated on BraTS data which shown training speed is 30 times faster.
K. He, X. Zhang [12] proposed a residual framework which simplifies the training of networks compared to previous models. This isattained by redeveloping the layers with reference as learning residual functions with reference to layer inputs. This work displayed that remaining networks are simpler to optimize and can achieve accuracy from enhanced framework depth. This is evaluated on ImageNet andanalysis is done on CIFAR-10 with varying layer parameters for different training depths.
Methodology
The 3D UNet is implemented in the Keras module in Tensor Flow. The given dataset was split into 80% for training and 20% validation. The data for each patient provided in four modalities T1, T2, T1ce, and FLAIR were first cropped to a size of 64x64x64 and preprocessed. The input to the model is in the form (4, 64, 64, 64) where 4 is the number of modalities. Data is loaded into the Unet model using Data Generator functions of Keras module. Post training for 100 epochs, random samples from the validation set are taken and the model predicts the tumor region and
outputs a file containing the segmented mask. The results of the predictions are then evaluated with the ground truth.
Figure 1.Brain Tumor Segmentation in MRI Images
The architecture of the 3D UNet model consists of convolution filter layers of size 3x3x3 and a pooling function Max Pool to decrease the size of the dataset by compression. The proposed methodology has two pathways. First one is the reduction path (or the encoder) which is applied to confine the context in the image. Then the symmetric expanding path (or the decoder)is applied to enable accurate localization using transposed convolutions.
This process makes it a fully convolutional network. Skip connections are used to duplicate features from previous layers into later layers precisely.
Figure2.Architecture of the 3D UNet model with encoding an encoding and an up sampling layer
Architecture of the 3D UNet model is shown in Figure 2. It consists of an encoding layer and an up sampling layer. Various convolution and max pool operations are applied in the encoder part.
Deconvolution is applied to get the original image size in the decoder part.
The activation function or the sigmoid functionwhich is applied after every convolution operation is given by:
Figure 3.The activation or the sigmoid function Dataset
The BraTS 2019Multimodal Brain Tumor Segmentation Challenge dataset is used. This challenge is co-held by MICCAI and University of Pennsylvania and considered on the segmentation of heterogeneous tumors in brain named Glioblastoma. The tumor is of two types
High Grade Glioblastoma (HGG) which is more malignant and cancerous; the other is Low Grade Glioblastoma (LGG) which is a lesser amount of malignant and less cancerous.
The BraTS scans are available as NIfTI files (.nii.gz) which has four modalities (i) Native (T1), (ii) Post-contrast T1-weighted (T1ce), (iii) T2-weighted (T2), (iv) T2 Fluid Attenuated Inversion Recovery (FLAIR) and the data wereobtained from various clinics and scanners. There are two regions of the tumor, Whole Tumor, and TumorCore. The WholeTumor represents the whole region of the tumor. The TumorCore is the center of the tumor where it has started to grow.
Pre-processing
The preprocessing first crops the image to a size of 64x64x64. Instance Normalization is done on the dataset. This technique reduces the model’s training time. This normalizes across each channel in each image. Advanced Normalization Tools (ANTs) N4BiasfieldCorrection has been used to remove the corruption of low frequency bias field that could be present in the scans of the brain tumor. This would improve the accuracy of the brain tumor segmentation. The data is finally converted to NIfTI format with image wise normalization and with bias field rectification.
Training
The image shape is reduced to 64x64x64. The dataset has 4 modalities (T1, T1ce, FLAIR, T2) in which the model has been trained on all modalities. The training is performed on AWS SageMaker with Nvidia K80 GPU consisting of 12GB of GPU memory. For training, 250 patients with High Grade Glioblastoma (HGG) are considered. The model has been trained with a batch size of 1 for 200 epochs, and the validation split is 8:2 ratio. Training for fewer epochs is due to cost factors. The model takes 80% of the data for training and 20% for validation. Other functions that have been performed are flipping of the data, augmentation to expand the size of the data in modified versions, which develops the capacity of the model to generalize. This then creates a model of the training in.h5 format.
The dice score is used to quantify the performance of image segmentation techniques. The results are achieved in the form of dice coefficients. This is calculated by considering the ground truth in the image and then comparing with the predicted segmentation of the brain from the algorithm measure which will identify the similarity between them.
2 × A∩B
A +|B| (1)
This equation (1)is used to estimatethe dice similarity co-efficient and it is evaluated with the ground truth. This can be represented as,
Dice Coefficient = (2 * Area of overlap) /(total number of pixels in both the images)
Loss function is the negative of dice coefficient. Post training for 100 epochs, random samples from the validation set are taken and the model predicts the tumor region and outputs a file containing the segmented mask. The results of the predictions are then evaluated with the ground truth.
Experimental Setup and Results
The experimental results show that all the components of this work are working properly and producing expected output.The dataset is given as input in a specific format containing a separate folder for HGG and LGG which in turn contains the folders of the scans of each patient. This model produces an output of detecting the tumor in brain MRI data and performs segmentation of the brain tumor. The result is in NIfTI(.nii.gz) format. This can be viewed with the use of third- party software. In this model,ImFusionis used for demo purpose. It is image analysis software for AI computing and computer vision. For validation, 15 patientsdata haves been selected at random by a function. This model outputs two predictions which are Whole Tumor and Tumor core. The average dice score is around 0.84/0.65 for the whole tumor and tumor corerespectively. This method has compared the predicted weights with the ground truth. This model is trained for fewer epochs due to cost factors, yet the model has still attained a reasonable dice score of the predicted tumor.
Figure 4.The result of training and validation loss after training for 100 epochs
Figure 5 plotted from the training log shows a quite steady decrease of the training and validation loss over 100 epochs. The values of the loss are negative as dice coefficient. This is used to calculate the similarity which has values ranging from 0 to 1. The loss is the negative of the dice coefficient. The main purpose of the training is to reduce the loss function. The main objective of this method is to get the lowest negative loss which refers a higher dice coefficient and hence a higher accuracy of segmentation.
Figure 5.Box Plot shows Dice coefficient of Whole Tumor and Tumor Core
Figure 5 shows box plot graph for the dice scores achieved from the training. This has been plotted from the dice scores obtained after training of the model. From the Figure (5), it is observed that the dice scores for Whole Tumor and Tumor core ranging from 0.70 to 0.90 and 0.40 to 0.75 respectively. The orange line in the box shows the mean of the dice score of both the tumors. The mean scores are 0.84 and 0.65 for Whole Tumor and Tumor Core respectively.
Therefore, this model has performed better in predicting Whole Tumor than Tumor Core respectively.
The trained model is experimented on the validation data. Then the dice scores for the predictions are calculated which are given in Table 1.The model is trained on HGG with 100 epochs which is using a GPU instance of Nvidia K80. The results of 15 patients that have been selected at random by a function. The model outputs two predictions, Whole Tumor and Tumor Core.
The average dice scoresare 0.84 and 0.65 for whole tumor and tumor core respectively. The whole tumor represents the segmentation of the tumor that has been predicted whereas tumor core represents the core malignant region of the tumor. For the given data set, this model has compared the predicted weights with the ground truth of the patients. Since we have trained for fewer epochs due to cost factors, the model has still achieved a reasonable dice score of the data.
Dice score represents how similar the objects are that is between ground truth and expected segmentation.The scores of this model have been compared with RDM-Net. The latter model has been trained on larger images even with 100 epochs, our model achieved a similar score comparing to the RDM-Net model.
Table 1.The Dice score values for Whole Tumor and Tumor Core of 15 random patients Patient ID Whole Tumor Tumor Core
1 BraTS19_TCIA10_637_1 0.838261 0.576673
2 BraTS19_TCIA09_451_1 0.937802 0.940081 3 BraTS19_TCIA13_653_1 0.876385 0.125546 4 BraTS19_TCIA10_346_1 0.941859 0.914518 5 BraTS19_TCIA09_312_1 0.891699 0.3232 6 BraTS19_TCIA10_282_1 0.84033 0.633642 7 BraTS19_2013_16_1 0.796569 0.269813 8 BraTS19_TCIA13_615_1 0.905068 0.393239 9 BraTS19_TCIA10_644_1 0.877374 0.648977 10 BraTS19_TCIA13_630_1 0.932203 0.655389 11 BraTS19_TCIA09_462_1 0.907801 0.569602 12 BraTS19_2013_0_1 0.772685 0.752207 13 BraTS19_TCIA10_410_1 0.39837 0.006346 14 BraTS19_TCIA12_480_1 0.9065 0.785981 15 BraTS19_TCIA13_618_1 0.863551 0.382423
Table 2.Mean Dice score for Whole Tumor and Tumor Core Whole Tumor Tumor Core
Mean Dice 0.8457 0.6556
RDM-Net 0.86 0.71
Findings and Discussion
The results have different outputs, and the prediction score is measured in the form of dice score.
The dice score is applied to quantify the efficiency of image segmentation techniques. This is calculated by have a ground truth in the image and then comparing with the predicted segmentation of the brain from the algorithm measure how similar they are.
The following figure 6 shows the manual segmented ground truth and predicted segmentation output of the model for a selected patient.
Figure 6.Ground truth and segmented map predicted by the model
Conclusion and Future Work
This methodologyprovides an automated approach that focuses on simplifying the work of radiologists by evaluating brain MRI scans. This reduces time and effort for evaluation of the brain MRI scans. This can be further developed and can be made more robust and accurate for differing datasets in the medical fields. A Convolutional Neural Network (CNN) is explored that is based on the UNet model. It has been trained to segment brain tumors of High Grade Glioblastoma (HGG) in 3D MRI Images. The results can be viewed in 3D view, which shows all the different views of the tumor. The model has been trained on 4 modalities which are T1, T1ce, T2, and FLAIR. The results are providing two predictions which are for Whole Tumor and Tumor Core. With fewer epochs and a low batch size, the training of the model has given a dice score of 0.84 and 0.65 for Whole tumor and Tumor core respectively. This seems to be reasonable predicted score due to the number of steps run and the number of patients has been considered for this work. Training the model with various patch shapes to increase efficiency and reduce memory usage during training is considered as future works.
Acknowledgement
We thank the University of Pennsylvania for approving our request to use the BraTS dataset for exploring and training the model.
References
[1] Lachinov, D., Vasiliev, E., &Turlapov, V. Glioma Segmentation with Cascaded Unet.
Computer Science, Springer, 2019.
[2] Ding, Y., Li, C., Yang, Q., Qin, Z., & Qin, Z. (2019). How to improve the deep residual network to segment multi-modal brain tumor images. IEEE Access, 7, 152821-152831.
[3] Cicek, O., Abdulkadir, A., Lienkamp, S.S., Brox, T., &Ronneberger, O. (2016). 3D U- Net: Learning Dense Volumetric Segmentation from Sparse Annotation. Computer Science, Springer.
[4] Zhao, X., Wu, Y., Song, G., Li, Z., Zhang, Y., & Fan, Y. (2018). A deep learning model integrating FCNNs and CRFs for brain tumor segmentation. Medical image analysis, 43, 98-111.
[5] Hu, K., Gan, Q., Zhang, Y., Deng, S., Xiao, F., Huang, W., & Gao, X. (2019). Brain tumor segmentation using multi-cascaded convolutional neural networks and conditional random field. IEEE Access, 7, 92615-92629.
[6] Ker, J., Wang, L., Rao, J., & Lim, T. (2017). Deep learning applications in medical image analysis. IEEE Access, 6, 9375-9389.
[7] Pereira, S., Pinto, A., Alves, V., & Silva, C.A. (2016). Brain tumor segmentation using convolutional neural networks in MRI images. IEEE transactions on medical imaging, 35(5), 1240-1251.
[8] Menze, B.H., Jakab, A., Bauer, S., Kalpathy-Cramer, J., Farahani, K., Kirby, J., & Van Leemput, K. (2014). The multimodal brain tumor image segmentation benchmark (BRATS). IEEE transactions on medical imaging, 34(10), 1993-2024.
[9] Havaei, M., Davy, A., Warde-Farley, D., Biard, A., Courville, A., Bengio, Y., &
Larochelle, H. (2017). Brain tumor segmentation with deep neural networks. Medical image analysis, 35, 18-31.
[10] Bakas, S., Akbari, H., Sotiras, A., Bilello, M., Rozycki, M., Kirby, J.S., &Davatzikos, C.
(2017). Advancing the cancer genome atlas glioma MRI collections with expert segmentation labels and radiomic features. Scientific data, 4(1), 1-13.
[11] Bakas, S., Reyes, M., Jakab, A., Bauer, S., Rempfler, M., Crimi, A., & Eaton-Rosen, Z.
(2018). Identifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the BRATS challenge. arXiv preprint arXiv:1811.02629.
[12] He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, 770-778.
[13] Un, L., Zhang, S., Chen, H., & Luo, L. Brain Tumor Segmentation and Survival Predictionusing Multimodal MRI Scans with Deep Learning. Frontiers in Neuroscience, 2019.
[14] Menze, B.H., Jakab, A., Bauer, S., Kalpathy-Cramer, J., Farahani, K., Kirby, J., & Van Leemput, K. (2014). The multimodal brain tumor image segmentation benchmark (BRATS). IEEE transactions on medical imaging, 34(10), 1993-2024.
[15] Ronneberger, O., Fischer, P., &Brox, T. (2015). U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, Springer, Cham, 234-241.
[16] Abirami, M.S., & Sheela, T. (2017). Improved Speedup Performance in Automated Segmentation of Kidneys on Abdominal CT Images and Extracting its Abnormalities. International Journal of Pure and Applied Mathematics, 116(21), 799- 810.
[17] Abirami, M.S., & Sheela, D.T. (2014). Analysis of image segmentation techniques for medical images. In Proceedings of International Conference on Emerging Research in Computing, Information, Communication and Applications.
