Efficient Transfer learning Model for Humerus Bone Fracture Detection
Sasidhar Aa*, Thanabal M Sb, Ramya Pc
a Department of Computer Science and Engineering , University College of Engineering, Dindigul, India, 624622.E-mail: [email protected]
b Professor, Department of Computer Science and Engineering ,PSNA College of Engineering and Technology,
Dindigul, India, 624622
c Department of Computer Science and Engineering , University College of Engineering, Dindigul, India, 624622.
Abstract: Bone fracture classification with the help of the machine learning and deep learning models are common today. With the availability of pre-trained models such as VGG19, DenseNet121, DenseNet169 the trend is towards the concept of transfer learning. A transfer learning approach is followed in this work where the pre-trained models are first compared for finding a better model in terms of humerus bone fracture classification. In the first phase of the work, the three models are tested with the MURA Dataset. VGG16 performs comparatively low and much difference is not observed between DenseNet121 and DenseNet169. The hierarchy of results is also found to be similar in case of a customized dataset created from the humerus dataset by removing the images with metals. Hence, for complexity reasons, DenseNet121 is chosen for customization. Changes were made in the higher layers of the model and the model is subjected to partial training. While the lower layers are kept untrained the higher layers are trained with the humerus bone dataset and the customized dataset derived from it. Better results are obtained in both.
Keywords: Deep Learning, Convolutional Neural Networks, Transfer Learning, Humerus Fracture Detection.
1 Introduction
Computer vision applications though used widely in recent days, the technology is not entirely new, it’s in use for the past two decades [1]. With the necessity of the automated systems that can detect fractures from the X-Ray images, which would be of great help to the clinicians, there were many models developed based on machine learning and deep learning to detect fractures.
The accuracy of those models depends on the size of the dataset. There are studies like [2, 3 ] which uses deep convolutional neural networks and achieves higher accuracies with dataset that has more than fifty thousand studies.
There were also models developed based on the pre-trained convolutional neural networks. This includes pre trained models that has less number of layers such as LeNet , VGG and models that has more than 100 layers such as ResNet and Highway networks [4-7]. A kind of convolutional neural networks built on dense blocks is introduced in [8]. Availability of these pre-trained models lead to the concept of transfer learning which customizes the pre-trained
The objective of this work is to utilize densenet model for bone fracture classification in specific, bone fractures that is in humerus bones. Transfer learning is applied, strategies employed are the decision on the method of pre-training the densenet model optimal for the employed dataset and the method of replacing the higher level layers for better performance. The paper is organized as follows; second section explains the state of art models in bone fracture detection, third section, in detail explains the dataset preparation and building the model. Fourth section depicts the results and the last section gives the conclusion and future work.
2 Related Work
[10] Uses two predefined networks called Googlenet and Alexnet for detecting the neck of femur fracture and speech recognition. Five different pre-trained models are employed in [11] and ensemble models are also created with three models among the five in different combinations, ensemble models performs better than the other models. A novel backbone network is created and combined with R-CNN to build a better feature map and hence better results in [12]. While most of the works consider it as a binary class problem, [13] consider it as a multiclass problem and classifies the humerus fracture in to four different classes. Generative adversarial networks with transfer learning are employed in [14]. The efficiency of the pre-trained model is analyzed in [15]. In [16] a deep convolutional neural network is created by extending the U-Net Architecture, differing from the other models, it generates probability value for the existence of the fracture, in addition to it, it also adds a heat map. In addition to the features extracted from the images, various other parameters such as metadata are also considered for classification in [17]. A multistage method is employed in [18] for ROI extraction which results in better feature extraction for thigh fracture detection. Edge detection mechanism, called as the canny edge detection, along with neural network is used in [19] for fracture detection. In [20] wavelet transform is used as the pre-processing for feature extraction followed by back propagation neural network for fracture classification. Convolutional neural networks is used in [21] for detecting the neck of femur fracture. [22] Specifies that increasing the size of the dataset through data augmentation techniques increases the performance of the convolutional neural network models for fracture detection. Densenet model with focal loss function is found to provide better results in [23]. Employing transfer learning and class activation maps results in better femur fraction detection in [24].
More specific classification for the different bones is done in [25]. A two stage model is created, first stage is used for identifying the bone type and it is followed by the second stage where seven different classifiers are used for every type of bone. Another model that uses two stage approach is[26] where again the first stage is used for identifying the bone and second stage is the classifier which is the pre-trained convolutional neural networks Resnet-50 and inceotionv3.
In [27] a new approach is proposed to find the humerus fracture. A process called exemplar division is done early stage of the model and various approaches are used for feature extraction.
Relief and neighborhood component analysis is used for final feature selection. Four different classifiers are used and higher accuracy is obtained. In addition to building of a model with that considers only the images, additional parameters are considered to the pathway of the entire process from the beginning where the injury occurs to doctor’s diagnostics is proposed in [28].Harris corner model is used in [29] for fracture detection which also includes edge detection as a sub process.
3 Methodology 3.1 Data preparation
The dataset used in the work is MURA Dataset. The dataset contains X-ray images of various bones Elbow, Finger, Forearm, Hand, Humerus, Shoulder and Wrist. The one considered in the current study is humerus bone data. The details of the number of images are given below.
Following figure represents the number of training images and the number of validation images in both the positive class and the negative class.
Fig. 1Training and Test Images
Two datasets are used in the model. Dataset1 (DS1) which uses the entire humerus dataset and the other which uses a special case of positive images. Dataset 2 (DS2) is prepared by removing the images that are positive and with hardware as shown below.
Fig. 2 Positive case with hardware 0
100 200 300 400 500 600
Training Images Testing Images
Positive Negative
This is done in order to consider the fact that training a model with additional hardware in bone and labeled with positive would bias the model towards it. So the model is also tested with a dataset which does not contain such images. Removing of such images leads to low number of positive cases, 240 images. Since the number of images for training should not be less , data augmentation is made by horizontal flipping and the number of positive cases is 480. 90% of the total images are used for training and remaining for testing. Both these datasets are subjected to the following pre-processing steps. Preparations of both the datasets are depicted in the following figure.
Fig. 3.a Dataset 1
Fig. 3.b Dataset 2
Fig. 3(a and b) Dataset Preparation
3.1.1 Pre-processing
The steps followed in pre-processing are given in the following figure 4.
Fig. 4 Pre-Processing Humerus Bone
Dataset from MURA
Pre-Processing
Humerus Bone Dataset from
MURA
Positive
Negative
Extraction of Fracture Images (By removing
images with Hardware) Pre-Processing
Dataset
Rescaling
Data Augmentation
Affine Transformation
Normalization
All the images are resized into 224 X 224 for uniformity. It is followed by the data augmentation in which the images are horizontally flipped in order to increase the size of the dataset, in case of the dataset1. In case of dataset2 only the positive images are subjected to data augmentation in order to address the class imbalance problem in this dataset. However the size of the dataset2 is considerably lower than the size of the dataset1. Affine transformation is done in order to avoid the geometric distortions. Affine transformation is done with the help of an affine matrix which can be represented with the following matrix.
𝛼 𝛽 1 − 𝛼 . 𝑐𝑒𝑛𝑡𝑒𝑟. 𝑥 − 𝛽. 𝑐𝑒𝑛𝑡𝑒𝑟. 𝑦
−𝛽 𝛼 𝛽. 𝑐𝑒𝑛𝑡𝑒𝑟. 𝑥 + 1 − 𝛼 . 𝑐𝑒𝑛𝑡𝑒𝑟. 𝑦 Where
𝛼 = 𝑠𝑐𝑎𝑙𝑒. cos 𝑎𝑛𝑔𝑙𝑒 𝛽 = 𝑠𝑐𝑎𝑙𝑒. sin(𝑎𝑛𝑔𝑙𝑒)
Here angle refers to the degrees to which the image should be rotated. Parameter scale refers to the scale factor. Centre refers to the point based on which the rotation will be made. This matrix is used in the affine transformation matrix, which is nothing but the linear transformation which is followed by the vector addition. Following is the process of normalization. Normalization is done by calculating the mean and the standard deviation of the pixel values of the individual images and applying the following formula.
(Pixels-mean)/standard deviation
The normalization process is applied for the individual images of the dataset.
3.2 Model Preparation
Convolutonal neural networks are a commonly used deep learning model in computer vision problems[30]. It has been observed from the literature that instead of training the model from the scratch, transfer learning can be applied to use the existing pre trained models in image classification approaches with suitable modifications [9]. There are also many variants of the pre-trained convolutional neural networks. Three different pre-trained convolutional neural networks are tested with the Dataset.
VGG16
Densenet121
Densenet161
These models are trained and tested in the humerus bone X-ray images of the radiographs with the default imagenet weights. The results obtained with the dataset 2 are given in the following figure 5.
Fig. 5 Accuracy of the different pre-trained models (dataset 2).
The results obtained with the dataset1, the original dataset, with the pre-trained models are given in the following figure 6.
Fig. 6 Accuracy of the different pre-trained models (dataset 1).
It has been observed from the results that there is no much difference between the default DenseNet169 model and Densenet121 model. So for the reasons of computational complexity, Densenet121 is considered and customized.
The model is built on two concepts.
56 58 60 62 64 66 68 70 72
DenseNet169 DenseNet121 VGG16
Accuracy
Accuracy
66 68 70 72 74 76 78 80 82
DenseNet161 DenseNet121 VGG16
Accuracy
Accuracy
Removal of the fully connected layer in the pre-trained model and adding the customized block.
Training partial convolutional base.
Both these would help in increasing the accuracy of the model. Any conventional pre-trained models based on convolutional neural networks have two parts, a convolution base and a classifier. It is inferred from [31] that different layers of the model would learn different features.
While the layers that are closer to the input of the model learns features that are generic, higher level layers learn more specific features. Following are the different transfer learning process.
Training all the layers, in the convolutional base as well as in classifier.
Training a part of the layers in the convolutional base and the classifier
Training only the classifier.
The method of transfer learning that adapts to the problem taken depends on the size of the data and the relevancy of it to the dataset that has been used to train the pre trained model. The pre trained model that we use here is Densenet121 and it is trained in the imagenet dataset. The dataset that is employed in the scope of the problem is X-Ray images. It is obvious that this dataset is not most relevant to the imagenet dataset. It has also been observed that the employed dataset is not considerably larger in size. Because of these reasons it is decided to train the part of the higher layers and leaving behind the rest without training. Training all the layers is not needed in this scenario, whereas training only the classifier is not enough. So the Densenet121 pre trained model is partially trained. The higher 60 layers are trained whereas the layers before it is not trained. The strategy followed is given in the following figure 7.
Fig. 7 Transfer learning method
Three dense blocks along with a softmax is added at the top of the model. The softmax layer added at the end is represented as follows.
𝐽 = − 1 𝑚
𝑚
𝑖=1
𝑦𝑗(𝑖)𝑙𝑜𝑔(𝑦𝑗_(𝑖)) + (1– 𝑦𝑗(𝑖))𝑙𝑜𝑔(1– 𝑦𝑗_(𝑖))
𝑛
𝑗 =1
Where
yj(i)is the 𝑖𝑡ℎ training label for output node j, yj_(i)is the ith predicted label for output node j, m is the number of training / batch samples
Convolution base
Classifier
Trained Untrained
and n is the number . The results obtained with this model in both the datasets are given in the next section
4 Experimentation and Results
The implementation of the model is done in python with the help of scikit-learn[32] and Tensorflow .the results obtained is given in the following graphs. Figure 8 represents the accuracy obtained with the dataset 1. It represents both the training accuracy and validation accuracy.
Fig. 8 Training and Validation Accuracy (Dataset1)
Following Figure 9 represents the training and validation accuracy obtained with the model in case of Dataset2.
Fig. 9 Training and Validation Accuracy (Dataset2)
The accuracy obtained with the model for both the datasets is given in the below figure 10.
Fig. 10 Comparison of the accuracy with two datasets
5 Conclusion and Future work
Transfer learning is the method by which the knowledge learned by the available pre-trained models can be used for the new computer vision applications in different domains. Three different pre-trained models VGG16, Densenet121 and Densenet169 are tested for their performance in bone fracture detection in humerus bones. With the results obtained, Densenet121 is chosen for further processing and it is customized by replacing the fully connected layers and partially training the model to learn better features from the given dataset.
This customized model is found to produce better results than the conventional pre-trained models. Future work would include model that includes enhancement of feature extraction which is believed to be better accomplished with few image processing techniques that better distinguishes the bones in the images from the surrounding region followed by deep learning model.
References
1. Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel.
Backpropagation applied to handwritten zip code recognition. Neural computation, 1(4):541–551, 1989
2. A. Sheryl Oliver, Kavithaa Ganesan, S. A. Yuvaraj, T. Jayasankar, Mohamed Yacin Sikkandar &N. B. Prakash, Accurate prediction of heart disease based on bio system using regressive learning based neural network classifier, Journal of Ambient Intelligence and Humanized Computing (2020), https://doi.org/10.1007/s12652-020-02786-2
3. M.Kavitha, T.Jayasankar, P.Maheswara venkatesh, G.Mani, C.Bharatiraja, and Bhekisipho Twala, ―COVID-19 Disease Diagnosis using Smart Deep Learning Techniques‖, Journal of Applied Science and Engineering (2021), vol.24,No.3.
http://dx.doi.org/10.6180/jase.202106_24(3).0001\
4. Priyanka Parvathy, D Kamalraj Subramaniam, G.K.D PrasannaVenkatesan, P. Karthikaikumar , 76
78 80 82 84 86 88 90 92
Dataset1 Dataset2
Training Validation
Machine Learning‖, Journal of Ambient Intelligence and Humanized Computing (2020).
https://doi.org/10.1007/s12652-020-02314-2
5. O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A.
Khosla, M. Bernstein, et al. Imagenet large scale visual recognition challenge. IJCV.
6. R. K. Srivastava, K. Greff, and J. Schmidhuber. Training very deep networks. In NIPS, 2015.
7. A.Sheryl Oliver,M.Anuratha,M.Jean Justus,Kiranmai Bellam, T.Jayasankar, ―An Efficient Coding Network Based Feature Extraction with Support Vector Machine Based Classification Model for CT Lung Images,‖ J. Med. Imaging Health Inf. ,vol.10,no.11.pp.2628–2633(2020).
8. J. Jayanthi · E. Laxmi Lydia · N. Krishnaraj · T. Jayasankar · R. Lenin Babu · R. Adaline Suji,‖ An effective deep learning features based integrated framework for iris detection and recognition,‖ J Ambient Intell Human Comput (2020).
9. J.Jayanthi, T.Jayasankar, N.Krishnaraj, N.B.Prakash, A.Sagai Francis Britto, K.Vinoth Kumar,
―An Intelligent Particle Swarm Optimization with Convolutional Neural Network for Diabetic Retinopathy Classification Model,” Journal of Medical Imaging and Health Informatics (2020), Volume 11, Number 3, March 2021, pp. 803-809, https://doi.org/10.1166/jmihi.2021.3362 10. Adams M, Chen W, Holcdorf D, McCusker MW, Howe PD, Gaillard F. Computer vs human:
Deep learning versus perceptual training for the detection of neck of femur fractures. J Med Imaging Radiat Oncol. 2019;63(1):27-32. doi:10.1111/1754-9485.12828.
11. Sangeetha .J, Jayasankar.T , Emotion Speech Recognition based on Adaptive Fractional Deep belief Network and Reinforcement Learning, Springer- Advances in Intelligent Systems and Computing - International Conference on Cognitive Informatics & Soft Computing (CISC- 2017),pp-167-174. https://doi.org/ 10.1007/978-981-13-0617-4_16
12. Bin Guan, Guoshan Zhang, Jinkun Yao, Xinbo Wang, Mengxuan Wang, ―Arm fracture detection in X-rays based on improved deep convolutional neural network‖, Computers & Electrical Engineering,Volume 81,2020, https://doi.org/10.1016/j.compeleceng.2019.106530.
13. Chung SW, Han SS, Lee JW, et al. Automated detection and classification of the proximal humerus fracture by using deep learning algorithm. Acta Orthop. 2018;89(4):468-473.
doi:10.1080/17453674.2018.1453714.
14. Lin, Ying-Jia & Chung, I-Fang. (2019). Medical Data Augmentation Using Generative Adversarial Networks : X-ray Image Generation for Transfer Learning of Hip Fracture Detection.
1-5. Proceedings of 2019 International Conference on Technologies and Applications of Artificial Intelligence (TAAI), 10.1109/TAAI48200.2019.8959908
15. Kim DH, MacKinnon T. Artificial intelligence in fracture detection: transfer learning from deep convolutional neural networks. Clin Radiol. 2018;73(5):439-445. doi:10.1016/j.crad.2017.11.015 16. E. Punarselvam, Mohamed Yacin Sikkandar, ·Mohsen Bakouri, N. B. Prakash T.Jayasankar and
S.Sudhakar,― Different Loading Condition and Angle Measurement of Human Lumbar Spine MRI Image Using ANSYS‖, Springer Journal of Ambient Intelligence and Humanized Computing (2020)https://doi.org/10.1007/s12652-020-01939-7.
17. M. Buvana, K. Muthumayil, T. Jayasankar, ―Content-Based Image Retrieval based on Hybrid Feature Extraction and Feature Selection Technique Pigeon Inspired based Optimization‖, Annals of the Romanian Society for Cell Biology (2021), Vol. 25, Issue 1, 2021, pp. 424 – 443,
18. Bin Guan, Jinkun Yao, Guoshan Zhang, Xinbo Wang,‖Thigh fracture detection using deep
learning method based on new dilated convolutional feature pyramid network‖, Pattern Recognition Letters, Volume 125, 2019, Pages 521-526,
https://doi.org/10.1016/j.patrec.2019.06.015.
19. A. Y. Yang, L. Cheng, M. Shimaponda-Nawa and H. Zhu, "Long-Bone Fracture Detection using Artificial Neural Networks based on Line Features of X-ray Images," 2019 IEEE Symposium Series on Computational Intelligence (SSCI), Xiamen, China, , pp. 2595-2602, doi:
10.1109/SSCI44817.2019.9002664.
20. Kamil Dimililer, ―IBFDS: Intelligent bone fracture detection system‖, Procedia Computer Science,Volume 120,2017,Pages 260-267, https://doi.org/10.1016/j.procs.2017.11.237.
21. William Gale , et.al,‖ Detecting hip fractures with radiologist-level performance using deep neural networks‖, arXiv:1711.06504
22. D. P. Yadav and S. Rathor, "Bone Fracture Detection and Classification using Deep Learning Approach," 2020 International Conference on Power Electronics & IoT Applications in Renewable Energy and its Control (PARC), Mathura, Uttar Pradesh, India, 2020, pp. 282-285, doi: 10.1109/PARC49193.2020.236611.
23. M. Lotfy, R. M. Shubair, N. Navab and S. Albarqouni, "Investigation of Focal Loss in Deep Learning Models For Femur Fractures Classification," 2019 International Conference on Electrical and Computing Technologies and Applications (ICECTA), Ras Al Khaimah, United Arab Emirates, 2019, pp. 1-4, doi: 10.1109/ICECTA48151.2019.8959770.
24. V. Gupta et al., "Using Transfer Learning and Class Activation Maps Supporting Detection and Localization of Femoral Fractures on Anteroposterior Radiographs," 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI), Iowa City, IA, USA, 2020, pp. 1526- 1529, doi: 10.1109/ISBI45749.2020.9098436.
25. El-Saadawy H., Tantawi M., Shedeed H.A., Tolba M.F. (2020) A Two-Stage Method for Bone X-Rays Abnormality Detection Using MobileNet Network. In: Hassanien AE., Azar A., Gaber T., Oliva D., Tolba F. (eds) Proceedings of the International Conference on Artificial Intelligence and Computer Vision (AICV2020). AICV 2020. Advances in Intelligent Systems and Computing, vol 1153. Springer, Cham.
https://doi.org/10.1007/978-3-030-44289-7_35
26. Tantawi M., Thabet R., Sayed A.M., El-emam O., El bake G.A. (2020) Bone X-Rays
Classification and Abnormality Detection. In: Ghalwash A., El Khameesy N., Magdi D., Joshi A. (eds) Internet of Things—Applications and Future. Lecture Notes in Networks and Systems, vol 114. Springer, Singapore. https://doi.org/10.1007/978-981-15-3075-3_20
27. Sukru Demir, Sefa Key, Turker Tuncer, Sengul Dogan, ―An exemplar pyramid feature
extraction based humerus fracture classification method‖, Medical Hypotheses, Volume 140, 2020,https://doi.org/10.1016/j.mehy.2020.109663
28. Moreno-García, C.F., Dang, T., Martin, K., Patel, M., Thompson, A., Leishman, L. And Wiratunga, N. 2020. Assessing the clinicians pathway to embed artificial intelligence for assisted diagnostics of fracture detection. In Bach, K., Bunescu, R., Marling, C. and Wiratunga, N.
(eds.) Knowledge discovery in healthcare data 2020: proceedings of the 5th Knowledge discovery in healthcare data international workshop 2020 (KDH 2020), co-located with 24th European Artificial intelligence conference (ECAI 2020), 29-30 August 2020, [virtual conference]. CEUR workshop proceedings, 2675. Aachen: CEUR- WS [online], pages 63- 70
29. A. Bekkanti, S. Karimunnisa, S. Gogulamudi, K. Kumar and C. Zeelan Basha, "Enhanced Computerized Bone Fracture Detection Using Harris Corner Detection," 2020 International Conference on Smart Electronics and Communication (ICOSEC), Trichy, India, 2020, pp.
572-576, doi: 10.1109/ICOSEC49089.2020.9215240.
30. Bengio, Y., 2009. Learning deep architectures for AI. Foundations and trends in Machine Learning, 2(1), pp.1–127.
31. Yosinski, J., Clune, J., Bengio, Y. and Lipson, H., 2014. How transferable are features in deep neural networks?. In Advances in neural information processing systems(pp.3320–3328)
http://jmlr.csail.mit.edu/papers/v12/pedregosa11a.html
32. Martin Abadi , et.al TensorFlow: Large-scale machine learning on heterogeneous systems,2015.
Software available from tensorflow.org.