Artificial Intelligence Survey for 2020 by ACR Data Science Institute

Authors

• Prabhakar Vagvala Alcon , Director and Head of Global ERP Operations, Dallas-Fort Worth Metroplex, USA

Abstract

An Outside of the healthcare industry, commercially available artificial intelligence (AI) algorithms have shown signs of racial, gender, and societal prejudice. The development of AI algorithms in the fields of radiologic sciences and healthcare is significantly impacted by these biases. The physician community should work with developers and regulators to create paths that guarantee algorithms sold for widespread clinical practice are secure, efficient, and free of unintended bias in order to prevent the introduction of bias in healthcare AI. Structured AI use cases with data elements have been developed by the ACR Data Science Institute to make it easier to create standardized datasets for AI testing and training at various universities. This project seeks to encourage the accessibility of a variety of data for algorithm development. Additionally, ACR Certify-AI and ACR Assess-AI, validation and monitoring services offered by the ACR Data Science Institute, integrate guidelines for minimizing algorithm bias and advancing health fairness. The ACR should support pricing methods for AI that guarantee access to AI technologies for all patients, regardless of their socioeconomic level or the resources available within their healthcare systems, in addition to promoting diversity. The ACR Data Science Institute conducted its first annual survey of ACR members to better understand how artificial intelligence (AI) is used in clinical practice and to create a baseline for tracking trends. Participants were asked to answer questions about their practice's demographics and to say whether or not they presently use AI in their clinical work, as well as in what capacity. AI has started to be used in mammography screening procedures. According to the report, larger practices were more likely to use AI than smaller ones. The majority of individuals who used AI in clinical practice did so to improve interpretation; the most frequent uses were for the early detection of cerebral hemorrhage, pulmonary emboli, and abnormalities in mammograms. According to the survey results, clinical practice has just recently begun to incorporate AI at a moderate rate. The data acquired from the poll will help researchers and business experts create AI solutions that can improve radiological practice, resulting in enhancements to the effectiveness and quality of patient care.

Introduction

Data science and machine learning have become indispensable technology in our daily lives, as evidenced by a variety of applications like automatic facial and traffic sign identification, speech recognition in cars and on mobile phones. and even in advanced algorithms for games like chess and, more recently, go, where humans are now consistently defeated. Large-scale data analysis utilizing search, pattern recognition, and learning algorithms provides insightful information on how processes, systems, nature, and eventually humans behave. This opens up a world of unprecedented possibilities. In fact, the concept of autonomous driving, which seemed distant in the past, has now become a tangible reality for many drivers. The use of adaptive cruise control and lane-keeping assistance systems in vehicles makes this possible, enabling a level of autonomous driving experience. The magnitude of advancements goes beyond what has been mentioned so far, particularly within the automotive industry. This becomes evident when considering the simultaneous investments made by Toyota and Elon Musk, the founder of Tesla, in artificial intelligence research and development, each amounting to one billion US dollars announced towards the end of 2015. These investments highlight the growing trend toward autonomous, interconnected, and artificially intelligent systems that can learn from data over time and make the best choices. The impact of these developments is nothing short of revolutionary and holds immense significance for numerous industries. The automotive industry, which is a key sector in Germany, is not exempt from this transformative wave. The ability to offer new technical and service solutions facilitated by data science and machine learning will become a determining factor in international competitiveness. The future landscape will witness the influence of these advancements on the automotive industry, shaping its trajectory and paving the way for innovative offerings.

In recent decades, artificial intelligence (AI) has become a significant science, mostly because of the vast array of practical uses it has. Medical diagnosis, facial recognition, robotics, internet applications, data mining, and industrial applications are a few famous examples. The significance of AI lies in its ability to tackle complex challenges across various domains. In the realm of scientific research, the trend of interdisciplinary collaboration has become increasingly prominent. Historically, several sub-fields operated independently, but researchers have realized that combining information from various scientific subjects results in more creative and efficient solutions. This fusion of several scientific fields has been extremely helpful in creating answers that are not only unique but also broadly relevant to the scientific community at large. To ensure correct training, operation, and regulation of AI-based systems and their applications, it is essential for engineers, medics, data scientists, and computer science researchers to work together. These systems offer a dual perspective, serving as both analytical tools and models of synthesis. As an analytical method, AI allows for the validation of various theories formulated on the functioning of biological systems. This validation occurs through simulation using different models, eliminating the need for direct intervention in these systems. By bringing together expertise from multiple disciplines, collaborative efforts can effectively advance the development and application of AI systems in healthcare and other fields. This integration of knowledge and skills is crucial for enhancing scientific understanding and achieving significant breakthroughs. As models of synthesis, AI-based systems enable the construction of solutions that mimic the problem-solving capabilities of biological systems. By harnessing large datasets, These systems make use of highly developed data science (DS) tools and techniques, such as deep learning (DL). These algorithms can effectively process vast amounts of unstructured data, enhancing generalizability while allowing feature extraction and detection of high-level abstractions. Accessibility to large datasets facilitates the use of DS technologies like machine learning (ML), across various research fields, including biomedicine, neuroscience, and robotics. ML algorithms have the potential to automate or address complex tasks in areas such as Time series analysis includes prediction, categorization, regression, diagnostics, and more. In essence, AI and DS technologies hold promise for supporting research endeavors by providing efficient and automated solutions to intricate challenges across multiple disciplines.

Artificial intelligence (AI) has the potential to revolutionize healthcare by gathering and analyzing information, improving diagnosis accuracy, optimizing treatment planning, and ultimately enhancing patient outcomes. Within the radiologic sciences, AI applications are rapidly expanding [1]. The American College of Radiology (ACR) founded the ACR Data Science Institute (DSI) in 2017 to lead the advancement of AI in radiological sciences and medical imaging. The ACR DSI's main objective is to promote the development, verification, and application of AI technology in these areas with the ultimate goal of enhancing the quality of life for patients, society, and the radiography profession [2]. The ACR DSI is essential in promoting innovation, easing the use of AI in radiology, and assuring the safe and efficient application of AI algorithms in clinical practice. The ACR Data Science Institute (DSI) has made it a top priority to put patients' needs first while developing and using AI technologies. This entails making sure AI algorithms don't display biases that could make them perform poorly for particular patient populations? The ACR DSI further stresses the significance of ensuring that all patients, regardless of their demographics or socioeconomic level, have fair access to AI algorithms that improve clinicians' capacity to deliver better care. The ACR DSI's tools for developing AI use cases, providing training data, validating algorithms, and monitoring them are intended to increase cooperation among radiology practitioners, developers, and regulators. The application of AI algorithms in clinical practice is carefully scrutinized, bias-free, and in line with patient safety and equitable healthcare delivery thanks to this collaborative approach. Machine learning advancements in medical imaging are progressing rapidly in both academic research laboratories and industry settings. Artificial intelligence (AI) techniques are essential in the field of diagnostic imaging for disease identification and categorization, picture optimization, radiation reduction, and workflow improvement. The adoption of AI and machine learning algorithms in clinical practice relies on demonstrating their tangible impact on patient care and improvement in radiologist workflow. Therefore, the objective of this study was to evaluate two key aspects: (a) the influence of implementing an algorithm for intracranial hemorrhage (ICH) detection on turnaround times in noncontract CT scans, and (b) whether the impact on turnaround time was influenced by the presentation of information within the radiologist workflow. The study sought to assess the potential benefits of integrating AI algorithms into radiology practice by analyzing their effect on workflow efficiency and turnaround times.

Materials a nd Method

The use of AI in diagnostic radiology is expected to rise exponentially, according to surveys done in the business literature [1]. There haven't been many surveys in the medical literature, but the ones that have been done have mainly concentrated on the potential uses of AI in clinical practice rather than how it is already being employed [3]. The ACR Data Science Institute carried out the largest survey of radiologists in the United States in 2020, which offers important insights regarding the state of AI adoption at the time. It is hoped that the survey's findings will be very helpful for ACR, federal authorities, and AI developers. The findings will help guide the initiatives of the ACR Data Science Institute, ensuring that they align with the needs and requirements of its members in the radiology community. The ACR Data Science Institute's study results show that, despite the considerable buzz surrounding AI, its use in clinical practice is still only marginally prevalent. The use of AI in radiologists' jobs was only reported by 30% of them. Although this precise topic was not asked in the study, it is important to note that some respondents may have thought that prior computer-aided detection (CAD) algorithms used in breast imaging were AI. It is anticipated that the overall market adoption of AI is currently at roughly 2% when extrapolating the findings of part 2 of the survey to the complete population of radiologists in the United States. This suggests that, as implementation constraints for AI are steadily removed and minimized, there is significant potential for AI developers. The survey highlights the need to further explore and address the challenges and barriers that impede the broader integration of AI in clinical practice. There is a sizable opportunity for AI developers to enhance the acceptance and impact of AI in healthcare when these restrictions are removed. According to the survey results, bigger practices are more likely to use AI than smaller ones, as expected. 5% of these larger practices were found to be using AI in research applications, despite the survey not directly asking about the academic character of the activities. The fact that the US FDA has approved over 80 algorithms for clinical usage is notable [2]. In the survey, it was shown that 27 locally generated algorithms and 40 FDA-cleared algorithms were both employed in clinical settings. Interestingly, the aggregate utilization rate of these "home-grown" algorithms (9.8%) exceeded the utilization rate of any individual commercially available algorithm (9%). These findings suggest that while commercially available algorithms have a presence in clinical practice, there is also a significant utilization of locally developed algorithms. This demonstrates the potential for in-house algorithm development and highlights the importance of fostering innovation within individual institutions and practices.

The largest cause of cancer-related fatalities among women is currently breast cancer, which also retains the distinction of being the cancer in which women are most frequently diagnosed worldwide [1]. The annual incidence of breast cancer was estimated to be 1.68 million cases in 2012, and projections indicate a further 30% increase by 2025 [1]. To address the impact of breast cancer, many countries have implemented screening programs that employ mammography for early detection and treatment. The goal of these programs is to reduce mortality rates and minimize the severe consequences of the disease. Evidence from randomized controlled trials has demonstrated that mammography screening has a substantial impact on mortality. According to these studies, mammography participation in screening programs resulted in a 20% reduction in mortality from breast cancer [2]. Mammography screening is now widely used as a vital weapon in the battle against breast cancer. contributing to the early detection and subsequent treatment of the disease, thereby positively influencing patient outcomes.

While mammography screening has demonstrated its effectiveness, it is important to acknowledge the potential drawbacks associated with this technique: False-positive recalls: Mammography may sometimes produce false-positive results, leading to additional imaging studies or biopsies. This can increase medical expenses and cause emotional stress for the patient. False-negatives: Breast cancers may not be detectable on mammography due to various factors, including tumor characteristics or interpretation errors. False-negative results can result in delayed diagnosis and treatment. Radiation exposure: Mammography involves exposure to ionizing radiation, which carries a small risk. While the benefits of screening generally outweigh the potential risks, it is essential to balance the advantages of early detection with the associated radiation exposure. Overdiagnosis: Mammography screening may detect certain cancers that may not pose a significant threat to a person's health, like ductal carcinoma in situ (DCIS), a low-risk condition. This can lead to overtreatment, as some cancers may not progress or cause harm during a person's lifetime. When choosing to undergo mammography screening, it is crucial for patients and healthcare professionals to be aware of these limitations and take them into account in the context of personal risk profiles and shared decision-making.

A multi-scale Convolutional Neural Network (CNN) technique for automatically segmenting magnetic resonance (MR) brain pictures was the subject of a study by Moeskops et al. [6]. CNN does not necessitate the intentional extraction of particular properties like intensity, shape, or texture, in contrast to feature-based techniques. As an alternative, it employs trained or predefined kernels with various patch sizes. The researchers used three different sizes of image patches in this investigation to maintain spatial information and collect nearby neighborhood voxels. For each patch size, corresponding kernel sizes were learned and optimized using weights and biases tailored to the corresponding patch and kernel sizes. Five unique sets of pictures, including three sets of volumetric-weighted MR brain images of preterm newborns and two sets of volumetric-weighted MR brain images of adults, were segmented using the CNN-based approach. The goal was to evaluate how well the multi-scale CNN technique performed in precisely segmenting the brain structures in these various datasets. The multi-scale CNN method demonstrated accurate segmentations across various tissue classes when an adequate amount of training data was available. The evaluation revealed that using only the smallest patch size led to spatially inconsistent results for hippocampus segmentation, while the largest patch size showed better consistency. However, the most accurate segmentation was achieved when all patch sizes were combined. Another study implemented an AI-based technology into an emergency room scenario to detect cerebral hemorrhage on non-contrast CT images. This integration was done to evaluate the diagnostic capabilities of the tool and how it affected clinical workflow in an academic setting. The tool aimed to assist healthcare professionals in identifying intracranial hemorrhage cases promptly, potentially improving patient care in emergency situations.Top of Form

In 2006, Brauers and Zavadskas presented the Multi-Objective Optimization on the basis of Ratio Analysis (MOORA) approach. Despite being a relatively new approach, MOORA has found use in a number of fields, including building, management, and economics. In order to address decision-making issues in the context of real-time industrial systems, Chakraborty (2010) used the MOORA technique. By employing MOORA, Chakraborty aimed to make effective decisions in dynamic manufacturing settings. Kracka et al. (2010) applied the MOORA method to the construction field, specifically focusing on energy loss in heating buildings. The authors utilized MOORA as a tool to solve problems related to energy efficiency and optimization in construction practices. These studies demonstrate the versatility and applicability of the MOORA method across different industries and problem domains, showcasing its potential as an effective decision-making tool. The goal of the research is to create a method for choosing external building walls and windows. In this situation, the MOORA technique was used by Brauers and Zavadskas (2009; Brauers et al., 2008b) to assess facilities sector contractors. The MOORA technique has additionally shown effective in identifying the best road design options (Brauers et al., 2008a). Both Brauers and Ginevicius (2010, 2009) and Brauers and Zavadskas (2010, 2008) have suggested using the MOORA approach in a variety of economic disciplines. These studies highlight the versatility and effectiveness of the MOORA method in different domains, including facilities evaluation, road design, and economic decision-making. By leveraging the MOORA method, the research aims to provide a robust approach for selecting external walls and windows that considers multiple criteria and objectives, enhancing decision-making processes in building design and construction. For project management in a transaction-based economy, Brauers and Zavadskas (2010) used the MOORA technique in their study. In addition, Brauers and Ginevicious (2009) defined an economic policy for attaining balanced regional development in Lithuania using the MOORA technique. The MOORA approach can be thought of as a compromise between the well-known Simple Additive Weighting (SAW) method and the widely-used Technique for Order of Preference by Similarity to Ideal Solution (TOPSIS) method in terms of ranking alternatives. Due of its placement, the MOORA technique is a practical and effective way to make decisions. Although the MOORA approach is still quite young, no extensions have yet been suggested. As a result, the MOORA approach is expanded in this study expressly for use with interval numbers. The goal of this expansion is to make the MOORA approach more useful and adaptable by enabling its application in situations where interval numbers are present.

The table presents market penetrance data for different AI algorithms across various fields. The information shows the surveyed use, percentage of total AI used, estimated total market use, and estimated total sales of data for each category. In the case of self-developed AI, 38 instances were surveyed, accounting for 9.8% of the total AI used. The estimated total market use of self-developed AI was projected to be 1063, but there were no sales of data reported, indicating a value of $0. Mammography screening, on the other hand, had 35 instances surveyed, representing 9% of the total AI used. The estimated total market use for mammography screening AI was 979, with data sales amounting to $98. For CT Chest (embolism), there were 25 surveyed uses, contributing to 6.4% of the total AI used. The estimated total market use for this application was projected to be 699, with data sales amounting to $70. MR Brain analytics had 23 surveyed uses, accounting for 5.9% of the total AI used. The estimated total market use for MR Brain analytics was 643, and the data sales were estimated to be $64. Lastly, CT Brain (Hemorrhage) had 22 surveyed uses, representing 5.7% of the total AI used. The estimated total market use for CT Brain (Hemorrhage) AI was 615, and the estimated sales of data were $62. It's important to note that these values are purely hypothetical and do not reflect real-world data.

The table presents normalized data for the market penetrance of AI algorithms in different fields. The values have been normalized between 0 and 1, providing a relative comparison of the surveyed use, percent of total AI used, estimated total market use, and estimated total sales of data for each category. In the case of self-developed AI, the normalized values indicate a total surveyed use of 0.5790, accounting for 0.5802% of the total AI used. The estimated total market use for self-developed AI is represented by a normalized value of 0.5792. However, no sales of data were reported, resulting in a normalized value of 0.0000. Mammography screening, with a normalized total surveyed use of 0.5333, represents 0.5328% of the total AI used. The estimated total market use for mammography screening AI is represented by a normalized value of 0.5334. The estimated total sales of data for this category have a normalized value of 0.6541. For CT Chest (embolism), the normalized total surveyed use is 0.3809, accounting for 0.3789% of the total AI used. The estimated total market use for CT Chest (embolism) AI has a normalized value of 0.3808. The estimated total sales of data in this category are represented by a normalized value of 0.4672. MR Brain analytics, with a normalized total surveyed use of 0.3505, represents 0.3493% of the total AI used. The estimated total market use for MR Brain analytics is represented by a normalized value of 0.3503. The estimated total sales of data for this category have a normalized value of 0.4272. Lastly, CT Brain (Hemorrhage) has a normalized total surveyed use of 0.3352, accounting for 0.3375% of the total AI used. The estimated total market use for CT Brain (Hemorrhage) AI is represented by a normalized value of 0.3351. The estimated total sales of data in this category have a normalized value of 0.4138.

Figure 1. Normalized Data

Figure 1 show that the normalized values allow for a relative comparison of the market penetrance of AI algorithms across different fields, taking into account their surveyed use, percentage of total AI used, estimated market use, and estimated sales of data.

The table shows uniform values of 0.25 for all categories across the Total surveyed use, percent of total AI used, estimated total market use, and estimated total sales of data. This suggests that each category has an equal representation in terms of surveyed use; percentage of total AI used, estimated market use, and estimated sales of data. It's important to note that these values might not accurately represent real-world data and could be used as placeholders or hypothetical examples

The table represents a weighted normalized decision matrix for different AI categories. The values in the table have been calculated by assigning weights and normalizing the data based on undisclosed criteria. Analyzing the values, it can be observed that the highest weighted normalized scores are given to Self-developed AI with 0.1448, mammography screening with 0.1333, CT Cheast (embolism) with 0.0952, MR Brain analytics with 0.0876, and CT Brain (Hemorrhage) with 0.0838. These scores indicate the relative performance or importance of each category based on the undisclosed criteria and weights applied. However, without specific knowledge of the criteria and weights used, it is difficult to interpret the significance of these values. It is important to note that the values presented in the table are hypothetical and do not reflect real-world data. The provided information is limited to the given context and the undisclosed criteria and weights applied.

Figure 2. Weighted normalized decision matrix

The table presents assessment values for different AI categories, reflecting their relative performance or effectiveness based on an undisclosed criterion. Each category is assigned an assessment value that indicates its level of success or accomplishment within the given context. According to the provided values, mammography screening receives the highest assessment value of 0.5634, suggesting that it demonstrates relatively strong performance. This implies that mammography screening AI is deemed more effective or successful in its application compared to the other categories. Following closely behind is self-developed AI with an assessment value of 0.4346. While not as high as mammography screening, self-developed AI still showcases a considerable level of performance. CT Chest (embolism) is assigned an assessment value of 0.4020, indicating a relatively moderate level of effectiveness. Similarly, MR Brain analytics has an assessment value of 0.3693, suggesting a similar level of performance. Lastly, CT Brain (Hemorrhage) receives the lowest assessment value of 0.3554, indicating comparatively lower effectiveness or performance within its category. It is important to note that the specific criteria or factors used to determine these assessment values are not provided, limiting the interpretation of the values solely to the given context.

Figure 3. Assessment value

Figure 3 shows that the mammography screening receives the highest assessment value of 0.5634, CT Chest (embolism) is assigned an assessment value of 0.4020, self-developed AI with an assessment value of 0.4346, MR Brain analytics has an assessment value of 0.3693, CT Brain (Hemorrhage) receives the lowest assessment value of 0.3554.

The table displays the rank assigned to different AI categories based on an undisclosed criterion. The ranks indicate the relative positioning of each category in terms of their performance, effectiveness, or any other relevant metric. According to the provided ranks, mammography screening is ranked first, indicating that it holds the highest position among the listed categories. This suggests that mammography screening AI is considered the most successful or effective based on the undisclosed criterion. Self-developed AI holds the second rank, showcasing a relatively strong performance but falling slightly behind mammography screening. CT Cheast (embolism) is ranked third, suggesting a moderate level of performance within its category. MR Brain analytics takes the fourth rank, indicating a slightly lower performance compared to the preceding categories. Finally, CT Brain (Hemorrhage) is ranked fifth, representing the lowest position among the listed categories. It is important to note that without knowledge of the specific criterion used to assign these ranks, their interpretation is limited to the given context.

Figure 4. Ranking

Figure 4 shows that the mammography screening is ranked first, Self-developed AI holds the second rank, CT Cheast (embolism) is ranked third, MR Brain analytics takes the fourth rank and CT Brain (Hemorrhage) is ranked fifth.

Conclusion

In the area of analytical data processing, we will gradually move away from the only use of decision-support systems and toward additional use of systems that make decisions on our behalf over the coming years. We are currently developing unique analytical solutions for particular problems, particularly in the field of data analysis, but these solutions cannot be used across different contexts. For instance, a solution developed to detect anomalies in stock price movements cannot be used to comprehend the contents of images. Although AI systems will merge separate interacting components and subsequently be able to handle increasingly complicated jobs that are currently designated exclusively for humans, this will still be the case in the future—a clear trend that we can already detect. The purpose of this initial AI survey for ACR members is to track the usage of AI in radiological practice. It is the first in a series of surveys

References

that will be conducted over time. Since we employed a limited number of questions to maximize responses, it was challenging to determine the true response rates, which might have contributed to some misunderstandings regarding what qualifies as artificial intelligence. For instance, it appears that some respondents classified legacy CAD for mammography as AI. We did not employ the classification system for practices recognized in the ACR Workforce Surveys, despite stratifying practice types by group size [8]. To better understand how AI is used in the various practice kinds, we will use the classification in the future. We also acknowledge that a survey of practices and individual radiologists should be balanced. In addition to asking radiologists' perspectives, future surveys will also target practice leaders, similar to the ACR Workforce Survey [8].

1. Hofmann, Martin, Florian Neukart, and Thomas Bäck. "Artificial intelligence and data science in the automotive industry." arXiv preprint arXiv:1709.01989 (2017).
2. Vinod, Dasari Naga, and S. R. S. Prabaharan. "Data science and the role of Artificial Intelligence in achieving the fast diagnosis of Covid-19." Chaos, Solitons & Fractals 140 (2020): 110182.
3. Górriz, Juan M., Javier Ramírez, Andrés Ortíz, Francisco J. Martinez-Murcia, Fermin Segovia, John Suckling, Matthew Leming et al. "Artificial intelligence within the interplay between natural and artificial computation: Advances in data science, trends and applications." Neurocomputing 410 (2020): 237-270.
4. Tien, James M. "Internet of things, real-time decision making, and artificial intelligence." Annals of Data Science 4 (2017): 149-178.
5. Allen, Bibb, Sheela Agarwal, Laura Coombs, Christoph Wald, and Keith Dreyer. "2020 ACR Data Science Institute artificial intelligence survey." Journal of the American College of Radiology 18, no. 8 (2021): 1153-1159.
6. Raschka, Sebastian, Joshua Patterson, and Corey Nolet. "Machine learning in python: Main developments and technology trends in data science, machine learning, and artificial intelligence." Information 11, no. 4 (2020): 193.
7. Dhar, Vasant. "Data science and prediction." Communications of the ACM 56, no. 12 (2013): 64-73.
8. Latif, Siddique, Muhammad Usman, Sanaullah Manzoor, Waleed Iqbal, Junaid Qadir, Gareth Tyson, Ignacio Castro et al. "Leveraging data science to combat COVID-19: A comprehensive review." IEEE Transactions on Artificial Intelligence 1, no. 1 (2020): 85-103.
9. Jordan, Michael I. "Artificial intelligence—the revolution hasn’t happened yet." Harvard Data Science Review 1, no. 1 (2019): 1-9.
10. Gruson, Damien, Thibault Helleputte, Patrick Rousseau, and David Gruson. "Data science, artificial intelligence, and machine learning: opportunities for laboratory medicine and the value of positive regulation." Clinical biochemistry 69 (2019): 1-7.
11. Akpofure, Nathaniel, and Opara Jude. "An explicit and efficient self-developed algorithm for IBFS of a transportation problem." Asian Journal of Multidimensional Research 11, no. 11 (2022): 58-72.
12. Kherad, Fahime Javdan, Hamid R. Naji, Mohammad V. Malakooti, and Payman Haghighat. "A new symmetric cryptography algorithm to secure e-commerce transactions." In 2010 International Conference on Financial Theory and Engineering, pp. 234-237. IEEE, 2010.
13. Yoon, Jung Hyun, and Eun-Kyung Kim. "Deep learning-based artificial intelligence for mammography." Korean journal of radiology 22, no. 8 (2021): 1225.
14. Rodríguez-Ruiz, Alejandro, Elizabeth Krupinski, Jan-Jurre Mordang, Kathy Schilling, Sylvia H. Heywang-Köbrunner, Ioannis Sechopoulos, and Ritse M. Mann. "Detection of breast cancer with mammography: effect of an artificial intelligence support system." Radiology 290, no. 2 (2019): 305-314.
15. Rodriguez-Ruiz, Alejandro, Kristina Lång, Albert Gubern-Merida, Mireille Broeders, Gisella Gennaro, Paola Clauser, Thomas H. Helbich et al. "Stand-alone artificial intelligence for breast cancer detection in mammography: comparison with 101 radiologists." JNCI: Journal of the National Cancer Institute 111, no. 9 (2019): 916-922.
16. Lång, Kristina, Solveig Hofvind, Alejandro Rodríguez-Ruiz, and Ingvar Andersson. "Can artificial intelligence reduce the interval cancer rate in mammography screening?." European Radiology 31 (2021): 5940-5947.
17. Bhattacharya, Mahua, and M. Chandana. "Analytical assessment of intelligent segmentation techniques for cortical tissues of MR brain images: a comparative study." Artificial Intelligence Review 37 (2012): 69-81.
18. El Naqa, Issam, Kristy Brock, Yan Yu, Katja Langen, and Eric E. Klein. "On the fuzziness of machine learning, neural networks, and artificial intelligence in radiation oncology." International journal of radiation oncology, biology, physics 100, no. 1 (2018): 1-4.
19. Mansour, Romany F., José Escorcia-Gutierrez, Margarita Gamarra, Vicente García Díaz, Deepak Gupta, and Sachin Kumar. "Artificial intelligence with big data analytics-based brain intracranial hemorrhage e-diagnosis using CT images." Neural Computing and Applications (2021): 1-13.
20. Mansour, Romany F., José Escorcia-Gutierrez, Margarita Gamarra, Vicente García Díaz, Deepak Gupta, and Sachin Kumar. "Artificial intelligence with big data analytics-based brain intracranial hemorrhage e-diagnosis using CT images." Neural Computing and Applications (2021): 1-13.
21. Seyam, Muhannad, Thomas Weikert, Alexander Sauter, Alex Brehm, Marios-Nikos Psychogios, and Kristine A. Blackham. "Utilization of artificial intelligence–based intracranial hemorrhage detection on emergent noncontrast CT images in clinical workflow." Radiology: Artificial Intelligence 4, no. 2 (2022): e210168.
22. McGinty, Geraldine B., and Bibb Allen. "The ACR data science institute and AI advisory group: harnessing the power of artificial intelligence to improve patient care." Journal of the American College of Radiology 15, no. 3 (2018): 577-579.
23. Allen, Bibb, and Keith Dreyer. "The role of the ACR Data Science Institute in advancing health equity in radiology." Journal of the American College of Radiology 16, no. 4 (2019): 644-648.
24. Allen Jr, Bibb, Steven E. Seltzer, Curtis P. Langlotz, Keith P. Dreyer, Ronald M. Summers, Nicholas Petrick, Danica Marinac-Dabic et al. "A road map for translational research on artificial intelligence in medical imaging: from the 2018 National Institutes of Health/RSNA/ACR/The Academy Workshop." Journal of the American College of Radiology 16, no. 9 (2019): 1179-1189.
25. Stanujkic, Dragisa, Nedeljko Magdalinovic, Sanja Stojanovic, and Rodoljub Jovanovic. "Extension of ratio system part of MOORA method for solving decision-making problems with interval data." Informatica 23, no. 1 (2012): 141-154.
26. Brauers, Willem Karel M., Romualdas Ginevičius, and Valentinas Podvezko. "Regional development in Lithuania considering multiple objectives by the MOORA method." Technological and economic development of economy 16, no. 4 (2010): 613-640.
27. KUNDAKCI, Nilsen. "Combined multi-criteria decision making approach based on MACBETH and MULTI-MOORA methods." Alphanumeric Journal 4, no. 1 (2016): 17-26.
28. Sinaga, Novendra Adisaputra, Heru Sugara, Ewin Johan Sembiring, Melva Epy Mardiana Manurung, Harsudianto Silaen, Pipin Sumantrie, and Victor Marudut Mulia Siregar. "Decision support system with MOORA method in selection of the best teachers." In AIP Conference Proceedings, vol. 2453, no. 1. AIP Publishing, 2022.
29. Pérez-Domínguez, Luis, KY Sánchez Mojica, LC Ovalles Pabón, and MC Cordero Diáz. "Application of the MOORA method for the evaluation of the industrial maintenance system." In Journal of Physics: Conference Series, vol. 1126, no. 1, p. 012018. IOP Publishing, 2018.
30. Attri, Rajesh, and Sandeep Grover. "Decision making over the production system life cycle: MOORA method." International Journal of System Assurance Engineering and Management 5 (2014): 320-328.
31. Brauers, Willem Karel M., Edmundas Kazimieras Zavadskas, Friedel Peldschus, and Zenonas Turskis. "Multi-objective optimization of road design alternatives with an application of the MOORA method." (2008).
32. Brauers, Willem Karel M. "Multi-objective contractor’s ranking by applying the MOORA method." Journal of Business Economics and management 4 (2008): 245-255.