Encouraging and astonishing developments have recently been achieved in image-based diagnostic technology. Modern medical care and imaging technology are becoming increasingly inseparable. However, the current diagnosis pattern of signal to image to knowledge inevitably leads to information distortion and noise introduction in the procedure of image reconstruction (from signal to image). Artificial intelligence (AI) technologies that can mine knowledge from vast amounts of data offer opportunities to disrupt established workflows. In this prospective study, for the first time, we develop an AI-based signal-to-knowledge diagnostic scheme for lung nodule classification directly from the computed tomography (CT) raw data (the signal). We find that the raw data achieves almost comparable performance with CT, indicating that it is possible to diagnose diseases without reconstructing images. Moreover, the incorporation of raw data through three common convolutional network structures greatly improves the performance of the CT models in all cohorts (with a gain ranging from 0.01 to 0.12), demonstrating that raw data contains diagnostic information that CT does not possess. Our results break new ground and demonstrate the potential for direct signal-to-knowledge domain analysis.
Bingxi He, Yu Guo, Yongbei Zhu, Lixia Tong, Boyu Kong, Kun Wang, Caixia Sun, Hailin Li, Feng Huang, Liwei Wu, Meng Wang, Fanyang Meng, Le Dou, Kai Sun, Tong Tong, Zhenyu Liu, Ziqi Wei, Wei Mu, Shuo Wang, Zhenchao Tang, Shuaitong Zhang, Jingwei Wei, Lizhi Shao, Mengjie Fang, Juntao Li, Shouping Zhu, Lili Zhou, Shuo Wang, Di Dong, Huimao Zhang, Jie Tian.
From Signal to Knowledge: The Diagnostic Value of Raw Data in the Artificial Intelligence Prediction of Human Data for the First Time.
Engineering, 2024, 34(3): 64-74 DOI:10.1016/j.eng.2023.02.013
The discovery of X-rays in 1895 ushered in a new era in the use of imaging for medical diagnostic purposes. Since then, noninvasive medical imaging technology has subverted the traditional palpation and cut-and-see approaches [1]. Astounding technological advances have been made in medical imaging over the past 120 years, and modern medical care is increasingly inseparable from imaging technology. Medical imaging is essential for clinicians to observe the patient’s body from images and thereby diagnose diseases. This process can be defined as a path from image to knowledge. However, it has recently been found that clinicians’ experience has become a bottleneck in this path, hindering the accurate diagnosis and treatment of diseases [2], [3].
The emergence of artificial intelligence (AI) technology has partially solved the problem of humans’ limited ability in the diagnosis process [4], [5], [6], [7]. AI can be used to automatically mine the radiographic patterns in imaging data related to the occurrence and progression of diseases, and it has been shown to match and even surpass human abilities in many clinical applications [8], [9], [10], [11], [12]. The essential reason why AI can surpass humans may be that it treats images as data rather than as visual images and extracts huge quantities of features for analysis [13], [14]. In comparison, a medical image is compressed or its data is filtered to fit the human eye, which may be insufficient or imperfect for diagnosis. Taking computed tomography (CT) as an example, the CT system first collects raw data (i.e., the signal) from the patient; then, a reconstruction method converts the raw data to images (i.e., signal to image) [15]. Therefore, both AI-based and human-based diagnosis are signal-to-image-to-knowledge processes. Medical images suffer from information distortion in both the acquisition and reconstruction processes. The current high sampling frequency greatly compresses the influence of factors such as motion artifacts in the acquisition process, so the main reason for the loss of resolution lies in operations such as interpolation and suboptimal statistical weighting in the reconstruction process [16]. In fact, the unprocessed data size of raw data is about 10-20 times larger than the size of the final CT images (e.g., 2 GB compared with 180 MB). In general, we can imagine such processes as following a chain from signals to images to knowledge (as shown in Fig. 1). In the signal domain, complete diagnostic knowledge is obtained. However, during the process from the signal domain to the image domain, the introduction of reconstruction errors leads to a loss of diagnostic knowledge, which is difficult to recover. Although AI-based models have been designed to continuously reduce their impact, losses are inevitable due to the nature of solving inverse problems [17]. In this way, no additional information is added in this process, but some information is lost. Hence, the huge amount of information present in the raw data is not optimally mined in current signal-to-image-to-knowledge processes, and there is great scientific interest in the question of how to analyze raw data.
Skipping the image process and going directly from signal to knowledge will hopefully lead to new breakthroughs in disease diagnosis [17]. This idea can be traced back to 2016, when Wang [18] proposed a direct pathway from data acquisition to therapeutic actions. Inspired by this idea, several studies have discussed the potential value of analyzing raw data [19], [20], [21], [22], [23] and thereby going directly from signal to knowledge. De Man et al. [21] conducted a simulation experiment to detect and estimate the vessel centerline from raw data in the signal domain. They achieved encouraging initial results that demonstrated the feasibility of raw data analysis for clinical CT analysis tasks. Through the simulation of raw data with CT images of pulmonary nodules, Gao et al. [23] found that pulmonary nodules could be detected in the signal domain in reality and obtained exciting results showing that this method could effectively solve the different-nodule-size problem encountered in common image-based diagnosis. Wu et al. [22] also demonstrated that an end-to-end network prediction scheme from the signal domain was superior in terms of both sensitivity and accuracy to an abnormality detection model developed with reconstructed images. We have reported our simulation results on lung cancer at the American Association for Cancer Research (AACR) conference [24]. However, most current studies are based on simulation; thus, there is no published research on signal-to-knowledge analysis in real clinical tasks based on patient data, as far as we know.
In this prospective study, for the first time, we developed an AI-based signal-to-knowledge diagnostic scheme for lung nodule classification directly from CT raw data (a flowchart of this process is shown in Fig. 1). The value of raw data alone and its added value to CT are studied for 276 patients. We found that the raw data achieved almost comparable performance with CT, indicating that it is possible to diagnose diseases without reconstructed images. Moreover, the introduction of raw data greatly promoted the performance of CT, demonstrating that raw data contains diagnostic information that CT does not possess. This research breaks the routinely used circle of image-based diagnosis and may open up a new signal-to-knowledge pathway for disease diagnosis.
2. Material and methods
2.1. Patients
In this prospective study, 626 patients who had a chest CT scan in the First Hospital of Jilin University from November 2019 to May 2021 were recruited. Eligible patients were included according to the following inclusion criteria: ① patients who had a pulmonary lesion larger than 2 cm with a contrast-enhanced chest CT scan, ② raw data obtained from a CT machine after the imaging examination, and ③ a pathological diagnosis of pulmonary lesion within a two-week interval from the CT scan. Patients were excluded based on the following: ① previous systemic antineoplastic treatments or ② CT images with poor image quality or an unreadable scan. After exclusion, a total of 276 patients were included in the modeling experiments.
The experiments were performed in accordance with the Standards for Reporting of Diagnostic Accuracy (STARD) and approved by the Ethics Committee of the First Hospital of Jilin University (AF-IRB-032-05).
2.2. Collection of CT images and raw data
Both the CT images and the raw data were collected from the First Hospital of Jilin University and were acquired using a NeuViz Prime CT system (Neusoft Medical Systems Co., Ltd., China). The system parameters of the CT scanner included a source-to-isocenter distance of 570 mm, a source-to-detector distance of 1040 mm, and a scanning field of view (FOV) of 500 mm. The imaging protocol included a contrast-enhanced CT of the chest with variable imaging parameters. Contrast-enhanced CT scans were performed in spiral scan mode using a 324 mA tube current, 100 kilovolt peak (kVp) tube voltage, 0.5 s ration time, and 0.9 spiral pitch. CT raw data were reconstructed using a kernel F20 at a slice thickness of 1.0 mm with an image pixel range from 0.59 to 0.98 mm and an image matrix of 512 × 512. In addition, we acquired the initial height and initial view angle of the CT detector each time the patient underwent a scan. Finally, CT images and raw data from each scanner were randomly stratified into one of three cohorts in a 6:2:2 ratio: a training cohort, a validation cohort, and a test cohort. Table S1 in Appendix A provides the CT scanner information, system parameters, and imaging parameters.
2.3. Lesion segmentations in CT images
The primary lesion segmentations were manually delineated across all the sections in the axial view using the annotation tool in IntelliSpace Discovery (ISD; Philips, Germany). The regions of interest were annotated and reviewed by four radiologists with 8-25 years’ experience with chest CT. All radiologists were blinded to any clinical or histopathologic information. The annotation was labeled into five common categories according to the lesions’ pulmonary lobe.
2.4. Realization of typical CT models
Many studies have been conducted on benign-malignant lung nodule classification in chest CT. We selected four typical papers from the major journals IEEE Transactions on Medical Imaging, Medical Image Analysis, and Nature Medicine to construct a CT model (CTM) using the multi-scale ensemble method (CTM 1) [25], the global and local information fusion method (CTM 2) [26], the loss function-based method (CTM 3) [27], and the multi-view fusion (CTM 4) method, respectively [28]. We further performed experiments on these four typical models with our dataset. All realization details are described in Appendix A.
2.5. Extraction of the lesion region from raw data
After acquiring the four CTMs, we proceeded to perform raw data gain experiments. The first step of the experiment was to select a projection surface containing the lesions in the raw data. The raw data of CT scans has three dimensions, which consists of 1D scan index and 2D projection data. Specifically, the scan index represents the acquisition order, and the projection data represents the detector receives the X-ray attenuation, for which the channel and row directions are defined as x and y, respectively. All lesion segmentation regions of the raw data were derived from the binarized segmentation of the CT image after being represented in a unified coordinate system. The complete derivation can be condensed into three steps: orientation, querying, and mapping (Fig. S1 in Appendix A).
2.5.1. Orientation
For the derivation of orientation, we took the segmented regions in the CT image as the research object. The orientation calculation mainly includes cross-sectional orientation and height orientation. For cross-sectional positioning, we set the center point between the CT source and detector as the coordinate origin (which is also the rotation center of the CT gantry), parallel to the cross-section of the CT image. Next, the motion trajectory was characterized by the scan index , the rotation radius , and the angle . In order to obtain these parameters, we first read the origin coordinates (, , and ) and calculated the offset values ( and ) through voxel spacing ( and ) and image size ( and ).
Next, with the help of the offset values and the voxel coordinates and in the CT image, the distance from the coordinate origin ( and ) was calculated as follows:
In the case of obtaining the above variable, the rotation radius , and the starting angle were obtained.
By introducing the scanning period of the machine, we obtained the angle change with the following relationship:
For height positioning, we directly obtained the initial height of the voxel through the coordinate z, the slice thickness, and the origin of the voxel point in the CT images.
2.5.2. Querying
Our purpose in this step was to determine the interval of the scan index tt in which the tumor voxels appeared in the raw data. Since there is a cone beam in the projection, we first calculated the change function of the voxels.
where is the distance from the voxel to the X-ray focal spot on the x-axis, and is the distance from the focal spot to the detector.
where is the number of detector rows and is the channel spacing along the y-axis.
Next, we determined the scan index t range of the voxels in the raw data by means of the following inequality.
where is the initial height at which the detector starts to scan, and is the height change in a scan.
To reduce the computational complexity, we first extracted the highest and lowest masks in the segmentation images and calculated the start and end indices of two voxels. Then, we initially located the range of index dimensions. Within this interval, we computed the mapping result of the voxels within the layer.
2.5.3. Mapping
Through the above calculation, we obtained the scan index interval corresponding to the voxels; then, the voxel appearing in the scan index was obtained by calculating the projection data of the scan index layer by layer. The coordinates of each voxel on the projection surface were defined as and , respectively. Since is related to the height , at the scan index t, and the number of detector rows in the detector, we determined its height difference relative to the detector by means of the following formula, and then calculated its coordinates in the projection.
Since the x-axis of the projection plane is equiangularly sampled, can be acquired through the angle at the scan index , the view angle of the detector, and the number of channels in the detector .
After obtaining the segmentation files of the lesions in the raw data, we saved the raw data segment through the initial scan index interval and used this as the training data for this gain experiment. It should be noted that there are different directions in the actual retrieval of raw data (i.e., from head to foot or from foot to head). We used the same spatial relationship to modify the inequality for different directions and then located the lesion.
2.6. Construction of a raw data gain model based on CT images
To explore whether the raw data contained unique information, we built residual fusion models through the raw data and fused it with the CTMs’ output to determine whether the raw data could improve the output. First, we built three feature extraction networks using the raw data. Based on the memory needed for the calculation, we sampled the index dimension of the raw data fragments containing lesions to one-eighth, and resampled the same size based on the average value by means of equal interval sampling. For the channel, we directly removed the data outside the reconstruction area from both sides and resampled with the row dimension at half the size. In the model building, we did not modify DenseNet121 (DN) [29], ResNet18 (RE) [30], or ResNeXt18 (RX) [31] in three dimensions in order to direct the direct gain of the raw data as much as possible. The training settings and parameters are detailed in Appendix A.
The primary aim of the residual fusion model was to achieve a correction of the CTM output; the origin of this idea was that the learning residual is easier, which is mentioned in ResNet. The probabilities of the CTMs predicting the patients as being positive were fused with the predicted probabilities of the raw data models (RDMs), which was performed during the training process. The name raw gain model (RGM) was given to the fusion model. All models have two output nodes, positive and negative, which were calculated separately. More specifically, the probability of predicting one patient as being positive was calculated as follows:
The probability of predicting one patient as being negative was calculated as follows:
After the output fusion of the CTM and the raw data mode, the Softmax function was used to keep the above two output sums equal to 1. In addition, the loss function was used to calculate the loss and optimize the model. The three feature extraction networks built with raw data were fused with each of the four representative CTMs described above to obtain four RGMs, for a total of 12 RGMs: RGB-DN (RGM-DN 1/2/3/4), RGM-RE (RGM-RE 1/2/3/4), and RGB-RX (RGM-RX 1/2/3/4). Next, the RGMs were compared with the CTMs to evaluate the benefits of the raw data.
2.7. Calculation of the average attention score
To calculate the average attention score of each voxel, we first used the segmentation data of the lesion in the raw data to obtain the non-lesion area by means of the unary complement. Next, we dotted and summed the segmentation data of the lesion and non-lesion areas with the attention matrix. Finally, the average attention score was obtained by dividing the total amount of attention in these two areas by the number of voxels in the segmented regions, respectively. It should be noted that we also normalized the average attention score of the lesion area and the non-lesion area in each piece of raw data to obtain a more intuitive comparison result.
3. Result
3.1. Clinical characteristics
The clinical characteristics are summarized in Appendix A Table S2. A total of 276 patients were included in this study, with 166 patients in the training cohort, 55 in the validation cohort, and 55 in the test cohort. Of the patients, 50% (n = 138) were female, and the mean age in the entire dataset was 58.48 years. Among the included patients, there were 21 (8%) cases of small cell carcinoma, 35 (13%) squamous cell carcinomas, and 149 (54%) adenocarcinomas. Among all the patients, the lesion location in most patients was identified as the right upper lobe (n = 89; 32%), followed by the left lower lobe (n = 67; 24%), and the left upper lobe (n = 64; 23%). For lung cancer diagnosis, most patients (n = 225; 82%) were evaluated as showing malignant cancer.
3.2. Performance of CT model and raw data gain model
This experiment explores the performance improvement shown by the residual fusion model based on both raw data and CT images, in comparison with the model based on CT images only. To further explore the repeatability and stability of this gain, we tested four different CT models (abbreviated as CTM 1-CTM 4) and adopted three backbone network architectures for raw data feature extraction, as follows: DN [29], RE [30], and RX [31]. For each CTM, three raw data gain models based on different backbone feature extraction networks were constructed. The performance of each RGM was compared with that of the original CTM. The receiver operating characteristic (ROC) curves and the area under the curve (AUC) of the four CTMs and the corresponding RGMs based on different backbone networks are shown in Fig. 2.
For each CTM, the residual fusion models based on different backbone networks obtained a better classification performance on the training, validation, and test cohorts. For the CTM 1 model, the fusion model that produced the maximum performance improvement for the training cohort was RGM-RX 1, with an AUC improvement of 0.051 (from 0.757 to 0.808). The fusion models that produced the maximum performance improvement for the validation cohort were RGM-RE 1 and RGM-RX 1, with an AUC improvement of 0.033 (from 0.756 to 0.789). The fusion model that produced the maximum performance improvement for the test cohort was RGM-RE 1, with an AUC improvement of 0.046 (from 0.807 to 0.853).
For the CTM 2 model, the fusion model that produced the maximum performance improvement for the training cohort was RGM-RX 2, with an AUC improvement as high as 0.109 (from 0.745 to 0.854). The fusion model that produced the maximum performance improvement for the validation cohort was RGM-DN 2, with an AUC improvement as high as 0.124 (from 0.698 to 0.822). The fusion model that produced the maximum performance improvement for the test cohort was RGM-DN 2, with an AUC improvement as high as 0.022 (from 0.760 to 0.782).
For the CTM 3 model, the fusion model that produced the maximum performance improvement for the training cohort was RGM-RX 3, with an AUC improvement as high as 0.083 (from 0.765 to 0.848). The fusion model that produced the maximum performance improvement for the validation cohort was RGM-RX 3, with an AUC improvement as high as 0.093 (from 0.760 to 0.853). The fusion model that produced the maximum performance improvement for the test cohort was RGM-RE 3, with an AUC improvement as high as 0.027 (from 0.773 to 0.800).
For the CTM 4 model, the fusion model that produced the maximum performance improvement for the training cohort was RGM-RX 4, with an AUC improvement as high as 0.035 (from 0.832 to 0.867). The fusion model that produced the maximum performance improvement for the validation cohort was RGM-DN 4, with an AUC improvement as high as 0.026 (from 0.756 to 0.782). The fusion model that produced the maximum performance improvement for the test cohort was RGM-RE 4, with an AUC improvement as high as 0.034 (from 0.833 to 0.867). Overall, using RX as the backbone network for the raw data feature extraction resulted in the maximum average performance improvement on the three cohorts.
3.3. Image feature distribution of the RGMs and gain stability analysis
We performed a t-distributed stochastic neighbor embedding (t-SNE) dimensionality reduction on all deep learning features obtained by the different feature extraction networks and counted the true positives, false positives, true negatives, and false negatives for each patient (Table 1). In addition, we assigned different colors and markers to visualize these metrics in the same coordinate system (Fig. 3). As shown in Fig. 3, the results of the various RGMs within each CTM are relatively similar, even though they come from different feature extraction networks. Table 1 shows the same situation. The gain of the RGMs inside each CTM is approaching the same trend, such as improving the malignant or benign detectable rates. Moreover, RGM-RE 1, RGM-DN 2, RGM-RE 2, RGM-DN 3, RGM-DN 4, RGM-RE 4, and RGM-RX 4 achieve a significant increase in the detectable rate of one category at the expense of a small number of the other category detectable rates.
Therefore, we calculated the optimization rate and error rate of each RGM for the CTM; we also calculated the proportion of at least two model optimizations to all optimization samples, which can reflect the stability of the raw data’s gain. All results are summarized in Appendix A Table S3. The results show that the analysis method incorporating the raw data has a high optimization rate for CTMs 1-3 that is greater than the error rate, which is also reflected in the improvement of the AUC. In addition, although different feature extraction networks were used to analyze the raw data, the proportion of at least two networks that can be optimized in each CTM is about 80%. Finally, we found that seven samples were mis-predicted within four CTMs. Of these seven samples, the input of raw data corrected the prediction results of six CTMs, and a corrected model existed in each CTM. In summary, the gain of the raw data for the CTM is very stable.
3.4. Visual statistics and analysis of the RGMs
To better explain the prediction process of the RGMs, we visualized the region of greatest interest in the RGM using gradient-weighted class activation mapping (Grad-CAM). The predictive results of the RGMs were most dependent on the information of the RGM-discovered suspicious areas. Fig. 4 illustrates the lesion masks and corresponding attention maps from different views of the raw data. From Fig. 4, it can be seen that the RGMs can always focus on the lesion areas for prediction, although the input data includes some non-lesion areas. We also calculated the average attention score of each voxel in the lesion and non-lesion areas in the raw data; the result showed that the attention score of the lesion area was 1-2 times greater than that of the non-lesion area.
3.5. Stratified analysis of different malignant subgroups
The results of the subgroup analysis for age, sex, and lesion size are shown in Table 2 and in Appendix A Table S4.
In the subgroup of patients aged 60 or lower, RGM-RX 4 and CTM 4 achieved a similar highest model performance, with AUCs of 0.837 (0.746-0.924) and 0.831 (0.749-0.904), respectively. In the over-60 subgroup, RGM-RX 4 achieved the highest model performance, with an AUC of 0.845 (0.713-0.949), outperforming the best CTM (CTM 4 with an AUC of 0.790). In the male subgroup, CTM 4 and RGM-RX 4 performed best, with similar AUCs of 0.804 (0.706-0.882) and 0.810 (0.707-0.897), respectively. In the female subgroup, RGM-RX 3 achieved the highest performance, with an AUC of 0.885 (0.818-0.945), far exceeding the best CTM (CTM 4 with an AUC of 0.823 (0.720-0.920)). In the subgroup with a lesion size less than or equal to 23 mm, RGM-RX 3 achieved the highest model performance, with an AUC of 0.847 (0.781-0.916), far exceeding the best CTM (CTM 4 with an AUC of 0.806). In the subgroup with a lesion size greater than 23 mm, CTM 4 and RGM-RE 4 showed a similar highest model performance, with AUCs of 0.819 (0.719-0.906) and 0.833 (0.703-0.925), respectively. For the lesion location subgroups, CTM 4 and RGM-RX 4 showed a similar performance in the subgroup of superior lobe of left lung, while the RGMs outperformed the CTMs, with an AUC of 0.840 versus 0.812 in the subgroup of inferior lobe of left lung, 0.849 versus 0.807 in the subgroup of superior lobe of right lung, and 0.872 versus 0.843 in the subgroup of inferior lobe of right lung.
4. Discussion
In this prospective study, we validated the potential value of raw data in real clinical practice for the first time. Interestingly, the raw data analysis showed a performance comparable with CT images, which indicates that leveraging non-image information holds promise as an alternative to image-based methods. Moreover, this study confirmed the value of adding raw data to CT images, indicating that the combination of non-image and image data will further promote the advance of disease diagnosis. This study proposed and validated a feasible method for diagnosis without image reconstruction, which has the potential to change existing imaging-based diagnosis and treatment strategies.
The classification of benign and malignant pulmonary nodules is a matter of great clinical concern [25], [32], [33]. This study explored the feasibility of raw data analysis in classifying indeterminate lung nodules greater than 2 cm in size. The results indicate that raw data can well discriminate malignant nodules from benign nodules. The AUCs of the raw data in the training cohort, validation cohort, and test cohort were 0.768 (95% confidence interval (CI) 0.681-0.851), 0.760 (95% CI: 0.558-0.922), and 0.782 (95% CI: 0.592-0.924), respectively (Fig. S2 in Appendix A), and there was no statistical difference between the performance of the raw data and that of CT. These results indicate that the classification of lung nodules may not need image reconstruction and clinician participation. However, it is still uncertain whether a convolutional neural network is the most suitable method for raw data analysis. The scanning mode and spatial structure of the raw data require a unique network structure that fits its characteristics, which is the main goal of our follow-up research. A future RDM could be applied to a wide range of grassroots hospitals that have mainstream CT systems but lack technical personnel and clinicians.
Our study shows that the introduction of raw data to CT resulted in an overall improvement over different CTMs, regardless of which backbone network was used. This indicates that raw data contains unique information that may be lost during reconstruction processing, which aligns with the results of a study by Gao et al. [23] showing that the fusion of raw data with image data resulted in an improvement in the model’s result. Moreover, the stability of the RGMs was increased on the training cohort, validation cohort, and test cohort compared with the CTMs. The combination of both non-image and image data made the model robust, which was reflected in the fact that each convolutional network indicated optimization.
The result statistics showed that RX had a better performance improvement than the other networks, which may be related to factors such as the network structure and number of parameters. By introducing group convolution, RX enables the model to learn more multiple feature representations [31], which is similar with the multi-head attention mechanism in Transformer. Nevertheless, the structural advantages of the network for analyzing raw data will need to be evaluated on more data in the future.
Our study used Grad-CAM to visualize the model decision; the results showed that the model could self-adapt to converge to the lesion and its surrounding area during the judgment. To quantify the assessment of the visualization, we calculated the average attention score and found that the mean importance of the voxels in the lesion area was about twice that of the voxels in the non-lesion area. As a gradient-based visualization method, Grad-CAM plays a strong role in the explainable field of deep networks, although it still has some defects that can be optimized [34], [35]. The question of how to associate the properties of multiple views with the network visualization algorithm is a very meaningful direction for raw data research.
We also performed intra-CT and inter-CT analyses. For intra-CTM, fused raw data prediction had a higher optimization rate than the error rate, which showed a similar gain trend, and about 80% of the optimized patients appeared in at least two feature extraction networks. For the inter-CTMs, 85% of the patients that all CTMs predicted incorrectly had optimizable RGMs within each CTM. The results demonstrated that the gain of raw data was stable across different convolutional networks and different CTM approaches. Therefore, exploring the drawbacks of post-reconstructed CT image analysis and developing models for direct diagnosis from raw data are the keys to future research. The results of the subgroup analysis showed that the RGMs performed better than the CTMs in most subgroups, especially in the subgroups of older, female, and smaller lesion size, indicating that the raw data provided valuable information that brought model gains in these subgroups, while this information may have been lost in the process of CT reconstruction.
This study has some limitations. First, it involved a small number of patients, and the proportion of positive and negative samples was unbalanced. Further study on large-scale multicenter datasets should be performed. Second, only patients with a single nodule were included in this study, so further validation of our method on patients with multiple nodules should be performed. Third, although the raw data had a comparable performance with CT, it still had a certain gap with the best CT diagnosis. Thus, there is an urgent need to develop novel AI methods specifically for raw data.
Strong computing power is a problem that cannot be ignored when calculating raw data. It is not realistic to read the complete high-frequency scanning data directly to a computing device. Designing appropriate preprocessing algorithms and building deep networks that align with the characteristics of raw data will be potential breakthrough points in the future. Finally, the CT scan scheme is designed for image reconstruction and may be not suitable for raw data analysis. Therefore, novel scan strategies, such as scanning for specific diagnostic purposes, should be developed to maximize the gain of raw data.
5. Conclusions
In summary, for the first time, we have validated the potential value of raw data in real clinical practice. Our raw data analysis showed comparable performance with CT images, indicating that leveraging non-image information holds promise as an alternative to image-based methods. Moreover, this study confirmed the value of adding raw data to CT images, demonstrating that the combination of non-images and images will further promote the advance of disease diagnosis. This study proposed and validated a new feasible direction for diagnosis without image reconstruction, which may facilitate the development of fully automated scanning and diagnostic processes.
Acknowledgments
The authors would like to acknowledge the instrumental and technical support of the multi-modal biomedical imaging experimental platform at the Institute of Automation, Chinese Academy of Sciences. This work was supported by the National Key Research and Development Program of China (2017YFA0205200, 2023YFC2415200, 2021YFF1201003, and 2021YFC2500402), the National Natural Science Foundation of China (82022036, 91959130, 81971776, 62027901, 81930053, 81771924, 62333022, 82361168664, 62176013, and 82302317), the Beijing Natural Science Foundation (Z20J00105), Strategic Priority Research Program of Chinese Academy of Sciences (XDB38040200), Chinese Academy of Sciences (GJJSTD20170004 and QYZDJ-SSW-JSC005), the Project of High-Level Talents Team Introduction in Zhuhai City (Zhuhai HLHPTP201703), the Youth Innovation Promotion Association CAS (Y2021049), and the China Postdoctoral Science Foundation (2021M700341).
Compliance with ethics guidelines
Bingxi He, Yu Guo, Yongbei Zhu, Lixia Tong, Boyu Kong, Kun Wang, Caixia Sun, Hailin Li, Feng Huang, Liwei Wu, Meng Wang, Fanyang Meng, Le Dou, Kai Sun, Tong Tong, Zhenyu Liu, Ziqi Wei, Wei Mu, Shuo Wang, Zhenchao Tang, Shuaitong Zhang, Jingwei Wei, Lizhi Shao, Mengjie Fang, Juntao Li, Shouping Zhu, Lili Zhou, Shuo Wang, Di Dong, Huimao Zhang, and Jie Tian declare that they have no conflict of interest or financial conflicts to disclose.
E.C.Ciccarelli, A.J.Jacobs, P.Berman. Looking back on the millennium in medicine. N Engl J Med, 342 (2000), pp. 42-49.
[2]
L.J.Lauwerends, P.B.A.A. vanDriel, R.J. Baatenburg deJong, J.A.U.Hardillo, S.Koljenovic, G.Puppels, et al. Real-time fluorescence imaging in intraoperative decision making for cancer surgery. Lancet Oncol, 22 (5) (2021), pp. e186-e195.
[3]
C.D.Lehman, R.D.Wellman, D.S.M.Buist, K.Kerlikowske, A.N.Tosteson, D.L.Miglioretti, the Breast Cancer Surveillance Consortium. Diagnostic accuracy of digital screening mammography with and without computer-aided detection. JAMA Intern Med, 175 (11) (2015), pp. 1828-1837.
[4]
W.L.Bi, A.Hosny, M.B.Schabath, M.L.Giger, N.J.Birkbak, A.Mehrtash, et al. Artificial intelligence in cancer imaging: clinical challenges and applications. CA Cancer J Clin, 69 (2) (2019), pp. 127-157.
[5]
A.Hosny, C.Parmar, J.Quackenbush, L.H.Schwartz, H.J.W.L.Aerts. Artificial intelligence in radiology. Nat Rev Cancer, 18 (8) (2018), pp. 500-510.
[6]
G.Litjens, T.Kooi, B.E.Bejnordi, A.A.A.Setio, F.Ciompi, M.Ghafoorian, et al. A survey on deep learning in medical image analysis. Med Image Anal, 42 (2017), pp. 60-88.
[7]
P.Lambin, R.T.H.Leijenaar, T.M.Deist, J.Peerlings, E.E.C. deJong, J. vanTimmeren, et al. Radiomics: the bridge between medical imaging and personalized medicine. Nat Rev Clin Oncol, 14 (12) (2017), pp. 749-762.
[8]
X.Liu, L.Faes, A.U.Kale, S.K.Wagner, D.J.Fu, A.Bruynseels, et al. A comparison of deep learning performance against health-care professionals in detecting diseases from medical imaging: a systematic review and meta-analysis. Lancet Digit Health, 1 (6) (2019), pp. e271-e297.
[9]
D.Killock. AI outperforms radiologists in mammographic screening. Nat Rev Clin Oncol, 17 (3) (2020), p. 134.
[10]
S.C.Rivera, X.Liu, A.W.Chan, A.K.Denniston, M.J.Calvert, H.Ashrafian, et al. Guidelines for clinical trial protocols for interventions involving artificial intelligence: the SPIRIT-AI extension. Lancet Digit Health, 2 (10) (2020), pp. e549-e560.
[11]
D.Dong, M.J.Fang, L.Tang, X.H.Shan, J.B.Gao, F.Giganti, et al. Deep learning radiomic nomogram can predict the number of lymph node metastasis in locally advanced gastric cancer: an international multicenter study. Ann Oncol, 31 (7) (2020), pp. 912-920.
[12]
Y.Q.Huang, C.H.Liang, L.He, J.Tian, C.S.Liang, X.Chen, et al. Development and validation of a radiomics nomogram for preoperative prediction of lymph node metastasis in colorectal cancer. J Clin Oncol, 34 (18) (2016), pp. 2157-2164.
[13]
W.Mu, M.B.Schabath, R.J.Gillies. Images are data: challenges and opportunities in the clinical translation of radiomics. Cancer Res, 82 (11) (2022), pp. 2066-2068.
[14]
R.J.Gillies, P.E.Kinahan, H.Hricak. Radiomics: images are more than pictures, they are data. Radiology, 278 (2) (2016), pp. 563-577.
[15]
B.Zhu, J.Z.Liu, S.F.Cauley, B.R.Rosen, M.S.Rosen. Image reconstruction by domain-transform manifold learning. Nature, 555 (7697) (2018), pp. 487-492.
[16]
G.Wang, J.C.Ye. B. De Man. Deep learning for tomographic image reconstruction. Nat Mach Intell, 2 (12) (2020), pp. 737-748.
[17]
C.Chung, J.Kalpathy-Cramer, M.V.Knopp, D.A.Jaffray. In the era of deep learning, why reconstruct an image at all?. J Am Coll Radiol, 18 (1) (2021), pp. 170-173.
[18]
G.Wang. A perspective on deep imaging. IEEE Access, 4 (2016), pp. 8914-8924.
[19]
M.Kalra, G.Wang, C.G.Orton. Radiomics in lung cancer: its time is here. Med Phys, 45 (3) (2018), pp. 997-1000.
[20]
G.Wang, J.C.Ye, K.Mueller, J.A.Fessler. Image reconstruction is a new frontier of machine learning. IEEE Trans Med Imaging, 37 (6) (2018), pp. 1289-1296.
[21]
Q. DeMan, E.Haneda, B.Claus, P.Fitzgerald, B. DeMan, G.Qian, et al. A two-dimensional feasibility study of deep learning-based feature detection and characterization directly from CT sinograms. Med Phys, 46 (12) (2019), pp. e790-e800.
[22]
WuD, KimK, DongB, LiQ. End-to-end abnormality detection in medical imaging. In:Proceedings of the 6th International Conference on Learning Representations (ICLR 2018)2018 Apr 30-May 3; Vancouve, BC, Canada. San Francisco: OpenReview; 2018..
[23]
Y.Gao, J.Tan, Z.Liang, L.Li, Y.Huo. Improved computer-aided detection of pulmonary nodules via deep learning in the sinogram domain. Vis Comput Ind Biomed Art, 2 (1) (2019), p. 15.
[24]
D.Dong, B.He, B.Kong, L.Zhang, L.Tong, F.Huang, et al. Abstract CT274: diagnosis based on signal: the first time break the routinely used circle of signal-to-image-to-diagnose. Cancer Res, 80 (Suppl 16) (2020), p. CT274.
[25]
X.Xu, C.Wang, J.Guo, Y.Gan, J.Wang, H.Bai, et al. MSCS-DeepLN: evaluating lung nodule malignancy using multi-scale cost-sensitive neural networks. Med Image Anal, 65 (2020), 101772.
[26]
D.Ardila, A.P.Kiraly, S.Bharadwaj, B.Choi, J.J.Reicher, L.Peng, et al. End-to-end lung cancer screening with three-dimensional deep learning on low-dose chest computed tomography. Nat Med, 25 (6) (2019), pp. 954-961.
[27]
L.Liu, Q.Dou, H.Chen, J.Qin, P.A.Heng. Multi-task deep model with margin ranking loss for lung nodule analysis. IEEE Trans Med Imaging, 39 (3) (2020), pp. 718-728.
[28]
Y.Xie, Y.Xia, J.Zhang, Y.Song, D.Feng, M.Fulham, et al. Knowledge-based collaborative deep learning for benign-malignant lung nodule classification on chest CT. IEEE Trans Med Imaging, 38 (4) (2019), pp. 991-1004.
[29]
HuangG, LiuZ, VanDer Maaten L, WeinbergerKQ. Densely connected convolutional networks. In: Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2017); 2017 Jul 21-26; Honolulu, HI, USA. New York City: IEEE; 4700-8.
[30]
HeK, ZhangX, RenS, SunJ. Deep residual learning for image recognition. In: Proceedings of the 2016 IEEE Conference on Computer Vision Pattern Recognition; 2016 Jun 26-Jul 1; Las Vegas, NV, USA. New York City: IEEE; 2016. p. 770-8..
[31]
XieS, GirshickR, DollárP, TuZ, HeK. Aggregated residual transformations for deep neural networks. In: Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2017); 2017 Jul 21-26; Honolulu, HI, USA. New York City: IEEE; 1492-500.
[32]
W.Shen, M.Zhou, F.Yang, D.Yu, D.Dong, C.Yang, et al. Multi-crop convolutional neural networks for lung nodule malignancy suspiciousness classification. Pattern Recognit, 61 (2017), pp. 663-673.
[33]
P.Mukherjee, M.Zhou, E.Lee, A.Schicht, Y.Balagurunathan, S.Napel, et al. A shallow convolutional neural network predicts prognosis of lung cancer patients in multi-institutional computed tomography image datasets. Nat Mach Intell, 2 (5) (2020), pp. 274-282.
[34]
ChattopadhayA, SarkarA, HowladerP, BalasubramanianVN. Grad-CAM++:generalized gradient-based visual explanations for deep convolutional networks. In: Proceedings of the 2018 IEEE Winter Conference on Applications of Computer Vision (WACV 2018); 2018 Mar 12-15; Lake Tahoe, NV, USA. New York City: IEEE; 839-47.
[35]
WangH, WangZ, DuM, YangF, ZhangZ, DingS, et al. Score-CAM:score-weighted visual explanations for convolutional neural networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2020); 2020 Jun 14-19; online. New York City: IEEE; 24-5.
RIGHTS & PERMISSIONS
THE AUTHOR
AI Summary 中Eng×
Note: Please be aware that the following content is generated by artificial intelligence. This website is not responsible for any consequences arising from the use of this content.
PDF
(2888KB)
