1. Introduction
Electroencephalography (EEG) monitors the neural activity of the brain by measuring its electrical fields with electrodes; in extracranial EEG, the electrodes are placed outside the skull, whereas in intracranial EEG (iEEG), the electrodes are implanted directly on the cerebral cortex. The complexity of EEG signals necessitates advanced signal processing and analytical methods for an accurate interpretation. Deep learning [
1] offers considerable advantages in analyzing EEG signals by learning feature representation and data abstraction to facilitate various applications, for example, epilepsy diagnosis and brain-computer interface, sleep, and cognitive monitoring [
2], [
3], [
4]. Deep learning architectures based on artificial neural networks (ANNs) are generally implemented on electronic computing platforms using central processing units (CPUs), graphics processing units (GPUs), field-programmable gate arrays (FPGAs), and application-specific integrated circuits (ASICs) [
5]. With the continuously increasing number of EEG signal channels and data scales, processing EEG signals using deep learning on electronic computing platforms is becoming more power intensive and time-consuming. Therefore, deploying these models on edge devices for portable and wearable applications is challenging owing to constraints on power supply and computing speed. In addition, the growth of electronic computing is becoming increasingly unsustainable as electronic transistors approach their physical limits [
6].
Photonic computing is considered a promising solution for future computing systems because it offers the advantages of a high energy efficiency and light-speed parallel processing [
7]. Using photonic computing, research on photonic neural networks (PNNs) and photonic integrated circuits has flourished in recent years [
8], [
9], enabling ultrafast and low-power artificial intelligence (AI) inference and providing critical support for high-performance computing in scenarios with strict limits on energy consumption. Various effective architectures for PNNs have been demonstrated, including spike neural networks [
10], convolutional neural networks [
11], [
12], reinforcement learning [
13], and reservoir computing [
14]. The linear weighted interconnection of PNNs can be implemented using a mesh of Mach-Zehnder interferometers (MZIs) [
15], diffractive surfaces [
16], [
17], [
18], [
19], [
20], [
21], [
22], and tunable attenuators connected with waveguides [
10], [
23], and so on. The nonlinear activation function of PNNs can be realized using methods such as laser-cooled atoms with electromagnetically induced transparency [
24], cavity-loaded interferometers [
25], and photodetector (PD)-driven MZIs [
26]. Currently, photonic accelerators enable massive parallel computing at speeds of trillions of multiply-accumulate operations per second [
27], [
28]. However, most existing PNNs support only simple neural network architectures with a limited number of neurons and have a lower performance in complex tasks than that of state-of-the-art electronic ANNs. Diffractive photonic computing [
16], [
19], [
21] supports millions of neurons and arbitrary linear transformations, and it can be utilized to construct more advanced architectures to perform complex AI tasks.
Epilepsy is a serious, repeated, and sudden chronic neurological disorder. Approximately one-third of cases are intractable using medication [
29]. Epilepsy patients are at high risk of serious physical injury or even death from accidental seizures. Therefore, performing EEG analysis to obtain automated and portable seizure detection devices is critical for alerting caregivers and protecting patients’ lives. Several seizure detection methods based on statistical features and machine learning classifiers have been proposed [
30], [
31], and ANN-based methods [
32], [
33] have achieved state-of-the-art detection performance. In contrast to the energy-intensive ANNs, PNNs can overcome the limitations of traditional electronic edge devices with respect to computing power [
34] and provide promising opportunities to design portable and power-saving seizure detection devices. However, no such devices been demonstrated to date.
Herein, we propose the first photonic architecture to process EEG signals and to detect epileptic seizures with high accuracy. The proposed EEG opto-processor, based on diffractive photonic computing units (DPUs), implements a PNN architecture to detect epileptic seizures using EEG and iEEG measurements (
Fig. 1). We developed a three-dimensional (3D) free-space DPU with a highly compact optical system, in which a high-data-throughput spatial light modulator (SLM) was used to implement input data encoding and diffractive modulation simultaneously (
Fig. 1(b)). Diffractive deep neural networks (D
2NNs) were constructed with a DPU by implementing the computational operations of diffractive-weighted interconnections optically, using photoelectric conversion for nonlinearity activation, and controlling the data flow electronically. Furthermore, we designed a two-dimensional (2D) integrated DPU with a metaline-based multilayer structure [
22], [
35], [
36], [
37] on a silicon photonics platform to achieve higher integration density and lower power consumption for wearable devices (
Fig. 1(c)). We also introduced an optical bias block to the design and validated its effectiveness in unbalanced classification tasks. We demonstrated that the DPU-based D
2NNs successfully processed 23-channel EEG and 30-channel iEEG signals and detected epileptic seizures with an accuracy of 98.96% on the Children’s Hospital Boston (CHB)-Massachusetts Institute of Technology (MIT) dataset [
38], [
39] and 94.49% on the Epilepsy-iEEG-Multicenter-Dataset [
40]. In addition, a channel selection scheme using random forest was developed to select the most important channels, and it was evaluated theoretically and experimentally. The system proposed in this study may aid in achieving economical, efficient, and comfortable personalized medicine for epileptic seizure detection.
2. Materials and methods
2.1. Epileptic seizure detection with EEG or iEEG signals
In the proposed design, we use photonic computing to analyze and detect epileptic seizures from recorded EEG signals (
Fig. 1(a)). EEG is one of the most used methods for monitoring brain activity and is considered the best indicator for epilepsy diagnosis and analysis [
41]. Extracranial EEG measures the voltage difference between scalp-mounted electrodes caused by ionic flow in brain neurons, whereas iEEG is the neuroelectrophysiological signal obtained from implanted electrodes. Different channels from the electrodes at the corresponding positions reflect the spatial and temporal information of the brain activity. Epileptiform EEG or iEEG patterns, such as spikes and sharp waves, contribute to seizure diagnosis. The effectiveness of the proposed EEG opto-processor was validated using both the EEG and iEEG modalities.
2.2. Design of the 3D free-space DPU
We developed a 3D free-space DPU to execute the task by preprocessing the EEG signals into 2D images for analysis using the constructed large-scale D
2NN architecture [
21], as shown in
Fig. 1(b). The main components include a large-scale reconfigurable SLM and an optoelectronic detector that can be programmed to support millions of diffractive neurons. The input data are encoded in the phase of the optical field to the input nodes and modulated via the diffractive layer parameters that can be trained according to specific AI tasks. The input nodes are connected to the output neurons via optical diffraction, and trainable synaptic weights are generated via the diffractive modulation. A nonlinear activation function occurs during the photoelectronic conversion of the diffractive computation results as the intensity of the complex fields and intensity-to-phase conversion at the next input layer are measured. The data flow is controlled electronically to build a multilayer PNN architecture that performs video-rate epileptic seizure detection. In contrast to the design described in Ref. [
21], both input data encoding and phase modulation are implemented with an SLM, which eliminates the additional input data encoding module and related relay optics, thereby significantly reducing the system complexity.
The experimental setup is shown in
Figs. 2(a) and
(b). A compact green laser diode (CPS532, Thorlabs, Inc., USA) coherent light source with a wavelength of 532 nm was used to generate the input optical field. The light beam was collimated and expanded using two lenses (AC050-008 and AC254-100, Thorlabs, Inc.), polarized using a polarizer (LPVISA050, Thorlabs, Inc.), and split using a beam splitter (CCM1-BS013, Thorlabs, Inc.). An SLM (P1920-400-800, Meadowlark Optics, Inc., USA) modulated the phase of the wavefront according to the input information and trained parameters of the D
2NN. The optical field was captured and photoelectrically converted using a complementary metal-oxide semiconductor (CMOS) sensor (GS3-U3-41C6M-C, Teledyne FLIR, Inc., USA) after a diffraction propagation of 10 cm. This process was multiplexed and programmed to construct an
N-layer D
2NN.
The diffractive computing process can be represented as
, where
, , , and
denote the input, trainable modulation coefficient, free-space propagation operator at distance
d (Section S1 in Appendix A), and output of the
ith layer of the D
2NN, respectively.
is fed to the SLM to modulate the phase of the light wavefront, and
is the light intensity acquired by the CMOS sensor (i.e., the diffraction computing results).
is the 2D feature map scaled to [0, 2π] of the EEG or iEEG signals, and
is obtained from
for
i>1 by a nonlinear activation function, given as
, where
and
are two trainable parameters. The incident light, which was collimated and expanded by the lenses, was circular in the experimental setup. To ensure consistency between the experimental and simulation results, we loaded blazed grating patterns onto the SLM to ensure that the incident light outside the region of interest (ROI) of
propagated in four oblique directions. As shown in Appendix A Fig. S1, the light outside the ROI diffracts away from the optical axis and does not interfere with the ROI diffraction. Finally,
represents the output plane of the
N-layer D
2NN, and the target region with the highest optical intensity indicates the epileptic seizure detection result.
Fig. 2(c) shows the diffractive computing process for a two-layer D
2NN. The network diffractive modulation coefficients were trained using pre-collected and labeled data. The mean squared error (MSE) between
and the ground truth, that is, 1 for the target detection region and 0 for the other regions, was defined as the loss function. The forward model of the D
2NN was simulated using angular spectrum propagation methods, and the errors were backpropagated to minimize the loss function by optimizing
, , and
using stochastic gradient descent algorithms.
We constructed a two-layer D2NN for epileptic seizure detection. For each input signal sequence, two target regions representing the seizure and non-seizure states were configured at the network output plane, where the target region with a higher optical intensity indicated the epileptic seizure detection result. Furthermore, we developed an adaptive training approach to deal with systematic errors, such as the instability of the laser power, optical aberration of the collimating lenses, phase modulation error of the SLM, device misalignment between the SLM and sensor, shot noise, and readout noise of the sensor. In the two-layer D2NN experiments, we deployed the network parameters of the first layer with the pretrained model and captured the experimental outputs. Subsequently, the modulation coefficients of the second layer were retrained with the experimental outputs of the first layer to correct the systematic errors of the first layer. To address the systematic errors in the second layer, for the two intensities of the detection regions obtained from the output plane of the D2NN after the experiments on the second layer, the intensity of the first detector region was multiplied by a factor c within [0.9,1.1]. The factor c was optimized to maximize classification performance on the training dataset, which was fixed in the inference process. This factor contributes to the errors caused by uneven light illumination. After training to learn the diffractive modulation coefficients, the D2NN performed light-speed inference, offering video-rate and energy-efficient epileptic seizure detection for patients.
2.3. Design of the 2D integrated DPU
We propose the design of a 2D integrated DPU based on metamaterials [
22], [
35] and on-chip optical devices to further reduce the system size and improve the computational efficiency of wearable EEG analysis, as shown in
Fig. 1(c). Specifically, coherent light from an on-chip laser [
42] is split into different single-mode waveguides using an array of multimode interferometers (MMIs) or Y-couplers. The one-dimensional (1D) features of EEG signals are encoded on the light amplitude in waveguides with on-chip modulators, such as MZIs. The weighted interconnections between the complex optical fields of the input and output waveguides are achieved with a diffractive computing block comprising multilayer metalines, which are passive optical structures for high-energy-efficiency computations. Each metaline is a 1D etched rectangular silica slot (meta-atom) array in the silicon membrane of a silicon-on-insulator (SOI) substrate. Both the amplitude and phase modulation coefficients of a meta-atom are learnable and can be programmed by adjusting the width and length of the slot. We trained only the width in this study (Fig. S2 in Appendix A). The diffractive computing block can be expanded vertically to receive higher-dimensional input features and horizontally to increase the diffractive parameters and learning capacities.
The integrated DPU was operated at a wavelength of 1550 nm. Because the modulation coefficients of each meta-atom were calculated with the periodic boundary condition using finite-difference time-domain (FDTD) simulations, we used binary modulation and combined three identical meta-atoms as one diffractive modulation neuron to facilitate fabrication and improve model accuracy. The height of the meta-atom was fixed at 400 nm, and the slot period was 300 nm. The slot width was chosen from 0 or 100 nm to implement binary modulation, and the corresponding phase modulation coefficients were 0 or −1.55 rad (Figs. S2 and S3 in Appendix A).
In addition to the weighted interconnection block, we introduced an optical bias block to enhance model capability. The bias coefficients were modulated at different wavelengths and coupled to the two output waveguides of the diffractive computing block. The incoherent summation of optical energy was collected using PDs to indicate the epileptic seizure detection results. The bias modules enable the adjustment of output thresholds, which is especially effective for imbalanced data [
43], that is, when the presence of the majority class is more prominent than that of the minority class in the dataset. For example, in an epileptic seizure detection task, the seizures of patients constitute a minority class, whereas the non-seizures are the majority.
3. Results
3.1. Epileptic seizure detection with the free-space DPU
We validated the effectiveness of the proposed optical epileptic seizure detection approach using EEG signals from the CHB-MIT dataset [
38], [
39] of Children’s Hospital Boston, Massachusetts Institute of Technology. It contains hours of seizure and non-seizure EEG recordings of 23 patients obtained using the 10-20 international system of EEG electrode placement. The CHB-MIT dataset is widely used as a benchmark for epileptic seizure detection [
31], [
32], [
33], [
44]. As in Ref. [
33], we selected the EEG recordings of eight patients with a sufficient seizure duration (more than 400 s) over 23 channels to generate the training and testing sets for the D
2NN. Signals from seizures and randomly selected two-hour non-seizures were used for evaluation. All EEG signals were segmented into one-second windows and randomly arranged. Half of the seizures and non-seizures of the same duration were used for training, whereas the remaining data were used for testing. The 1D vectors of the time-series EEG voltage signals were preprocessed to extract and generate effective 2D features as the D
2NN input. Because epileptic EEG signals are nonstationary and multicomponent, extracting meaningful statistical features from raw 1D time series signals is critical for their detection. We adopted the short-time Fourier transform (STFT) to preprocess the raw signals and extract effective 2D features containing both time and frequency characteristics [
45], [
46]. STFT can be easily implemented in wearable devices using digital electronics, and optical solutions have also been demonstrated to achieve high speeds and bandwidths [
47], [
48]. Epileptic seizures cause changes in certain frequency bands, such as the
δ (0.4-4.0 Hz),
θ (4.0-8.0 Hz),
α (8.0-12.0 Hz),
β (12.0-30.0 Hz), and
γ (30.0-70.0 Hz) bands [
45], [
49]. The energy of the EEG signals is mainly concentrated in the low-frequency bands; therefore, we set the frequency range of the STFT to 0-50 Hz and the sliding window width to 25 samples to evaluate the free-space DPU. For each time window of the EEG signals, signals from different channels were transformed into STFT spectra, stitched together into one image, and fed to a D
2NN to detect epileptic seizures. The spectral energy was normalized and the 2D features were resized to 400×400 pixels as the D
2NN input.
Selecting informative EEG channels is critical for building wearable healthcare devices and for achieving personalized medicine for epileptic seizure detection. Generalized epileptic seizures involving the entire brain can be seen in every channel of the EEG recordings, whereas partial seizures can be seen only in a few channels whose corresponding electrodes are near the lesion. Full-channel EEG monitoring in wearable devices is associated with considerable challenges because it is expensive, time-consuming, computationally intensive, and uncomfortable. Moreover, invalid channels add noise to the signals and make detection more difficult. Channel selection can be achieved by medically diagnosing the location of the lesion or using statistical methods [
44], [
50]. This process should be completed only in the preliminary analysis, and it is not repeated in subsequent epilepsy monitoring.
We applied a random forest channel selection method [
44] to analyze the D
2NN performance for EEG analysis on the CHB-MIT dataset with different channel numbers (
Fig. 3). Feature selection with a random forest combines the qualities of the filter and wrapper methods and offers advantages such as higher accuracy, less overfitting, better generalization performance, and easy interpretability [
50], [
51]. The channel selection pipeline is shown in
Fig. 3(a), where the EEG signals were first divided into subsequences with a one-second time window. Next, the power spectral density of each EEG channel in each time window was calculated, representing various bands (i.e.,
δ,
θ,
α,
β,and
γ). The signals of each time window were utilized as samples with 115 attributes (23 channels and five bands). A random forest [
52] containing 1000 decision trees was used to learn the features of each sample, that is, a vector with 115 attributes, and to detect epileptic seizures, which has been proven to be robust to learning irrelevant features [
53]. During the process of updating the random forest, that is, minimizing the sum of the impurities, redundant features are not selected for node splitting, and informative features appear more frequently in those trees. Therefore, the feature contribution percentage can be obtained by investigating the normalized total reduction of the impurities in the random forest caused by that feature, which is also known as the Gini importance. The channel contribution percentage is the sum of the five feature contribution percentages. For example,
Fig. 3(a-iv) illustrates the contribution percentages of the 23 channels for patient chb01 (Table S1 in Appendix A); the most important channel is 17. The channels with the highest contribution percentage were selected for seizure detection using the D
2NN. In addition, the random forest is also an effective machine learning classifier for epileptic seizure detection [
31] and serves as a performance reference for the D
2NN-based approach.
Using channel selection, we evaluated the performance of D
2NN on the CHB-MIT dataset. The statistical evaluations, including accuracy, sensitivity, specificity, and
score, are used for evaluating the binary classification performance (Section S2 in Appendix A). In the medical domain, the
score (typically
β=0.5,1.0,2.0) is a weighted harmonic mean of the precision and recall that is more useful than accuracy, especially when the dataset is imbalanced [
54]. We adopted the
F2 score (i.e.,
β=2), to enable the recall to have a larger weight, considering the identification of disease episodes constituting only a minority of cases [
55]. We investigated the performance of the D
2NN under different layer numbers with one-channel signals selected using the random forest (Fig. S4 in Appendix A). All simulation models were numerically implemented using Python (v3.6.13) and TensorFlow (v1.11.0) running on a desktop computer (Nvidia TITAN XP GPU, AMD Ryzen Threadripper 2990WX CPU with 32 cores, 128 GB of random access memory (RAM), and Microsoft Windows 10 operating system). The modulation coefficients of the diffractive layers were optimized using stochastic gradient descent and error backpropagation. The Adam optimizer and a learning rate of 0.01 were used with the loss functions of MSE or cross-entropy. Each D
2NN was trained for 1000 epochs. The training time of the free-space D
2NN was approximately 4 h. With 0.16 million diffractive neurons in each layer, the average accuracy, sensitivity, specificity, and
F2 score were 97.89%, 91.92%, 98.13%, and 0.8553, respectively, for the two-layer D
2NN, which are comparable to those of the three-layer model. A further increase in the number of layers exhibited minimal performance improvement. In addition, the one-layer D
2NN did not converge during training and was not suitable for this task. The same conclusion was drawn for larger channel numbers. Therefore, the two-layer D
2NN model was adopted for the numerical evaluations and experiments.
The accuracy and
F2 score of the D
2NN numerical model for various channel numbers are visualized as box diagrams in
Figs. 3(b) and
(c), respectively. We compare the performance of three methods, Rand-D
2NN, RF-D
2NN, and RF-RF, which denote random channel selection and classification with the D
2NN, channel selection with a random forest and classification with D
2NN, and simultaneous channel and classification with the random forest, respectively. Both the accuracy and
F2 score of RF-D
2NN outperformed those of Rand-D
2NN and RF-RF when a few channels were used, demonstrating the effectiveness of the channel selection method using the random forest. The performance of all models improved with an increase in the channel number, particularly for the Rand-D
2NN and RF-RF models. The RF-D
2NN model achieved a sufficiently high performance using one-channel EEG signals with an average accuracy of 97.89% and an
F2 score of 0.8553, which are close to those of the full-channel RF-D
2NN model with an average accuracy of 98.96% and an
F2 score of 0.9161. The full-channel RF-RF achieved an average accuracy of 98.88% and an
F2 score of 0.9335. In addition, the one-channel RF-D
2NN performed relatively stably in different patients, that is, the accuracy varied from 95.43% to 99.31% and the
F2 score varied from 0.7351 to 0.9261. The same conclusion was drawn for the sensitivity and specificity metrics (Fig. S5 in Appendix A). The simulation results verify that the D
2NN with random forest for channel selection can fully use the information of one-channel signals to perform high-quality epileptic seizure detection.
We conducted experiments using a two-layer model and single-channel EEG signals to facilitate efficient computation and convenient signal acquisition. The results are shown in
Fig. 4. To eliminate systematic errors, the modulation coefficients of the second layer were fine-tuned via adaptive training using the experimental results of the first layer. Additionally, the intensities of the two target detection regions were multiplied by two factors, one for each region, to reduce the impact of uneven illumination and improve the detection accuracy.
Fig. 4(a) illustrates the experimental results of the D
2NN on patient chb01 using one-channel EEG signals. As expected, the D
2NNs successfully distinguished between the seizure and non-seizure samples and concentrated the light into predefined regions. The image characteristics of the experimental results aligned closely with those of the simulation results.
Fig. 4(b) illustrates the pretrained parameters of the two-layer D
2NN and the fine-tuned parameters of the second layer.
Fig. 4(c) shows the normalized intensity of the two detector regions for 200 seizures and 200 non-seizures. This distinct difference rendered the system robust to small noise disturbances. Furthermore, we evaluated the numerical and experimental performance of the one-channel model for all patients. As shown in
Fig. 4(d), the experimental accuracy, sensitivity, specificity, and
F2 score were 96.84% ± 1.79%, 92.81% ± 6.74%, 96.98% ± 1.87%, and 0.8174 ± 0.0868, respectively, which are comparable with the respective values 97.89% ± 1.22%, 91.92% ± 7.62%, 98.13% ± 1.27%, and 0.8553 ± 0.0753 in the simulation. The accuracy, specificity, and
F2 score were reduced by approximately 1%, 1%, and 0.04, respectively, whereas the sensitivity did not decrease in the experiments. Detailed results for each patient are shown in Appendix A Table S2, verifying the accuracy and effectiveness of the one-channel approach and experimental setup.
In addition to the EEG modality, we utilized a DPU-constructed D
2NN to detect epileptic seizures using iEEG signals. The performance was evaluated using electrocorticography (ECoG) recordings from six patients selected from the Epilepsy-iEEG-Multicenter-Dataset [
40] with a sufficient seizure duration and clear onset and offset labels (Tables S1 and S2). They were segmented using five-second sliding windows with an overlap of 4 s and were randomly arranged. The training and testing sets were divided in the same manner as that for the EEG recordings. For the evaluation of the free-space DPU, the frequency range of the STFT was set to 0-50 Hz because of the higher sampling frequency of the iEEG signals, and the sliding window width was set to half of the sampling frequency. For the evaluation of the integrated DPU, each time window of one-second signals was divided into four parts, and the energy of the four spectral bands (0-6, 6-14, 14-22, and 22-30 Hz) in each part was calculated to form a 1D feature with 16 attributes. Each iEEG signal sequence of these patients has a time duration of a few minutes near an epileptic seizure with large numbers of seizure signals, resulting in a more difficult detection than that with the CHB-MIT dataset, which contains long-duration non-seizure periods. We adopted the same feature extraction, channel selection method, and D
2NN model as for the EEG signals; the performance corresponding to various channel numbers is shown in
Fig. 5(a). With channel number settings of 1, 5, 10, and 30, the average accuracies were 90.93%, 94.35%, 94.31%, and 94.49%, respectively; average sensitivities were 92.50%, 93.32%, 93.51%, and 94.03%, respectively; average specificities were 90.38%, 96.06%, 96.53%, and 95.68%, respectively; and average
F2 scores were 0.9132, 0.9341, 0.9351, and 0.9395, respectively. The performance improved slowly with an increase in channel number. In addition, the one-channel performance was sufficiently high and close to the 30-channel performance. Therefore, as with the detection of epileptic seizures using EEG signals, we adopted single-channel iEEG signals to facilitate signal acquisition and system simplification. As shown in
Fig. 5(b), the experimental accuracy, sensitivity, specificity, and
F2 score were 87.51% ± 4.65%, 89.00% ± 5.86%, 87.35% ± 7.91%, and 0.8755 ± 0.0483, respectively, which are also comparable with 90.93% ± 3.58%, 92.50% ± 4.51%, 90.38% ± 7.07%, and 0.9132 ± 0.0340, respectively in the simulation. The accuracy, sensitivity, specificity, and
F2 score were reduced by approximately 3%, 3%, 5%, and 0.04, respectively in the experiments. Detailed results for each patient are presented in Table S2. The generalization ability of the model for different patients is further demonstrated in Appendix A Fig. S6. Furthermore, the performance of the DPU-constructed D
2NN is compatible with the state-of-the-art convolutional neural network AlexNet [
56] (
Table 1 and Fig. S7 in Appendix A).
3.2. Epileptic seizure detection with the integrated DPU
The integrated DPU comprises a one-layer metaline for the on-chip epileptic seizure detection. There are 1800 meta-atoms (i.e., 600 diffractive neurons), for processing EEG or iEEG signals, and the DPU width is 540 μm. The distance between the input plane and output plane is 200 μm, and the metaline is placed at the center. The input waveguides are placed with an interval of 15 μm, and the interval of two output waveguides is 270 μm.
We applied the integrated DPU for epileptic seizure detection to EEG signals from the CHB-MIT dataset. To reduce the input port numbers of the DPU, the time and frequency resolution of the STFT is reduced during the pre-processing, and only one channel selected by the random forest is utilized to generate a 16-dimensional feature for each time window. We trained the integrated DPU with 16 input waveguides.
Fig. 6(a) illustrates the details of the optical field propagation simulated using FDTD. Sufficiently matched modulation profiles between the analytical model and FDTD are shown in Fig. S3. We numerically evaluated the integrated DPU with an optical bias block, as shown in
Fig. 6(b) and Table S3 in Appendix A. When the same training platform as the free-space D
2NN was used, the training time of the integrated D
2NN model was approximately 1 h. The average accuracy, sensitivity, specificity, and
F2 score were 97.68%, 88.92%, 98.01%, and 0.8385, respectively, but they decreased to 89.11%, 76.61%, 89.61%, and 0.5173, respectively, without the optical bias block. In addition, the performance fluctuated significantly in different patients without the optical bias block. We compared the performance with a three-layer electronic fully connected neural network (3-layer FC), which contained 64 and 16 neurons in the first and second hidden layers, respectively, two output neurons, and a rectified linear unit (ReLU) activation function. The average accuracy, sensitivity, specificity, and
F2 score of the 3-layer FC were 98.24%, 89.32%, 98.61%, and 0.8547, respectively. The results demonstrate the effectiveness of the optical bias block for integrated DPU and the success of its application for epileptic seizure detection.
Similarly, the results of the iEEG signals from the Epilepsy-iEEG-Multicenter-Dataset are presented in
Fig. 6(c) and Table S3. The average accuracy, sensitivity, specificity, and
F2 score were 86.07%, 82.82%, 86.90%, and 0.8328, respectively, but they decreased to 80.60%, 76.45%, 84.70%, and 0.7665, respectively, without the optical bias block. Compared with the CHB-MIT dataset, less performance degradation was observed because the durations of the seizure and non-seizure periods were similar in this dataset. The 3-layer FC achieved an average accuracy, sensitivity, specificity, and
F2 score of 84.50%, 81.88%, 86.48%, and 0.8196, respectively, which were also close to those of the integrated DPU with an optical bias block. Performance comparisons with state-of-the-art electronic neural networks are summarized in
Table 1.
4. Discussion
4.1. Computing speed and energy efficiency
We evaluated the computational performance of a 3D free-space DPU using the constructed two-layer D2NN. Each layer of the D2NN generates a feature map with element numbers of 400×400 from an input image of the same size, which performs a 4002 × 4002 matrix multiplication, that is, the 5.12×1010 operations of pixel-by-pixel multiplication and summation in the optical domain. Because diffractive optical computations are performed at the speed of light, the computing speed is determined by the refresh rate of the SLM and acquisition frame rate of the camera. The SLM frame rate was 30 Hz and the camera exposure time was 1 ms in the experiments. The STFT pre-processing and data-flow controlling time was 2.78 ms. Therefore, the total system latency of each working cycle was 37.11 ms, corresponding to the system frame rate of 27 Hz. The computing speed of the free-space DPU was 1.38×1012 operations per second (1.38 TOPS). The powers of the laser, SLM, camera, and controller were 1.65, 12.00, 4.50, and 65.00 W, respectively; that is, the total system power was 83.15 W and the energy efficiency was 0.02 TOPS/W. Furthermore, the SLM in the free-space DPU system supported 1920×1152 diffractive neurons (i.e., 9.78×1012 operations in each cycle of diffractive computing). Consequently, the maximum diffractive computation speed was 264.06 TOPS and the maximum energy efficiency was 3.17 TOPS/W. With a high-speed SLM (e.g., HSP1920-600-1300-HSP8, Meadowlark, 422.4 Hz), the energy efficiency could be further enhanced more than ten-fold.
The 2D integrated DPU in
Fig. 1(c) executes a 16×2 weight matrix with passive metalines, that is, 64 operations in each cycle. Considering the modulation and photodetection rate of 30 GHz based on the existing silicon photonic foundry, the computing speed was 1.92 TOPS. Typical powers of the on-chip laser source and modulators were 10 and 15 mW, respectively [
57]. The integrated DPU required two lasers and 18 modulators. Therefore, the total power was 290 mW and energy efficiency was 6.62 TOPS/W. The energy efficiency of the state-of-the-art GPU Tesla V100 is 0.4 TOPS/W (Table S4 in Appendix A). The integrated DPU therefore achieves more than 15-fold improvement in energy efficiency.
4.2. Limitations and future works
We demonstrated a DPU-based opto-processor for effectively processing large-scale EEG signals, which can facilitate its application in different areas. For example, in brain-computer interfaces [
58], the iEEG signal channel counts are increasing from thousands to hundreds of thousands, placing a high demand for high-performance computing processors. Furthermore, we demonstrated that the one-channel EEG signal with the channel selection method exhibited sufficiently high performance for epileptic seizure detection, that is, only one electrode attached to the scalp is required for the acquisition system, facilitating its applications in wearable healthcare and personalized medicine. The volume of a free-space DPU can be reduced by designing ASICs for EEG signal acquisition and data-flow control. Time-frequency-based methods are more efficient than spiking-based engines because of feature extraction in both the time and frequency domains and more comprehensive mining of EEG signals [
45]. In addition, different time resolutions can be achieved by adjusting the time window length of the STFT to achieve a local or global understanding of EEG signals.
The reconfigurable free-space DPU includes millions of diffractive neurons, which can construct more complex neural network architectures, such as diffractive recurrent neural networks [
21], for better performance in processing time-series signals and various health monitoring tasks. The integrated implementation can be fabricated using deep-ultraviolet lithography, which is highly compatible with electronic integrated circuits. Reconfigurable non-volatile materials such as phase change materials (PCM) [
27], [
59] can be used to design metalines with programmable network parameters and increased flexibility. Although the one-channel model was primarily considered in this study, more channels may be necessary for tasks that are more complex. The data throughput of the proposed EEG opto-processor can be further expanded based on wavelength-division multiplexing (WDM) [
27], [
28], and optical processing of various channels can be accomplished at individual wavelengths.
5. Conclusions
In summary, we demonstrated the successful use of a DPU to construct PNNs for EEG analysis and applied it for epileptic seizure detection of EEG and iEEG signals with high performance. This study is anticipated to facilitate advances in the use of photonic computing for the development of various healthcare devices.
Acknowledgments
This work was supported by the National Major Science and Technology Projects of China (2021ZD0109902 and 2020AA0105500), the National Natural Science Fundation of China (62275139 and 62088102), and the Tsinghua University Initiative Scientific Research Program.
Compliance with ethics guidelines
Tao Yan, Maoqi Zhang, Hang Chen, Sen Wan, Kaifeng Shang, Haiou Zhang, Xun Cao, Xing Lin, and Qionghai Dai declare that they have no conflicts of interest or financial conflicts to disclose.
Appendix A. Supplementary material
Supplementary data to this article can be found online at
https://doi.org/10.1016/j.eng.2024.01.008.
PDF
(3063KB)
