Journal Home Online First Current Issue Archive For Authors Journal Information 中文版

Strategic Study of CAE >> 2024, Volume 26, Issue 1 doi: 10.15302/J-SSCAE-2024.01.011

Device-Cloud Collaborative Intelligent Computing: Key Problems, Methods, and Applications

1. School of Software Technology, Zhejiang University, Hangzhou 310013, China;

2. College of Computer Science and Technology, Zhejiang University, Hangzhou 310013, China;

3. Taobao (China) Software Co., Ltd., Hangzhou 310012, China;

4. Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai 200240, China;

5. Shanghai Institute for Advanced Study of Zhejiang University, Shanghai 201203, China

Funding project:科技创新2030—“新一代人工智能”重大项目“大小模型端云协同进化与系统”(2022ZD0119100);中国工程院咨询项目“新一 代人工智能及产业集群发展战略研究”(2022-PP-07) Received: 2023-11-10 Revised: 2023-12-20

Next Previous

Abstract

Device-cloud collaborative intelligent computing, an emergent result of the development in big data, cloud computing, and edge computing, offers significant improvements in data utilization while protecting user privacy. This approach synergizes the realtime response capabilities of intelligent computing with service robustness. The study explores the application value of this computing paradigm, highlighting technical challenges such as optimizing on-device learning efficiency, mitigating overfitting with limited samples at the device, customizing on-device models, learning false associations under distributional discrepancies, and balancing communication overhead with computational efficiency. We systematically review the progress in mainstream methods within devicecloud collaborative intelligent computing, encompassing efficient computation hardware as the application cornerstone, device-centric collaborative computing, cloud-centric collaborative computing, bidirectional device-cloud collaborative computing, and trustworthy device-cloud collaborative computing. The study also summarizes applications in vertical domains such as recommendation systems, autonomous driving, security systems, and  educational models. Looking toward the future of device-cloud collaborative intelligent computing, it underscores the need for focused research on cloud resource application strategies in device model personalization, multi-objective optimization algorithms for device-cloud collaboration, and optimized collaborative strategies between devices and the cloud.

Figures

图1

图2

图3

References

[ 1 ] 中华人民共和国国民经济和社会发展第十四个五年规划和2035年远景目标纲要 [EB/OL]. (2021-03-13)[2023-11-15]. https://www.gov.cn/xinwen/2021-03/13/content_5592681.htm.
The outline of the Fourteenth Five-year Plan for national economic and social development of the People´s Republic of China and the vision 2035 [EB/OL]. (2021-03-13)‍[2023-11-15]. https://www.gov.cn/xinwen/2021-03/13/content_5592681.htm.

[ 2 ] 张依琳, 梁玉珠, 尹沐君, 等‍. 移动边缘计算中计算卸载方案研究综述 [J]. 计算机学报, 2021, 44(12): 2406‒2430.
Zhang Y L, Liang Y Z, Yin M J, et al. Survey on the methods of computation offloading in mobile edge computing [J]. Chinese Journal of Computers, 2021, 44(12): 2406‒2430.

[ 3 ] 贾晓千, 陈刚, 李白冰‍. 边缘计算在视频侦查中的应用 [J]. 计算机工程与应用, 2020, 56(17): 86‒92.
Jia X Q, Chen G, Li B B. Application of edge computing in video investigation [J]. Computer Engineering and Applications, 2020, 56(17): 86‒92.

[ 4 ] 曹行健, 张志涛, 孙彦赞, 等‍. 面向智慧交通的图像处理与边缘计算 [J]. 中国图象图形学报, 2022, 27(6): 1743‒1767.
Cao X J, Zhang Z T, Sun Y Z, et al. The review of image processing and edge computing for intelligent transportation system [J]. Journal of Image and Graphics, 2022, 27(6): 1743‒1767.

[ 5 ] 王智, 夏树涛, 毛睿‍. 基于边缘智能的沉浸式元宇宙关键技术与展望 [J/OL]. 大数据, [2023-11-13]. https://link.cnki.net/urlid/10.1321.G2.20231110.1017.004.
Wang Z, Xia S T, Mao R. Edge-intelligence-based immersive metaverse: Key technologies and prospects [J/OL]. Big Data Research, [2023-11-13]. https://link.cnki.net/urlid/10.1321.G2.20231110.1017.004.

[ 6 ] 高祥云, 孟丹, 罗明凯, 等. 支持隐私保护的端云协同训练 [J]. 华东师范大学学报 (自然科学版), 2023 (5): 77‒89.
Gao X Y, Meng D, Luo M K, et al. Privacy-preserving cloud-end collaborative training [J]. Journal of East China Normal University (Natural Science), 2023 (5): 77‒89.

[ 7 ] 杨洋, 况琨, 陈政聿, 等. 基于端云协同体系的预训练大模型及其服务化 [J]. 人工智能, 2022, 9(6): 103‒120.
Yang Y, Kuang K, Chen Z Y, et al. Pre-training model based on end-cloud collaboration system and its service [J]. AI-View, 2022, 9(6): 103‒120.

[ 8 ] Lecun Y, Denker J S, Solla S A, et al. Optimal brain damage [C]. Denver: Advances in Neural Information Processing Systems 2, 1989.

[ 9 ] Han S, Mao H Z, Dally W J. Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding [EB/OL]. (2015-10-01)[2023-11-15]. https://arxiv.org/abs/1510.00149.

[10] McMahan H B, Moore E, Ramage D, et al. Communication-efficient learning of deep networks from decentralized data [EB/OL]. (2016-02-17)[2023-11-15]. https://arxiv.org/abs/1602.05629.

[11] Li Y G, Yu R, Shahabi C, et al. Diffusion convolutional recurrent neural network: Data-driven traffic forecasting [EB/OL]. (2017-07-06)[2023-11-15]. https://arxiv.org/abs/1707.01926.

[12] Li T, Sahu A K, Talwalkar A, et al. Federated learning: Challenges, methods, and future directions [J]. IEEE Signal Processing Magazine, 2020, 37(3): 50‒60.

[13] Yao J C, Wang F, Jia K Y, et al. Device-cloud collaborative learning for recommendation [EB/OL]. (2021-04-14)[2023-11-15]. https://arxiv.org/abs/2104.06624.

[14] Sim K C, Zadrazil P, Beaufays F. An investigation into on-device personalization of end-to-end automatic speech recognition models [EB/OL]. (2019-09-14)[2023-11-15]. https://arxiv.org/abs/1909. 06678.

[15] Finn C, Abbeel P, Levine S. Model-agnostic meta-learning for fast adaptation of deep networks [EB/OL]. (2017-03-09)[2023-11-15]. https://arxiv.org/abs/1703.03400.

[16] Tan A Z, Yu H, Cui L Z, et al. Towards personalized federated learning [J]. IEEE Transactions on Neural Networks and Learning Systems, 2023, 34(12): 9587‒9603.

[17] Deng S G, Zhao H L, Fang W J, et al. Edge intelligence: The confluence of edge computing and artificial intelligence [J]. IEEE Internet of Things Journal, 2020, 7(8): 7457‒7469.

[18] Pimenidis E, Polatidis N, Mouratidis H. Mobile recommender systems: Identifying the major concepts [J]. Journal of Information Science, 2019, 45(3): 387‒397.

[19] Shuja J, Bilal K, Alasmary W, et al. Applying machine learning techniques for caching in next-generation edge networks: A comprehensive survey [J]. Journal of Network and Computer Applications, 2021, 181: 103005.

[20] Cisco global cloud index: Forecast and methodology, 2016—2021 [EB/OL]. (2018-01-15)[2023-11-15]. https://virtualization.network/Resources/Whitepapers/0b75cf2e-0c53-4891-918e-b542a5d364c5_white-paper-c11-738085.pdf.

[21] Le Duc T, Leiva R G, Casari P, et al. Machine learning methods for reliable resource provisioning in edge-cloud computing: A survey [J]. ACM Computing Surveys, 52(5): 94.

[22] Sze V, Chen Y H, Yang T J, et al. Efficient processing of deep neural networks: A tutorial and survey [J]. Proceedings of the IEEE, 2017, 105(12): 2295‒2329.

[23] Ananthanarayanan G, Bahl P, Bodik P, et al. Real-time video analytics: The killer app for edge computing [J]. Computer, 2017, 50(10): 58‒67.

[24] Ha K Y, Chen Z, Hu W L, et al. Towards wearable cognitive assistance [C]. Bretton Woods: Proceedings of the 12th Annual International Conference on Mobile Systems, Applications, and Services, 2014.

[25] Cao J, Xu L Y, Abdallah R, et al. EdgeOS_H: A home operating system for Internet of everything [C]. Atlanta: 2017 IEEE 37th International Conference on Distributed Computing Systems, 2017.

[26] Dey S, Mondal J, Mukherjee A. Offloaded execution of deep learning inference at edge: Challenges and insights [C]. Kyoto: 2019 IEEE International Conference on Pervasive Computing and Communications Workshops, 2019.

[27] Xu Z C, Zhao L Q, Liang W F, et al. Energy-aware inference offloading for DNN-driven applications in mobile edge clouds [J]. IEEE Transactions on Parallel and Distributed Systems, 2021, 32(4): 799‒814.

[28] Pacheco R G, Couto R S, Simeone O. Calibration-aided edge inference offloading via adaptive model partitioning of deep neural networks [C]. Montreal: ICC 2021-IEEE International Conference on Communications, 2021.

[29] Pacheco R G, Oliveira F D V R, Couto R S. Early-exit deep neural networks for distorted images: Providing an efficient edge offloading [C]. Madrid: 2021 IEEE Global Communications Conference, 2021.

[30] Banitalebi-Dehkordi A, Vedula N, Pei J, et al. Auto-split: A general framework of collaborative edge-cloud AI [C]. Singapore: Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, 2021.

[31] Asheralieva A, Niyato D, Xiong Z H. Auction-and-learning based Lagrange coded computing model for privacy-preserving, secure, and resilient mobile edge computing [J]. IEEE Transactions on Mobile Computing, 2023, 22(2): 744‒764.

[32] Shyuan J, Lim W, Xiong Z H, et al. A double auction mechanism for resource allocation in coded vehicular edge computing [J]. IEEE Transactions on Vehicular Technology, 2022, 71(2): 1832‒1845.

[33] Lu Y, Shu Y C, Tan X, et al. Collaborative learning between cloud and end devices: An empirical study on location prediction [C]. Washington DC: Proceedings of the 4th ACM/IEEE Symposium on Edge Computing, 2019.

[34] Ding C T, Zhou A, Liu Y X, et al. A cloud-edge collaboration framework for cognitive service [J]. IEEE Transactions on Cloud Computing, 2022, 10(3): 1489‒1499.

[35] Lyu Z Q, Zhang W Q, Zhang S Y, et al. DUET: A tuning-free device-cloud collaborative parameters generation framework for efficient device model generalization [C]. New York: Proceedings of the ACM Web Conference, 2023.

[36] Yan Y K, Niu C Y, Gu R J, et al. On-device learning for model personalization with large-scale cloud-coordinated domain adaption [C]. Washington DC: Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2022.

[37] Luy Z Q, Chen Z Y, Zhang S Y, et al. IDEAL: Toward high-efficiency device-cloud collaborative and dynamic recommendation system [EB/OL]. (2023-02-14)‍[2023-11-15]. https://arxiv.org/abs/2302.07335.

[38] Li Y N, Yuan H T, Fu Z, et al. ELASTIC: Edge Workload Forecasting based on Collaborative Cloud-Edge Deep Learning [C]. New York: Proceedings of the ACM Web Conference 2023, 2023.

[39] Zhou Z, Chen X, Li E, et al. Edge intelligence: Paving the last Mile of artificial intelligence with edge computing [J]. Proceedings of the IEEE, 2019, 107(8): 1738‒1762.

[40] Nguyen D V, Tran H T T, Thang T C. A delay-aware adaptation framework for cloud gaming under the computation constraint of user devices [C]. Thessaloniki: International Conference on Multimedia Modeling, 2019.

[41] Basiri M, Rasoolzadegan A. Delay-aware resource provisioning for cost-efficient cloud gaming [J]. IEEE Transactions on Circuits and Systems for Video Technology, 2018, 28(4): 972‒983.

[42] Kassir S, de Veciana G, Wang N N, et al. Joint update rate adaptation in multiplayer cloud-edge gaming services: Spatial geometry and performance tradeoffs [C]. New York: Proceedings of the Twenty-second International Symposium on Theory, Algorithmic Foundations, and Protocol Design for Mobile Networks and Mobile Computing, 2021.

[43] Dionisio J D N, Gilbert R. 3D Virtual worlds and the metaverse: Current status and future possibilities [J]. ACM Computing Surveys, 45(3): 34.

[44] Lee L K, Braud T, Zhou P Y, et al. All one needs to know about Metaverse: A complete survey on technological singularity, virtual ecosystem, and research agenda [EB/OL]. (2021-10-06)[2023-11-15]. https://arxiv.org/abs/2110.05352.

[45] Jakob N. Usability Engineering [M]. San Francisco: Morgan Kaufmann Publishers Inc., 1994.

[46] Bao G M, Guo P. Federated learning in cloud-edge collaborative architecture: Key technologies, applications and challenges [J]. Journal of Cloud Computing, 2022, 11(1): 94.

[47] Zhou X, Lei X Y, Yang C, et al. Handling data heterogeneity in federated learning via knowledge fusion [EB/OL]. (2022-07-23)[2023-11-15]. https://arxiv.org/abs/2207.11447.

[48] Li T, Sahu AK, Zaheer M, et al. Federated optimization in heterogeneous networks [C]. Austin: Conference on Machine Learning and Systems, 2020.

[49] Ye M, Fang X W, Du B, et al. Heterogeneous federated learning: State-of-the-art and research challenges [J]. ACM Computing Surveys, 56(3): 79.

[50] Sun Z T, Kairous P, Suresh AT, et al. Can you really backdoor federated learning? [EB/OL] (2019-11-18)[2023-11-15]. https://arxiv.org/abs/1911.07963.

[51] Lyu L J, Yu H, Ma X J, et al. Privacy and robustness in federated learning: Attacks and defenses [J]. IEEE Transactions on Neural Networks and Learning Systems, 2022: 1‒21.

[52] Chen Y, Gui Y J, Lin H, et al. Federated learning attacks and defenses: A survey [C]. Osaka: 2022 IEEE International Conference on Big Data, 2022.

[53] Kulkarni V, Kulkarni M, Pant A. Survey of personalization techniques for federated learning [C]. London: 2020 Fourth World Conference on Smart Trends in Systems, Security and Sustainability (WorldS4), 2020.

[54] Marfoq O, Neglia G, Vidal R, et al. Personalized federated learning through local memorization [C]. London: Proceedings of the 39th International Conference on Machine Learning, 2022.

[55] Pillutla K, Malik K, Mohamed A, et al. Federated learning with partial model personalization [EB/OL]. (2022-04-18)[2023-11-15]. https://arxiv.org/abs/2204.03809.

[56] Chen Z Y, Yao J C, Wang F, et al. Mc2-sf: Slow-fast learning for mobile-cloud collaborative recommendation [EB/OL]. (2021-09-25)[2023-11-15]. https://arxiv.org/abs/2109.12314.

[57] Kahneman D. Thinking, fast and slow [M]. London: Penguin Books Ltd., 2011.

[58] Madan K, Ke R N, Goyal A, et al. Fast and slow learning of recurrent independent mechanisms [EB/OL]. (2021-05-18)[2023-11-15]. https://arxiv.org/abs/2105.08710.

[59] Zhang S Y, Feng F L, Kuang K, et al. Personalized latent structure learning for recommendation [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023, 45(8): 10285‒10299.

[60] Gan Y L, Pan M J, Zhang R Y, et al. Cloud-device collaborative adaptation to continual changing environments in the real-world [C]. Vancouver: 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.

[61] Qian X F, Xu Y, Lv F Y, et al. Intelligent request strategy design in recommender system [C]. Washington DC: Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2022.

[62] Ding Y C, Niu C Y, Wu F, et al. DC-CCL: Device-cloud collaborative controlled learning for large vision models [EB/OL]. (2023-03-18)[2023-11-15]. https://arxiv.org/abs/2303.10361.

[63] Lyu C F, Niu C Y, Gu R J, et al. Walle: An end-to-end, general-purpose, and large-scale production system for device-cloud collaborative machine learning [EB/OL]. (2022-05-30)[2023-11-15]. https://arxiv.org/abs/2205.14833.

[64] Zhang S Y, Jiang T, Wang T, et al. DeVLBert: Learning deconfounded visio-linguistic representations [C]. Seattle: Proceedings of the 28th ACM International Conference on Multimedia, 2020.

[65] Zhang S Y, Feng X S, Fan W J, et al. Video audio domain generalization via confounder disentanglement [C]. Washington DC: Proceedings of the AAAI Conference on Artificial Intelligence, 2023.

[66] Zhang S Y, Jiang Z Q, Yao J C, et al. Causal distillation for alleviating performance heterogeneity in recommender systems [J]. IEEE Transactions on Knowledge and Data Engineering, 2023: 1‒16.

[67] Naghiaei M, Rahmani H A, Deldjoo Y. CPFair: Personalized consumer and producer fairness re-ranking for recommender systems [C]. Madrid: Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2022.

[68] Huang H L, Wei S, Feng Y H, et al. Active client selection for clustered federated learning [J/OL]. IEEE Transactions on Neural Networks and Learning Systems, [2023-07-28]. https://doi.org/10. 1109/TNNLS.2023.3294295.

[69] Yang Q. Federated recommendation systems [C]. Los Angeles: 2019 IEEE International Conference on Big Data, 2019.

[70] Liu S S, Liu L K, Tang J, et al. Edge computing for autonomous driving: Opportunities and challenges [J]. Proceedings of the IEEE, 2019, 107(8): 1697‒1716.

[71] Deng Y, Zhang T H, Lou G N, et al. Deep learning-based autonomous driving systems: A survey of attacks and defenses [J]. IEEE Transactions on Industrial Informatics, 2021, 17(12): 7897‒7912.

[72] Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition [EB/OL]. (2014-09-04)[2023-11-15]. https://arxiv.org/abs/1409.1556.

[73] He K M, Zhang X Y, Ren S Q, et al. Deep residual learning for image recognition [C]. Las Vegas: 2016 IEEE Conference on Computer Vision and Pattern Recognition, 2016.

[74] Chen Q Y, Wang Z, Su Y, et al. Educational 5G edge computing: Framework and experimental study [J]. Electronics, 2022, 11(17): 2727.

[75] Wang C, Wang D. Managing the integration of teaching resources for college physical education using intelligent edge-cloud computing [J]. Journal of Cloud Computing, 2023, 12(1): 82.

[76] Caines A, Benedetto L, Taslimipoor S, et al. On the application of large language models for language teaching and assessment technology [EB/OL]. (2023-07-17)[2023-11-15]. https://arxiv.org/abs/2307.08393.

[77] Qian C, Yu Y, Zhou Z H. Subset selection by Pareto optimization [C]. Montréal: Advances in neural information processing systems, 2015.

[78] Gronauer S, Diepold K. Multi-agent deep reinforcement learning: A survey [J]. Artificial Intelligence Review, 2022, 55(2): 895‒943.

[79] Wu F, Souza A, Zhang T Y, et al. Simplifying graph convolutional networks [C]. Long Beach: Proceedings of the 36th International Conference on Machine Learning, 2019.

Related Research