期刊首页 优先出版 当期阅读 过刊浏览 作者中心 关于期刊 English

《工程(英文)》 >> 2023年 第25卷 第6期 doi: 10.1016/j.eng.2022.01.012

基于普通器件实现快1000倍的相机与机器视觉

Department of Computer Science and Technology, National Engineering Laboratory for Video Technology, Peking University, Beijing 100091, China

收稿日期: 2021-02-19 修回日期: 2022-01-03 录用日期: 2022-01-05 发布日期: 2022-04-12

下一篇 上一篇

摘要

在数码相机中,我们发现了一个重大缺陷,即从胶片相机继承的图像和视频模型阻碍了相机捕捉快速变化的光子世界。我们提出了一种新的视觉形式,称为视象(vform),这是一个比特序列阵列,其中每个比特表示光子的累积是否达到了一个阈值,从而可以记录和重建任何时刻场景的光强。仅使用消费级CMOS(互补金属氧化物半导体器件)传感器和集成电路,开发了一种比传统相机快1000 倍的脉冲相机。将视象看作生物视觉中的脉冲序列,进一步开发了基于脉冲神经网络的机器视觉系统,它可以将机器的速度和生物视觉的机理结合起来,从而实现了比人类视觉快1000 倍的高速目标检测和跟踪,并通过辅助裁判和目标瞄准系统证明了脉冲相机和超级视觉系统的效用。视象模型和芯片有望从根本上改变图像和视频的概念以及摄影、电影和视觉媒体等相关行业,并开启一个全新的基于脉冲神经网络的速度自由的机器视觉时代。

图片

图1

图2

图3

图4

图5

图6

参考文献

[ 1 ] Haykin S, Van B. Signals and systems. New Jersey: John Wiley & Sons; 2007.

[ 2 ] Chakravorty P. What is a signal? IEEE Signal Process Mag 2018;35(5):175‒7. 链接1

[ 3 ] Stump D. Digital cinematography: fundamentals, tools, techniques, and workflows. Boca Raton: CRC Press; 2014. 链接1

[ 4 ] Itatani J, Quéré F, Yudin GL, Ivanov MY, Krausz F, Corkum PB. Attosecond streak camera. Phys Rev Lett 2002;88(17):173903. 链接1

[ 5 ] Bradley DK, Bell PM, Landen OL, Kilkenny JD, Oertel J. Development and characterization of a pair of 30‒40 ps x-ray framing cameras. Rev Sci Instrum 1995;66(1):716‒8. 链接1

[ 6 ] Wässle H. Parallel processing in the mammalian retina. Nat Rev Neurosci 2004;5(10):747‒57. 链接1

[ 7 ] Masland RH. The neuronal organization of the retina. Neuron 2012;76(2):266‒80. 链接1

[ 8 ] Litwiller D. CCD vs CMOS. Photon Spectra 2001;35:154‒8.

[ 9 ] Lamb TD, Pugh EN. Phototransduction, dark adaptation, and rhodopsin regeneration theproctor lecture. InvestOphthalmol Vis Sci 2006;47(12):5137‒52. 链接1

[10] LeCun Y, Bengio Y, Hinton G. Deep learning. Nature 2015;521(7553):436‒44. 链接1

[11] Maass W. Networks of spiking neurons: the third generation of neural network models. Neural Netw 1997;10(9):1659‒71. 链接1

[12] Roy K, Jaiswal A, Panda P. Towards spike-based machine intelligence with neuromorphic computing. Nature 2019;575(7784):607‒17. 链接1

[13] Marr D, Poggio T, Ullman S. Vision: a computational investigation into the human representation and processing of visual information. Cambridge: MIT Press; 2010. 链接1

[14] Palmer SE. Vision science: photons to phenomenology. Cambridge: MIT Press; 1999. 链接1

[15] Li Z. Understanding vision: theory, model, and data. New York City: Oxford University Press; 2014.

[16] Davies ER. Computer and machine vision: theory, algorithm, practicalities. 4th ed. London: Academic Press; 2012. 链接1

[17] Sonka M, Hlavac V, Boyle R. Image processing, analysis, and machine vision. 4th ed. Stamford: Cengage Learning; 2015.

[18] Medathati NVK, Neumann H, Masson GS, Kornprobst P. Bio-inspired computer vision: towards a synergistic approach of artificial and biological vision. Comput Vis Image Underst 2016;150:1‒30. 链接1

[19] Tsodyks MV, Markram H. The neural code between neocortical pyramidal neurons depends on neurotransmitter release probability. Proc Natl Acad Sci USA 1997;94(2):719‒23. 链接1

[20] Tsodyks M, Pawelzik K, Markram H. Neural networks with dynamic synapses. Neural Comput 1998;10(4):821‒35. 链接1

[21] Costa RP, Sjöström PJ, van Rossum MCW. Probabilistic inference of short-term synaptic plasticity in neocortical microcircuits. Front Comput Neurosci 2013;7:75. 链接1

[22] Mi Y, Fung CA, Wong KM, Wu S. Spike frequency adaptation implements anticipative tracking in continuous attractor neural networks. Adv Neural Inf Process Syst 2014;27:505‒13.

[23] Tavanaei A, Maida A. BP-STDP: approximating backpropagation using spike timing dependent plasticity. Neurocomputing 2019;330:39‒47. 链接1

[24] Song S, Miller KD, Abbott LF. Competitive Hebbian learning through spiketiming- dependent synaptic plasticity. Nat Neurosci 2000;3(9):919‒26. 链接1

[25] Rumsey CC, Abbott LF. Synaptic equalization by anti-STDP. Neurocomputing 2004;58:359‒61. 链接1

[26] Adelson EH, Bergen JR. The plenoptic function and the elements of early vision. In: Landy MS, Movshon JA, editors. Computational models of visual processing. Cambridge: MIT Press; 1991. 链接1

相关研究