Robot Cognitive Learning by Considering Physical Properties

Fuchun Sun: Writing-review; editing; Supervision; Project administration; Conceptualization. Wenbing Huang: Writing-original draft; Methodology; Formal analysis. Yu Luo: Writing-review

doi:10.1016/j.eng.2024.10.013

Engineering ›› 2025, Vol. 47 ›› Issue (4) :179 -191. DOI: 10.1016/j.eng.2024.10.013

Research Artificial Intelligence—Article

research-article

Robot Cognitive Learning by Considering Physical Properties

Fuchun Sun: Writing-review ^a
, editing ^b
, Supervision ^a
, Project administration ^a
, Conceptualization. Wenbing Huang: Writing-original draft ^a
, Methodology ^a
, Formal analysis. Yu Luo: Writing-review ^c

Author information +

History +

PDF (1947KB)

Abstract

Humans achieve cognitive development through continuous interaction with their environment, enhancing both perception and behavior. However, current robots lack the capacity for human-like action and evolution, posing a bottleneck to improving robotic intelligence. Existing research predominantly models robots as one-way, static mappings from observations to actions, neglecting the dynamic processes of perception and behavior. This paper introduces a novel approach to robot cognitive learning by considering physical properties. We propose a theoretical framework wherein a robot is conceptualized as a three-body physical system comprising a perception-body (P-body), a cognition-body (C-body), and a behavior-body (B-body). Each body engages in physical dynamics and operates within a closed-loop interaction. Significantly, three crucial interactions connect these bodies. The C-body relies on the P-body’s extracted states and reciprocally offers long-term rewards, optimizing the P-body’s perception policy. In addition, the C-body directs the B-body’s actions through sub-goals, and subsequent P-body-derived states facilitate the C-body’s cognition dynamics learning. At last, the B-body would follow the sub-goal generated by the C-body and perform actions conditioned on the perceptive state from the P-body, which leads to the next interactive step. These interactions foster the joint evolution of each body, culminating in optimal design. To validate our approach, we employ a navigation task using a four-legged robot, D’Kitty, equipped with a movable global camera. Navigational prowess demands intricate coordination of sensing, planning, and D’Kitty’s motion. Leveraging our framework yields superior task performance compared with conventional methodologies. In conclusion, this paper establishes a paradigm shift in robot cognitive learning by integrating physical interactions across the P-body, C-body, and B-body, while considering physical properties. Our framework’s successful application to a navigation task underscores its efficacy in enhancing robotic intelligence.

Graphical abstract

Keywords

Robot learning / Physical basis / Cognitive learning

Cite this article

Download citation ▾

Fuchun Sun: Writing-review, editing, Supervision, Project administration, Conceptualization. Wenbing Huang: Writing-original draft, Methodology, Formal analysis. Yu Luo: Writing-review. Robot Cognitive Learning by Considering Physical Properties. Engineering, 2025, 47(4): 179-191 DOI:10.1016/j.eng.2024.10.013

登录浏览全文

4963

注册一个新账户忘记密码

1. Introduction

Humans interact with the world in a complex and highly intelligent manner [1]; they employ their eyes for sight and their hands/legs for action, with both working collaboratively under brain control. Navigating a cluttered room, for example, necessitates specific abilities: perception, through which the individual detects doors, walls, and obstacles; behavior, which determines the person’s navigation while avoiding collisions; and cognition, in which the person’s body issues commands for perception and behavior via brain functions. Acting as feedback systems [2], humans refine their cognitive abilities through the reciprocal processes of perception and behavior. Trial and error enhance planning when the observed outcomes deviate from the expected results.

Mimicking such closed-loop multifunctional systems in robots is challenging. Artificial intelligence and robotics researchers often model robots as agents [3], [4]. Observational input (e.g., camera images and force sensors’ tactile feedback) is mapped to actions through policies optimized using imitation learning (IL) [5] or reinforcement learning (RL) in order to maximize feedback rewards. While these methods enable robots to perform real-world tasks, such as navigation [6], locomotion [7], and Rubik’s cube manipulation [8], they have limitations. First, current approaches yield static, one-way mappings from observation to action, ignoring the intricate interaction between robot-embodied perception, behavior, and the environment. Human systems, in contrast, exhibit complex physical integration in which perception, behavior, and cognition interact and evolve holistically. Second, existing methods lack the capability to represent human-like cognition dynamics.

In this paper, we introduce the concept of physics-informed robot cognitive learning to address these limitations. Our approach boasts several advantages over previous methods. We conceptualize the robot as a three-body physical system comprising a perception-body (P-body), a cognition-body (C-body), and a behavior-body (B-body). Each body engages in physical dynamics, with the P-body receiving environmental information and altering its configuration for observation, the C-body handling cognition and generating sub-goals based on the P-body’s states, and the B-body executing behavior directed by the C-body’s sub-goals.

Significantly, our approach introduces three critical interactions among these bodies:

•C-body to P-body: The C-body relies on the P-body’s states, reciprocating with future sub-goals to enhance the P-body’s perception policy.

•C-body to B-body: The C-body guides the B-body’s actions through sub-goals, and P-body-derived states following the B-body’s actions facilitate cognition dynamics learning in the C-body.

•B-body to P-body: The B-body interacts with the environment and changes the state, causing the P-body to acquire new information that updates its perception.

Fig. 1 illustrates the overarching framework, encompassing the P-body, C-body, and B-body interactions.

This unique interaction framework fosters joint evolution, leading to optimal design. Although it shares similarities with multi-agent RL (MARL) [9], our integrated system presents distinct features. The three bodies—perception, cognition, and behavior—possess diverse dynamics and pursue different sub-goals, whereas regular MARL agents are homogeneous, with common decision processes and rewards.

To validate our approach, we apply it to a navigation task employing a four-legged robot, D’Kitty [10], with a movable global camera. The task demands intricate coordination between the camera and D’Kitty, showcasing the effectiveness of our formulation. The experimental results, while indicative, underscore the potential applicability of our method to diverse practical tasks, including air–ground robot systems for emergency rescue.

2. Related works

2.1. Vision perception

In recent years, encouraging advances have been demonstrated in various vision applications, such as image classification [11], [12], [13], object detection [14], [15], [16], [17], [18], and instance segmentation [19], largely thanks to the rapid development of deep learning. Here, we review recent developments in object-detection tasks, in which algorithms require an image as input and then deliver the category and location information of a certain object of interest. Typically, traditional object-detection algorithms fall into two categories: two-stage object detectors and single-stage object detectors. Both types involve the steps of extracting features, generating proposals, regressing position coordinates, and classifying classes. One representative two-stage object-detection method is the faster region-based convolutional neural network (RCNN) [20], which proposes an anchor mechanism for predicting object boundaries and objectness scores per location instead of selective search methods. Another example worth mentioning is the feature pyramid network (FPN) [21], which exploits the fusion of features between different depth feature layers to give better object-detection results at various scales. As a single-stage object-detection approach, the “you only look once” (YOLO) object-detection system [22] drastically reduces detection time compared with two-stage detection methods, albeit at the cost of lower accuracy.

To name a few more one-stage detection methods, the single-shot detector (SSD) [23] extracts feature maps and utilizes a prior box of different scales and aspect ratios for detection, while RetinaNet [24] introduces focal loss to resolve the extremely uneven sample distribution issue and thereby improves the accuracy of the single-stage detector. Anchor-free object-detector methods [25], [26], [27] also exist, which employ features of the network structure to replace the anchor. The recent development of transformers [28], [29] has also promoted research on object detection [30], [31].

The aforementioned methods come under the passive visual perception category, into which most existing algorithms fall. Unfortunately, passive visual perception methods depend on a well-captured still image as an input, which is not easy to obtain in practice. To be specific, passive visual perception methods struggle to locate and classify objects accurately in complex and dynamic circumstances, hindered by noise interference such as occlusion, overexposure, or moving objects. In contrast, humans are able to adjust their position and viewpoints to obtain images of interest—an observation that has inspired the study of active perception trends. In our previous works, we explored a robot with a fixed vision sensor [32], [33], [34], promoting the development of active object detection. However, these methods depend on a handcrafted reward design, such that the agent trained by RL needs to maximize the return to yield better perception performance [35], [36]. In our architecture, the P-body is able to move and seek a desirable configuration for better detection, which falls into the category of active perception methods. Moreover, our P-body is both data-driven and knowledge-driven. In particular, the perception policy of the P-body is not only learned from data by RL under the guidance of the perception reward but also influenced by the sub-goal guidance generated from the C-body. Conditional on the generated sub-goals, the active perception process of the P-body will leverage the knowledge from the C-body, and such knowledge will be continuously updated and optimized through interactions between the B-body and the environment, until the final task goal is attained.

2.2. Reinforcement learning

RL has made impressive advances in artificial intelligence, surpassing human performance in various domains (Atari, Go, etc.). It formalizes the problem of goal-seeking intelligence, which corresponds to maximizing accumulated rewards in the task and environment respectively [37]. RL algorithms are generally divided into the two categories of model-free RL and model-based RL. In model-free RL, agent-training methods can be further divided into two predominant approaches:

(1)Value-based methods. In these methods, the agent learns a state (s)–action (a) value function approximator

Q_{θ} (s, a)

with

θ

as the neural network parameters and selects an action accordingly. The deep-Q network (DQN) [38] was the first deep learning model to successfully learn the value function directly from high-dimensional sensory input.

(2)Policy-based methods. These methods focus on modeling and optimizing the policy

π_{θ} (a | s)

directly. For example, REINFORCE [39] updates the policy parameter via gradient ascent on an estimated return (using episode samples collected by Monte–Carlo methods).

More specifically, actor-critic methods interpolate between policy evaluation and policy optimization. For example, deep deterministic policy gradient (DDPG) agents [40] concurrently learn a deterministic policy and a Q-function and use each to improve the other, while soft actor critic (SAC) agents [41] learn a stochastic policy by incorporating entropy regularization and some other tricks. Unlike previous model-free RL methods, our B-body in the Bcent framework can be guided by the planning prompts from the C-body and thus achieve better exploration performance when interacting with the environment.

Model-based RL methods usually achieve better data efficiency but have a weaker asymptotic performance than their model-free counterparts. Model-based algorithms are typically grouped into three categories according to the various usages of the learned dynamical models [42]:

(1)Dyna-style algorithms. In Dyna algorithms, the learned dynamics are used to generate imagined data, and the training iterates between policy optimization with imagined data and model correction with real samples. Model-based policy optimization (MBPO) [43] utilizes an ensemble of neural networks to model the dynamics and adopts SAC [41] as the policy optimization algorithm.

(2)Policy search with back-propagation through time. These methods exploit the model derivatives and improve the policy based on its analytic gradient. The iterated linear quadratic Gaussian (iLQG) method [44] assumes the reward function to be quadratic and the dynamics to be linear and then derives a controller from these simple parametrizations using dynamic programming.

(3)Shooting algorithms. These algorithms sample candidate actions from a designed distribution, evaluate the sampled action under a model, and choose the most promising action in order to deal with nonlinear dynamics and non-convex reward functions. In a recommender system (RS) [45], the agent generates candidate action sequences from a uniform distribution, while the cross entropy method (CEM) [46] and probabilistic ensembles with trajectory sampling (PETS) [47] iteratively adjust the sampling distribution. The knowledge base in the C-body of Bcent can be regarded as an extension of these learning dynamics models. Moreover, our knowledge base can provide sub-goals to guide the P-body and B-body to improve the training efficiency.

3. Methodology

As depicted in Fig. 1, our architecture comprises three essential physical components—P-body, C-body, and B-body—that intricately collaborate within a closed loop. In essence, the P-body acquires comprehensive information from the environment and distils it into low-dimensional states for the C-body. Subsequently, the C-body dispatches specific sub-goals to the B-body based on the received states. Eventually, the B-body enacts precise actions to follow the sub-goals designated by the C-body. This cyclical process persists as the environment evolves, iteratively advancing until the robot attains its ultimate objective.

The forthcoming sections delve into the comprehensive particulars of each body’s role and functionality.

3.1. The perception-body

The primary objective of the P-body is to provide the C-body with a low-dimensional representation after processing high-dimensional observations obtained from the environment. The physical implementation of the P-body may involve various types of sensors, such as cameras for visual input, force arrays for tactile feedback, and voice recorders for audio data collection. A fundamental characteristic of the P-body is its dynamic perception process. The term “dynamic” signifies that the P-body possesses the capability to adjust its configuration to enhance its observation quality if necessary. For example, if the P-body is realized as a movable camera, its observations are closely tied to its configuration, including the camera location and orientation. We refer to the policy-governing configuration changes as the “perception policy.” Furthermore, as previously mentioned, the perception policy is influenced by cues (i.e., sub-goals) received from the C-body.

The computational flow within the P-body is mathematically defined as follows:

(1)

\begin{matrix} s_{p} (t) = f (o (t) | c_{p} (t)) (state extraction) \end{matrix}

(2)

\begin{matrix} a_{p} (t) = π_{p} (s_{p} (t) | s_{c} (t + Δ t)) (p e r c e p t i ° n p ° l i c y) \end{matrix}

(3)

\begin{matrix} c_{p} (t + 1) = g_{p} (c_{p} (t), a_{p} (t)) (p e r c e p t i ° n d y n a m i c s) \end{matrix}

where a_p is the action of the P-body’s policy. In the above equations,

t

represents the time step (

t = 0, . . ., T

); T is the task horizon;

o (t)

represents the high-dimensional and multi-modal observation from the environment, such as images, depths, and audio;

s_{p} (t)

is a low-dimensional abstract state vector that characterizes the perception state;

c_{p} (t)

is the perception state defined as a combination of the sensors’ parameters and configurations, such as the position and the inner and outer parameters of the sensors, which are determined based on the policy

π_{p} (s_{p} (t) | s_{c} (t + Δ t))

, where

s_{c} (t + Δ t)

denotes the future sub-goal from the C-body after a delay time

Δ t

; and

f

is an embedding function that maps the high-dimensional observation vector

o (t)

to a low-dimensional and abstract state vector

s_{p} (t)

, conditioned on the configuration

c_{p} (t)

. Subsequently, the P-body transitions to a new configuration,

c_{p} (t + 1)

, based on the action taken from the current configuration, as governed by the transition probability function

g_{p} (c_{p} (t), a_{p} (t))

. The underlying principle of this formulation is rooted in the concept of a Markov decision process (MDP).

3.1.1. Model-free perception policy learning via P–C interaction

In the context of an MDP, training the policy

π_{p}

typically involves two primary approaches: model-based methods [48] and model-free methods (e.g., proximal policy optimization (PPO) [49] and trust region policy optimization (TRPO)) [50]. In the case of the P-body, we opt for the model-free strategy because of the potential complexity and cost associated with learning the transition probability of the observations

o (t)

. An important question arises regarding the provision of the reward signal,

r_{p}

. While one option is to set

r_{p}

as a measure of how accurately the perception state

s_{p} (t)

estimates the ground-truth state

s^{*} (t)

, this criterion may not suffice in many scenarios.

3.1.2. P–C interaction

Given that robots typically operate with long-term goals, such as reaching a target position, it becomes essential to ensure minimal perception error throughout the entire movement trajectory, especially at the target point. Moreover, in the Bcent framework, the C-body will transmit sub-goals to the P-body in order to provide guidance and thus improve the perception performance. Compared with conventional active perception methods [50], [51], our P–C interaction offers distinct advantages:

•Efficiency: The P-body’s explicit modeling of the perception dynamics allows the C-body’s planning information to aid the P-body’s movement through the dynamic model, enhancing the interpretability and effectiveness of the perception. In contrast, active perception methods using RL often guide sensor movement solely based on maximizing the perceptive reward, resulting in high exploration costs.

•Flexibility: The generality of our framework (Eqs. (1), (2), (3)) accommodates various policy implementations, including predefined rules, control methods, and RL.

3.2. The cognition-body

The C-body plays a pivotal role in our model, akin to the human brain’s responsibility for coordinating and directing various bodily functions. More specifically, the C-body’s key tasks involve extracting and updating knowledge based on accumulated experiences gained through interactions with the environment by the other two bodies. In return, the C-body provides guidance and instruction to the other two bodies.

The computational flow within the C-body is defined as follows:

(4)

\begin{matrix} s_{c} (0) = f (o (0) | c_{p} (0)) (i n i t i a l s t a t e e x t r a c t i ° n) \end{matrix}

(5)

\begin{matrix} a_{c} (t) = π_{c} (s_{c} (t) | k_{c}) (c ° g n i t i ° n p ° l i c y) \end{matrix}

(6)

\begin{matrix} s_{c} (t + Δ t) = g_{c} (s_{c} (t), a_{c} (t) | k_{c}) (c ° g n i t i ° n d y n a m i c s) \end{matrix}

In the above equations,

s_{c} (t)

is the planning instruction from the C-body, aimed at guiding the exploration of the P-body and B-body; and

a_{c} (t)

is the action of the C-body at each planning instant, which then leads to the generation of the next planning instruction. Furthermore,

Δ t \in N^{+}

(is a positive integer) denotes the update interval in the C-body, with the temporal values of

t = 0, Δ t, . . ., K Δ t

(

K

steps) as possible time points;

π_{c}

and

g_{c}

refer tothe cognition transition probability function in the C-body, respectively; and

k_{c}

represents the knowledge base in C-body.

It is important to note that

s_{c} (t)

is of multiple forms to guide the exploration and execution of the P-body and B-body, representing planned sub-goals or instructions generated by the cognition dynamics. Unlike the P-body (and the upcoming B-body), which employs a unit update interval (

Δ t = 1

), we deviate from this default setting by allowing for

Δ t > 1

. This modification is motivated by the idea that the C-body focuses more on long-term, coarse-level goals and evolves over a larger temporal scale.

It is worth highlighting that both the cognition policy and the dynamics are contingent on the knowledge of the C-body, denoted as

k_{c}

. While this paper currently defines

k_{c}

as the parameters of the cognition transition probability function

g_{c}

, it can also take the form of memory buffers, knowledge databases, knowledge graphs, or other informative constructs. The incorporation of knowledge

k_{c}

enables the C-body to mimic the functioning of a human brain, encompassing knowledge utilization and knowledge updates.

We further specify the planning instructions in Fig. 2, Fig. 3. In particular, on the right side of Fig. 3, a D’Kitty robot would start from the start point and focus on reaching the target point, passing by an obstacle represented by an orange area. Then, in the knowledge base, we can divide this task into a triple skill sequence, “Move to obstacle,” “Move around obstacle,” and “Move to target”, each equipped with specified action primitives and parameter spaces.

More specifically, in the first skill, “Move to obstacle” the action primitives include detecting the distance and shape of the obstacle, approaching the turning point 1, and then arriving in front of the obstacle. It should be noted that the obstacle is a short wall or a pit that the robot cannot pass through. Thus, the robot must move around the obstacle to reach the target. When reaching the first turning point, the D’Kitty robot initiates the second skill, “Move around obstacle,” which includes newly detecting the obstacle for its coordinates, planning to pass by the obstacle, and tracking the planning trajectories in this process. Finally, the D’Kitty robot performs the third skill, “Move to target,” by detecting the target point and arriving at the target. By using the three skills in sequence, the D’Kitty robot can autonomously detect both the obstacle and the target point, plan a reasonable route based on the detection results, and finally perform actions to achieve the sub-goals one by one. In this case, the C-body acts as the skill policy in order to generate the planning instructions, and the B-body acts as the parameter sub-policy in order to eventually reach the target point.

3.2.1. Knowledge utilization

The objective of knowledge utilization within the C-body is to generate sub-goals

{\{s_{c} (k Δ t)\}}_{k = 1}^{⌊ \frac{T}{Δ t} ⌋}

conditioned on the task goal

g

, where

⌊ x ⌋

computes the greatest integer less than or equal to

x

, and

k

is a positive integer. This is achieved through model-based RL in this paper. Given our focus on goal-based tasks, the reward

r_{c}

is computed by measuring how closely the perception state

s_{p} (T)

at time

T (T = K Δ t)

approaches the target state

s^{*}

. The reward is defined as

r_{c} (s^{*}, s_{p} (T)) = - d (s^{*}, s_{p} (T))

, where

d (\cdot, \cdot)

represents the distance measure function in the perception state space, such as the Euclidean norm used in our implementation. We opt for model-based policy learning in C-body for two main reasons:

•The cognition dynamics operate within a concise and low-dimensional state space, rendering it more tractable to learn a model

g_{c} (s_{c} (t), a_{c} (t) | k_{c})

in this abstract domain.

•Learning the cognition dynamics is imperative, as we will soon outline that the C-body is responsible for supplying both the P-body and B-body with sub-goals, which depend on the outputs of the cognition model.

Various model-based RL techniques have been proposed; we select the iLQR method [52], which has been widely applied in robot control. The generation of sub-goals involves formulating an optimal control problem:

(7)

\begin{matrix} {\{s_{c}\}}_{k = 1}^{⌊ \frac{T}{Δ t} ⌋} = \arg \max_{s_{c}} \sum_{k} r_{c} (s^{*}, s_{p} (T)) \end{matrix}

(8)

\begin{matrix} s . t . s_{c} ((k + 1) Δ t) = g_{c} (s_{c} (k Δ t), s_{p} (k Δ t) | k_{c}) \end{matrix}

(9)

\begin{matrix} s_{c} (0) = s_{p} (0), s_{c} (T) = s^{*} \end{matrix}

The solution results in a sub-goal sequence

\{s_{c} (0), s_{c} (Δ t), . . ., s_{c} (T)\}

, which is then conveyed as guidance to the P-body and B-body.

3.2.2. Knowledge update

In the Bcent framework, initialization of knowledge can occur in two ways: learning from prior knowledge or random initialization. When prior knowledge exists, it can be encoded into the knowledge base via function approximation or hard coding. Otherwise, for general cases without prior knowledge, initialization can be performed with random hyperparameters to the knowledge base. Then, the question arises: How do we update knowledge

k_{c}

—more specifically, the parameters of the cognition model

g_{c} (s_{c} (k Δ t), s_{p} (k Δ t) | k_{c})

? In essence,

s_{c} (t + Δ t)

signifies the expected state projected by the C-body and should closely approximate the actual feedback

s_{p} (t + Δ t)

. In other words, the learning objective of the cognition model is to minimize the residual error

{‖ s_{c} (t + Δ t) - s_{p} (t + Δ t) ‖}_{2}^{2}

via the mean squared error. Given that

s_{c} (t + Δ t) = g_{c} (s_{c} (t), a_{c} (t) | k_{c})

, we propose to instantiate

a_{c} (t)

with the actual state

s_{p} (t)

during the model learning process, as our experiments have shown that this instantiation accelerates learning significantly. Overall, the model learning problem can be formulated as follows:

(10)

\begin{matrix} k_{c} = \arg \min_{k_{c}} \sum_{k = 1}^{⌊ \frac{T}{Δ t} ⌋} {‖ s_{p} ((k + 1) Δ t) - g_{c} (s_{c} (k Δ t), s_{p} (k Δ t) | k_{c}) ‖}_{2}^{2} \end{matrix}

where

{\{s_{c} (k Δ t)\}}_{k = 1}^{⌊ \frac{T}{Δ t} ⌋}

is derived from the knowledge utilization in Eq. (7).

3.2.3. P–C interaction

Once the optimal cognition transition probability model

g_{c}

is learned, it can facilitate policy learning in both the P-body and the B-body. The sub-goals are integrated into the perception policy (Eq. (2)) and the long-term reward

r_{p}

for the P-body for active sensing.

3.2.4. C–B interaction

For the B-body, the sub-goals

{\{s_{c} (k Δ t)\}}_{k = 1}^{⌊ \frac{T}{Δ t} ⌋}

will be harnessed as the expected states to guide behavior policy optimization, which will be introduced in the subsequent subsection.

In the aforementioned P–C and B–C interaction processes, the C-body leverages historical information and prior knowledge to update the cognition body

k_{c}

. This feature underscores the advantages of our formulation over prior works:

•Imitation of human cognition: The C-body emulates the environment’s characteristics and refines the knowledge of the planning process, aligning more closely with human cognitive growth.

•Sample efficiency: The introduction of

k_{c}

taps into historical information and prior knowledge, enhancing the sample efficiency of cognition training.

•Cooperativity: The C-body imparts planning information to both the P-body and the B-body, enabling them to execute their actions with increased precision and efficacy.

3.3. The behavior-body

The B-body orchestrates its actions in accordance with the sub-goals

{\{s_{c} (k Δ t)\}}_{k = 1}^{⌊ \frac{T}{Δ t} ⌋}

obtained from the C-body. To streamline this explanation, we will outline the process of reaching the sub-goal

s_{c} (Δ t)

; the computational flows for other sub-goals follow a similar pattern. We consider the MDP below, where

t = 0, 1, . . ., Δ t

(11)

\begin{matrix} s_{p} (t) = f (o (t) | c_{p} (t)) (s t a t e e x t r a c t i ° n) \end{matrix}

(12)

\begin{matrix} a_{b} (t) = π_{b} (s_{p} (t) | s_{c} (t + Δ t)) (b e h a v i ° r p ° l i c y) \end{matrix}

(13)

\begin{matrix} o (t + 1) = g_{b} (o (t), a_{b} (t)) (b e h a v i ° r d y n a m i c s) \end{matrix}

We have duplicated the state extraction stage from the P-body for the sake of clarity. Apart from this,

a_{b} (t)

is the action of the B-body derived by behavior policy, which is applied to the environment, leading to the next observation

o (t + 1)

π_{b}

is the behavior body’s policy and

g_{b}

is the behavior dynamics model.

The B-body is motivated by two primary objectives. First, it aims to accurately track the sub-goals provided by the C-body. We refer to this reward as the goal-guided reward

r_{c 1} = - d (s_{p} (k Δ t), s_{c} (k Δ t))

for each

k

th phase to measure the distance from

s_{p} (k Δ t)

s_{c} (k Δ t)

. Second, in addition to reaching the target position, the B-body must fulfil other requisites (e.g., in navigation, the robot needs to learn how to stand up and avoid obstacles before reaching the target location). Therefore, we introduce another reward

r_{c 2}

that can be obtained from the environment. In combination, we compute the reward

r_{c} = λ_{b} r_{c 1} + (1 - λ_{b}) r_{c 2}

, where

λ_{b}

is a weight parameter that balances the significance of the two rewards. To facilitate the learning of the behavior policy

π_{b}

, we lean toward employing model-free RL techniques. This choice is influenced by the typical complexity of behavior dynamics, which are often challenging to accurately characterize. In cases where the dynamics are straightforward or known, opting for model-based policy learning can offer the advantage of enhanced exploration efficiency.

After receiving the guided sub-goals from the C-body, the B-body divides the original long-term complex task into short-term simple sequential subtasks. Within each subtask, the B-body performs actions to reach the given sub-goal. When completing the subtask, the B-body feeds the real exploration results back to the C-body in order to update the knowledge base.

We summarize the advantages of the B-body compared with conventional RL methods:

•Reducibility: By leveraging planning information, the B-body can decompose the original task into a series of simpler sub-tasks. The successful completion of each sub-task contributes to the successful achievement of the overarching goal.

•Interactivity: Following interaction with the environment, the interactive residual error computed by the B-body effectively contributes to the update of the C-body’s knowledge.

•Enhanced efficiency: The task decomposition into sequential sub-tasks enables the B-body to enhance the interaction efficiency, facilitating effective and streamlined execution.

4. Experiments

All experiments were conducted using a four-legged robot called D’Kitty. The primary objective of the robot was to navigate to a predefined target location. Below is a description of the implementation of our experimental framework:

•The overhead camera functions as the P-body and is responsible for tracking the location of D’Kitty. To enhance its tracking capabilities, the camera can freely move along the three-dimensional (3D) directions—up and down, left and right, and forward and backward.

•D’Kitty, which has 12 degrees of freedom for locomotion, serves as the B-body. Its task includes learning how to stand up and move forward, based on the angles of its legs detected by internal sensors and the two-dimensional (2D) location coordinates provided by the P-body.

•The planner, acting as the C-body, is tasked with generating a virtual trajectory of sub-goals leading to the target location. This trajectory guides the movements of both the P-body and the B-body.

The subsequent sections present a series of experiments to demonstrate the necessity of interactions between the three bodies in the context of the navigation task. More specifically, we conduct evaluations for the perception task, followed by an investigation into the navigation task.

When conducting these experiments, we instantiate the symbols in Bcent framework as follows:

c_{p} (t)

represents the internal and external parameters of the camera;

o (t)

is a top-view image received by the camera;

s_{p} (t)

denote the position and velocity of D’Kitty extracted from the image, as well as the angle and angle velocity of D’Kitty detected by the internal sensors;

a_{p} (t)

is the control strategy of the camera configuration;

k_{c}

represents the parameters of the prior knowledge structure of D’Kitty and the task;

s_{c} (t)

is the expected position and attitude of D’Kitty in the next

Δ t

step;

a_{c} (t)

is the cognition strategy of the position and attitude of D’Kitty and

a_{b} (t)

is the control input of D’Kitty, such as the joint torques.

While our scenario may seem straightforward, with only one obstacle, it presents considerable complexity. Traditional control-based methods struggle to meet our task requirements due to two primary challenges. The first challenge involves the robot’s need to perceive its environment, such as identifying the obstacle’s location and size. Without the perception module (the P-body), relying solely on low-level sensing makes obstacle avoidance difficult. The second challenge arises from the unique configuration of the D’Kitty robot, which is equipped with four legs and 12 degrees of freedom. In our task, the robot must know how to stand up and move. It is difficult to control all four legs by means of traditional learning-free methods.

In general, both the perception and the behavior capabilities mentioned above are better achieved through learning-based methods rather than traditional control-based approaches.

4.1. Evaluations of perception

In our experiment, we enable the camera to move in three dimensions, allowing us to thoroughly assess the significance of active perception in the robot tasks and fully explore the potential of active perception in a perfect setting. At time

t

, the P-body (represented by the camera) receives a top-view image of the environment denoted as

I (t) = (o (t) | c_{p} (t))

, based on the internal and external parameters of the camera

c_{p} (t)

. Using this input, the P-body employs an object-detection neural network to detect the 2D state

s_{p} (t)

of the moving D’Kitty, as described by

s_{p} (t) = f (I (t))

in Eq. (1). Our P-body exhibits two distinct advantages compared with previous object-detection methods:

•Active perception: The P-body has the capability to move and ensure that D’Kitty remains within its field of view. This ability is achieved through its perception policy

π_{p}

•P–C interaction: The trajectory envisioned by the C-body serves as valuable guidance for the P-body, enabling it to better track D’Kitty’s behavior. This results in an improved policy

a_{p} (t) = π_{p} (s_{p} (t) | s_{c} (t + Δ t))

, where

s_{c} (t + Δ t)

represents the sub-goal at time

t + Δ t

in Eq. (4).

In this section, we will demonstrate the benefits of these specific designs. An overview of the experimental flowchart is provided in Fig. 4, Fig. 5.

We compiled two distinct datasets for our training: an offline dataset and an online dataset. The offline dataset was constructed using DeepMind Control Suite and MuJoCo in combination with the Robel environment. This was achieved by integrating a fixed camera into the setup. The implemental details of P-body are shown in Fig. 6. After training on this offline dataset, the passive perception module was established and kept static for the subsequent training of the active perception module. The online dataset was generated through real-time interactions between the environment and the moving camera. It should be noted that the online images do not need to be stored, since the active perception model dynamically engages with the environment. All images have a consistent input size of

480 \times 480

4.1.1. Evaluation metrics

Our evaluation employs five key metrics: precision, recall, F₁ score, F₂ score, and mean average precision (mAP). The utilization of the mAP is a prevalent practice in the computer vision research community for assessing the performance of object-detection models.

4.1.2. Results and analyses

We present the quantitative outcomes in Table 1 [33], [52], highlighting the best results in bold font. In this table, the method labelled as “fixed” [33] denotes the scenario where the camera is stationary and unmoving. The “fixed” approach delivers the weakest performance because D’Kitty consistently moves out of the camera’s field of view. The term “random” signifies the case where the camera is set in motion by initializing three vectors to determine the extent of movement along the x, y, and z axes. The “active” method [52] refers to active perception, where the perception agent is trained by an RL method within hand-crafted rewards, in order to adjust the sensors’ configurations and positions. The designation “policy” refers to a variation of our method that lacks the guidance of sub-goals, while “policy + sub-goal” represents the complete implementation of our proposed method.

Our observations reveal that our method outperforms all other three baselines across the most fundamental and rigorous evaluation metric—namely, mAP. This outcome substantiates the superiority of our approach. It is noteworthy that the recall of the “policy” method surpasses that of our method. We speculate that, being devoid of assistance from sub-goals, the “policy” method tends to gather an excessive number of samples at the beginning. This likely contributes to its improved recall value but diminished precision in comparison with our method.

4.2. Evaluations of locomotion and navigation

The experiments conducted in this subsection provide a comprehensive validation of D’Kitty’s capabilities with respect to both locomotion (i.e., tasks such as standing up and walking) and navigation (i.e., strategies for reaching a designated target location).

4.2.1. Observation and sub-goal spaces

In the context of these evaluations, the complete observation space

o (t)

for D’Kitty at a given time step

t

encapsulates a rich set of variables. This includes D’Kitty’s current position and velocity, and the degree of alignment of its heading with respect to the target location. In addition, information about the 12 joint angles and their corresponding angular velocities are encompassed within this space. Moreover, the state vectors

s_{p} (t)

and sub-goal

s_{c} (t)

are intricately linked to the 2D location of D’Kitty within the environment.

4.2.2. Predictive planning

Upon receiving the initial state and target position, the C-body engages in the formulation of a predictive trajectory of states, specifically represented as

\{s_{c} (0), s_{c} (Δ t), . . ., s_{c} (T)\}

. Subsequently, contingent upon each predictive state

s_{c} (t)

, the behavior policy of the body (i.e., the B-body) orchestrates a series of actions, with the objective of attaining the subsequent sub-goal

s_{c} (t + Δ t)

within the course of

Δ t

steps.

4.2.3. Knowledge update

Following an interaction window spanning

Δ t

steps, an accumulated residual error, characterized by

s_{c} (t + Δ t) - s_{p} (t + Δ t)

, is conveyed to the C-body. This information serves as the catalyst for the updating of its knowledge parameters

k_{c}

through Eq. (10). Concurrently, the B-body undertakes the task of refining its policy to mitigate this error, thereby facilitating improved performance during subsequent training iterations.

In essence, this cohesive interaction loop seamlessly integrates perception, planning, and action, fostering the acquisition of enhanced knowledge by the C-body while concurrently enabling the B-body to fine-tune its policy for optimized execution.

4.2.4. Implementation details

Within the C-body, the cognition transition probability model

g_{c}

is instantiated using a multilayer perceptron (MLP). During each episode, the C-body provides sub-goals that guide the interactions of the B-body with the environment. The experiences obtained from these interactions are stored in a cache

C

for self-supervised C-body training. To overcome non-stationarity issues caused by the simultaneous training of the P-body and B-body, an “all-in all-out” training strategy is employed. This involves the cyclic processing of the cache, wherein experiences are divided into training and test sets for iterative model training. Besides, the implement details of C-body and B-body, as well as their interaction, are illustrated in Fig. 7.

For the B-body, the task is divided into a sequence of goal-conditioned subtasks with intervals of

Δ t

. Each subtask incorporates a sub-goal from the C-body to facilitate interactions with the environment. The behavior policy is trained using PPO, where a three-layer MLP serves as the policy approximator. All hyper-parameters are listed in Table 2.

4.3. Evaluation results

To assess the training and testing stages, the D’Kitty robot’s initial location is fixed at the origin (0, 0) with a standard stance, and target locations are chosen randomly within the x–y coordinate space. After every 50 training episodes, ten tests are conducted, and the average cumulative reward and success rate are computed for presentation.

4.3.1. Reward function and success indicator

The reward (r) function is defined as follows referring to Ref. [54]:

(14)

r (s, a) = r_{u p r i g h t} - 4 d_{t, g o a l} + 2 h_{t, g o a l} + r_{b o n u s_s m a l l} + r_{b o n u s_b i g}

(15)

r_{b o n u s_s m a l l} = 5 (d_{t, g o a l} < 0.5 o r h_{t, g o a l} > c o s 25^{°})

(16)

r_{b o n u s_b i g} = 10 (d_{t, g o a l} < 0.5 o r h_{t, g o a l} > c o s 25^{°})

where

r_{u p r i g h t}

is the reward encouraging the robot to keep upright and

r_{u p r i g h t} = 1

α_{u p r i g h t} = c ° s 15^{°}

r_{u p r i g h t} = - 500

α_{u p r i g h t}

is the cosine of the angle between the body normal and the ground,

d_{t, g ° a l}

represents the distance from current state to the target,

h_{t, g ° a l}

is the cosine of the angle of the robot's orientation in relation to the target line and

r_{b ° n u s_s m a l l}

and

r_{b ° n u s_b i g}

are the bonuses for reaching the desired target. From this reward, it can be seen that this task requires the simulated D’Kitty to reach the target and maintain its balance.

The success indicator is whether the goal distance is within a certain threshold and D’Kitty is sufficient upright at the last step of the episode:

(17)

\begin{matrix} E_{τ \sim π} [1 (d_{T, g o a l}^{(τ)} < 0.5) \times 1 (h_{T, g o a l}^{(τ)} > c o s 25^{°})] \end{matrix}

where

E i s e x p e c t a t i ° n;

τ is the state-action trajectory of single task;

d_{T, g ° a l}^{(τ)}

is the distance from the current position of D’Kitty to the target when task completes; and

h_{T, g ° a l}^{(τ)}

is the cosine of the final angle when task completes.

4.3.2. Results and analysis

The evaluation results for the success rate and reward are depicted in Fig. 8, Fig. 9, respectively. And, we also provide visualization results in Appendix A Video S1 to show the dynamic motion process. Two variants of the method are compared: one employing the ground-truth states, denoted as CB-body (the C-body and the B-body) , and the other employing the predicted states by the P-body, denoted as PCB-body (the P-body, C-body, and B-body). Several baseline methods utilizing PPO [49], SAC [41], or temporal difference model (TDM) [53] for individual B-body control are also included. CB exhibits better control performance and success rates, with the method showing resilience to local optima. PPO, however, converges to a lower reward due to limited exploration, resulting in the robot learning only to stand and orient itself toward the target, without mastering the walking motion.

When comparing PCB-body with CB-body, a minor performance drop is observed. This is attributed to the fact that the states predicted by the P-body do not precisely match the ground truth, leading to a slight degradation in PCB-body’s performance. Without ground-truth perception information such as the position and velocity of the D’Kitty robot and the coordinates of the target and obstacle, the P-body will enable information extraction from the images. Then, the C-body and B-body will receive the perception information from the P-body and drive the robot to the target. In short, the PCB-body (P-body, C-body, and B-body) presents in a more challenging environment than CB-body, while the performance degradation is slight, which demonstrates the effectiveness of the inherent cooperation of the three bodies in the Bcent framework. Nevertheless, PCB-body significantly outperforms the baselines, highlighting the efficacy of the interplays among the three bodies in enhancing the control performance and task success rate.

We also report the training wall time and the computing infrastructure in Table 3, to show the computational efficiency. From the results, it can be seen that, even within the synchronous training of three bodies, Bcent can be trained efficiently in general computing infrastructures.

5. Discussion and Conclusions

In this paper, we introduce a pioneering decision-making framework that harnesses the collaborative development of a P-body, a C-body, and a B-body. Our framework transcends the conventional agent paradigm by incorporating the C-body, enabling a more faithful emulation of human cognitive processes and dynamic decision-making. The physical synergy among these three bodies contributes to their collective enhancement. Through extensive experiments involving D’Kitty’s navigation and locomotion, we validate the necessity and effectiveness of each proposed component compared with existing methods and alternative variations.

This work lays a foundation for numerous future explorations in cognitive robotics:

•The collaboration of multiple cognitive bodies: Drawing inspiration from the biology of creatures like octopuses, in which different brains serve distinct purposes, a fascinating avenue would be to explore how multiple cognitive bodies can synergize to achieve superior performance.

•Human–robot collaboration in dynamic environments: As robots increasingly collaborate with humans and fellow robots in diverse tasks, there is a compelling need to study how robots can effectively interact with dynamic environments while working alongside humans.

•Distributed sensing and learning in the cloud: The concept of cloud robots is gaining momentum. The integration of distributed local sensing with the global sharing of information through the cloud could revolutionize robotic capabilities.

•Non-Markov decision-making: Traditional decision processes are often Markovian, with the agent lacking the capability to remember past trajectories. Embracing non-Markovian decision processes could allow robots to leverage their complete history and experiences, leading to more informed and holistic decision-making.

In conclusion, our framework introduces a ground-breaking perspective in the field of cognitive robotics, emphasizing collaboration among distinct cognitive bodies. This work opens the door to a multitude of exciting avenues for future research and innovation. Our proposed Bcent framework is generally versatile and can be applied to various types of robots or tasks by selecting specific state and action spaces for the three bodies involved. In human–robot scenarios, potential challenges exist in two aspects: In the aspect of perception, in order to better communicate with humans, a robot should understand human intentions and know what the humans plan to do during human–robot interactions. In the aspect of behavior, the robot should be able to learn from human demonstrations and imitate skills from humans in order to improve its behavior capability. In our Bcent framework, in addition to perception and behavior, we utilize cognition to update the knowledge in order to realize continuous learning and life-long learning, which can help Bcent deal with complex human-robot interaction scenarios.

CRediT authorship contribution statement

Fuchun Sun: Writing – review & editing, Supervision, Project administration, Conceptualization. Wenbing Huang: Writing – original draft, Methodology, Formal analysis. Yu Luo: Writing – review & editing, Visualization, Validation, Formal analysis. Tianying Ji: Writing – review & editing, Software, Data curation. Huaping Liu: Supervision, Resources, Formal analysis. He Liu: Supervision, Resources, Formal analysis. Jianwei Zhang: Methodology, Investigation, Conceptualization.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgments

This work was jointly funded by the National Science and Technology Major Project of the Ministry of Science and Technology of China (2018AAA0102900) and the “New Generation Artificial Intelligence” Key Field Research and Development Plan of Guangdong Province (2021B0101410002).

Appendix A. Supplementary material

Supplementary data to this article can be found online at https://doi.org/10.1016/j.eng.2024.10.013.

References

Publishing order | Descend order by publishing year | Descend order by cited within

[1]	Miriyev A, Kova Mč.Skills for physical artificial intelligence.Nat Mach Intell 2020; 2(11):658-660.

[2]	Murray RM.Feedback systems: an introduction for scientists and engineers.Princeton University Press, Princeton (2010)

[3]	Sünderhauf N, Brock O, Scheirer W, Hadsell R, Fox D, Leitner J, et al.The limits and potentials of deep learning for robotics.Int J Robot Res 2018; 37(4–5):405-420.

[4]	Wang W, Siau K.Artificial intelligence, machine learning, automation, robotics, future of work and future of humanity: a review and research agenda.J Database Manage 2019; 30(1):61-79.

[5]	Osa T, Pajarinen J, Neumann G, Bagnell JA, Abbeel P, Peters J.An algorithmic perspective on imitation learning.Found Trends Robotics 2018; 7(1–2):1-179.

[6]	Kretzschmar H, Spies M, Sprunk C, Burgard W.Socially compliant mobile robot navigation via inverse reinforcement learning.Int J Robot Res 2016; 35(11):1289-1307.

[7]	Kohl N, Stone P.Policy gradient reinforcement learning for fast quadrupedal locomotion.In: Proceedings of the IEEE International Conference on Robotics and Automation; 2004 Apr 26–May 1; New Orleans, L A, US A. New York City: IEE E; 2004. p. 2619–24.

[8]	Akkaya I, Andrychowicz M, Chociej M, Litwin M, McGrew B, Petron A, et al.Solving rubik’s cube with a robot hand.2019. arXiv: 1910.07113.

[9]	Zhang K, Yang Z, Basar T.Multi-agent reinforcement learning: a selective overview of theories and algorithms.K.G. Vamvoudakis, Y. Wan, F.L. Lewis, D. Cansever (Eds.), Handbook of reinforcement learning and control, Springer, Berlin 2021; 321-384.

[10]	Ahn C, Kim E, Oh S.Deep elastic networks with model selection for multi-task learning.In: Proceedings of the IEEE/CVF International Conference on Computer Vision; 2019 Oct 27–Nov 2; Seoul, Republic of Korea. New York City: IEE E; 2019. p. 6529–38.

[11]	He K, Zhang X, Ren S, Sun J.Deep residual learning for image recognition.In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; 2016 Jun 27–30; Las Vegas, N V, US A. New York City: IEE E; 2016. p. 770–8.

[12]	Howard AG, Zhu M, Chen B, Kalenichenko D, Wang W, Weyand T, et al.MobileNets: efficient convolutional neural networks for mobile vision applications.2017. arXiv: 1704.04861.

[13]	Krizhevsky A, Sutskever I, Hinton GE.ImageNet classification with deep convolutional neural networks.In: Proceedings of the 26th Annual Conference on Neural Information Processing Systems; 2012 Dec 3–6; Lake Tahoe, N A, US A. Trier: the dblp computer science bibliography; 2012. p. 1097–105.

[14]	Girshick R.Fast R-CNN.In: Proceedings of the IEEE International Conference on Computer Vision; 2015 Dec 7–13; Santiago, Chile. New York City: IEE E; 2015. p. 1440–48.

[15]	Girshick R, Donahue J, Darrell T, Malik J.Rich feature hierarchies for accurate object detection and semantic segmentation.In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; 2014 Jun 23–28; Columbus, O H, US A. New York City: IEE E; 2014. p. 580–7.

[16]	He K, Zhang X, Ren S, Sun J.Spatial pyramid pooling in deep convolutional networks for visual recognition.IEEE Trans Pattern Anal Mach Intell 2015; 37(9):1904-1916.

[17]	Redmon J, Farhadi A.YOLO9000: better, faster, stronger.In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; 2017 Jul 21–26; Honolulu, H I, US A. New York City: IEE E; 2017. p. 7263–71.

[18]	Redmon J, Farhadi A.YOLOv3: an incremental improvement.2018. arXiv: 1804.02767.

[19]	He K, Gkioxari G, Dollár P, Girshick R.Mask R-CNN.In: Proceedings of the IEEE International Conference on Computer Vision; 2017 Oct 22–29; Venice, Italy. New York City: IEE E; 2017. p. 2961–9.

[20]

Ren S, He K, Girshick R, Sun J.Faster R-CNN: towards real-time object detection with region proposal networks.In: Proceedings of the Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems; 2015 Dec 7–12; Montreal, Q C, Canada. Cambridge: The MIT Press; 2015. p. 91–9.

[21]	Lin TY, Dollár P, Girshick R, He K, Hariharan B, Belongie S.Feature pyramid networks for object detection.In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition-2017; 2017 Jul 21–26; Honolulu, H I, US A. New York City: IEE E; 2017. p. 2117–25.

[22]	Redmon J, Divvala S, Girshick R, Farhadi A.You only look once: unified, real-time object detection.In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition-2016; 2016 Jun 27–30; Las Vegas, N V, US A. New York City: IEE E; 2016. p. 779–88.

[23]	Liu W, Anguelov D, Erhan D, Szegedy C, Reed S, Fu CY, et al.SSD: single shot multibox detector.In: Proceedings of the European Conference on Computer Vision; 2016 Oct 11–14; Amsterdam, the Netherlands. Berlin: Springer; 2016. P. 21–37.

[24]	Lin TY, Goyal P, Girshick R, He K, Dollár P.Focal loss for dense object detection.In: Proceedings of the IEEE International Conference on Computer Vision; 2017 Oct 22–29; Venice, Italy. New York City: IEE E; 2017. p. 2980–8.

[25]	Law H, Deng J.CornerNet: detecting objects as paired keypoints.In: Proceedings of the European Conference on Computer Vision (ECC V 2018); 2018 Sep 8–14; Munich, Germany. Berlin: Springer; 2018. p. 734–50.

[26]	Zhou X, Zhuo J, Krahenbuhl P.Bottom-up object detection by grouping extreme and center points.In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition-2019; 2019 Jun 15–20; Long Beach, C A, US A. New York City: IEE E; 2019. p. 850–9.

[27]	Zhu C, He Y, Savvides M.Feature selective anchor-free module for single-shot object detection.2019. arXiv: 1903.00621.

[28]	Dosovitskiy A, Beyer L, Kolesnikov A, Weissenborn D, Zhai X, Unterthiner T, et al.An image is worth 16 ×16 words: transformers for image recognition at scale.2020. arXiv: 2010.11929.

[29]	Liu Z, Lin Y, Cao Y, Hu H, Wei Y, Zhang Z, et al.Swin transformer: hierarchical vision transformer using shifted windows.In: Proceedings of the IEEE/CVF International Conference on Computer Vision; 2021 Oct 10–17; Montreal, Q C, Canada. New York City: IEE E; 2021. p. 10012–22.

[30]	Fang Y, Liao B, Wang X, Fang J, Qi J, Wu R, et al.You only look at one sequence: rethinking transformer in vision through object detection.In: Proceedings of the 35th Annual Conference on Neural Information Processing; 2021 Dec 6–14; online. San Diego: Neural Information Processing Systems; 2021.

[31]	Song H, Sun D, Chun S, Jampani V, Han D, Heo B, et al.ViDT: an efficient and effective fully transformer-based object detector.2021. arXiv: 2110.03921.

[32]	Jing M, Ma X, Huang W, Sun F, Yang C, Fang B, et al.Reinforcement learning from imperfect demonstrations under soft expert guidance.Proc Conf AAAI Artif Intell 2020; 34(04):5109-5116.

[33]	Kong T, Sun F, Liu H, Jiang Y, Li L, Shi J.Foveabox: beyound anchor-based object detection.IEEE Trans Image Process 2020; 29:7389-7398.

[34]	Liu H, Wang F, Guo D, Liu X, Zhang X, Sun F.Active object discovery and localization using sound-induced attention.IEEE Trans Industr Inform 2021; 17(3):2021-2029.

[35]	Bajcsy R, Aloimonos Y, Tsotsos JK.Revisiting active perception.Auton Robots 2018; 42(2):177-196.

[36]	Liu H, Den Y, Guo D, Fang B, Sun F, Yang W.An interactive perception method for warehouse automation in smart cities.IEEE Trans Industr Inform 2021; 17(2):830-838.

[37]	Silver D, Singh SP, Precup D, Sutton RS.Reward is enough.Artif Intell 2021; 299:103535.

[38]	Mnih V, Kavukcuoglu K, Silver D, Rusu AA, Veness J, Bellemare MG, et al.Human-level control through deep reinforcement learning.Nature 2015; 518(7540):529-533.

[39]	Sutton RS, McAllester D, Singh S, Mansour Y.Policy gradient methods for reinforcement learning with function approximation.In: Proceedings of the Annual Conference on Neural Information Processing Systems (NIP S 1999); 1999 Nov 29–Dec 4; Denver, C O, US A. Cambridge: The MIT Press; 1999.

[40]

Lillicrap TP, Hunt JJ, Pritzel A, Heess N, Erez T, Tassa Y, et al.Continuous control with deep reinforcement learning.In: Proceedings of the 4th International Conference on Learning Representations, ICL R 2016; 2016 May 2–4; San Juan, Puerto Rico. Trier: the dblp computer science bibliography; 2016.

[41]

Haarnoja T, Zhou A, Abbeel P, Levine S.Soft actor-critic: off-policy maximum entropy deep reinforcement learning with a stochastic actor.In: Proceedings of the 35th International Conference on Machine Learning; 2018 Jul 10–15; Stockholm, Sweden. New York City: Proceedings of Machine Learning Research; 2018. p. 1856–65.

[42]	Wang T, Bao X, Clavera I, Hoang J, Wen Y, Langlois E, et al.Benchmarking model-based reinforcement learning.2019. ar Xiv:1907.02057v1.

[43]	Janner M, Fu J, Zhang M, Levine S.When to trust your model: model-based policy optimization.In: Proceedings of the Annual Conference on Neural Information Processing Systems; 2019 Dec 8–14; Vancouver, B C, Canada. San Diego: Neural Information Processing Systems Foundation, Inc.; 2019. p. 12498–09.

[44]	Tassa Y, Erez T, Todorov E.Synthesis and stabilization of complex behaviors through online trajectory optimization.In: Proceedings of the 2012 IEE E/RSJ International Conference on Intelligent Robots and Systems; 2012 Oct 7–12; Vilamoura, Portugal. New York City: IEE E; 2012. p. 4906–13.

[45]	Zhou Z, Yan N.A survey of numerical methods for convection-diffusion optimal control problems.J Numer Math 2014; 22(1):61-85.

[46]	De PT Boer, Kroese DP, Mannor S, Rubinstein RY.A tutorial on the cross-entropy method.Ann Oper Res 2005; 134(1):19-67.

[47]

Chua K, Calandra R, McAllister R, Levine S.Deep reinforcement learning in a handful of trials using probabilistic dynamics models.In: Proceedings of the Annual Conference on Neural Information Processing Systems; 2018 Dec 3–8; Montreal, Q C, Canada. Red Hook: Curran Associates Inc.; 2018. p. 4759–70.

[48]	Yildiz C, Heinonen M, Lähdesmäki H.Continuous-time model-based reinforcement learning.In: Proceedings of the International Conference on Machine Learning; 2021 Jun 18–24; online. New York City: Proceedings of Machine Learning Research; 2021. p. 12009–18.

[49]	Schulman J, Wolski F, Dhariwal P, Radford A, Klimov O.Proximal policy optimization algorithms.2017. arXiv: 1707.06347.

[50]	Schulman J, Levine S, Abbeel P, Jordan M, Moritz P.Trust region policy optimization.In: Proceedings of the International Conference on Machine Learning; 2015 Jul 6–11; Lille, France. New York City: Proceedings of Machine Learning Research; 2015. p. 1889–97.

[51]	Chang D, Johnson-Roberson M, Sun J.An active perception framework for autonomous underwater vehicle navigation under sensor constraints.IEEE Trans Control Syst Technol 2022; 30(6):2301-2316.

[52]	Amos B, Jimenez I, Sacks J, Boots B, Zico Kolter J.Differentiable MPC for end-to-end planning and control.In: Proceedings of the 32nd Annual Conference on Neural Information Processing Systems (NIP S 2018); 2018 Dec 2–8; Montreal, Q C, Canada. San Diego: Neural Information Processing Systems; 2018.

[53]	Pong V, Gu S, Dalal M, Levine S.Temporal difference models: model-free deep RL for model-based control.In: Proceedings of the International Conference on Learning Representations; 2018 Apr 30–May 3; Vancouver, B C, Canada. Trier: the dblp computer science bibliography; 2018.

[54]	Ahn M, Zhu H, Hartikainen K, Ponte H, Gupta A, Levine S, Kumar V.ROBEL: robotics benchmarks for learning with low-cost robots.In: Proceedings of the Conference on Robot Learning; 2020 Nov 16–18; online; 2020.

RIGHTS & PERMISSIONS

THE AUTHOR

PDF (1947KB)

Part of a collection:

Supplementary files

supplementary data

6983

Accesses

Citation

Detail

Sections

Recommended

Browse

Authors & reviewers

About the journal

Abstract

Graphical abstract

Keywords

Cite this article

1. Introduction

2. Related works

2.1. Vision perception

2.2. Reinforcement learning

3. Methodology

3.1. The perception-body

3.1.1. Model-free perception policy learning via P–C interaction

3.1.2. P–C interaction

3.2. The cognition-body

3.2.1. Knowledge utilization

3.2.2. Knowledge update

3.2.3. P–C interaction

3.2.4. C–B interaction

3.3. The behavior-body

4. Experiments

4.1. Evaluations of perception

4.1.1. Evaluation metrics

4.1.2. Results and analyses

4.2. Evaluations of locomotion and navigation

4.2.1. Observation and sub-goal spaces

4.2.2. Predictive planning

4.2.3. Knowledge update

4.2.4. Implementation details

4.3. Evaluation results

4.3.1. Reward function and success indicator

4.3.2. Results and analysis

5. Discussion and Conclusions

CRediT authorship contribution statement

Declaration of competing interest

Acknowledgments

Appendix A. Supplementary material

References

RIGHTS & PERMISSIONS