《1 Introduction 》

1 Introduction

Disruptive technology is considered a revolutionary force with “game-changing” and “future-transforming” results [1]. It has several broad and profound impacts on human society: promoting the evolution and development of civilization, affecting the rise and fall of world powers, dominating the survival of organizations, and changing people’s production activities and lives. Before the advent of modern society, disruptive technologies had been continuously created and had always changed human producing activities and lives, as well as promoted the progress of human society; however, people had not formed a mindset of actively pursuing disruptive technologies that go through a long “natural development stage.”

With the development of technology and the competition between the US and the Soviet Union, the US government took the lead in disruptive technological innovation and pushed the development of disruptive technologies into the fast lane. After the Second World War, the carrying out of nuclear tests and successful launch of satellites by the Soviet Union alarmed the US. The US government reexamined military science and technology innovation systems and realized the need to consciously and systematically cultivate high-risk but high-reward technologies to avoid surprises from the Soviet Union’s technological development. In this context, the US established the Advanced Research Projects Agency (ARPA) in 1958; it was renamed as the US Defense Advanced Research Projects Agency (DARPA) in 1972. DARPA’s mission is to develop game-changing, high-risk, but high-reward technologies for “avoiding technological surprise and creating technological surprise for adversaries.” To this end, a large number of disruptive technologies produced by DARPA, such as those related to the Internet, global positioning, and stealth weapons have played an important role in not only maintaining technological leadership of the US military, but also safeguarding national security [2,3].

The DARPA model has gradually matured, and its influence has expanded continuously. It has been promoted by the US government and enterprises and is followed by the world’s major powers. The US government has established a DARPA-like advanced research project bureau in departments such as the Department of Homeland Security and the Department of Energy, and has formed a unique program support system. In recent years, the development of information technology, represented by the Internet, has accelerated knowledge accumulation and technological innovation, and disruptive technologies have emerged in an endless stream. This has led to growing attention from enterprises and the public. Many companies have been involved in actively cultivating disruptive technologies. Under the combined effect and long-term integration of “national guidance, technology promotion, and market traction,” the US has formed a disruptive technological innovation system that originates in its national defense strategy, is rooted in the national innovation environment, extends to the national economic system, and fully caters to the characteristics of disruptive technologies. DARPA is the core and essence of this system. This study analyzes the historical events surrounding the formation of DARPA’s management model of disruptive technological innovation, discusses the reasons for the success of DARPA from the perspective of evolutionary theory, and obtains relevant insights.

《2 Description of DARPA》

2 Description of DARPA

《2.1 Mission and positioning》

2.1 Mission and positioning

DARPA, an administrative agency under the Department of Defense, is responsible for the development of high-risk but high-reward technologies that are “game-changing”; the focus is on “avoiding technological surprise and creating technological surprise for adversaries,” as well as maintaining technological superiority of the US military. DARPA was established in 1958 and has more than 200 staff members, most of whom are top-notch experts and scholars in various disciplines [3,4].

《2.2 Organizational mode and operational mechanism》

2.2 Organizational mode and operational mechanism

DARPA is a typical organization featuring a flat management system. Its operation is based on the “project manager system” and adopts a two-level decision-making system of “professional sector department and technical project manager” as the core organizational structure; this leads to high flexibility. In terms of staff, DARPA offers four types of employment: permanent full-time, permanent part-time, appointed full-time, and appointed part-time. As for the management of project managers, DARPA adopts a method wherein the “project manager shall be with the project every step of the way” based on a principle of “limited management.” The project management is carried out through the following steps: recruitment; vision development; project kick-off; combined management; and technology transition. In the processes of project decision making, investment, promotion, and exit, DARPA adheres to innovation-oriented guidelines and carries out policy innovations through brainstorming sessions [3−6].

DARPA has seven technical project offices and around 100 project managers, with an estimated budget of USD 3 billion allocated to more than 200 research projects annually. It receives less than 1% of the national research and development (R&D) funds and less than 4% of the Department of Defense’s research, development, testing, and evaluation (RDT&E) budget. The organizational structure of DARPA is shown in Fig. 1 [4].

《Fig.1》

Fig. 1. Organizational structure of DARPA.

《3 Analysis of DARPA’s disruptive innovation experience》

3 Analysis of DARPA’s disruptive innovation experience

In the context of “Everybody loves DARPA,” Chinese and foreign consulting organizations, think tanks, academic circles, and DARPA itself have begun to explore DARPA’s sources of innovation, project management, organizational model, or transition mechanism to verify the feasibility of replicating DARPA’s success. Some departments or companies have even hired DARPA’s former directors or project managers to prepare and manage new institutions; these include the first Deputy Director of Homeland Security Advanced Research Projects Agency (HSARPA), Jane Alexander (former Deputy Director of DARPA), and Regina Dugan, head of Google’s Advanced Technology and Projects group (formerly DARPA’s Director). However, so far, there have been few achievements that can match those of DARPA’s. HSARPA, an organization older than 10 years, has always been criticized. In general, there is no consensus on how to build another organization as brilliant as DARPA, and there is no known way to emulate its achievements.

From the practical experience of various entities that imitated DARPA’s model in the past ten years, DARPA’s dominant organizational structure and operational mechanism appear to be the bases of DARPA’s success, but not its essence. After several consultations, we interpreted the reasons behind DARPA’s success by employing the environmental and strategic perspectives discussed in the following sections.

《3.1 A harsh living environment was the main driving force for DARPA’s innovation and change》

3.1 A harsh living environment was the main driving force for DARPA’s innovation and change

The environment of organizations shapes them, and DARPA is no exception. In particular, DARPA had faced serious survival challenges, especially in the first ten years of its existence, as discussed below.

(1) It faced cut-throat competition for survival with the armed forces. The 1950s was the most intense period of infighting in the US military. The problems caused by repeated reorganization and fragmentation were extremely serious. The efficiency of research and development was low. At that time, President Eisenhower detested the internal strife and seized the opportunity to establish DARPA when the Soviet Union successfully launched a satellite that came as a great shock to the US. Although one of the objectives was to end the struggles resulting from conflicts of interest among the forces, it led to subsequent conflicts between DARPA and the armed forces. Ensuring its survival in the competition with the armed forces had also become a core interest of DARPA. History also shows that DARPA came close to being abolished.

(2) DARPA lost a key business for settling down in the early days of its establishment. In October 1958, the National Aeronautics and Space Administration (NASA) was established, and the Department of Defense immediately assigned DARPA’s military space mission (a core work of DARPA, accounting for nearly two-thirds of the funding) to NASA. For the first time, DARPA faced a challenge that threatened its survival.

(3) The living environment in the US deteriorated. First, because of the Vietnam War, the relationship between the military and academic circles had become increasingly tense. As a result, the expectations of a majority of university researchers from the Department of Defense and national security policies had gradually been belied, and these researchers refused to participate in defense research projects. Second, in terms of the economic environment, the wage gap between employees of the US federal government and the industrial circles had widened significantly, resulting in a serious brain drain in federal government agencies. The high unemployment rate was accompanied by a decrease in the flow of talent from the industry and universities; consequently, DARPA could not attract talent. In addition, high-level changes and shifts in the Department of Defense’s science and technology management policies had steadily weakened DARPA’s status, and its projects were neither of concern to the Secretary of Defense nor were they related to major national policy issues. The combined result of these factors was that DARPA was no longer seen as a “scientific highland” in the US.

In the first ten years of its establishment, DARPA had experienced several crises threatening its survival and was in danger of being abolished several times. Faced with tremendous environmental pressures, it focused on the following three major changes.

(1) Motivating the original leaders to think and stick to the uniqueness of DARPA: Director Lukasik, the director during its period of crises, believed that their experience could be likened to embracing the “kiss of death”; this is because uniqueness is hard to recognize. However, it was this insistence that gave DARPA its innovative quality and gained the recognition of a large number of “science enthusiasts” who wanted to change the world and realize self-fulfillment.

(2) Undertaking high-risk and challenging tasks that the other forces were unwilling to do: after losing its main business, DARPA had to find a way out of its predicament and began to engage in difficult, cross-service, and national-security-related projects that other military organizations were unwilling to take on and ultimately established its business orientation and status.

(3) Weakening departmentalism: this enabled DARPA to break its own interest barriers; continuously improve relations with the armed forces, industrial circles, and the scientific community; as well as improve its management to respond in a timely manner. It was these difficult processes that enabled DARPA to break through the restrictive walls that hindered common interest, discipline, education, and talent, all of which are necessary elements for disruptive technological innovation to succeed.

《3.2 Remaining true to its original aspiration is DARPA’s way of avoiding system obsolescence and pursuing innovation 》

3.2 Remaining true to its original aspiration is DARPA’s way of avoiding system obsolescence and pursuing innovation

Even under the combined pressure of the armed forces and the persistence of scientists, DARPA has always remained true to its original aspiration and focused on the mission of “developing future weapon systems,” “maintaining technological superiority of the US military, and avoiding surprise.” It has strived to prevent system obsolescence and pursue innovation.

(1) Remaining true to its original aspiration makes DARPA adhere to its unique mission and ensure that its system does not become obsolete. Moreover, DARPA has always adhered to the most challenging Science and Technology (S&T) phase, which refers to the “Science and Technology Plan” of the Department of Defense. After the project becomes successful or is verified, it is handed over to the military for further research, development, and final application. This “let the matured projects go” working concept prevents DARPA from sinking into innovation obsolescence. Further, DARPA has always adhered to its unique positioning, and has not expanded for profits. For example, from the 1960s to the early 1970s, other people have occasionally suggested that the basic research of the Department of Defense should be centralized and placed under the unified management of DARPA; this was strongly opposed by several directors. Although DARPA was almost abolished in the early days because of its preference for funding basic research, it became well-known that DARPA was not the National Science Foundation to the Department of Defense.

(2) Remaining true to its original aspiration makes DARPA effectively refrain from self-expansion, thus avoiding innovation from being “overwhelmed” by complex organizations. DARPA has no laboratory or research facilities. However, the Department of Defense has granted DARPA relevant authorizations, and DARPA did build and own some facilities, depending on projects’ requirements. However, the difference is that DARPA’s laboratories or facilities are not retained for a long time; they are transferred to other organizations as a project’s technology is transitioned. Although this may appear counterintuitive, it does inhibit DARPA’s expansion, preventing excessive self-interest and avoiding system obsolescence from hindering innovation.

(3) Remaining true to its original aspiration makes DARPA firmly hold its belief, maintain its vision, and support several major disruptive technologies. Disruptive technology-oriented operational concepts and equipment seriously impact existing habits or operational doctrines. The resistance in the military is high, and DARPA is required to lead the development of disruptive technologies and complete demonstrations in order to be accepted by the military. In addition, owing to immature technology and incomplete support, the R&D process continuously faces severe challenges and numerous setbacks. For example, in the case of the stealth aircraft, DARPA withstood the opposition of the US Air Force, overcame many difficulties, and developed the “Have Blue” stealth demonstrator. It is because of this unwavering faith and foresight that the US military took the lead in stealth technology and gradually changed the style of air operations.

《3.3 A behavioral pattern transcending military services, industry, and field, and forward-looking strategic vision are important factors that support disruptive technological innovation 》

3.3 A behavioral pattern transcending military services, industry, and field, and forward-looking strategic vision are important factors that support disruptive technological innovation

The basic path for DARPA’s project development is problem−thought−talent−project (group). The “problem” is an important aspect in DARPA’s pursuit of disruptive technological innovation. DARPA’s unique positioning enables it to transcend military services, industries, fields, and even science schools. It considers the most pressing issues under the national security strategy and is more likely to realize the vision of disruptive technologies and promote new ideas and theories for disruptive technological innovation. In the late 1950s, the US military’s consensus was that its command and control system could not meet the urgent need for rapid decision making in an increasingly complex and rapidly changing military environment. In 1961, President Kennedy asked the military to improve this system. After the major issue of national defense security was raised, the Department of Defense assigned the responsibility for the project to DARPA. To this end, DARPA established the Information Processing Technology Office (IPTO) and invited J.C.R. Licklider, a professor from the Massachusetts Institute of Technology, to serve as its first director. Although the project was urgent for the military and constituted an imperative for the president, DARPA did not succumb to the demands of the military. Instead, based on the idea of “man–computer symbiosis” proposed by J.C.R. Licklider, which states that man–computer interaction is the essence of command and control, DARPA carried out a long-term, continuous research work on this project. Since then, IPTO has gradually explored several new fields in computer science and information-processing technology by following the ideas of J.C.R. Licklider; further, it developed epoch-making disruptive technologies, such as ArpaNet, which continues to have far-reaching effects.

Many DARPA directors emphasized the importance of the “problem.” For example, in an interview in 2006, Robert Sproull argued that the lack of big problems restricted other institutions from following the DARPA model [7]. In addition, the idea of turning problems to solutions is the key to disruptive technological innovation. A remarkable feature of DARPA is that it always, at the right time, pushes down a wall that blocks the convergence of ideas, breaks the barriers of imprisonment, and fosters its ability to recognize good ideas. This is even more important than the ability to come up with great ideas.

《3.4 High turnover rate is a foundation for maintaining innovative vitality》

3.4 High turnover rate is a foundation for maintaining innovative vitality

DARPA has an average annual staff turnover rate of approximately 25%. The scope of the director’s job is more extensive, and has an actual average term of approximately 30 months. First, ideas come from talented individuals, while a high turnover rate constantly introduces “fresh blood,” brings in new ideas, and further stimulates exchange of ideas. Second, a high turnover rate breaks the inherent barriers to innovation: conventional thinking and intellectual laziness. DARPA believes that conventional thinking and intellectual laziness are inherent barriers to innovation, meaning that no one can continuously innovate; new ideas come with personnel turnover. Finally, a high turnover rate, to some extent, prevents the growth of bureaucracy. To avoid bureaucracy and keep the structure streamlined, DARPA’s employment period for researchers is generally no more than six years.

《3.5 Technological faith is the spiritual support for DARPA’s innovation》

3.5 Technological faith is the spiritual support for DARPA’s innovation

(1) Value identification based on technological faith is the core cohesive force within DARPA. DARPA is a platform for the pursuit of innovative dreams, not a place to seek high salaries. What the “mad scientists” expect from short-term work at DARPA is to turn ideas into actual disruptive successes that can affect history to some extent, thereby gaining a sense of accomplishment. They value “unlimited imagination and creation” over monetary rewards. Based on such technological faith, DARPA attracts a large number of outstanding talents, regardless of salary differences or salary reductions; however, some benefit-oriented talents are discouraged.

(2) Technological faith allows for failure, which ignites the “enthusiasm for innovation” in DARPA’s researchers. When referring to the innovation culture leading to DARPA’s disruptive technologies, its innovative attitude of “allowing for failure” is widely recognized [8]. Indeed, DARPA’s investment failure rate is surprisingly high, but it cannot simply be attributed to this attitude. DARPA’s tolerance for failure is based on a trend-oriented strategic vision and an extraordinary creative ability. It is not overly focused on success or short-lived, mediocre results; this inspires enthusiasm and challenges researchers’ innovativeness, creating an atmosphere conducive to innovation.

(3) An innovative spirit that “emphasizes trust” based on technological faith plays a crucial role in the “orderly development” of team-based innovation research. Compared to traditional research institutions, DARPA grants greater autonomy to project managers and researchers. When a researcher produces an adventurous idea about a project, he can communicate this idea equally to colleagues or project managers, and the project manager can effectively communicate the mature concepts from the “brainstorm phase” to the director of DARPA for his or her support. Owing to the DARPA value of “emphasizing trust,” project researchers are more willing to be the promoters of DARPA, creating a “mutually beneficial union.”

《4 Enlightenment 》

4 Enlightenment

The story of DARPA is remarkable. In the past, DARPA was an inconspicuous “small institution” with numerous problems that posed serious challenges to its survival. On several occasions, it was on the verge of being abolished. For example, the loss of a core business pushed it to reexamine its purpose and define its identity. It had severe competition with the armed forces for a long period of time and could not arrive at the right positioning of its business; its poor compensation package meant that it could not attract the best talent (the average age of DARPA project managers in the 1960s and 1970s was around 30 years). The director’s position remained vacant for a long time; a director was even expelled from the Pentagon by the Department of Defense and became an “exile” in Siberia. There is a discrepancy between DARPA’s history and its current image as a “shining star.” However, the truth usually seems counterintuitive; it is precisely because of this “counterintuitive phenomenon” that disruptive technological innovation can be accomplished. The DARPA story provides the following enlightening insights.

《4.1 The biggest obstacle to disruptive technological innovation is the organization itself》

4.1 The biggest obstacle to disruptive technological innovation is the organization itself

As a revolutionary force that drastically alters tradition in various aspects, the process of achieving disruptive technology comes with two major conflicts in the fields of technology and of management. On the one hand, as a new “rail-changing” technology, disruptive technologies often adapt neither to the supporting technological and industrial systems in existence, nor even to commercial bases and business models. In its growth process, there is a huge conflict with the existing technological systems. On the other hand, the existing management system hinders or even rejects the development of disruptive technology in the following ways. (1) The growth of the existing value network stifles emerging disruptive technologies. New disruptive technologies restructure existing models and patterns, thereby facing serious conflicts over resources, processes, and values within the organization [9]. This, in turn, prohibits further development of disruptive technologies. (2) Conflicts of interest make it impossible to lay a solid foundation for disruptive technological innovation. The constraints on thought, conflicting interests, and divisions among departments prevent an organization from allowing a disruptive technology to emerge. (3) Institutional obsolescence erases the vitality of disruptive technological innovation. The obsolescence of organizational structures and styles of operation creates strong path dependence. It is difficult to stimulate disruptive technological innovation in such organizations.

Owing to major conflicts related to technology and management, it is difficult for disruptive technology to grow within traditional organizations. In the face of disruptive technological changes, the better the company is on its original path and the better the quality of management, the faster it would fail while pursuing disruptive technological innovation. Industry giants that dominate talents, technology, and capital are often the losers of disruptive change. For example, Kodak “invented a digital camera, but was subverted by its invention.” The DARPA story also supports this conclusion. When DARPA was established, the US military-industrial complex faced the divergence of interests in innovation and of innovation itself becoming obsolete. It was difficult to develop “game-changing” disruptive technology. In order to deal with the technological surprise from the Soviet Union, DARPA was established as an independent organization. An important factor behind DARPA’s success is its self-transformation amid an extreme environment, achieved by suppressing selfish departmentalism, breaking various walls, preventing organizational obsolescence, overcoming some of the inherent shortcomings of traditional organizations, and maintaining the vitality of innovation. The DARPA story tells us that the biggest obstacle to disruptive technological innovation is the organization itself. To successfully carry out such innovation requires a profound self-revolution.

《4.2 Accelerating the development of disruptive technology requires a unique management approach—becoming an “independent” organization 》

4.2 Accelerating the development of disruptive technology requires a unique management approach—becoming an “independent” organization

The success of DARPA reveals that successful disruptive technological innovation requires a unique management model, that is, being an “independent” organization (DARPA is an “independent” organization in the Department of Defense), that is not subject to the existing value network. Currently, in the face of disruptive innovation, both national and industry giants, such as DARPA, Google X, and the popular X Lab have chosen new management structures, in addition to the existing systems. The experience of DARPA shows that an “independent” organization may be guided by the following principles.

Mission orientation: With innovation as the mission, mobilize the resources of the whole society as a means to create innovation in the entire ecosystem.

Business positioning: Reach beyond all businesses and departments, do not pursue a specific direction, adhere only to the most challenging S&T phase, transfer the project to other departments after verification, and do not get tied to the innovation chain.

Institutional mechanism: Establish specialized “institutions” that operate outside the existing value network (resources, processes, and values). Keep the scale of the organization small, the management flat, and the project type flexible.

Implementation guarantee: Abandon field classification, suppress selfish departmentalism, avoid depending on a specific business direction and department, adopt a systematic behavioral pattern, and a long-term vision. Streamline organizations, keep the entity small, maintain flexibility, and avoid getting stuck in organizational obsolescence. “Push down the walls,” topple all kinds of walls that hinder gathering of ideas and talents, including the walls of thoughts, interests, departments, businesses, and groups.

《Acknowledgement 》

Acknowledgement

The historical materials cited in this paper are mainly taken from the DARPA story, compiled by Huang Simin, a professor of Tsinghua University and the consulting report The Road of Disruptive Technological Innovation of DARPA, authored by Mr. Huang. We express our heartfelt thanks to Mr. Huang for offering guidance and help during the writing process.