Requirements, Objectives, Both, or Neither: How to Formulate Complex Design Problems for Innovation Contests

Technical organizations often search for new ideas when designing new, complex systems [1,2]. While the literature describes various innovation paradigms to do so internally, scholars have also suggested opening up the earliest stages of this process to outsiders. Specifically, activities like innovation contests² allow individuals not part of the system’s domain or the organization to contribute [8,9]. Since individuals with different (combinations of) expertise perceive the problem differently [10], their input can find or inspire unique, and better, solutions [11,12]. For example, industries like additive construction [13] and robotics [4] have seen ground-breaking designs that outperformed what the domain’s experts expected.

Reaching (the right) outsiders can be challenging, however. The literature recommends that these contests be used to find new knowledge [11]: sourcing new solutions to existing problems or addressing new problems entirely. Here, two uncertainties are amplified as the organization—or solution-seeker—ventures into new (to them) areas of knowledge. First, they might be unable to rely on their expertise for design choices that lead to the desired performance: not knowing what a good solution looks like could negatively impact how they assess the incoming solutions [14]. Second, the seeker might be unable to forecast what expertise and capability will be required to create a good solution: not knowing what constitutes a good problem solver could negatively impact a search for them [3]. With limited information on what good solutions look like or who can produce them, the seeker must instead rely on an unknown crowd with unknown combinations of expertise and capability.

The nature of innovation contests amplifies these uncertainties. In particular, a solver must often deliver a working solution without the seeker’s design feedback [15]. Normally, stakeholder input is part and parcel of iterating a design [10]. In the contest environment, however, other participants might question the fairness of each information exchange with the seeker. Alternatively, solvers’ competitive advantage could be compromised if all is made public. As such, the seeker (effectively) has only one opportunity to decide on and communicate their problem before solutions are created and deployed [16]: at the contest’s release. It is the seeker’s chance to address the uncertainties of solutions and solvers, ensuring that the contest searches for useful, new knowledge from capable solvers. In practice, the seeker will often spend considerable time and budget on formulating the problem to get it right [17,18] because getting it wrong lowers the chances of useful solutions [19] and could mean wasted efforts for both seeker and solvers.

The innovation contest’s rules document is crucial in this formulation process. This design artifact communicates the seeker’s problem across boundaries to the solvers, representing the one opportunity to describe the need to those trying to solve the problem. It contains the contest’s rules and their “numbers” [20]: the desired system performance laid out as a set of constraints describing important design parameters and their thresholds³. If not carefully managed, the crowd might focus on different problem(s), risking significant uncertainty in the solutions [21].

Yet, despite the importance of the rules document and its contents, theoretical guidance to create these has been sparse [15]. To address this gap, we explored the process of formulating an innovation contest’s most important design artifact. Specifically, we asked, “How does the seeker create problem constraints for unknown outsiders?” We proposed an inductive model to answer this question, based on a multiyear fieldwork⁴ that captured several formulation processes.

Per our findings, the seeker’s attempts to mitigate design uncertainty at a granular level drove the formulation of each constraint. In this process, they balanced the need to communicate detailed information on the design problem with the risk of excluding valuable solvers and their solutions. Here, we identified three formulation approaches: impose, incentivize, and subsume. Their use depended on the seeker’s knowledge of potential solutions and (estimates of) solvers’ capabilities. These findings unpack a part of innovation contests that is still poorly understood and illuminate a path to resolve a longstanding debate in the engineering design literature. They also guide practitioners to navigate the formulation process in their own contexts.

A system’s success rests on correctly formulating its design problem [19,22]. In this process, the designer—the equivalent of the contest’s seeker—must translate an abstract need into specific constraints to progress the design. Here, they decide “what is part of the problem and what is not” [23]. The designer identifies the need within their broader context, then decomposes, and defines (un)acceptable designs [24,25]—naming and framing what will be designed [26]. Their decisions constrain the solution space and drive the subsequent design efforts. These also communicate the intended design to all involved, allowing the designer to delegate⁵ parts to team members—be they individuals or other organizations. However, uncertainty reigns and the stakes are high during this part of the design process: making wrong decisions on the design or delegation might induce system failures or costly rework [28,29].

Engineering design and systems engineering have long studied two mutually exclusive formulation strategies to navigate this process successfully [30,31]: requirement allocation and objective allocation. These project-level strategies mitigate the designer’s uncertainty by prescribing processes that translate their need into constraints. Both help designers shape the solution space and delegate the solving tasks (where required) to arrive at the best designs. Both strategies should produce the same design if the designer has perfect knowledge of the solution space. Yet, they differ in two important ways: how they restrict the exploration of the solution space and how they communicate the interfaces between the parts of the design. Since designers can only choose one of these strategies per design problem, scholars have debated the merits, drawbacks, and reasons for choosing for decades [32,33].

However, these strategies offer coarse direction, potentially obscuring the designers’ underlying process—especially in innovation contests. Scholars present requirement and objective allocation as mutually exclusive design philosophies—being opposing choices, they are always applied in separate contexts. Yet, the contest’s separation of formulating and solving a design problem could show different dynamics. In particular, to create a rules document that could result in useful solutions, the seeker must carefully shape the problem according to their knowledge and uncertainties [13]—guiding solvers on what aspects of the design to explore and what to incorporate as described. Faced with the granularity of this task, the seeker may use both strategies in one project, leveraging their strengths as needed.

As such, our exploratory study addresses the question of problem formulation by seekers while acknowledging the dynamics of innovation contests. We first review the requirement allocation and objective allocation strategies. We then summarize why—separately—they might not be able to guide a contest’s formulation, thus making a case for an inductive study of how this process works.

Per the requirement allocation strategy, constraints convey the specifics of the system’s design. Under this strategy, designers rely on their expertise to define design parameters and their thresholds to precisely define (each part of) the design through its (spatial, energy, data, and material) interfaces [25,34]. With high-level constraints defining firm boundaries, designers then allocate these to lower-level parts of the design [35,36]. The resulting statements are, ideally, “unambiguous, testable, and measurable” [31]: for example, “the system shall weigh less than X kg” [32]. These decisions divide the solution space into regions of acceptable and unacceptable designs, bounding what can be explored.

This strategy’s constraints facilitate communication across boundaries [28,37]. Their definitive nature helps a design team readily comprehend how all parts of the system interact and how they fit within that structure [25]. It also clarifies that a failure to interface as specified will result in a poor overall design [38]. Generally, the clarity of this strategy’s output has helped it gain favor among designers. Both professional communities and organizations have created reference materials that use requirements allocation [39,40]; customized implementations of this strategy include quality function deployment, spiral models, or agile development [see, e.g., 27,36].

However, the constraints’ nature also presents a significant risk: unambiguous, but wrong, bounds on the solution space can result in a poor design. These constraints are created early in the design process when the designer has the least knowledge of the impact of different design aspects [41]. Here, the designer can incorrectly declare a valuable region of the solution space unacceptable, precluding exploration that would deliver better solutions. Thus, a lack of appropriate solution knowledge risks bounding poor(er) regions of the solution space, resulting in poor(er) designs [42].

Relatedly, the definitiveness also prevents the team from making system-wide design tradeoffs. The bounds imposed on the design are strict, and knowledge sharing is limited between parts of the design [25]. The design team cannot forecast how their design choices impact the whole, as there is inherent uncertainty in the dynamics involving all decomposed parts of the design. Nevertheless, they must adhere to the designer’s constraints. Thus, though the team might make ideal design decisions for their delegated part, they lead to poor(er) system-wide outcomes. These and other drawbacks—like specifying conflicting requirements [43]—have prompted scholars to envision a different approach to formulate complex designs.

Per the objective allocation strategy, constraints convey the value of the system’s design. Unlike the aforementioned strategy, where definitiveness is required, the designer now decides what the system needs to accomplish to be broadly successful [44]. Accordingly, this strategy’s constraints are open ended: e.g., “the system should have minimal weight” [32]. Thus, a design’s value—and the value of each constituent part—is determined by how well it meets the specified design objective(s) or function(s) [30,33].

The open-ended nature is deliberate: objective allocation converts the design process into an optimization problem. No region of the solution space is explicitly in or out of bounds. Instead, these constraints encourage the design team to broadly explore all design parameters to find the solution that best meets the stated objective. Here, scholars have leveraged insights and techniques from multiple fields, including expected utility theory [30,42], game theory [45], and multidisciplinary design optimization [46].

However, this strategy also presents challenges that have limited its practical adoption. First, decomposing the objective and the system according to this strategy’s intent is hard. A designer may easily derive an objective for the whole system. However, describing the sub-objectives for the related subsystems in an open-ended manner is challenging. As mentioned earlier, one begins this process by decomposing the system into defined parts, prescribing large swaths of the solution space through specific interfaces. Thus, the designer must make decisions that this strategy tries to avoid. Second, these objective functions and their related constraints are also much harder to communicate [33], which could lead to miscommunication and poorly performing designs. While scholars have proposed techniques to simplify a set of objectives to allocate them across the decomposed parts of the problem [e.g., 28], implementation challenges remain [47,48]. Finally, without the comprehensive view of the system that requirement allocation provides, designers might make rational design choices in their part of the problem that negatively impacts the whole [49].

The success of a design stemming from an innovation contest also rests on correctly formulating its problem [50], but guidance to do so is scarce [15]. Constraints are this paradigm’s “sine qua non” [51]: collectively, they are the seeker’s most important lever to influence this process because they communicate the seeker’s need across various boundaries. Yet, earlier work has not examined the specifics of formulating these constraints. Instead, they recommend selecting the right part of the problem to open up [3], focusing on the characteristics of the solvers [52,53], or simplifying the problem to be widely understood [11]. While insightful, these suggestions are too coarse to guide seekers through the formulation process.

However, seekers cannot simply turn to the established formulation strategies either. Generally, the intent and output of the formulation process are the same for a contest and in traditional design—communicate the (intended) design and delegate its parts. But a seeker might be unable to employ one of these formulation strategies as a designer would: iterating the problem based on new information from solvers—part and parcel of a normal design process [54,55]—can be untenable in a contest.⁶ The rules document is final, with little chance for course correction: the seeker throws the problem over a proverbial fence once it is formulated [15]. The iterative process of sensemaking, formulation, and solving is interrupted—there is little (or no) room for change once the problem has been shared.

Without these iterations, there is an increased risk of “errors of the third kind” [19] compared to traditional design: designing the right solution to the wrong problem. Several issues drive this risk. First, imposing detailed constraints too early could (unfavorably) shape the way others solve and (unduly) constrain potential solutions. Organizations often look to innovation contests to address new, or new-to-them, designs for the chance at novel ideas [56]. Yet, the early stages of designing a new system can be uncertain. As such, the seeker might, correctly, hesitate to provide the detailed information required to accurately communicate the problem [36,57]. This could result in a relatively open-ended contest problem, producing a wide variety of solutions that might not address the seeker’s need.

Second, estimating the capabilities of potential solvers can be challenging. In traditional design, the design team often knows what expertise and design infrastructure resides with their individual or organizational members. This knowledge is crucial to appropriately delegate tasks among team members to achieve systems of the “highest quality at lowest cost” [36]. However, participants in an innovation contest can be a heterogeneous mix of individuals and teams, possessing varying (levels and domains of) expertise and infrastructure. Even domain-specific problems see a broad range of solvers [4]. The inability to precisely know whom to delegate to means that the seeker must guess who will self-select to solve while crafting their constraints—and their specific capabilities might vary widely [cf. 7,51]. As such, if the seeker’s constraints necessitate high expertise and uncommon infrastructure, they will preclude many solvers from participating [11]. Yet, problems that do not require expertise and niche tools might be easily solved internally. Both extremes could invalidate the innovation contest approach.

Lastly, the communication challenges in an innovation contest risk overconstraining the seeker’s problem. Successful communication between design team members in the same organization often requires detailed information to be translated or transformed as it crosses domain boundaries [58,59]. Innovation contests engage individuals across organizational and domain boundaries [50], but crossing both boundaries requires even more details to adequately communicate the problem [21]. With the seeker supplying the additional information, they increase the risk of (inadvertently) bounding the problem too narrowly.

In short, correctly formulating constraints is challenging and a common concern. Even Albert Einstein proclaimed that “the formulation of a problem is often more essential than its solution” [60]. A set of constraints—e.g., a requirements document, objective function, or a contest’s rules—communicate a design’s intent and desired output across boundaries. Engineering design and systems engineering have developed strategies to guide their formulation. In an innovation contest, however, seekers face a unique challenge: balancing the need to communicate detailed information on the design problem with the risk of excluding valuable solutions and solvers, with only one opportunity to get it right. As such, it is unclear if or how these existing strategies apply. Considering the gap in the academic literature, we pursued an inductive study to ensure that our insights are grounded in the actual dynamics of that process. We explain our setting, data, and our analysis methods in Sec. 3.

We addressed our research question inductively. As described earlier, how seekers formulate an innovation contest’s problem—the phenomenon we are studying—is poorly understood. In these cases, understanding the context is crucial to explaining these dynamics [61]. Additionally, the phenomenon we were after would only be found in decision-makers’ recollections of their decisions, project documentation, and interactions between individuals. Thus, we pursued a qualitative approach to gather and analyze the necessary data and extract the important patterns in the context [62]. Below, we describe our setting, the data we gathered, and how we analyzed them.

Our setting is the Centennial Challenges Program (CC): the National Aeronautics and Space Administration (NASA)’s flagship for complex innovation contests. The program finds topical technology problems across the agency—specifically where external design ideas, development activity, and funding will be useful—and launches innovation contests to address them. CC contest submissions range from paper designs to small satellites to be deployed in orbit; their prize purses range from $50K to $5M. Solvers form industry, academic, or nonaffiliated teams to create their submissions; the same team rarely participates in multiple contests. Thus, CC contests vary in topic, level of development, participants, and the (NASA and non-NASA) stakeholders involved in formulating each contest.

We studied two multiyear CC contests for this work: the 3D-printed habitat (3DPH) challenge and the CO₂-to-Glucose (CO₂) challenge. The former challenged solvers to design and demonstrate additive construction technologies for Mars, which could 3D print the needed infrastructure on the planet’s surface. This capability will massively reduce NASA’s launch costs and the risks of radiation exposure to astronauts. This contest launched in 2015 and ended in 2020. The latter challenged solvers to find and demonstrate an efficient pathway for converting carbon dioxide to glucose. Using this process as a basis, a NASA crew could manufacture many valuable materials during their long-duration stay on Mars. This contest launched in 2018 and ended in 2021. Our fieldwork spanned the formulation processes of both contests.

This setting was ideal for addressing our question. First, we captured various formulation processes. The 3DPH and CO₂ challenges had multiple contest stages where solvers could win a prize. While each stage focused on the same topic as its parent contest, they covered different aspects of the technology, asked for markedly different deliverables, or re-evaluated previous constraints. At each stage, a team of experienced subject matter experts (SMEs)—our setting’s seeker—formulated multiple constraints making up its rules document. Here, we observed four formulation processes related to the 3D Printed Habitat (3DPH) challenge and one related to the CO₂ challenge: these five covered a range of topics, saw a spread of prize purses, and were carried out by different SMEs, both inside and outside of NASA.

Second, Subject Matter Experts (SMEs) were deliberately engaging non-NASA outsiders. Generally, NASA SMEs in our sample wanted to engage outside the aerospace industry, noting how they “fight constantly to get outside the known group of people and companies that we deal with.” They recognized CC contests as a path to do so and formulated the constraints for the 3DPH and CO₂ Challenges accordingly.

Lastly, the contests and their outcomes mattered to those who launched them. For years before the 3DPH and CO₂ challenges, NASA ran technology development projects on the contests’ topics. During those challenges, the projects’ staff were heavily invested in formulating the related constraints [63]. Here, SMEs saw this as a prime opportunity to improve the chances of getting good solutions: shaping the constraints would shape the contests’ outcomes. That way, solvers would play “a very complementary role to what we’re doing,” per one of our interviewees.

In our context, formulating meant translating an internal need into a contest’s constraint. Here, SMEs decided on the problem’s scope, scale, and deliverables during this process, impacting both who participated and what solutions SMEs received. This translation happened over many months and meetings across the five formulation processes in our data. Additionally, various stakeholders both inside and outside NASA contributed to each contest’s formulation process. Thus, capturing the relevant data for this analysis required a wide net.

To this end, our data consisted of interviews, project documents, and first-hand observations. First, we collected over 65 h of semi-structured interviews. These included interviews with all NASA and non-NASA formulation team members for both contests and the CC personnel who supported them during this process. While our questions about their context, work, and the contests were wide ranging, we asked them to describe their discussions on the rules in detail—capturing what they decided and why. To understand the effects of their decisions on the solvers, we also interviewed various teams across both contests. However, their responses did not inform the formulation model in Sec. 4.

Second, we collected upward of 3500 pages of NASA documents. These included multiple drafts of all contest rules, accompanying frequently asked questions, and the contests’ programmatic documents. Additionally, we collected the weekly meeting minutes of challenge teams—where the constraints and contest details were decided—and the program documents of related projects. Combined, these formed a contemporaneous picture of the formulation process and its context.

Lastly, we also conducted direct observations during several contest activities. We observed and took notes at contest formulation sessions, planning meetings, and contests themselves. These gave us a first-hand account of the formulation process and the contest’s dynamics.

We analyzed these data across several steps. First, to establish a record of problem formulation within the two contests, we created analytical chronologies [62,64]: detailed, vetted⁷ narratives that captured what happened, when it did, and why. These triangulated the formulation processes across our various sources. We used these as a baseline to inform our subsequent analyses.

Next, we extracted instances of constraint formulation. To do so, we pored over the rules document creation processes that the chronologies captured. Here, we extracted how SMEs created each constraint: how they deliberated and decided what should, or should not, be posed to the crowd. Each instance we identified centered on one design parameter, like “printed material strength” or “footprint of the conversion system.” For example, in a rules document for the 3DPH challenge’s design challenge, SMEs prompted solvers to include specific volumes dedicated to life support systems in their designs. The threshold they set on this design parameter—underlined in Fig. 1 ⁸—was three spaces of 45 ft³ each. We extracted 33 such constraints across the five rules documents.

Then, we performed two rounds of open coding [65]. In the first, we explored how the SMEs formulated. For each constraint, we first returned to our data, asking how the relevant SMEs translated their technical needs into the statement in the rules document. These frontline codes described how they did, or did not, implement their knowledge onto a particular constraint. For example, “SMEs discouraged powder bed printing methods by disallowing related approaches,” or “SMEs added a significant (negative) modifier to the solvers’ score based on the usage of water as part of the feedstock.” We then started grouping the frontline codes. The commonality that emerged was the leeway each constraint allowed. In particular, the groups described how stringent the SMEs were in their implementation: e.g., “SMEs let solvers define the parameter” or “SMEs set the parameter to a specific state.” Through a few rounds of refining the categories, both their labels and their contents, we arrived at a set that did not change as we sequentially compared them to frontline codes: impose, incentivize, and subsume. Since these categories saturated all 33 constraints, we stopped refining and used these three to describe how SMEs formulated. Examples of the approaches and the coding process appear in Table 1.

During the second round of open coding, we asked why SMEs decided to take each approach. The codes that emerged during this round suggested that SMEs based their decisions on the solutions they thought would be successful. They also indicated that the depth of the relevant knowledge played a role: differences between knowing what a solution must do versus the details of what design(s) accomplished that—its function versus its design thresholds—were apparent. As such, we grouped these codes into two categories of solution knowledge: high, where the formulation team knew a solution’s function (or design objective) and design, and low, where they only knew the former.

Not all codes involved the SMEs’ knowledge, however. In the second round, codes like “the resource constraints are too high for challenge participants” also emerged. These suggested that the solvers’ capabilities, specifically the lack of them, also influenced SMEs’ actions. We grouped these under two categories of solver knowledge: high, where the SMEs strongly believed that no solver possessed the resources or knowledge to address a constraint, and low, where the SMEs did not know, or care, how limited solvers were. We provide examples of the solution and solver knowledge categories and part of their coding process in Table 2.

Finally, we used axial coding to connect the categories we created [65]. Specifically, we traced the causal relationship of the formulation process, connecting SMEs' solution knowledge, their solver knowledge, and the resulting approaches. Here, we established that the SMEs’ uncertainties about successful solutions and the solvers’ capabilities influenced how they translated each parameter into the constraint. We provide an overview of our methods in Fig. 2, and we describe the SMEs’ formulation process next.

To create each rules document, SMEs translated their abstract need into a set of specific constraints. Here, SMEs intended to push past existing performance limits to address the agency’s technology needs. At the same time, they recognized the difficulty that this work entailed. SMEs had to choose which design parameters to specify and how to specify them. Per one of our interviewees, ‘the rules team [was] strategically planning for a set of rules to push the technology to its limits while reducing barriers to entry for the competitors.’ Our findings unpacked this process.

At its core, each constraint represented a careful balancing act between two uncertainties. On the one hand, the seeker’s uncertainty about the thresholds of a particular design parameter: Do I know what design will meet the need? On the other hand, the seeker’s uncertainty about the solvers’ ability to meet those needs: Do I know whether solvers can create that design? Our empirical work found three approaches, each representing a different balance between those uncertainties: impose, incentivize, and subsume. Notably, SMEs in our context did not implement one formulation approach to create a rules document: rather, they used all three. In doing so, they selectively chose the parts of the problem where they were willing to accept risk. We summarize these findings in Fig. 3 and describe this process in the following section.⁹

For nearly half of our constraints, SMEs knew what a successful solution needed to do but did not know what design threshold(s) could meet that. In these cases, a solution that lacked these functions would bypass important technical hurdles that SMEs set out to solve, resulting in a poor solution for NASA’s aims. At the same time, SMEs acknowledged that further work was needed to find specific, suitable designs: imposing their current (estimates for) design thresholds would likely result in premature or nonoptimal solutions. Nevertheless, SMEs could articulate their need as maximizing or minimizing: they knew that better solutions demonstrated more, or less, of a particular parameter. Even small steps in a particular direction would be desirable.

To nudge solvers accordingly, SMEs chose the incentivize approach, formulating a constraint with a scoring gradient to match the desired performance.¹⁰ As one SME explained, the nature of CC contests meant that “teams [were] going to go after the maximum points’ to win. As such, SMEs would create a scoring gradient for a design parameter, which described how they would reward solvers” solutions. This was a proxy for how well their designs addressed the intended function: sometimes, a better performance translated directly to more points; other times, rewards involved point multipliers, which impacted how other parameters would be scored. For example, using resources available on the Martian surface instead of launching them from Earth significantly lowers the mission’s cost. Thus, to encourage their use, SMEs awarded multipliers according to the mass fractions of teams’ printing: using more in situ resources meant more points. Regardless of how SMEs implemented the gradient, its use signaled that a wide range of solutions would still be accepted, even those falling short of the highest rewards.

The incentivize approach mirrored the objective allocation strategy, where the seeker shifted the uncertainty of design onto the solvers. For these design parameters, SMEs wanted to avoid declaring design thresholds that might, ultimately, be poor design choices for the system. Yet, they needed to ensure that their functions would be incorporated into the solvers’ designs. Incentivize facilitated this balance: its constraints communicated the importance of a design parameter and its directionality but omitted any related design thresholds. Here, SMEs intended for solvers to explore solutions along a gradient, making choices that (try to) maximize or minimize the design’s related performance. That way, solvers could draw on their nonspace expertise to create designs while designing with the seeker’s functional preferences in mind. The risk of exploring and creating appropriate solutions, thus, shifted to the external solvers.

For the other half of our constraints, SMEs were confident about what designs would meet their needs. SMEs on the formulation teams had decades of experience in related topics, like Mars’ surface conditions and geology, human habitation in space, off-planet manufacturing, and synthetic biology. These projects helped SMEs better understand the system’s features, like the timely deposition of the printer’s feedstock and feedstock material choices. SMEs would derive design thresholds for specific parameters based on their work and expertise. For example, a habitat size of 1000 ft² to support a crew of four—any smaller, and the space would be less comfortable or less productive. In these cases, SMEs believed only solutions that met these thresholds would address their need. Yet SMEs’ concerns were not entirely laid to rest. While they were confident that these would lead to appropriate designs, they were uncertain whether participating teams could comply with certain thresholds in their design. We explore both cases below.

SMEs chose the impose approach to force solvers to comply with their designs. SMEs translated their certainty in the design thresholds into a certainty in the constraints: these would be incorporated despite any barrier to entry they presented.¹¹ Arming solvers with these would allow them to build on the SMEs’ work, as it would force solvers to overcome obstacles that SMEs would have faced. As such, SMEs accepted the resulting increase in the expertise, resources, or both needed to comply with SMEs’ design. Of the three approaches, this approach had the most definitive impact on solving: it forced solvers to incorporate, or exclude, the stated design. These statements used words like “shall,” “must,” or “are required,” and—in the worst case—would disqualify noncompliant solutions.

The impose approach mirrored the requirement allocation strategy, sharing the uncertainty of that particular constraint between the seeker and the solver. Using this approach, SMEs limited solvers’ ability to explore by explicitly defining the (un)acceptable (range of a) design threshold; they intended these to be rigidly adhered to. Since bounding the wrong region(s) of the solution space was also a concern for them, they relied extensively on their expertise to guide these decisions—the resulting constraints represented their current understanding of the best design choices. Yet it was up to solvers to implement SMEs’ design choices, demonstrating how these thresholds do, or do not, spur the right design performance improvements. Thus, the uncertainty of realizing the design was shared between SMEs and solvers: the seeker bore the risk of choosing the design threshold and the solvers bore the risk of making it work within their system.

There were also constraints where SMEs were confident in their design but particularly mindful of the effort and expertise required to create them. Converting carbon dioxide to glucose and printing a Martian habitat would require a development effort that many outside the space industry could not meet or afford, potentially encouraging shirking or discouraging participation. Additionally, finding individuals who had already solved the same problem in a different context—the easiest way to participate successfully—would likely be impossible. SMEs expressed this concern in both challenges: while the thresholds to meet their needs were important, what they asked solvers to do “also [had] to be commensurate with the amount of money we’re offering and the timeframe we’re doing it in.”

To navigate this tension, SMEs chose one of two approaches to create a constraint: best fit (a special case of impose) and subsume. Both approaches drew on their deep knowledge about the relevant parameter to accommodate solvers’ capabilities while progressing the technology past the “infant stage.” They differed in their approach to mitigating the solvers’ uncertainty. We explain both approaches below.

Best fit: This special case of impose forced solvers in the right direction, even though they would not meet the SMEs’ goals. Thinking that their intended design thresholds would create too a high barrier to entry, SMEs imposed easier, more accessible ones to mitigate the risk of getting few (or no) solutions: like reducing the scale of the 3DPH challenge’s printed habitat from 1000 to 100 ft². The approach was, on its face, somewhat counterproductive to SMEs’ needs: while it made the problem more accessible, the incoming solutions would not fully address SMEs’ need. Yet, SMEs felt that partially meeting the need was better than not at all. They understood that tackling the best fit constraint would be an early step in the development process for these technologies—the approach ensured solvers would not avoid important technical aspects. Even if the resulting designs fell short of the need, SMEs “felt like [these designs] would still push the technology, but [at a level] the people could actually do and have a chance at being successful at.”

While the impose approach generally mirrored the requirement allocation strategy, best fit did not. Recall that this special case accommodated solvers by imposing designs tailored to their capabilities, not what the seeker’s expertise dictated. Here, the seeker deliberately asked for the “wrong” design—solutions that incorporated that threshold would not address their need. However, this approach had a meaningful benefit. Compromising the design in these areas made creating the overall solution easier while maintaining some of its essential elements. In these cases, the uncertainty of realizing the design was not (quite) shared between the seeker and solver: solvers did not bear the risk of incorporating the needed threshold in their solution. Instead, best fit pushed solvers to explore a region of the solution space that was acceptably inadequate in the eyes of the seeker. To the seeker, a solution that fell short in predictable ways was better than getting nothing at all.

Subsume: When SMEs were confident about a design but questioned the solvers’ capabilities to comply with them, they would also omit—or subsume—a parameter from the rules documents.¹² During the formulation of the two contests, SMEs discussed various parameters important to off-planet manufacturing and synthetic biology. However, despite their importance, they did not incorporate all of them into the final constraints. In their telling, these were “not viewed as germane to this contest.” SMEs felt that the effort to comply with even partial designs would “overwhelm the whole thing and it would disperse the focus too much.” Some SMEs also believed these parameters were not the solvers’ responsibility but instead were “something that NASA owns at this juncture.” The resulting rules document “dropped,” “eliminated,” or “didn’t ask teams to account for” the parameters in question. By omitting any specification of these parameters, solvers could make design choices more aligned with their capabilities. The slack in this part of the problem allowed solvers to focus their resources on other, harder parts of their solution.

How did the SMEs ensure the solvers’ solutions would (help) fulfill the need without explicit direction? SMEs architected their systems to absorb the range of expected solutions, ensuring they would meet NASA’s need when integrated.¹³ Doing so required extensive knowledge of that design parameter and its dynamics. SMEs drew on their knowledge to estimate the range of designs they could receive and how to convert that range to meet their needs. Sometimes, converting the solution’s output into a more suitable one required an interface between it and the SMEs’ systems. Other times, SMEs would revise the solver’s solution to meet the need. For example, instead of burdening solvers with NASA’s flammability standards in the rules document, they would “start an interchange” with the developer of an (otherwise) promising material to add flame retardants to its mix. In short, SMEs took this approach knowing that (nearly) any submitted solution could meet their needs; solvers did not need to be influenced accordingly.

The subsume approach did not mirror either of the existing formulation strategies. Guided by their expertise, SMEs decided that certain design parameters were their responsibility, not the solvers’. In these cases, SMEs reconfigured their (planned) internal design to accept a range of design thresholds per these parameters—creating an interface that could absorb the variety of solutions that solvers could submit. In contrast to the approaches and strategies mentioned earlier, subsume did not communicate preferences or specifics. Instead, SMEs obscured or deemphasized the parameter in question; neither gradient nor threshold was specified in the rules document. From the solvers’ perspective, their freedom to explore the solution space was total; they shared no risk of searching or implementation. Solvers could make design choices that made sense in light of their capabilities and specific solution (concerning the parameter in question). Thus, the risk of creating an appropriate system incorporating solvers’ solutions fell completely on SMEs.

Our work studied how seekers formulate an innovation contest’s most important design artifact—the rules document. In this article, we inductively described this process. In particular, the seeker formulated each constraint per one of three approaches: incentivize, impose, or subsume. The approaches differed in balancing the need to communicate detailed information on the design problem with the possibility of excluding valuable solutions and solvers. To choose between them, the seeker relied on their knowledge of potential solutions and the solvers’ capabilities, which varied significantly across different design parameters within each problem. Using only one approach to formulate the whole problem—as the engineering design and systems engineering literature recommend—would not adequately represent (or communicate) the seeker’s knowledge and would lower the chances of good solutions. Instead, the seeker employed multiple approaches within each rules document, formulating more granularly than the literature currently describes. With these findings, we unpack how seekers (can) use this open innovation activity to find new knowledge to solve a complex design problem.

We expand on the implications of our findings below. First, they shed light on how seekers employ innovation contests in addition to describing how they formulate them. Our findings also impact the debate between requirement and objective allocation. Finally, we describe these findings’ limits and conclude with future work directions.

While our findings describe how seekers formulate, they also shed light on their desired outcomes more generally. First, rather than formulating to maximize solution variety [e.g., 51], the seeker deliberately narrowed the space of potential solutions to increase their likelihood of solving the problem. The more the seeker knew about potential solutions, the more they wanted to control the variety of solutions they received—in these cases, solvers explored a smaller solution space. SMEs’ reason for narrowing was not (just) because it wasted their effort [14]: we showed that the seeker did not want to waste the solvers’ exploration effort either. Here, they narrowed the high-confidence regions of the solution space and opened up more uncertain parts. Thoughtful usage of these formulation approaches—or other strategies to selectively narrow how solvers explore the solution space [52]—can direct the solvers’ efforts more effectively, resulting in better outcomes.

Second, the seeker did not aim for the broadest possible participation, as some scholars have suggested [e.g., 9,11]. The combination of skills needed to solve the problems they faced simply did not reside in a large swath of the population [3,4]. While the seeker can work to lower entry barriers, some must remain. Complex problems, even when decomposed, cannot be solved by everyone in a crowd. Additionally, many complex designs require design validation—demonstrating that a concept performs as expected is crucial to believe the solution. As such, the resources required to meet the needs are out of the reach of many and may even prompt cost–benefit analyses within entities that can afford it. To get useful solutions, the seeker could identify domains that might provide good solutions and tailor the problem to attract its members. That way, their aim is useful depth instead of maximum reach.

Previously, scholars have framed requirement and objective allocation two as overarching, mutually exclusive strategies to guide the design process [32]; in fact, the objective allocation strategy was created to address requirement allocation’s shortcomings [33]. Since each strategy stems from a different view of the engineering design process [30], scholars have created separate frameworks and tools to implement them [cf. 47,66]. As a result, practitioners are steered toward (or away from) employing one or the other.

However, our findings suggest that one design project can benefit from both requirement and objective allocation. One way to reconcile these two strategies might lie in how our approaches managed the seeker’s uncertainties. In our setting, the seeker’s knowledge of solutions and those who would design them was never uniform: they had high confidence in certain parts of the design but not others, or they had low confidence that solvers could comply with a particular design threshold but not others. Since their knowledge was unevenly distributed within a single problem, they could not employ a strategy that required consistent knowledge throughout if they wanted the challenges to succeed. To reflect this variability, the seeker selectively leveraged the approaches according to that distribution: formulating each constraint separately to address specific design parameters, not consistently following one strategy across the rules document.

The formulation approaches also allocated the risk of producing a feasible design in a granular manner. By choosing these approaches, the seeker shared, shifted, or took on the risk associated with specific design parameters. Incentivize allowed the seeker to set an optimization direction—like the objective allocation strategy, the approach shifted that design risk to the solvers. For example, SMEs in the CO₂ challenge incentivized teams to maximize the mass of glucose their solutions could deliver—letting teams decide how much they could produce competitively. Impose allowed the seeker to set a mandatory threshold for a design parameter—like the requirement allocation strategy, the approach shared that design risk between themselves and the solvers. For example, SMEs in the 3DPH challenge imposed a minimum ratio of indigenous materials on solvers’ printing feedstock, but expected solvers to demonstrate its feasibility by printing a habitat. Finally, subsume avoided setting a design parameter’s direction altogether—the approach absolved solvers of that design risk. For example, SMEs in the 3DPH challenge omitted any language concerning the printer feedstock’s flammability—solvers did not need to consider this. Instead, SMEs decided to tackle this issue in-house after the Challenge if a solver’s material showed promise.

In short, using (all three) formulation approaches in the same design showed how requirement and objective allocation can coexist. The seeker applied the approaches in a granular manner, formulating each constraint according to their uncertainties. As a result, they allocated the risk of a feasible design unevenly—some of it was shared, some shifted to solvers, and some remained with the seeker. This contrasts with the output of either requirement or objective allocation strategies, where the risk is either fully shared or fully shifted.

While we expect the formulation approaches we found to be universal across innovation contests, our model of how seekers chose them might not. First, our formulation model might not apply when the seeker can iterate a design based on (partial) information from the solver(s)—collaborative crowdsourcing activities or user communities are good examples. In these settings, the design artifact that facilitates communication between seeker and solvers is frequently updated as uncertainties are addressed: the solution space is clarified, the constraints are reformulated, and the solving tasks are (re)allocated based on the relative capabilities of those involved. The risks of appropriately formulating the first iteration of their (equivalent of a) rules document are much lower than in an innovation contest, which might lead the seeker to make different choices.

Furthermore, our model might not apply when a contest challenges a simple problem. In a search for new knowledge, the uncertainty of the solvers is proportional to the complexity of the problem: as less expertise and capability are required for a simple problem, more individuals are able to solve it. Indeed, reducing the complexity of a problem is an established strategy to open it up to a broad audience [11]. With solver uncertainty at its lowest (and knowledge of their capabilities at a maximum), these scenarios fall outside of our model’s scope—those did not exist in our data.

Finally, our model might not explain a contest searching for known knowledge either, i.e., gig work. In these cases, both solution and solver uncertainties are at their lowest: the seeker knows exactly what they want and who can deliver those solutions. For example, NASA has sourced small-scale mechanical design tasks from the crowd using the GrabCAD platform, with relatively good results from its members [67]. However, as neither the solution nor solver uncertainties impacted the seeker’s formulation decisions, our model would strain to explain their choices in that case.

Our work described how the seeker creates an innovation contest’s most important design artifact: the rules document and its contents. Relying on inductive methods in an empirical setting, we described a choice between three approaches—impose, incentivize, and subsume—to create one of the rules document’s constraints. Notably, though impose and incentivize mirrored existing formulation strategies, the seeker used (both of) them at a level of granularity not previously reported. Further work will assess whether their synergy can be leveraged outside of the innovation contest setting. Further work will also assess whether subsume is unique to contests or may have broader importance for design in general.

• National Aeronautics and Space Administration, Grant No. NNX13AR06G.

• National Science Foundation, Grant No. CMMI-1535539.

There are no conflicts of interest.

The datasets generated and supporting the findings of this article are obtainable from the corresponding author upon reasonable request.