Making systems

Table of contents

• Part V: Development methodology and life cycles
• Chapter 21: Introduction to life cycle patterns
• 21.1 Introduction
• 21.2 Life cycle and development methodology
• 21.3 Key ideas
• 21.4 Purpose of life cycle patterns
• 21.5 A model for patterns
• 21.6 Documenting life cycle patterns
• 21.7 Work steps and artifacts
• 21.8 Life cycle and teams
• 21.9 Life cycle and planning
• 21.10 Principles for a life cycle pattern
• Chapter 22: Development methodologies
• 22.1 Purpose
• 22.2 Methodologies
• 22.3 Practical considerations
• Chapter 23: Example life cycle patterns
• 23.1 Introduction
• 23.2 Whole project life cycle
• 23.3 System development patterns
• 23.4 Post-development patterns
• 23.5 Detail patterns
• 23.6 Comparisons and lessons learned
• Part VI: Reference life cycle
• Chapter 24: Introduction
• 24.1 Projects with proposals
• 24.2 Project-wide decisions and reviews
• Chapter 25: Project preparation
• Chapter 26: Project support
• Chapter 27: Development
• 27.1 What is development?
• 27.2 The development phase
• 27.3 Purpose development
• 27.4 Concept development
• 27.5 Development methodology
• 27.6 System feature development
• 27.7 Recursion to component development
• 27.8 Feature development variations
• 27.9 Acceptance
• Chapter 28: Operation
• 28.1 System production
• 28.2 System production examples
• 28.3 System deployment
• 28.4 System deployment examples
• 28.5 System operation
• 28.6 System operation examples
• 28.7 System evolution
• 28.8 System evolution examples
• 28.9 System retirement
• 28.10 System retirement examples
• Chapter 29: Project ending
• 29.1 Project cancellation
• Chapter 30: Using the reference life cycle
• 30.1 Meeting life cycle principles
• 30.2 Relation to NASA life cycle
• Bibliography

Part V: Development methodology and life cycles

What life cycles and development methodologies are. It covers what goes into a life cycle pattern and how the patterns relate to other parts of how a project operates. It defines development methodologies and how these relate to life cycles. The last chapter lists several example life cycle patterns, which lead into a comprehensive reference life cycle in Part VI.

Chapter 21: Introduction to life cycle patterns

21.1 Introduction

System building in general follows a common story.

A project to develop a new system begins when someone has an idea that people should make the system. At this initial moment, the system is largely undefined. There is a vague concept in a few minds, but all the details are uncertain.

The project then moves the system from this initial concept through to an operational system, and through the system’s operational life and eventual retirement. During development, the team will need to ensure steps are taken in order to produce a correct, safe system. Designs will be checked. Implementations will be tested. The system as a whole will be verified before being deployed into service. At the same time, the resources spent on building the system must be used efficiently, doing the work that needs to be done and avoiding the work that doesn’t need to be done.

Many projects continue system development beyond the first operational version, with ongoing development or problem fixes. Some projects include the steps to shut down and dispose of the system once it has completed its functions.

The life cycle is how a project organizes the way the team moves through this story. It is a pattern that defines the phases and steps in the work: what will come first, what will done before something else, when checks will happen. It provides checklists to know when some step is ready to be done, and when it should wait for prerequisites. It provides checkpoints and milestones for reviewing the work, so that problems are found and dealt with in a timely way. It provides an overall checklist to ensure that all the work that needs to be done is in fact done.

Section 20.3 introduced the basic ideas for life cycle patterns. These include:

• The life cycle usually includes multiple patterns for different parts of the work.
• Work is organized as a set of phases or steps, with milestones or checkpoints included in some steps. (I use the terms phase and step interchangeably; I generally think of a “phase” being longer than a “step”.)
• Phases or steps can be dependent on each other, with A depending on B meaning that work done in A will build on work done in B.
• A phase can have conditions that should be met before it starts, or that should be true when it is complete.
• Many phases and steps can be worked on in parallel.
• A path from one step to another following dependencies does not necessarily mean that the steps must be performed serially.

Each project will use its own life cycle patterns. The patterns may incorporate a framework that is standard for the industry or the parent organization. Selecting and documenting the patterns is an essential part of starting up a project, and people in the project should review how well the patterns are working for them from time to time and may want to improve the patterns.

In this part, I discuss life cycles in general. In Part VI, I present a reference life cycle pattern.

21.2 Life cycle and development methodology

Life cycle patterns are related to, but separate from, the development methodology that a team chooses to use, such as waterfall, spiral, or agile methodologies. I address these methodologies in the next chapter (Chapter 22).

Speaking broadly, the development methodology determines how the work is organized in time: in a single sequence or iteratively, synchronized tasks or separate tasks, how far ahead to plan. The life cycle patterns reflect some of those methodology decisions and encode how to do different tasks.

Put another way, the life cycle patterns help organize what work the project has to do, and what dependencies there are among different steps in the work. The development methodology organizes how that work is planned and scheduled. As a result, the two go hand-in-hand but are distinct from each other.

21.3 Key ideas

Almost all project life cycle patterns, for both whole systems and for components, follow a similar overall flow. Abstracting from the story in the introduction, there are phases:

1. Working out how the project will operate
2. Identifying purpose
3. Developing a concept
4. Refining concept into specification and design
5. Implementation
6. Verifying the result
7. Operating the system or component
8. Evolving it over time
9. Retiring the system or component at end of life
10. Shutting down the project

For a whole system, this looks like:

Note that this flow starts with the system or component’s purpose. Good engineering always begins with having a clear understanding of what a thing is for. I have watched many engineers rush into designing and building a component without putting time into understanding what the component is going to be used for. By random chance their design has occasionally worked out to match what the component actually needed to do, but only rarely.

Understanding a system’s purpose or a component’s purpose also provides a way to bound the work. If one doesn’t know what a component is for, it is easy to keep working on a design without stopping because there isn’t a clear way to know when the design is good enough to be called done.

There are many points in this flow where one might add checks. At these times one can check on the correctness of the work. These checks improve system quality by building in the opportunity to discover and correct flaws before other work builds on the flawed work. Finding minor problems quickly usually means the cost of correction remains low.

There are also points where a project might have project-wide decisions--go/no go decisions or key decision points. These provide opportunities to check the entire project progress, sometimes occurring in the middle of other work, or at times when irrevocable actions are to be taken, such as funding, launch, or public announcements.

This general pattern applies recursively. One can start by creating a specification and design for the system. The system design will decompose the system into high-level components (Section 6.4). The act of defining a set of components implies identifying a purpose for each one, then specifying and designing each high-level component. The design of a high-level component might in turn decompose into a set of lower-level components, which in turn need a purpose, then specification and design.

The overall flow shows a move from high uncertainty at the beginning to lower uncertainty as the work proceeds. I will address managing using uncertainty in ! Unknown link ref.

Finally, a project’s life cycle patterns will reflect the development methodology that the team has selected. Waterfall, spiral, and agile development all affect the contents of the patterns. I discuss this more in Chapter 22.

The life cycle is provides a general set of patterns for how work should proceed, but it should not define exactly how each work step should be done. That is left to procedures (Section 20.4), which should provide step-by-step instructions for how to do key parts of the life cycle. For example, if a life cycle phase indicates that a design review and approval should occur before the end of a design phase, then there should be a corresponding procedures for design reviews. That procedure should indicate who should be involved in a review, what they should look for, how those people will communicate about the results, who is responsible for approving the design, and how they indicate approval.

The life cycle patterns are the basis for the project’s plan (Section 20.5). The patterns are a set of building blocks that people in the project can use to develop the plan. The plan, in turn, guides tasking: the selection of which tasks (as defined in the plan) people should be working on next.

21.4 Purpose of life cycle patterns

Life cycle patterns address problems that projects have. They can help the team have a predictable and reproducible flow to how work should be done, so that everyone shares the same understanding of how the team works.

There are six ways that life cycle patterns help a project.

1. Quality of work. The team must build a system that addresses the customer’s purpose, and in doing so must meet quality, safety, security, and reliability objectives.
2. Efficiency. The project will be expected to deliver the final system as quickly as possible, at the lowest reasonable cost, while meeting the quality objectives. This means that the team needs to be kept busy doing useful work.
3. Team effectiveness. People on the team need to know how to work together. Building trust depends, in part, on having shared expectations of how each person will do their work.
4. Management support. Project management will need to plan and track the work in order to ensure the team meets deadlines and that they have sufficient resources to do the work.
5. Customer and regulatory support. The customer may have specific milestones they expect the project to meet in support of the customer’s acquisition processes. Regulators often have similar expectations if a system must be certified or licensed for operation.
6. Auditing support. The project’s work may be audited to check that the processes followed meet regulatory requirements, certification requirements, or as part of a legal review.

Gaining these benefits is not a result of using life cycle patterns per se; rather, it comes from using patterns that are designed to provide the benefits. For example, if the customer has an acquisition process that specifies certain milestones, then the top-level life cycle pattern for the project should incorporate those milestones. If the project is likely to have auditing requirements, then the patterns should include tasks to generate and maintain auditing records.

21.5 A model for patterns

Each project will have several life cycle patterns, each covering a different part of the work.

Each pattern is defined by its purpose, the circumstances in which it applies, the phases or steps involved, and the dependencies among the steps. It should also include rationale that explains why the pattern is structured the way it is. In a previous chapter I used the example of a simple pattern for building one component:

This pattern applies to building one low-level component where the purpose of the component is already known, and the component is straightforward to design and build in house. Similar but slightly different patterns might apply when the component has to be prototyped before deciding on a design, or when the component is being acquired from a supplier outside the project. This pattern would be used as one part of a larger pattern for building a higher-level component that includes this one.

Each phase of a pattern defines a way to move part of the work forward. It should have a defined purpose that defines what work should be achieved in that phase.

The details of the phase are defined by:

• Preconditions: what conditions should be true when a phase is ready to start. A list of conditions for starting the phase, beyond the artifacts that should be ready.
• Input artifacts that should be available at the beginning of the phase, and their maturity or completeness level.
• Actions that are to be taken to begin the phase.
• What work should be done in the phase, and the artifacts developed in the phase.
• Milestones to be met during the phase.
• Actions taken to end the phase.
• Output artifacts that should be available at the end of the phase, and how complete each one should be.
• Postconditions that define what should be true when the phase has completed.

Each action should also indicate who is responsible for performing that work. The responsibility will usually be defined as a role, not a specific individual. For example, a component design phase might involve three actions: design the component, review the design, and approve the design. The design action would be the responsibility of the component developer; the review action would be the responsibility of the developers responsible for components that interact with the one being designed, and the approval would be the responsibility of a systems engineer overseeing some higher-level component of which this one is part.

The rationale for this example design phase might say:

• The purpose is to work out a design for the component that meets its specifications, including its relationships with other components in the system structure and its safety, security, reliability, and performance objectives.
• A separate design step between specification and implementation gives time to think specifically about this component, and to document its design for future developers who may need to revise the design.
• The review actions provide an independent check that should improve the likelihood of this component working correctly as part of the system by looking at the design from a point of view that has not been intimately involved in working out the design.

The actions defined for the phase should reference the procedures for doing those actions, when those procedures are defined. For the example design review action, the procedure might be:

• The component developer notifies the review group that their review is needed;
• Each reviewer in the review group acknowledges that they will provide the review;
• Each reviewer checks the design against the relationship that component has with their own components, and against the specification for the component being reviewed;
• Reviewers give feedback to the developer, indicating whether they are satisfied or not;
• When all reviewers are satisfied, they inform the developer.
• If one of the reviewers detects a more serious problem with the design that cannot be resolved by feedback to the developer, the reviewer should use another procedure to raise the issue up to a higher level.

The procedure might also name the tools to be used (an artifact repository for the design, a review workflow tool for the reviews).

21.6 Documenting life cycle patterns

A team needs clear documentation of the phases if they are to execute them properly. A team can’t be expected to guess at what they need to be doing, or how their work will be reviewed; it needs to be spelled out.

This documentation is assembled during the project preparation phase. The details are usually not completely worked out before any other work is begun; rather, “project preparation” more often proceeds in small increments, working out the rules shortly before the associated work begins.

Each life cycle pattern should have a purpose, and the steps or phases in the pattern should be checked that they can achieve that purpose (and that there is no extraneous work in the pattern).

A pattern should also have an explanation of when it applies and when it does not. For example, there may be multiple patterns for designing a component: one for a simple component that is built in house; one for a component that is outsourced to a supplier; one for a high-level component that is made up of several lower-level components; one for a component that requires investigation or prototyping before deciding on a conceptual approach to its design. All these patterns likely have a lot in common, but procuring an outsourced component will have contracting steps that an in-house component will not.

Someone using the documentation should be able to tell accurately whether they are using the correct version of the patterns. The life cycle patterns will be revised from time to time—as the team grows and as people find ways to improve how they work together. This means that the material that a user sees should indicate not just a revision number but have a clear indication of whether the version they are looking at is not longer current.

The form of the documentation is not as important as the content. It can be a written document. It can be made available electronically, with structured access and search capabilities (such as in a Wiki). Some companies offer tools that help define and document development processes or life cycle patterns, including definitions of phases. What matters is that each person who needs to use the documentation can do so conveniently and accurately.

21.7 Work steps and artifacts

Each phase or step has a number of artifacts that the team must develop. At the end of a phase, some of those artifacts need to be complete (allowing for future evolution), and others need to have reached some defined level of maturity. The work in a phase consists of the tasks that develop those artifacts.

I discussed artifacts in Chapter 17. The artifacts are the products of building the system, including the system being delivered as well as documentation of its design and rationale, records of actions taken during development, and information about how the project operates.

These artifacts are the inputs and outputs of the work specified in life cycle patterns (and the associated procedures). Using the component design step example, the work uses:

• The purpose and rough concept developed for the component;
• The specifications developed for the component;
• Documentation of the relations between this component and others, both functional and non-functional; and
• The purpose of the system and of higher-level components of which this one is part.

The design step produces:

• A design document for the component;
• Analyses of the design showing whether it will meet its specifications or not;
• Records of the rationale for why the design was chosen; and
• Records of review and approval of the design.

In general, every artifact involved in building the system should be a product of some work phase or step, and every input or output of work steps should be included in the set of artifacts the team will develop. Ideally, the life cycle patterns will be checked for consistency with the list of artifacts the project uses.

Artifacts are developed at different times during the course of a project. A few artifacts should be worked out as the project is started—especially those recording the initial understanding of the system’s purpose and initial documentation of how the project will operate. These will be refined over time. Other artifacts are developed during the course of development, and the life cycle patterns indicate which ones are to be worked out before others. The artifacts will be in flux during development: the team learns about the system as it designs and develops it; the customer or mission needs often change over time; flaws get discovered in designs or implementations.

Many of the project’s artifacts support how people work together, and the life cycle patterns should reflect these communication needs. For example, one person may work out the protocol that two components need to use to communicate with each other. Two other people may design and implement the two components. The interface specification that the first person develops serves to communicate the details of the interface among all three people. The patterns should record that the design and implementation work steps depend on the work to develop the interface specification. Later, if one of the component developers identifies a flaw in the interface, the people involved can work through how to revise the interface—and the revised specification artifact informs each person how to update their work to match the change. The pattern helps to show how information about a change to the interface specification triggers rework on dependent artifacts.

A good life cycle pattern has procedures to manage the change in artifacts, and how those changes affect other artifacts downstream from them. There are two separate problems these procedures must address:

1. Managing how changes are coordinated across multiple artifacts and through the team while a part of the system is in development; and
2. Ensuring that when a part of the system is complete, all the artifacts are consistent with each other.

Different life cycle patterns approach this in different ways, which we will discuss in later chapters on different patterns. The most common approach is to maintain different versions of an artifact, with at most one version being designated as a baseline or approved version, and other versions designated as works in progress. Many configuration management tools have a way to designate a baseline version, and many software repository tools provide branching and approval mechanisms to track a stable version.

21.8 Life cycle and teams

What is the team size and background? How is it expected to change over time? A small team can often be a little less formal than a large team, because the small team (meaning no more than 5-10 people) can keep everyone informed through less formal communication. A large team is not able to rely on informal communication, so more explicit processes and communication mechanisms are important. Many teams start small when the project is first conceived, but grow large over time. A team that will grow will need to communicate more formally from the beginning than they otherwise might so that as they add people to the team, the larger team works smoothly.

Conversely, if the life cycle patterns indicate that some action will be performed by some person, does the team actually have the staff to do that work? When a project says that some work is to be done and then does not staff that function sufficiently, it sends a message to the team that they should not take the process as written seriously. This undermines the team’s trust. If the function is actually needed, either the team will find an ad hoc workaround or the function will not get done adequately. Either way, there will be a disconnect between what is written down and what actually happens.

21.9 Life cycle and planning

The life cycle patterns are just patterns that provide a guide to work that goes in the project’s plan. The plan is the actual definition of the tasks to be done. When the plan needs to be updated, the patterns provide a template for the work that goes into the plan.

Assembling the plan, however, takes into account many inputs, of which the pattern is only one. Planning involves deciding on the priority and deadlines for work, which is based on project deadlines, risk or uncertainty, and the project’s development methodology.

Chapter 61 discusses in detail how the plan is developed and maintained, including how the life cycle patterns get incorporated.

Consider the following example of how a pattern gets incorporated into the plan. This example shows how the pattern is only a template, and there are many decisions that will depend on other information.

This pattern defines what should happen when a customer requests a change. The basic pattern is that first someone on the team should evaluate the request; this may involve working with the customer to clarify the request, and with other engineers to estimate the scope and cost of the work. The project can then make a decision whether to accept the change or not. If the decision is to make the change, work to build, release, and deploy the update will follow. If not, there is another pattern for how to communicate with the customer that the change will not be made.

The activity starts when the project receives a change request. Based on this, the plan can be updated to include three tasks right away: the evaluation, review, and decision tasks.

At the same time, the planner must make decisions: who should each task be assigned to? What priority should the flow of tasks have? The pattern can indicate the roles involved in the tasks, such as there being a small team responsible for evaluating change requests and a customer representative from the marketing team, but it doesn’t determine which specific people. That’s for the planning and tasking efforts to determine. Similarly, the pattern does not specify how the work should be prioritized relative to other work the same people are doing. The planner incorporates information about how urgent the customer’s request might be and the importance of the customer into the decision. The project might have decided, for example, that there should be a queue of outstanding change requests and they should be evaluated in their order in the queue.

Determining who should be involved in a review of the evaluation might depend on the results of the evaluation. The pattern might indicate that the evaluation should be reviewed by engineers responsible for each high-level component that will be affected by the change. This means that the decision about who specifically will be tasked with the review can’t be made until the evaluation has worked out the scope of the change.

The decision to proceed with making an update will depend in part on whether the team has the time and resources to make the update. The team will need to determine whether adding the work to the plan will cause a problem with meeting deadlines that have been established already, or if it will overload a team that is already busy. This determination will involve analysis of the current plan—something that the life cycle pattern can help with only to the extent that the patterns can help with generating estimates of the work that would be involved.

When the project takes the decision to go ahead with developing an update for the request, the pattern shows that work steps follow to develop a change and then release and deploy the update. When the decision gets made, this will trigger the planning activity to add development and release work into the plan. These are high-level work steps with little detail. The planner will find patterns for these steps and populate those patterns into the plan.

Decisions about the work involved in development will depend on the development methodology that the team has selected to follow. If the update will involved extensive changes and the team is following a spiral-style methodology [Spiral], the development plan might consist of two or three development rounds. Each round would design and implement part of the changes, with a milestone at the end of each round showing how the partial changes have been integrated into the system.

Decisions about the release and deployment work will also incorporate policy decisions about how the team works. Will each change request result in a separate update release? Or will updates be bundled together into releases that combine several updates, perhaps on a schedule defined in advance?

21.10 Principles for a life cycle pattern

In this section I list some principles to consider when designing a workflow pattern.

The act of designing—or refining—a life cycle pattern is an opportunity to think deliberatively about how the team should get its work done. Life cycle patterns are the templates for the project’s plan, and so they should be designed to achieve the work that is needed to move the project forward well.

Designing the patterns ahead of time means having time to define good work patterns. The pattern does not have to be worked out under pressure, as a reaction to something unexpected happening in the project. It can be discussed among multiple team members to get different perspectives and to ensure everyone’s needs are met. Working in advance gives time to check that the steps in the pattern are consistent with each other. It means that there is time to think about what exceptional situations might happen and define what to do in those cases.

Note that if an organization already has an approach to life cycle patterns, whether documented or not, one should aim for continuity with that approach. Anyone already in the organization will know that approach to organizing work; making a major change would mean losing the advantage of established team habits. On the other hand, if the current approach is not working well, then a new project is an opportunity to improve.

The life cycle patterns encode principles and methodology that encourages good work. Principles to consider include:

1. Know the purpose for something before developing it.
2. Build in time for and incentivize deliberative thinking.
3. Assign decision-making authority to an appropriate level based on the nature of the decision.
4. Build in ways to check work, and design them so they are a team norm and not prone to triggering defensive reactions.
5. Build for the longer term.
6. Build in project-wide decision points.
7. Think about exceptions that might happen, how to handle them, and when to change course.
8. Define the work so that everyone on the team can agree when a step has been completed.
9. Give a clear definition for each step of the quality considerations by which the work can be judged.
10. Make the pattern as light-weight as possible without compromising quality.

Chapter 22: Development methodologies

22.1 Purpose

A project should choose life cycle patterns that fit with its development methodology.

A development methodology is the overall style of how a project decides to organize the steps in developing the system. This includes decisions like whether to develop the system in increments of functionality, whether to design everything before building, whether to synchronize everyone’s efforts to a common cycle, and so on. These decisions are reflected in obvious ways in the life cycle patterns a project uses.

There are many methodologies named in the literature: waterfall, spiral, agile, and so on. Different sources interpret each of these differently, and they are rarely compared on a common basis. Some of these, like waterfall methodology, have evolved over time and do not have a single clear source or definition. Others, such as agile development, have a defining document (manifesto) to reference.

All of the methodologies I know of have come to be treated as dogma, and are more often caricatured than treated thoughtfully. This is unfortunate because each of the methodologies has something useful to offer, while all of them are harmful to project effectiveness if taken as dogma or used without thoughtful understanding.

These methodologies can be organized and compared based on a few characteristics.

Rather than try to argue for or against any specific methodology—which is difficult, since most methodologies are hard to pin down among many published variants—I focus on these characteristics, and argue for choosing a methodology that has the characteristics that a project needs.

Size of design-build cycle. Methodologies like waterfall use “big design up front”, where the entire system is specified and designed before implementation begins. Other methodologies break up development into many specify-design-implement cycles.

Size of design-build cycle.

The argument for doing as much design up front as possible is that errors are easier and cheaper to catch and correct before implementation than after. The arguments against are that in some complex systems the design work is exploratory and requires implementing part of the system to learn enough to know how to design—or not design—critical system parts.

Many iterative methodologies claim to be better at supporting adaptation as system purposes change.

Coupled or decoupled design-build. Some iterative methodologies plan to complete adding a feature to the system in one iteration, by executing an entire specify-design-build-integrate cycle for that feature. Other methodologies break up that cycle into multiple steps, and allow those steps to spread across multiple iterations.

Coupled or decoupled design-build.

Advance planning. Some methodologies emphasize planning out work activities as far as possible into the future, while others focus on planning as little as possible in order to adapt as needs change.

Planning to different horizons.

The argument for planning as far as possible into the future is that it gives the team stability: they have a reasonable expectation of what they should be working on now and have a sense of how that work will flow into other tasks soon after.

The argument for planning to shorter horizons is that someone will come along and change priorities or system purpose, and so the work will need to be changed to adapt. Planning too far ahead is wasted effort, it is argued, and gives teams a false sense of stability.

Synchronization across project. Some methodologies that break up development into multiple iterations align all the work being done at one time so that the iterations begin and end together. Other iterative methodologies allow some work iterations to proceed on different timelines from other work.

Synchronized versus unsynchronized tasks.

Synchronizing work iterations across the whole project can provide common points to check that work is proceeding as it should and to share information about progress. However, it can also break up tasks that run far longer than others and result in a perception that the synchronization is wasteful management overhead rather than something useful.

Shared short-term purpose across project. Iterative methodologies can focus the entire team on one set of features across all the work going on at one time, or they can allow different streams of work to have different focuses in the short term.

The argument for this practice is that the more people share a common goal, the more they will be motivated to work together to meet that goal and to defer work that does not address that common goal. The argument for having multiple work streams with different focuses is that too often a project will involve work from different specialties and on different timelines: mechanically assembling an airframe and building a flight control algorithm have little in common.

Shared purpose.

22.2 Methodologies

I present three of the most commonly discussed development methodologies in order to illustrate how they can be characterized. Each of these methodologies has many variants, and all are the subjects of debates comparing tiny details of each variant. The purpose of this section is to illustrate how they can be analyzed, not to capture all nuances of every methodology in use.

22.3 Practical considerations

Most projects actually choose to use a hybrid among the different methodologies. They may start from one of the generally available methodology definitions, but they adapt that template based on the needs of their project and their own experience.

I have been part of several projects that had hard decision points, reflected in project milestones. At these points, the project was expected to provide information that would lead to the project continuing or being canceled: the decision to award the team a contract to build the system, or decisions to continue funding. These decision points impose a degree of waterfall-like structure on the work. For example, one project had to present a proposal to a government agency in order to get a contract to perform detail design and prototype implementation. That proposal involved developing the concept for the system, showing how it met the customer’s objectives, and showing that there was a likely feasible design.

However, once the contract had been awarded, the team used a spiral development methodology to build a sequence of increasingly capable versions of the system, and used those to demonstrate the completion of defined features at the end of each spiral. Within each spiral, the software team used an agile-like approach based on two-week sprints.

Every successful project I have been part of has done some degree of planning and high-level design well ahead of detail design and implementation. When the project used spiral or iterative development, the general flow from one spiral to another provided guidance to keep the work on track to reach the defined end point. When the project used agile methods to manage tasks in the short term, the longer-range plan kept the tasking decisions in each sprint from going off track by providing a basis for prioritizing the work.

No matter what methodology the projects have followed, they have all had some kind of regular cycle for checking in so that project leadership could find out when there were problems, and so that team members could maintain awareness of progress across the whole project. In some cases this looked a lot like agile practice, with short daily stand up meetings and regular, more in-depth discussions. In other cases the nature and schedule for checking in depended on the part of the system people were working on—from continuous interaction in some parts of software development to weekly updates from people doing safety analysis.

All the projects I have worked on have also had tasks varied widely in duration. Many software-related tasks were short, in the range of hours to a few days, while testing or hardware implementation often required weeks to complete. Most of the projects did not try to make all these tasks fit into a synchronized schedule across the project.

In practice, therefore, the projects I have seen that have been successful have applied common sense to the choices they make about how they chose the design methodology for their specific project.

Chapter 23: Example life cycle patterns

23.1 Introduction

In this chapter I survey some of the many different life cycle patterns in use.

The patterns have different scopes. Some cover the whole life of a system, from conception through retirement. Some are concerned only with developing a system. Others focus on more narrow parts of the work.

I group the patterns in this chapter into four sets, based on scope. The first group covers the whole life of a project, without much detail in the individual steps. The second dives into the development process. The third addresses post-development processes—for releasing and deploying a system; these patterns overlap with development processes. The fourth and final group is for patterns with a narrow focus on some specific detail of building a system.

Patterns with different scopes can potentially be combined. Most patterns that cover a system’s whole life, for example, define a “development phase” but do not detail what that is. One of the patterns for developing a system can be used for the details.

Each of the examples will include a comparison against the following baseline pattern for the whole life of a project.

The baseline phases are the same as in Section 21.3:

• Project preparation. Work out and document the processes, rules, standards, and life cycle patterns that the team will use.
• Concept development. Determine what the system needs to be or do in order to be useful to its users and to those who will operate it.
• System development. Design, build, and verify the system.
• Operational acceptance. Review and acceptance by the customer, indicating that the system is ready for operation.
• System production. Build and deploy the system, using the artifacts from system development.
• System operation. Support the system in operation, including fixing flaws and supporting problem resolution.
• System evolution. Add capabilities to the system.
• System retirement. Take the system out of operation and ensure that all artifacts are archived, destroyed, or recycled. Shut down the project.

23.2 Whole project life cycle

These patterns organize the overall flow of a project, from its inception through system retirement and project end. I have selected two examples: the NASA project life cycle, which is used in all NASA projects big and small, and the Rational Unified Process, which arose from a more theoretical understanding of how projects should work.

23.2.1 NASA project life cycle

The NASA life cycle has been refined through usage over several decades. It is defined in a set of NASA Procedural Requirement (NPR) documents. The NASA Space Flight Program and Project Management Requirements document [NPR7120] defines the phases of a NASA project.

The NASA life cycle model is designed to support missions—prototypically, a space flight mission that starts from a concept, builds a spacecraft, and flies the mission.

NASA space flight missions involve several irreversible decisions, and this is reflected in how the phases and decisions are organized. Obtaining Congressional funding for a major mission can take months or years. During development, constructing the physical spacecraft, signing contracts to acquire parts, and allocating time on a launch provider’s schedule are all expensive and time-consuming to reverse. Launching a spacecraft, placing it in a disposal orbit, and deactivating it are all irreversible. These decision points are reflected in where there are divisions between phases, and when there are designated decision points in the life cycle.

There are several life cycle patterns for NASA projects, depending on the specific kind of program or project. I focus on the most general project life cycle [NPR7120, Fig. 2-5, p. 20], which is reproduced below:

The pattern includes seven phases. There is a Key Decision Point (KDP) between phases. Each decision point builds on reviews conducted during the preceding phase, and the project must get approval at each decision point to continue on to the next phase.

The key products for each phase are defined in Chapter 2 of the NPR and in Appendix I [NPR7120, Table I-4, p. 129].

Phase F (Closeout). In this phase, any flight hardware is disposed of (for example, placed in a graveyard orbit or commanded to enter the atmosphere in order to destroy the spacecraft). Data deliverables are recorded and archived; final reviews of the project provide retrospectives and lessons learned.

This pattern of phases grew out of complex space flight missions, where expensive and intricate hardware systems had to be built. These missions often required extensive new technology development. The projects involved building hardware systems that required extensive testing. The NASA procedures for such missions are therefore risk-averse, as is appropriate.

I have observed that many smaller, simpler space flight projects have not followed this sequence of phases as strictly as higher-complexity missions do. Many cubesat missions, where the hardware is relatively simple and more of the system complexity resides either in operations or in software, have blurred the distinctions between phases A through C. In these projects, software development has often begun before the Preliminary Design Review (PDR) in Phase B.

At the same time, I have observed some of these smaller space flight projects failing to develop the initial system concept and requirements adequately before committing to hardware and software designs. This has led to projects that failed to meet the mission needs—in one case, leading to project cancellation.

The phases in the NASA life cycle compares with the baseline model presented earlier as follows.

The NASA life cycle splits the system development activities across four phases. The NASA approach does this because it needs careful control of the design process, in particular so that agency management can make decisions whether to continue with a project or not at reasonable intervals. The NASA approach also places reviews throughout the design and fabrication in order to manage the risk that the system’s components will not integrate properly. Many NASA missions involve spacecraft or aircraft that can only be built once because of the size, complexity, and expense of the final product; this makes it hard to perform early integration testing on parts of the system and places more emphasis on design reviews to catch potential integration problems.

The NASA pattern is notable for some initial work on a mission concept starting before the project is officially signed off and started. There are two reasons for this. First, because all NASA missions have common processes, there is less unique work to do for each individual project. Second, NASA is continuously developing concepts for potential missions, and this exploratory work is generally done by teams that have an ongoing charter to develop mission concepts. For example, the concept for one mission I worked on was developed by the center’s Mission Design Center, which performed the initial studies until the concept was ready for an application for funding.

23.2.2 Unified Process

The Unified Process (UP) was a family of software development processes developed originally by Rational Software, and continued by IBM after they acquired Rational. Several variants followed in later years, each adapting the basic framework for more specific projects.

The UP was an attempt to create a framework for formally defining processes. It defined building blocks used to create a process definition: roles, work products, tasks, disciplines (categories of related tasks), and phases.

The framework led to the creation of tools to help people develop the processes. IBM Rational released Rational Method Composer, which was later renamed IBM Engineering Lifecycle Optimization – Method Composer [IBM23]. A similar tool was included in the Eclipse Foundation’s process framework, which appears to have been discontinued [EPF]. These tools aimed to help people develop processes and then publish the process documentation in a way that would let people on a team explore the processes.

While the UP and its tools gained a lot of attention, their actual use appears to have been limited. I explored the composer tool in 2014, and found that it remarkably hard to use. It came with a complex set of templates, which were too detailed for project that I was working on. Another author wrote that “RUP became unwieldy and hard to understand and apply successfully due to the large amount of disparate content”, and that it “was often inappropriately instantiated as a waterfall” [Ambler23]. Certainly I found that the presentation and tools encouraged weighty, complex process definitions and that they led the process designer into waterfall development methodology.

The UP defined four phases: inception, elaboration, construction, and transition.

1. Inception. The inception phase concerns defining “what to build”, including identifying key system functionality. It produces the system objectives and a general technical approach for the system.
2. Elaboration. This phase is for defining the general system structure or architecture and the requirements for the system. The results of this phase should allow the customer to validate that the system is likely to meet their objectives. This phase may be short, if the system is well defined and or is an evolution of an existing system. If the system is complex or requires new technology, the elaboration phase may take a longer time.
3. Construction. This involves developing detailed component specifications, then building and testing (verifying) the components. This includes integrating the components together into the whole system and verifying the result. The result is a completed system that is ready to transition to operation. RUP focuses on constructing the system in short iterations.
4. Transition. This phase involves beta testing the system for final validation that the customer(s) agree that the system does what is needed, and deploying or releasing the final software product.

The UP does not directly address supporting production, system operation, or evolution; however, the expectation is that, for software products, there will be a series of regular releases (1.0, 1.1, 1.2, 2.0, …) that provide bug fixes and new features. Each release can follow the same sequence of phases while building on the artifacts developed for the previous release.

The four phases in UP compare with the simple model presented earlier as follows:

The Unified Process provides lessons for defining life cycle patterns: keep the patterns simple, make them accessible to the people who will use them, and put the emphasis on what they are for, not on tools and forms. The basic ideas in UP are good—carefully defining a life cycle, and building tools to help with the definition. I believe that these good ideas got lost because the effort became too focused on elaborate tools and model, losing focus on the purpose of life cycle patterns: to guide the team that actually does the work.

23.3 System development patterns

Some patterns focus only on the core work of developing a system. These patterns generally begin after the project has been started and the system’s purpose and initial concept are worked out. The patterns go up to the point when a system is evaluated for release and deployment. In between, the team has to work out the system’s design, build it, and verify that the implementation does what it is supposed to.

These examples all share the common basic sequence of specifying, designing, implementing, and verifying the system or its parts. Some of the examples include similar sequences of activity to evolve the system after release.

23.3.1 Systems V model

This pattern is used all over in systems engineering work. It is organized around a diagram in the shape of a large V. It is used in many texts on systems engineering; it has also been used to organize standards, such as the ISO 26262 functional safety standard [ISO26262, Part 1, Figure 1].

In general, the left arm of the V is about defining what should be built. The right arm is about integrating and verifying the pieces of the system. Implementation happens in between the two. One follows a path from the upper left, down the left arm, and back up the right side to a completed system.

There is no one V model. There are many variants of the diagram, depending on the message that the author is trying to convey. Here are two variants that one often encounters.

The first variant focuses on the sequence of work for the system as a whole:

The second variant focuses on the hierarchical decomposition of the system into finer and finer components:

The key idea is that specifications, of the system or of a component, are matched by verification steps after that thing has been implemented.

In general this model conflates three ideas that should be kept separate.

1. Development follows a general flow of specification, then design, then implementation, then verification.
2. System development proceeds from the top down: start with the whole system, and recursively break that into components until one reaches something that can be implemented on its own.
3. Development follows a linear sequence from specification and design, through implementation of components, followed by bottom up integration of the components into a system (with verification along the way).

The first two ideas are reasonable. Having a purpose for something before designing and building it is a good idea. There are exceptions, such as when prototyping is needed in order to understand how to tackle design, but even that exception is merely an extension to the general flow. The second idea, of working top down, is necessary because at the beginning of a project one only knows what the system as a whole is supposed to do; working out the details comes next. Again there are exceptions, such as when it becomes clear early on that some components that are available off the shelf are likely useful—but again, that can be treated as an extension of the top down approach.

The third idea works poorly in practice. It is, in fact, an encoding of the waterfall development methodology into the life cycle pattern, and so the V model inherits all the problems that the waterfall methodology has.

In particular, the linear sequence orders work so that the most expensive development risk is pushed as late as possible, when it is the most expensive to find and fix problems. By integrating components bottom up, minor integration problems are discovered first, shortly after the low-level components have been implemented when it is cheapest to fix problems in those low level components. Higher-level integration problems are left until later, when complex assemblies of low-level components have been integrated together. These integration problems tend to be harder to find because the assemblies of components have complex behavior, and more expensive to fix because small changes in some of the components trigger other changes within those assemblies already integrated.

Development methodologies other than waterfall address these issues better, as I discussed in Chapter 22.

23.3.2 Systems or software development life cycle (SDLC)

There are several life cycle definitions for system development, primarily of software systems, that go by the SDLC name. They generally have similar content, with variations that do not change the overall approach.

I have not found definitive sources for any SDLC variants. It appears to be referenced as community lore in many web pages and articles.

The core of the SDLC consists of between six and ten phases, depending on the source, that give a sequence for how work should proceed in a project. The phases are:

• Initiation(*): Identify a need or opportunity for a system.
• Concept development(*): Work out the system’s purpose and scope, along with a rough concept of how it might work. Do feasibility and cost-benefit analyses. Some variants merge this phase with the next two.
• Plan: Develop how the project will operate, including management plans, procedures, and resource estimates.
• Requirements(*): Develop requirements (specifications) for the system based on customer needs developed in earlier phases.
• Design: Develop a design for the system that will match requirements identified in the Plan phase. Includes making build-versus-buy decisions. Some SDLC variants separate this into Preliminary Design and Detailed Design phases.
• Implement: Write software that follows the designs from the previous phase, or acquire it from outside sources. Some variants separate this into Implementation and Unit Test as one phase and Integration as a second phase.
• Test: Perform testing on the software to verify that it meets the system objectives and requirements. Some variants mention review and analysis of the implementation artifacts as part of this phase.
• Deploy: Release or manufacture the system artifacts and put them into operation.
• Maintain: Fix problems identified in operation, and develop new features to extend the system.
• Dispose(*): Take the system out of service, dispose of any physical assets, and potentially preserve any data generated by or stored in the system.

Phases marked (*) are not included in all sources.

Most discussions of SDLC stress that the pattern is meant to help organize a project’s work, not to dictate the sequence of activities. Some sources then discuss how the SDLC relates to development methodologies. A project using the waterfall methodology would perform the phases in sequence. Iterative and spiral development would lead to a project repeating parts the SDLC sequence multiple times, once for each increment of functionality that the project adds to a growing system. A project using an agile methodology would perform tasks at multiple points in the SDLC sequence in any given iteration, as long as for any one part or function of the system the work follows that sequence. I discussed how life cycles fit with development methodologies in Chapter 22.

23.4 Post-development patterns

23.4.1 EVT/DVT/PVT

Many electronics development organizations use a set of development and testing phases:

• Engineering Validation and Testing (EVT)
• Design Validation and Testing (DVT)
• Production Validation and Testing (PVT)

This set of phases is intended for developing an electronic hardware component, such as an electronics board. Developing this kind of hardware differs from developing a software component: while software source code can be compiled and tested immediately, a board design must be built into a physical instance before much of its testing can happen. Simulating the board can be done earlier, of course, but much testing is only done on the physical instance. This is especially true for integrating multiple boards together.

This pattern also addresses not just the design and testing of the component itself, but also the ability to manufacture it—especially when the component is to be manufactured in large numbers. The NASA, V, and SDLC patterns do not address manufacturing specifically; this pattern can be combined with those if a project involves manufacturing.

23.5 Detail patterns

The last two patterns have to do with managing changes to the system: when errors are found, and when customer needs change.

Both these patterns apply to specific, short parts of a project. They apply as needed—when a error report or a change request arrive. Both also potentially involve repeating parts of the overall development life cycle pattern. Both may be used many times in the course of a project.

23.5.1 Defect or error management

This life cycle applies when someone reports a defect or error in the system. It includes fixing the problem and learning from it.

Common practice is to use an issue or defect tracking tool to keep track of these reports and the status of fixing them. Many of those tools have an internal workflow, and parts of this life cycle pattern end up embedded in that internal workflow.

There are two different times when people handle error reports: when errors are found during testing, before an implementation is considered as being verified, and later, when a verified design or implementation must be re-opened. In the first case, the people doing verification are expected to be working closely with the people implementing that part of the system; the pattern for that activity amounts to reporting an error, fixing it, and verifying the fix.

The general pattern for addressing later errors is:

1. Reporting. Someone finds an error and reports it into the tracking tool.
2. Triage and ranking. Determine what to do about the report. Someone investigates the report to determine whether it is understandable and actionable. This may involve communicating with whoever reported the problem to get more information, either about their expectation or about what they found. The result is either accepting or rejecting the report. If accepted, the report is given a priority level (typically one of four or five levels) and a likely part of the system affected. If rejected, the result is an explanation of why the report will not be acted on.
3. Assignment. Who will be responsible for resolving this error? Most projects have a standard procedure: whoever is responsible for the component identified as the likely source, or people can pick up reports when they have time, or a manager can make assignments.
4. Analysis. What is the actual problem that caused the report, and how can a fix be verified? This investigation might involve working to reproduce the problem. The analysis may find that the source of the erroneous behavior is a defect in a different part of the system than originally determined during triage, which may lead to assigning the responsibility to someone else. The analysis may show that the problem is broadly systemic or that it arises from multiple defects in multiple parts, in which case several people will be involved with fixing the problems and overall responsibility for the report may be moved to someone who can oversee all the affected parts.
5. Fix. Making changes to the system amounts to recapitulating the overall life cycle pattern for building a part of the system. This can be seen as an instance of rewinding progress, as discussed in Section 20.3. The fix might be simple—there is one part of one component that is reimplemented, a test is developed to check the change, and it can be reviewed and approved. On the other extreme, the problem might come from a high-level design decision; the fix may involve changing that design, which in turn changes the specifications for multiple components, each of which must have their designs updated, their implementation and tests updated to match.
6. Verify. Once a fix has been implemented, the changes are verified. The fix is verified to ensure that it actually addresses the reported problem, and that it hasn’t created new problems. The changes may have affected how components integrate together, in which case the verification status of integration and interactions among them may be invalidated by the change, and the integration must be revalidated.
7. Review and accept. Once the fixes are complete and have been verified, the fix can be reviewed and accepted. At this point the updated designs and implementations are baselined (that is, made as the current mainline working version). The record of the error can be marked as completed.
8. Learning. Is there something that can be learned from the error and its fix that can be used to avoid similar errors in the future? This may be informal learning by the people involved, or it may be important enough to document and used to educate others on the team.

23.5.2 Change requests

From time to time, someone will request changes to the system. The request may come from a customer, asking for a change in behavior or capability. The request may come from the organization or funder, reflecting a desire to meet a different business objective. The request might even come from a regulator, when the regulations governing a system change or when the regulator finds a problem when reviewing the system.

The pattern for handling a change request has much in common with the one for handling a defect report.

After receiving a request, someone evaluates the request to ensure that it is complete and that they understand the request. After that, there is a decision whether to proceed making the change and, if so, what priority to put on the change. After making the decision to proceed, there are steps to design, implement, and verify the changes and eventually release the new version of the system.

Change requests differ from defect reports in two ways. First, requests for changes do not reflect an error in the system as it stands. The team can proceed building the system to meet its current purpose and defer making changes until after the current version is complete and released. Second, most requests are expressed as a change in the system’s purpose or high-level concept rather than as a report that a specific behavior in a specific part of the system does not meet its specification or purpose. A high-level request will have to be translated into, first, changes in the top-level system specification, and then propagated downward through component specifications and designs to work out how to realize the changes. This sequence of activities to work from the change of objective to specifications to designs to implementations is essentially the same as the activities to specify, design, and implement the system in the first place. In the pattern shown above, the “develop update” step amounts to recapitulating the overall system development pattern.

The decision to proceed with a change or to reject it depends on whether the change is technically feasible and whether it can be done with time and resources available. This depends on having an analysis of the complexity involved in making a change. Ideally, the team will be able to estimate the complexity with reasonable accuracy and little effort. Analyzing a change request will go faster and quicker if the team has maintained specification and design artifacts that allow someone to trace from a system purpose, down through system concept, into specifications and designs, to find all the parts of the system that might be affected by a change. If the team has not maintained this information, someone will have to work out these relationships from the information that is available—which is difficult and error-prone.

23.6 Comparisons and lessons learned

The life cycle patterns in this chapter have all been developed in order to guide teams through their work. To meet this objective, they have to be accessible and understandable by the teams using them; they can’t be explained in legalistic documents that include many layers of qualification and exceptions. Some of these have passed this test and have been used successfully. Others, such as the Unified Process, have not caught on.

Some of the patterns cover the whole project, while others address specific phases or activities. One pattern often references other patterns: for example, a high-level pattern like the NASA project life cycle uses lower-level patterns for developing components or handling change requests. Some low-level patterns, such as handling change requests or error reports, can end up using or recapitulating higher-level patterns.

The specific patterns that a project uses depend on that project’s needs. A software project that is expected to be continuously reactive to new customer needs works differently from a project that is building an aircraft, where rebuilding the airframe can cost lots of money and time. The NASA approach is influenced by the US Government fiscal appropriation and acquisition mechanisms, which require programs to have multiple points where the government can assess progress and choose to continue or cancel a program.

All of these patterns implicitly start with working out the purpose of some activities before proceeding to do detailed work.

These patterns also implicitly reflect the cost of making and reversing a decision (Section 21.10). The NASA life cycle puts design effort before a decision to spend money and effort building hardware. The change request and defect report patterns place evaluating the work involved ahead of committing to make a change.

Part VI: Reference life cycle

A reference life cycle pattern for projects. It models what a full life cycle contains, and can be the basis for developing an actual project’s life cycle.

Chapter 24: Introduction

The previous chapters have introduced the ideas of life cycle patterns and development methodologies, along with the ways that the two affect each other. Chapter 22 introduced a number of characteristics that one can choose to match a project. Chapter 23 presented a number of example life cycle patterns, along with a rough framework for comparing the examples.

In this chapter, I present a reference model of development methodology and life cycle patterns. This approach is based on the approaches I have used myself or have observed others using in successful projects, along with learning from projects that have gone poorly. These recommendations do not attempt to follow any of the development methodologies dogmatically, instead taking the parts from several of them that work well. In other words, I have tried to distill a pragmatic set of solutions from the many options available.

The reference life cycle covers the entire life of a systems-building project. It has four high-level phases: preparation, development, operation, and ending.

Project preparation is about setting up the project: how it will work, who is sponsoring it, who is funding it. Development covers working out what the system is for and then designing and building it, until it is ready for use. Operation is about producing the system, deploying it, using it, and evolving it. Ending is about shutting down the project when its work is done.

This reference also includes a project support “phase”, which includes all the activities that go on throughout the project to support operations.

Some projects are only concerned with building a system; once the system has been implemented and tested, it goes into production or operation and is no longer the concern of the development team. Those projects skip the operations phase. Most projects, on the other hand, have some level of involvement after the system is deployed and in operation, such as fixing bugs or enhancing the system. These projects involve all the phases.

The phases in the top-level life cycle in turn expand into more detailed patterns. Development consists of working out a purpose and a concept for the system, then developing a system to match, ending with a review to determine that the system is acceptable for putting into operation. Operation expands into a pattern of several phases, which I will discuss below.

Some projects will spend most of their time in development, while others spend most of their time in evolution after the system is in operation. Exploratory spacecraft missions usually consist mostly of development, since once the spacecraft is launched there is little opportunity to change the spacecraft beyond the occasional software update. Mass-market consumer software, on the other hand, often spends as little time as possible on initial development and can spend years developing upgraded versions to keep consumers satisfied. This reference life cycle fits both kinds of projects.

The arrows in this diagram show how information and artifacts flow from one phase to another, but they do not necessarily indicate complete temporal orderings. For example, the project preparation phase often lasts quite a while, and overlaps early parts of the development phase. Within operation, different customers might deploy and operate their own instances of the system, and the project may be working on multiple system improvements at once.

Two of these phases—system development and system evolution—involve designing and implementing parts of the system. These are the two phases where a development methodology applies.

I will discuss each of the top-level project phases in turn in the coming chapters.

24.1 Projects with proposals

Some projects require developing a proposal to get funding or approval to proceed.

The life cycle for this kind of project adds a phase between preparation and development to develop a proposal. Developing the proposal typically involves developing the purpose and a preliminary concept for the system, so that the potential customer or funder can understand what they will getting if they agree to fund developing the system. The initial concept is then documented as part of the proposal itself, which is typically a document (often a large document) explaining what the system will be, how it responds to the customer’s requirements, how long it will take to develop, and how much it will cost.

Much has been written about how to do proposal development well. There is best practice for how to organize a proposal development team and what kinds of reviews are helpful.[1]

After the customer or funder has agreed to the proposal, system development proceeds as it does for other kinds of projects.

24.2 Project-wide decisions and reviews

Projects have times when there will be a decision whether to continue the project, end it, or continue with significant changes. Some examples: whether to start a project, when additional funding is needed to continue, or at periodic progress reviews.

These are often not driven by progress on making the system. They can be driven by external considerations, such as the need for funding, or by a regular cadence of progress checks.

Such reviews or decision points do not fit neatly into the flow of phases defined in the life cycle pattern. When multiple steps are in progress concurrently, as happens during most of the development phase, the decision often happens in the middle of several of them. Preliminary specification or design reviews are also common; they happen part way through specifying or designing part of the system. Design reviews often mean that design should have reached a given level of completeness for the top X layers of components in the system.

I will note some representative decision points in this reference lifecycle, but the actual milestones are project-specific.

Chapter 25: Project preparation

The project preparation is for getting together the things that the team will need to operate.

Chapter 26: Project support

Project support covers all the various things done continuously in the project to keep the team working. Project support starts with the beginning of the project and ends when the project ends.

This phase includes work to monitor and manage parts of the project. Teams are one example (Section 19.3.1); maintaining plans and tasking (Section 20.5) is another. Tracking project risk (Chapter 63) and technical uncertainty (Chapter 62) supports planning.

Other elements of project support include:

• Artifacts: setting up and maintain the tools that store artifacts and handle artifact-related workflows.
• Tools: defining how to acquire and maintain tools, and how people will be trained to use them.
• Teams: hiring, on boarding, and off boarding procedures; how to maintain a team directory (Section 19.4).
• Space: the office and work space needed for the team.
• Partners: how to maintain relationships with partner organizations, such as subcontractors.
• Finance: tracking budgets, working with accounting organizations.
• Production and vendors: how to select and work with vendors.
• Relations: who will be responsible for maintaining relationships with customers, the organization, funders, and regulators (Section 16.2).

These efforts, similar to project preparation, will usually start small and develop over time. Similar principles apply to the timing of work on project support.

Chapter 27: Development

The development phase sees the project work out what the system is supposed to do, and then build the system to meet that objective.

27.1 What is development?

Before going into the sub-phases that make up the development phase in the large, it’s worth thinking about how a system actually gets developed. A great many systems have been built over the centuries without the benefit of methodologies; with some experience, good systems engineers usually have intuition that guides them through development.

Development starts with a rough idea of what the system is for: what problem the system will solve, or what it can do for people. An aqueduct begins with the idea that something should transport water from a source to a town. A pump driven by a steam engine starts with an idea that a machine could pump water out of a mine better than a human- or animal-driven pump, and thus allow mines to go deeper than they had before.

The people thinking about the problem to be solved also often have some approaches in mind that might be applied. Someone responsible for moving water into a town might know about aqueducts that have already been built. Steam pumps were developed incrementally by many people over a period of over two hundred years.

For developing modern complex systems, the development process still begins with a general idea of what the system might do and what problems it might solve, perhaps with some key technical approach in mind.

The team needs to get from this general idea to a clear and precise definition of what they need to design and implement. This does not occur in one step; the detailed design of the system does not spring fully-formed from the chief engineer’s head. Instead, the team starts with a vague understanding and refines it bit by bit until it is clear enough for design and implementation to start.

The team does need to understand the system’s purpose before working out how the system should work. However, in practice these are often parallel efforts, where some people work with customers and other stakeholders to clarify the system purpose while some people begin to brainstorm ideas of what kind of system might meet that purpose. As the understanding of purpose becomes clearer, those who are investigating what the system might look like—the concept of the system—will refine their ideas. Those who are working on the system concept track updates to the purpose, often feeding questions back to stakeholders when they find something potentially ambiguous or when they suspect that some part of the purpose might not yet be worked out.

The system concept represents the bridge between understanding the customer’s needs and building the details of the system. The concept sets the general approach that the team will use. Working out the concept is a time for creativity, when the team can entertain many possible ways to build the system, eliminating those that aren’t likely and refining those that are promising. The team evaluates these possible approaches along the way to see if they are likely feasible to build and to meet the system purpose.

The team may be tempted to turn the concept-building exercise into a full system design exercise. This is unwise. First, the techniques used to develop a system concept are meant to be fast and fluid, not working to the degree of rigor that design and implementation require. Second, this can lead to a concept and design period that drags on and on when the team needs instead to make a decision about the high-level structure and then move on to investigating design based on that decision. Third, stopping to review the basic concept before committing to it makes for a better concept that will better guide the team later.

This means that a system concept will be (and should be) incomplete. It should show some of the big ideas of the system’s structure, and it should show that these ideas are likely to meet the stakeholders’ needs and are likely to be technically feasible. It should be accurate, in that anything named in the concept should in fact be a necessary part of the system, but it should not be precise, having all of the details worked out.

Once the team has a concept, it is a good time to step back. Is this system still worth building? Is it likely to be feasible? Is it going to be a good answer to customer needs? And is it plausible that the resources needed will be available?

As the development work moves forward, the team will refine the concept. They will find things missing in the concept and have to find designs that fill those gaps. They will find inconsistencies or mistakes, and they will have to correct them. At the same time, customer needs may change—so the initial concept will always be different from the final system.

The level of detail and analysis needed in the concept depends on the project. A project that is building a revolutionary system for potential future customers probably only needs a rough sketch of the system, since investigations will continue for months or years into what those customers really need. On the other hand, a project that is answering a request for proposal typically needs a much more developed concept in order to explain to a funder what they will get and why their funding will be a reasonable risk.

Once the purpose and concept are completed, the team can turn to actually developing the system itself. In practice this is rarely a sharp transition; instead, some part of the team may begin moving forward in working out a system specification even before the concept is finalized, or they may begin prototyping parts of the system that seem especially uncertain.

27.2 The development phase

Development consists of many sub-phases. Purpose development comes first, in which the team determines what customer needs the system will address. After determining the system’s purpose, the team develops a high-level concept for the system, then builds the system itself. The development phase ends when there is agreement that the built system is ready to be produced, deployed, and put into operation.

The first two steps set the direction for the system development work. Purpose development establishes a record of who the stakeholders are for a project, and what each of them needs the system to do. This record of the system’s purpose will be incomplete, initially, but it must be accurate at the time it is documented. The concept phase then provides the time to explore different ways that a system might be built to meet those needs. The concept records a high-level picture of how the system will behave, the environment in which it operates, and some of the main top-level components that will make up the system. The concept phase is also the time when constraints related to security and safety are refined, turning general objectives coming from the customer or other stakeholders into more precise statements of what those objectives mean. Part way through or at the end of concept development is a good time for a review and decision about whether to continue the project.

The system development step in turn consists of many tasks. In this reference approach, the development phase is organized first into a number of system feature development phases, using the development methodology to determine what those phases are. Each system feature development phase, in turn, is organized as a sequence of specify-design-implement-verify patterns.

In this section, I will first discuss the development phase as a whole, then go into more detail about each of the subphases and development methodology.

27.3 Purpose development

The purpose development phase is for working out in detail what the system is to be, in terms of what it will do for its users and who those users are (Chapter 9).

The people responsible for working out the purpose work with the customer (or a proxy for customers, when the customers are hypothetical; see Section 16.2.1). This requires the team to work directly with the customers, in order to understand not just what the customers are saying they need but also to identify implicit needs and to find constraints on the system that the customers may not be able to articulate.

The team does similar work with other stakeholders. They identify the objectives that their organization has: is it to make a certain level of profit? Are there time constraints on demonstrating capability? Who might be the funder, and what are they looking for? And finally, who might have regulatory authority over the system, and what regulation or standards apply? All this information creates constraints on how the system can be built and what it can do, and will be considered when determining whether these other stakeholders will agree for the project to continue.

The needs found in this phase define objectives that the system should try to address. The constraints, on the other hand, define things that must be true about the system. ! Unknown link ref

I discuss working out system purpose further in Chapter 31.

27.4 Concept development

The concept development phase is the transition between working out system purpose and beginning to design the system in detail. It is a time to work out an initial, rough idea of how a system might be built to meet the purpose and constraints worked out in the purpose development phase. It is a time to brainstorm many different possible approaches and to be creative. These different approaches can be evaluated and narrowed down to one concept. That concept is the start, not the end, of design; it will guide the work in the subsequent system development phase.

The system concept is a sketch of the system on paper or similar media. It should cover all the major behaviors of the system, but it should not go into great detail about how those will be achieved.

The concept has two general parts: an external view and an internal view. The external view takes a black-box perspective of the system, and includes:

• The users and external interfaces of the system, documenting who will use the system;
• The externally-visible behaviors; and
• The environments in which the system will operate.

The internal view is an initial sketch of the insides of the system’s black box. This view includes:

• The first-level parts or components of the system, and how they are connected; and
• Behaviors that these components should have.

The concept does not usually go more than one or two levels deep in the component breakdown.

This information can be recorded in different forms, and it usually takes more than one to capture it adequately.

• Key use cases capture information about users and expected behaviors.
• A structure diagram, showing the major components and their relations, captures some internal information.
• A concept of operations provides a narrative of how the system will operate and illustrates both internal and external behaviors.
• A requirements matrix or requirements tracing tree shows how the system concept meets the system purpose.

Documents recording analyses complement these records. The whole collection of concept documents also records the rationale for decisions taken, and perhaps includes records of alternative designs that were considered and not chosen.

The concept is used for three purposes. First, it reveals whether there is likely to be a feasible approach for implementing a system that meets the customer needs. Second, it provides an illustration to customers and other stakeholders that they can use to validate whether the concept meets what they expect their needs to be. Third, it provides a guide as the team begins to specify and design parts of the system for real.

A likely-feasible concept is one where there is likely to be some way to design and implement each of the high-level parts of the system, and that combining those parts will likely satisfy stakeholder needs. The concept can only be likely, because it is supposed to be developed quickly; the uncertainties about whether the concept will actually work are not completely resolved until the whole system has been built and verified. The process of developing the concept can generate a list of what technical uncertainties people have found or suspect. (These uncertainties guide work planning as the project moves into the system development phase.)

The concept gets reviewed by stakeholders, including customers or customer proxies. While a stakeholder might look at the list of their needs as generated in the previous phase and think it complete, I have found that when they step through how a system concept will operate they get a different perspective and come to realize things they missed in the list of needs. When they find that a system concept appears to meet all their needs, the act of validating the concept with them provides them confidence in the project.

Finally, the road that the team follows from initial idea to a complete design (and implementation) has to start somewhere. The concept provides that starting point. The high-level components identified in the system concept become the starting point to specify, design, and build all of the rest of the component in the system.

Chapter 32 discusses the work involved in making and documenting the concept. To summarize that chapter, developing a system concept involves brainstorming many possible approaches to meeting customer needs and sketching them out. These different approaches get evaluated and compared to find out how well they meet the system purpose and how feasible they are; this often involves doing simple analyses. The evaluations show where there are gaps in meeting customer needs or in the technical solution. The best possibilities get refined or combined and improved until the approaches have been narrowed to one best option.

I have said that the concept should be likely feasible, and that the technical uncertainty and project risk uncovered in the investigation should be acceptable. The obvious next question is, how likely or how much uncertainty? In fact these uncertainties and risks are not generally quantifiable, as they deal in unknowns and the point of the concept development exercise is to expose unknowns and not to work them out. Qualitatively, some projects can accept more risk than others: a startup that is developing a speculative new technology can accept far more risk than a project proposing a system for a fixed-price contract. The decision will require a judgment call on the part of project leaders.

27.5 Development methodology

The system development phase is about creating the system based on the concept worked out in the previous phase. At the end, the project has the artifacts for a working system ready to hand off to production and deployment. Along the way, the project may need to meet other milestones—preliminary and critical design reviews for government customers, or feature demonstrations for funders.

The reference development methodology structures how the team does the work to design, implement, and verify that system. It is based on the spiral or incremental methodology. Project leadership works out a set of intermediate milestones where the team builds and demonstrates some set of system features working—usually integrating different parts of the system along the way. There is a life cycle phase leading up to each of these milestones, in which the team does the tasks needed to add features to the system. These are called feature development phases. Each feature development phase has an expected duration. If it appears not to be on track to meet that deadline, the team takes this as a signal that corrective action is needed. Unlike in the spiral methodology, this methodology leads to multiple overlapping feature development phases, running in parallel on different timelines and working toward different milestones.

This approach was motivated by several goals.

1. Provide mesoscale guidance to the development process, so that there is continuity and planning while providing a way to adapt when needed. This level of guidance operates on a time scale longer than a few days or a couple weeks, but less than the whole project; it is intermediate between the time scales used in agile and waterfall development.
2. Allow threads of activity (feature development phases) to operate on longer or shorter timelines as is appropriate to the work.
3. Allow different feature development phases to follow the microscale methodology appropriate to the work in that phase. For example, software development often benefits from short, agile-like sprints within one feature development phase, while fabricating large mechanical structures is better done using techniques derived from construction industries.
4. Promote interaction and collaboration across parts of the system during design, while avoiding involving people working on unrelated parts of the system. Those who are actively working together to build some interconnected capabilities in a few components need to communicate frequently, but people working on some other part of the system do not need to sit through meetings discussing things they aren’t working on.
5. Support integration-first and uncertainty-first planning practices (Chapters 47 and 62).
6. Support a partial plan—one that is concrete in the short term and less so in the medium to longer term, with growing specificity as the project progresses (Chapter 64).

Compare this approach to waterfall and agile development methodologies.

Waterfall development, practiced strictly, does not handle uncertainty or adaptation well: the system is designed up front, and implementation follows thereafter. In practice, projects nominally using the waterfall methodology often develop intermediate milestones to organize the work.

Agile development, on the other hand, can lead teams to constantly change direction—unless they develop a plan with some longer-term objectives. When they do so, agile development ends up looking a lot like this reference methodology. Short sprint periods can also work poorly for parts of a project doing work that does not complete within one sprint, like building an airframe or developing detailed analyses.

Example. Consider the following example, taken and simplified from a spacecraft project I worked on. The mission involved multiple spacecraft working together to perform a science mission.

The mission’s concept development defined the overall design of the system: multiple spacecraft, communication links between them, communication with ground stations, and so on. The concept also defined an initial breakdown of the system, where the spacecraft had a set of major subsystems like structure, power, avionics, sensors, flight software, and so on. The concept identified some existing software and hardware designs that could be re-used for this mission.

The development phase, then, was about building hardware, software, and operational procedures that would implement that concept.

The team worked out the major steps that had to happen to build the system, such as designing the avionics, designing the structure, testing and integrating them, and putting sample spacecraft units through environmental testing (heat, vacuum, vibration). The project also would build software to run each spacecraft, which involved tasks like prototyping algorithms for attitude control and then verifying that they would work in testbed equipment. These major steps were partly worked out based on experience on previous missions, and partly from working backwards from the high-level system design to determine major functions to be implemented.

The following shows the first part of the sequence of feature development phases for the main flight software (simplified and abstracted from the original). The flight software had a series of milestones that started with the basic software infrastructure and a simulation environment for testing it. Later milestones then added capabilities one after another. Each milestone integrated new functions across several different components. In most milestones, the work involved behind the scenes was as important as what was overtly demonstrated; for example, the first demo was as much about establishing a software configuration management and build system as it was about demonstrating simple software running.

This project made extensive use of software skeletons or scaffolds, mockups, and emulations. This is typical of a project that prioritizes integration over feature depth. In this case, the main spacecraft control software for the first couple of demos was a simple skeleton of what it would become. The software modules involved could start up, and interact with some others in simple ways, but there was no real logic in the control. Building this part first reduced integration risks that the control software modules would not interact properly with the middleware and operating system on which they ran—and indeed showed middleware bugs that cause the system to crash. By the third demo, the team added basic attitude control logic to the control software. This attitude control still only had limited function; its purpose was as much to show that the control software could interact with (emulated) sensors and actuators.

27.6 System feature development

A system feature development phase is a stream of work that adds a defined set of features (the purpose of the phase) to the system, ending in a milestone with those features implemented, integrated, and demonstrable. It starts with design work that has already been done and the purpose of the work, and ends with system artifacts updated to meet the phase’s purpose.

This approach to organizing development is focused on the features rather than on the components or component breakdown. One feature development phase usually involves several components (and their subcomponents). It promotes the integration of work across parts of the system.

Reference pattern for feature development. A feature development phase recapitulates the life cycle of the overall system development life cycle. It starts with purpose, works out a concept, then proceeds into the specification, design, implementation, and verification of parts of the system to build in that purpose.

The concept for a feature development phase includes working the general design approach for adding the phase’s features. As with the system concept, the feature concept involves brainstorming different ways to implement the features, along with evaluations of the alternatives until the team selects one concept. The concept for the features should give a general idea of what components will be modified or created in this phase, along with the internal structure among those components and a narrative of how they will interact (the concept of operations for the features).

Identifying the components that will be affected is key to being able to scope how much effort will be required to implement the features, and who will need to be involved in the work.

The next step is to develop or modify specifications for the components involved (Chapter 33). These detail how the components are to behave and the non-functional attributes they are to provide. This may involve adding to or modifying the top level system specifications, or flowing those specifications down to components. Security, safety, and reliability specifications are particularly important.

Design follows specifications, working out how each component can be built to provide the behaviors and properties it is specified to have (Chapter 37). Design may require evaluating alternatives, perhaps by modeling or prototyping (Section 8.3.5; Chapter 41).

Two separate and independent implementation steps follow. One step implements components and changes to components, following the design. The other step works out how to verify the features in the feature development phase, including verifying both the individual components by themselves (using unit tests, for example) and the features that are provided by the components integrated into the system. If the verification implementation runs ahead of the component implementation, the component implementers can verify as they go (using test-driven development).

As parts of the feature set are implemented, they are verified. By the end of the feature development phase, the components created or changed in the phase and the features the phase is adding are all verified.

The feature development phase ends when the team successfully demonstrates that the system now has the features they have worked to implement. This demonstration might amount to showing that the new system version has passed its verification checks, but doing an actual demonstration gives the people who did the work an opportunity to show the rest of the project what they have done and for the project as a whole to celebrate their work.

Once again, note that this work is organized around the features, not the components. This methodology does not necessarily mean implementing each component’s changes in isolation, verifying those, and then verifying their integration. Rather, the team can order the work however works best for the particular task at hand. For example, an integration-first approach might lead the team to build simple skeletons or mockups of component changes and focus on checking out how the components will interact before implementing detailed changes to the components—which means verifying integration before verifying the unit components. (Of course, the finished changes still need to be verified as a whole before the verification work is done.)

The reference pattern for the feature development phase, in the diagram above, includes review milestones for each of the steps (concept, specification, design, implementation, verification) involved. These reviews serve two purposes. First, they are an opportunity for someone independent to check the work in order to find things the team doing the work might miss. Second, they provide an opportunity for the team working on the features to pause long enough to ensure that they all understand the work in the same way.

Finally, the team responsible for a feature development phase may decide that the phase is large enough that it should be split up into subphases. Each of the subphases might have its own milestone goals; those subphase goals build on each other to reach the features of the main feature development phase. These subphases might focus on individual components or smaller groups of components, or they might split the work into sequential steps, or some combination of the two. These subphases follow the same pattern as the higher-level feature development phase of which they are part.

27.7 Recursion to component development

A system feature, in the end, is made up of behaviors and properties of a number of components. That is, system features are emergent from the individual components involved.

The work to implement a system feature is thus made up of the work on each of the components, along with the effort to integrate those components and their changes. The team working out the concept for the feature determines how parts of the high-level feature are allocated to components. That is, they work out what behaviors or properties are needed from each component so that together they produce the high-level feature. Along the way during concept development, the team works out what components are affected by the feature development work.

The feature development life cycle pattern for the high-level feature applies for developing the changes to each of the affected components. Just as the feature as a whole has concept, specification, design, and implementation steps, so do each of the components. Developing the concept for the feature includes developing a concept for each affected component. Developing the specification for the feature leads to developing specifications for each component, and so on. The implementation of the feature is the implementation step for each component.

The people who are working on all these component pieces coordinate their work so that it all integrates properly and produces the desired features.

That coordination means that the work on each component moves at a pace at least partly constrained by the work on other components: for example, the specification step for any one component cannot be completely finished until the specifications for all the affected components are finished. Otherwise, the specification work in some other component could reveal a surprise that affected the specification that was thought to be finished.

At the same time, teams rarely just stop and sit idle when the work on some component lags. They proceed from specification to design to starting implementation, accepting the risk that some surprise may happen that will require them to re-do some amount of work. The choice of how much work to do at risk has to be made based on the usual estimates of likelihood and consequence. If the work on some other component is almost done and is in the final stages of cleaning up details, the likelihood of finding something that will require a change to other components is unlikely. On the other hand, if the work on some other component is just getting started, then the chances of a surprise are high. If part of the component in question appears to be fairly immune to changes in other components, then there is little risk of having to redo that work. For example, if the component will definitely need to communicate over a network with other components, then getting network communication designed is low risk.

The figure above illustrates how the work for a feature is coordinated across all the components. The top row shows the steps or phases for the feature as a whole. that work is broken down into the work for two components, shown in the middle two rows. The components each follow the feature development pattern of concept, specification, design, implementation, and verification. The last row covers the thread of work done to address integrating the changes to individual components, and it follows a reduced form of that pattern. The feature integration thread of work is primarily about checking that the work on the components properly combines to produce the high-level system features, and so it focuses on verification methods for this integration.

The figure also shows that the concept development work for the high-level feature and the affected components may often be done as a single task. If the feature and components are simple enough, a small group can work out the concept together and produce one set of concept artifacts that cover both the feature as a whole and its effects on specific components. In this case, the artifacts for each component will reference the shared concept artifacts; after a while, the records for a component may reference several concepts for different features.

If the feature or the components are more complex, the work may need to be divided up so people can work on different parts in parallel, combining and reconciling the pieces before the concept is completed. The artifacts for the components will then reference their own concept for that feature as well as the high-level feature concept documents.

27.8 Feature development variations

The feature development pattern in the last section covers the simplest case: when the team is designing and building a straightforward feature. There are three variants to consider: when the component carries enough uncertainty that prototyping is warranted; when the component will be acquired from outside the project rather than built in house; and the specific needs for implementing hardware components.

Prototyping. Prototyping is used when there are possible technical approaches to designing some part of the system, and the technical uncertainty is too high. In these cases, taking steps to reduce the uncertainty before committing to one particular design can lead to better outcomes.

The uncertainty can take different forms. In one case, the team might have an idea, but they don’t know if it will work correctly. In another case, they may not have an idea for a solution, and they need to explore and learn in order to find possible solutions. Or the team might have a solution, but lack skills essential to completing design or implementing it. Finally, the team may have a solution that is not technically mature enough, and they need to validate its suitability. In each case, developing a prototype of some kind can help.

The prototyping effort is added to the design step. The prototype might take the form of a simple implementation, or of a model of a possible solution. Any prototyping effort should have a clear purpose: to see if an idea works (and working out what it means “to work”) or the like. The focus must be on learning what is needed as quickly as possible. The work should prioritize speed of learning over quality of the prototype implementation.

Prototyping can be a necessary part of learning about a design and managing its uncertainty, but its contribution to the system is indirect—by leading to a good design. The amount of effort or time spent on the prototype should be bounded so that the prototyping effort does not take over the development effort.

The principles about prototyping (Section 8.3.5) apply. The prototype artifacts should be built as quickly as possible to maximize efficient learning, without putting in effort to make them high quality. The artifacts that come out of the prototyping work must not end up in the real implementation.

Acquired component. Sometimes components are best acquired from somewhere else rather than being designed and built by the team. This might involve reusing a component from another project, or using an open source design, or purchasing a component from a supplier. Acquiring a design or component can take advantage of work that others have already done, reducing development costs. It can take advantage of expertise that the team does not have itself, such as a supplier that can manufacture an electronics board or a software vendor that has developed a component with a particular algorithm.

The pattern for an acquired component proceeds with developing a concept for what is needed and a specification for the component. The specification is the basis for a request for proposal (RFP), which is sent out to potential suppliers that are expected to offer potential solutions. The suppliers in their turn use the specification to develop a design, which might simply be an off-the-shelf product or might involve development work on their part. Once the suppliers have a design, they respond to the team. The team evaluates whether the design in fact meets the specification and determines which option is best, if there they have more than one potential choice. In many cases the team will build a simple prototype using a supplier’s prototype implementation, if they have one, as part of the evaluation. After that, the supplier implements, builds, and delivers the component. In other words, this pattern moves the design and implementation work away from the project team and onto the supplier.

The team, however, still does some amount of verification once they have received the implementation. This acceptance testing may be more limited than it would be for a bespoke design, if the supplier provides information about the verification steps they have taken. Nonetheless, the team should spot check any verification work that the supplier has done and must check that the supplied component integrates as expected into the rest of the system.

Acquiring components like open-source designs or software do not have to go through the process of developing a formal RFP. However, these components do still require evaluation before deciding whether to use the design or not. The team must ensure that the license terms are compatible with the system. The team must also ensure that the potential component meets the specification of what is needed of it. Finally, the team must evaluate the quality of the component—which for open-source components, includes not just the quality of the artifact itself but also its governance and supply chain security [Goodin24][CVE24].

This pattern involves support roles that I have not detailed out elsewhere. For example, the acquisition might involve someone who manages contracting or payment. The acquisition will likely involve checking that the license terms and intellectual property rights associated with the component are appropriate for the system the team is building, which may require legal expertise.

Hardware components. Hardware development has different constraints than some other kinds of component development, and so a different development pattern applies. The primary cause of the differences is that a hardware component involves physically building one or more artifacts, which can take time and resources. This makes iterating on a design to work out bugs or to change features much more expensive than it is for software or higher-level designs. In addition, some verification testing is destructive, putting a component in increasingly harsh environments or under harder loads to determine when it fails.

Hardware development also differs from other kinds of component and feature development in the way terms like “design” are used. A design for an electronics board is a full description of how it is to be implemented; in some cases, it can be sent to an automated production system to create a complete physical board. Similarly, many mechanical designs are complete enough to send to a CNC machine or additive printer to create the physical artifact. By comparison, a software design is more abstract; it cannot be directly translated into a working program. Software source code is closer to mechanical or electronic designs, as source code can be sent to compilation tools that produce the executable artifact.

These constraints have led to disciplines about how to organize hardware development. I discussed the EVT/DVT/PVT pattern earlier (Section 23.4.1), which defines a sequence of phases for developing and verifying a hardware component. The NASA approach uses different language [NASA16, p. 124] to describe the sequence of hardware artifacts to be developed and verified. The two approaches are similar, with one naming the phases and one naming the artifacts.

This approach splits up the design, implementation, and verification phases into multiple iterations. There are typically four iterations.

1. Preliminary development: produces a breadboard or brassboard, which are low to medium fidelity versions of the component. This version is focused on function but often has a form unlike that of the final. It may use off-the-shelf components that will not be used in the final version. These may be subscale or digital models. This step may be repeated multiple times, adding features at each iteration.
2. Engineering unit development (or EDU, engineering demonstration unit): produces a version that closely resembles the final version in both function and form. It is put through EVT (engineering validation and testing), which verifies that the version mostly meets its specifications. The engineering unit may have a small number of defects, but at the end of verification the team should have confidence that these can all be corrected to produce the final version.
3. Qualification unit development: this produces one or more units that are built to the final design. These units are typically built manually in small numbers. They are put through DVT (design validation and testing), which involves thorough testing of the function and form of the component. For some systems, some of these units will be tested to failure in order to show that the design can function correctly across the full range of environments in which it will operate. Aircraft wings, for example, are tested by flexing them until they fail. Spacecraft electronics are subjected to heat, vacuum, radiation, and vibration beyond what they will experience in use, and those tests are likely to damage the components. These units are also used for certification with regulators or similar industry organizations.
4. Production or flight unit development: this step produces a small number of components that can be deployed into operational systems. These are used to verify final manufacturing processes, using the PVT (production validation and testing). This includes matters like supply chain operation, manufacturing logistics, and delivery and storage of the final units. These units are put through acceptance testing, which verifies that the manufacturing process builds components that are identical to those built for qualification.

Figure 27.2: Hardware development phases

The fourth step, producing production or flight units that can be deployed, can occur as part of development or later, in a production phase after the system has been accepted (Section 28.1). If a component is going to be mass produced, verifying the manufacturing methods is worth doing before declaring that the component is complete. After acceptance, the manufacturer will build more units. On the other hand, if only a handful of units will be built and they are expensive to build, such as with individual spacecraft, delaying the production of those units until after acceptance can manage risk.

Finally, the development of a hardware component is part of the development of the larger system. This leads to two ways that the hardware development steps can be organized, depending on how the hardware development will be synchronized and integrated with other parts of the system.

The first way is to plan out the hardware component development as its own thread of work. This way has the advantage of keeping the team focused on designing and building the component.

The second way is to break up the hardware development thread into smaller steps, and put some or all of those steps in feature development threads. For example, when building a circuit board that will run a control system, it will be hard to verify that the board works without some version of the software that runs on it or the interfaces to sensors and actuators of what it controls. In other words, verifying the integration of the hardware component with other parts of the system is an essential part of checking that the component actually works. This is the way virtually every project I have worked on has actually planned out its hardware development work.

As an example, this sequence of feature development steps is loosely based on two different control system implementations in projects I have worked on. The sequence shows how different hardware and software components come together to implement increasingly complex features. This approach integrates the hardware and software parts in incremental steps.

27.9 Acceptance

The acceptance phase is the time for final checks that the developed system is indeed ready for production and deployment. It is the last step in the overall system development life cycle.

There are three kinds of checks involved: that the system can be put into production and deployed; that the customer (or their surrogate) validates that the system is what they need; and that regulators approve the system, if needed.

The check for production and deployment involves verifying that the manufacturing and distribution process is ready for operation, and that all the procedures and tools are in place to install a manufactured product for customer use. For a software-only product, the manufacturing and distribution procedure involves packaging the software release and putting it on distribution servers (or manufacturing distribution media if it is not distributed over networks). The deployment readiness involves verifying that the packaged software has prominent and understandable instructions on how to install it and start using it. On the other hand, for a mass-produced hardware product, verifying manufacturing and distribution involves checking that the manufacturing line can correctly build the system, that it has the proper supply chains in place to support the manufacturing, and that the products can be shipped and warehoused before delivery to customers.

Validating that the system meets customer needs involves customers trying out an instance of the system—not just looking at documentation about the system. This often involves getting one or more customers to use a test installation of the system to do the tasks that the customers need. For some systems, this kind of validation can be done by beta testers, who are given an almost-ready version of the system and try it out in their environment while providing feedback of what works or doesn’t. Other systems that involve more installation and setup can involve setting up test installations that the customers come to use.

Regulatory approval involves different procedures in different industries. An aircraft, for example, must be reviewed and certified by the appropriate civil aviation authority. A spacecraft mission typically requires licenses for launch, communication, and certain kinds of earth observation. Other systems may need approval by an industry safety organization. Most of the work to get these approvals or licenses is part of the development phase, and the acceptance phase is the final check that the necessary approvals are in place.

Once these checks are completed, the final milestone is for the organization and the project to decide whether to proceed to production and deployment or not. Many systems are designed and built, but in the end the organization behind the project decides that the result does not justify the investment in production. Many commercial aircraft, for example, are designed and built, but in the end there is not sufficient sales interest to start production and the aircraft model is quietly retired.

Chapter 28: Operation

Once the system has been developed and verified, it is ready to be manufactured, deployed, and put into use. The initial work of building is done, but there is much more to go. There are several ways the operation phase can proceed, depending on the kind of system, kind of customer, and the role that the organization that developed the system plays.

The general flow is first to manufacture or produce the system using the artifacts that have been developed, then deploy instances of the system. After that, the system instance is in operation. Further development of the system, to evolve it or to fix problems, continues in parallel with customer operation. Finally, at some point, the customer will decide to retire and dispose of the system instance. The steps of deploying, operating, and retiring system instances can occur multiple times in parallel for different customers.

28.1 System production

The production phase covers manufacturing the artifacts to be deployed.

Bear in mind that this is a brief overview of manufacturing, intended to explain the main points that people like systems engineers or project managers will need to know in order to understand the general scope of the work, and to understand how the manufacturing steps are related to other parts of the system-building work. Manufacturing has been studied and refined for a couple centuries, and there is an extensive literature with far more information.

There are several kinds of production that different projects might use. These include:

• One or small batch production, often done mostly in house. For example, a project might manufacture one spacecraft and a backup for a mission, or a company might build a one-off system for internal use.
• Mass production, often by outside manufacturing lines. Physical consumer or office products use mass production.
• Digital systems with little or no physical artifacts. For example, software that people will download over networks.
• Hybrid, with physical artifacts and digital added to them. This is the most common today: systems that have both physical artifacts that are manufactured and software or configuration that is loaded into them, and that software may be updated in the future.

Production of a new system for a new installation can also differ from production of parts for maintaining or upgrading an existing installation. A new system might consist of a complete collection of hardware components that will be assembled from scratch for the installation. Producing replacement or upgraded parts, on the other hand, consists only of manufacturing a few parts and making them available for deployment into existing installations.

A review and approval to begin production milestone checks that the project has everything ready before committing to production, as discussed below. The review checks that the system development has completed all its milestones and that a system will be ready to deploy when manufactured. It also checks that everything needed for production itself is ready: the manufacturing tools and people, suppliers, testing. It also checks that the organization is prepared by being able to pay for supply and manufacture, and that people are ready to deploy systems once their parts have been manufactured, so that capital does not remain tied up in unneeded inventory.

Production relies critically on security of the supply chain, management of the developed artifacts, the manufacturing process, and the delivery mechanisms. All these elements of the production process have been attacked in recent years. For example, the SolarWinds attack [Zetter23] compromised the production process for their software, which was then distributed to and installed by many other organizations and led to attacks on those other systems. There are other reports of fake hardware components (e.g. pressure sensors [Control19]) being injected into a supply chain. These attacks can result in loss of system components, delaying deployment to a customer, exposure of intellectual property, deployment of a faulty or dangerous system, or creation of security problems for the system’s customer.

The overall production process has the following steps:

• Take in implementation artifacts, input stock supply, and manufacturing procedures.
• Acquire input stock from suppliers or inventory.
• Perform manufacturing procedures.
• Perform acceptance tests.
• Record tracking information for each manufactured unit (such as serial number, build history, or test history).
• Deliver deployment-ready artifacts to inventory or directly to customer.

This flow depends on the supply chain of parts used in manufacturing or production. Any physical parts or stock used must be on hand to perform manufacturing; this implies that the stock is in inventory, and that it has been supplied from some qualified source. Sourcing implies selecting the suppliers and setting up contracts for them to provide the stock. The contracts with the suppliers should include clear specifications of exactly what stock or components are to be supplied, along with evidence that the delivered parts meet the specification.

Procedures for receiving materials from suppliers and maintaining inventory are part of the definition of manufacturing procedures. The procedure will typically need some amount of space for maintaining this input stock, along with managing information about what stock is on hand and what should be used next. The storage space maintains the input components or stock in an environment that will keep the material in its designed storage conditions. The procedures include determining when to order more stock. The receiving and storage facilities should have security that ensures that material is not stolen or replaced.

The production process relies on accurate configuration or version management. The artifacts used to manufacture the production components should have consistent versions, and those should match the versions used for final verification during development. If inconsistent implementations were manufactured, the components might not work together—and the resulting problems are often subtle.

The manufacturing procedures specify who does what steps, in what order, using what tools. These procedures are designed during system development and verified during production verification testing (see the section on hardware development above).

After system components have been manufactured, they are checked to ensure that there are no manufacturing defects. This is typically called acceptance testing. For many hardware components, this involves putting the component through a set of tests that are defined during system development. These tests do not stress the component to a level that will induce faults, like testing at high temperatures or voltages; the tests only look for potential manufacturing problems. Some mechanical or electrical components go through a “burn in” period, which operates the component long enough to catch early component (“infant mortality”) failures. For some other kinds of components, only a sample of each batch of components gets tested, under the assumption that manufacturing defects will tend to cluster in one production batch (for example, one day’s production shift).

The production process involves a significant amount of record keeping. Each produced component has its own set of records. These records start with the component’s identity, typically represented as a serial number. The record identifies what version of the input development artifacts were used, often by associating a release version number or code with the serial number. The records include when, by whom, and using what equipment the component was built, so that if parts start failing an analysis can identify other components that may be at higher than expected risk of failure. The records track what parts or stock were used to manufacture the component: the serial number of components used, if appropriate, or the supplier, model, and batch number of stock.

In addition, each manufactured component must be identifiable. That typically means that it should be clearly labeled with its model or version information and serial numbers, at minimum. The labeling is typically in both human- and machine-readable forms.

Once a component has been manufactured and checked, it is placed in inventory and later delivered for deployment. The components in inventory are stored in secure spaces that maintain the components in their designed storage environment—often dust-free, within a particular temperature and humidity range, and so on. The inventory is managed to know what components are in stock and ready to send for deployment.

The production process needs to be resilient to disruptions. One company I worked for was building hardware systems outside the US, and investors asked the company how they would handle a political or military disruption in that country. (The answer was that the company would go out of business because it had no alternative manufacturing option.) Many production or manufacturing processes are also in places that can be vulnerable to natural disasters, including earthquakes and storms.

Finally, the manufacturing process is generally a human process, and processes involving humans have a tendency to drift over time away from their originally-intended procedures (see e.g. Leveson [Leveson11, Chapter 12]). This drift can come from changes in how people are trained, people finding potential simplifications in the procedures, changes in the environment in which the people are working, and many other causes. The designs of robust, safe manufacturing procedures include periodic audits to check that people are performing the procedures as originally designed, and to re-design the procedures if they are found to have problems in use.

28.2 System production examples

I mentioned earlier that different projects follow different kinds of production patterns. Here are a few examples that show some of these different approaches.

28.3 System deployment

The objective of deployment is to set up a system instance for a customer and get them successfully using that system.

There are several kinds of deployments. The first variation is: who is doing the deployment? Consumer products are set up and installed by the customer. More complex systems are delivered and set up by a team that is part of the project. I will refer to this as “assisted deployment”. Other systems are deployed and used internally by the organization that created them. The second variation is whether one is deploying a complete new system installation, or installing an upgrade into an existing system.

The overall flow of events is the same for all these variants:

1. Deliver and install the system (or its new components) at a customer’s location;
2. Verify that the installed system works as expected;
3. Train users on how to use the system properly;
4. Migrate any data or material if needed; and
5. Put the system into operation.

A system is deployed into an environment. That environment might be a customer site for a physical system; it might be spread over multiple sites; it might be an attachment to a launch vehicle; it might be resources on a compute server somewhere. In all those cases, the customer finds the places where the system can be installed. The deployment team and the customer usually interact before deployment starts to let the customer know what is required for the system, and for the customer to let the deployment team know what is available.

The environment for a software system might include the number and kind of compute servers used, the amount of memory or storage on each, the reliability and security of the servers, and the reliability of each server.

The environment for physical systems might include physical space, along with the temperature and atmosphere in that space. It might include the mechanical mounting needed, along with electrical, water, networking, and other supply lines.

Some customers will be migrating from an existing system to the new system being deployed. The migration might include moving information from the old system to the new system, or it might involve moving physical artifacts or supply from one to the other. Developing the migration procedures are a development activity on their own; in effect, they are a second mini-system to design and implement.

Complex systems will have users who need to be trained in order to operate the system safely and correctly. The initial group of users are trained during deployment, so that they can verify that the system works correctly and can take over its use once the installation has been accepted. Other users will learn to work with the system later, perhaps years later.

The installed system includes education and training materials for these users. These materials are assembled during the development phase of the project.

Different kinds of users may interact with the system. At the simplest, there are users who directly command and use the system’s primary behavior. A system may also have administrators who are responsible for specialized tasks, such as managing the set of users or the system’s security. It may have people who are responsible for maintenance and repair. I likely has other people who set policy for how the system should be used. All these people use the education and training material, and that material must address each of their needs.

Deployment presents a number of ways that someone could attack and compromise the system. The system components will be in transit from the production facility or warehouse and could be tampered with; they will be received at the customer site and might be accessed before being installed. The system components may be partially installed but not fully configured to be secure during the deployment process. The deployment procedures themselves could be altered or hijacked. All these potential exposures mean that the deployment procedure must be designed with security in mind, and that security must be evaluated as part of the system requirements.

Deployment includes setting up the customer on a customer service system. Once the system has been installed and the initial users have been trained and given access, the customer will begin to take over system operations. As they do this, they are likely to find they do not actually understand some parts of the system and have questions. They will use the customer service system to communicate with the project team for questions and to report problems.

Accidents and incidents may happen during system operation. When these happen, the customer works with the team that developed and maintains the system to investigate what happened. If the accident is serious enough, regulatory agencies may be involved. During the deployment process, the team establishes the necessary working relationships with the customer that will help the customer to detect when accidents have happened and to bring in the team for investigation. The investigation may determine that there is a flaw in the system, in which case a problem report and change requests are sent to the team to guide fixing the flaws. Section 28.7 below addresses how the team handles such changes.

Setting up the customer for ongoing success using the system is the last part of deployment. Once the system is in operation, the customer’s users are responsible for the ongoing safe and secure use of the system. Users of complex systems tend, over time, to find shortcuts and workarounds for how they use the system. They may forget part of their training, and new users may not be trained fully correctly. The environment in which the system operates may also change—parts might be moved, air conditioners changed out, or electrical feeds changed, for example. All of these can slowly change how the system is working and lead to accidents. Regular monitoring or auditing of system and user behavior is necessary to detect and correct these drifts and avoid accidents, and this auditing must be backed by management policy and actions. (See Leveson [Leveson11, Chapter 12] for background.) The deployment activities, therefore, must include working with the customer to establish the necessary monitoring activities and to establish necessary management policies.

Customer deployment. The components for these systems are delivered to the customer, who is responsible for installing or upgrading the system. The process includes:

• Delivering the system, along with deployment procedure instructions and training material.
• The customer sets up the environment for the system. This might involve creating space for machines, providing power, providing network connections, providing cooling, and so on.
• The customer follows the deployment procedures to install and test the system installation.
• The customer might contact the project’s organization for assistance, but most customers should be able to follow the instructions on their own.
• The customer migrates any materials or information from an old system. The customer is generally responsible for determining how to do this, though the system deployment instructions might include procedures for common cases.
• The customer puts the system into service.

Assisted deployment (internal or external). When someone from the project team does the deployment, the process is similar to customer deployment.

The process includes:

• The team gives the customer requirements for the environment the system will require, such as server room, logistics bay, assembly bay, or an open field. The customer is responsible for arranging that environment before deployment starts. The team may verify that the environment is ready before starting delivery or deployment.
• The team delivers system components to the customer site, then assembles them following deployment procedures.
• The team performs basic testing to ensure assembled system works.
• The team provides training on operation and maintenance to the customer.
• The team and customer may work together to migrate information or materials from a previous system.
• The customer performs acceptance testing, and accepts the deployed system when no issues are found.

28.4 System deployment examples

Deployment follows many different patterns, depending on the kind of system and customer. The following four examples illustrate some of the range of ways that the general deployment step can happen.

Complex system, shared deployment responsibility. The previous examples have been simple, performed entirely by the customer. The next example covers a more complex deployment.

Consider an information system that supports a repair and maintenance workshop. This example is based loosely on a system I worked on for local government public works agencies, which maintained a wide range of equipment from buses to lawn mowers to backhoes.

The repair and maintenance organization had multiple shop sites. Some shops were specialized for working on particular kinds of equipment.

The system provided record-keeping support for managing work orders (repair orders), scheduling resources like work bays or large equipment, and managing parts inventory. It also interfaced with the customer’s other IT systems: security and user authentication systems, and systems to place orders to buy parts and to pay for them.

For a particular installation, the customer asked for a set of features to be added to an existing software package. The development phase of the project for this customer involved work to determine their specific objectives and changes, implement changes to the base system, and then validate the customized system with the customer. Once the customer accepted the changes at the end of the development phase, a production phase generated the software installation packages and other materials for the deployment.

Physically, the system consisted of a small set of servers in a server room, plus workstations of different kinds at the workshops. This equipment used communication equipment between the sites and the server room. The server room provided power, cooling, communications, and support services like backup and security for the servers.

The customer wanted to perform a phased installation and roll-out, where initially only a few people would use the system and over time its use would be extended to more and more sites. The goal was to minimize risk by avoiding disruption to the shop’s existing work, and to contain any problems that might come up as the shop users learned to work with the system. A phased installation would also allow the customer and deployment team to monitor the performance of the servers and communication systems in order to identify unexpected behaviors before they caused problems. The customer decided to continue using their existing (paper-based) system for all existing work, so no data would be migrated into the new system.

The project’s deployment team was responsible for installing and configuring the initial system, and for training the initial users. The customer installed servers and communications, along with workstations at the shop sites. The customer was also responsible for adding users to the system and would take over training and configuration after the system had been rolled out to half the shop sites.

The deployment process proceeded as follows:

• The customer installed servers and networking to host the shop management software.
• The deployment team installed the software and verified that it was running correctly.
• The customer installed workstations for the system administrators, and the deployment team trained the administrators.
• The customer installed workstations at one test shop site. The deployment team trained users at that site. The customer’s IT staff and the deployment team helped the new users and monitored system performance for the one shop site.
• The installation and training process continued with a second site.
• After half the sites were set up and users trained, the customer accepted the deployment and took over responsibility for operations. The customer installed workstations at more shop sites and trained the sites’ users.
• The deployment team remained available to help with questions and problems under a support contract.

This system thus had a phased transition between deployment and operation, rather than a hard split between one phase and another.

28.5 System operation

In this phase, the system is placed into operation. The customer uses the system, performing administration and maintenance as needed. Most of the system operation is the customer’s responsibility; in this section, I focus only on what the project does to support the customer’s operation.

The system operation phase affects the project team in two ways. First, the team will sometimes support the customer during operation. Second, point of the team’s work is to build a system that can go into operation, which means that the system’s design supports all the activities that the users will do. This includes the rare and exceptional activities, not just everyday usage, so these activities are included in the concept and specification to which the system is built.

The customer is responsible for maintaining the system. That may mean only following procedures for periodic checking, but for many systems maintenance can be far more intrusive, and involve regular replacement of some components. The customers rely on maintenance procedures that are designed as part of the system to keep the system operating safely; these maintenance procedures are designed to take safety, security, and reliability constraints into account. The customer also periodically orders replacement parts to install into their system.

The customer also takes care of their users. This includes adding and removing user access to the system and training those users. The project team supports these tasks by including features to manage users and their roles as part of the system. The team also develops training material that the customer uses when bringing on new users.

The system may have problems from time to time. These may reflect flaws in the system, improper usage, wear and tear, or combinations of all three. The customer, as the system owner, is responsible for handling the problems. However, the project team sets the customer up to be able to address problems by developing instructions for detecting and diagnosing problems, and training some of the customer’s staff on how these work. The project may also provide services to help diagnose and repair problems. The project also provides some form of customer support that the customer can use to report problems back to the project.

Most complex systems have human elements—users who operate the system and in doing so act as a control system that manages system behavior. As I noted in the previous section, these users can change how they interact with the system over time, finding shortcuts or using the system in ways they are not expected to. The customer establishes usage policies and performs monitoring and auditing tasks that check that people continue to interact with the system in safe and secure ways. The project team sets the customer up to perform this work by documenting what constitutes safe and secure system usage, including the rationale for why some interactions are acceptable and others are not.

Accidents happen. When some loss or injury occurs due to the use of the system, both the customer and the project team have a responsibility and interest to determine why the accident occurred in order to avoid future accidents. The accident investigation may also be mandated by regulation, in which case regulators are involved. The customer may be able to pursue the investigation on their own, if they have sufficient information about how the system is supposed to be used safely. The project assists in the customer’s investigation by providing that information, which includes the documentation of how to use the system safely, and why. However, for serious accidents, the investigation often requires a more in-depth understanding of the system’s design and implementation. The project prepares for supporting these investigations by maintaining complete records about the system’s concept, specification, design, and implementation, including explanations of the rationale for why choices were made and safety or security analyses that the team did about the system’s design.

Finally, the customer may find that their needs change over time, or that there is some aspect of the system that does not work as well as they had planned. These changes can be externally driven; for example, regulatory changes that affect the customer’s industry can affect what the customer needs from the system. The project team can receive change requests (along with problem reports) through a customer service mechanism.

28.6 System operation examples

Operations vary widely depending on the kind of system. Here are some examples illustrating the range.

28.7 System evolution

The system evolution phase is about making changes to the system after it has been released and potentially deployed to customers. These changes can happen for many different reasons—such as a planned roadmap for adding to the system over time, requests for changes from customers, fixing problems, or changes in regulation. System evolution can occur in parallel with system deployment and operation.

Overall, system evolution is a recapitulation of system development (Chapter 27). It involves working out a purpose for the change, a concept for how the system will work when changed, leading to specification, implementation, and verification. These steps use information about what has already been specified and implemented in the system, along with the reasons why it is that way, to work out how to make changes that achieve the desired results without disturbing the system’s existing behaviors.

Making changes starts with a change request. In whatever form the request takes, it identifies who is asking for a change, what their purpose is in the change, and why it is worth doing. In practice change requests are usually maintained in a database. Requests can come from many sources. They may be part of the project’s long-term plan to continue developing the system. They may come from customers, who ask for new or changed capabilities. They may stem from the investigations into reported problems or accidents, in order to avoid problems in the future.

A project does not act on all change requests. Some of them will be technically impossible; some will be infeasible because of time or resources. Others might be reasonable requests that have to wait until higher-priority requests have been addressed. The team looks at each request received to determine its importance, its feasibility, and its cost, and makes decisions about whether to accept or reject the request based on the analysis. If a request is accepted, the team determines a relative priority compared to other work or a potential deadline. These are used in planning the team’s upcoming work (Section 20.5).

Determining whether a request is feasible involves determining how much of the system will be affected by a potential change. While working out the concept for the change, a team member determines what parts of the system will be affected, using documentation about the system’s structure and design (Chapter 12). The result is a preliminary analysis listing the set of components that will be changed and the general nature of those changes. This information is then used to estimate the effort that will be needed to design and implement changes for the request.

Changes happen iteratively. There may be multiple iterations in progress concurrently if multiple changes have been accepted. Handling multiple concurrent iterations requires careful configuration management discipline (Section 17.4).

Making the changes involves changing the specifications and designs for affected components. These changes can be difficult to make accurately because they are done to an existing, complex set of relationships between components. Making a change without causing flaws depends, then, on being able to accurately understand the structure of the system and how parts of that structure contribute to emergent properties like safety constraints. This relies on having rationales, analyses, and earlier designs available, so that people can work from an accurate information base.

Once a change has been specified, designed, and implemented, it is verified. Verifying the work for a change has two parts: ensuring that the modified system meets the new specifications and the purpose of the change request, and ensuring that the rest of the system continues to work correctly.

Once the changes have been verified, they can be deployed to customers as an upgrade or incorporated in new deployments, using the production (Section 28.1) and deployment (Section 28.3) patterns already discussed.

The team continues to evolve the system until the team is relieved of responsibility for fixing problems or when the system is taken out of operation.

The overall process for system evolution includes:

• Receiving change requests. These may include problem analyses, such as root cause analysis or accident investigation reports.
• Triage the requests to determine the importance of addressing each one. Perform a basic analysis of the affected scope and the feasibility of a change.
• Decide on which change requests to address and which to reject, and the relative priorities for requests to be addressed.
• Reiterate the development pattern.
  • Update the purpose and concept to include the changes. Identify what parts of the system will be changed to address this as part of the concept.
  • Modify specifications and designs.
  • Update implementations.
  • Verify the changes. This includes verifying manufacturing changes, training changes, maintenance procedure changes, and similar material.
  • End with acceptance test. This might include a beta test with customers.
• When development is done, reiterate production and deployment. Manufacture the update packages: new hardware components to be deployed, software updates, new procedures and training. Deploy the updates, either by moving in new hardware or applying digital updates, along with training.

28.8 System evolution examples

28.9 System retirement

No system lives forever, and most are deliberately taken out of service when their usefulness has ended.

Most systems continue in operation until there is a decision to retire them. For some systems, this comes when the purpose for the system has been completed—for a spacecraft mission, for example. For others, it comes when the system has worn out enough that ongoing maintenance and repair costs outweigh the cost of replacement, such as for vehicles that wear out. Yet others are replaced because newer systems become available that can meet the customer’s need better.

A system being retired and disposed of typically goes through three periods. In the first period, the system is in normal operation, but the decision has been made to retire it. During this period people plan how to shut the system down and transition its functions or information. They should conduct dress rehearsals to verify that the procedures will work as expected. The system then enters the second period, where it is no longer in normal use but may remain at least partly operational to support transition and archival. Once those are verified complete, the system is shut down for the last time, is dismantled, and its resources are disposed of.

There are two primary aspects of retiring a system to consider: what to do with information or materials that should be migrated to a new system, or archived, and how to dispose of the artifacts that make up the system.

I discussed migrating into a system in Section 28.3 above. The task of migrating out of a system is part of the same process, involving developing a plan for migrating information or materials from the old system and into the new.

There will be other information that people will want long after the system has been retired, in many cases. This can include logs of system activity or user access that may be needed for later accident investigations or legal inquiries. It can also include information or materials that the system processed that are not being migrated to another system, but that may be valuable in the future. This information is moved from the working system to some kind of archive. Developing the procedures to archive the information, how the information will be organized, and the system to hold the archive requires development on its own, just as migrating information from one system to another does. This development phase involves determining what information needs to be archived and how it will be used once it has been stored, which in turn leads to a concept, then specifications, then a design.

Archived information is usually retained for a long term. If a system has been used for business or manufacturing, retention is mostly governed by regulation—anywhere from one to 30 years in the US, depending on the kind of information. Scientific and medical data is often of value indefinitely, though legal retention requirements may be shorter. Scientific data is often re-evaluated decades after it was first gathered; for example, data collected from the Viking landers on Mars in the mid-1970s was re-interpreted thirty years later after other missions gathered more information about Martian soil composition [Navarro-Gonzalez10]. This particular example also illustrates a problem with many data archives: the mission data were recorded on microfilm and had to be scanned to get digital data to process.

Long-term archival media often have two problems. First, the media wear out and decay over time, which has led to information believed to be safely archived to be found to be unreadable [Purdy24]. Second, even if the media are readable, there may no longer be machines that can read them. I have a number of backup tapes for which I have not been able to find a drive to try reading them.

Sometimes physical artifacts are retained from a retired system. It is common to keep parts of aircraft and spacecraft in museums after they are retired, for example.

Disposing of system artifacts can range from trivial to complex. Erasing a software application and its data, for example, is easy; once the storage media have been erased, there is no further meaningful trace of the system remaining. Disposing of a system that processed hazardous biological or chemical materials, on the other hand, can be difficult.

The retirement and disposal procedures must be secure. An unauthorized attempt to shut down a working system can cause major losses, and can lead to safety hazards. Information and materials are being moved around during migration and archival, and are potentially accessible to being copied or corrupted. Physical artifacts that are being decommissioned can carry confidential information about both the way the system works and about the customer that has been using the system.

28.10 System retirement examples

There are many different ways systems are retired. Here are three examples that illustrate different approaches.

Chapter 29: Project ending

When a project ends, there are three objectives: completing obligations and support for stakeholders (Section 16.2); saving information and artifacts that might be needed in the future; and releasing resources that the project used.

Note that ending the project is separate from retiring any particular instance of a system. Ending the project is about stopping development and support for a system product, independent of whether there are instances of that system in operation or not. Some projects will combine these, such as for exploration space missions that build and fly one spacecraft.

A project might end for one of many reasons. It might have a fixed term or have completed a defined system deliverable. It might run out of money or time. It might no longer fit the organization’s or funder’s strategy, perhaps because a better replacement system is planned. Competitors might have won over customers and there is no longer demand for the system. The team might be unable to deliver, with the project behind schedule or over budget or lacking key features.

The first step is a decision to wind down the project. This is typically a decision made by the organization that hosts the project, or its funders; the project staff generally do not make the decision on their own.

A decision to end the project is followed by a plan for how to do so, which defines the steps the team will take to meet the final objectives. The plan typically gets review and approval before proceeding. (In some environments, at least part of the plan must be worked out early in the project, long before any decisions are made.)

The following sections list some of the steps that are involved in ending a project. The specific steps will depend on the project; for example, not all projects have contracts with funders that must be closed out. The team can use this list to help build the plan, bearing in mind that some steps should be done in particular orders. Customers should be notified of the project’s impending end before ending contracts; shutting down production should happen only after all upgraded components have been manufactured.

29.1 Project cancellation

Some projects end because they are canceled, even before they have completed their development phase. Anecdotally, it seems that more projects are canceled than go to completion—this is a consequence of using competitive approaches to programs, and the net effects of competition are generally regarded as valuable. The information in this chapter applies to canceled projects just as to other projects.

Consider two examples, based on projects I have worked on.

In the first project, the team was writing a proposal for a US DoD spacecraft system. In the proposal-writing phase, the team has to establish the basic architectural and management approaches for the project, show they meet the department’s needs, and establish the price at which the team proposes to build the system. The team progressed through establishing the initial concept and architecture for the system, and we began evaluating the solution to see how good a job it would do for the customer and how much it would cost to build it.

We had a checkpoint milestone where we reviewed what we had found. At that review, it became clear that while our team had a decent solution for the needs, we did not have a great solution, and that other companies we expected to propose designs would likely have better solutions (because they had more experience in a couple of key technical areas). We made the decision not to pursue the proposal.

This was a good decision. Assembling a proposal is not a small task; we had a team of about 15 people working long hours. For US government projects, the proposer generally pays for the proposal development. Choosing to spend our team’s time and money on this project meant that the team couldn’t work on some other project. We judged that the opportunity cost was not matched by the probability of successfully winning the contract, so we freed up the team to work on a different system that did prove successful. If we had continued to work on the original proposal, we would have spent the budget available to develop proposals and could not have spent it on the proposal that succeeded.

In the second example, a different US DoD spacecraft program, the team was about two years into a multi-year contract. The team had performed excellently in a competitive first prototyping phase, and was the only team to be selected to move on to a second phase for building an initial working version. A key subcontractor on the team had staffing and management problems, and were not delivering results. Within the team we were struggling to fix the execution problems or find another way to build the necessary components, all the time keeping a large staff on payroll and running through budget. While the technological solutions for many system capabilities were probably sound, the team could not deliver. The customer observed the problem, and after working with the team to try to resolve the problems, went through the process to cancel the project.

This was also a good decision. In hindsight, the team lacked necessary capability in the subcontractor and in the project management team. If the project had been allowed to continue, it is unlikely that the team would have solved the problem and more money would have been spent without benefit in the end.

The take away from these examples is that there are many sound reasons for canceling a project. Sometimes the cancellation is designed in (as with competitive acquisition); other times it is because continuing to invest money, time, and the care of the team building the system has become unlikely to pay off.

For a more general discussion of US DoD project failures, see the report by Bogan et al. [Bogan17].

Chapter 30: Using the reference life cycle

The last several chapters have presented a reference life cycle pattern. This pattern is intended to inspire thoughts about how a project can organize its own work. It is not in itself ready to use off the shelf. Each project will have its own needs, its people will have preferred ways to work, and some projects will have life cycles mandated by regulation or industry standards to follow.

XXX purpose of the life cycle – get a system built, deployed, and sustained that meets customer and other stakeholder needs – in doing so, develops the whole system including all its development artifacts, not just the end product – plea for thinking about flexibility with discipline, and the idea of the artifact edifice

The reference pattern does not discuss the roles involved. A full definition of each phase will include definitions of who performs different tasks in each phase, and in particular who is responsible for milestones. I argue elsewhere (e.g. Section 8.2.6) that, to be meaningful, reviews must be done by people with an independent perspective on the material being reviewed, and that approvals must reflect a check on the work fitting into the project’s big picture.

The objective for any project is to develop and adopt a life cycle that meets its needs. In the next section I discuss several principles that a good life cycle will follow; these can help people evaluate a life cycle they are considering. Some other considerations include:

• Is this project intended to deliver a system once, perhaps with upgrades and fixes after delivery, or will it evolve the system through multiple releases for a long time? A system that is expected to have multiple releases needs more thinking about how to sustain the development over time and how to support customers at the same time as developing new releases.
• What make-or-break events are foreseen during the project? These include things like getting funding, aircraft type certification, or spacecraft launch. The life cycle should include these milestones explicitly, and include the work to be done to prepare for them. In many cases these will interrupt what might otherwise be a continuous development flow. (See Section 30.2 below for examples.)
• What regulatory constraints are there on the project or the system? Are there work products or milestones needed to satisfy regulators?
• What development methodology will the project use?
• Is the system intended for a single customer, or is it meant to be a product marketed to multiple customers? If there is a single customer, they can be included directly in the life cycle—especially if the customer is internal. If there are to be multiple customers, a marketing team may have to stand in for customers, and they may need time to investigate particular market segments before they can accurately reflect potential customers.

The project works out what life cycle patterns it uses, and documents the patterns. This effort starts during project preparation (Chapter 25). It does not necessarily need to define the entire life cycle all at once; it can be done iteratively, as long as the work keeps ahead of what the team needs. In practice I have found that enough should be completed in the project preparation phase that the team understands the general complexity of the work ahead, has chosen a development methodology, and can name major milestones they will need to meet. The remainder of the life cycle can be worked out during purpose and concept development and likely will be refined or adjusted as the project moves along. (I worked on one project that had a limited budget, and spent much of that budget writing elaborate management and engineering plan documents before even beginning to work out the high-level system concept. The result was a pile of such documents that were never looked at again, which was a waste of their efforts.)

In no case should the team get ahead of the defined life cycle. See Section 8.1.5—Principle: Team habits for a discussion of this principle.

The life cycle patterns have value only if the team actually uses them. This means that the team must know that the patterns exist, understand them, and agree that they are useful. The people in the team must also understand that they have a responsibility to follow the patterns, or to raise an issue when they find a problem with the life cycle’s definition. Achieving these means educating people as they join the team about what the life cycle is and how to learn about it, as well as monitoring that everyone actually follows the patterns. The team can also learn about and accept a life cycle a bit more easily if they are involved in developing the patterns; at minimum, they should be able to give feedback before the patterns are adopted.

The life cycle patterns are documented in a way that the team can find them and learn about them when they are joining the team and when they need to refresh their understanding of how some step works. The documentation is an artifact that should be managed using the principles in Section 17.4: it should be versioned and under change management; it should be stored in a way that the team can find it when needed; and it should be secure enough that it will not be tampered with.

There is no one right way to document them, as long as the documentation is well-organized and accessible. Some organizations prefer to define the life cycle in a prose plan document, which can be printed in its entirety if needed. I have had some success maintaining the documentation in a wiki or in a collection of web documents; the advantage of these is that they allow linking between parts of the document. The patterns should be explained and listed explicitly; they should not be hidden in a workflow system that doesn’t let team members see and understand the whole context for their work (see Section 4.6 for an example).

The documentation for each phase or step in the life cycle should include the information listed in Sections 21.5, 21.6, and 21.7.

30.1 Meeting life cycle principles

In Section 21.10, I listed principles that a life cycle pattern should meet. The reference life cycle pattern in this part reflects these principles, though it cannot address all of them. Here are ways that a life cycle built using this reference as a base can address them.

30.2 Relation to NASA life cycle

As many people are familiar with the NASA life cycle, or may be obliged to use it (or a variant of it), in this section I discuss how the canonical NASA life cycle compares to this reference life cycle. I will use the general NASA life cycle defined in NPR 7120.5 [NPR7120, Figure 2-5]. I presented an overview of this life cycle in Section 23.2.1.

The NASA life cycle is divided into seven major phases:

1. Pre-Phase A: concept studies that define a potential mission.
2. Phase A: concept development that results in the definition of a feasible and useful mission.
3. Phase B: high-level design of the system for a mission and evaluation of the technology available for the mission.
4. Phase C: the bulk of development, in which the system components are developed, verified, and manufactured.
5. Phase D: assembly, integration, and test of the flight and ground systems; launch and initial on-orbit checkout.
6. Phase E: operation in flight.
7. Phase F: close out, including flight system disposal.

This life cycle was developed over several decades as NASA learned how to develop and operate complex missions. Elements of this approach have been adopted by many other organizations—terms like “System Requirements Review” and “Preliminary Design Review” have become nearly ubiquitous in the aerospace industry.

The overall flow of the NASA life cycle is organized around two constraints: fitting in with the US Federal funding cycle, and managing risk for a few highly expensive steps. The funding constraints come at the transition from Pre-phase A to Phase A, when the mission is approved and funded enough to develop its concept, and between Phases B and C when the agency commits to funding the full mission [NASA16, Section 3.5, p. 25]. The distinction between Phases C and D comes with Phase C covering development of designs and fabricating components, but actual assembly of a spacecraft does not start until Phase D, at which point there should be little residual risk that the system design will not work out.

The NASA approach was, however, developed for hardware-heavy systems and people who today develop spacecraft or aircraft that have a greater amount of software components sometimes find it difficult to map software project best practices onto the NASA approach. There are usually two issues: software development best practice puts integration earlier than the way many people interpret the NASA model; and many software developers combine design and implementation, especially for novel software functions. I show one way to reconcile these approaches in the mapping in this section.

The reference life cycle I have presented is organized around types of work—conception, specification, design, and so on. The NASA life cycle is organized at the highest level around milestones that check progress early, allowing corrections before committing agency resources. This means that the NASA life cycle splits several of the early phases in the reference life cycle in two, with a major review or checkpoint of the project’s progress before continuing. These two approaches are compatible: almost every project will have some kind of project-wide milestones alongside the milestones specific to the work phases.

In the following, I present how each of the NASA phases maps to the reference life cycle.

30.2.1 Outside the NASA life cycle

The reference life cycle defines the project preparation phase and project support “phase”. The preparation phase involves a rough definition of the project and establishing basic operations abilities. Project support covers support functions, like managing teams, finances, or artifacts.

In the NASA environment, the initial support is provided by one or more agency centers and external collaborators, using budget, tools, space, and people for general concept exploration. Each center has its own procedures for starting up a concept exploration project.

Similarly, the NASA agency provides essential support services to its projects.

In one project I worked on, the NASA Ames Research Center had a Mission Design Center that was charged with exploring potential mission concepts. A small group developed the mission idea and explored ways it could be realized. Ames and the agency provided all the key support infrastructure: staffing, finance, office and lab space, and IT services, for example.

30.2.2 Pre-phase A—Concept studies

The Pre-phase A work develops a concept for a mission, presumably in response to NASA agency priorities. It is expected to limit its work to the concept of a mission: what it might achieve, who would benefit from the mission, and high-level technical approaches that might support such a mission.

There is one major review in this phase: the Mission Concept Review (MCR). This checks that the potential mission is well formulated and that there is sufficient interest to justify funding “project formulation”—working out a detailed concept and high-level design.

At the end of Pre-phase A, after the MCR, the agency makes a decision whether to continue the project and fund it for “formulation”: the phases where the concept and high-level designs are worked out. This involves greater financial commitment than the early studies, and is the start of the “real” mission.

Pre-phase A maps to the purpose development phase (Section 27.3) and part of the concept development phase (Section 27.4). The purpose development phase covers identifying what the mission might do, and who the mission stakeholders might be. The concept development produces an initial sketch of a mission concept, without breaking the concept down into great detail.

30.2.3 Phase A—Concept and technical development

This phase is the first of two that are about developing a feasible high level design for a mission and ensuring that necessary technologies are available. Phase A includes developing a complete mission concept and high-level system designs. The team identifies any new technology that the mission will require and works out what will be needed for it to be ready to use in flight.

The depth of design and requirements is not clearly specified in the NASA procedural documents. However, my experience is that it is generally taken to include the spacecraft and its major subsystems, ground systems and their major subsystems, potential launch vehicles, and testing and other ground support equipment to a similar level. The exercise is intended in part to develop the general structure of the system and its likely cost, and in part to find those parts of the system that will require new technology.

Phase A includes developing a list of new technology that will be used for the mission, an evaluation of its maturity, and plans to develop that technology so that it will be mature enough for flight.

This is the first phase where a NASA project is funded for itself, as opposed to using resources allocated for general mission concept development. The various management and development plans required by NASA procedures get developed in this phase.

Phase A includes two key reviews:

• System Requirements Review (SRR): checks that high-level requirements for the mission will support the intended mission, and that they are complete and consistent.
• Mission or System Definition Review (MDR or SDR): checks the overall structure proposed for the system, and how requirements flow to major subsystems. Checks that initial measures of the design are reasonable. Also checks that management plans are in place.

The NASA Phase A maps to the second part of the concept development phase in the reference life cycle, along with concept and specification and preliminary design steps for the highest-level components in the system.

30.2.4 Phase B—Preliminary design and technology completion

Phase B continues the work from Phase A, completing a preliminary design and refining any new technology to the point where it is sufficiently mature to use in flight. This often involves building models and prototypes of parts of the system.

Phase B also involves developing the safety and security of the mission. The high-level design should incorporate designs for safety, security, and other critical mission success factors, and the design should be backed up by analysis showing why the design is sufficient. (See Chapter 43 for more on safety design.)

At the end of Phase B, the project should have a high-level design for the entire mission. That design should meet all the mission objectives, be technically feasible, and fit within cost and schedule available.

After Phase B, the agency allocates money to actually implement the system. The process can be complex and time-consuming, potentially involving legislative approval. The estimates for cost and schedule should be accurate enough that the project is unlikely to exceed them, which would require repeating the process to find more funding or time. This imposes limits on how much risk the project can carry going from Phase B to Phase C.

There is one key review in Phase B:

• Preliminary Design Review (PDR): checks that the high-level system design is complete and meets mission objectives, that any new technology needed is ready, and that safety and security requirements are met by the design.

The end of Phase B maps to a slice across the development phase in the reference life cycle. It includes the concept, specification, and preliminary design of the first two or three levels of components in the breakdown hierarchy (Section 11.3; Chapter 38). In general this might include the major spacecraft subsystems—payload, structure, propulsion, attitude control, and so on. The portion of the design step includes prototyping or modeling of components that pose technical risk, and the design may go to deeper levels of the breakdown hierarchy if needed to understand and address that risk.

The mission-level PDR follows reviews of the component-level preliminary designs.

30.2.5 Phase C—Final design and fabrication

This phase is when most of the development and production work is done. It involves designing, building, and verifying all the components in the system, to the point where they are ready to be assembled into the working spacecraft and ground systems.

Phase C is designed around the spacecraft being difficult and expensive to assemble, involving building large structures, using complex manufacturing tools, threading complex wiring harnesses through the structure, and putting large amounts of money at risk during the assembly. This leads to organizing the final assembly work to avoid as much risk as possible by ensuring that all the components are ready to assemble before committing to the final assembly steps.

During this phase, the team completes all of the designs and implementations of the system components, and verifies all of them. This usually includes producing engineering and qualification units of hardware components (Section 27.8) for testing, including destructive testing for some parts. It also usually includes integrating all of the engineering or qualification units and the corresponding software into a testing version of the entire spacecraft in order to verify the entire integrated system.

Verification in Phase C typically includes verifying the human interfaces in the system. Can an operations team use the ground systems to accurately control the spacecraft, using simulated telemetry showing the spacecraft in different conditions (including off-nominal conditions).

Phase C is typically divided into two parts: the first part for completing all the designs, and the second part for implementing and producing the components. The Critical Design Review separates the two parts, where all the designs are checked.

I have seen the Critical Design Review milestone cause confusion: how far should work progress before the CDR? What is the boundary between “design” and “implementation”? For hardware components, such as an electronics board, engineers work on the board design: the layout of the components and traces that will be fabricated. The NASA CDR definition ([NPR7123, Table G-7, pp. 113-4]) indicates that the CDR should include “integrated schematics” and “fabrication, assembly, integration, and test plans”, which would indicate that the board design is complete. That the document also indicates that the CDR and Production Readiness Review are often coupled lends credence to the interpretation that the CDR reviews the board designs.

If this same interpretation were applied to software, it would imply that the software would be essentially complete by CDR. Software source code is the equivalent of electronics board design: while it is thought of as implementation, it must be processed through a build system to produce the actual executable software, just as a board’s design file is used to manufacture the boards.

However, the NASA Systems Engineering Handbook states that the CDR for a software component should occur “prior to the start of coding of deliverable software products” [NASA16, Section 3.6, p. 29]. In other words, the documents appear to disagree, though NPR 7123.1 is presumed to have precedence.

Further, software is often developed iteratively, implementing one version after another, each version adding some amount of functionality over time. Some lower-level functionality is left as a mockup, perhaps not even fully designed, until some of the higher-level integrated functionality has been implemented and verified (the idea of integration-first development (! unknown reference XXX), done to reduce risk as quickly as possible). Software development best practice also has verification proceeding continuously throughout implementation, with feedback to the implementer as early as possible. This often implies having some of the hardware components built and available for testing the software before the software is completed.

An official answer to how a team should resolve the discrepancies and interpret the CDR for a NASA project will have to come from the relevant NASA authorities.

However, in practice, I have found that focusing on the review before implementation is more useful for components in the upper and middle levels of the breakdown hierarchy. For example, this might include the major components with a subsystem, such as power distribution or generation within the electrical power subsystem, or attitude control algorithms in the guidance, navigation, and control subsystem. Components at these levels realize the important relationships between components in the system structure (Section 12.2) and the way components work together to produce emergent properties (Section 12.4). Analyzing these designs allows one to check whether key system behaviors will be met, and that properties like safety or reliability are handled correctly. These are the properties that are difficult to change if the implementation is found during verification not to meet them. The design and implementation of low-level components should be reviewed, but as long as there is an obvious, low-risk approach for them their review need not block the design reviews of the system as a whole. This interpretation, of performing the critical design review before implementation, means that the team is then free to implement software components incrementally if that is the best approach for that part of the system.

There are three reviews in Phase C:

• Critical Design Review (CDR): a set of reviews of all the specifications and designs for the system, from the lowest-level component to the system as a whole. If a component’s design passes this review, it is ready to be produced. Each major component in the system—both flight and ground systems—has its own CDR, working from the bottom up until finally there is a CDR for the system as a whole.
• Production Readiness Review (PRR): for components that will be produced in some quantity (typically three or more), checking that the production procedures, tooling, and people are ready to build them. This is sometimes conducted alongside the CDR for the component.
• System Integration Review (SIR): checking that all of the components are on track to be assembled into the system, and that the procedures, tools, and people needed for assembly are ready. “Integration” in this review means “integration into the final assembly”, and not the more abstract integration that occurs during development and verification.

The NASA Phase C maps to completing the development phase, the acceptance phase (Section 27.9), and the system production phase (Section 28.1) in the reference life cycle.

The CDR milestone maps to a slice through the system and component development phases, at the end of the design step for most or all of the components. The PRR for a component is equivalent to a review at the end of the production unit development step (Section 27.8). Note that the reference life cycle has a manufacture and deployment check milestone in the acceptance phase; this applies when the entire system is manufactured together, rather than the model implied in the NASA life cycle where different hardware components go to production individually. Finally, the SIR is equivalent to the deployment readiness review that is at the beginning of the deployment phase (Section 28.3) in the reference life cycle.

30.2.6 Phase D—System assembly, integration, and test, launch and checkout

Phase D covers the work between the end of designing and building all the parts and having a spacecraft on orbit ready to begin its mission proper. This includes assembling the spacecraft and ground systems, and verifying that they work (and work together). The verification typically involves testing the assembled spacecraft in vacuum, under strong vibrations, and in thermal environments equivalent to what it is expected to handle in flight—but not testing beyond those levels, in ways that might damage the vehicle. After testing, the team proceeds onward to integrating the spacecraft with its launch vehicle, final preparations, launch, and starting operations on orbit. The team on the ground finally checks the spacecraft out before declaring it ready to begin its mission.

Some missions build a second copy of the spacecraft to be used on the ground for debugging issues with the one in flight and to test possible commands before sending them to the operational spacecraft. The duplicate is typically assembled in Phase D. It might use qualification units for hardware that were used for testing in Phase C, rather than flight-ready units.

There are several reviews in Phase D. All of them are final checks that some part of the mission is ready for taking an irrevocable step. These include:

• Operational Readiness Review (ORR): that mission operations is ready for the flight to start.
• Mission or Flight Readiness Review (MRR or FRR): that the flight parts of the mission are ready to go.
• Launch Readiness Review (LRR): that the launch vehicle and supporting systems, including the site and launch range, are ready.

The NASA Phase D maps directly to the deployment phase in the reference life cycle. It takes in manufactured components and procedures, assembles them into a working system, tests that it has been assembled properly, and starts it in operation. The milestones in the NASA Phase D are different from the deployment phase milestones mainly because they are specific to launching a spacecraft.

30.2.7 Phase E—Operations and sustainment

In this phase, the team operates the mission through its end.

There are two kinds of reviews that occur in Phase E:

• Critical Event Readiness Review (CERR): reviews before specific mission events where something could go wrong. This might be adjusting an orbit, or uploading software updates.
• Decommissioning Review (DR): reviewing the decision to end the mission.

Phase E is equivalent to the system operation (Section 28.5) and evolution (Section 28.7) phases in the reference life cycle. The Decommissioning Review is equivalent to the decision to retire the system at the beginning of the system retirement phase (Section 28.9).

30.2.8 Phase F—Close out

The final phase in the NASA life cycle involves retiring and disposing of the flight systems, retiring or releasing ground systems, archiving mission data, and closing out the project.

There is one review identified in the NASA life cycle:

• Disposal Readiness Review (DRR): determining that the mission is ready to command the disposal of the spacecraft.

This phase corresponds to the system retirement (Section 28.9) and project ending (Chapter 29) phases in the reference life cycle.

Bibliography