Making systems—3: Development
IX   Design
Chapter 46   Safety and security design

46.1 Introduction

This chapter covers how one goes about designing safety and security into a system.

Almost every system I have worked on has been designed, at least in part, with the assumption that the system is designed correctly as long as it performs the functions people want, while people use the system as expected.

This assumption is wrong, and the systems built this way are not designed correctly.

A correct system design is one that performs its desired behaviors and that does not perform other undesired behaviors. This second part is just as important as the first.

The process of designing a system, as I have discussed in previous chapters, involves determining the system’s purpose in terms of the needs the system should meet, and the functions by which it can do so. Defining what the system should not do is equally important.

Safety and security are two of the common labels for things that a system should not do. They are two different aspects of the same concept, as I will discuss. Both involve defining harms that the system should not cause. A harm is the mirror image of a need: it is something that stakeholders want for the system not to do.

In this chapter, I discuss what safety and security design is and the general approach for achieving it. This is only an overview, which illustrates how this aspect of design is related to the overall work of building a system. I strongly recommend that those who actually design or analyze system safety—which includes most of the people working on systems projects—pursue this topic with more specialized sources.

46.1.1 Principles

There are three basic principles for saftey or security to bear in mind.

  1. Everything must be presumed unsafe or insecure until shown otherwise.
  2. If an unanticipated accident or breach happens, that still counts as unsafe or insecure system operation.
  3. Safety and security are always specific to the system. Generic mechanisms may be building blocks but they are not in themselves what makes a system safe or secure.

I will discuss these principles throughout this chapter.

46.2 Why

Complex systems are usually high-value systems; otherwise why go to the effort to make them? These systems often also have real-world consequences if there are problems with them, whether malfunction or just outages. The liabilities, both economic and moral, can be large when such systems misbehave.

This implies that the systems must minimize how and how often they misbehave. This does not happen accidentally; it requires deliberate design.

The least costly time to address how to keep a system from misbehaving is early in the project. The most expensive is after a system has been implemented, deployed, and actually malfunctions: not only is there the cost of the malfunction, but the pressure to address the problem is high (making it difficult to make careful design choices) and the effects of making changes are hard to fully work out.

Safety and security are not isolated parts of the system-building work. They are woven into every step of the work, from understanding a system’s purpose through design and implementation through deployment and operation.

46.3 What safety and security are

Safety and security are about ensuring that the use of the system does not lead to harms. The process starts with stakeholder policy or needs. The system is safe or secure enough if the harms that do occur match the policy.

undisplayed image

The process of building safety and security into a system starts with defining the harms that the system must avoid, defining the hazards that can lead to those harms, and then designing features and behaviors in the system to eliminate, reduce, or mitigate the hazards. The system will then be deployed and used. From time to time the usage will result in harms.

Put another way, safety and security are dynamic properties of the use of the system. The design and implementation can lead to safe operation, but they are not safe in themselves—the design and implementation are static elements that only have effects when used.

It is important to keep safety or security policy and design separate. What the harms are is policy. They are defined externally and are generally not within the team’s authority to define or change. The team is responsible for how the system meets this policy by appropriate design and implementation.

In general terms, a harm is an anti-need. The needs a system addresses are things that will produce value for stakeholders. Harms are things that will reduce value for stakeholders. Colloquially, a harm is something that people want not to happen.

The common conception is that safety is about avoiding injury, to people, other living things, or property. Security is about avoiding disclosure of or damage to information. These are all valid harms, but there are usually many more that must be considered.

Many harms are defined by external policy or regulation. In aviation, for example, there are defined maximum acceptable rates for collisions between aircraft, or accidents causing human injury or death. For spacecraft in Earth orbit, there are regulations related to collisions on orbit and damage caused by re-entering debris. Financial transactions must comply with security standards to protect payment information, in order to avoid the harm of fraudulent transactions.

Some of those policies or regulation focus on the means for protection as opposed to the harms to be prevented. However, the harms themselves are usually referenced or can be deduced from the mechanisms.

Stakeholders are usually concerned about more than just the harms defined by external policy. For example, many aviation safety regulations are concerned with human injury or death, but an aircraft manufacturer is concerned about the operational cost of damaged aircraft, or the reputational harm that comes from an accident even if no one is injured.

Other harms can only be defined by stakeholders. The value of intellectual property, for example, varies a lot and each stakeholder may have to define their specific needs. The kinds of property harm that matter depends on the kind of system and where it will be used. For example, one recent system I worked on defined five different kinds of information with different levels of confidentiality needs for each, ranging from publicly visible to restrictions on which parties could know about it.

Once the harms are defined, the next step is to use that information in just the same way other stakeholder needs are used to build a good system. Safety and security affect the system concept; they are incorporated into system and component specifications, which lead in turn to aspects of designs; safety and security properties are verified as the system is implemented.

Neither the system design nor its operation will be perfect for all time. The design can and will miss hazards. Mitigations may not work as expected. Some hazard that was considered unimportant may turn out to be important. New harms or threats will emerge. Entropy will have its effects on the parts, both physical, conceptual, and human.

A system might have an excellent design, but result in harms because of problems with how it is used:

This means that a safe or secure system must include continuous learning, monitoring, fixing, and evolving. A safe system must include ways that it will catch and correct situations where its usage, maintenance, or management are drifting out of the range that will keep the system working safely. It also must include ways to update the system to address the design or implementation problems that will be found.

46.4 What they are not

Safety and security are not about jumping in during design or implementation and checking for certain classes of problems. I was once in a meeting at a major aircraft manufacturer where their staff, who had recently worked on one of their new major passenger aircraft, defined security as performing certain software checks on their flight systems. (Two days later I was flying home on one of those aircraft and was not reassured.)

Addressing security is not about adding an authentication and encryption protocol to a software system and declaring the system secure (though those can be a part of making a secure system).

Safety is not about just training operating staff on how to use part of a system, though again that can be part of a safety solution.

Safety and security are properties of a whole system. They are emergent properties that only happen when many different parts of the system are deliberately designed to ensure that the parts combine together properly. Using a security mechanism in one part, such as encrypting network communication, may be one piece of a solution but is not the entire solution. (This is part of the third principle I listed above.)

A safe or secure system requires developing evidence that the behaviors and features designed into the system are sufficient to avoid a well-defined set of harms. Claims based on one design aspect, such as data encryption, provide no evidence on their own that the system is secure; only the proper analysis that connects the mechanisms and the total set of harms is meaningful.

Sidebar: Safety versus security versus assurance

Safety and security are largely similar: both are about defining possible harms and avoiding them.

However:

  • Safety is about inadvertent accidents, where the harm is caused either by events within the system or unintentional external events.
  • Security is about harms caused by malicious actions by an intelligent actor (the attacker), whether that actor is inside or outside the system.
  • Security must assume that some attackers are sophisticated (but not omnipotent).
  • Safety can often address harms that can occur repeatedly, so that one can define a safety objective to keep rates of some harm occurring below some rate.
  • Security, and other kinds of security, generally addresses harms that are sufficiently serious that even one occurrence is too many.

In many cases it is difficult to distinguish whether some hazardous situation is a safety matter, a security matter, or a combination. While there are differences between how one analyzes and defends against a security problem and a safety problem, the two often end up being combined.

Both safety and security are system properties that require assurance. Systems can have other similar properties that could require similar arguments and evidence. A system might have objectives for availability or real-time performance that could be addressed in ways similar to how safety or security are addressed.

46.5 Objectives of safety and security analysis and design

When analyzing and designing a system for safety or security, there are a few objectives to the effort.

It is not good enough to design and build a system that happens to meet safety or security requirements. There must be evidence backing up claims that the system meets those requirements.

A great many systems have been built that were claimed to be safe that turned out not to be. Consider, for example, the many “fireproof” buildings that were built in the US in the late 19th and early 20th century. They were built using non-flammable steel, brick, and stone, and had a number of feature to prevent fires. Some of them turned out to be susceptible to fire after all, such as the San Francisco Call Building that burned in the 1906 earthquake [Klages93]. A claim is therefore not enough; a safe system needs solid evidence showing why and how it is safe.

Safety and security analysis is the process of determining the harms the system must address, working out how they could possibly occur, and then determining whether the system will adequately prevent those situations from happening. If the system doesn’t, the analysis provides the basis for changing the design to address the problems. When the analyses find that the system does address the situations well enough, the analyses provide the evidence that the design or implementation is acceptable. The analysis and design process usually iterates many times before the design is good enough.

The system will need to be updated from time to time. People will find flaws, where the system doesn’t meet its security or safety objectives. New harms, new causes, or new threats will develop in the world. When it is time to update the system, people who did not work on the original system design may be doing the work, or enough time may have passed that people don’t remember the details of how safety and security is designed into the system—especially the subtle details. Providing these people with a record of why and how the system is safe will help them make changes without breaking safety. If there has been an accident, this record will help them track down what didn’t work as designed and provide them a foundation on which to design and implement improvements.

Sidebar: Safety and security terms

Harm: A kind of undesired effect; a kind of loss.

Accident: An undesired and unplanned event that results in harm or loss. (Some standards, such as ICAO Annex 13, use the term incident as the general term [ICAO20].)

Hazard: A combination of system state or condition and environment conditions that will lead to an accident. (Also: threat.)

Risk: An evaluation of the seriousness or importance of a hazard, based on its likelihood of leading to an accident and severity of the harm caused.

Mitigation: A feature of a system design that addresses one or more hazards, reducing risk compared to omitting this mitigation.

Actor: Some entity that can participate in a hazard or an accident.

Attacker: An actor that can create a hazard intentionally. (Also: threat actor.)

46.6 Analysis process

Stripping away the details of specific methodologies, safety or security analysis and design has several steps:

  1. Identify harms.
  2. Define accidents based on harms.
  3. Identify the threat actors that could cause those accidents, if appropriate.
  4. Identify the hazards that can lead to those accidents.
  5. Prioritize which hazards to address.
  6. Work out how to address these hazards.
  7. Update design and specifications.
  8. Repeat the last few steps until all the important hazards are addressed.
undisplayed image

A thorough analysis can define a vast number of harms or hazards. In practice it is not worthwhile to address all of them. It is not generally worthwhile, for example, to consider the harms to an ordinary financial transaction system that would be caused by an asteroid strike on the Earth. The number of hazards is also usually far larger than what the team can address all at once. The team needs to prioritize which hazards to address, in what order, and which hazards are not worth addressing.

46.6.1 Hierarchical application

This process is repeated at multiple levels of the system.

Stakeholders’ security and safety objectives, and the associated harms, apply to the system as a whole. In the end, what matters is that these whole-system objectives can be shown to be met.

The safety or security properties, however, emerge from the behaviors of the components that make up the system. Each component will have its behaviors, and the safety analysis must show that these behaviors combine to address the hazards that have been identified. (Chapter 12 discusses the structure of components and emergent properties.)

undisplayed image

This argument depends in turn on each of those components accurately performing the behaviors they have been assigned. Failing to perform one of those behaviors is then a harm at the level of that component. The process of identifying the accidents that could produce that harm, that is, how the component could fail to behave as needed, leads to identifying component-level hazards that are then addressed and mitigated.

This approach to decomposing safety or security follows the same pattern as decomposing the system itself into components, then into subcomponents, and on recursively. (Chapters 11 and 41 discuss this hierarchical structure.)

As an example, consider air traffic management for UAS (drones) in Figure 46.1. Some of the harms to be avoided include damaging a UAS, damaging another aircraft, damaging property on the ground, or injuring people on the ground. (There are many other harms to be considered in the real systems, but this is adequate for the example.)

undisplayed image
Figure 46.1: Harms and hazards in a UTM system

These harms can occur in several kinds of accidents. These might include:

Considering the accident of two UASes colliding, there are several corresponding hazards:

To address the first hazard, UAS operators use a shared UAS traffic management (UTM) system (Figure 46.2). The UTM system is a component of the overall airspace system, along with the UASes, their operators, and other aircraft. The operators submit a flight plan to the UTM system before their flight, and the UTM system checks for potential collisions. The UTM system will allow plans that don’t collide, and will deny plans that might.

undisplayed image
Figure 46.2: UAS safety as emergent property of airspace components

The overall argument for avoiding harm, therefore, depends in part on the UAS operators filing accurate plans, the UTM system performing these checks properly, and the UAS then following the plans. If the UTM system does not correctly check the plans it receives, then UASes could collide and result in the first accident listed above.

The safety analysis and design process repeats for the UTM system, as a component of the overall airspace. The harm that the UTM system could cause is to approve two UAS flight plans even though they lead to collision. One can define accidents specific to the UTM system, work out the conditions when they can occur, and then define mitigations in how the components that make up the UTM system behave. The process can then recurse down to those components, and repeat until the potential hazards have been addressed.

46.6.2 Adversarial mindset

Before going into details, it is worth discussing the attitude that one must bring to the task of designing and analyzing safety or security.

Analyzing safety is about looking for any and every way that the system could cause harm. This is an uncomfortable and unfamiliar mindset for many people who are used to building systems, who are usually looking to show that what they are designing or implementing performs the functions it is supposed to, that is, for the presence of desired functions.

Safety and security are about ensuring the absence of harms when the system is used.

This is related to the second principle above. A harm that happens but was unanticipated is still a harm.

One of the main ways to develop this evidence is to find all the scenarios where a harm could happen and trace out what could cause those scenarios. This requires being thorough and imaginative about ways things could go wrong.

It starts with identifying all of the steps and dependencies that the desired behavior takes—most importantly, identifying ones that people usually don’t think about. On several occasions, for example, I have worked on safety in network-connected systems. The designers, reasonably, thought about how networks function most of the time. But basic network communication depends on many, many layers of systems working correctly, such as the many routing mechanisms that direct packets through the network, or the name system that maps system-meaningful names to network addresses. The designers did not consider how the Internet’s structure lends itself to common-mode failures where large portions of the network become disconnected. (A later example discusses another example related to vehicle control.) The exercise of identifying all of these dependencies is usually an exercise in finding these things that one usually takes for granted.

Completeness matters for this exercise. For security, leaving even one way to attack a system is enough to let someone compromise it. For safety, eliminating one hazard may reduce the number of number of accidents but even one accident that injures or kills many people is too much most of the time. This puts a premium on meticulous analysis to find all these ways.

Safety and security analysis is an exercise in identifying and countering confirmation bias (Section 59.2). People are generally looking for evidence that the system works, not that it does not do things it is not supposed to. Many times I have encountered the argument that some situation “cannot happen because…”. This is a common sign that this situation really is worth considering. What is the evidence that the situation really cannot happen? What if it could?

This is where first principle I listed above comes from: that everything must be presumed unsafe or insecure until there is good evidence otherwise.

In the Inquiry Board report for the failure of the first Ariane 5 launch, the committee wrote [ESA96, p. 6]:

An underlying theme in the development of Ariane 5 is the bias toward mitigation of random failure. […] The exception which occurred was not due to a random failure but a design error. The exception was detected, but inappropriately handled because the view had been taken that software should be considered correct until it is shown to be at fault. […] The Board is in favour of the opposite view, that software should be assumed to be faulty until applying the currently accepted best practice methods can demonstrate that it is correct.

Making a system safe or secure also requires focusing on more than “failure”. Many unsafe situations occur when there is no failure in a system, when every component functions exactly as designed—but the design itself was wrong. Incidents that occur when there is no failure in operation are often the hardest to diagnose and fix.

I will discuss methodologies for how to organize the analysis and design activities so that they can be as complete and accurate as possible (Section 46.10.1).

46.7 Identify harms

Identifying harms is listing the harms that the system should avoid causing. Some of these harms are defined by regulation or by organizational policies, as noted earlier.

In most cases, defining the list of harms is not a technical decision that is within the authority of the team designing the system. Decisions about which harms to include in safety or security is fundamentally a business or social policy decision. It is part of how an organization determines it should position its products compared to other products, and how it should respond to societal expectations. I have found it difficult to get company leadership to take responsibility for these decisions; in multiple cases the leadership was interested in questions of market fit and profitability, but did not want to be bothered with setting company policy about product safety or security. Be prepared for this to be a difficult conversation inside some companies.

A harm has two parts: something that can be affected (a subject) and what should not happen to them. The subject can be people, animals, equipment, property, environment, reputation, or anything else seen as worth some kind of protection. The effect can be anything that would be seen as damaging the subject: injury to people or animals, damage to equipment, theft of property, loss of reputation, and so on.

harm = subject + effect on subject

The majority of harms are defined implicitly by the system’s purpose. For example, in one recent system for air traffic management the primary purpose was to “deconflict drone flights so that no two drones would be operating in the same airspace volume at any given time.” The system provided deconfliction in order to minimize the number of mid-air collisions between drones. The harm for this system was therefore having “two or more drones authorized to operate in overlapping airspace volumes.” This harm was not injury or damage; it was an abstract condition. The larger airspace system had safety objectives of avoiding harm to other aircraft or people on the ground, and the role of the traffic management system was to avoid a more specific harm.

In another system, related to autonomous road vehicles, there were several harms: injury or death to people, property damage outside the vehicle, damage to the vehicle, damage to property in the vehicle, and interfering with road usage.

Harms are usually ranked in order of seriousness. Injury or death of a person is typically more serious than a scratch on a vehicle, for example. Having some understanding of the relative seriousness of two harms becomes important when one must choose between avoiding one harm or the other—for example, choosing to risk scratching a vehicle in order to avoid colliding with a pedestrian. Sometimes the seriousness of a harm depends on its scale: injuring one person is less serious than injuring thousands in one event. Many ethical dilemmas arise when trying to rank harms.

Security harm categories. People working on information security have developed categories to help organize harms and prompt their discovery. This is the “CIA” approach [FIPS199]: confidentiality, integrity, and availability.

  • Confidentiality is protecting information from unauthorized access and disclosure;
  • Integrity is protecting against “improper modification or destruction, including non-repudiation and authenticity”; and
  • Availability is protecting “timely and reliable access to […] information”.

The referenced standard also qualifies each category by its impact.

While these categories were formulated specifically for information security, they are useful guides for general security. A nation or company might want to keep the existence of the system secret, or an aircraft should be difficult to detect. A road vehicle depends on the integrity of its steering, throttle, and braking systems—that they respond to commands as designed. An airline depends on the availability of the aircraft it operates, on the fuel or electricity is uses for those aircraft, and on the availability of airports from which to operate their flights. In other words, these apply to physical subjects beyond just information.

Absolute versus rates. For some harms in some systems, the goal is to eliminate the harms entirely or to reduce it as much as possible—with good enough being subjective.

In other systems, harms that cannot be eliminated must be reduced to meet a specific target. The goal will be to reduce the rate at which the harm occurs to below some defined threshold. For example, in the US, regulation requires that “catastrophic” events, leading to multiple fatalities, just occur at a rate no more than 10−⁢9 per flight hour [FAA11].

Meeting rates or not only affects the steps for identifying harms and mitigating hazards. Identifying accidents and hazards do not need to consider whether a harm has an associated rate.

46.8 Define accidents

The next step is to define accidents based on harms. Accidents are the events when harms occur. They provide context and specifics to the situations where harm occurs.

An accident can usually be expressed as an event that can cause one or more harms:

accident = event + harm(s) caused

In the autonomous road vehicle example, the list of accidents was long and included things like:

The UAS air traffic management example earlier listed accidents that system was intended to avoid (Section 46.6.1).

Security accidents might include:

46.9 Identify security threat actors

Security accidents do not occur randomly; they are caused by actors taking deliberate action. The intentions and capabilities of different threat actors affect what kinds of accidents they are likely to try to cause. For example, a nation-state actor is likely to have greater resources and expertise than a random disgruntled person. A criminal organization will be motivated by financial gain more than destruction, whereas a nation-state will be motivated by destroying an opponent’s systems.

Identifying the threat actors helps identify hazards, discussed next. It also informs the risk analysis that guides prioritizing which hazards to address.

A threat actor can be characterized by:

Working out the threat actors leads to identifying accidents, and these in turn can lead to updates to the system purpose.

Consider a few examples.

Consider an application that is meant to run on a personal device. That application will contain some information that its owner will want to keep confidential in order to avoid minor embarrassment or discomfort, but if disclosed the information will not lead to significant financial loss or physical injury. This information might be of interest to a relatively limited set of threat actors, who would likely not thing it worth putting a lot of effort or taking a lot of risk to obtain. However, they might do so by trying to hack into the personal device, stealing a copy of a backup of the device, or stealing the device itself. (The threat actor might also steal the device just for the value of the stolen material, with the information in question being a side-effect.)

By comparison, consider a criminal organization that has access to significant cyberattack capabilities. Their motivation is financial gain by extracting a ransom by corrupting an organization’s operational data. They are not, however, motivated to subvert someone inside the organization to act as their agent; their best interest is served by trying to attack as many organizations as possible with the least effort for each one. These actors are not likely to try to gain physical access to any parts of the system; instead, they are likely to try to attack and subvert some system, such as an employee’s laptop, which will give them a foothold from which to attack further into the organization. This kind of threat actor will not be interested in the personal information in the previous example.

At the extreme, a nation-state or a terrorist organization might value causing significant physical damage somewhere. They will be willing to use nearly any mechanism they can to achieve this aim, though they may wish for their attack to be hidden for some time before being activated. They will be willing to take the risks involved to obtain physical access to part of the system—to steal or sabotage some component, or to install malicious software that can spread.

These different kinds of threat actors create different kinds of security hazards.

Understanding the different potential attackers leads to updates to the system purpose, and from there to concept and specifications. Some years ago I was working on the high-level design for an autonomous aircraft system. During an analysis of potential threat actors, it became clear that if the aircraft’s control or guidance were subverted it could be used to damage buildings or ground infrastructure by crashing at high speed. This revealed that protecting the aircraft against this kind of subversion was a basic part of the system purpose, so we added that to the purpose document and began revising the system concept and high-level specifications to account for it.

46.10 Identify hazards

The third step is to identify the hazards that can lead to each of the accidents, that is, the list of situations that can lead to the accidents. This list is typically long; there are usually many ways that an accident can happen.

A hazard can be expressed as a set of conditions that can lead to an accident, which in turn causes harms:

hazard = set of conditions + accident

= set of conditions + event + harm(s) caused

Identifying all the hazards in a system is key to making the system safe or secure, but identifying hazards is something of an art. I learned how to do this by doing it, haphazardly and not very well early on and getting better over the years. That is not the best approach for learning how to do it.

There are several methodologies that are used to identify hazards. However, no methodology ever substitutes for good judgment, which arises from understanding the problem. I start with the general idea and then discuss specific methods (starting in Section 46.10.1).

The basic approach I have learned for identifying hazards is:

  1. I start with the harms and try to imagine all the ways those harms could happen, which leads to a list of accidents.
  2. After identifying an accident, I then plot out how the system functions when it avoids the accident, then enumerate all the lower-level functions that contribute to this desired emergent behavior.
  3. Next, I look at things that could interfere with these lower-level functions. These reveal the conditions that can lead to the accident, and thus identify hazards.

For identifying hazards, I often try to do this in a somewhat organized way. For example, with an autonomous road vehicle I go through one drive: getting into the vehicle, starting it up, getting it onto the road, driving along the road, and so on. In each phase of operation I try to find all of the ways each harm could happen. I also try to find all the operations that the vehicle will do; for example, a vehicle will undergo maintenance and it will remain parked somewhere for a period.

I also try to find similar systems or similar environments and discover what hazards have been observed in them. Battery problems in electric road vehicles, for example, are a well-known cause for accidents.

Note that, for complex systems, identifying all the significant hazards will likely not be complete after one short exercise. My experience is that a concentrated investigation produces many hazards in a short time, but that one discovers more hazards over time, at a rate that generally decreases over time. Continuing to look for potential hazards all through development and after a system is put into service is important.

How I understand nominal system behavior is best illustrated. Consider the example from before, of an autonomous road vehicle. The accident to be avoided is causing damage or injury by colliding with road infrastructure—roadside barriers or bridge piers, for example (Figure 46.3). To do so, the vehicle must move in a way that departs its planned lane and intersects the infrastructure.

undisplayed image
Figure 46.3: Vehicle colliding with road infrastructure

The vehicle’s normal behavior is to follow an overall path to its destination (which streets, for example). In the short term it develops motion plans for how it should move to follow the road lane. It then manipulates the wheels to exert forces on the vehicle to move it along that plan.

undisplayed image
Figure 46.4: High-level functions in controlling vehicle

The vehicle has one or more control systems that take in inputs about the desired short-term path, the vehicle’s state, and its environment, computes what forces should be applied to keep the vehicle on its desired path, and outputs commands to steering, braking, and accelerator actuators to achieve those forces (Figure 46.4). The control system recomputes these values quite frequently.

One can trace through these functions to find situations that could cause the accident. Some examples:

A vehicle might collide with road infrastructure:

  1. When the vehicle’s sensing system did not sense the correct location of the road infrastructure (sense environment).
  2. When the vehicle’s sensing system did not sense the road infrastructure at all (sense environment).
  3. When the vehicle’s sensing system did not lead to a correct estimate of the vehicle’s location and motion (model state).
  4. When the path planning correctly planned a path into the infrastructure in order to avoid a worse harm, such as colliding with a person in the road (modify objective).
  5. When the path planning created an incorrect path that intersected the road infrastructure (plan short-term path).
  6. When the vehicle’s path planning created a path that required the vehicle to perform a maneuver it could not, such as turning too sharply or braking harder than possible (compute control).
  7. When the road surface was more slippery than normal, leading to the vehicle not following a planned path (compute control, actuation to vehicle state).
  8. When the vehicle’s guidance and control system failed, so that the vehicle did not follow planned paths (several functions).

Such collisions can also be caused by attacks—in the grey zone between safety and security hazards. For example:

In these examples, the reason why a threat actor causes a hazard affects what one does to address the hazard.

One can follow each of these hazards downward to lower levels. Consider if the sensing system does not produce a correct estimate of the vehicle’s state. The driving process takes in the vehicle’s state: position, attitude, and velocity. The state might also include the state of braking actuators, tire inflation, steering angle, or current accelerator setting. If any of these are wrong, it can lead the control algorithm to make a wrong decision and guide the vehicle outside the expected path.

The vehicle state information could be out of date (delayed), or missing, or incorrect. It could be missing because:

The state information could be wrong because (among other things):

One can follow this kind of reasoning to find many different ways there can be problems with keeping to a lane.

This procedure will produce a large number of hazards. I have found that it is necessary to take care in organizing the procedure and its results. One may well discover the same hazard multiple times.

I have found it helpful to organize hazards hierarchically, starting with whole-system hazards at the top and then recording lower-level hazards that can cause those to occur.

46.10.1 Methodologies

Finding and fixing all the ways that a system can result in harm is one of the challenges in building a safe and secure system. Leaving one attack vector unidentified and unaddressed means the system is vulnerable to attack. Missing one major safety hazard means the system can result in an accident.

Of course, perfection is not generally possible—especially since harms or threats might not yet be known. It is generally better to identify and address as many hazards as possible rather than do nothing.

There are several methodologies for addressing safety and security harms. These methodologies all aim to provide a systematic process that will identify and resolve all significant hazards when used properly. In practice most methodologies have limitations that one should understand.

However, no methodology ever substitutes for good judgment; methodology can inform judgment, but the people building the system are still responsible for safety.

The methodologies work from different views of a system to find hazards. I discuss five:

In practice I often use all these methods, because each one is good at identifying and addressing different kinds of hazards.

In the next few sections I give an overview of these methods. The overview is not meant to be sufficient as instruction on each of them; instead, it explains how each of these methods work in general terms and what it is good for, and includes references to where one can find more information on how to apply them in real projects.

46.10.2 Control-theoretic

These methodologies view safety or security as properties that emerge from the behaviors of the components that make up a system (Section 12.4). A system then has a controller that manages the component behaviors so that they jointly produce the desired safe behaviors. This controller can be understood as a feedback system that senses system state, applies a control algorithm to decide on responses, and produces commands to move the system to the desired state.

This control system might be explicitly designed or might arise and be discovered from component designs, but either way the controller guides the components. It might not look like a regular feedback controller; it can take other forms, such as communication patterns that ensure that some condition is checked before taking a potentially-hazardous action.

The System-theoretic Process Analysis (STPA) methodology [Leveson11] provides a good explanation of this way of analyzing systems and identifying hazards.

undisplayed image

Hazards, in this model, arise when the control mechanism doesn’t work as expected. [Leveson11, Fig. 8.6, p. 223] illustrates the different problems that the control can experience. These include:

The control-theoretic approach is better for identifying some classes of hazards than other approaches. For example, problems arise from component interactions—where all components work as specified, but they do not combine to produce safe (emergent) behavior in some circumstances.

46.10.3 Flows

These methodologies put the focus on how things move through a system—including energy, information, mass, or people.

These flows of things follow the relationships among components (Section 12.2), especially the functional flows. As a reminder, there is a functional relationship between components A and B when an event in A can cause an event in B. The events might be sending and receiving information, moving thermal energy from one to the other, or moving a fluid fuel between them.

The approach I follow is to consider a use case or scenario, trace how its effects move through system components, and consider what could cause a problem at each step. The flows follow paths through the system, following functional relationships, to accomplish something. (I discussed this idea as it relates to views on a system in Section 13.2.)

Consider the example in Figure 46.4, for controlling an autonomous road vehicle, which shows a set of software components and some of their functional relationships. One scenario is to drive along the road in a way that meets a short-term objective, such as following the current lane for the next kilometer. (Other flows might change that objective to something else, like changing lane or stopping, but those are considered separately.) In the example components, this involves three paths that together start from sense environment and end with actuate. The paths are:

  1. sense environment → model state → compute control
  2. sense environment → model environment → plan short-term path → compute control
  3. compute control → actuate

The exercise is then about identifying all of the assumptions that are needed for the flow to behave correctly. Put another way, it is about imagining all of the ways that something could invalidate the design assumptions for that component.

undisplayed image

For each of the functions, consider:

This approach has much in common with the control-theoretic methodologies, but with a focus more on a general flow through part of a system than on a control mechanism. In practice I have found that the two methods often overlap.

Identifying implicit relations. Consider the example I’ve used of the autonomous road vehicle steering itself to follow a lane and thereby avoid collisions with roadside infrastructure. This is done through a classic control system, that takes inputs about the vehicle’s location and motion, computes reactions, and sends commands to actuators in response.

When I first sketched this example, I thought of it in those terms; in particular, that the control output is a set of commands to steering and other actuators. When driving, I think in terms of how I need to manipulate the steering or brake pedal to “make the car go where I want.”

This feedback system looks like:

vehicle state → sensor → sensor values communication → control system → control computation → actuator commands → actuators → vehicle state.

However, this isn’t actually quite correct. The steering or braking actuation is not the point of the control system; the point is to use them to create forces that will move the vehicle. If the vehicle were not in contact with the road, the control mechanism could actuate all it wanted and it would not affect how the vehicle moved. Moreover, the control mechanism has within it a model of how actuating steering or brakes will cause desired forces. That model might be implicit, encoded in experimentally-derived parameters within the control calculations, but it nonetheless has that model.

A more complete set of relations is:

vehicle state → sensor → sensor values communication → control system → control computation → actuator commands → actuators → forces on vehicle → vehicle state.

If one doesn’t do the analysis in terms of forces changing vehicle motion, then the analysis will miss important hazards: errors in the controller’s model and cases where the environment does not match assumptions embedded in the controller’s model. For example, if the road surface is slippery it will not provide the friction expected to turn the vehicle. This can lead to the vehicle not turning as quickly as needed, or for the vehicle to start skidding.

Recognizing that these hazards exist is necessary for addressing them.

This is an example of how leaving out essential causal relationships when trying to discover hazards results in missing hazards. It is worth the time to trace out the low-level causal steps involved in, for example, getting a vehicle to turn. Each of the steps is a point in the system where something can happen that interferes with keeping the system in its desired state.

46.10.4 Structure

These methodologies put the focus on the structure of the system. They are particularly useful for security analyses, where part of the purpose is to identify how someone could get into a system, or a part of it, to attack. However, structural analyses are also useful for analyzing safety properties when part of the system design is to compartmentalize problems, discussed below.

I first encountered this approach in an organized form in The Security Development Lifecycle [Howard06]. That book covers much more than identifying security hazards: it also addresses organizational issues, testing methods, software development and testing strategies, and so on. While the book focuses on security for software systems, many of its ideas apply generally. That work has been continued since that book was published and further resources can be found on line.

The structure methodologies seek to identify threat vectors into a system. A threat vector is a means by which an attacker can get access to a part of the system in order either to subvert part of the system or to extract something from the system. A hazard arises when an attacker can access a threat vector and use it to cause an accident. Part of mitigating the hazard is then to address that vector. (The collection of all the threat vectors is the attack surface of the system, and one goal is to minimize the attack surface.)

The approach is useful for more than just addressing malicious attacks. The ways that the effects of a failure propagate through the system follow similar paths, and so considering structure can be useful for containing problems that would lead to safety hazards.

The approach looks for the ways in and out of the system. These are the channels by which someone could inject malicious information, physically manipulate part of the system, and so on. The first step is to identify all the explicit, expected interfaces to the system. These interfaces should include all the interfaces: data, energy, mass, physical access. The second step is to identify all the implicit interfaces.

These steps need to work both from the top down, starting with the high level view of the whole system, and from the bottom up, looking at the ways each individual component could be accessed. For example, at the high level a user could use a web interface to communicate with a system, or an autonomous vehicle could communicate with an operations center over a cellular link. It is by digging into the lower-level details that one discovers unobvious vectors. For example, a part of a web service could be deployed on a virtual cloud server, with the cloud provider having control interfaces for managing the virtual server. An autonomous vehicle could have an access hatch that allows physical access to the on-board electronics bay.

The analysis has to begin with identifying the information or functions that need to be protected (Section 46.7), as already discussed.

Start with the system, as defined by its boundaries (Chapter 10).

undisplayed image

This first step identifies the major interfaces and dependencies, but it does not reveal lower-level dependencies or hazards internal to the system.

The next step is to recurse one level down the breakdown structure (Chapter 41) to the components that make up the system. Follow the functional and non-functional relations between components at that level, looking to determine their inputs, outputs, and dependencies. Each of these inputs and dependencies is a potential source of hazards.

undisplayed image

Continue this exercise on lower-level components, until one reaches components that do not have reasonable inputs or dependencies separate from what has been identified at higher levels. For example, it may be sufficient to stop at one electronic assembly rather than following further to the individual chips in that assembly.

As an extremely simplified example of this process, consider the autonomous road vehicle I have discussed before. Figure 46.5 shows the top level view of the vehicle and external actors, and Figure 46.6 shows the main components that make it up and how those connect to the external actors identified at the top level.

undisplayed image
Figure 46.5: Top level structure of the autonomous vehicle, with system boundary and major interfaces
undisplayed image
Figure 46.6: First-level components in autonomous vehicle

At the top level (Figure 46.5), the system—the vehicle—has a data connection with the outside world over which it receives commands and sends monitoring information. The vehicle also interacts with its environment by receiving data through sensors (such as cameras, LIDAR, and GNSS receivers) and producing forces through how it moves tires against the surface.

There are, however, other less obvious channels by which the vehicle can be affected. The vehicle can be affected by wind, by radiation or RF energy. Over time one builds up a list of these kinds of effects that cross the boundary from the environment to the vehicle.

Even at this first step, the first three identified interfaces can produce potential hazards.

The command and monitoring interface:

The sensing interface:

The force interaction interface:

One can then explore each of these vectors to work out specific hazards. The sensors’ integrity could be attacked, for example, by blinding the cameras or LIDAR. The integrity of the command interface could be attacked by inserting unauthorized commands into the stream, or modifying authorized commands that have been sent.

The next step is to open up the system and look at the first level of components that make it up (Figure 46.6). For each of these components, identify the interfaces and dependencies they have, and find the hazards these can produce. The interfaces and dependencies are the functional and non-functional relations (Section 12.2).

Consider the commanding component. It has obvious inputs and outputs, as recorded in functional relationships for data, energy, or mass flows. It receives messages from outside the vehicle via the communication component. These messages are expected to contain commands that include things like setting a destination, setting driving or navigation parameters, updating maps, or allowing the vehicle to start operation. The integrity of these commands is important: giving it the the wrong map information can lead the vehicle to drive somewhere unsafe.

The commanding component might be designed so that it assumes that the communication channel with the outside command-and-monitor actor is implemented so that command messages cannot be inserted into the stream, messages cannot be altered in transit, and only the real external actor can send the commands. This is a dependency on the design of the communication component.

The commanding component also has less obvious interfaces. It is a software component that runs on a compute node. That compute node likely provides an operating system, supporting software packages, low-level communication mechanisms, data storage, and secure information storage. The compute node has interfaces for updating its software, receiving electrical power, and physical access to check or replace the node. The operating system may collect activity logs and provide them over an administrative interface. The commanding component likely has additional interfaces to communicate with other software components in the vehicle to monitor that it is working correctly.

Every one of these dependencies and relationships has the possibility of creating a hazard. Each of them must be investigated to find the actual hazards and work out how they are mitigated.

Sidebar: External and environmental dependencies
  • Software environment
    • OS and libraries
    • External software used
    • Supply chain to deliver software to the deployed system
    • Administrator privileges and access
  • Hardware environment for software
    • Physical access to data
    • Network connections
  • Physical environment
    • Access to hardware
    • Atmosphere, dust, radiation, electromagnetic energy
  • Access environment, not under system control
    • User systems that access the system
    • Browsers
    • Messaging applications
    • Physical kiosks
    • Printed material distributed outside the system
  • External services
    • Cloud environment
    • Identity and authentication mechanisms
    • Authorization mechanisms
    • Time and clocks
  • Communication environment
    • External networks
    • Routing
    • Name services
    • Messaging
  • Other protection services
    • Backup, disaster recovery
    • Intrusion detection
    • Monitoring and logging

46.10.5 Failure

The failure-focused methods are some of the oldest in common use today. They originated in designing systems for reliability or fault tolerance, rather than safety or security.

The general approach is to start with a failure, either of the system as a whole or of a component, and reason about its impacts or causes. The top-down methods start with a system-level failure (sometimes called an “undesirable state” or “system failure condition”), and look at what events or conditions could lead to that failure. The bottom-up methods start with a failure at one component and look at what higher-level failures could be caused by that failure.

Top-down methods. The top-down methods for failure analysis are related to the other top-down safety methods, such as STPA or structural methods. Indeed, all these methods can blend together in actual practice. They differ mainly in what conditions they look for, and how they address safety, security, or failure as an emergent property that can involve many components. The control-theoretic methods look at a safety property as a control mechanism; structure methods look at them in terms of boundaries or attack vectors; failure methods focus on arbitrary “failure” conditions.

Fault tree analysis (FTA) is a top-down methodology that starts with an identified hazard and works out what events or conditions might lead to that hazard. It is a useful way to organize identifying how one function depends on another. In one project, I was analyzing the conditions where an autonomous vehicle could fail to follow guidance commands. This failure could happen if the vehicle throttle failed, its brakes failed, the steering function failed, or the low-level control electronics failed. I could then take each of those subsystems and identify how they could fail: steering could fail if the ability to communicate with the steering column failed, or if the steering column failed mechanically, and so on. I then could use models of failure rates to estimate which failure modes were the largest contributors to system failure.

The FTA approach has the benefit that it organizes the process of discovering relationships among functions: function A in component B depends on function C in that same component and on function D in component E. It is useful for estimating the relative contribution to high-level failure rates, which can be used to guide where effort should be put in addressing problems. It has the downside that by itself it does not guide one toward identifying and addressing interaction problems rather than failures.

Bottom-up methods. The bottom-up methods, on the other hand, put the focus on one component. The method is generally to identify ways that the component can fail, and then trace the effects of each of those failure modes out into the system. This method has the advantage of putting close attention on one individual component. It can use expert knowledge about that component to identify ways that it could fail that a generalist working top-down might not know about. The disadvantage is that the focus on one component makes it hard to reason about how problems with one component can interact with problems in a different component—for example, these methods have been found not to identify component interaction accidents (see [Leveson11, p. 8]) where every component functions per specification but the interaction between the components nonetheless causes an accident.

Failure mode and effects analysis (FMEA) is a bottom-up methodology for identifying the effects of a fault in one component. It is the basis for some safety lifecycle standards, including ISO 26262 [ISO26262]. The methodology focuses on one component, identifies ways that the component can fail, and then projects the effects of that failure outward into the system. The importance of addressing a component’s failure mechanism can then be based on the system effect.

The FMEA approach has the advantage of taking advantage of specialist understanding of a component’s design for guiding design effort. It is often used to guide the amount of effort to be put into failure tolerance; for example, in automotive electronic systems an “ASIL D component” is one that is intended for the most failure-critical uses.

46.10.6 Formal methods and automation

Automated tools are beginning to assist projects in identifying hazards. Tools that can analyze a formal model of a component or an implementation can rapidly explore potential ways that the component can misbehave, identifying hazards more quickly than a human can. Automated methods can also find hazards that human imagination does not catch.

I first used semi-formal methods when designing low-level communication and synchronization mechanisms for building parallel and distributed software systems. While those efforts were fully manual, they revealed design errors that I had not found by inspecting the designs. They found error cases that I had not thought of. This convinced me of the importance of formalizing correctness, safety, security, and similar analyses where possible.

Consider two examples of using formal methods: TCAS II and SeL4.

TCAS. The Traffic alert and collision avoidance system (TCAS) is equipment that is required to be installed on all large transport aircraft. Its mechanisms supplement a pilot’s responsibility to detect other aircraft and remain clear of them, and of air traffic control to provide guidance and information to keep aircraft separated. A TCAS system detects the presence of other nearby aircraft that could pose a collision threat and alerts pilots to ascend or descend in order to maintain a safe distance. The TCAS system, therefore, has strict safety requirements.

Early versions of the decision logic in TCAS were implemented as a complex set of rules, and then tested (in simulation and in flight) to verify that they were sufficiently accurate. However, as time went by, each improvement complicated the logic and the rule set became expensive to maintain and test. A second generation of TCAS used very simple decision logic that was driven by a table of potential reactions in different situations [Kochenderfer11]. This table was machine-generated based on a large body of potential encounter scenarios and weights for how desirable different alerts would be. The result was a design that roughly halved both the rate of potential collisions and the rate of false alerts.

TCAS thus provides an example of using automated mechanisms to design and implement a safety-critical system component. The process eliminated many cases where people could make design errors.

SeL4. The SeL4 operating system is a microkernel software environment for high assurance systems. While it is very high performance, it is best known for being formally proven correct in its security properties. The functional correctness of both the design and of the binary implementation have been proven.

The team that built it formally proved properties like:

  • One application cannot interfere with the integrity of another.
  • Information confidentiality is maintained except as authorized.
  • One application cannot interfere with the ability of another to run.
  • Authority cannot leak from one application to another
  • Applications can execute in real time, with strict deadlines.

These properties provide a trustworthy foundation on which higher-level system security properties can be built.

The central piece of this work is still the initial functional correctness verification of seL4 in the theorem prover Isabelle/HOL. This property is stronger and more precise than what automated techniques such as model checking, static analysis or kernel implementations in type-safe languages can achieve. It not only analyses specific aspects of the kernel, such as safe execution, but also provides a full specification and proof for the kernel’s precise behaviour down to its C implementation on the ARM platform.

[The results] extend this proof in the strongest sense: they directly compose with the formal functional correctness statement and yield end-to-end theorems about the C source code semantics in the theorem prover Isabelle/HOL.

—Klein et al. [Klein14]

The work used several existing specification languages and theorem provers to verify the system. They used “interactive, machine-assisted and machine-checked proof” techniques, rather than fully-automated verification, which is beyond the current state of the art. The objective was to show that a high-level model of the system was formally refined into a C source code implementation, and that that is in turn was refined into a correct binary implementation.

The project developed a design and implementation approach that supported both kernel developers, often focused on the low level, and those who verified the system, who focused on top-down properties. This allowed them to build a checking suite that they used to verify that all proofs remained valid after every commit to the code base.

While this required significant effort, they found that it was less than a third of the cost of assuring a system using traditional engineering.

The future. Both of these examples, TCAS and SeL4, indicate that designing for verifiability or assurance is key to achieving it. It is hard to use formal methods to any arbitrary design—the space of possible states can grow too fast to be processed using current tools. Instead, both projects created simplified methods for specifying the system that limited what could be expressed (encounter rules in TCAS; kernel design and formal models for SeL4).

I speculate that, as with the development of software languages that restricted what a developer could express, ways of expressing system designs may evolve that prevent designers from expressing poor designs and that make the designs more amenable to checking.

46.10.7 Summary of methods

The top-down methods presented in this section for identifying hazards have many similarities. They look at how some high-level behavior or property maps onto the components that implement it. The top-down methods differ mainly in what kinds of high-level things one looks for: emergent properties, flows, structure, or failure. In practice there is no sharp line dividing one of them from the others.

FMEA is the exception, working bottom-up. This helps reveal low-level details that can affect high-level and that might be missed only working from the top down.

Reasoning about high-level properties sometimes requires some degree of formalism, whether that involves true formal methods and automated processing, or semi-formal manual proofs.

I generally find myself using multiple methods at the same time. It isn’t all that hard to find problems that arise from one behavior at one component if one takes care and works methodically. It is harder to catch problems that arise from multiple concurrent causes—when two components behave just as designed but together they cause a hazard. Some of these non-local interactions will be more evident from one perspective or another.

46.10.8 Continuous re-evaluation

The SeL4 example shows how hazard analyses need to be re-evaluated continuously during development. Each change to requirements or design can affect some property relevant to security or safety. New kinds of safety-related accidents or hazards will be discovered as a system gets used. New security hazards will arise as threat actors develop new capabilities. Team members will discover gaps and mistakes in the analysis as they work on the system.

The team can record the hazard analysis in a way that makes it easier to use it and update it regularly. Section 46.18 discusses this further.

46.11 Risk analysis and prioritization

Complex systems usually lead to a large number of hazards. Tackling all of them at once is not feasible; the team has only so many people who can work on them, and addressing one hazard (discussed in the next section) can interact with another hazard or create new hazards.

The better alternative is to tackle only a few hazards at a time, with a small enough team that they can maintain a common understanding of their work.

Most safety design methodologies include a risk analysis, where each hazard is given some kind of risk score based on how serious the harms are and how likely they are to occur. The team can then focus on the hazards with the highest risk scores first, working their way down to less potentially serious hazards over time.

If the associated harms have a maximum acceptable rate, as discussed above, then the gap between the required rate and the rate an which the unmitigated hazard will occur is a rough indicator of importance. Note that the required rate is related to the top-level harm, not to the hazard, and so one must consider how much one individual hazard contributes to the total rate of the harm occurring. (The FTA method can help estimate the contribution.)

Some hazards will not be worth addressing. A hazard that can lead to mildly annoying some users, for example, may not be worth putting a lot of effort into addressing, or addressing that hazard might be deferred to a later version. While the goal should be to build as good a system as possible, limits on time and resources mean it will not be perfect.

Other hazards can be quite serious, but there is no point in addressing them, such as the earlier asteroid collision example. The rate of asteroid collisions is sufficiently low that it does not appreciably contribute to the risk of many harms.

46.12 Mitigate hazards

The final step is to determine how to address the hazards. This can be done by (in decreasing order of preference):

Eliminate hazards. The best option is to make an accident impossible, by ensuring that some collection of component behaviors or properties cannot lead to the corresponding hazards.

Some examples:

  • Spacecraft re-entering the atmosphere could impact property or people on the ground, causing damage or injury. This hazard can be eliminated by designing the spacecraft so that it will disintegrate in the upper atmosphere.
  • An electrical system can cause shock or electrocution to a user or technician. If the system only uses low voltages (under ~50V) serious injury or death will not happen.
  • An intruder could access and damage confidential information in a server. This can be avoided if the server is not connected to any networks and is in a physically-guarded location (though this approach is not as effective as once believed).

The safest system is often one that does nothing. An airplane cannot crash if it never takes off. A server cannot be compromised if it is never turned on. Unfortunately this way of eliminating hazards is useless. One needs to eliminate hazards while still meeting the system’s purpose.

Reduce likelihood. Sometimes a hazard cannot be eliminated For example, if a system really must use high-voltage electricity, such as in an electric transport system, the hazards associated with high voltage cannot be eliminated. The next-best option is then to reduce the chances of the hazard occurring.

This is done by adding new components to the system, or adjusting properties or behaviors of the set of components that can interact to cause the hazard. This is also often done by adding a control function into the system to manage component behaviors.

Examples:

  • When a large spacecraft must re-enter the atmosphere, include reliable control systems to deorbit it to a safe location. (Point Nemo, in the South Pacific Ocean, is commonly used.)
  • In most spacecraft, their internal components will only work in a fairly narrow temperature range. Some electronics will function in a range of -20º to +50º C, while fuels like hydrazine must be kept above ~5º C. These spacecraft have control systems that manage internal temperatures through managing spacecraft attitude, heaters, and radiators.
  • When using high-voltage electricity, introduce interlocks or procedures to de-energize components before they need to be manipulated.
  • For a network-connected information system, use firewalls, authenticated access, encrypted communication to control access.
  • For aircraft navigation, use electronic navigation systems (ILS, GNSS, ground proximity warning, TCAS) to maintain ability to navigate when pilot cannot do so visually.

Note that control systems take active steps to reduce the likelihood of an accident if they work. Problems with the active control system can lead to the accident it was meant to avoid (and these problems can cause new accidents as well). An interlock in a high-voltage electrical system only makes a hazard less likely when it works correctly. To show that the mitigations work requires evaluating their safety or security (Section 46.6.1).

Consider the hazard identified earlier for an autonomous road vehicle: a collision with road infrastructure because the road was sufficiently slippery that the vehicle could not follow a path that avoided the infrastructure. This hazard can be broken down into sub-hazards:

  • Occurring when there is ice on the road.
  • Occurring when there is water on the road.
  • Occurring when there is non-weather debris (oil, mud, small particulate matter) on the road.

The first hazard can be (almost) eliminated by mandating that the vehicle never operates in temperature below 10º C, or when there are reports of remaining ice in a region. If the vehicle does not operate when there might be ice then the first sub-hazard is not possible.

This hazard is only almost eliminated because the mitigation strategy is dependent on weather and temperature information. If that information is wrong then the prevention mechanism won’t be accurate and the vehicle might encounter ice. This might happen if weather sensors malfunction, if communication about weather forecasts is disrupted, or if there are no recent reports about road conditions. These create third-level hazards: When the road surface was more slippery than normal, leading to the vehicle not following a planned path and when the road is slippery because of ice and when temperature information sources have malfunctioned.

This recursion continues until one can provide evidence that the remaining hazards are unlikely enough.

The second hazard can be made less likely by disallowing the vehicle from operating when the chances of rain are above some threshold. However, this does not eliminate the hazard: rain can occur without much warning, and water could be on the road for other reasons like dew in the morning or water flowing across a road from a roadside water leak. The rain-related sub-hazards might be eliminated by including precipitation sensors on the vehicle; water from other sources might be detected by sensors that could evaluate the quality of the road surface. Each of these mechanisms can result in residual hazards that must be addressed.

Reduce harm. One can also try to reduce the harm caused by a hazard, usually in addition to reducing its likelihood.

Examples:

  • For high-voltage electrical systems, provide people with protective equipment such as insulated work wear.
  • In a network-connected system, use internal compartmentalization, backups, and disaster recovery to reduce how much harm can be done when a server is attacked.
  • In motor vehicles, use seat belts, crumple zones, and air bags to reduce forces acting on passengers, reducing the severity of injuries.
  • For UAS, limit the mass that the aircraft can have if flying over people, impose a maximum velocity, and design it to break up to lighter parts if it crashes.

These mitigations also only help when they work, and so their effectiveness has to be evaluated.

A mitigation’s effectiveness can depend not just on its technical design but also on whether people will actually use it. Electrically insulating work wear is only effective when people wear it. Similarly, a seat belt only reduces injury if it is worn.

In the example of a road vehicle colliding with road infrastructure due to wet conditions, steps can reduce the harm caused. Harm is generally related to the kinetic energy, so reducing speed can reduce damage. If an onboard rain sensor detects rain, the vehicle can slow down so that it is less likely that the vehicle will lose traction and that any collision will occur at lower energy. One might also include a sensor to detect loss of traction—that is, detecting that the hazard may be present—and initiating some kind of traction control response.

Reduce harm through compartmentalization. One way to reduce harm is to contain the spread of some effect through the system. This is

Several well-known design patterns reflect this idea.

  • Using two or more redundant components can limit the problems caused by one of them having a problem. If one component has a detectable failure or misbehavior, the other can take over and continue providing service.
  • Using multiple independent sensors to estimate some input measurement can reduce the effect of a problem. In the Mars Polar Lander loss [Leveson11, p. 8], noise from a landing sensor led to early engine shutdown. Adding a second input source to the shutdown decision, such as using radar altimetry, could have prevented the problem.
  • Communication firewalls and isolated computing environments can limit what an intruder or software bug can affect. There is a story of an attack on a casino’s data systems that entered through an insufficiently secured, network-connected fish tank controller [Schiffer17]. Once the intruders has compromised the controller, they were able to connect to other systems on the internal network and extract information from a database server. Better compartmentalization between the systems would have protected the database contents.
  • Defense in depth cybersecurity strategies layer multiple ways of defending attack vectors so that if one layer is compromised, the others can still protect the system.

None of these patterns completely eliminate the hazards they address, but they can reduce either the rate, the effect, or both.

Containment often depends on independence. Multiple sensors provide benefit if they are likely to have different errors. Redundancy for reliability works when the redundant components do not have common failure modes. Firewalls defend parts of a system if the parts are truly separated.

In practice complete independence is not possible. Components that are part of the same system always share the same environment. Attempts to build multiple independent implementations of a component rarely help much [Knight86]—independently-developed components usually share common flaws that derive from unclear or incorrect requirements that affect every design.

Instead, one should focus on what kind of containment or independence actually matters and find ways to ensure that those properties are designed and implemented correctly.

Detect and respond to harms. Finally, if the accident occurs anyway despite all the steps taken to eliminate it or reduce harm, the system should at minimum detect and report the accident. This should lead to a process to recover the situation—at worst, bringing an emergency response team into action.

If an autonomous road vehicle collides with another vehicle or a person, it is considered necessary for the autonomous vehicle to stop and notify someone to evaluate the situation and initiate an emergency response if needed. (This is the same standard to which human drivers are held.) Doing so requires significant capability in the autonomous vehicle: in many situations it may be difficult for the onboard systems to distinguish between a collision and a pothole in the road, for example.

For all the safety features in modern transport aircraft, failures still happen. Most aircraft remain in contact with air traffic control throughout their flight, and many have telemetry continuously reporting location and status. This information can be used to detect when an aircraft has crashed. ATC or airline staff can then follow emergency response procedures to locate the aircraft, rescue people, or address problems on the ground.

Many network-connected computer systems have monitoring tools that can detect when a system has been damaged by watching for changes to data content or behavior. These tools can notify people who take steps to determine whether there is a problem, isolate the system if needed, and protect other systems.

As a final example, many spacecraft are designed with a “safe mode” or “safe hold”. When some accident occurs where the spacecraft might lose control of essential functions, it enters a special mode that allows the spacecraft to maintain enough power to keep functioning, maintains the spacecraft’s temperature in operating range, and sets up for communication with ground resources. This does not eliminate any hazards or reduce their likelihood; it only provides a safety net for recovering after the accident has occurred. In many spacecraft the safe mode is implemented using separate avionics hardware that takes control if the primary systems might have a problem. The intent is that the remaining functionality is just enough to let people on the ground know that an accident has happened, diagnose the problem, and send up commands that will restore the spacecraft back to regular function. Without a safe mode, many missions would have ended in failure, but instead the spacecraft were restored to function and the missions proceeded.

All these approaches share some capabilities.

  • A way to detect that an accident has occurred. The detection mechanism should be reasonably accurate: too many false positives will lead people to ignore reports, while missing accidents means no one responding.
  • A way to report the accident.
  • Procedures to address the accident.

Mitigation design. Consider designing mitigations for a hazard at some level of the system. The mitigations will change the design of those components so that the desired safety or security properties hold.

This can be done by changing the properties or behaviors that some of the components exhibit. This amounts to changing the specification of those components, which leads to changes in their internal design and implementation.

The mitigation can also be done by adding and replacing components to change how that system level is designed. The new components will have properties and behaviors specified that define how they contribute to mitigating the hazard in question.

Afterwards one must check that the changes actually mitigate the hazard, and that they don’t introduce new problems. Meeting these new properties or behaviors becomes a safety objective for the component that has changed, and the process of analyzing safety repeats at a lower level.

Conflicts. The safety or security objectives can conflict. The first step is to find where there are conflicts, and then choose which objective to satisfy.

Conflict is built into some hazards. Some situations require choosing between two potential harms, such as in the case listed earlier when an autonomous vehicle must collide with a human or with road infrastructure but cannot avoid both.

In some cases a means of addressing one hazard causes a different hazard or makes another one more likely. Leveson uses the example of doors on a passenger train. The doors should only open when the train is stopped at a station platform, so that people are not injured by falling from a moving train or by falling down to the track side from a stopped train. At the same time, the doors must open in an emergency in order to avoid injuring people by trapping them on board a damaged train car [Leveson11, p. 190]. An interlock that prevents the doors from opening addresses the first hazard but can cause the second hazard, and vice versa.

In the end, I know of no simple rule to resolve these conflicts. The choice can often be guided by the relative seriousness of the alternative harms. However, in many other cases the choice requires careful design judgment.

The way that conflicts are resolved should always be well documented. These choices are likely to be revisited as a system evolves or when someone analyses an accident to understand how to improve the system.

Alternate mitigations. Often there will be multiple ways to address a hazard, and one has to choose which way to use.

The choice can be influenced by:

  • How effective each choice is at addressing that hazard;
  • What undesired effects or residual hazards it causes;
  • How difficult it will be to design and implement;
  • How well it might adapt to future changes; and
  • How much resource it will use in operation.

Sometimes addressing one hazard will also address another hazard.

In some cases it makes sense to adopt two approaches that complement each other. That way if one of the approaches fails to work, the other will still keep the system safe or secure—a belts-and-braces solution.

Again, documenting the choice and the rationale for why that choice was made is important for people who have to come later to understand what was done.

46.13 Update design and specifications

The process of identifying hazards and working out mitigations proceeds iteratively. The team finds and addresses some of them, and makes changes to the appropriate concept and specifications. These changes lead in turn to changes in design. It can lead to changes in implementation, and verification if those have already been started.

These changes to specification and design can in their turn feed back into changes in the analysis.

The team repeats this process many times, each time addressing more and more of the hazards and getting closer to ensuring that all the system’s harm objectives are met. It is usually helpful to address high-priority hazards first (Section 46.11). The repetitions stop when the analysis shows that the design is “good enough”.

The number of identified hazards usually starts small when the system is first being developed and grows rapidly as the team explores the possible accidents and as they develop the details of the system. It will also grow as they begin to design hazard mitigations—one mitigation can lead to multiple smaller hazards in lower-level components. The rate at which new hazards are identified should begin to drop off at some point.

Those managing the project’s progress should monitor the rate of identifying new hazards and the number of unmitigated hazards. In some cases, the system design may be fundamentally flawed in a way that leads to the number continually growing. (This same behavior can occur with bug fixes as well.) If the system is designed so that there is some accident that simply cannot be mitigated, the team may keep trying one thing and another to address it. Each of the changes might increase the system complexity and lead to even more hazards—while not yet adequately addressing the fundamental accidents. I observed this in one project (Section 4.1) that could not meet its stakeholder needs because those needs were underspecified and conflicting. I have seen similar problems in another project that needed a redesign and reimplementation of a major subsystem.

There are three ways to decide when the design is good enough.

  1. When all of the harms (Section 46.7) that have been identified have been met adequately. When a harm has a maximum acceptable rate, meeting that rate is an indication that the harm has been addressed.
  2. When no high-priority hazards remain unmitigated. This uses information gathered in risk analysis (Section 46.11).
  3. When the time or budget for designing and building the system runs out.

Most of these decisions are somewhat subjective. The priority of a hazard depends on the risk that people assign to it, which usually involves subjective judgments about impact. Whether the system is good enough after running out of time, when there are no other objective criteria, is also subjective.

Abandoning the project. In rare cases, the process of analyzing safety or security and addressing hazards will reveal that a system cannot be completed, or cannot be completed to an adequate level of safety in the time or resources available.

The signals that this is the case are often subtle: inability to find a way to address some critical accident, or a list of hazards that keeps growing when it should begin to shrink. These indications are generally not obvious to project managers because they are low-level and technical. Project management may only see that schedules are slipping or that some progress metric is not on track, but these indications will not in themselves indicate clearly that the project has an intractable problem.

This situation places responsibilities on team members to detect and communicate about this kind of problem (Chapter 58) and on project management to be able to detect and address it (Chapter 69).

46.14 The completed analysis

The end result of safety or security analysis is a safety case (or security case). This is the complete argument that the system is sufficiently safe or secure. “Sufficiently” means that it meets its safety objectives, as defined initially by stakeholders and documented in the system purpose.

The argument in general has the form:

The system meets its security objectives because:

Harm 1 objectives met and Harm 2 objectives met and …

Harm n objectives met because:

Accident n1 addressed and Accident n2 addressed and …

Accident nm addressed because:

Hazard nm1 mitigated and Hazard nm2 mitigated and…

Hazard nmk mitigated because:

List of the design features used to address the hazard, and evidence that the design properly avoids or reduces the associated risk.

The argument for a hazard mitigation at the system level is that the components in the system have their own safety or security objectives, which when combined mitigate the system-level hazard, plus the argument that each of these components meets its component-level objectives—which is this same pattern repeated recursively.

The argument, in other words, amounts to assembling and organizing all the pieces of the safety or security analysis so that a reviewer can trace through its content in a way to verify each claim that the system meets its objectives.

The organized analysis is also useful for tracing out other implications. If someone is considering modifying some component, they can determine how changes might affect the overall safety or security.

This information can be presented in many different ways. I have found two ways particularly useful: as a database, and graphically.

A database for recording aspects of a safety or security analysis is much like one for recording requirements (Chapter 37). It can consist of tables of harms, objectives, accidents, and hazards for the system and each level of components.

Goal Structuring Notation [GSN] is a graphical notation designed specifically for recording the arguments associated with assurance cases, the general term for safety or security cases. Figure 46.7 shows a rough sketch of such a diagram, illustrating how each goal (statements in the argument) depend on lower-level goals being met, recursively until one gets to the evidence that specific hazards have been mitigated. See the standard documents for explanation of the details of the notation, and be aware that this figure is only a rough sketch.

undisplayed image
Figure 46.7: Sketch of a Goal Structuring Notation diagram

46.15 Learning, maintaining, updating

In practice, much of what we know about how to build safe or secure systems comes from learning after accidents have happened.

Learning comes after a system has been designed an implemented. Some lessons will be learned during verification, when the implemented system is put through its paces. More lessons come after the system has been deployed and is in use. The objective should be to learn as much as possible before deployment, but it is never realistically possible to catch every possible accident in advance.

For truly safe or secure systems, it is not enough to design and implement once and then be done with the system. There are three drivers of the need to maintain and adapt a system: ongoing maintenance, of both people and equipment, to keep the system running as designed; safety or security flaws discovered in use; and changing environments leading to different needs for safety or security.

Someone, then, will have to learn from these things and update the system. The system builder might be responsible, or the customer or user, or the responsibility might be shared. At worst, no one is responsible or able to make the necessary adjustments and the customer has to abandon the system or accept that its use is unsafe.

The system must support whoever is responsible for maintaining and updating the system, whether that is the system-builder or user. A system should include ways to:

In other words, the designs for safety or security response are part of the system, just as much as the primary functions of the system. The design and implementation of the system includes mechanisms and processes to detect and address accidents. It also includes the processes for protecting information or artifacts that can be used to determine what happened. It includes the processes to request system changes.

Diagnosing and fixing a system relies on knowing what the system actually is. This is another reason why the design of the system needs to be recorded, well organized, and available. This is also why the rationales and analyses that informed the design need to be documented—so that after an accident one can determine whether there is a flaw in the system, and where that flaw might be.

Many industries have adopted mechanisms for learning from accidents. Aviation is one where many mechanisms are required by legislation—and the resulting improvement in aviation safety has been remarkable. (As have the effects when there have been lapses in these mechanisms.)

Most governments have established organizations that are responsible for investigating accidents. In the US, the National Transportation Safety Board (NTSB) is responsible for investigating aircraft accidents, determining their probable cause, and developing recommendations to avoid similar accidents in the future [49CFR800]. The NTSB was set up to be independent of the aviation regulatory agencies, in order to minimize conflicts of interest if an investigation led to findings of problems with the regulatory agencies. Aircraft manufacturers, operators, and agencies pay attention to the NTSB’s findings, and those regularly lead to regulatory actions to correct errors that have been found and to changes in aircraft design and operator procedures.

Aircraft operators, such as airlines, are usually required to have a Safety Management System (SMS) [14CFR5]. An SMS encodes a learning feedback system in which the operator documents:

46.16 Designing for other system properties

This chapter has been written in terms of safety and security. However, this same process applies to other emergent system properties.

For example, consider a system with a reliability need. Reliability is a stronger property than availability: it addresses not just what fraction of the time a function can be used (the availability), but for how long it will be usable at a stretch. The classic example of the difference is that a system could be 99.9% available if it fails for 1 ms out of every second. No useful function might complete in the other 999 ms the system is operating.

The analysis process for reliability can proceed by:

46.17 Using safety and security design

Safety and security work occurs throughout the entire system-building workflow. It is just as important to address safety as to discover and build the system’s primary functions. As with other stakeholder needs, safety and security needs are addressed iteratively as part of all the other steps to work out a system concept, specification, or design.

undisplayed image
Figure 46.8: Relationship between safety and security analysis and system development phases

Figure 46.8 shows how the steps of safety and security analysis map into the phases of the system development lifecycle.

Purpose development (Chapter 33). Identify the stakeholders affected by safety or security, as well as those who have explicit functional needs. Capture the safety and security needs of those stakeholders in the system’s purpose. Identify threat actors here if appropriate. Begin performing the security analysis here, to define the harms they care about.

This work is done by the part of the team that gathers and organizes stakeholder needs. Some experience in interpreting safety and security needs is helpful to understand what the stakeholders actually need and to uncover needs that they may have trouble articulating.

Concept development (Chapter 34). Include how the safety and security needs might be addressed in the concept. Begin working out the accidents and hazards that the concept can exhibit, and work out some of the high-level mitigations.

The team members who develop the concept should have some experience in organizing the objectives, harms, and so on, and in working out how different hazards will affect the concept. Sometimes it is helpful to bring in specialists to help the team get the analyses started.

Specification (Chapters 36 and 37) and design (Part IX). Perform more complete safety and security analyses as part of these phases. Complete the list of harms, accidents, and hazards (and threat actors), along with prioritization.

There is usually an iterative feedback loop between the specifications and design: start with the general safety or security requirements, try out a design idea, analyze the safety/security, work out the detailed requirements that will realize the high-level requirements, iterate on the design to address mitigations, and repeat.

Because this phase is when most of the safety or security analysis is developed, it needs people who have the training to develop these arguments in detail. If the people developing the analyses are separate from the team developing the design, they must work closely together: identifying hazards comes from understanding design, and mitigating hazards comes by modifying specifications or design.

This work is particularly sensitive to confirmation bias (Section 46.6.2). It can be helpful to have some of the people who are identifying accidents and hazards, and who are defining how to mitigate them, be independent of the people who are designing the affected components in order to reduce this kind of accidental bias.

Implementation (Part X). Safety and security aspects get built in the implementation—and components are reviewed or tested to ensure that they meet the safety or security properties and behaviors specified for them.

Verification (Part XI). Verification involves verifying both that the analyses are correct and that the design and implementation correctly implements the mitigations identified in the analyses.

The safety and security aspects of the implementation are verified along with all the other specifications that it is expected to meet. Where possible, perform verification tests or analyses to show that defensive safety or security functions work. Analysis by automated tools can be helpful here.

The ability to detect regressions matters, so expect to perform the verifications many times.

Production and deployment (Chapter 29). Ensure that assumptions made during design and implementation are met during production, such as that authentic parts are used. Supply chain attacks in this phase can compromise the built system.

Deployment involves setting up the environment in which the system will operate. Ensure that the system is deployed in a way that matches safety and security requirements, such as ensuring that a physical system is deployed in an appropriate space, training people on how to use the system safely, or connecting system security mechanisms to appropriate external authorization and authentication mechanisms.

Operation (Chapter 29). The system in operation is expected to maintain safety and security. Continued operation requires maintenance of physical systems, and correct use by its users, who require training. [Leveson11] discusses operational safety and security in depth.

Uses monitoring and problem detection mechanisms to let people know when an accident needs to be resolved or when the system needs to be updated to address hazards that weren’t planned for. This process must be explicitly designed into the system; it is part of the system, not an adjunct.

System evolution (Sections 21.4 and 29.7). Safety and security problems identified in operation are fed back to evolve and improve the system. The team analyzes accidents and near-misses that happen, addresses new threats or safety problems identified after the system was implemented, and adds safety and security capabilities not addressed in previous development.

The analyses lead to change requests to update the system accordingly.

46.17.1 Use in fixing problems and upgrades

When making changes to a system, the new work must not undo the careful work already done to address hazards. The people making the changes need to know what hazards have been addressed, why they were included (and why others were not), and how they were addressed.

undisplayed image
Figure 46.9: How updates affect the security analysis

When an accident occurs, the first question will be whether the accident falls within the scope of accidents that the system was supposed to avoid. If the system was supposed to avoid the accident, then the next question is why the system happened anyway. Information about the hazards that system was designed to handle can help determine whether there is a hazard that was not considered, or whether the hazard was known but its mitigation did not work as intended, or whether there was an error in the safety analysis. If accident was outside the system’s intended scope and a decision is made that the system should be updated to avoid accidents like this in the future, then the analysis of accidents and hazards needs to be updated. In either case, the information in the analysis helps the team work out where to focus attention.

Similarly, when new harms, accidents, or hazards are discovered, perhaps because the system’s operating environment has changed or new threats have evolved, these can be added to an existing safety or security analysis. This helps scope what parts of the design need to change, which in turn informs estimates of time and effort required to make the changes. These costs help guide planning decisions (Chapters 67 and 68).

In the other direction, when the system’s design changes the system must remain safe and secure. The team uses the safety and security analysis to determine whether the design change invalidates some part of how a hazard was handled.

After making a change to the design of the system, the team must be able to show that the new version still meets its safety or security objectives. This means that there must be a new safety or security analysis, combined with evidence that the hazard mitigations will function as needed. It is far easier to update an existing safety analysis than it is to reverse-engineer a complete new safety analysis.

46.18 Artifacts and documents

There are several artifacts related to making a safe and secure system. Many of them are parts of artifacts already discussed.

The system and each component in the system has each of these artifacts. The ones for the system as a whole are generally far more complex than the ones for components. The artifacts for a component derive from the component or system of which it is part.

There are two kinds of safety- and security-related artifacts. Some artifacts are developed for other purposes and contain some safety content, such as the system purpose, concept, or design. Other artifacts make up the safety or security analysis, and stand separate from those other artifacts.

The safety or security analysis, made up of the definitions of harms, accidents, hazards, and mitigations, can be documented in many different ways, as I discussed in Section 46.14 above.

Table 46.1: Safety and security artifacts
Artifact Content
Safety and security objectives The objectives that stakeholders have for the system.
Part of: System purpose (Section 33.7)
Used for: Determining whether to build the system or not; validating system design; system acceptance; guiding system changes when needs have changed
List of harms The harms that the system is to avoid, including maximum acceptable rates when applicable.
Part of: System purpose (Section 33.7)
Used for: Developing and updating list of accidents; determining priority for hazards; determining whether design sufficiently addresses harms; adding to the list when unanticipated incidents happen in operation
List of accidents The accidents that the system should avoid, including derivation from associated harms.
Part of: System purpose (Section 33.7)
Used for: Developing and updating lists of hazards; determining priority of hazards; adding to the list when accidents occur in operation that were not anticipated
Threat actors (optional) For security analyses, the actors that might cause accidents, along with descriptions of their motivations and capabilities.
Part of: System purpose (Section 33.7)
Used for: Helping reveal what accidents or hazards might be caused and how to mitigate them; prioritizing security hazards based on threat actor motivation and capabilities; updating hazards and priorities as threat actors change
List of hazards The list of hazards that can cause accidents. Each hazard has a relation to the accidents it can cause.
Used for: Identifying and prioritizing problems that will affect design; checking that specification/requirements and design address the hazards; identifying what is affected by a change in harms, accidents, or threat actors
Risk analysis For each accident or hazard, an analysis of the associated risk (severity and likelihood), used to prioritize addressing hazards, along with the decision whether to address or not.
Used for: prioritizing which hazards to mitigate, thus guiding project plans; re-evaluating risk and thus priority as objectives, threat actors, or observed accidents change
List of mitigations For each hazard being addressed, indication of what parts of the design form the solution and how they mitigate the hazard. The general description of a mitigation is then linked to the corresponding elements in the specification of the affected component(s).
Used for: Determining how to update system or component design to address a hazard, leading to specifications/requirements
Concept The initial, high-level ideas for system structure and function provide the information for beginning safety analysis by roughly defining what components there might be in the system, from which some initial hazards can be defined.
Part of: Concept (Section 34.5)
Used for: Defining how mitigations might work at a high level; identifying dependencies and new hazards caused by mitigations
Safety and security specifications These are elements in system or component specifications, such as safety requirements or security requirements.
Part of: Specifications (Section 36.8), Requirements (Chapter 37)
Used for: Tracking how hazard mitigations must be reflected in component designs; designing verification steps to check whether the resulting components adequately provide mitigations
Design System and component designs implement the safety and security mitigations recorded in their specifications.
Part of: Design (Section 40.3)
Used for: Defining how a mitigation will be implemented; decomposing a high-level mitigation into the effects it has on lower-level components, and discovering the lower-level hazards that must be addressed in those components; mapping out the effects and costs of addressing a new harm or hazard; reviewing whether a design accurately meets its specified mitigations
Safety and security verification The evidence that the implemented system or component will meet its safety and security objectives, as recorded in its purpose. This is typically a combination of analysis and testing, with analysis showing that subcomponent behaviors and properties should address the right hazards, combined with testing evidence that the subcomponents in fact behave as needed.
Part of: Verification evidence (Chapter 54)
Used for: Checking whether a component or the system can be accepted for manufacture or deployment; after an accident, determining whether a component that likely contributed to the accident had a design or implementation error that was not caught by verification

When one is deciding on the form and format for these artifacts, it is important to consider how each one will be used. In addition to the initial analysis when first designing a system, consider:

The way the artifacts are recorded should help team members do these tasks. They should be able to work their way up and down the hierarchy of safety and security information to see what is related. They should also be able to find relevant kinds of information quickly—for example, if some component has misbehaved and caused an accident, people will want to find hazards that are likely related to that compoent quickly and accurately. A long, unstructured list of hazards will not help them.

Many of these documents need to record not just the bare facts or lists, but the rationale that lies behind the facts. The list of security harms, for example, needs to include not just the list of information to protect and the kinds of protection each requires, but why that protection is needed. In one recent project, we identified that the system should protect the confidentiality of authorizations to perform flight authorizations. Why some entities should be able to see which information about those authorizations involved finding regulatory requirements that governed some parties, or functional requirements for other systems that would access the information.

Similarly, the analyses for identifying hazards develop information about how one component depends on others. Keeping track of these dependencies is important for later checking whether a mitigation will address the hazard or not, or for evaluating whether a change to the system design will affect a hazard or its mitigations.