This post presents a conceptual model of flight test activity to provide a common framework for organizing test management and conduct.
First, a heirarchy of scopes and a set of decisions associated with flight test activity.
A couple definitions, drawn from abstractions used in discrete event modeling and simulation:
Event
A time-bound object associated with a single time
Examples of events: state transition, threshold transition, phase transition, indication of unexpected behavior
Activity
A time-bound object with a beginning, a middle, and an end, bounded by start and stop times
Activity start and activity stop are also examples of events.
Examples of activities: run, mission/session, campaign, project, program, portfolio
Scopes
The smallest useful scope of test activity is a “Run.”
Anything less than a “run” doesn’t serve as an organizing unit of action that provides value for meeting test objectives.
From there, it is often useful to label collections at different scopes with different names. The graphic shows 5 levels above run.
It’s important to note that:
the collections can be as few as one element, and there’s no upper limit conceptually, though too many elements may make organizing activities unweildly
the collections don’t assume a linear sequence of its elements, which may executed in parallel and with arbitrary spacing
Short descriptions of each scope:
“Run”
Beginning: Setup and Configuration Check
Middle: Test Maneuver
End: Recovery
“Operation” is a collection of runs and equivalent to a flight test mission, a simulator session, or a propulsion test sequence
Start: Brief and Pre-Operation Actions
Middle: Run Execution
End: Post-Operation Actions and Debrief
“Campaign” is a collection of operations, typically organized around a type of testing like low-speed taxi, first flight, flutter, loads envelope expansion, etc
Start: Campaign Kick-Off Meeting
Middle: Test Operations
End: Campaign Outbrief or Report
“Project” is a collection of campaigns, typically organized around the system under test like a suite of software upgrades, a new airframe, a new subsystem integration
Start: Project Kick-Off Meeting
Middle: Project Management
End: System Delivery
“Program” is a collection of projects, typically an ongoing activity organized around family of systems and their accompanying support components like “F-16,” “AMRAAM,” “787”
Start: Program Definition, Design, Plan
Middle: Implementation, Evaluation, Monitoring
End: Program Closeout and Resource Disposition
“Portfolio” is a collection of programs, typically organized around a set of stakeholders like “domestic programs,” “commercial transport,” “IRAD”
Start: Portfolio Definition
Middle: Portfolio Management
End: Portfolio Closeout
What would you call a collection of portfolios? I’ve seen “Group” but that strikes me as too generic. Most flight test organizations don’t have that level of scope, anyways.
Decisions
It’s tempting to conceptualize continuation decisions as a simple GO/NO-GO choice, as made popular in accounts of NASA launch status checks from movies and TV shows.
Complexity enters when there’s more than one “event” in play, more than just a “launch".”
An activity has a Before-During-After timeline. “During” is further decomposed into Beginning-Middle-End.
The consequences of decision timing can be very high in terms of safety, schedule, and budget, so explicitly defining terms can reap large benefits.
GO/NO-GO still have a place in the pantheon, but a more precise definition, and the addition of one more concept, will provide the basis for the different decisions available.
I believe this is a comprehensive list.
The 3 states below must be continuously assessed before, during, and even after the activity.
GO: All resources required for an activity are available and meet continuation criteria. PROCEED with activity.
NO-GO: Any resource required for an activity is unavailable or does not meet continuation criteria. ABANDON activity.
NOT-YET: One or more resources for an activity is not currently available or does not currently meet continuation criteria, but there is reason to believe the state will transition back to GO.
The continuation decisions are phrased as directive statements for brevity to indicate test team action.
The timing of the state change affects which continuation decision is made. The figure provides a visual of the timeline. In roughly chronological order, the decisions and their definitions:
CANCEL: A NO-GO decision to REMOVE the activity from the timeline
CONTINUE: A GO decision to PROCEED with the activity AS BRIEFED
CLEARED…: A GO decision to TRANSITION from BEFORE to the BEGINNING or from BEGINNING to the MIDDLE of the activity
RECOVER: A GO decision to TRANSITION from the MIDDLE to the END of the activity
SKIP: A NOT-YET decision to TRANSITION directly to the END of the activity before transitioning to the middle of the activity, without completing the BEGINNING or MIDDLE of the activity; often followed by an assessment of whether to attempt a repeat of the activity
HOLD: A NOT-YET decision to maintain current conditions and to standby for an update; may or may not be directly associated with the activity
ABORT: A NO-GO decision to CEASE activity and urgently JUMP to an abort procedure. Often triggered by meeting one or more safety-related abort criteria
CEASE TEST: A NO-GO decision to transition to the END of an activity EARLY before the middle of the activity is fully complete without urgency; often triggered by not meeting technical constraints or efficiency expectations
KNOCK-IT-OFF: A NO-GO decision that the activity is no longer relevant and to transition to safe condition
KILL: A NO-GO decision to disable the system under test
PAUSE: A SIM & TRAINING decision to pause the scenario to discuss or change the activity
RESUME: A SIM & TRAINING decision to resume the scenario as discussed
Very well stated and explained