Course on Applications of Probability Management

Naval Postgraduate School, Glasgow Hall 102, 1 University Circle, Monterey, CA 93943 (campus map)
Friday April 1, 2022

Schedule (all times PDT)

Core Concepts of Probability Management

8:00 AM - 8:15 AM: Introduction to Chancification - Sam Savage
8:15 AM - 8:30 AM: SIPmath 3.0 Standard - Sam Savage
8:30 AM - 9:00 AM: The Metalog Distribution - Tom Keelin
9:00 AM - 9:30 AM: The FrankenSME: Synthesizing Expert Opinion - Doug Hubbard
9:30 AM - 9:45 AM: Break

Defense

9:45 AM - 10:15 AM: From Ready or Not to How Ready for What - Connor McLemore
10:15 AM - 10:45 AM: Probability Management at Lockheed Martin - Phil Fahringer
10:45 AM - 11:15 AM: Chance Informed Aircraft Fleet Management - Steve Roemerman
11:15 AM - 11:30 AM: Break

Pandemic Modeling

11:30 AM - 12:00 PM: Making CDC Forecasts Actionable - Eng-Wee Ethan Yeo

12:00 PM - 1:00 PM: Lunch Break

Infrastructure 

1:00 PM - 1:30 PM: Assessing and Improving Operational Resilience of Critical Infrastructure - Dave Alderson
1:30 PM - 2:00 PM: Modeling Risk Probability for NAVFAC Construction Cost and Schedule - LCDR(s) Karl Coulson, P.E., CEC, USN; LCDR Tim Dahms, P.E., CEC, USN; and LCDR Scott Sobieralski, P.E., CEC, USN

2:00 PM - 5:00 PM: Breakout Rooms

Abstracts

Sam Savage: Chancification

The discipline of probability management represents uncertainties as data structures called SIPs that obey both the laws of arithmetic and the laws of probability. This enables analysts with statistical training to estimate probability distributions for use by non-experts in decision analysis applications, much as experts generate electricity for use by non-experts in lightbulbs and other appliances.

The current SIPmath Standard is based on storing potentially millions of Monte Carlo trials. This has now been enhanced by three complementary breakthroughs in simulation technology, the Metalog distribution from Tom Keelin, the HDR portable random number generator from Doug Hubbard, and the new 3.0 Open SIPmath™ Standard from ProbabilityManagement.org. Just as the shift from Direct to Alternating current in the 1890s led to widespread Electrification, the increased efficiency of these open technologies could lead to widespread “Chancification,” providing organizations with a more coherent approach to modeling risks and opportunities.

Tom Keelin: Introduction to Metalog Distributions

Introduced in 2016, the metalog distributions are now the most flexible and easiest to use of probability distributions. Metalogs make it simple to model expert assessments with a smooth continuous distribution, simulate it, and gain otherwise-unavailable insights from simulated or empirical data. With a single set of simple, closed-form equations, metalogs can displace traditional distributions (like the Normal, Lognormal, Beta, Student t, Gamma, and others) by being more flexible, easier to use, easier to interpret, and faster to simulate. The metalogs allow users to choose among unbounded, semi-bounded, and bounded forms; are fit to data with ordinary least squares; and enable Bayesian inference in closed form. This presentation provides an introduction to the metalog system, including strengths, limitations, and practical-use guidelines.

Connor McLemore: From Ready or Not to How Ready for What

Although the purpose of the Department of Defense (DoD) is accepted broadly to be “to provide ready and sustainable military forces to protect the nation’s vital interests,” the meaning of that statement is largely reliant upon the definition of the word “ready.” Yet it is generally unclear what it means to be ready. Ready for what? How ready? By when? To address this problem, we recommend the DoD adopt a simple, interpretable, and actionable data framework using stochastic scenario libraries that permit calculation of the probabilities of military readiness for specified missions at uncertain future times across unit types and military branches. It is based on the concept of auditable, stochastic scenario libraries long in use in financial engineering and the insurance industry. If implemented by the military, such a framework could allow mathematically coherent readiness estimates to better communicate “how ready for what” combinations of military assets are. Additional details can be found in our paper published in MOR Journal 2021 Vol. 26, #1, “Military Readiness Modeling: Changing the Question from ‘Ready or Not?’ to ‘How Ready for What?’”

Phil Fahringer: Probability Management at Lockheed Martin

Lockheed Martin is faced with numerous forecasting and decision making that are vital to our business, our shareholders and our customers – such as, forecasting revenue, forecasting inventory requirements for our fielded equipment that we support, forecasting maintenance requirements for fielded equipment, forecasting production and sustainment costs, deciding how to best invest in product and service improvements to maximize customer and business value based on how much budget is invested. All of these forecasts and decisions are layered with multiple levels of uncertainty around the assumptions and factors that influence the outcomes. Traditionally, we have relied on a combination of simple average values and applying some arbitrary level of additional uncertainty as a hedge to the final outcomes. To be sure, we have applied Monte Carlo simulation in certain circumstances such as inventory analysis requirements, but even this is heavily dependent on average assumptions. Primarily this has been because averages are well understood, and easily computed and updated, as well as managed and incorporated in analysis. Further – they are easily explained and stored and shared across the user community. It is well understood that using appropriate probability distributions for our underlying assumptions and critical factors for our forecasts and decision-making processes would yield superior insights. Unfortunately, probability distributions have not had any of the ease-of-use characteristics of averages, and therefore they have not been incorporated in many instances where they could potentially add tremendous value. But this is changing with the help of Probability Management.org and the vision of Probability Management. In this presentation I, along with my colleagues, will profile several use cases where we are employing and expanding the application of Probability Management, and we will discuss the anticipated value from these employments.

Steve Roemerman: Chance Informed Aircraft Fleet Management

Managing a fleet of assets requires a difficult blend of determinism and probabilistic thinking. It might be possible to agree 15% of an aging fleet will need engine replacement next year. But it will be very difficult to correctly choose which specific aircraft should be overhauled. The difficulties lie in the hyper dimensionality of the problem: deployment needs, unplanned failures, aircraft upgrade plans, congressional whims, and aircraft retirement plans. At the end of the F-111 fleet, the USAF flew newly upgraded Aardvarks from Grumman to the Yuma AMARG (Aerospace Maintenance and Regeneration Group) “the bone yard.” Careful study concluded prior USAF decisions made this the most economical alternative. Sadly, these problems are not isolated to the F-111 or the Air Force.

This presentation will discuss a “chance informed” approach, with illustrations from several Navy efforts, primarily featuring the management of the F-18 fleet. Benefits of considering a full span of uncertainty will be contrasted with point estimates. Benefits discussed will include improved accuracy, better risk information for decision-making, reduced human bias, and reduced organizational conflict.

Doug Hubbard: The FrankenSME: Synthesizing Expert Opinion

How do you combine the estimates of multiple expert opinions? How do you measure the performance of individuals, teams and methods in estimating and decision making? How do we know what works? The most popular methods of using teams to estimate are worse than just averaging different estimates or picking the best individual estimator. But improved algorithmic and behavioral methods exist which can make the estimates and decisions of a team better than the best individual.

Using over 60,000 trivia question responses gathered from training individuals to estimate probabilities, over 30,000 estimates of cybersecurity risks, and a review of extensive previous research, we analyze the performance of various methods for combining estimates of teams of individuals.

We build on this to propose an approach for the most important metrics that are usually never measured: the performance and ROI of decision-making methods and metrics programs themselves.

Eng-Wee Yeo: Making CDC Forecasts Actionable

Managing healthcare resources under uncertain COVID-19 surges is difficult and it is tempting to plan for surges in demand based on the average or best guesses of contagion forecast models. Unfortunately, this leads to systematic errors induced by the Flaw of Averages. This presentation provides the impetus and an approach to transform the uncertainty in daily COVID-19 hospitalization forecasts provided by the CDC into actionable data for making chance-informed healthcare resource decisions.

Dave Alderson: Assessing and Improving Operational Resilience of Critical Infrastructure 

In this talk, I present a view of infrastructure resilience that is tied to the operation (or function) of an infrastructure as a system of interacting components and that can be objectively evaluated using quantitative models. Modeling infrastructure operation in this way makes it possible to systematically evaluate the consequences associated with the loss of infrastructure components, and leads to a precise notion of “operational resilience” that facilitates model verification, validation, and reproducible results. Using a simple example of a notional infrastructure, we demonstrate how to use these models for (1) assessing the operational resilience of an infrastructure system, (2) identifying critical vulnerabilities that threaten its continued function, and (3) advising policymakers on investments to improve resilience.

Karl Coulson, Tim Dahms, and Scott Sobieralski: Modeling Risk Probability for NAVFAC Construction Cost and Schedule 

A data analytical design study to reduce risk on future Naval Facilities Engineering Systems Command (NAVFAC) construction cost estimating and scheduling.  Additionally, the proposed analytical tool can narrow a list of projects from NAVFAC's ieFACMAN database with the highest predictive risk for leadership to proactively mitigate project cost overruns and project schedule.

Speaker Bios

David Alderson
Professor, Operations Research Department and Founding Director, Center for Infrastructure Defense at the Naval Postgraduate School (NPS)

Dr. Alderson has been the Principal Investigator of sponsored research projects for the Navy, Army, Air Force, Marine Corps, and Coast Guard. He has held research positions at the California Institute of Technology (Caltech), the University of California Los Angeles, the Xerox Palo Alto Research Center (PARC), and the Santa Fe Institute. Dr. Alderson received his doctorate from Stanford University and his undergraduate degree from Princeton University.

LCDR(s) Karl Coulson, P.E., CEC, USN
Graduate Student, Stanford University

Lieutenant Coulson enlisted in the United States Navy in 2003 and after completing four years of service as an Aviation Electronics Technician (AT), LT Coulson attended the University of Washington and graduated with a Bachelor of Science degree in Civil Engineering. He was commissioned through Officer Candidate School as an Ensign in the Civil Engineer Corps and served in various tours in locations around the world. LT Coulson is a registered Professional Engineer in the state of Arizona, is qualified as a Seabee Combat Warfare officer, DAWIA Contracting Level III Certified and Public Works Level III Certified. His personal decorations include the Navy/Marine Corps Commendation Medal (5), the Navy/Marine Corps Achievement Medal, the Inherent Resolve Campaign Medal, and other personal and service awards. He is currently a graduate student in Stanford University’s Sustainable Design and Construction program.

LCDR Tim Dahms, P.E., CEC, USN
Graduate Student, Stanford University

Lieutenant Commander Tim Dahms graduated from the University of Wisconsin-Platteville in 2010 with a Bachelor of Science Degree in Civil Engineering with a minor in Business Administration. LCDR Dahms was commissioned in the United States Navy via Officer Candidate School in April of 2011. Upon completion of Civil Engineer Corps Officers School, LCDR Dahms served in various tours in multiple locations. Currently, LCDR Dahms is a Master of Science candidate at Stanford University under the Civil and Environmental Engineering Department’s Sustainable Design and Construction program with a management focus. Lieutenant Commander Dahms is a licensed Professional Engineer in the state of Wisconsin and Seabee Combat Warfare Officer qualified. He has also achieved certifications for Public Works Level II and DAWIA Contracting Level II. His awards include the Navy Commendation Medal (2) and Navy and Marine Corps Achievement Medal (2). He was also named the 2017 NAVFAC Marianas Military Engineer of the Year and recipient of the 2018 Society of American Military Engineers (SAME) National Young Member Medal and SAME Pacific Region Vice President Medal in 2018. He currently serves on the SAME National Young Professional Community of Interest as the Vice Chair for Credentialing and the Engineer and Construction Camp Community of Interest Liaison.

Philip Fahringer
Strategic Modeling Fellow, Lockheed Martin Aeronautics

Philip Fahringer serves as both an internal and external expert consultant providing strategic level decision support to senior business and government leaders to maximize revenue and performance, while assessing risk and controlling budgets. He applies the skills and disciplines of Process Mapping and Analysis, Discrete Event Simulation, Continuous Process Improvement, Monte Carlo Simulation, Portfolio Analysis, Optimization, Probabilistic Risk Assessment, and Interactive Decision Support Applications to develop and then communicate insights. He holds master’s degrees in operations research and strategic planning and is retired from the US Navy.

Douglas W. Hubbard
Founder, Hubbard Decision Research

Mr. Hubbard is the inventor of the Applied Information Economics (AIE) method and author of one of the best-selling business statistics books of all time, How to Measure Anything: Finding the Value of Intangibles in Business. He is also the author of The Failure of Risk Management: Why It’s Broken and How to Fix It, Pulse: The New Science of Harnessing Internet Buzz to Track Threats and Opportunities, and co-author of How to Measure Anything in Cybersecurity Risk. Mr. Hubbard’s consulting experience and financial analysis totals over 27 years and spans many industries including pharmaceuticals, insurance, banking, utilities, cyber security, interventions in developing economies, mining, federal and state government, entertainment media, military logistics, and manufacturing.

Tom Keelin
Managing Partner, Keelin Reeds Partners

Tom Keelin, inventor of the Metalog distribution, has combined a career in decision analysis practice with innovations to advance the field. Using decision analysis, modeling, and probabilistic simulation, he has developed asset valuation, portfolio management, and business development deal terms methodologies that have enabled greater success for dozens of client companies. Tom is a Fellow of the Society of Decision Professionals, and a founder and director of the Decision Education Foundation, a not-for-profit organization that helps youth learn good decision skills for life. Tom holds three degrees from Stanford University: BA in Economics and MS and PhD in Engineering-Economic Systems.

Connor McLemore
Principal Operations Research Analyst, CANA Advisors

Connor has over 12 years of experience in scoping, performing, and implementing analytic solutions. He is a published author on artificial intelligence and machine learning in peer-reviewed scientific journals and national security platforms. He is a former Naval Postgraduate School Military Assistant Professor and Operations Research Program Officer, former naval officer and graduate of the United States Navy Fighter Weapons School (Topgun) with numerous operational deployments during 20 years of service. Previously Connor was with the Office of the Chief of Naval Operations Assessment Division (OPNAV N81) in Washington D.C. He holds Master’s degrees from the Naval Postgraduate School in Monterey, California and the Naval War College in Newport, Rhode Island.

Steven Roemerman
CEO, Lone Star Analysis

Steven has served on the boards of a number of corporations, authored dozens of papers on technology and management, and he holds patents in the defense, telecommunications and energy sectors. Much of his work deals with large, complex systems, whether human institutions, computer systems, networks, or systems of systems. He holds a degree in Applied Mathematics with post graduate studies in mathematics, business, telecommunications and signal processing. He is a Senior Member of the IEEE, a Life Member of the NDIA, and a member of the SPE.

Sam L. Savage
Executive Director, Probability Management.org

Dr. Savage led the development of the open SIPmath Standard for storing probability distributions as auditable data and founded ProbabilityManagement.org, a nonprofit devoted to making uncertainty actionable. He is author of three books, Chancification: How to Fix the Flaw of Averages, The Flaw of Averages: Why We Underestimate Risk in the Face of Uncertainty, and Decision Making with Insight, as well as numerous articles. He is an Adjunct Professor in Civil and Environmental Engineering at Stanford University and a Fellow of Cambridge University's Judge Business School. He received his Ph.D. in computational complexity from Yale University.

LCDR Scott Sobieralski, P.E., CEC, USN
Graduate Student, Stanford University

Lieutenant Commander Sobieralski earned a Bachelor of Science in Aeronautical Engineering from Embry-Riddle Aeronautical University in 2010. He was commissioned an Ensign in the Civil Engineer Corps in April 2011 through the Navy’s Officer Candidate School and served in various tours in locations around the globe. He is currently pursuing a Master of Science in Civil Engineering at Stanford University. Lieutenant Commander Sobieralski is a Seabee Combat Warfare qualified officer, Registered Professional Engineer in the state of North Carolina and a certified DAWIA Level III Acquisition Professional. His personal decorations include the Navy and Marine Corps Commendation Medal (four awards), and the Navy and Marine Corps Achievement Medal.

Eng-Wee Ethan Yeo
Principal, Technology Risk Modeling & Methodology, Kaiser Permanente

Eng-Wee is responsible for modeling, simulating and assessing non-financial technology risks to support decision making. He has over 20 years of experience in Information Security. Prior to joining Kaiser, he was Manager of Information Security Governance at Surescripts and spent over 12 years consulting and serving as a trusted advisor to Fortune 500 companies in the financial and retail industries, working across a diverse range of security architecture, assessment, compliance, engineering, research, and risk-related projects.