Decison Stucturing & Indicators

• We’re gonna talk about decisions, but before we do that, how many of you all know one other person this room? Okay, so most of you have met one other person. Three other people before you came here? So a couple of people. More than five? Okay, and then, that’s hopeful, to just kinda get a sense, so a lot of you all are meeting each other. How many of you all have ever taken a policy analysis class or a decision analysis class? Okay, and then how many of you all would classify your research as being focused more on understanding and forecasting to better understand how the world works. More on the side of the basic science piece of it, how many of you are more focused on forecasting to directly support decisions, and you know what those decisions are? I think Allie, is that right? You’re, I think, one of the few academics I saw raise your hand related to decisions.

01:22 - Can you give me a sense of the decision that you’re working on? - [Allie] Yes, a lot of my work is focused on land use decision making and the implications of land use decisions to be affirmed. I just use transmission and anything practicable that landowners can do with their own property, like private landowners, to influence this interest in their environment. - Okay, awesome, and then are you working with stakeholders or other folks outside of the university? - I’m working with the Maine Woodland Owners Association, which is a forester, forest landowner organization, and I also work with a lot of social scientists on it. - Awesome, a couple of my folks that I had raise their hand that they work on more decision applications. Who were they again? Yeah, so, Jen, can you describe a little bit about what you do? - [Jen] Yeah, not to ruin my exciting legging talk. - It’s okay.

(everyone chuckling) 02:21 - - [Jen] I work in the Endangered Species Act Listing Program so pretty much all of our projects, or all the research in our projects, is geared towards answering the question whether or not we should list a species. - Okay, a couple of you all mentioned endangered species. There’s a couple of you all that work on that, is that right? Tony, you’re another one. Does anyone have another application? Justin? - [Justin] Yeah, I work on primarily malaria but disease intervention, so my research is on decision in part for maximizing effectiveness of actions toward malaria. - Okay, so you’re directly thinking about whether or not something that someone will do will have an impact. And then I think I saw your hand.

03:07 - Mike, right? - [Mike] Yeah, so we’re looking at decisions for mitigation and restoration habitat around the Great Lakes coastline for migratory banking or migratory activity for the bridges, and so– - Okay, well, fabulous. I mean, I think you’re getting a diversity. I mean, this is just a snapshot of some of the decisions that can be made, but, I mean, similar to what my team works on, I mean, what you’re hearing is that these are multi-stakeholder problems, they’re really hard, there’s science that’s really important, and it’s not always easy to understand how to best support those decisions. But I think one of the things that we’re gonna talk about is this links into forecasting, but it also is just, like, links into how you can make smart decisions in your everyday life. I mean, honestly. (audience chuckling) I mean, there are tons of really hard decisions, and I have used this process to decide whether or not to take a tenure-track position at an R1 university versus taking a postdoc with one of my academic heroes. Spoiler, 10 years ago I took the postdoc.

04:21 - It was a good decision with a good outcome. And I’ve used this to think about how to make choices about what vacation I go on, whether to do big moves. I mean, how to buy a house and what house you wanna buy given all the trade-offs of school districts, and price, and proximity of friends, commute time, all of these things that matter when you’re making these choices, and as soon as you get beyond one or two objectives, it just gets really complicated really fast. So today in the session that I have, they’re not very math-focused because it’s really hard to just think about how to do this and to structure it. It seems simple, but a lot of the thinking is just on getting the pieces right.

05:11 - The math of this is actually fairly easy, and just because I love books, I’ll give you a sneak peak. So Mike has in his book a chapter on ecological forecasting. We refer to it a little bit. Two of my favorites of many, my whole bookshelf if you want more. Individual decision-making, so you’re thinking about making smart choices for your own life. Links in, same process, as how we think about it for environmental decisions. Environmental decisions are just even harder. Here’s a great book that you can read literally on the airport ride home that will give you a big overview of things that we’re talking about. Thinking about this within the context of environmental management decisions with multiple stakeholders and how science feeds in, “Structured Decision Making,” related to environmental management. All of the examples in here are about the environment. There’s lots of other books out there, but this one I think breaks it down pretty well, and we’re talk about where forecast feed into this.

06:13 - Mike set this up beautifully because, you know, we wouldn’t really necessary need forecast if we could just have clairvoyance that would tell us exactly what would happen in the future. But forecast ends up being really critical to thinking about decisions for two reasons. One is we can’t perfectly predict the future, so we need forecast to help us do that. Part of the reason why we can’t predict the future is that there’s lots of choices that we as people make that impact the trajectory that we pursue. So if you think about climate projections, the largest uncertainty with those projections is what choices we make in terms of the greenhouse gases that we either emit or mitigate.

06:57 - It’s not as much the answer with the model; it’s the uncertainties with the scenarios and what we choose to do that will affect our pathway in 2021. So thinking about decisions, I think it’s really useful to break this down because we don’t have just science-based decisions, because inherently in any decision that we make, there’s science that’s part of it. But critical to a decision, and it can’t be divorced from a decisions, are our values: either our personal values or stakeholder values. What do we care about? That’s the whole reason why we’re making decisions, is we care about something and we want to maximize the likelihood that what we care about will happen. And then with especially environmental problems, and sometimes with our personal decision opportunities as well, there’s constraints that are imposed.

07:52 - Some of these might be regulatory, some of them might be geographic, but there’s constraints that have to be considered. So when we think about supporting decisions, there’s a lot of pieces that go into it. We’ve talked about the values. We’ve talked about models and projections. There’s clearly uncertainty associated with it. There’s information about sort of the environment and what’s possible, and possible responses that one might have. There’s considerations like cost and revenue that end up being particularly important. When we talk about decision support, it includes two pieces. We have and we will talk about some of these tools, such as scenarios and thinking about scenario planning, data management visualization, ecological forecast modeling, integrated assessment models: things that feed into decisions that help to aid those decisions. But just as important as these tools are, are the processes to think about how we make decisions. So these are processes where scientists and stakeholders work together to understand these decisions and problems and how to scope the problem space and the decision opportunities that you have.

09:16 - It’s thinking about co-production of knowledge. Ecological forecasters are one group of scientists that have a lot of knowledge, but if you’re forecasting in Boston for something in Guam, it might be useful to also ground truth some of the things that you’re doing with local knowledge of people who have lived there many generations, and so being able to think about how you build that local knowledge in. So part of what you’re doing is understanding how to co-produce knowledge together so that you can represent that fully within the decision models and choices that you’re making. Thinking about these ways of assessing the current state of knowledge and linking scientists and stakeholders together, there’s lots of literature on how to do this well and far more examples of how to do this poorly. So we’re not gonna get into a lot of these details, but just think about this: when we talk about decision support, it’s not just constructing a bunch of models that feed into a decision.

10:19 - It’s both how we set that process up to interact with scientists and stakeholders as well as how we build the scientific knowledge to actually make really smart decisions. And the whole idea of this is that if we do this well, we should have two things that result. One is you want good decision outcomes. The whole reason why you’re doing this is you’re trying to increase the likelihood of a good outcome. Can’t guarantee a good outcome, but you’re trying to increase the likelihood of good outcome, and that can, I hate this word consensus. I can’t believe I didn’t edit it out in the National Climate Assessment report before it came out.

11:00 - And understanding of the problem, the objectives, the option, and the scientific information that goes into it. So people being on the same page about how the decision is made so that then you have a understanding. It doesn’t mean that everyone’s, it’s gonna be a win-win option, but it means everyone understands how the outcome was, how the decision was made and then the outcomes that result as a result of that decision. The other thing is is if you do this well, you have process outcomes, and that means that you’re building and strengthening relationships between scientists and stakeholders, you’re getting buy-in for your decisions so you don’t run into roadblocks just because you’ve created this model. You’ve linked it with a decision model. It has output, your preferred alternative.

11:48 - Great, we make that decision, and then you have a bunch of other people that are like, “Whoa, whoa, whoa, whoa, wait! “Did you consider what I care about? “I wasn’t part of this process.” So really making sure that we’re thinking about both of these so that then you have outcomes that are achievable and good at the end. So during breaks and things, if you’re interested in some of the more process element things, we can have some side conversations about how to set that up and manage it effectively. So I think it’s really useful to then put some context around what decision analysis and decision science is. So descriptive science, we’re just gonna go out in the field and look at stuff and measure stuff.

12:31 - It’s literally, what do we see? What is it? And that goes all the way to normative science. How should we make decisions if we are rational actors? These are process-based, physical models. So these kinds of normatives of how should the world work given sort of a theoretical understanding. Experimental science, I think this is a bit we’re ecological forecasting. You’re creating experiments about the world within a computer environment and then testing out whether or not those happened.

13:04 - What I work on is prescriptive science, and that means that I’m really focused on how can we make, improve the processes to make better decisions. So how can we develop better methods, work more effectively with stakeholders, better integrate models to make better decisions. And so that means as we think about how we design and create ecological forecast models, there’s a couple of different ways we can do that. There’s the sort of scientist perspective of we want to better understand the world and so we’re gonna construct these forecasts, and we’re gonna push ‘em out into the world. So this is not dissimilar from sort of the weather forecasting model: We develop really good forecasts, and we push it out into the world, and other people figure out how to use it.

13:56 - There’s a purchase where what we do is we work really closely with stakeholders, we understand the science gaps that they have, and we design model or scientific information that directly feeds into what they need. What I’m gonna encourage you all to do as you all think about and work on your projects is actually a model where you think about both. So we think about the kind of science that needs to be developed to improve our scientific understanding, but we do that within the context of understanding what decisions may be influenced by that information and whether or not there’s a way of designing the models and constructing the outputs such that it’s relevant and useful for those decision context. In some cases, you may be working directly with a federal agency that wants to use that information. In other cases, the decision context might be a little further off.

14:58 - But if you’re thinking about the decisions in addition to the scientific understanding, I think it creates an opportunity to do more. Part of the reason why I think this is just so incredibly important is because this just links perfectly into adaptive management, because we have not fully realized this dream and this philosophy of adaptive environmental management, because it’s really hard. But Holling in 1978 put in his book when talking about adaptive management, “is concern with how to deal with qualitative “and quantitative data, “how to use this knowledge of fundamental processes “to construct models that can serve as laboratory worlds “for the testing and evaluation of intrusions, “or interventions, developments, and policies.” This is why models are so powerful in environment-decision making, as we can’t always monkey around with the world, and so we have to figure out a way of creating an invisible world that we can screw up as many times as we want until we figure out the best way of making some of the changes that will lead to the outcomes that we want to see. I mean, how many times have we talked about introducing species into ecosystems only to have them become invasive, ginormous pests? Models, (chuckles) expert knowledge.

16:43 - I mean, these are things that can be brought into decisions so that we’re not making poor choices because we’re not considering all of the implications of what might happen. We’re not scoping it too narrowly. So, with that, I think… One of the things that, I was trained as an undergrad as an environmental scientist, but I really wanted to work on policy questions, and what I saw when looking at policy questions is that so many of the environmental decisions that I saw implemented relied heavily on models. So I thought, “That’s what I’ll do. “I’ll do water quality modeling,” because I was super interested in freshwater systems and how to make them better. But my last year of undergrad, almost every person I talk to has a story like this, but here’s mine. (chuckles) So last year of undergrad, I decided school is really fun, and, guess what, they’ll pay you to go to school. That’s really awesome.

17:54 - No one in my family has ever gotten a PhD, but this sounds like an amazing deal because it’s the most money I’ve ever made in my life. Why don’t I go around and see if some of the people who I wanna work with are doing really cool things in the next five years and wanna take me on board as a student. So I went around and I basically visited Duke. Okay, so my decision process (laughing) for my PhD is not one I recommend to any of, because Yale was my backup and I only applied to two schools. So I went to Duke and I met with who became my advisor, Ken Reckhow.

18:33 - He works on water quality and water quality modeling but with a policy emphasis. I met with a couple of other people. One of the people I met with just said, “Everything that you’re describing “is focused on decision analysis. “Have you ever heard of this?” I was like, “No.” They’re like, “Read this book,” and it was a book written by Ralph Keeney, K-E-E-N-E-Y, no relation, but I’m sure we have some Irish roots somewhere. We just changed the spelling. And he wrote this book called “Value-Focused Thinking.

” 19:07 - So I ordered it and spend the weekend reading it, like, I don’t know, most undergrads, and by the time I got to about, I think chapter five is where the math started, and I was like, “I need to figure out how to do this “because everything I’m reading here “is a formal way to describe “how I think about the world and the decisions, “and how we can structure making good decisions.” So I say that because I think there’s two things with this, is, like I said in a lot of the sections that we’re gonna deal with, it’s not as focused on the math because I think the thinking about how to structure this is really hard, and I think it’s useful to kinda go through that. It’s also because I view math as means to an end, not the end of itself, and so I don’t learn through equations. I really learn through context, and decisions, and applications, and then the math is more just a way of getting there. Is there other people who kinda think that way? Okay, so I have a couple of fellow compatriots (chuckling) in the room. We’re gonna get along well.

(laughing) 20:25 - But I think the thing is is that this whole thing also applies to ecological models. So we’re gonna think about it within various decisions, but you all were making decisions throughout this week, and you’re making decisions about how to structure your model, what’s included, what’s excluded, how you think about the real world and how you’re abstracting it for models, and so you’re gonna making a lot of those kinds of decisions as you go forth. We’re just gonna break down a rigorous process to think about how to do that better. We can talk during some of the project time, if it’s useful, about how you can incorporate that in terms of modeling and thinking about that. So, yes, decision analysis, magic. It was really great when I was at Duke because part of the beauty of that was they had phenomenal environmental folks.

21:19 - I got to work with people in Jim’s Clark Lab and met Mike. He TAed my Bayesian statistics class. (laughing) But I was able to spend half my time in the business school with decision scientists who found environmental problems a really neat application. So (laughing), so it all depends on your perspective of what’s novel, but that gives us some sense of what’s interesting. I think the one other thing that I want to emphasize before I move to the process is we make a real strong distinction between decisions and outcomes. So I mentioned this earlier, what we’re trying to do is make really good decisions that we think will lead to outcomes, but what outcomes occur are based off of the choices that we make. They are only influence by those choices.

22:18 - Once I learned that, I can’t tell you how freeing it was because it meant that when I was making really hard choices about what home to buy, what jobs to take, all of these big things in life, if I had thought vigorously and made the best decision I could based off of the information that I had right now, then if it led to a bad outcome, there’s nothing I could have done to influence that because I’d gone through all of the steps that I thought I could do to lead to the outcome that we wanted. I think that’s something we should keep in mind as we think about environmental decisions because they’re even more complex, and it’s really tricky, and there’s a lot that we can do to make better decisions, but we’re still gonna have bad outcomes, and that’s an opportunity to sort of learn and adapt. When we think about our processes, that’s something that we can do. A lot of the diagrams that I’m using are coming out of structured environmental decision-making, largely because I want you to be able to use that as a reference if you wanna come back to this because you find it as awesome as I do. And so what we’re gonna do is we’re gonna focus on this method called PrOACT.

23:46 - PrOACT is basically the process that we’re gonna go through. It stands for problem, objectives, problem, objectives, alternatives, consequences, and trade-offs. Problem, objectives, alternatives, consequences, and trade-offs. So then when you map that, we’re gonna look at two conceptual models because they link to structured environmental decision-making and ecological forecasting. The way that they represent problem, clarifying the decision context, really understanding, what’s your decision choice? What’s your decision opportunity? Objectives, defining objectives and measures.

24:46 - So this is getting to both what do we care about and how do we measure what we care about. So objectives also includes performance measures, indicators, attributes, metrics, criteria. Okay, these are all words to more or less indicate the same, that are pointing to the exact same thing that we’re gonna be talking about. So you’ll see these words: criteria, metrics, attributes, indicators, performance measures. If we use those words, they’re more or less the same thing, but you’ll see it in structured environment decision-making and the ecological forecasting book as performance measures.

25:38 - Developing alternatives, here’s our choices: estimating the consequences. Here’s where ecological forecasting comes in, really understanding what’s gonna happen, whether it’s predictions or projections. Evaluating your trade-offs and selecting, so how do you compare your different choices? And then really thinking about this within an adaptive measurement context because with environmental decisions, like, guarantee, we’re probably not gonna get it right the first time. And even if we do, we wanna show that we did. (laughing) So developing processes to monitor review, this is also where iterative ecological forecasts come in because as we’re collecting new data, we should be updating our forecast and understanding how they both improve our forecast but also how that feeds into these decision processes. Mike, in his book, has similar layout.

26:34 - I’ll highlight two things that are different, but they map here as well. So generate alternatives and scenarios. So the reason why scenarios are brought in here is if you’re thinking about ecological forecast, the boundary conditions matter. So you can’t think about the consequences if you’re not controlling for the conditions that you’re trying to set up. So the land use scenarios, policy interventions, you have to be able to be comparing similar conditions in order to really make smart trade-offs. And then forecasting consequences. Consequences and decision models involve forecast, but there are a lot of objectives that are outside of the domain of ecological forecasts.

27:26 - There are objectives related to aesthetics and recreation with our national parks. There are objectives related to cultural values associated with a lot of our lands that are on current or former indigenous locations. All of those things matter, and there’s space for them in decision models. It’s just one of those things where you can’t just forecast something and expect it to give you the answer ‘cause it’s one piece of this larger decision that we’re making, and there’s a lot of objectives that we need to care about when thinking about those choices. And then evaluating trade-offs. I would say with this, there are, but not represented in this diagram, these loops related to adaptive management and iterative forecasting. So just keep that in mind.

28:20 - It looks linear but it’s really everything’s a giant circle. So I thought it would be useful to actually go through an example. We’re gonna think about a decision of a flight to ESA and breaking down how we make that decision. If you’re not going to ESA, any kind of professional meeting. Let’s get a sense from everyone about what you think about when making a decision about a flight. Jeff, can I start with you? - Sure, I usually think first about cost. - Okay, great, cost. Ian, is that right? - [Ian] Yeah, how much grief you get from the PI for how much you spend on it. (everyone chuckling) - Okay, so cost plus perception of (chuckling) other people, like grief from the PI. Rushi, what do you think about? - [Rushi] What time I’m leaving and arriving. - Mm-hmm, so the amount of time. Is it Jenny? - [Jenny] It there’s a lot of layovers, just length of layover.

29:32 - - So if it’s not a direct flight, whether or not there’s layovers– - Yeah, layovers. - And the time. - [Jenny] Where you might eat, yeah. - Yeah, Whitney? - Mm-hmm. - Okay. - [Whitney] I broke cost up into a couple of categories ‘cause these days you often have luggage fees in addition to your airline ticket, and sometimes you have multiple choices of airports, like going to Washington, D.C. area, so you might consider your destination and how that relates to public transportation, or Ubering, or additional costs to get to your hotel. - Excellent. You know, additional fees like baggage fees as well as if you have a choice of airports, almost like how convenient it is to where you’re meeting and the like. Leland. - [Leland] Unsure of who’s paying for it. So as a grad student, I would always book the cheapest and with the most horrible layovers as possible, generally, just because I was saving cost and saving money, but now that I have some grant funding, I can spend a little bit more.

30:34 - I’ll spend more (Melissa’s laughing drowning out speech). - Okay, so who’s paying for it, which I think can be an important factor. - Alice. - Alice, thank you. - [Alice] Maybe pre even flight decision, is driving an option? I live in California and I looked at a rent-a-car price. - Yeah, so we’re gonna come back to that, Alice, because– - [Alice] I gave you an alternative. - No, you’ve reframed the decision. What you’ve done is you’ve changed it from a decision about what flight do I buy to how do I travel to this meeting, which may create both some additional options but may also introduce some additional objectives.

31:22 - So we’ll come back to that ‘cause I actually wanna build that out, but for right now, we’ll focus on the flight because the travel adds a additional layer of complexity. Naomi. - Yes. Sometimes I think about the airline. I think, “I might be stuck flying Spirit Air,” ‘cause I waited so long to buy my flight, which is really suboptimal. - And for what reason is that suboptimal? - Because– - Because they’re the worst. (Melissa laughing) (everyone chuckling) - [Naomi] There’s fees for everything. Have you ever flown Spirit Airlines? - No, no, and there’s many reasons I have not.

32:04 - - [Naomi] You’d know why it’s suboptimal if you had. - Right, so lots of fees. I think I heard something about– - [Naomi] The seats are super small. - Okay, so there’s a factor of, yeah, uncomfortable. - You can’t use your laptop because the tray table is like this big, so you can’t do any work on the plane. I’m usually doing it on the way to the conference. It’s usually to finish my talk. - (laughs) No judgment.

(laughing) 32:32 - Okay, okay, so, yes, whether or not that this space will actually be comfortable to do things. Yeah, Justin. - [Justin] Kinda related to that, what I need to bring. Do I need to bring a poster? Do I need to check a bag? Things like that. - Yeah, so what do I need to bring and then what’s associated with bringing that stuff. Other things, yeah, Winslow. - For ESA specifically, I think about what is the probability that I’ll get scheduled for a Friday morning time.

33:02 - (Melissa laughing) That weird half day session. - Okay, yeah, yeah, which I’m guessing you’re trying to buy flights before the program is even put out, so there’s an additional layer of uncertainty. You might be able to buy a cheaper flight, but you– - Or get home at a better hour. You know, when you get home at midnight, but, like, six in the morning instead. - Yup, okay, so considerations of the meeting and the meeting time and then certainty associated with that. Other things, yeah, Toni.

33:35 - - So in addition to the airline, do you wanna collect miles for a certain airline? - Yeah, and that can be, that can be particularly important for some people, so miles or loyalty and implications of that. Yeah, Mike. - [Mike] So if you happen to be somewhere at a particular time or home at a particular time, you have to sort of backtrack which flights are gonna work in that context. - Right, so you’re thinking almost of the constraints that end up happening as a result of– - [Mike] Yeah, so if you have to take a bus two hours in the city or something like that, you have to sort of figure out what the last bus is, and then that’s the last time you can get in, et cetera. - Yeah, yeah, so figuring out, yeah, all of the additional linked decisions that end up happening as a result of a choice of a flight. Zoe, right? - [Zoe] Yeah, I think how jet lag will affect your seat schedule for a red-eye flight, overnight, or if it’s a time zone change.

34:44 - - Yeah, so not just the amount of time that you’re traveling, but the time that you would arrive and how that then affects your ability to be productive during that time. Other, yeah, Whitney. - [Whitney] I’m often traveling with other people. It’s also I have labmates and my husband going to the same conference. Then it’s almost another can of worms, where it’s my preference for each of these decisions, but it’s also theirs as well. - Yeah, yeah, and you can think about that in two ways.

35:16 - You can think about that as a constraint, so what’s gonna be feasible for other people and how does that constrain what’s possible, but you can also think about that as almost like a couple of individual decisions that then you’re trying to develop some consensus and negotiate what’s the best choice for everyone where it might not be optimal for any one person. So, yeah, you’ve basically created a giant complexity in terms how we would structure a decision model (laughing) in that context. (laughing) I love ecological forecasters. They always think about these (chuckling) much more complicated situations. Toni. - This may be more relevant for me, but whether or not you’re flying a domestic or an international airline, if you’re constrained by– - Fly America, yeah, we have that with our grants as well. - When you route through Japan, (everyone chuckling) you think about your options.

36:14 - - Yeah, any other things people want to add to this list? - [Winslow] So one thing I actually think a lot about is the ethics of the different airlines. So ever since United pulled that person off the airline, I’ve avoided them, so I try to stick to that. - Okay, ethics of the airline. And, Jake, I think you had another one? - [Jake] Yeah, I think about legroom a lot. (everyone laughing) - You know, funny, that doesn’t factor into my decision too much. (laughing) (everyone chuckling) - Shorter flights and I won’t care as much for this one, but I’ve paid extra just to get some legroom. - You capture things, that’s swell. - Yes, well done, well done, Chris. This is only a decision about a flight.

(laughing) 37:05 - And most of the time, if you travel a lot, you don’t necessarily go through and brainstorm all the things that you might want to consider. You probably have developed some mental shortcuts around these choices or potentially might be constrained. You might have narrowed the decision context a little bit so that certain factors may not come into play. But this kind of (laughing), layovers stink, I love the commentary. (laughing) So let’s take this and actually think about how they link to objectives, because this is brainstorming, and what you see here are a lot of things that matter, but they’re more means to what we fundamentally care about.

37:56 - So let me take a couple of colors, and let’s circle a few things. So let’s do the easy ones. Okay, I see things about cost. Okay, we see things about the cost of the flight. We see this about the sort of grief about the boss, but we’re gonna keep that a little bit separate. There was one about baggage fees around here. Actually, Chris, can I ask you to come up and circle where these are (chuckling) ‘cause I think you’re gonna be able to find them faster than me.

38:29 - - Yeah, someone mentioned cost again, and they’re like the associated cost besides the flight, so that was kinda there, yeah. - Okay, so one of the fundamental objectives that I’m seeing, and it might show up in other places, but how do we minimize costs? And that can be the cost of the flight, but it can also be associated with baggage fees. You might also think about it also in terms of partially related to airline miles because you can actually calculate, basically, the monetary worth of an airline mile and that could feed into your linkage to cost. So if it’s an airline that you fly frequently and you’d actually get value out of that versus an airline you don’t fly frequently, you can actually calculate, you know– - [Chris] You want me to circle that? - That, sure, yeah. We’ll give it a half circle ‘cause I think there’s another piece that goes into that.

39:25 - Okay, so minimizing cost as one of your key objectives. What do you all see as another objective? - [Woman] Comfort and convenience. - Comfort and convenience, excellent. So that gets into this: airline quality, the small seats, the legroom, how horrible the airline is. I think ethics is different. - [Chris] More legroom. - Legroom, there we go. Another sort of fundamental objective that you see coming out of this list. - Time? - Time, yes. Time, and, yes, layovers, the amount of time that the flight takes, whether or not there’s additional time that’s added based off of the airport that you choose to go into. - [Chris] These are kind of the same.

40:20 - - Mm-hmm, so the amount of time that it takes. - [Woman] Jet lag goes well. - Yeah, I think jet lag fits into that, but that might also be something where we wanna think about how we consider that and measure that. I think that’s a really important one. So lets think about it in terms of time, but we might actually pull it out, relate it to something else. Other objectives that you’re seeing in here? - [Naomi] Maybe maximizing your productivity or maximizing what you get out of the conference? - Okay, in what sense for the choice of an airline flight? - [Naomi] For more like the jet lag thing. That’s what I was thinking about. Or coordinating with your coworkers or deciding flights before knowing session day and time.

41:11 - - Okay, excellent, so maximizing productivity during the conference. - You want me to circle that one? - [Naomi] Or just what do you get out of the conference? - [Chris] Oh, I already wrote that. - Other objectives that you see? - [Allie] Maybe considerations involving other people in some fashion? - Yeah, so I think that was linked into the productivity, but I think you’re thinking about it slightly differently, Allie. Can you give me a little bit? - [Allie] The ones involving the boss and the boss’ reception as well. - [Chris] Oh. - Yeah, so almost like, so you can frame it as a constraint.

41:54 - You can also frame it as an objective of almost minimizing– - Clientation? - Like social irritation? - (laughs) Social irritation. (laughing) We can wordsmith these later, but, yes, something about minimizing social irritation. (laughing) Which I think is a good one because if you think you’re gonna get grief from the boss about certain things. - [Whitney] Would the Fly America go into that as well? Kind of in the theme of making the higher ups happy? Or following rules that are set from above? - I would consider that more of a constraint, Whitney, and the reason is is that it limits the number of options that you can pursue, yeah. And I think there’s one other one that relates to Winslow’s comment.

42:50 - - [Winslow] To ethics? - Yes, the ethics. - I circled that one. - Maximizing sort of a ethical considerations. Exactly how they get worded, usually you come back and wordsmith. So let’s actually erase this. Can I ask for, thank you, Chris. (clapping) - You’re welcome. (audience applauding) Can I ask for another repertoire for this? Okay, so let’s actually take this brainstorming and let’s build out our objectives. So let’s actually list our objectives. This is what we’re trying to create. This isn’t the only way that you can construct things, but it’s super useful. So what we do is we list our objectives.

43:43 - So these aren’t things like legroom; instead, it’s things like maximizing comfort. We have our performance measures, which we’re gonna go through. We have the units and where we might find the information. We also sometimes indicate the magnitude if we haven’t included it in the objective or if our performance measure is different. So you’re trying to maximize legroom, but the way that you measure it may have a different relationship. And then various alternatives up here.

44:19 - Okay, so we’re gonna start with our objectives. So we had minimize time, maximize comfort, minimize social irritation, and then maximize ethical behavior. Okay, so these are what we call fundamental objectives. These are things that we care about because it is fundamental to our decision. Now how we get to that, part of what we’ve talked about are means, and this gives us a sense of potentially some of the performance measures we would wanna use in order to evaluate those different objectives.

45:01 - For minimized cost, what are some of the ways that we could measure whether or not time, sorry, cost has been minimized? - [Man] Dollar spent? - Dollar spent. And dollar spent for what? - [Man] Even from door to door. Literally walking around your house, all the cost you incurred, including the plane and baggage fees, and then coming back. - So you would wanna include the flight costs, the drive or taxi costs, parking, baggage fees, other fees that Spirit Airlines adds on to things because they’re really annoying, but basically your door-to-door expenses, and when you’re doing this, sometimes it’s helpful to just make sure you’re listing all of those things just so that if someone else were to look at your decision model, they would understand all of the factors that you’re considering. It’s not just your flight cost, it’s also the door-to-door cost.

46:04 - Minimizing time, how would you wanna measure that? Yeah, Jenny. - [Jenny] I guess how many layovers you had, or if, yeah, you could avoid, just have a direct flight. Actually, yeah, what time you had to wake up to catch your flight, sometimes to not have a morning flight. - Okay, so I’m hearing things related to the total time that you’re spending flying, and then I’m hearing something related to the, yeah, the convenience, ‘cause I think you’re hearing things with comfort, comfort related to me personally with the flight times that might constrain the options that I consider and considerations once I get onto the plane. We had a bunch of ideas around maximizing comfort that were highlighted: legroom, comfort of seats, size of the drop down table, yeah, the tray, things related to sort of the service that you get from an airline. Yeah, Ian.

- So in some of these, 47:12 - they’re potentially qualitative. Are you gonna talk about how we deal with qualitative and quantitative measures? - Yeah, so maximizing comfort, I think this is what you’re getting at, is there’s things that you could do to quantitatively measure each of those. - [Ian] Like inches of seat or something like that isn’t maybe how we think about it, and just some of the more inherently qualitative. - Yeah, so let’s actually use comfort as a way of doing a constructed scale. So we think about multiple things with comfort.

47:43 - Some of those we can measure, some of them we qualitatively assess, but what we could do is we could do a constructed scale of Spirit Airlines to, like, you know? - [Ian] Or like window/middle/aisle seat. That’s not easily quantified. So maybe something like that. - Yeah, Zoe. - [Zoe] I feel like the comfort during the flight might be a separate scale of the disturbance it causes in your life before and after the flight, like the waking up early sort of thing. - Okay, so I think what you’re then highlighting is that there might be two indicators or performance measures that we might want to use. So one of ‘em might be sort of a disturbance indicator. You know, how much it disturbs our life both in terms of traveling but also once you get there, like whether or not you have jet lag or things of the sort, and then the second is a comfort.

48:42 - Like once you’re on the flight or that flight experience, how comfortable it is. With this, there are certainly things that one could quantify, and you could list out lots of different types of indicators and be fully comprehensive, but for this example, let’s just think about it as a qualitative scale because I think most of you all, when you were talking about this, you were like, “Well, I kind of know with certain airlines “how comfortable it’s gonna be.” Emirates Air, it’s gonna be pretty luxurious. Spirit Airlines is going to be pretty hideous. So we can think about that in terms of a qualitative scale of maybe a one to five ranking of one being the worst and five being the best.

49:31 - You could also think about that with the disturbance indicator, so whether or not the flight is gonna have a negative impact on all of these other factors that you care about. Yeah, Naomi. - [Naomi] I feel like I could put a dollar value on those things by some sort of willingness to pay. Well, because I have, like I don’t necessarily have it clearly articulated, but when I looking at flights, it’s like, “Okay, well, Delta costs $200 more than Spirit. “I won’t pay that much more to go on Delta, “but if it’s $50 more, I would pay to go on Delta.” So you can put a dollar on those comfort things of what premium will you pay for them.

50:14 - - Yeah, which is I think, Naomi, you’re jumping ahead to how do we think about trade-offs? And that’s not a bad thing. Some of it you can do through dollar values, but some of it you can also just do through the natural measures or the proxies that you’re using. But you’re basically saying, “Okay, I know how to trade off costs and comfort.” - [Naomi] Yeah, well, more so than I know how to say, like, “Rate these things from one to five.” Maybe it’s just ‘cause we go straight to the trade-offs when we’re thinking about these things and thinking about it in this way. One to five is maybe less natural.

50:50 - - Well, and I think here’s where it comes into environmental decision-making. Okay, so we can jump ahead and think about all of the alternatives and how we would make these trade-offs, but that’s really easy to do if it’s me. I know my preferences and values really well, but I’m sitting in a room with 20 other people who have slightly different… We might we might agree on these broad objectives. This might be something that we kind of agree.

51:25 - We might be able to create agreement on how we would measure it. There’s choices that we have, but the trade-off comes in is the way that you would trade off cost versus comfort is not the same way I would trade off cost versus comfort. And so that process of how you do that, there’s an element of wanting to be transparent and rigorous of that process, but there’s also an element of, let me, yeah, let me just stop there. I think it’s really important especially for environmental decisions with multiple people involved and a lot of scientific information to be really clear about how you’re developing the science to make these assessments, but also how you’re making those trade-offs, because you may choose to fly Spirit Airlines because you’re like, “Ugh, it’s hugely uncomfortable, “but it costs me, you know, “it’s $200 less, so that’s worth it to me,” whereas I would be like, “No.” (laughing) And that’s absolutely fine but what you would want is if you’re constructing this and multiple people are using it, it may not be, it’s not necessarily important when we think about how am I going to make a personal decision to fly to ESA; it’s really important when we think about how do we make choices about how we manage our environment because what we want to do is keep each of these steps transparent and separate.

53:00 - We wanna be able to be really clear about these fundamental objectives that we care about. We wanna be really clear about how we’re going to measure those, not calculating what those are, not making assignments, but what’s the way to measure the things that we care about. We wanna think really clearly about what our choices are. One of the reasons why we don’t start with our choices is for two things. One is this is where advocacy in a bad way comes into play, of like, “I have predetermined what I want to happen, “and I will find facts “and ways of justifying the choice I have already made.

” 53:41 - When you think about your goals and objectives, it creates an opportunity to expand the way that you think about your alternatives. We see that, actually, in the example that Alice pointed out, of like, “Well, I might not fly. “Could I could I drive instead?” That might be a more environmentally sustainable option which might be another objective that I have when making a choice about how to travel to a meeting, and that expands my option set of both driving, it might also include options like trains and other things. So that a lot of the reason why we go through this process, is not because for these kinds of things, it necessarily helps us to make a better decision, but it helps to break things down so that then when we’re dealing with even more complex things, with multiple people in the room, and a lot of really uncertain science, and we have people coming at this and calculating consequences in different ways, we can point out where the fallacies are in the decision-making process. Is it because we’re having conversations about fundamental values, and we’re talking past each other because we haven’t actually had the conversation about the things that we care about? Is it because we’re thinking about what to measure? And we’re thinking about what to measure in really different ways, and so we’re not actually able to compare and talk about things.

55:07 - Are we bringing different science to bear or the different boundary conditions or other factors that aren’t gonna lead us to think about these things concretely? Are we thinking about the full range of alternatives that that can occur? So this is part of the reason why we wanna break it down like this and not jump to making the trade-offs. So, clearly, these are important. We’re just gonna assume that they’re basically the same. So we’re gonna give ‘em the same number so they don’t end up factoring in. So let’s look at three alternatives. Flight one is gonna be $600, flight two is gonna be $300, so we know what Naomi is gonna pick, and (chuckling) flight three is gonna be 650. In terms of minimizing time, this is gonna be total flying time, it’s not gonna be total door-to-door time, but what we’re gonna assume is that the door-to-door time is the same so that if we just calculate total flying time, it’s a perfect proxy.

56:17 - It’s a perfect substitute for thinking about total time. So this is three hours, this is six hours, and this is 10 hours, and then maximizing comfort. So we’re going to do this on a constructed scale, so one is the worst and five is the best. This is four, this is four, and this is a new airline that we don’t really know, and we’ve never flown before, and none of our friends have flown, so we literally have no information, so all we can say is that it’s somewhere between one and five because I really don’t know. Giant question mark. Giant question mark. Let’s just focus about it as the during flight experience.

57:01 - This disturbance indicator, really important. We’re just gonna put this as one we’re not gonna consider right now, or we’re gonna assume it’s exactly the same. That’s just so that we don’t get too far down the rabbit hole before talking about environmental examples. So, with this, we have three options. We have three alternatives. I find it super easy, and this is actually something they do in the military. We’re just going to put a green around the best in all cases because it’s not always the lowest or highest number. These are kinda the same. And then the worst.

57:45 - This, this, and I have no idea what that is. So what we can do from this is let’s first eliminate objectives that don’t matter because we have figured out some of these are more or less the same. Okay, so we’re not talking about minimizing social interaction because we’ve given that the same score, and ethical behavior is the same even though we know it’s not. And, similarly, we’re not gonna think about disturbance. That said, here, what we see is that these two are exactly the same, and this one we don’t really know much about.

58:30 - So I would say for this one, we can’t really make an assessment of differentiation of levels of comfort between the airlines, so we can eliminate that objective from consideration. Now we have, you know, three alternatives into objectives. What we can see is this is a worst on all of ‘em, so this one’s dominated. That means there’s no way that this would ever be the best choice so we can eliminate immediately. Jeff. - I did question about you just wind up maximizing comfort, but it seems like you could potentially say, “Well, how risk averse are you? “I really feel my best and loose. “I might attack it and go for it.

” 59:12 - - Yes, so this gets into, in this case, we we’ve simplified the problem and created it sort of as a discrete entity, but, yes, if you’re risk averse and you’re like, “Hey.” It’s maybe the new Emirates. Sure, let’s try it out. Maybe comfort is really important to me. No one knows anything about it, and I can learn a lot. So, absolutely, there are cases where that level of uncertainty is not necessarily a bad thing because being able to have an experience where you would have a comfort level of five is worth it. We’re gonna say for a group like this, we’re not gonna consider that, but you’re absolutely right. It absolutely depends on how risk averse you are and what you see in terms of the value of learning.

00:03 - So once you have a set of, you know, basically you’ve constrained your set of options because you’ve eliminated objectives that don’t matter or you’ve eliminated options that are dominated. So, in this case, we are risk averse. We’re not risk taking. We are gonna eliminate this option from consideration. This comes down to a trade-off between cost and time. So given your own values and whether or not you typically have to pay for your travel or it’s coming off of something else, how many of you would take the $300 flight that would be six hours? How many of you would take the $600 flight that would be three hours? And so, yes, what you’re seeing here is that people would make different choices, and let me give you one of the cases where I actually would consider another objective when making this choice. So if this is coming out of my personal funds, and I do not wanna spend a huge amount of expendable income on professional activities, I would probably choose this cheaper flight.

01:33 - However, if I’m doing consulting, and the consulting company pays for my travel time, and my hourly rate is $200 an hour, this is a better option because it’s cheaper overall for that company, so you can then think about almost how much your own time is worth in certain situations, which may be different depending on who pays and the context of how they’re paying, to factor into how you would make these choices. So this is breaking things down in a particular way. The thing that’s really powerful about this is it separates the scientific pieces from the values, and it recognizes because we’re making decisions that values are equally important. You can’t make decisions without ‘em. It also means that the science that you’re developing, the information you’re developing, directly relates to the objectives that are relevant in the decision. Doesn’t mean that if you’re developing a forecast and someone’s going through this kind of rigorous process that they wouldn’t pull that forecast in, but if you’re designing specifically for a decision, you’re pulling in information that that fits in and helps you to understand the likelihood of being able to achieve that objective.

02:57 - So we talked about a few things because Alice brought up this question of travel and that can bring up additional objectives. So we talked about almost environmental sustainability. So you could do that related to greenhouse gas. and the equivalent greenhouse gas emissions that would occur as a result of that travel choice. Yeah, so I think with this, with the decision framing, if you’re expanding it, part of what you’re oftentimes doing is potentially adding more objectives that you care about and increasing the number of alternatives that you would consider, so you’re considering alternatives you wouldn’t. Naomi.

03:42 - - [Naomi] The consequences step, is that the $600 versus 300, and the three versus six, those are the consequences? Like, what are the measurable outcomes of taking alternative one versus alternative two? - Absolutely. - So that’s the forecast? - Right. - In this sense we’re not, it’s just how much the flight is. - Right, I mean, in this very– - [Naomi] Like think of the odds of a delay too or something. - Absolutely, I mean, you can construct these probabilistically because it’s also, like, maybe you’re buying your flight within a week and so you’re trying to get a sense of what flight you would wanna buy. There’s uncertainty about whether prices would jump or potentially get get lower, and you can look at the volatility in prices by looking at KAYAK or some other sites like that.

04:28 - And you can look at the time because flying through Chicago in winter. You know. (scoffs) Yeah, exactly, exactly. So you also consider those aspects because it might be three hours, but if it’s January and you’re flying through Chicago, you might wanna add some uncertainty bounds on that because that might be a minimum time, and it might actually be a couple of days before you get out. The Chicago airport is not very comfortable. (chuckles) So, yes, yes, we’re looking at it deterministically, but, yes, you can absolutely look at it probabilistically and through that lens. Chris. - [Chris] So this example is kind of boiled down to this simple trade-off of things.

05:17 - How heuristically or maybe algorithmically do you kinda do these when you have more than two things on the table? ‘Cause my brain starts to freeze up if you change those ones on the bottom to the numbers. - Yes, dealing with more than two things is super difficult. It wasn’t a problem that was solved until 50 years ago. And so the short answer, and we’ll actually talk about this more on Friday, is what you end up doing is using something called an index of desirability, also talk about this in terms of utility, and what you’re trying to do is normalize these different objectives on a scale of zero to one, or zero to 100, whatever. Some kind of normalized scale so that then you can say, okay, let’s see, 300. 650. This is the most desirable. So this is gonna be one. Least.

06:30 - So with this, 300 is the most desirable, 650 is the least. The question comes in of whether or not a change from a dollar of 300 to $301 is the same as a change from 649 to 650. So is this linear or does this have a different shape to it? - [Chris] So it’s more than just a simple normalization. You’re actually, it’s like the step, yeah, there’s just function as you go from worst to least. - Yeah, and so with money, you know, with companies and large decisions, they should have a linear index of desirability.

07:15 - A dollar here and there should not matter when you’re dealing with billions of dollars of corporate revenue. It may matter a lot for our own personal finances and decisions. And then, similarly, so comfort is on this qualitative scale, and I think, Ian, you were pointing out qualitative skills are really important, but the difference between one and two may not be the same as the difference between four and five. So you might think in terms of comfort, being able to get from a one to two is a big jump. This me having my knees up in my nose versus being able to have ‘em against the seat, so that ends up being a big jump, but from a four to five, not so much.

08:08 - So it might have this this function where you get a lot of gain from going from one to two, but four to five, pretty minimal. This comes into play when you make trade-offs because what you will eventually wanna do is be able to normalize based off of the utility or an index of desirability, and this is not the same for everyone. So for this jump in comfort, it might matter a lot to Jake because he’s really tall, I’m not so much, and so this is likely to look different for me for a one to two. I probably am still not touching the seat in front of me. The thing about this is is that you’re constraining it based off of best and worst case scenarios, values for this. It’s not infinite; it is constrained.

09:07 - This is all based off of preferences; this is not scientifically determined. And so when you get into trade-off models, there are rigorous scientific processes that you can use to construct and make those choices, but if any decision analyst tells you it’s a scientific determination, they’re fooling themselves because this is all, like, the way that you translate these values is based off of what you care about. And so we talked about this in terms of even swap, so this is what you’re talking about of, like, “I’d be able to calculate my willingness to pay.” An additional three hours is maybe $100, so if I’m looking this, this is a much better choice. So I can do what they call an even swap. So I can make a calculation and determine what would be the sort of equivalent.

10:00 - This is a more sophisticated way of doing it that ends up being really important when we’re working with environmental questions. Let’s go through a couple of examples to make this concrete for environmental decisions. All of these are coming from that structured decision-making book. So we talked about this as fundamental versus means. So think about legroom and tray tables as being aspects of comfort, but ends objectives or fundamental objectives are what we ultimately care about.

10:34 - So here’s an example of related to sort of fishing experience and lakes. So what we’re trying to do is one of the things we care about is maximizing fishing experience. So the way most ecologists might think about this is we wanna increase the numbers of fish, or increase lake productivity, or increase lake nutrients, or fertilize the lake, so you might hear all of these things, but what you wanna ask is if our idea is what we wanna do is maximizing fishing experience, the question is, how do we achieve that? What are the pieces that matter? If we’re starting here, because this is easier to think about, it’s like all the different things, the question is, why is that important? And when you stop is when someone responds, “Because it is. “It just is.” So increasing the number of fish. The reason why we care about it is we wanna maximize fishing experience, and why do we care about it? Because it is important. It’s just fundamentally important to what we care about.

11:41 - This ends up being really important because what we wanna do is focus our decisions on these end objectives while realizing these help us to construct our performance measures that help us to understand whether or not we’re going to achieve that end. But if we think about it only in terms of productivity, we’re not necessarily including our measuring of some of the things that we care about, and we might constrain our decision too narrowly. Okay, so characteristics of a good objective, and I think we got there are a lot with our flight example, and so we want it complete. We wanna include everything that matters. We want concise, so we don’t want anything that’s unnecessary or redundant, so things that would be outside of the scope of what we’re really considering with this particular decision, such as like, I don’t know, how much fun we’re gonna have at the conference, because we’re focused on a decision about how we get there. Controllable, so these are things where there is some choice to be made, so you have the ability to make a decision about the level of comfort you would have on a flight.

12:45 - The airline has set parameters, but you have a choice about which airlines you would fly. And the other thing is is that it’s affected by the alternatives that you’re considering, so this is why we eliminated some of these objectives. So we thought all about this, but given the options that we were looking at, some of these objectives ended up not mattering. So we can eliminate them so that we’re only focusing on the ones that we could actually trade off and make choices about. Understandable, you would want this to be both understood by you but also someone else, and especially when you’re working in a multi-stakeholder context. And then preferentially independent.

13:26 - This means that I don’t need to know what’s happening with cost in order to make a determination about time. These are independent of each other. We can assess these. These do not depend on each other. Performance measures and indicators. So these are critically important, and this helps you to understand what matters in the decisions and maybe some of the outcome variables that you would wanna consider in your ecological forecast. So thinking, again, about the decisions that one makes and what people care about in those decisions may give you a sense of, you know, I’m looking at phenological changes, but being able to predict the start of spring ends up being really important for certain types of decisions, and being able to do that and the lag time affects agricultural planting, and whether or not we are able to hit our Cherry Blossom Festival on time, and other things that end up being important. So you can think about the performance measures that one would think about in terms of the ecological forecast. So let’s go through some examples. Objective, we’re gonna minimize negative health effects.

14:43 - So that’s a fundamental objective that we’re trying to do. So there’s three ways we can think about that. So one of the ways is you can just think about that in terms of fatalities per year, so just the number of people. The example two, this is where we got with some of this, where we were building out and adding complexity. So one could think about it in terms of fatalities per year, reduced activity days in the average population, reduced activity days amongst sensitive populations, hospital visits, respiratory events, cardiovascular events, asthma events.

15:15 - So you could build this out in a much more complex way. And then you could also build it out but just not quite as detailed, so fatalities per year and then reduced activity days in the average population. So what they’ve done is they’ve collapsed all of these activity days and these aspects, and then thought about it in a more aggregate sense. So three different valid ways of assessing the objective of minimizing health effects. Let’s look at another example. Candidate performance measures for maximizing water quality. One of my favorite issues.

15:53 - You can think about it in terms of concentration of dioxin and use the original units, and then the cost of making those improvements. You can also think about it in terms of number of exceedances, so this comes into play when we have environmental regulations, and you have some cut off. You can think about it whether or not it exceeds some kind of environmental standard. And then you can divide it based off of ratepayers and then calculate how much it is per household, which may impact a decision slightly differently than if you’re thinking about total costs. This really depends on the decision, the decision context, because the performance measures you choose matter, because whether you’re thinking about it in terms of the average cost of an individual versus a municipality is really different.

16:42 - And people, because of mental shortcuts they use and things that we know about how the framing is constructed, they may have different preferences based off of how you construct those. So natural versus proxy. So for environmental considerations, oftentimes with ecosystem services or think about structure and function, and let’s monetize it because then it’s included in decisions, which is true. If you’re doing cost benefit analysis and that’s the way that you’re making trade-offs, we can make trade-offs in slightly more sophisticated ways than cost benefit analysis. Same kind of general concepts; we just don’t have to monetize everything. So here’s three objectives. Let me read these. I think it’s hard to see in the back. Maximizing population of boreal caribou. A natural measure, this is something that it naturally measures that specific objective, such as minimize cost, use cost.

17:38 - This is abundance or the number of caribou. That’s really hard to do (chuckles), so oftentimes we might use a proxy, like the area of habitat. Now when you use a proxy, as Mike pointed out earlier in his forecasting, there’s uncertainty around that proxy. It’s not a perfect measure of what we’re trying to achieve, and you would want to account for that when making those considerations. Minimizing air quality, so this might be the number of respiratory cases, but you could also look at this as a proxy in terms of emissions of particulate.

18:12 - So different ways of getting at the same objective. And then minimize impacts on Aboriginal culture, this is getting at some of those social factors that are critically important. So, here, there’s no obvious natural measure. (chuckling) This is to your point, Ian, of how, what? How do you do that? (laughing) A proxy might be number of culturally significant sites that are affected, so that might be a suitable proxy. It might not be a proxy that one of your stakeholder groups accepts, and so you might need to move to something like a constructed scale.

18:47 - So, for example, you could do that for the Aboriginal cultures, but here’s an example of the way they’ve constructed a scale for public support of a facility siting. So this is whether or not there’s support for siting hydropower. And so what they’ve done is a constructive scale: one, zero, negative one, negative two, negative three. One is their support; no groups are opposed. That’s awesome. Neutrality groups are indifferent or uninterested. You might have something that’s kind of in between one and zero, but you have. One is there’s controversy, two is there’s action-oriented oppositions working against you, and then three is strong action-oriented opposition of two or more groups. So there’s a way of then classifying this and then being able to include this in the decision process. Another environmental example, let’s think about eutrophication status. Eutrophication manifests itself differently in different systems, and so you can describe the characteristics of an oligotrophic versus like a hypereutrophic lake, but the characteristics of how you would map to that kind of categorical scale, it’s not as easy as just taking nitrogen phosphorus, algae, Seki depth, and just being perfect. This lake is mapped as eutrophic.

20:16 - It’s a constructed scale where there’s not a perfect way of of perfectly measuring it. Does that make sense? So three different ways you can think about constructing performance measures. What you wanna do is make sure that you’re not just taking data or model output and trying to jerry-rig it into an objective. You’re really thinking about what would be some of the best ways of measuring this. Are there ways of doing that, or are there ways of configuring models and other things that we have so that fits up? Alternatives, so we talked about why it’s so important to start with objectives and performance measures, and the reason is this theory called value-focused thinking that has been empirically shown of if you start with what you care about, you will expand the number of options and alternatives that you consider. So think about the ESA example.

21:12 - If you reframe that as a decision about professional development, ESA as a conference might be one way that you choose to get that kind of professional development experience, but there might be other mechanisms, and there may be particular goals that you have about getting new scientific information, potentially meeting with new colleagues, being able to generate new funding sources as a result of it that then allow you to do new science. So there may be means and fundamental objectives that you care about, but then they broaden the scope of your decision and broaden the kind of options that you would consider to determine whether or not taking a week out of a year would be worthwhile to go to the conference or whether it would be better to, I don’t know, take an ecological forecasting class, or go on vacation, or other things. And then, finally, consequences are where ecological forecasts come in but understanding all of these pieces then allows you to think about what you’re forecasting and how it feeds into decisions. Understanding how to do some of the trade-offs and how ecological forecast fit with the decision models, I think, is really important, and we’ll get to that on Friday. There’s ways of thinking about value of information, the methods that you would use to be able to assess whether or not improving your forecast has value for a decision, and the sensitivity of the scientific information to that decision.

22:46 - And there’s ways of then thinking about how we link adaptive management and iterative forecasting. We’ll get to that on on Friday. But what I want you to think about this week is think about these kinds of decisions. Think about who might use the forecast that you’re generating. Think about the measures, the performance measures, that they might use. Are there ways of developing your forecast and quantifying the consequences, whether they’re predictions or projections, that may lend itself more easily to being able to be incorporated into a decision.

23:26 - It’s not the case that every forecast is gonna be decision relevant. There’s a lot of forecasts that just improve our understanding of ecological systems, but so many forecasts could be so useful. And if we can think about how we can better incorporate this kind of approach into the ways that we’re doing ecological management and really making robust choices about our environment, I think we can kind of transform the way that we’re both forecasting, but how we’re making decisions and improving our natural systems for future generations. I honestly believe that ecological forecasts are the frontier for how we operationalize adaptive management. So when we think about that, I’m gonna bring up Holling again because Holling wrote, “Modeling involves two fundamental phases.

” 24:28 - And so when we think about this, the first phase is an inductive phase. This is a creative, synthesis, constructive phase. This is where we map from how we think the world works to how we are abstracting it in terms of a model. Then comes this deductive phase of literally we’re turning the crank. It’s a mechanical phase. We’re using these mathematical analyses and simulations to sort of turn the crank and develop this.

24:57 - So the piece that most people don’t do is they don’t go back because it takes so long to get that deductive engine to work. You don’t go back and say, “Okay, let me compare “what I’ve gotten from my models “with this expectation of reality “and really understand the inconsistencies “and how I can learn from those processes “to improve the model.” These are deliberate decisions that you all are making in terms of how you improve the model. So learning is really critical in this. Decision scientists were right along with the weather forecasters. Part of the reason why the weather forecasters got better is because decision scientists were helping to understand how well those models performed, because we had a natural experiment with local weather forecasters making predictions, and, initially, the people outperformed the models.

25:58 - And that’s probably gonna be the case with the ecological forecast. If we take your predictions versus an expert group, I would hypothesize that the expert group is probably gonna outperform for a while. But if we keep going at this and we think about this in terms of learning and how we improve our understanding, how we make better choices about the model, how we link it back to reality, so we’re being very deliberate about how we design this, we can accelerate learning from failure so that then these models are actually useful in decision processes, and we can operationalize them into adaptive management processes so that as we get new data, the models are updated, but we’re also revisiting our environmental management decisions so that we’re understanding whether our decisions led to good outcomes, and if they didn’t, we’re actually making better decisions because we’re getting our new information, we’re understanding whether it works, and we’re going back and improving it. So that’s my that’s my call to you all this week, is to really think deeply about these things because it’s not just being able to represent the system. I think it’s fundamentally important for improving our decisions. So thank you all. .