Trade-offs, Value of Information, Adaptive Management, & Decision Support Communication

• We are going to go whirlwind-through some examples with this PrOACT approach. We’re gonna talk about three different methods to assess decision trade-offs, cost benefit analysis, dominance methods, and multi-attribute criteria analysis, or multi-attribute utility analysis. We won’t be able to get into a lot of the details and weeds, but there’s a couple references at the end and that structure and decision making book I think is a really great one for being able to look at some of these approaches. With these, what I’m gonna do is I’m gonna introduce a concept, give you an example from my own research because it’s fun to talk about your own work, and introduce the idea of a value of information. We’re not gonna get into it in detail, and we’ll talk a little bit at the end of how do we thin about communicating these forecasts effectively, all in an hour, ideally.

01:06 - So, cost benefit analysis, so this is an approach to compare the relative cost and benefits of a particular solution. So you’re looking at one alternative and you’re looking at, for that particular solution, what are the various costs that are associated, what are the various benefits that would be associated with selecting that option? It quantifies all of the objectives on the same value metric, and that’s usually monetary. That’s why a lot of people like it, it puts it into dollar values. That does mean that when you’re doing this there are things that are difficult to put into dollar values that you have to put into dollar values to be able to use this method. And then the way that you select a preferred alternative or eliminate ones that are less preferred is that the preferred option is the one that maximizes benefits relative to cost.

01:57 - So that’s oftentimes ascribed as a higher benefit to cost ratio and if you have multiple alternatives that have more benefits than costs, then the one with more benefits than the costs, then that’s the option that you choose. Kind of straightforward and intuitive. So what I’m gonna do is go through a quick example where we used this approach to assess this question of is urban stream restoration worth it? What we saw is that in cities like Baltimore City they were using stream restoration and counting it as water quality benefits. And it just seemed like a really expensive way to achieve nitrogen, phosphorous, and sediment reductions. So that was the question, was, is it worth it, largely for these water quality benefits? So we used that cost benefit analysis to look at the pros and cons and then we looked at three major objectives. The first was water quality benefits. We used an approach called the least cost feasible alternative, which I’ll discuss in a second, infrastructure benefits, which used the same approach, and then aesthetic and recreation benefits.

03:04 - This is one of those hard to quantify, and we used economic approach called contingent valuation to get the willingness to pay. We didn’t consider other objectives. So you all might say, “Well, what about habitat protection, or other things that stream restoration might provide?” Absolutely important, for the context of this, they weren’t considered in the analysis. And this is something you wanna be aware of with cost benefit analysis, it’s only as good as what you come in with. This is the thing with these types of models, is that they’re based off of values. The science, supports, and assessments of how you make those types of decisions all based off of your values.

03:45 - So if you’re missing values that are important you have a model that’s not gonna tell you what option would be best based off of your own values. It gives you a sense of what would be best based off of a more limited set. And then costs, we were able to get data from urban stream restoration projects in Baltimore. So the water quality model, this is one of the ways that we went from things like how much nitrogen and sediment reduction one could expect from streams into a dollar value that’s linked to a dollar per liner foot of restored stream, was we had to think about it in terms of what are best management practices that would also achieve nitrogen and phosphorous reductions. So we developed a BMP sizing model, we looked at the size that those best management practices had to be in order for that to happen.

04:40 - We then had to cost those based on the size using, sort of standard engineering restoration cost approaches, and then you’re looking at how much did it cost in total, and that’s basically an assessment of how much it would cost to achieve the same objective. You look at the stream restoration characteristics, how much nitrogen and phosphorous it would produce, and then you get a sense of what you would need as a replacement to be able to achieve the same objective. So looking at that, we looked at a range of different best management practices and we were able to make calculations, both with land costs and hypothetical land cost scenario, and I won’t give away the punchline, but in general, it’s a lot cheeper to do this than to do some of the stream restoration approaches. Infrastructure, so a lot of times with stream restoration projects, pipelines or other infrastructure is run right along the stream because you now you have gravity supporting the movement of water. So a number of stream restoration projects have co-benefits related to infrastructure. protection of bridges or pipelines.

05:54 - So we looked at, okay, least cost alternative. So we’re not thinking about other benefits, we’re only thinking about what is the cheapest way to achieve this benefit? That’s what you have to think about. And so what we said was, well, rip rap the stream, sides of the stream, that stabilizes stream banks, you won’t have impacts to your roads and bridges and pipeline along it, and you can calculate what the cost of rip rapping that stream is. So you get a cost for that. Aesthetic and recreation benefits are a little trickier. So what we had to do was, this was a multi-phase process, is that people don’t view stream restoration approaches the same way. So we looked at four different scenarios.

06:41 - We called one high and dry with meadow, so mostly no trees or few trees. High and dry stream banks with tree cover, so stream banks are higher, but you have trees surrounding it. Low and wet with meadow, so you can think about these as wetland, or not quite wetland-like systems. And low and wet with tree cover, so highly shaded. We had people assess what they liked best, and what we found was the most favored approach is high and dry with trees.

07:16 - This is almost like at Disneyland scenario, and the least favored, despite it’s potential bio-geo-chemical benefits, is the low and wet sort of meadow conditions. What we then did was we did a willingness to pay survey. There are economists that study this whole approach. We worked with an economist to do this because I’m not an economist and we needed to make sure we were employing this right, but this was set up as how would you vote for a one-time tax to pay for this project? And so then you add different tax values, whether they would vote yes, no, or would not vote, they didn’t care, they were kinda indifferent. And we set a number of the conditions as constant.

08:00 - So we said some likely to improve, water quality, it would protect existing infrastructure, so we set a number of things constant so we could really get a sense of what they would be willing to pay for the most favored and the least favored stream restoration approach. And so what’s clever about that is what your able to do is calculate and aesthetic premium. So, what’s the difference that people would be willing to pay for something that would offer them greater aesthetic and recreation benefits? So not how much are they willing to pay for the restoration, but how much they’re willing to pay for that difference between their most favored and their least favored approach. And so we ended up being able to calculate how much they would vote. We did it for all of the City of Baltimore in this area that’s right close by.

08:53 - You can incorporate all three of those approaches and basically build it into a low and high scenario situation. The crux of this is that if you’re basing these decisions off of infrastructure or water quality improvements, given the cost of stream restoration in Baltimore and a lot of areas around, what it says is that you can’t justify those projects on those objectives, alone. You can justify it in some situations with the low cost being the low cost of restoration and the higher cost being, sort of the high cost per linear foot of restored stream. You can justify it in some cases depending on the willingness to pay. So this helps you to understand what benefits or things you might need to consider in an option in order for it to be an appropriate solution.

09:52 - It also means that it can highlight where there might be cheaper alternative to achieve the same objective. So cost benefit analysis, I know none of these are approaches you’re gonna apply right now, but just so that you know how these forecasts are being incorporated, I think it’s useful to just kind of see these different approaches. Dominance methods, Pareto optimization, this is in Michael Dietz forecasting book. A couple of things to know about dominance methods. First is, it’s an approach to identify alternatives that would be dominated by all other options.

10:32 - So it’s really almost a screening approach. If you have tons of options, you have a hundred different options, that’s really labor-intensive to be able to go through, are there ones that are always gonna be worse off than all of the others on the objectives that we care about? So again, it’s still based off of the objectives that we care about and we’re trying to eliminate the ones that would never rise to the top on any dimension. Usually we do this through optimizing across objectives and assessing the performance on those objectives. You can also include constraints, so regulatory or environmental constraints and that. The key thing is that this does not tell you what to do, like, even if you included all of your objectives, what it does it eliminate choices that aren’t very good.

11:18 - Because the reason it doesn’t tell you what to do is the relative importance that you put on different objectives depends on your values and what you care about. And so because this approach doesn’t assess the weight or the relative importance of those objectives, it’s not gonna be able to tell you this option is better than that one. It narrows the set of options that need a more in-depth analysis. So the way that this looks if you’re doing a Pareto frontier is that you might have all of these different options and then you have this sort of optimal location. So we’re looking at two objectives here. Here this means that this objective is preferred over any of these other objectives here.

12:03 - So anything that’s in this space is dominated by solutions that would outperform it. Let’s look at an example where we did this with a cost analysis looking at large-scale diversion structures on the Mississippi River delta. This was done in collaboration with a couple of geoscientists at UT Austin and University of Illinois Urbana-Champaign. So the motivation for this was there’s been tremendous amount of land loss in the Louisianan delta and there’s been a lot of discussion about how a lot of this land is gonna sink within the next century, and how might we be able to build up some land that would create some of the buffering capacity that we’ve lost for a number of the areas in this location? Just to give you a sense, so we talked about there’s sediment loss to the deep gulf because it’s going off and landing into larger shelves. There’s degradation in barrier islands, we’re losing wetland habitat and the benefits that come with that as well as swamps and mangrove.

13:11 - And this is one of those questions that has a huge number of stakeholders, federal, state, local governments, NGOs, whole wealth of different kinds of decision makers, private land owners, outside scientific groups, and there’s lots of people weighing into that and there’s a lot of different objectives that need to be considered, perfect sort of multi-stakeholder objective. And so one of the questions that arises and some of the discussion that happened was “You know, we think we can build a massive amount of land using smaller scale diversion structures.” and what we wanted to do was ask this question is, since we know better how deltas are built and how you can build large amounts of land based off of our geoscience knowledge, is would engineered diversion structures that are either single projects or portfolios, combinations of options, so an alternative can be a single option, or it can be combinations of options, for these large scale avulsions, which one’s gonna give you the biggest bang for your buck? Something that’s super deep into the water column and pretty costly, or something that’s shallow and deep? And this is a real conversation that was happening sometimes still happening about how you think about restoring some of Louisianan delta. So what we did was we were looking at cost-efficient options, so that Pareto frontier idea by looking at water and sediment diversions, coupling that with a land building model, and then coupling that with a diversion cost model. So let’s go through these super fast. So again, this is an optimization approach. So let’s look at our objectives.

15:03 - So, cost, we wanna minimize cost, sounds reasonable. We wanna maximize land, so we’re gonna minimize cost subject to maximizing land, but when you have a diversion structure, you’re not just dumping sediment, you’re also dumping water. Mississippi River is one of the largest economic engines because of the amount of goods and services that go up and down that river. And so you still have to maintain navigation within the high-flow period of time. And so what we did was we made that subject to a constraint on the amount of water that could be released.

15:44 - And that ended up being really important because what we needed to do is not potentially release massive amounts of water to be able to achieve land building objectives, we did need to put a constraint on it so that you could then maintain that critical navigation. So the water and sediment diversion, this comes up because there’s more sand at depth. When you’re looking at a water column, in order to build land from scratch you can’t build it with clay. You need, ideally, sand, silt is okay, but you want as much sand as possible. And so since we have a limited amount of water, there was a really important question about the depth and the width and how much sand and water would come out of the structure as a result of that.

16:36 - And that’s really, because, again, I mentioned, we have more sand the deeper we are in the water column, so building deeper captures a higher fraction of what we are trying to achieve. If you couple that and you know the amount of sand and water coming out of those diversion structures, you can build that into a mechanistic land building model that is forecasting the amount of land that would be built given an assessment of the local sea level rise that would happen over the next 50 years and subsidence that is occurring in that area. That was a collaboration with geoscientists. And part of what this shows is that also because you have the bathymetry, so it slopes downward, not intensely until you get to this drop-off in the areas that we were looking at. You know, as you get farther from the shore it takes a longer amount of time to create land because you have to build up more over time.

17:45 - So then the diversion cost model, this was just a simple empirical model where what we did was we took existing diversion structures in this area, got their depth, width, and cost, and then just developed a regression model to be able to understand the relationship. And it confirmed this hypothesis that we had that there’s scale of diseconomies with cost. It costs more to build bigger and deeper, and so it’s more expensive to build deeper than it is to build wider. Which makes sense, if you think about it, going deep into a river like the Mississippi delta is really complex and expensive, going wider is easier. What this allowed us to do was to explore a range of different options, again, single options and combinations of diversion structures that would allow us to optimize the objective that we were trying to achieve. So let’s see what that looks like.

18:48 - Okay, so a little hard to see, circles indicate deep, so these are deep diversion structures. Squares indicate medium diversion structures, so kind of halfway through the water column. Shallow indicate more like freshwater diversion structures, so breaks and levies, or things that are more shallow. Additionally, if it’s white, then it’s a small amount of water that’s being diverted, if it’s black it’s a lot more water that gets diverted for that particular option. So this is the amount of water that’s diverted.

19:21 - What you see is that if you are trying to maximize land relative to the cost, when you are in the region where you’re not trying to build a massive amount of land, shallow diversion structures are cheaper to build than deeper diversion structures. They release a lot of water, but they’re cheaper. As you get up into the Pareto frontier, you really can’t achieve larger-scale land building until you start building medium to deep, and the more land that you wanna build, because of that water constraint, you really get into deep. So what we did here is we basically, for each 100 kilometers squared of land, we looked at what was the absolute best option on all of those dimensions relative to the water constraint. And so here, again, when we talk about this in the 100-200 kilometers squared amount of land, what we see is shallow, you can build wider because you’re not running into water limitations.

20:31 - But you quickly run into this water constraint. And this is one of the drivers of what approaches ended up being optimal, largely because you have to maintain that navigation channel. And so it meant that to build larger amounts of land you ended up needing a deep diversion structure more quickly than I might have predicted, otherwise, simply because to be able to capture the amount of sand that you need to build the land you quickly maximize the amount of water that you are using. You know, this was a generic approach so it doesn’t consider land rates, or preferred locations, exactly where this is sited, and if you were to do this for real, then you would wanna consider all of those factors. But to narrow down and create, from a scientific perspective, what is actually feasible and to put some science into this conversation where people are saying you can build hundreds of kilometers of land by doing shallow diversions, we can simply say, “That’s not feasible if you want to maintain your navigation channel.

” 21:41 - So, dominance methods, a really great way of sort of constraining and focusing on the particular alternatives that matter. So, multi-attribute utility analysis, economics is great when you can collapse things into a single-value. For a lot of the environmental problems that we deal with there’s multiple objectives that are just really difficult to monetize and quantify into something that allows for that comparison. Multi-attribute utility analysis is one of the approaches that helps you to sort of solve that problem. It’s a consistent, rigorous method to making decisions given multiple competing objectives from lots of different stakeholder perspectives and scientific information that’s uncertain that you wanna include in the process.

22:33 - When we think about this, again, this links really nicely to this PrOACT framework. Forecasts and information that predicts the consequences of alternatives, this is explicitly incorporated into these particular models. And when you do that, similar to how we did that consequence table, you can identify options that might be dominated from that approach, but these consequences, similar to the dominance study I just showed you, it doesn’t tell you what option is the best. It doesn’t incorporate the values explicitly, what it does is it limits the choices. So these types of forecasting consequences coupled with dominance methods can be really useful but they have limitations if you wanna use them to help support a more rigorous trade-off analysis to think through how you might select decisions.

23:29 - So, the benefit of multi-attribute utility analysis is it doesn’t require collapsing those performance measures into the same unit. Instead what we do is we normalize the indicators through a utility function, thus, utility analysis. Utility is basically an index of desirability. So we talked a couple of days ago about expert elicitation, those are probability distributions based off of what we believe as scientists. And by belief, I’m not talking about religion, I’m talking about our scientific knowledge that we’re incorporating.

24:05 - Preferences are different, so there’s a number of the same approaches you would use to preference elicitation, but it’s not a probability distribution. You will sometimes see them from zero to one I prefer to represent them as a unit-less value between zero and a hundred, simply because it’s really confusing if you have forecasts with probabilistic information that you’re incorporating and then you have a preference assessment that’s from zero to one. If you change it, since it’s a unit-less value, to zero to a hundred, it helps to keep those mentally separate. So you can normalize, so now you have a basis for comparing between objectives. But wait, this isn’t money, so it doesn’t give you a direct comparison, what you have to do is because you’ve translated things to an index of desirability, you have to weight the relative importance of the objectives.

25:03 - Weighting is relative to the other objectives, and there’s rigorous methods to do this using swing weighting, we will not get into those, but just know, lots of science and approaches for how to do this, call a decision scientist. Now, the way that you calculate this is through a linear additive model, the math is super simple. So usually the limitation here is not doing the math and constructing the model, again, it’s the structuring the problem and making sure that you’re including all the things that are important. Because if you’re using this as a way of helping to select the best option or portfolio of options, it will only give you a result that is meaningful if you have included everything in your model that matters. And everything in your model that matters is the values that are important for decision making and consequences that are linked in meaningful ways to assessing those values.

26:06 - It’s a decision model, so it’s fundamentally different. So let’s go through an example. Water quality standards, so the Clean Water Act sets a water quality standard as a narrative designated use. Think about this as a goal, your objectives. Narrative or quantitative criteria, proprium, it’s measures, and an antidegradation clause, which is a constraint, just don’t make it worse. A number of years ago, 15-ish, EPA was really pushing to move narrative criteria for nutrients or nutrification problems to something that’s qualitative. And nutrification’s a little tricky.

26:51 - The reason why they didn’t do this is because it manifests itself differently in different systems and it’s not a clean relationship across an ecosystem. And so the way that the recommended it was through an approach where you map a distribution, of course, it looks normal, and if you’re creating summary statistics based off of all the lakes you set your criterion at the lower 25th percentile of the distribution. If you have selected out what a reference lake is and define that, it’s set at the upper 25th percentile of the distribution. What this doesn’t recognize is inherent in these water quality policy problems is that this is both a science and a values question. So, by that, I mean when we think about how we wanna set water quality standards, from a prescriptive standpoint we want something easily measurable, so something we can actually collect that serves as a good proxy, a surrogate for the water quality goal that we care about, the designated use, and it should serve as an accurate predictor of whether or not a water body is impaired.

28:06 - From a pragmatic perspective, we’re gonna get some nonattainment. You’re just have variability in these systems. There’s very few systems that will be in compliance all the time, and so you wanna think about this in terms of risk of nonattainment. And so you’re thinking about this partially in terms of what’s the probability of nonattainment and how can you think about that within the context of these problems? So that brings the question of how should you set these criteria? So we’re gonna go through a mitigation treatment decision. We’re gonna build out a decision tree and I’ll show you how we combine these models.

28:44 - So from a lake we’re gonna observe some water quality value that I’ll call y. From that measurement what we’re gonna do is we’re gonna make a decision about whether or not a lake is classified as unimpaired, so it requires no additional treatment, or it’s impaired, so it requires some level of treatment. If it requires treatment, that’s gonna incur costs it’s gonna cost something to improve it. After that we observe the lake’s eutrophication status, conditional on the treatment given the eutrophication status to the designated use and how much did it cost. So to be able to do this what we did was we developed a simulation model to help us choose the optimal criteria for this water quality value.

29:33 - So we chose a candidate criterion level, and it was across a distribution of values, that’s what this indicates, and then we randomly selected a water quality variable to represent a chosen lake in this study. And then we had this mitigation treatment based off of what we observed and the criterion that was set, is it classified as unimpaired or impaired, and then how does that then map to eutrophication status and then what can we say about the value or utility related to the designated use and cost? So what this shows you is we have a water quality model component of this and a multi-attribute utility model that we linked together in order to solve this problem. This does not involve forecasting, but one could see sort of similarly how you could think about it within a forecasting paradigm. So within water quality we had a range of different data basis given what states collect. What we did was we use existing water quality, these data to predict the future water quality state if it was classified as impaired.

30:48 - So we looked at some treatments and said, “Okay, if we were to improve this, what would it look like with improvement?” And we used my dissertation advisor’s models to be able to do that. To be able to predict eutrophication status, this is where expert elicitation came in. And the reason is is that because it doesn’t translate one to one like measured water quality variables to a particular constructed scale of eutrophication status, the experts were able to help us understand the direct and indirect effects and then be able to classify probabilistically. So you can have them, when they do this, sometimes what you’ll do is you’ll say what’s the most likely category? What we did was we had them assess the probability over these categories, ‘cause they’re looking at a region. And since it manifests itself differently, it’s not just one category, it’s oftentimes multiple categories with a probability distribution.

31:45 - So that gives us our expert elicitation category, ei, and then what we used was, we used a binomial regression model to predict the eutrophication category given these water quality data. We don’t observe, we don’t collect data on eutrophication status, so we need to be able to map from the data to eutrophication status. So we use this model and magic came out, and then we’re able to use that and link it to the multi-attribute model. So I’ll spend more time on this. So, again, talking about the designated uses, so the two designated uses that we looked at for North Caroline are primary contact recreation, so when you go swimming, like you guys are gonna do later today, and there’s a small possibility of consuming water, that’s what primary contact recreation is. Secondary contact recreation, boating, fishing, other uses where you might get dermal contact but you’re not likely putting your head underwater and consuming water, and then cost.

32:48 - So remember, we talked about utility functions being a way of mapping from things that we measure or goals that we have to how much we care about them, how desirable is a particular state. So let’s look at costs, first. So you can look at cost in millions of dollars. If you’re gonna spend money on a restoration project and it’s from zero to five million dollars, what’s the best option? - [Audience Member] Zero? Zero, zero, yes we wanna spend no money! So that’s why this has, with our utility functions they go from zero to one, but you can think about it in terms of zero to one hundred. So this would be like one hundred. What’s the worst? - [Audience Member] Five million. - Five million, so here’s our bounds, boom.

33:43 - Similar, kind of in terms of elicitation as you think about with expert elicitation, but again, preferences. Now, cost is linear, because if you’re dealing with a large government agency they should not care about the difference between zero and one dollar and five million and five million minus a dollar. So that means that the difference is linear. To assess it for things like eutrophication status, this is a little bit more tricky. What we did is we worked with four different decision makers in North Carolina, assessed their utility functions individually, and then had the ability for them to meet together and develop a consensus so then we could compare and aggregated versus consensus versus the individual, it’s kind of awesome.

34:33 - And so when they did this, eutrophication status where you have a oligotrophic lake was viewed as most desirable for swimming and those kinds of activities. A hyper eutrophic is least desirable, but it’s not linear. So it really drops off after you get to sort of eutrophic good into sort of eutrophic bad. So that’s where you see it drop off, and it drops off a little sooner for primary contact recreation than it does for boating and fishing, in part because with fishing a little bit more productivity gives you bigger fish in lakes, and it’s also one where some of the clarity issues may not matter as much to a particular person. In the elicitation we did swing weighting where what we did was we were able to quantify their weights for each of these different three objectives, do it as a group and then we looked at the average.

35:30 - So weights are indicated as k, and so our multi-attribute model is the utility given the eutrophication status and cost. It’s a simple linear additive model, you know? Given i, and then weights, which in this case we call k, and utility, it’s a simple linear additive model because what you’re doing is you’ve normalized for what you care about, in this case we care about designated use but we measure it through eutrophication status, which then maps to the water quality variables, and then we weight it based on how much we care about these different things. So you use this approach, you combine them together and then what you’re able to do is then calculate, in this case, the expected value or the expected utility of different levels of a water quality standard. In our case what we’ve found was total phosphorus was a near perfect correlate, like 95% correlated with eutrophication status for reservoirs in North Carolina, so we could use it as a near-perfect proxy, and then we could map levels of total phosphorus onto expected value and that then means that it provides an approach for how you would think about developing a standard that recognizes the importance of science, but also recognizes the values and this risk of nonattainment come into play when setting those standards. Needless to say, this approach was not adopted but, (laughing) but gives an example of how one might be able to do that. Lots of different approaches.

37:18 - One of the things I wanted to touch on briefly is value of information. Different groups think about this differently. Some like to monetize it, if you’re using multi-attribute methods there are approaches to do this quantitatively and you guys would be able to look at the books and pick it up quickly, but just to give you a high-level overview of what this means is from a decision analytic perspective, what we’re looking at is does better information potentially change a decision? Not whether it improves our understanding of fundamental processes or improves our model, or improves our predictive capability, but does it change a decision? And what you can do is you can look at this from the value of perfect information, so we know what will happen, and the value of imperfect information like forecasts. However, let me just say, with this, oftentimes the calculation for imperfect information is a little bit more tedious. If you can calculate the value of perfect information and it tells you that it’s not gonna improve your decision, you don’t have to calculate the value of imperfect information because it’s also not gonna improve your decision.

38:32 - So you can use both of these approaches and use this kind of simplifying, because the value of perfect information is basically the maximum value of information that you could obtain. And then one of the bounding conditions is value of information can never be less than zero. If someone tells you, “You know, I can’t make a decision until I know for sure whether or not something is gonna happen.” The correct response is we make decisions based off of imperfect or uncertain information all the time, and do you want me to calculate your value of information? One last thing that I wanted to go over is presentation of forecasts. So a lot of us are producing forecasts, I know folks have mentioned Shiny and the development of decision support.

39:24 - What I want to do is give an example of how we’ve worked with the NOA Climate Prediction Center to revisit the way they’re visualizing their temperature and precipitation outlooks to make them more useful for people who could potentially use them in decision making and improve their interpretations. So this is all based off of this idea that a lot of times with decision support products we improve it based on feedback or intuition. And intuition doesn’t always match best practices very well. And that’s in part because we have to understand who our users and our audiences are and understand how they understand and comprehend the information. And the way that you guys comprehend information amongst each other is fundamentally different than the way that other people outside of this room and ecological forecasters think about it, or ecologists.

40:23 - So a couple of quick tips, it’s easier to distinguish color than shape. So if you’re doing something that’s fundamentally different color is pretty powerful, because color is really powerful you wanna be very careful about how you use it. If you’re presenting something what you want is if you’re presenting different things you can use different colors. If you’re presenting the same thing you do not want to use different colors because you are indicating that you are representing different things. And this, then, comes into play also when we think about maps.

40:59 - I don’t know why scientist love rainbow maps, but they end up being really problematic because what we do is we inherently think about changes in the hue or the color imply more difference than changes in intensity. So if we’re changing color, this means something different than if we’re just changing gray scale and the intensity, the saturation of what we’re showing. This ends up being really important. You know, if you look at this, you’re like, “Hot, cold, in the middle.” These are things that we automatically jump to, whereas if you see this you automatically think about it in a different way. So think about your users and presenting it.

41:43 - They use this all the time to talk about how scientists think about and present our work, think about the standard scientific paper, present all of this information, and the context and all of the details, and then results and conclusions we tell you exactly what happens. Public, think about your newspaper article or if you’re reading scientific information outside of your field, kinda want this bottom line, so what, and then you want all the details. You don’t want all the details to be able to understand what you really need to care about. Same is true, we would argue, for scientific visuals. So exact same principles because if you’re designing visuals you don’t need to include absolutely everything upfront for someone to be able to understand it it is very likely they will not come to the same conclusion.

42:36 - So being able to think about that simple message ends up being really important for thinking about designing visuals. So my example, all right, temperature and precipitation outlooks. How many of you guys know these? These are the seasonal forecasts, NOA’s asking for comments on it, if you have comments about seasonal forecasts, I highly recommend that you submit comments. What these are are their forecasts on a seasonal basis, so extended range, so six to 10 days, eight to 14 days, to mid-range, three to four weeks, to long range forecast, which are one to three, to out to a year in advance. So they call them one or three month forecasts but they’re predicting out a year in advance.

43:25 - They do it both for temperature and precipitation but what you’re presenting here is geospatial uncertainty and that’s super hard because people don’t really get probabilities and uncertainty and now you’re presenting it in a map, and that’s really tricky. So it’s an open question of how to present that the best. And so what my team did was we used decision science and visualization science to really think through the users and how we might be able to design these graphics with them in mind. So what we did was we built off of literature that exists related to how you classify different visualization problems. We coupled that with interviews and surveys where we were able to get the self-reported, what people said they had problems understanding and what they didn’t like or what they said, other people had problems understanding and they had to translate for them.

44:21 - So we worked with some people who take the forecasts and contextualize them given different decisions. And revealed understandability, think about this as a test, can you get questions right that are important to understand those graphics? And so that gave us an ability to then identify the challenges, trade-offs, major concerns, really diagnose the problems with those particular graphics. So the diagnosis for those graphics were you can’t use white space inside the US and outside the US because people, if they mean different things because people don’t get it. The understanding of normal, so they were using normal or near-normal conditions inside the US as white space but even how you understand, what does it mean to be near-normal in terms of temperature and precipitation can be confusing, clarity and clutter, you can get that from the graphic, and probability versus intensity, this is a really key one to understanding these graphics because they’re presenting probability of being in one of these three bins, probability of being near-normal conditions, above normal, or below normal. They’re not giving you how much, the intensity.

45:40 - And so making sure that that didn’t get mixed up ends up being an important communication challenge. So here’s the original, what we did was to look at breaking down two pieces, was all of the graphics that we did, we collaborated literally every two to four weeks with the NOA Climate Prediction Center because to be able to actually test graphics it doesn’t matter if it’s an academic ideal. What matters is whether or not they can operationalize it thorough their system and that it meet some of the constraints that they have in what they can display by their agency. So we worked really closely with them and they redesigned the graphics to make sure that they would actually meet those operational requirements. So the first one was a simple modification where what we did was we said, “Well, if it’s going out to multiple users, then,” just to give you a sense, this goes out easily to hundreds of thousands of people everyday.

46:41 - We looked at it with expert users from ag, emergency management, water resources, and energy sectors. And it goes out and is reinterpreted by the public, so it ended up being in my inbox with a heating and cooling air company convincing me that I needed a package for my AC for the summer because it was gonna be in above average conditions. It showed up in a bathing suit forecast in a fashion magazine, and it has been reinterpreted for Weather Channel and other things. So these get used and reinterpreted and how they get reinterpreted, similar to your forecasts, once you put out into the world, some of it’s out of your hands. So how you communicate it matters. So we thought, “Okay, well, if it’s going out to these broad public audiences, maybe as a default, you can always add layers of complexity, it would be better to simplify and do these bins of leaning towards normal, where it’s between 33 and 50% chance, leaning below normal, or leaning above normal, or likely above or below normal where it’s 50% and above.

” 47:52 - I chose the term leaning because it had not been tested in the uncertainty qualitative language before and also if you saw the New York Times election dial, that used the term leaning, and when I sort of informally tested with people it seemed to play better. Likely was interpreted different than leaning. So that meant that you’re blocking off coloration and aggregating things and then the different probabilities are represented through contour lines. Here what we did was we kept the color gradation, we simplified the near-normal because these higher values are never represented because it’s near impossible to be able to produce more higher probabilities of near-normal for these forecasts, and we made sure that if you’re doing normal conditions versus equal chances that you’re representing those differently. And then what we did, we tested that, that gave us some information, and then we worked with NOA because this was a lot harder to do, it required more time on their end to be able to develop a combined where we got rid of places where you don’t have data.

49:04 - If you don’t have data don’t show it in your graphic. We moved up the legends and provided information. We were clear about the probability distributions, this is an older version, we also included the qualitative language, over here, but because color is really powerful, we found that people don’t look at the contour lines for probability, you really need the color differentiation for them to get those probabilities. So looking at the results, what we did was, because you have this you can do control versus treatment testing if you set up your experimental design right. We did, so what we could do is we could look at how people interpreted the white color mapping and whether they got that right.

49:48 - We could look at how they interpreted near-normal with our combined graphic. They were more likely to get that right, so as we made design improvements we were able to improve that. This is, again, the simplified versus that discrete legend. Color is really powerful and we specifically designed this state-specific, to pick a state where they had to look at the contour lines. People don’t look at the contour lines. That informed our decision for how we needed to develop the combined one. And similarly we were able to share this.

50:22 - In general, what you find with the results, and this was some of the phenomenon were a little bit unexpected, but in total, the combined ended up being better, is experts, in general, got this right more often than not. So they usually outperformed the public. You saw the largest gains in improvement by the public and a few of these cases, because you’re completely redesigning these graphics you saw slight dips in the way that the experts interpreted it that were statistically significant and we aren’t able to formally know why this happened, but the hypothesis that is a literature-based one is when you have experts that have seen graphics for the past 25 years and know what those graphics are, think about websites and when they redesign websites, even if it’s exactly what you want, you can’t find stuff for a little while, it takes a little longer. I think this is what’s happening with this, is if you significantly redesign the graphics it’s a little harder to find the information you were expecting. So it’s not unanticipated to find a dip, if we were to do this as a time series I would expect for those to go up after three to six months. So in general, what this did, because we worked so closely with the NOA Climate Prediction Center was we were able to use a really robust method to be able to understand how we could redesign a forecast based off of how users interpreted it and how we could improve that understandability.

52:05 - And that provides a really strong evidence-based way of being able to improve the way that you’re visualizing and presenting these forecasts. So if they’re going out to lots of different types of decision makers and you’re producing this, especially for very high profile federal government products, it’s worth it to make sure that you’re producing and presenting something that other people that you intend to use it actually understand correctly. With that we have completed the PrOACT approach. We’ve talked about presentation and design of, and the consequences and information, and I would just say if we’re thinking of these things in terms of interactive forecasts and decision models, these ideas of adaptive monitoring, adoptive management, how forecasts get used in decision making, all of this is with the idea that I think we would be successful if in 10 to 15 years we figured out a way of operationalizing adaptive management and if ecological forecasts were part of the solution to be able to achieve that. So I hope to be part of that. I hope that this gives you an opportunity to at least have one friendly decision scientist who might wanna work with you on things related to decision support, but if you’re interested in more of these things, let me know.

53:32 - Because I think this is where collaborations between forecasters and social scientists really come to bear is if we wanna do some of these bigger things none of the big dominance methods, whether or not stream restoration is worth it, I used to be a water quality modeler, and I can do statistics, so I did those, but those kinds of collaborations, you’re not able to solve these problems unless you’re collaborating. And so that’s why we work a lot with ecologists and geoscientists. So, with that, if you are interested in personal decision making, I had a couple questions related to that, “Smart Choices,” simple methods related to even swaps that we talked about in this PrOACT method is presented, here. “Structured Decision Making,” this is the one, if you’re interested in environmental decisions, Robin Gregory is one of my academic heroes, ask him for a post-doc, he has gone through multiple examples of things that he and other have done to think about these approaches and how you might be able to structure rigorous analysis and understanding of values into the way that you think about helping people to make really hard decisions. And if you’re interested in making hard decisions and really interested in the decision models, Bob Clemen, who is one of my collaborators, he wrote the first textbook on decision analysis and a lot of this is framed within a modeling paradigm.

55:07 - So I think it’s a pretty approachable way of breaking down and understanding these methods. And if you’re interested in value of information and imperfect information, I would use this as your reference. So with that, thank you. (audience applauding) .