Why Forecast?

• There was a nice Tweet from Ethan White slab last week I think, where he got a review back saying, “This is all great, “but why should we forecast as ecologists? “Why bother?” So I wanna give a perspective on why we should bother and why particularly, I think iterative forecasting is an important direction for ecology to go. So as Moses said, decisions are fundamentally about the future. So one of the main but not by no means, only reason that we were interested in ecological forecasting, is because there are clearly pressing societal needs to make better environmental decision making that range from High Level National International Policy down to every day decisions made by individuals in the public. All decisions are fundamentally about the future, and if we want environmental and ecological understanding and knowledge to inform those decisions, one way that’s really important is for those to inform predictions and projections, because ultimately, you don’t make a decision about what happened in the past. You make a decision based on what you think is going to happen in the future.

01:14 - And a forecast is making explicit what we think is going to happen in the future. So part of the reason we make forecast is to make our science more relevant, and to respond to the clear societal need to make better environmental decision-making. That said, I’ve come more and more around the opinion that ecological forecasting is a win-win, not just for better decision making, but for better basic science. So one of the emphases of NEFI, the name Near-term Ecological Forecasting Initiative is on the Near-term. As I said in my intro, I’ve spent most of the last decade of my career focused on models that are making projections out to 2100.

01:58 - I feel like the ecological community, when they think about forecasting has gotten fairly good but not necessarily good at making predictions, but fairly good about thinking about projections out on the timescale of 2100. But, after spending a decade making projections out to 2100 I don’t know anything more about whether I’m any good at making predictions at the 2100, because it’s not 2100 yet. And I have not validated whether I’m actually any good at that. By contrast, I’ve come around to the perspective that if I make a projection for next week, I know pretty well pretty quick whether I’m any good at making a prediction for next week. If I make a production for the this season, then you know, the next…

02:42 - The inter annual variability or the next few years, this phenological variability over different seasons that these are timescales are actually timescales where we can actually validate whether we’re any good at making predictions. And I’ve also come around to the perspective that a viewing, forecasting and iterative forecasting in particular is fundamentally as a expression of one way of thinking about how the scientific method itself works. So in the scientific method, we start by, we form a hypothesis. We use that hypothesis to make a prediction about how we think it’s gonna happen in the world. We test that hypothesis against what we actually see in the world and we interpret the results.

03:24 - Well, in order to forecast, the models we use to make forecasts are at their essence, a distillation of our hypotheses about how we think the system works. And for me, I think that’s an important perspective to have when you think about models. Models are essentially just the formalization of our hypotheses. So if a model is our formulation of our hypothesis, then we use those to make predictions. Those are an expression of our hypothesis about how we think the world works.

03:54 - I think in ecology we have let ourselves off easy a lot when it comes to formulating hypothesis and making predictions. I feel the majority of the theory we work with is very qualitative. I feel like the majority of the predictions we make from those theory are fairly qualitative. And the way we test those are often not super strong tests. You know, by contrast, using a model to make predictions actually puts a quantitative number on what you think is going to happen.

04:29 - And with acknowledgements, you will probably be wrong most of the time. But when you’re wrong, you learn something. So we test this hypothesis by making predictions, we interpret this and one of the things I’ve been thinking more and more about is by making it or forecast, by continually making predictions we continually get feedback about how we’re doing. And so the idea here, the belief here is that by making near-term forecasts on an interval basis, we have the potential to actually accelerate the pace of science. So not only make that science more relevant to society, but also try to make the pace at which we improve our understanding accelerate.

05:10 - And for me, I showed the (murmur) diagram earlier, one of the things that was striking about that was that if you look at the 2007 version of that in 2014 version of that, the biggest difference in seven years was the color scheme. They were otherwise pretty much indistinguishable. Suggesting that in seven years of effort, the trash, the global trestle carbon cycle community essentially didn’t get any better at what they were doing. That, at a time of rapid environmental change, we cannot rest on our world. We cannot just wait to get better. We need a way of getting better faster in responding to these needs faster.

05:50 - So, one of the ways that I think forecasts help us do better basic science is because forecasts are a priority. So we are actually saying before we see what happens if we make a prediction in space or prediction the time, that we don’t know what that data is at the time that we’re making the prediction. I think this is in contrast to something we’ve seen in a lot of sciences, not just, you know, it’s been very prominent in fields like psychology, but I don’t think ecologists are in any way immune to this is that there’s been problems with reproducibility of results. There’s been problems with accusations of p-hacking and accusations of over fitting of models. And I think with a small percentage of times, those are deliberate.

06:41 - I think in the vast majority of times those are completely inadvertent. But when you have all of the day that you have in hand, it’s very easy to just keep fitting models and testing models and you end up testing all these hypothesis and you can end up with a model that fits data great, but you have no idea if it actually is reproducible, if it’s actually general. By making your predictions a priority, I think it is in some way a stronger test because you don’t know. It’s much harder to over fit when you don’t know what’s going to happen at the time that you’re making the prediction. It’s not perfect, but I do think it provides a stronger test.

07:26 - Also think that trying to make forecasts will force us to synthesize our understanding of ecological systems better. So to give an example, like I said, I started as a plant ecologist, plant ecologists and their brethren in agronomy have been making nitrogen addition experiments for over a century now. We dumped nitrogen on plants. Hypothesis, nitrogen affects plants, stronger hypothesis plants like nitrogen they grow faster. Great, can anyone not believe that hypothesis? It’s pretty much come up consistently for over a hundred years that when you add nitrogen to plants, they tend to grow faster. If you do a nitrogen experiment today, you started today and you use traditional statistical methods, your null hypothesis still remains nitrogen has no effect on plants.

08:29 - When you get a P value, it’s refuting the hypothesis. Nitrogen has no effect on plants. You did not believe that hypothesis before you started. So why (laughs) why in the world would you use that as the basis of your model? By contrast, let’s say we actually went back and synthesized that a hundred years of research on nitrogen addition experiments and came to the conclusion, that in this particular experiment I’m dumping a specific amount of nitrogen on a specific crop at a specific location, and I say based on what I understand, I predict 25% increase in yield plus or minus 5%. I’ve now made a quantitative prediction, I hypothesis with a number around it and an uncertainty around it. Under the null hypothesis, a 25% increase in growth and a 50% increase in growth are indistinguishable.

09:20 - They both refute the null hypothesis that nitrogen doesn’t affect plants. By contrast, 25% supports our existing understanding of the system. 50% rejects our compute. It says something is missing and I’ll completely understand that traditional statistics don’t just give us dumb no hypothesis. They don’t let us realize when we have and have not learned something new. By contrast, no response of nitrogen is non-significant under traditional hypothesis testing.

09:52 - But if this is what you expected is it is a very novel result. You have, I dump nitrogen on a system that my current understanding says is nitrogen limited and it doesn’t respond. That to me should be a highly significant result, not an insignificant result. At the moment we’re not very good at this (laughs) And one of the reasons, like I said, we’re not very good about that. This is because we make predictions over a long timescale.

10:21 - We’re not getting that feedback that we need to learn. So if you look at any field that’s gotten good at forecasting, whether it’s weather forecasting, economic forecasting will set aside whether they’re actually good at it or not. But any sort of field that, whether it’s in the natural sciences, biological sciences, social sciences, any discipline that gets good at forecasting, it’s very clear that that requires feedback. You need some learning process, in places where we’d make predictions and we don’t have a process of getting feedback about how we’re doing can result in us being very overconfident, falsely over confident in our ability to make predictions. Because we do it a lot, but we didn’t ever get any feedback about whether we’re good or not.

11:06 - An as was highlighted in Collins video, the experience has been in fields, like weather forecasting is that, that process of getting feedback does result in improvement. I mean, there’s a lot of things you could lay at the feet for why weather forecast has gotten better. They do have more powerful computers, they have more observations, But I genuinely believe a lot of why weather forecasts have gotten steadily better over decades, is that every day, both technically six times a day, at this point, they put out a global hypothesis about how they think the world works. And then they confront that hypothesis with reality, six times a day over the whole globe. If we were doing that, we sure as heck would learn a lot about what we do and don’t understand.

12:00 - So, to me the question that I’ve been thinking about is, can we forecast ecology likely forecast weather? So what I want to spend the rest of this chunk of time before morning break talking about, is thinking about this from a theoretical perspective. Give a quick introduction to some of the methods we’ll learn about this week. And most importantly emphasize the idea, the importance of thinking probabilistically, when it comes to trying to think about how we bring models and data together, and how we make forecasts. This comes from the book, but if you had never worked with models or tried to make forecasts before, your perspective on modeling probably consists of, well, there’s data and there’s theory. We put data in theory together. We make a model, we run that model in the forward to make a forecast, and then that miraculously somehow informs decisions. Life is never that simple.

12:58 - And we’ll go over the course of the week, Many of the components of, why life is not that simple, but ones that I think I really wanna emphasize, is because everything that goes into these models has uncertainty associated with it. There’s uncertainty in our theory that reflects into uncertainty about how models are actually structured. Unlike the physical sciences, we don’t have physical laws behind a lot of our models. A lot of them are empirically inferred. There’s therefore uncertainty about the parameters, there’s uncertainties about the drivers, uncertainties about the current status of the world, and uncertainties about any of the scenarios they go into making projections. Because of that, those uncertainties compound, and we result with uncertainties in our projections. If we think about….

13:47 - Tying to make forecast, one of the questions will obviously. comes up is, are we any good at this? Or how do we measure the predictability of different ecological systems? And one of the things I’m gonna put out as one of the important measures of predictability is the uncertainty in those predictions. Rule of thumb, when you make a forecast, the uncertainty should increase as you go into the future. If you’ll become more confident about the future than you are at the present, you’ve probably done something wrong (laughs) So we’re gonna assume, and there’s some math by this, uncertainty is going to increase as we go into the future. So one way of measuring predictability is not just the uncertainty and the prediction, but also the rate at which that uncertainty increases.

14:32 - A process that becomes, where the uncertainty increases rapidly, is one that we’re gonna have a shorter horizon, that we can make effective forecasts over, process where that uncertainty grows slowly, gives us more time over which those forecasts are likely going to be effective. So how do we characterize systems as being predictable or not? So throughout this course I’m going to emphasize the use of dynamic models and forecasting. I understand very clearly, that they are not the only class of models that we use to make predictions, but I think they are particularly common in pretty much every subdiscipline of ecology, where we often think about predicting some state of the system, some Y, at some point in the future, T plus one, that some function of the state of the system right now, plus co variates or drivers given some parameters and some uncertainty. So if I take this basic framework, I can break down the uncertainties into the future. But what you can do is, you can break down, each of those things that go into that dynamic model, in terms of their contribution to the overall uncertainty.

15:49 - So we end up with a contribution coming from the internal dynamics of the system, these external drivers. And then what I’m gonna often do is separate the uncertainties and the parameters in a model into two parts, what I’ll call the uncertainty about the parameters themselves often. What is the mean of this perimeter, and distinguished it that from heterogeneity and variability in processes that cause heterogeneity and variability in parameters, often what’s the statisticians would call the random effects. So being able to separate, what is the mean of this parameter from how does the process and the parameters vary in space and time? And then just the remaining unexplained error. And I’m also gonna often separate out a traditional residual air into observation air as separate from process air.

16:44 - So observation errors, the errors in the data themselves, which is separate from errors in the process. The fact that our understanding of systems is always incomplete, so there’s always something our models do not capture. And so that’s what I’m gonna mean by process error. So each of these five terms follows a very similar pattern, whereby the uncertainty, the contribution of that term to the future involves two parts. The uncertainty about that part,,, So say for example, the uncertainty in the initial conditions, translates into the uncertainty in the future, and then the sensitivity of the projection to that particular item. So every that particular term. So every term involves this.

17:31 - What is the uncertainty about that component, and what is the sensitivity to that part? The internal component involves uncertainty about the state, which is the initial condition, uncertainty and the sensitivity of the forecast to that state, which is actually equivalent to what we all learned in intro ecology, of system stability. To make this a little more concrete, I’m also gonna…. So what I’m going to do next is, I’m gonna kind of walk through the implications of each of those uncertainties, for how we make ecological forecasts. But in doing so, I’m gonna rely on a very common model that we’ve all, again, all seen since intro ecology, the logistic growth model as an example for how we characterize uncertainties and propagate them. I’m also gonna kind of…. One of things we’re gonna see over the duration of this course, is the difference between fitting models as processes, fitting them as dynamic time series processes with uncertainties versus what you would get if you fit that model as a function.

18:42 - So if you take the logistic growth modeling, just treat it as a function, and you fit it to this data in black, the green line actually ends up being the best fit, which is amazingly bad (laughs). By contrast, if you fit that model as a process, by which populations grow iteratively over each year with uncertainties, you can actually, you’re your best fit. You can make get to follow the data. But again, that’s because you have access to the data. What are each of these terms and how they affect the predictability of ecological systems? So, like I said, the first term has to do with internal stability of the system, where this idea of stability is the classic one we’ve all seen thousands of times. As an ecologist, like anyone else cannot resist the urge to draw pictures of balls rolling into valleys and off Hills.

19:37 - (audience laughs) Everything we learn in intro ecology theory, is actually relevant to forecasting. But it’s not the only thing relevant to forecasting, but it does matter because the stability of systems does affect their predictability a good bit, as well as combine that with our needing to understand, what’s the state of the system right now. Qualitatively, we can divide that distinction between stable and unstable into making two different predictions about how our forecast uncertainty is gonna change over time. So this is time on the x-axis and this is how the predictive certainty is growing. If the system is unstable or chaotic, that predictive error is going to grow exponentially as we move forward in time.

20:27 - This is exactly what characterizes weather forecasts. Whether the atmospheric system is unstable, it’s chaotic, and therefore the uncertainty about the current state of the atmosphere, will grow in time and come to dominate the forecast error. It’s actually because of this realization that the atmosphere is chaotic, that one can characterize atmospheric weather forecasting as an initial condition problem. So the nature of the forecasting problem in atmospheric sciences is that initial condition problem. If I want to reduce uncertainties in my weather forecast, essentially the way that I do that, is I reduce uncertainties about the initial conditions.

21:10 - Because of those five terms, that is the one that dominates weather forecasts. I pause it, that is not the case for most ecological forecasts. So one really important distinction between ecological forecasts and weather forecast, is I think the relative importance of this term relative to the others is very different. That said, I do think there are places, such as emerging epidemics and ecology where there is evidence that systems are chaotic, where this high sensitivity, the initial conditions really does matter. And so there may be cases in ecology where that initial condition uncertainty dominates.

21:50 - But I do think most of ecology falls in this case of having some form of stabilizing feedbacks. I might not know exactly the state of the system right now, but I know, as it evolves through time, that there are stabilizing feedbacks, that keep those bounds. So if I think about processes like succession, succession has stabilizing feedbacks whereby trajectories of systems, their uncertainties are bound, they don’t diverge exponentially. Other thing to note is this internal stability term because it is a feedback, is the only one that grows or declines exponentially to a first approximation all over other uncertainties grow linearly. but we don’t know their relative importance. So here’s a example in a logistic model.

22:39 - If we have uncertainty about the initial condition, this is kind of a constant envelope showing the overall uncertainty converging as you move to that carrying capacity. in these dash lines are just individual ensemble member realizations of that process. One of the things that’s interesting about the logistic model, is it also has the capacity to be set parametrically into that chaotic range as well. If you take this model and put it into chaotic range, then this is just one realization. If I make an ensemble projection with some amount of uncertainty and initial conditions, we can see, you can almost looks like you’re doing pretty good and then suddenly you have no idea what you’re doing.

23:19 - But in practice I’m starting from a very small uncertainty, and that uncertainty is growing exponentially. And because I’m making a prediction, the mean of that essentially just goes back to the background means. So kind of at this point, you’re not doing any better than the ecological equivalent to climatology. You know, I know what season it is. I know the, the range of variability this process can predict, but I don’t know anything about where in that range I might be, and that that’s decaying at an exponential pace. And if I wanna predict this future, this process out further in the future, the way I do that is I need to crank down that uncertainty about the initial conditions.

24:00 - And if I crank that down, I can make predictions further out. But again, they eventually reach some limit of predictability. It’s an idea of what’s limited predictability is actually a very much more easy to define in systems like weather forecasting, because they so rapidly go from, “Oh it looks like I’m doing pretty good.” so I have no idea what’s going on. And in fact, one of the classic proofs by Lorenza, is that the weather itself has a limited predictability of about two weeks. You can crank down those uncertainties really tight and they will still blow up after about two weeks, which is in some sense the distinction between weather forecasting and…

24:37 - So like I was saying, one of the things that we saw, is that weather forecasting is this initial conditions problem, it blows up exponentially. Because of that, everything that weather forecasters have done over the 60 years, almost 60 years, that numerical weather forecasting has existed, Once they figured out which uncertainty dominated, they’ve essentially invested tens, if not hundreds of billions of dollars, on constraining initial conditions. Everything weather forecasters do, how they build their models, how they’d deploy networks or sensors, every satellite they’ve launched I’m the space, some of them make pretty maps that guy’s on TV point to, but that’s not why they have those satellites. Those satellites exist for the sole purpose of feeding data into an initial condition constraint. That’s what they do. That’s how they do forecasts. Knowing what dominated the uncertainty, drove this whole field into the direction of knowing, this is the thing as a field we need to do to improve our predictions. And that’s part of why. I think there’s an analogy in ecology.

25:55 - We’re trying to make forecasts and we don’t know what type of forecasting problem we have. Second term was the exogenous stability. So how sensitive systems are to their external drivers. When easy take home message from this, is that those systems are more predictable if they are insensitive to environmental drivers, or there’s not much uncertainty in those environmental drivers. So when does the latter happen? I think the latter happens in systems that where the environmental forcing is predictable, such as diurnal cycles, tidal cycles, annual season, seasonal cycle. So systems that are really locked to strong cycles, are gonna be ones that are more predictable. Because there’s not much uncertainty.

26:44 - Like, I can run solar geometry calculations out over billions of years. I know where the sun’s gonna be until it explodes. It’s not going anywhere. The tides are gonna be predictable, seasonal cycles, annual cycles, things like that. Another important take home from this, and I’ll classify this one is under, even if you don’t ever make ecological forecast yourselves, important to understand how the data we collect informs ecological forecasts. What does this term tells us? Well, it tells us that to understand the uncertainty in a global focus, we need to know the uncertainty in those drivers, in our inputs, and the sensitivity of that system? Well, what is the sensitivity? Sensitive is just a slope, right? Well, if to make forecasts, we need to know what those slopes are, we need to collect data that allows us to quantify what those slopes are.

27:45 - So in many ways, if you wanna make a forecast the hype, the question is not, does X affect Y? But, how much does X affect Y? So if you’re doing an experiment… In a Nova Design where there’s controlled treatment, allows you to answer the question, does X affect Y? But doesn’t allow you to quantify how much. It doesn’t allow you to estimate that slope. One of the things that…. I advocate very simple thing we can do to advance ecological forecasting, is to make more use of regression designs and experiments. Because they allow us to get this quantification of what is that slope? What is that sensitivity? The other thing we need to remember, is we need to know this uncertainty about the drivers, which comes back to, you need to report the uncertainties in your data.

28:35 - This is something I will again reinforce throughout everything we do, that to be able to make forecasts, we need to know the uncertainty in the data, and we need to know the uncertainty in the model. Without the two of those we can’t make forecast. And as we’ve gotten better as a field at making data more open, we very often have not gotten good as a field as as figuring out how to report the uncertainties about those data. This is actually one of the areas where I think neon is really pushing us as a field is, is figuring out ways to report, to quantify and report those uncertainties as part of just standard practice. Other thing we learn, if we have a covariate in our prediction, we need to be able to predict that covariate in the future too.

29:18 - So whatever your drivers are, we also need to be able to forecast those. And for example, in ecological forecasts, a lot of the things we care about, do depend on weather and climate, and so we’re tied to the predictability of those systems. It does point to a few interesting possibilities though. For example, you might actually use a different set of co variates to forecast a system, then you might use it to post hoc explain the system, because you might have a variable that explains the system very well, but that thing itself is unpredictable. Past economic activity might explain logging, but good luck forecasting future economic activity.

29:59 - Now I think that’s even harder than forecasting climate. So maybe you rely on other proxies that are more predictable. Other important takeaway, we all in stats probably learn something about model selection. You probably heard AIC this, you know whatever tests. You know there’s tests to choose between different models of different complexities, in an all forms of model selection explicitly or implicitly, there’s some penalty for complexity.

30:25 - This term, the uncertainties in the co variates, is in none of those standard statistical model selection terms. Which means, if you’re trying to make a forecast and there’s uncertainty about your co variates, you are not including that uncertainty and therefore you are always selecting for models that are too complex. Other important thing is, because you have to forecast X in the future , because uncertainty about anything increases as you forecasting the future. The relative importance of this is expected to increase, because you’re tying your uncertainties to something else whose uncertainty has to increase with time. ‘Cause that was one of the things we started with.

31:04 - The other thing, I think is useful to think about in these first two terms, this internal dynamics versus external sensitivity, is that it really, in many ways, bringing this back to again, classical ecological theory and thinking about endogenous versus exogenous forcing as not a dichotomy but as a continuum. What is the relative importance of these two factors in the predictability of a system? I mean I don’t expect any system to be a 100% internally driven or a 100% externally driven, but you have two terms that tell you how the relative importance of those two factors. Okay, parameter and certainty. I think this one is relatively straight forward, because it comes back to basic stats 101 concepts. As you increase your sample size, uncertainty about parameters declines under traditional sampling theory, which doesn’t always apply to every model. But under official sampling theory, you end up with one over square root of N, you square even in the denominator, so air comes predictably.

32:10 - So for a lot of problems, as long as there’s sufficient data, the parameters should, uncertainties should not be the dominant problem, but there are always going to be certain classes of ecological forecast problems, such as emerging infectious disease, invasive species, where there will always be data limited problems. And therefore will always be parameter limited. We’ll always have perimeter air as one of the things that needs to be taken into account. So if I deal with carbon cycle model, the set archive has petabytes of data. I can constrain parameters down to negligible uncertainty, but if I’m trying to forecast, some emerging infectious disease or invasive species has never been here before, I have very little data on how that system is gonna behave.

32:58 - So just again showing with the logistic case, if we have uncertainty about our parameters, we can sample over that and generate uncertainties about, how those uncertainties propagate, and this should be the part that they we’re most familiar with, because propagating parameter error into predictions, this is what a confidence interval is. You know, we’ll cover ways to deal with this, in more complex models. But that’s the same… It’s concept of what a constant rule is. It comes from propagating parameter. And in last bit, the errors in the processes, which can be both the due to inherent stochasticity and systems. Perimeter air will decrease asymptotically, but the inherent stochasticity in a system, may be somethings irreducible. Heterogeneity and systems may result irreducible uncertainties.

33:55 - Structural uncertainties in models, technically reducible, but every model is always an approximation of reality. No model is perfect. It will never be perfect. It’s not supposed to be perfect because if it was perfect, then it would be as complex as the real world. So there will always be structural uncertainty in models. I’ll throw out that, my personal hypothesis with some data behind it, but it’s still to be determined, still to be bought out is, when I think about these five terms, I think this one, the heterogeneity and variability, this is statistical random effects, is actually going to be one that’s going to dominate a lot of ecological forecasting problems. So if I were to put my money on it, I think ecology is a random effects forecasting problem, not an initial condition forecasting problem.

34:48 - It’s something we all encounter when we work in the field. You know, I study this plot, this watershed, this population, and I got this question in prelims and it stunned me. You know, if I take, I map every tree in this whole watershed and like someone on my committee asked, “What about the next watershed over?” “Can you predict what’s going on over there?” And it’s like, I don’t know. I have absolutely no idea. After measuring every tree and this whole watershed, whether it has anything to do with what’s going on in the next watershed. And I think one of the challenges that ecology is, you know, we’re coming from a discipline that very much historically was focused on that very small scale, heterogeneity, my plot, my watershed, my population and have not fought. We’ve been struggling.

35:32 - The one of the growth pains of equality is struggling to think about how we scale this up, how we deal with the heterogeneity and variability. The fact that when I move over to the next watershed, yeah, the carrying capacity is a little bit different, why? Well we don’t understand yet, but we need to accommodate those uncertainties even if we can’t explain them yet because they will have a big impact on our forecasts. So here’s a simple simulated experiment. So here are envision 10 plots, populations, whatever your favorite thing you sample measured over 10 time periods. So I’m going to call these sites in years. I’ve cooked up this example so that this set of time, 10 times series, and this set of time, 10 times series have the exact same variance.

36:17 - So if all I’m looking at is residual unexplained variants, they are identical. But in this case, most of the variability is site to site. In this case, most of the variability is year to year. I don’t understand why their site to site variability here. I might not understand why, what’s causing the year to year variability over here, but I can accommodate that variability even if I can’t yet explain what’s causing it and it impacts prediction.

36:45 - So if I am at this plot, and trying to predict next year, I actually can predict it fairly well. By contrast, if I am in year 10 but trying to predict what’s happening at a new site, I might have a lot of uncertainty. But what’s happening in a new site, because there’s a lot of site to site variability, they don’t yet understand. By contrasting this system, if I’m trying to predict a new year, for a site I already know, I might not have much confidence in predicting a new year. But if I want to predict what’s going on at a new site, in this 10th year I might actually be fairly confident.

37:22 - So again, the idea here, random effects can account for accommodate, account for the variability. Partition the variability that we can’t yet explain and it does have real impacts on predictions. That said, predict new sites and new years. Are equally uncertainty. You go back to the spec they’re identical. So here’s the show. You got the logistic model showing, for example accommodating, just additive process here at the end versus accounting for parameter variability.

37:51 - The idea that you know R and K themselves, may be changing from year to year. And that would be a representation of variability in the process here. It might be inter-annual variability in the process that we don’t yet understand, versus just residual variability. So one, take home from this, is we take a simple model that we all were exposed to in our probably first ecology class, if not an intro bio class. It’s a very simple model, I’m predicting and population size, given R and K growth rate and carrying capacity and I need some initial condition.

38:25 - So this model has three one state variable and three parameters. Okay, Well we just walked through, I needed to quantify 11 uncertainties, to forecast that problem with one state variable and three parameters. So I guess one of the things I want to throw out, as a take home here, is that when you move from theoretical modeling, we’ll just like has it been beat to death in theoretical modeling, to making a forecast with that model, all of your emphasis suddenly shifts, to the uncertainties. That’s where the bulk of the work actually ends up being done. So yeah, again, the message think, when you think about forecasting, you’re thinking about probability, you’re thinking about distributions a lot.

39:14 - Last bit related to the theory, was to point out in that overall term, simple version of the model of the derivation I showed that had five terms. I left all the covariance terms, but they’re in there. And the covariance terms are important and I think they could potentially tell us a bit about, how ecological processes might scale and definitely how ecological forecast should scale. So for example, as we move up in scale, we’re gonna average over the uncertainty and things like drivers and heterogeneity and process here. As we move up an average over variability, though should become less important, because they’re becoming less uncertain.

39:53 - So things like internal stability might increase in importance. It also tells us that if you want to scale things up when we make forecasts, it’s going to be very dependent, upon those co-variance terms. So the spatial and temporal auto across correlations. So that covariance term, gets really important when you move up. And so one of the things I’ve been, thinking a lot about is when I make, when I think about how a forecast scales is a first and foremost, what is the covariance? So like how, how does the information measured here, decay in space as I move further away.

40:25 - So what’s the rate at which information at one location is actually relevant, further on and then what might cause those correlations? So if you think about it this way, if the world was simple, I could predict what’s going on at this plot, predict what’s going on this plot, predict what’s going on with that plot and they’re all independent, in which case scaling is just summing up. So what causes that not to be the case, is the fact that there are correlations between these different things. If there weren’t correlations, then scaling would just be summing up. So what causes those correlations is key, to thinking about how forecasting scales. So to come back to the (mumble) Stan diagram, and think about it from what we’ve just learned, we now can see that, one of the things we see in this spread of ensembles, is when we have one single projection for each model, we’re confounding the structural area, the driver area, and the parameter area.

41:26 - Because each of these models themselves, should have a confidence interval around it. We have an initial condition present. We didn’t use any data to constraint. These initial conditions were constrained by the assumption the world was an equilibrium in 1850, which it wasn’t. There is no representation of process here. There’s no representative heterogeneity and variability in the system and any of these projections. And if we added competence intervals, which might be most important, we’re trying to predict in this case, I’m trying to predict the global carbon cycle, I don’t know, which of those five terms is driving the uncertainty. There’s evidence that structure matters.

42:07 - But again, we can’t say that for sure because this spread confounds, the fact that the models have different parameters, in the models have different drivers. Because they’re coupled have different atmospheres. The other thing that I want to point out about this framework, is it’s not just a qualitative one that makes qualitative predictions, but it’s also a quantitative framework that can be used to actually partition uncertainties, in real forecast. So this is a very simple forecast I made, of carbon flux at the Willow Creek flux tower in Wisconsin, using just a dynamic linear models. No, no complexity here. Just linear model of, current flux + temperature + light, I think.

42:49 - But I can partition out that uncertainty is in cumulative. This case is cumulative carbon flux into the contribution to that process here, uncertainty in the drivers, parameters, initial conditions. So we see that initial condition uncertainty dropping exponentially, as we expected we see the driver uncertainty increasing through time, because the weather forecast, gets more important and we see the process here , this is cumulative carbon flux. We see the process air declining over time because the random errors from every half hour start averaging out. So to come back and think about, kind of reinforce something I said earlier, one of the things we need as ecologists trying to think about, making forecast is to understand what type of forecast problems we have.

43:34 - And I think this is going to be really important because it allows us to think about, from a theoretical perspective what drives that Amex of systems, and how general are they, across different processes and across different locations. Also very practical. We need to be able to classify what sorts of problems are predictable, so we know how to tackle new problems. If we can go, well we learned that this certain class of population models are all dominated by, uncertainty X and therefore if we encounter a new problem that’s like that, it’s likely to be dominated by those sorts of uncertainties, and we can focus our attention on those. One of the challenges in ecology and ecological forecasting, is the temptation to say everything matters. Everything interacts with everything, and there for everything matters. But not everything matters equally.

44:29 - So we need to be able to say, if I’m encountering a new problem, where should I focus my efforts? What’s likely to matter? Cause I can’t measure everything. If I know a certain type of problem is going to be initial condition dominated, I may make a very different set of measurements, that if I know it’s dominated by driver, uncertainty or if I know it’s dominated by site to site heterogeneity, how I make measurements and how we make forecasts, how I monitor that may be very different. And so that leads me to the last bit about what we measure, how we build our models and how we assimilate data into models, is ultimately going to be driven by which uncertainties dominate, different classes of problems. Which brings us back to… The actual theory behind the the NEFI project. I kind of laid out, we’re forecasting a bunch of different stuff.

45:19 - Why are we’re forecasting the bunch of different stuff, essentially the things that were as different as we could find in the neon data catalog, to make forecast is independent of each other, to ask the question, are there common patterns of predictability? So if I learn something about predictability, in small mammals and ticks, does that tell me anything about my ability to protect, harmful (mumbles)? Are there common patterns of predictability? And so actually for me right now I would say, in this project is one of the most exciting periods of my career. Because not only have I kind of grown to becoming just a broad ecologist, but I’ve done it because, for the first time I actually see connections across ecology. I see that we are often working in individual silos, and not always learning from each other. And that’s because the way that we do work often doesn’t make it clear how work that one of you is doing, could actually affect our understanding of how other systems are working. And so you know, the kind of what we, the place we got to, I call kind of the first principles derivation of predictability, is that left us with a place where, because we don’t have a single model that applies to all ecology, but we have a framework, that applies to most ecological problems.

46:40 - we couldn’t take that derivation further to the point to say, like in the weather forecast where they could say, “ it’s always going to be initial conditions.” Instead, I think we’ve reached a point where understanding predictability, is going to be a comparative problem. We need, we need enough literature, enough synthesis, enough examples, of ecological forecast to start understanding, from a purely comparative perspective what is and is not predictable, in ecological systems. And so we’re trying to start building that library is are asking, do common things affect the predictability of different systems. Let me quickly in the interest of getting you guys to a break. Go through some of the methods.

47:24 - We’ve talked about the theory, we’ve talked about predictability. So a little bit about the how. So what are the methods that we use to make forecasts? So I’m going to, to focus on in, this idea of iterative forecasting, is the idea that forecast should be updated when new data become available. So we make a prediction, new observations, we put things into the model and make a prediction. The forecast, new observations become available. We want this ability to update those predictions, every time new observations become available.

47:55 - The parts of this class that I teach, which are not all of it, are going to emphasize that to take a very Bayes and perspective because Bayes theorem has this nice, inherent structure to it where, you take your prior understanding of the system, you update it with new data and you get an updated understanding of the system conditioned on that new data. And then this current understanding of the system can then become the Prior for our next round. So it’s one of the reasons that I think Bayes approaches are popular in ecological forecasting, is because Base theorem is itself, an inherently iterative way of doing inference. It is always taking what we know and building on it. That said, Barbara is going to come in tomorrow and give you a machine learning perspective, that is very much deliberately brought in because that is the other perspective that is being used in ecology, and it’s not how I think.

48:48 - And so I brought, I was like, I can’t teach you that. I want someone who can. So to show you how I think about things that I say. Here’s my current uncertainty in the system. I make a forecast that uncertainty gets larger, I make some new observation, of some new data with uncertainty and then by base theorem I can combine the uncertainty in the forecast and in certainty and the data to update. I get an updated (mumbles) system, which I can then, re forecast from.

49:16 - High level take homes, As I was saying earlier, we have to know the uncertainty in the model. We have to know the certainty in the data because the thing that we get out of data simulation, essentially is, that the precision of those two things controls the answer. If the model is imprecise, the forecast goes to the data. If the data is in precise, the forecast goes to the model. The relative uncertainty, So if you were failed to report uncertainties, about a model or failed to report uncertainties about a data, it gives that thing too much weight. It makes it overconfident.

49:54 - The other thing that I think is really powerful about this data simulation approach, is it’s way that it lets us combine observations. So this idea of data fusion. So a simple example, imagine that I have a model. Again, I’m very plant focused where I’m making a prediction of the fraction conifer on the landscape and above ground biomass and landscape. And let’s say that the forecast of that, that forecast has a correlation. So forecasts that predicted higher conifers, predicted more biomass, forecast that predicted lower conifers had lower biomass.

50:28 - These blue distributions are just the marginal distributions here. So if I just take these points and make a density of them, so then let’s say, imagine I make an observation of the fraction conifer on the landscape that could come from remote sensing inventory plots. In our paleo work, it’s coming from fossil pollen. I can combine that data with a forecast to get an updated estimate of the system, again, using Base theorem, but the powerful thing is, because these two things are correlated, I can also update above ground biomass, in proportion to the strength of this correlation. So data simulation lets you update what we call Leighton indirectly observed variables, based on the correlations in the forecast.

51:14 - And this works in both directions, If I observe above ground biomass, I can update fraction conifer. If I observe both, then the constraint on each comes from the direct constraint from that observation and the indirect constraint, by the things that it’s correlated with. And this is actually kind of, why weather forecasting works? Cause if you think about the weather forecast, they need to initialize millions of volume grid cells of the whole earth. They pixelate the whole earth in three dimensions, and they needed to know about a dozen state variables about every single pixel in the whole earth. Well, they can’t possibly make direct measurements on all of those every six hours, they’re inferring those a lot based on the correlations with other things that they can’t measure.

52:04 - The weather forecasting is in under specified problem. They don’t make enough observations to actually say what is the state of the system. They have a lot of indirect inferences. And I think this is something that we, potentially as a strength, that we can leverage an ecological problems as well because I think in a lot of ecological problems, we have many data sources, that each provide us partial information about the state of the system. And so I view model data simulation as kind of this prob… I call it as model as scaffold problem, where I have different observations about the state of the system.

52:40 - In this case, again, I think about the carbon cycle where I might have, remote sensing at a kilometer resolution and fossil pollen at a 50 year resolution, and in flux towers at a 20 Hertz temporal resolution, leaf level measurements over two square centimeters. And I can’t run a regression, across any of these things because they operate on such different temporal and spatial scales. But they each tell me something about how the system works. But if I have a model that makes predictions across multiple spatial and temporal scales, I can use that model as a way of letting different data sources talk to each other. And in some sense the model itself just becomes a giant covariance matrix, and it’s really a data driven model of how we allow different observations to talk to each other.

53:26 - So to wrap things up, ecological forecasting, it’s more than just running a model forward into the future. That’s actually the easy part. It requires this fusion between models and data, has to deal with multiple sources of uncertainty and variability. There’s multiple… We’ve emphasized five key ones, and the importance of thinking probabilistically. Those uncertainties are important. I think a lot of what I end up doing when I work on any forecast problem, is spending a lot of time thinking about, probability distributions and how… I guess one way to think… I didn’t say this earlier, thinking about the world probabilistically is important whether you believe the world itself is deterministic or not.

54:08 - So if you could believe the world is deterministic, but our understanding of that is incomplete, or knowledge of that is incomplete. So even if the world itself is fundamentally to terministic, we can use probability distributions to capture our current lack of… The fact that we have incomplete knowledge and incomplete understanding. So for me, thinking probabilistically is very different than slapping stochasticity on a model. So this is not about stats slapping stochastic processes on a model, It’s about reusing probability distributions to characterize and quantify, what we do and don’t understand about the world right now.

54:50 - And always viewing that as a work in progress. .