Can You See a Thought? Neuronal Ensembles as Emergent Units of Cortical Function

DARIO GIL: All right. Welcome, team. Good afternoon and good morning in Almaden. We’re livestreaming the event as well. And good day, everybody, for those of you who are connecting remotely. I’d like to welcome all of you to the second seminar of the year of our distinguished speaker series. Really, it’s my great pleasure today to welcome Professor Rafael Yuste. Rafael is a leading neuroscientist and professor of biological sciences at Columbia University.

00:33 - He has pioneered optical methods to measure and modify the activity of neural circuits of the cerebral cortex on the quest to understand how brains work. Rafael Yuste obtained his MD at the University of Autónoma in Madrid and his PhD from Rockefeller University and was a postdoctoral student at Bell Labs. He joined Columbia University in 1996 and is currently the director of its Neurotechnology Center and co-director of its Kavli Institute for Brain Circuits. In 2011, Rafael led a small group of researchers to propose the Brain Activity Map, precursor to the U.S. Brain Initiative. And in 2016, he helped coordinate the launch of an International Brain Initiative.

01:17 - He’s presently involved in establishing ethical guidelines for neurotechnology and artificial intelligence, what he refers to as neuro-rights. Raphael received multiple awards for his work, most recently sharing the Eliasson Global Leadership Prize. Today, he will present the use of optical methods to selectively image and manipulate the activity of neural populations in 3D in vivo and to alter behavioral choices. I had the good fortune of visiting Rafael at his lab at Columbia University earlier - late last year. I can tell you that it deeply impacted me.

01:54 - It impacted me for the possibilities and the capability of the science and the technology but also because of the consequences of what this is all going to mean to each of us and to our society. I could not be more delighted to have Raphael here today and share his pioneering research with our IBM community. Following this seminar, actually, Rafael is going to participate in the AI Ethics Board that we have in IBM to discuss this very topic in more detail and discuss some of the implications and actions that we can take and collaborate together. Rafael, the stage is yours. And welcome. >> RAFAEL YUSTE: Thank you, Dario, for inviting me. I’m delighted to be here. The story today starts with magnets. This is something that many of you are probably familiar with. I’m talking about ferromagnetism.

02:48 - You know that magnets have this very weird property, which is that if you split them apart into individual atoms, the atoms are not magnetic. But if you put them together, something happens and the system becomes magnetic. This is an example of an emergent property, also known as a collective property, which is a property that, by definition, is not present in the individual elements of the system but emerges when the system is put together. Magnetism 100 years ago gave a lot of headaches to physicists because they said how can you get something out of nothing - if there’s no magnetism in the atom, how can you get the magnetic material - until this undergraduate student Ernst Ising proposed a very simple mathematical model. This is the critical equation of the model in which he describes energy of the magnet, and you can imagine and understand energy as the propensity of a system - a physical system to change.

03:42 - So you have a lot of energy, you’re likely to change. He defined it with this negative integral of the sum of the spins of each pair of atoms times the coupling coefficient, this term J, which essentially measures how strongly coupled atoms are with each other. The higher the J, if the spins are coherent, the larger the term, which means the more negative the energy, which means the more stable the system is going to be. Armed with this equation, he predicted this existence of the states of matter, the spin classes in which parts of the material becomes magnetic because they have coherent spins that actually are, in a way, self-generated. There territories emerge, they’re an emergent property of the interactions of the atoms. They’re not magical.

04:33 - They just come from interactions of things. This is an example of - a classical example of an emergent property that has been understood scientifically. But now if I draw your attention to neuroscience, and you probably know that we don’t understand how the brain works, but there are a lot of things that make us think that the brain is the mother of all emergent systems, because if you want to build an emergent system, what do you do? You build a system that has many particles, many units, as many as you can, and then you enhance all these interactions because the emergent properties depend on the direction. You make them as strong as possible and as rich as possible. Guess what? If you look at the evolution of the central nervous system, nature is systematically increasing the numbers to the point that, in our brains, we have approximately 86 billion neurons.

05:32 - Okay? Astronomical numbers of neurons. And the connectivity of the brain is still unknown, but, on average, our neurons are probably each connected to about another 100,000 other neurons. It’s not at all to all connectivity, but it’s a huge number of connections. That’s exactly what you need if you’re playing this emergent property game. It’s very likely that the brain is optimized for the generation of emergent properties.

06:02 - As I told you, it’s critical for scientists to understand the brain for three reasons. One is because it turns out that this organ happens to be the substrate of our mental activity, of our minds. As humans, we define ourselves by our minds. We are the ultimate cognitive animal. Essentially, everything that you are, your identity, your thoughts, your imagination, your perception, your emotions, everything comes from this generation of activity of these neurons that you have in your skull. If we understood how the brain works, we would actually understand who we are for the first time from the inside scientifically. We’ll understand what is a human.

06:46 - What does it mean to say that word human? The second reason has to do with medicine. You probably know from personal experiences with friends or family members that mental or neurological diseases have no cure. It’s the dark corner of medicine. They afflict a large proportion of the population, which is increasing as the population is aging. The reason we cannot help these patients is because, as doctors, we don’t understand the pathophysiology of the system, of the nervous system, which is when the physiology, the normal function turns wrong, is defaulted, and that’s the pathophysiology. If you cannot understand the physiology, good luck with the pathophysiology. There’s no way you can solve the problem.

07:32 - That’s another reason to understand the brain. And the third reason probably you realize working at IBM that has to do with the secrets of bio-inspired algorithms that are in there because brains not just of humans but of all animals are computing all kinds of sophisticated optimization problems with hardly any energetic cost. I think our brain is the wattage of a small light bulb and we can achieve feats of computation that are unheard of for technology. There has to be - the ways in which brains compute have to be critical to invent new technologies in the future. Why don’t we understand how the brain works? It turns out neuroscience for the last 100 years has been anchored in a theory and a paradigm which is called the neuron doctrine, which assumes that the unit of structure and function of the brain is the individual neuron.

08:40 - It was actually called doctrine because it was a religious belief. It was something that had to be true. It wasn’t a hypothesis. It was a doctrine. Pioneers like Cajal and Sherrington using methods that revealed the structure and the function of individual neurons started the wave of research that continues to this day in which people have been taking apart brains one neuron at a time, describing how the neurons look like, like this beautiful image from Cajal’s drawings, to recording the activity of individual neurons with electrodes. Sherrington was the first person to develop the method to record the activity of individual axons of individual neurons. What we’ve been doing for 100 years is to record the activity of one neuron in one animal and correlating that with the behavior of the animal or in one patient and correlating that with the pathology of the patient, for example. If this is an emergent system, to do that, it’s a little bit like trying to watch a movie on a TV by looking at a single pixel.

09:48 - The image in a TV is an emergent property of the pixels. It is built with correlations in space and time and color of the individual pixels. By definition, it doesn’t exist in the pixels themselves. This is another example of emergent property. Imagine the foolishness of trying to understand something like this by looking at single pixels. Unless you capture the pixels all at once, you won’t be able to see these interactions. Because of that, people already 100 years ago started to think that maybe the neuron doctrine was wrong. It was actually someone in Cajal’s lab, Lorente de Nó, who was studying these circuits which Cajal called the impenetrable jungles where many investigators have lost themselves. He was speaking about himself because he could never figure out the logic of the connectivity of the cerebral cortex. He could never break the code there. Like a good disciple, Lorente did the opposite of what he was told and spent his whole life studying this impenetrable jungle and he came to the conclusion that most of the connections in the cortex of vertebrates were recurrent loops, were feedback connections.

11:06 - This is something that he called chains and he imagined that the whole purpose of this is to generate what he called reverberations, which essentially means that you have a circuit motif - I have an example here, like this one here on the top right - in which you have a neuron that’s receiving input from the left. It’s getting turned on and it’s passing that information to another neuron to the right, but it’s also sending these feedback loops of connections that activate other neurons which activates itself again. In a way, by doing that, this is not a trivial design principle, because when you do that, you can generate the state of activity which is independent of the input because it’s self-exciting. What Lorente thought, he argues that the purpose of the whole nervous system is to do that, that it’s all wired like that, to generate this internal state that he called reverberation. The reason this is important is because if you can do that, then you have something inside your head which exists independently of the world.

12:19 - That’s neat sensory input to get turned on. It can be activated independently. If you can do that, you can imagine how evolution could then use these states of activity, these endogenous states, as building blocks and as symbols for things in the physical world. And from that point on, instead of manipulating the physical world, you manipulate the model. You have a model in which these are pointers that are symbolizing things that happen outside. That was the critical insight that Lorente put on the table, and this was also picked up by Turing who came out with the same conclusion. This is all feedback business.

13:04 - This is an example of an emergent property because it doesn’t exist in the individual neurons. You need the neurons to connect to themselves to generate these intrinsic states. The same idea was picked up by Hebb who, quoting Lorente, changed the name, and instead of chains, reverberating chains, he called these things neural assemblies, but the idea was identical. You have a group of neurons - in this case, are the notes of this graph, and the edges are the connections between the neurons - that are all connected to themselves in one way or the other, so that means you can turn the whole thing on and then it remains on regardless of the input. What Hebb did propose, the originality of Hebb, is to argue that these things will build automatically if you link these neurons through a synaptic plasticity rule.

13:57 - The Hebb case is the famous Hebb Rule that the neurons would - neurons that fire together will wire together. If you have neurons that are firing together because maybe they’re responding to a stimulus outside, and eventually they can wire together to the point they can self-excite themselves independently of the outside stimulus. Again, at that point, you essentially are off from the physical world and you can start building a mental world. Going back to magnetism, John Hopfield in ‘82 published a very influential paper in which he took the Ising model wholesale and he applied neural networks. He argued that if you have a neural network - in this case, these little triangles, the neurons, and these are the axons - if this neural network was connected in a feedback fashion, in the ideal case where the connectivity graph is complete, and if these connections are symmetric, he argued this is isomorphic with the Ising model. He defined a concept of energy.

15:04 - In this case, it’s not thermal energy, but computational energy. But again, it captures the tendency of a physical system to change. He defined it just like Ising did with a negative term. The Vs here are the activity of each pair of pre and post synaptic neurons. The Ts are the Js, the coupling coefficients. In this case, are the synapses.

15:30 - If these neurons are strongly connected and their activity is congruent, you can have a very big term here, a very negative term, and he predicted the existence of these spin glass like states in the nervous system which he called attractors which with these stable states of activity with a group of neurons would be firing in a stable or semi-stable duration. He took this model and said if you can do that - well, he predicted that this would happen the minute you have these conditions and said, well, you can do that and then you can build yourself a universal computer because you can build a system of attractors, stable states in the dynamics of the neurocircuits, you can, in fact, understand the internal computation as an example of classical mechanics, and these stable states could be imagined to be the solution to computation or a memory. If this topo map would represent all the potential activity states of your brain, let’s say, and imagine you have these two valleys, these are the attractors, and we’re looking at the energy defined, as I told you, with the Ising equation, and then imagine that your activity at any given time is like a ball running through this landscape. As it gets close to the valley, it gets attracted, it drops down automatically to the bottom. That’s why these things are called attractors.

16:58 - This way of computing has the property of adding completion. You can arrive to the computation or to the memory with partial information. You don’t need to specify the entire path. This was the Hopfield model. I was lucky that I overlapped with young Hopfield when I was at Bell Labs. We were in the same department. I had many lunches with him. I came from the neuron doctrine camp. In fact, my thesis advisor was the great, great scientific grandson of Sherrington, so like straight through.

17:33 - I listened to John talk about these things. The brain is not completely connected. The synapses are not symmetric. And said, no, no, no, they don’t have to be completely connected. You can achieve the same thing with a random connectivity. If the synapses are not symmetric, you cannot prove the map. That doesn’t mean that this doesn’t work like that. It’s just that our map is poor. It doesn’t mean that the brain is not doing it. I talked to him, how can we prove you wrong? Okay? You have a new theory, you have to falsify it. He laughed and said, well, it’s going to be difficult because you have to record from every neuron in the brain to map all the attractors first and then you have to be able to move the ball to any pint in the landscape and predict/test what’s going to happen. Anyway, so I kept that in my mind. And then many years later, actually, this has resulted in the Brain Initiative. I’m not going to talk about that. But the Brain Initiative, essentially, it’s the research program to develop methods to do exactly that, to map the activity of every neuron in the brain, to see the entirety of the screen, and to be able to put the ball wherever you want, to be able to manipulate the system in a multi-dimensional space so that you can actually achieve any position in this dynamic landscape.

18:52 - I’m going to spend the rest of my talk telling you what we’re doing in my own lab at Columbia to try to advance that program, to explore if there’s emergent properties. This starts also with chemistry. It turns out that when I was at Rockefeller working with Larry Katz, we found out - well, how can we measure these properties? We have to put electrodes in every cell. This is not going to happen. You’re going to turn the brain to Swiss cheese. We decided to use optics, to use light. It turns out that you see high affinity calcium chelators. You can make them fluorescent by attaching the fluorophore.

19:34 - And when they bind calcium, they change the spectral properties, which means that if you make the spectrum of this dye, you can tell how much calcium there is. Working with Larry Katz at Rockefeller in the Whistle Lab, we found out by chance that you can use calcium indicators to label every neuron in a neural circuit. In this case, it’s a brain slice. This is a brain slice of the mouse cortex. Every one of these white dots is a neuron. It’s labeled. And they’re alive. They’re labeled with this calcium indicator.

20:04 - The calcium indicator gets into the neurons and then you can take images with a camera or a microscope and see in real time the concentration of calcium in the neurons, which was interesting if you care about calcium. What was really lucky is our chance discovery that if we took images of the calcium concentration of these neurons over time, these neurons seemed to flicker, the calcium concentration could increase and decrease. This has to do with the fact that whenever a neuron fires an action potential, so this is an electrical recording of the activity of a neuron as a function of time, and these lines are the famous spikes or the action potential of the electrical signals the neurons generate. If you measure calcium at the same time, every time there is an action potential, there is an increase in the intracellular or free calcium concentration, which means that we can take movies. This is a process movie. Every dot is a neuron. Ten minutes in the life of a piece of brain of a mouse visual cortex.

21:08 - And whenever these neurons are activated, they turn red. The reason is because they fire action potentials and they change their calcium. Through measuring calcium, we can indirectly interrogate who is firing when. This is 500 neurons in the brain of a mouse. The mouse has about 100 million neurons. It’s a little corner of that TV screen of the mouse. But in that little corner, we can see every neuron. It’s the beginning of the program to essentially watch the entire TV screen and explore if there are emergent properties. I was also lucky that when I was at Bell Labs, I worked with Winfried Denk who invented two-photon microscopy. Two-photon microscopy uses consultative physics, uses a nonlinear absorption of photons by excitation of fluorescence by infrared ultrafast photons that have two advantages for microscopy of living tissues. One is because it’s infrared light. It penetrates deep into living tissues. You want to image with regular microscopy the brain.

22:25 - It has the applicable properties of a glass of milk, so you cannot see. But with infrared, yes, you can go in and see. The second property is the nonlinear excitation generates a point spread function, essentially a focal point, which is narrowly restricted in space to a tiny little area. That means that you have like a magic wand that you can use to focus the light deep into tissue without being dominated by scatter. These are two fundamental advantages which, again, when we worked on this, we didn’t know they were going to be that great. Just one of those things that happened. Luck. Serendipity.

23:08 - But nowadays, many labs in the country, in the world, are using two-photon calcium imaging to measure the activity of neurons with these methods in living animals. Going back to the gist of the talk about these emergent properties, the emergent properties in brain tissue. Using two-photon calcium imaging, we started to look first in brain slices, just pieces of the brain taken from the cortex of mice. We can keep them alive for a few hours, label them with calcium indicators, use two-photon, and see who is firing there. When we did that, we noticed these big changes in fluorescence.

23:47 - In this case, they are plotted negatively, but they correspond to the action potentials. You can build plots like this. This is what we call a Rasta plot where in the Y axis you have neurons, so every Y value is a neuron, and X axis is time. This is a graphical representation of a movie like the one that you just saw where you have the neurons firing. In this case, every black dot is whenever the neuron fires at least one action potential using this calcium imaging method. You can see these lines of black dots that run down, and this is a coherent activity by a group of neurons that fires together during a period of time for a reason that we don’t understand.

24:36 - This is something that we call ensembles, but this is the same idea that people have defined with other terms. You could call them chains, attractors. It’s essentially a coherent co-activation of a group of neurons. It’s an emergent property because you would have never seen that if you were looking at individual neurons. You have to know what the other neurons are doing, and realize that when you fire, you’re part of a group. We found these, discovered these ensembles in these tissues and they had very peculiar shapes. Some of them were grouped.

25:10 - This case is an example of the neurons in red are the ones that belong to one of these ensembles. In this case, they’re a group near each other. In this case, they form a layer or a little column. Most of the time, the groups of neurons that form these ensembles were spread out through the tissue, through the slice. They would look at it in different parts and they would somehow know to fire together. This is, in brain slices, spontaneous activity. We weren’t doing anything to the tissue. They just did that continuously. We couldn’t stop it. Now because it was in brain slices, many people thought this must be an artifact of cutting the brain, taking a piece of brain out of the animal. But it turns out we’ve done the same experiments in living - we did it in mice and we see the same thing. In the top, you see a mouse that’s awake, his head skull is attached, head fixed to the microscope. He’s running on this spherical treadmill and he’s watching this TV screen where we’re showing him some visual stimulation and simultaneously there is a laser, an infrared laser that you don’t see, that’s going through the skull and we’re taking data.

26:19 - This is the raw data coming out of the microscope. This is a two-photon microscope. These are the neurons. They’re labeled with calcium indicators. If you look at the raw data, you can see how these neurons are flashing. These are the action potentials. You can now build - this is the analyzed version of that movie which we color in red. The neurons have statistically significant changes in fluorescence corresponding to this action potential. If you look at this movie, you see these groups of neurons firing together.

26:49 - See how the neurons are not firing individually? They fire in groups. That’s the ensemble. That’s what we call an ensemble. In fact, I bet you that in your brains right now, that’s what’s going on. This is exactly the challenge of neuroscience, how to decipher these coherent patterns of activity and understand how they relate to your thoughts, to your memory, how they build your mind. In a gist, this is what neuroscientists - the holy grail of neuroscience is to understand this code. But the good news is that we can actually measure this in awake behaving animals.

27:26 - Lo and behold, as I told you, we see the same thing we saw in slices. The way we analyze this data with a computer program, we tag the position of every neuron in the movie. For every neuron, we compute the fluorescence intensity as a function of time here. And then using an algorithm, we estimate the probability that the neuron has fired an action potential as a function of time. These are these spike probability plots. And then we build a Rasta plot like the ones that I just showed you earlier in which every line is a neuron as a function of time.

27:56 - You can see how this Rasta plot is dominated by these vertical stripes which you can see when you collapse the activity of all the neurons in this histogram. This is a percentage of neurons that are active as a function of time. You see how the activity of the cortex of this mouse - this is in our lab in New York - is dominated by these little earthquakes. These are the ensembles. This is what I’m talking about, this moment of neurons, about 6%-10% of the neurons that, for reasons we don’t understand, they fire together for a little bit and then someone else fires together. It’s essentially this jumping between ensembles that we are finding.

28:35 - This happens when you show the animal a visual stimulus like the one that you saw in that movie. This is an example of Rasta plots for gradings where we’re showing the animal these high contrast gradings. The ensembles are painted in red. In other words, the action potentials that are statistically belonging to an ensemble with very rigorous conservative criteria are colored in red. You appreciate that the majority of the spikes are part of this ensemble. If we use criteria which is less rigorous, less conservative, then practically all the spikes are part of this ensemble.

29:15 - That happens also if the animal is watching a movie that resembles the natural scenes that mice probably are watching in the wild. We use BBC documentaries of nature - this is an example - so that they represent the right spatial and temporal frequencies of natural images. This also turns on all these ensembles. But look at the top case. This is spontaneous activity with the light off. The animal is in the dark or the animal is looking at the gray screen, homogeneously gray screen that doesn’t change. Notice how this whole - it’s also filled with these ensembles.

29:55 - The brain is not turning off if there’s no input. It continues to fire, and it continues to fire with these groups of neurons. In fact, if you analyze the position and the properties of these ensembles and their spontaneous activity versus the ones that are generated during evoked activity, you find that they’re the same. Okay? This is an example of two ensembles from a movie of a mouse. The neurons in red are the ones that are part of the ensemble.

30:25 - This is an ensemble at some point, and a little few seconds to a minute later, this other ensemble happens. But then if you analyze the data on the spontaneous activity of the same mouse when the mouse was in the dark, you find that this ensemble has happened before. That’s the one on the top. The neurons that have the green circle are the ones that are repeating between this spontaneous ensemble and this evoke ensemble. Ensembles actually never repeat exactly. There is a liquid quality to this activity, so that when it repeats, you can see a core of neurons are still the same and that’s how you know that it’s repeating, but there’s always some new neuron that comes in and some other neuron that goes out. There is some plasticity in this type of representation.

31:12 - But our hypothesis is that we can identify these visually evoked ensembles in this spontaneous activity. The idea is that when the cortex builds a response to a visual stimulus, it’s using as building blocks patterns that are already there. It goes back to this other hypothesis from Lorente and Hebb and Hopfield. We have these attractors already there and they are doing some function. And then when stimulus comes in, we line them up in particular ways, but we don’t come out with a new type of representation.

31:53 - We essentially are using these internal building blocks. This raises the issue of what are these attractors doing, these ensembles doing? To test that, for us, the dream experiment is something that we call to play the piano, which is the following. You make movies like the one I showed you where you capture these ensembles in blue on the top and then you activate them artificially and play back as if each neuron was a piano key and you play back the same melody that you’ve seen - that you heard by watching these movies. And then you ask the question what happens if you play that back? If the ensembles are important, we play this back, something is going to happen. If there is some sort of noise or a phenomenon of the statistics of firing the brain, then who cares? If you don’t play it back, it’s like playing back noise.

32:49 - To do that experiment, we needed a way because these ensembles, as you’ve seen, are intermixed in these cortical territories. You see how there is one ensemble, another one. They’re all mixed together. Neurons near each other can belong to different ensembles. You have to have singles of precision and turn on one ensemble without turning on the other. That’s something we achieve with special light modulators building a two-photon hologram in which we can write words with two-photon lasers with a phased device - this is the four-year equivalent of that work - and generate the two-photon word project Into the Brain. This is a digital hologram.

33:35 - The student who pioneered this, Nikolenko, is a brilliant guy. He decided to paint pictures of Cajal, get the face picture, and project it down into the brain as if the brain would care that there was a light with Cajal’s picture on it. That’s what the graduate students do when you’re not looking around. This is our piano to turn on ensembles. And then we also used an opsin, a protein that we express into the neurons, and it’s activated by light and makes the neuron fire. This is an example of two neurons that are expressing this opsin, C1V1, and with the holographic piano, we can turn on both neurons simultaneously. They’re near each other in space. This is the worst possible scenario. They’re only 20 microns apart.

34:24 - We have the selectivity thanks to the nonlinearity of the photon excitation and the digital holography of turning one neuron on or the other with no crosstalk so that we can really play the piano. We started to play the piano in this experiment putting a mouse in the microscope and two lasers, one laser to image the calcium, this is the top imaging path, and another laser through the SLM to generate the holography to play back those patterns. We capture the pattern first and then we compute the face mask, put it in the SLM and turn those neurons on in the same order, in the same precision that nature has played them. In the control experiments, at the beginning, we decided we were going to play the piano with our elbow, trying to activate many neurons at once. These are very difficult experiments. You have to go through the skull of the animal. You need two two-photon lasers. The whole thing is really complicated.

35:28 - This experiment - Luis Carrillo, who was a [indiscernible] in the lab, decided to stimulate the neurons on the left side of the image all at once just as if he was playing the piano with your elbow. This is the calcium imaging results. Lo and behold, whenever he turned on the laser, he got all these neurons on the left to fire. He was happy. He was like I can play the piano successfully with activating all these neurons at once in one-half of the field. And then he comes to me and said, Rafael, do you remember the experiment that I was turning all these neurons at once? I said yeah, sure, what happened. Guess what? These neurons are still firing together after I stopped stimulating them.

36:10 - It turns out that he discovered by chance that if you activate these neurons 50 to 100 times, they bind together, they glue together up, and they become an ensemble artificially. They start to fire spontaneously even if there is no stimulus. That happens even a day later. Okay? This is not a minor change in the reprogramming of the circuit. This is already there, still there a day later. These are the calcium traces of these neurons. The dotted line represents when one of these spontaneous imprinted ensembles is firing. This showed us that these ensembles can be created when we activate the neurons with our piano. And then Luis said, okay, let me - this is a gross experiment activating all the neurons at once. Maybe that’s an artifact. Let’s just play the piano one neuron at a time. And then he started activating the neurons one by one.

37:16 - In this example, he’s turning on this little neuron here. Whenever he turns on the laser, this is the calcium trace of that neuron, the neuron fires. He was happy that he was playing the piano with one finger and that was working. But then when he played the piano with one finger in the group that he had imprinted before, very often when he played one neuron, the entire group came together. He did pattern completion. This is an example of a pattern completion experiment.

37:48 - The neurons in pink are the ones that belong to an ensemble. And that neuron 25 is the one that when he plays it, it trips the whole ensemble. It doesn’t happen all the time, but it does happen. This is an example of the Rasta plots. These pink lines are when the laser is on and it’s turning on neuron 25, which is this one here in the middle. This guy here. You turn it on here and nothing happens. Turn it on here, nothing happens. But look what happens here. You turn it on and, boom, bingo, you start to get all these other neurons to turn on.

38:21 - They are the ones that belong to the same ensemble. We hit ourselves directly with pattern completion. We build these ensembles artificially unbeknownst to us and they have pattern completion. This looks like a picture-perfect attractor, like the Hopfield model. We think that the way we do pattern completion is that we’re activating these neurons together and somehow their connections are getting strengthened, maybe through [indiscernible], and then we turn on one and we bring in everyone else.

38:50 - This is sort of illustrated here in this diagram in which you have a period. You can think of this as memory storage. You have a period in which you imprint. By imprinting the ensemble, we’re really imprinting a memory, and then we read it out by turning on only one of the neurons. This pattern completion was very robust. It was there even a day later. This is an example of a similar experiment. We’re turning on the neuron in blue, tripping on this ensemble where there is a star. We bring the animal back to the animal facility.

39:26 - We test the animal the next day and turn on this neuron 19 again, and, bingo, we get the ensemble back. We don’t know for how long this pattern completion occurs. It’s still there. But it’s probably many days. It’s a long-term change in the circuitry. That means that these ensembles are not just some sort of fake result from imaging. You can build them. They’re physical things you can actually generate. And they have these properties of pattern completion which are not coincidentally some of the key properties that theorists had identified in these models of emergent properties.

40:05 - But we still don’t know if they’re good for anything, if they’re used by the brain to do any computation. We have to do this piano experiment I was telling you about. It’s essentially go between the behavior, measurement, manipulation, and see what happens to the behavior. This closed - this loop, closed loop. The behavior we chose is to show the animal stripes of light - in this case, vertical stripes - and whenever he sees them, the animal licks. I don’t know if you can appreciate the animal is licking because he saw these vertical stripes of light.

40:37 - We show the animal horizontal stripes of light. We train him not to lick. This is what we call a go/no go task in which showing the animal one stimulus, one visual stimulus or the other, the animal behaves in one way or the other. In this behavior, he’s telling us what he saw. This is a very robust training. In one week or two, we trained the animals close to 90% success. Simultaneously, we’re imaging with two-photon calcium imaging in these awake behaving experiments the activity of these neurons. This is the experimental design.

41:19 - On the top, we show him these vertical stripes of light in blue, and that turns on these blue ensembles of cells that correspond to the vertical stimulus and the animal licks. In the middle, we show him horizontal bars, a different ensemble which is in between and interspersed with the ensemble, it turns on, the green one, and the animal doesn’t lick. This is the no go. And then the bottom experiment is like we don’t show him anything, we turn on our piano, and we play back these patterns to see what he does. This is the first experiment we did. It was to activate the no go ensemble. In this plot, it’s just one example, one experiment in which the blue bars is when the animal licks. The short red bars is when we show him the go stimulus.

42:13 - The top trays, every time there is a - not every time, but many times when there’s a go stimulus, the animal is licking. The tall red bars is the no go stimulus. When we show him the no go bar, the animal doesn’t lick. The same animal, the same stimulus, and we’re playing with our piano the no go ensembles, and these are about 10 neurons. Again, in the brain of a mouse, it has about 100 million neurons. When we do that and turn on these neurons, the animal stops licking altogether.

42:48 - We’re blocking the licking by activating the no go ensemble. This is illustrated here. Maybe it’s a complicated slide, but maybe look at the performance of the mouse. This is the behavioral performance, the licking. This might start with 70 to close to 100% success in licking for a visual stimulus. When we show him the visual stimulus with playing the no go ensemble, there is a significant decrease in the behavior.

43:20 - We’re controlling the behavior and making him stop licking or licking less by turning on the wrong neuron, so to speak. The next experiment is, okay, now we’re going to turn on the right neurons and we’re going to show him the right stimulus, but we’re going to lower the contrast of the stimulus so it becomes really hard for the animal to see that there is, in this case, these vertical bars of light. The ones you show are high contrast, 100% contrast. If you lower the contrast, not even you will be able to tell the difference if there is a bar there or not. We lowered the contrast. We showed him the stimulus. The animal doesn’t lick because he doesn’t see the stimulus.

44:01 - And then in the bottom, we’re showing him the same low contrast stimulus and turning on the go neurons and look what happens. The animal now suddenly seeing what he didn’t see before. He’s essentially licking to this low contrast stimuli. It’s quantified again in this complicated slide. But if you just look here, this is the performance of the animal with low contrast. That’s why the performance is very low, about 50%. If we activate this go ensemble, we increase the performance significantly. The final experiment is no stimulus. We turn off the stimulus, we turn on our piano, and then we activate the go ensemble. To do that, we use pattern completion. We turn on the neurons that we know are pattern completion neurons. We tried to do it with one. We couldn’t do it with one, so we did it with two.

44:59 - We were activating literally only two neurons in the brain of this mouse with optogenetics, with this holographic device. In this case, these two neurons, neuron nine and eight. Whenever we turn on our hologram, they fire. Not all the time. In this case, they only fired - nine only fires. I don’t know why. This is biology. They both fire. Nine is more reliable. Most of the time, nothing happens. But look what happens here. These two guys can trip the ensemble, the go ensemble. When that happens, the animal licks. That’s the star.

45:35 - This is the position of these two neurons, the position of the neurons in the go ensemble, and the quantification, I think you should look at this plot B. The performance of the animal when we activate these two pattern completion neurons and we recall the ensemble goes from 20% to 80%. It’s not seeing anything with his eyes. We’re just turning on the two key neurons that are tripping that go ensemble and that generates the whole behavior. If we quantify that behavior, it’s identical to the behavior that he had when he was seeing this go stimulus. No difference in the delay to lick, the velocity at which he’s licking, the duration of the licking.

46:16 - For all we can see, he interprets this as a visual stimulus to lick. In a way, we’re putting a hallucination in his brain by turning on these two neurons. Again, two neurons in 100 million neuron brain. We presume that this pattern completion of turning on one ensemble is generating a whole avalanche of pattern completions that is propagating through the rest of the brain until he moves his mouth and licks. When we were doing these experiments, we noticed in one lucky case that there was a mouse in which the go ensemble turned on spontaneously.

46:57 - Now the animal wasn’t looking at a screen, we weren’t - we didn’t have our piano on. This go ensemble, in this case, just lit up spontaneously. Every time that happened, the animal licked. This is endogenous activation of that ensemble. We don’t know why. But it correlates causally with the licking of the animal. That’s it in terms of what I wanted to tell you. In summary, I think we are finding emergent properties of brain circuits as predicted by theorists for 100 years, and that these cortical activities are organized in these groups of neurons, these ensembles or attractors that are spontaneous states. I didn’t show you, but individual neurons can belong to different ensembles, so they have a little bit of a combinatorial quality to them. The temporal spatial patterns are not identical, so they also have this flexibility, and they can do pattern completion, which I think would be critical to understand how the brain works if you think of it as a system of pattern completion. The visually evoked ensemble resembles spontaneous ones.

48:14 - The ensembles can be imprinted and recalled for several days. This is an example that a physicist would call a phased transition with generated phased transition in the circuit by activating a group of neurons together. These ensembles are necessary and sufficient for visual behavior, so they’re not an epiphenomenon of the circuit. They’re actually causally related to visual behavior. Our hypothesis in red here is that these ensembles are a unit of perception.

48:42 - Here I want to distinguish between what you’d call a sensation, which is the activation of your sensory system by let’s say visual stimulus or sensory stimulus, and your perception, which is your interpretation of that sensory stimulus. In this case, we can dissociate both things. We have experiments that I showed you where there’s activation of these internal states without any external stimulus. We can do that artificially or they happen sometimes spontaneously. That’s why we think that we’re dealing not with sensation but with perception. And then once you’re in perception, you’re essentially into memory. It’s very hard to distinguish when you see something between your perception or your memory of that object. When you’re looking at your grandmother, for example, are you looking at her or are you looking at your memory of her? I think this could be essentially right in the line of fire set out by people like Lorente, in a way Turing, and Hebb, and Hopfield, that these intrinsic internal units, the one we should concentrate on because they can enable us to take apart that edifice of the mind. And just to finish how this shows a very close link between the methods and the paradigms. The people that pioneered the single neuron methods, Cajal and Sherrington, are the same people that proposed the neuron doctrine in which you can have a single neuron method, everything is a single neuron.

50:14 - Now if we expand the palette of methods, like the Brain Initiative is doing, then you develop methods to watch the entire TV screen of the brain, then you can start seeing these emergent properties and start getting a new paradigm come in. I wanted to highlight that this was a lot of work done by mostly a lot of people. I would highlight Jae-Eun Miller and Luis Carrillo. They were the two who did all the critical work in the last decade in my lab on these ensembles. We have a relatively sizeable team. We’re funded by federal funding sources. And hopefully about to sign a contract on an IBM Columbia Data Science Institute contract for a conference on brain computer interfaces.

51:02 - I have a conflict of interest, a patent that has to do with this piano business, the holography stimulation using two-photon light. I wanted to finish with a quote from my mentor in England, Sydney Brenner, who died recently who argued that progress in science depends on new techniques, new discoveries, and new ideas, probably in that order. Thank you. .