!!Con West 2020 - Jeremy Apthorp: Curses!!

All righty. Hi. My name is Jeremy. Let me just start my presentation here. So I want to talk to you today about terminals. So if you’re a programmer, you’ve probably encountered the terminal before. Looks like this. It’s a prompt. You know, the computer asks you to type something in. And then maybe you type something, and the computer does something and prints a response back. So if you’ve written code before, you might have written something that looks like this.

00:58 - You invoke the name, the secret name of your program, and in this case, hello, in the computer – your program writes something out to the terminal. Hello world. Traditional incantation to the UNIX demons to bless your programming career. But like… Some terminal programs do some really wild stuff. They do colors, animations, move the cursor around, take over the screen. Some even have mouse support. So how the heck does that work? So this talk is an unreasonably detailed answer to that question. Or at least as unreasonable as I possibly could be within ten minutes. So to really start at the beginning of answering this question, we need to go back to about 1840. (laughter) And the beginnings of the electrical telegraph, one of the first uses of electrical engineering. So this is a telegraph key, something that a highly trained operator would use to transmit Morse code over a wire, and another trained operator on the other end would listen to the incoming sequence of, like, long and short dashes, and would transcribe it as text. They would write it down into something that somebody like a normal human could read. And it was slow. It was laborious.

02:18 - You needed trained operators on both ends. And so in the early 1900s, a group of engineers pitched to the Morton Salt Company of all things this idea they had of making telegraphs easier and faster to send. And so their idea was that they would hook up two electromechanical typewriters to each other, and you could just type on a keyboard, like a regular typewriter, and on the other end, those characters would appear instantly readable. And so they developed this machine, and that allowed the electrical telegraph to really take off. And it was especially – starting in about the ‘20s, it was really used a lot by the press, newspaper companies, and by financial companies. So that exploded.

03:11 - And that actually – this system kind of remained popular well into the ‘80s. So 50 years after the invention of the teletypewriter, this guy, Douglas T Ross, in 1956, was working on this computer called the Whirlwind at MIT. And this was like this marvel of engineering. It took 200 people two years to build this computer. It had 5,000 vacuum tubes. It ran at, like, a blistering 20 kilohertz and weighed 10 tons and crashed about every 20 minutes.

03:48 - But it was special in this one really important way, which is that it was one of the first computers that was ever designed for real time input. So instead of writing your commands to the computer on a stack of punch cards, putting it in the queue and sort of coming back and collecting the output later, the Whirlwind was designed for flight simulators, to control dials in a simulated cockpit, which is not what Doug Ross was doing with it, but that is what it was paid for to do. And so Doug had this idea that this computer was fast enough that you could sort of have a conversation with it. You could have this realtime back and forth between computer and human. So he hooked up a teletypewriter, a Freedom Flexawriter, to be specific, to the computer. It says PDP-11 on the top.

04:39 - They were used on PDPs as well, but he hooked it up to the Whirlwind. So he would write a message and the computer would write back instead of a human operator on the end. This worked really really well. People loved it. Within a few years, Teletypewriters were in every computer lab around the world. People would hook them up over telephone lines so the students and researchers could access the computer from their home. Super popular. CRTs did exist at this time, but they were not used for terminals, because memory was really, really expensive.

05:15 - That core memory that you saw earlier was the state of the art in RAM. Until about 1969, when RAM finally became cheap enough that you could reasonably build almost consumer grade hardware. This is the DataPoint 3300, one of the first video terminals. So at first, these video terminals just sort of emulated the paper and ink terminals. You would type into them, and instead of the response from the computer being printed out on paper, it was just displayed on the screen.

05:53 - So because VTs, video terminals, displayed characters on a screen, it meant they were a lot more flexible than a teletypewriter. You could move the cursor around, delete and insert characters, you could scroll through a long document without having to print the whole thing out. It would enable this whole new class of interactions that humans could use computers through. So Teletypewriters were still popular, though. To stay compatible with the Teletypewriters, video terminals used the same protocol, which was more or less ASCII.

06:32 - The computer would send ASCII characters and they would be displayed on screen. But ASCII is only 128 characters, and there are… So 32 and up are all printable characters. And these are the control characters that were available to the computer for things that are not printing characters. There are 33 of them. The DataPoint 3300, one of the first video terminals, already used 18 of these to do things. And the manufacturers wanted to do more than 33 different things. So they used this escape hatch. The hexcode 1b in ASCII is “escape”. When the terminal wanted to do something that was not print a character, it would send this escape code, and then followed by a sequence of instructions that would sort of describe what to do next. So in this case, it sends escape and then [1d, and on my terminal at least, this means move the cursor one step to the left. So there was like… One standard that all of the terminal manufacturers got together and agreed on a series of escape codes, and what each one would do. No, no. That’s not at all what happened. (laughter) There are hundreds of different companies manufacturing terminals, and hundreds of different models and they all had different capabilities and every one of them wanted to differentiate their terminal so they would invent something their terminal could do that no one else’s could do and they would choose whatever escape code they thought made sense for their terminal.

08:11 - So in 1978, this guy, Bill Joy, was writing a text editor, and he wanted it to work on lots of different terminals. No matter what lab you were in, what computer you’re using, what terminal you were using to access it, he wanted that text editor to work on that terminal. So he started out by hard coding tests for, like, are you using a DeckWriter? Are you using a DataPoint? And he would program in the appropriate escape codes to use for that terminal. But then he got sick of that and he invented terminfo. So terminfo is this database of escape codes, sequences, and some other information about a whole bunch of different terminals, and it’s on all of your computers right now in this directory. You can go check it out.

08:55 - It is actually complicated enough to describe all these different ways of handling different kinds of terminal escape codes that it has its own programming language with conditionals and variables and arithmetic and stuff. And compatibility bugs. And there’s a tool called infocomp that you can run. At least on Macs. I think this also comes on most Linux distributions installed by default. And by default, it’ll show you the information for the terminal that you’re currently running. You can also ask it for a different terminal.

09:27 - So this one – you can actually see the terminal I’m using is called iTerm 2, but it has given me the information for xterm 256 color, so my terminal is pretending to be an xterm even though it isn’t. And here you can see clear screen is ‘escape [H escape [2J’. So if I send that sequence of characters, my terminal should clear the screen. At least, that’s what the database says I should send if I want to clear the screen. So the terminfo database on my screen has 2500 entries in it, including for the DataPoint 3300, which is super cool.

10:03 - My 2019 Macbook Pro knows how to control a 1965 video terminal that I don’t know if any of them even exist anymore. But in case somebody hooks it up to my computer, don’t worry. It knows how to drive it! Also, your computer comes with a manual on this format and a bunch of the other tools. So if you just type ‘man terminfo’, you’ll get this really quite good description of this database and how it’s arranged and how to read it. So you now know everything about ncurses. No, there’s a ton more that’s not stored in terminfo. But ncurses is Open Source. You can just go and read the code.

10:45 - Mouseinfo is something that’s not in terminfo. I think that’s super cool. All this information is available and it’s not beyond your grasp to know. So thanks for listening. You can follow me on Mastodon, and source code for this talk, which is all custom, is JavaScript pretending to be ncurses. So it reads the terminfo database and everything. You can find the code for that there. And go read some code! Nothing is beyond your grasp. Thank you! .