!!Con West 2020 - Matías Lang: Delete all code! 100% testing coverage, the lazy way!

Okay. Hi, everyone! So my talk idea started one day at my company, while reviewing year end bonuses. This is a monetary compensation. Based on the accomplishment of certain objectives the company had. So as any good employee, I tried to find a way to automate this thing. Oh, not working. Okay. To automate this thing. Let’s find a way to minimize the work I had to do. But maximize the profit I could get from it. Obviously. So…

01:10 - One of the objectives we had was to get 100% testing coverage in our product. And if we think for a while, it seems that is an automatable metric. So I had this revelation. That testing code is measured based on how many lines of your code are being executed when you run your test suite. So if we just delete every line that it’s uncovered, that means that because the lines weren’t tested, or executed, my tests will pass anyway, and because the lines don’t exist anymore, we’ll get 100% coverage for free. So… Yesterday in Nick’s talk, we saw how to write programs that write other programs.

02:02 - And today, I’ll be talking about how to write programs that delete other programs, basically. So… To give an idea of what I expect my program to do, let’s take this simple example. This is a simple if-else Python code. We’ll be using Python. And we see here that the else branch of my code is not being executed in my test. It’s not called. So what I would expect from this tool is, given the source code of the program, and the coverage information – to delete the else, the oops(), and this is not tested. No evidence of it. For example, we have a function that – it looks like it can be tested, but it has some unreachable goal.

02:55 - In this simple goal, we can just remove the last two lines. We will have 100% testing coverage. So to do this kind of tool that does source code transformation, I used the best tool for this. Regular expressions. Just kidding. Please don’t use regular expressions to do this. What I used instead was Abstract Syntax Trees, so ASTs. It’s basically a tree-based data structure designed to represent your source code.

03:34 - In this case, a simple hello world will be translated into a function call, where the child – for one side, the function name, and for the other the arguments to the print function. That in this case is a binary operation where children are the hello and the world string. So simple enough. This AST. So just using the Python standard library, it is possible to, when I have this tree, it’s a syntax tree, essentially, so this means I can work it and transform it just as we’ll do with any kind of tree. This simple code that you see here is trying to get the – pass this print hello world into an AST. Then it will visit each node in the syntax tree, and when it sees some bin operation, it will hard code the right part of it to the !!Con string.

04:41 - So in this case, the print hello world thing will be translated into this hello !!Con thing. One thing I want to mention is that the Python standard library has an AST module, but it doesn’t allow me to transform, to convert this transformed AST back into source code again. So for this, I had to use this Google library called Pasta. I wanted to mention it, because the maintainers have been extremely supportive. I had to report many issues, because what I was doing was apparently kind of strange.

05:17 - So I hit a lot of corner cases related to indentation. But the maintainers helped me a lot, and fixed all the bugs almost instantly. So this is essentially what I had in my code. I wrote a program that does exactly that. It’s GPL licensed. And it’s not so tiny, as I will expect before. I thought it was going to take me a few hours. And it turned out to be way harder. Because what I had to do wasn’t just removing the else of an if or removing the original code. But I had to deal with more complex AST transformations. For example, I had to deal with side effects in the if test expressions. Or I had to deal with the Python yield statement -- because it forms functions into generator functions, even when it’s not tested, even with the yield line, it’s not executed. It will impact on your source code. So I cannot trust it. One thing I also found that was a bug in the Python coverage library that incorrectly reported my call as being uncovered, because when I deleted the line, my code broke, because it was actually called.

07:10 - Apparently this was related to a CPython optimizer machine bug. I have no idea what it did. But… Eventually I fixed these issues, and we got this project. So I think it’s time for a demo, how this works. Here I have… Can you see okay? Here I have a simple Python project I chose just because it has a fast test suite. So I will run the test suite to see. Yay. All tests are passing. Just as we would expect. But if we… Whoa. If we see the report, it has only 46% coverage. That’s not so good. My boss wouldn’t be so happy. But we have magicov to the rescue. Oops. I think I didn’t… Okay. I… Yeah.

08:22 - I want to avoid any misunderstanding of the tool. So I shall run this. Oh, no! Magicov, what did you just do with my code? You messed it up! Yeah. A lot of the deleted lines… No! This can’t be good! But if we run the test suite again… Let’s wait a second. Oh, all tests are passing. That’s good. And if we see the report… Whoa! So yeah. I have time for one more demo. So if we think this tool, magicov, was designed using test driven development, so it has a pretty good test suite. If we run it… It says all tests are passing. And 86% coverage. It’s good enough. But not good enough for 100% testing coverage! So let’s run the tool. Yeah. Here we have… It did some code deletions here. And if we see… Oh, I forgot the tool. No. Oh, I didn’t run the test. Sorry. I had to run this again. I see the report and…

09:55 - Yeah! So thanks! We just Magicov’ed Magicov. So as I hope you got by now, this tool wasn’t really designed to get a profit. It was just an excuse to make a tool that…it’s useless It has no purpose. But it was really fun to do that. So what I want you to take from this talk is that the world is full of rational, useful, pragmatic software. But that doesn’t mean you have to write useful software! This was really fun.

10:54 - I learned a lot of AST manipulation, source code transforming. Even if it’s useless, I learned a lot, and it was really fun doing it. So I want you encourage you to write useless software. Thank you very much! .