In early October I gave a keynote at Python Brasil in Belo Horizonte. Here is an aspirational and lightly edited transcript of the talk. There is also a video available here.

I love bugs

I’m currently a senior engineer at Pilot.com, working on automating bookkeeping for startups. Before that, I worked for Dropbox on the desktop client team, and I’ll have a few stories about my work there. Earlier, I was a facilitator at the Recurse Center, a writers retreat for programmers in NYC. I studied astrophysics in college and worked in finance for a few years before becoming an engineer.

But none of that is really important to remember – the only thing you need to know about me is that I love bugs. I love bugs because they’re entertaining. They’re dramatic. The investigation of a great bug can be full of twists and turns. A great bug is like a good joke or a riddle – you’re expecting one outcome, but the result veers off in another direction.

Over the course of this talk I’m going to tell you about some bugs that I have loved, explain why I love bugs so much, and then convince you that you should love bugs too.

Bug #1

Ok, straight into bug #1. This is a bug that I encountered while working at Dropbox. As you may know, Dropbox is a utility that syncs your files from one computer to the cloud and to your other computers.

1
2
3
4
5
6
7
8
9
10
11
12

        +--------------+     +---------------+
        |              |     |               |
        |  METASERVER  |     |  BLOCKSERVER  |
        |              |     |               |
        +-+--+---------+     +---------+-----+
          ^  |                         ^
          |  |                         |
          |  |     +----------+        |
          |  +---> |          |        |
          |        |  CLIENT  +--------+
          +--------+          |
                   +----------+

Here’s a vastly simplified diagram of Dropbox’s architecture. The desktop client runs on your local computer listening for changes in the file system. When it notices a changed file, it reads the file, then hashes the contents in 4MB blocks. These blocks are stored in the backend in a giant key-value store that we call blockserver. The key is the digest of the hashed contents, and the values are the contents themselves.

Of course, we want to avoid uploading the same block multiple times. You can imagine that if you’re writing a document, you’re probably mostly changing the end – we don’t want to upload the beginning over and over. So before uploading a block to the blockserver the client talks to a different server that’s responsible for managing metadata and permissions, among other things. The client asks metaserver whether it needs the block or has seen it before. The “metaserver” responds with whether or not each block needs to be uploaded.

So the request and response look roughly like this: The client says, “I have a changed file made up of blocks with hashes 'abcd,deef,efgh'”. The server responds, “I have those first two, but upload the third.” Then the client sends the block up to the blockserver.

1
2
3
4
5
6
7
8
9
10
11
12

                +--------------+     +---------------+
                |              |     |               |
                |  METASERVER  |     |  BLOCKSERVER  |
                |              |     |               |
                +-+--+---------+     +---------+-----+
                  ^  |                         ^
                  |  | 'ok, ok, need'          |
'abcd,deef,efgh'  |  |     +----------+        | efgh: [contents]
                  |  +---> |          |        |
                  |        |  CLIENT  +--------+
                  +--------+          |
                           +----------+

That’s the setup. So here’s the bug.

1
2
3
4
5
6
7
8
9
10
11
12

                +--------------+
                |              |
                |  METASERVER  |
                |              |
                +-+--+---------+
                  ^  |
                  |  |   '???'
'abcdldeef,efgh'  |  |     +----------+
     ^            |  +---> |          |
     ^            |        |  CLIENT  +
                  +--------+          |
                           +----------+

Sometimes the client would make a weird request: each hash value should have been sixteen characters long, but instead it was thirty-three characters long – twice as many plus one. The server wouldn’t know what to do with this and would throw an exception. We’d see this exception get reported, and we’d go look at the log files from the desktop client, and really weird stuff would be going on – the client’s local database had gotten corrupted, or python would be throwing MemoryErrors, and none of it would make sense.

If you’ve never seen this problem before, it’s totally mystifying. But once you’d seen it once, you can recognize it every time thereafter. Here’s a hint: the middle character of each 33-character string that we’d often see instead of a comma was l. These are the other characters we’d see in the middle position:

1

l \x0c < $ ( . -

The ordinal value for an ascii comma – , – is 44. The ordinal value for l is 108. In binary, here’s how those two are represented:

1
2

bin(ord(',')): 0101100  
bin(ord('l')): 1101100  

You’ll notice that an l is exactly one bit away from a comma. And herein lies your problem: a bitflip. One bit of memory that the desktop client is using has gotten corrupted, and now the desktop client is sending a request to the server that is garbage.

And here are the other characters we’d frequently see instead of the comma when a different bit had been flipped.

1
2
3
4
5
6
7
8

,    : 0101100
l    : 1101100
\x0c : 0001100
<    : 0111100
$    : 0100100
(    : 0101000
.    : 0101110
-    : 0101101

Bitflips are real!

I love this bug because it shows that bitflips are a real thing that can happen, not just a theoretical concern. In fact, there are some domains where they’re more common than others. One such domain is if you’re getting requests from users with low-end or old hardware, which is true for a lot of laptops running Dropbox. Another domain with lots of bitflips is outer space – there’s no atmosphere in space to protect your memory from energetic particles and radiation, so bitflips are pretty common.

You probably really care about correctness in space – your code might be keeping astronauts alive on the ISS, for example, but even if it’s not mission-critical, it’s hard to do software updates to space. If you really need your application to defend against bitflips, there are a variety of hardware & software approaches you can take, and there’s a very interesting talk by Katie Betchold about this.

Dropbox in this context doesn’t really need to protect against bitflips. The machine that is corrupting memory is a user’s machine, so we can detect if the bitflip happens to fall in the comma – but if it’s in a different character we don’t necessarily know it, and if the bitflip is in the actual file data read off of disk, then we have no idea. There’s a pretty limited set of places where we could address this, and instead we decide to basically silence the exception and move on. Often this kind of bug resolves after the client restarts.

Unlikely bugs aren’t impossible

This is one of my favorite bugs for a couple of reasons. The first is that it’s a reminder of the difference between unlikely and impossible. At sufficient scale, unlikely events start to happen at a noticable rate.

Social bugs

My second favorite thing about this bug is that it’s a tremendously social one. This bug can crop up anywhere that the desktop client talks to the server, which is a lot of different endpoints and components in the system. This meant that a lot of different engineers at Dropbox would see versions of the bug. The first time you see it, you can really scratch your head, but after that it’s easy to diagnose, and the investigation is really quick: you look at the middle character and see if it’s an l.

Cultural differences

One interesting side-effect of this bug was that it exposed a cultural difference between the server and client teams. Occasionally this bug would be spotted by a member of the server team and investigated from there. If one of your servers is flipping bits, that’s probably not random chance – it’s probably memory corruption, and you need to find the affected machine and get it out of the pool as fast as possible or you risk corrupting a lot of user data. That’s an incident, and you need to respond quickly. But if the user’s machine is corrupting data, there’s not a lot you can do.

Share your bugs

So if you’re investigating a confusing bug, especially one in a big system, don’t forget to talk to people about it. Maybe your colleagues have seen a bug shaped like this one before. If they have, you might save a lot of time. And if they haven’t, don’t forget to tell people about the solution once you’ve figured it out – write it up or tell the story in your team meeting. Then the next time your teams hits something similar, you’ll all be more prepared.

How bugs can help you learn

Recurse Center

Before I joined Dropbox, I worked for the Recurse Center. The idea behind RC is that it’s a community of self-directed learners spending time together getting better as programmers. That is the full extent of the structure of RC: there’s no curriculum or assignments or deadlines. The only scoping is a shared goal of getting better as a programmer. We’d see people come to participate in the program who had gotten CS degrees but didn’t feel like they had a solid handle on practical programming, or people who had been writing Java for ten years and wanted to learn Clojure or Haskell, and many other profiles as well.

My job there was as a facilitator, helping people make the most of the lack of structure and providing guidance based on what we’d learned from earlier participants. So my colleagues and I were very interested in the best techniques for learning for self-motivated adults.

Deliberate Practice

There’s a lot of different research in this space, and one of the ones I think is most interesting is the idea of deliberate practice. Deliberate practice is an attempt to explain the difference in performance between experts & amateurs. And the guiding principle here is that if you look just at innate characteristics – genetic or otherwise – they don’t go very far towards explaining the difference in performance. So the researchers, originally Ericsson, Krampe, and Tesch-Romer, set out to discover what did explain the difference. And what they settled on was time spent in deliberate practice.

Deliberate practice is pretty narrow in their definition: it’s not work for pay, and it’s not playing for fun. You have to be operating on the edge of your ability, doing a project appropriate for your skill level (not so easy that you don’t learn anything and not so hard that you don’t make any progress). You also have to get immediate feedback on whether or not you’ve done the thing correctly.

This is really exciting, because it’s a framework for how to build expertise. But the challenge is that as programmers this is really hard advice to apply. It’s hard to know whether you’re operating at the edge of your ability. Immediate corrective feedback is very rare – in some cases you’re lucky to get feedback ever, and in other cases maybe it takes months. You can get quick feedback on small things in the REPL and so on, but if you’re making a design decision or picking a technology, you’re not going to get feedback on those things for quite a long time.

But one category of programming where deliberate practice is a useful model is debugging. If you wrote code, then you had a mental model of how it worked when you wrote it. But your code has a bug, so your mental model isn’t quite right. By definition you’re on the boundary of your understanding – so, great! You’re about to learn something new. And if you can reproduce the bug, that’s a rare case where you can get immediate feedback on whether or not your fix is correct.

A bug like this might teach you something small about your program, or you might learn something larger about the system your code is running in. Now I’ve got a story for you about a bug like that.

Bug #2

This bug also one that I encountered at Dropbox. At the time, I was investigating why some desktop client weren’t sending logs as consistently as we expected. I’d started digging into the client logging system and discovered a bunch of interesting bugs. I’ll tell you only the subset of those bugs that is relevant to this story.

Again here’s a very simplified architecture of the system.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

                                   +--------------+
                                   |              |
               +---+  +----------> |  LOG SERVER  |
               |log|  |            |              |
               +---+  |            +------+-------+
                      |                   |
                +-----+----+              |  200 ok
                |          |              |
                |  CLIENT  |  <-----------+
                |          |
                +-----+----+
                      ^
                      +--------+--------+--------+
                      |        ^        ^        |
                   +--+--+  +--+--+  +--+--+  +--+--+
                   | log |  | log |  | log |  | log |
                   |     |  |     |  |     |  |     |
                   |     |  |     |  |     |  |     |
                   +-----+  +-----+  +-----+  +-----+

The desktop client would generate logs. Those logs were compress, encrypted, and written to disk. Then every so often the client would send them up to the server. The client would read a log off of disk and send it to the log server. The server would decrypt it and store it, then respond with a 200.

If the client couldn’t reach the log server, it wouldn’t let the log directory grow unbounded. After a certain point it would start deleting logs to keep the directory under a maximum size.

The first two bugs were not a big deal on their own. The first one was that the desktop client sent logs up to the server starting with the oldest one instead of starting with the newest. This isn’t really what you want – for example, the server would tell the client to send logs if the client reported an exception, so probably you care about the logs that just happened and not the oldest logs that happen to be on disk.

The second bug was similar to the first: if the log directory hit its maximum size, the client would delete the logs starting with the newest instead of starting with the oldest. Again, you lose log files either way, but you probably care less about the older ones.

The third bug had to do with the encryption. Sometimes, the server would be unable to decrypt a log file. (We generally didn’t figure out why – maybe it was a bitflip.) We weren’t handling this error correctly on the backend, so the server would reply with a 500. The client would behave reasonably in the face of a 500: it would assume that the server was down. So it would stop sending log files and not try to send up any of the others.

Returning a 500 on a corrupted log file is clearly not the right behavior. You could consider returning a 400, since it’s a problem with the client request. But the client also can’t fix the problem – if the log file can’t be decrypted now, we’ll never be able to decrypt it in the future. What you really want the client to do is just delete the log and move on. In fact, that’s the default behavior when the client gets a 200 back from the server for a log file that was successfully stored. So we said, ok – if the log file can’t be decrypted, just return a 200.

All of these bugs were straightforward to fix. The first two bugs were on the client, so we’d fixed them on the alpha build but they hadn’t gone out to the majority of clients. The third bug we fixed on the server and deployed.

📈

Suddenly traffic to the log cluster spikes. The serving team reaches out to us to ask if we know what’s going on. It takes me a minute to put all the pieces together.

Before these fixes, there were four things going on:

1. Log files were sent up starting with the oldest
2. Log files were deleted starting with the newest
3. If the server couldn’t decrypt a log file it would 500
4. If the client got a 500 it would stop sending logs

A client with a corrupted log file would try to send it, the server would 500, the client would give up sending logs. On its next run, it would try to send the same file again, fail again, and give up again. Eventually the log directory would get full, at which point the client would start deleting its newest files, leaving the corrupted one on disk.

The upshot of these three bugs: if a client ever had a corrupted log file, we would never see logs from that client again.

The problem is that there were a lot more clients in this state than we thought. Any client with a single corrupted file had been dammed up from sending logs to the server. Now that dam was cleared, and all of them were sending up the rest of the contents of their log directories.

Our options

Ok, there’s a huge flood of traffic coming from machines around the world. What can we do? (This is a fun thing about working at a company with Dropbox’s scale, and particularly Dropbox’s scale of desktop clients: you can trigger a self-DDOS very easily.)

The first option when you do a deploy and things start going sideways is to rollback. Totally reasonable choice, but in this case, it wouldn’t have helped us. The state that we’d transformed wasn’t the state on the server but the state on the client – we’d deleted those files. Rolling back the server would prevent additional clients from entering this state but it wouldn’t solve the problem.

What about increasing the size of the logging cluster? We did that – and started getting even more requests, now that we’d increased our capacity. We increased it again, but you can’t do that forever. Why not? This cluster isn’t isolated. It’s making requests into another cluster, in this case to handle exceptions. If you have a DDOS pointed at one cluster, and you keep scaling that cluster, you’re going to knock over its depedencies too, and now you have two problems.

Another option we considered was shedding load – you don’t need every single log file, so can we just drop requests. One of the challenges here was that we didn’t have an easy way to tell good traffic from bad. We couldn’t quickly differentiate which log files were old and which were new.

The solution we hit on is one that’s been used at Dropbox on a number of different occassions: we have a custom header, chillout, which every client in the world respects. If the client gets a response with this header, then it doesn’t make any requests for the provided number of seconds. Someone very wise added this to the Dropbox client very early on, and it’s come in handy more than once over the years. The logging server didn’t have the ability to set that header, but that’s an easy problem to solve. So two of my colleagues, Isaac Goldberg and John Lai, implemented support for it. We set the logging cluster chillout to two minutes initially and then managed it down as the deluge subsided over the next couple of days.

Know your system

The first lesson from this bug is to know your system. I had a good mental model of the interaction between the client and the server, but I wasn’t thinking about what would happen when the server was interacting with all the clients at once. There was a level of complexity that I hadn’t thought all the way through.

Know your tools

The second lesson is to know your tools. If things go sideways, what options do you have? Can you reverse your migration? How will you know if things are going sideways and how can you discover more? All of those things are great to know before a crisis – but if you don’t, you’ll learn them during a crisis and then never forget.

Feature flags & server-side gating

The third lesson is for you if you’re writing a mobile or a desktop application: You need server-side feature gating and server-side flags. When you discover a problem and you don’t have server-side controls, the resolution might take days or weeks as you push out a new release or submit a new version to the app store. That’s a bad situation to be in. The Dropbox desktop client isn’t going through an app store review process, but just pushing out a build to tens of millions of clients takes time. Compare that to hitting a problem in your feature and flipping a switch on the server: ten minutes later your problem is resolved.

This strategy is not without its costs. Having a bunch of feature flags in your code adds to the complexity dramatically. You get a combinatoric problem with your testing: what if feature A is enabled and feature B, or just one, or neither – multiplied across N features. It’s extremely difficult to get engineers to clean up their feature flags after the fact (and I was also guilty of this). Then for the desktop client there’s multiple versions in the wild at the same time, so it gets pretty hard to reason about.

But the benefit – man, when you need it, you really need it.

How to love bugs

I’ve talked about some bugs that I love and I’ve talked about why to love bugs. Now I want to tell you how to love bugs. If you don’t love bugs yet, I know of exactly one way to learn, and that’s to have a growth mindset.

The sociologist Carol Dweck has done a ton of interesting research about how people think about intelligence. She’s found that there are two different frameworks for thinking about intelligence. The first, which she calls the fixed mindset, holds that intelligence is a fixed trait, and people can’t change how much of it they have. The other mindset is a growth mindset. Under a growth mindset, people believe that intelligence is malleable and can increase with effort.

Dweck found that a person’s theory of intelligence – whether they hold a fixed or growth mindset – can significantly influence the way they select tasks to work on, the way they respond to challenges, their cognitive performance, and even their honesty.

[I also talked about a growth mindset in my Kiwi PyCon keynote, so here are just a few excerpts. You can read the full transcript here.]

Findings about honesty:

After this, they had the students write letters to pen pals about the study, saying “We did this study at school, and here’s the score that I got.” They found that almost half of the students praised for intelligence lied about their scores, and almost no one who was praised for working hard was dishonest.

On effort:

Several studies found that people with a fixed mindset can be reluctant to really exert effort, because they believe it means they’re not good at the thing they’re working hard on. Dweck notes, “It would be hard to maintain confidence in your ability if every time a task requires effort, your intelligence is called into question.”

On responding to confusion:

They found that students with a growth mindset mastered the material about 70% of the time, regardless of whether there was a confusing passage in it. Among students with a fixed mindset, if they read the booklet without the confusing passage, again about 70% of them mastered the material. But the fixed-mindset students who encountered the confusing passage saw their mastery drop to 30%. Students with a fixed mindset were pretty bad at recovering from being confused.

These findings show that a growth mindset is critical while debugging. We have to recover from confusion, be candid about the limitations of our understanding, and at times really struggle on the way to finding solutions – all of which is easier and less painful with a growth mindset.

Love your bugs

I learned to love bugs by explicitly celebrating challenges while working at the Recurse Center. A participant would sit down next to me and say, “[sigh] I think I’ve got a weird Python bug,” and I’d say, “Awesome, I love weird Python bugs!” First of all, this is definitely true, but more importantly, it emphasized to the participant that finding something where they struggled an accomplishment, and it was a good thing for them to have done that day.

As I mentioned, at the Recurse Center there are no deadlines and no assignments, so this attitude is pretty much free. I’d say, “You get to spend a day chasing down this weird bug in Flask, how exciting!” At Dropbox and later at Pilot, where we have a product to ship, deadlines, and users, I’m not always uniformly delighted about spending a day on a weird bug. So I’m sympathetic to the reality of the world where there are deadlines. However, if I have a bug to fix, I have to fix it, and being grumbly about the existence of the bug isn’t going to help me fix it faster. I think that even in a world where deadlines loom, you can still apply this attitude.

If you love your bugs, you can have more fun while you’re working on a tough problem. You can be less worried and more focused, and end up learning more from them. Finally, you can share a bug with your friends and colleagues, which helps you and your teammates.

Obrigada!

My thanks to folks who gave me feedback on this talk and otherwise contributed to my being there:

• Sasha Laundy
• Amy Hanlon
• Julia Evans
• Julian Cooper
• Raphael Passini Diniz and the rest of the Python Brasil organizing team

I often hear junior engineers or newcomers to a company say that they’re not getting enough feedback. It’s a common feeling, especially for people who are just leaving school and the predictable assignment->work->grade feedback loop that it creates.

There’s lots of good advice for managers, mentors, and senior engineers about giving specific, actionable feedback – but it’s important to know what kind of feedback someone needs. I’ve found that people asking for more feedback are generally looking for one of two very different things:

The first kind of feedback is strategic feedback. The engineer asking for strategic feedback means something like this: “I feel like my work is going okay, and I’m wondering if I can be more effective. Are there strategies I can change to be even better?” The person asking these questions probably feels open, secure, and calm. They’re eager to grow and want to know if there’s anything they’re missing. Ideally, this engineer is asking specific questions for the feedback they’re seeking, like “Did my architecture doc clearly explain our project? Was my last pull request the right size and scope? What are the most important problems that our team is facing?” Feedback for this person should certainly be specific and actionable.

The other kind of feedback is less often dicussed – belonging feedback. An engineer seeking belonging feedback might be asking “Do you have any feedback for me?” but means something like, “Are things going ok? Do people like me? Am I making dumb mistakes?” In this state, they probably feel vulnerable. They might not yet feel comfortable with their coworkers. They might even be worried they’re going to get fired.

As a feedback-seeker (whether you’re a new engineer or an experienced hand), the more clear you can be about what you’re looking for, the more likely you are to get it. I once sat down for a one-on-one with my manager to discuss an incident, and said, “My agenda for this meeting is how it happened, what our plan for remediation is, and my feelings.” For my manager and me, this worked great. Being clear about what you want also helps you determine whom to ask for it – different problems might go to your direct manager, a more senior engineer, someone who joined the company at the same time you did, or a friend outside of work.

As a manager, mentor, or senior colleague, the best thing you can do is understand what kind of feedback is being requested. You don’t want to tell someone seeking strategic feedback, “Don’t worry! Everything’s great!” Similarly, you don’t want to give a list of ten areas for improvement to someone who’s already being too hard on themselves. What kind of feedback is being sought isn’t always obvious, and probably requires some follow-up questions.

Differentiating between these two types of request allows everyone to have a more pleasant experience and get better feedback at the same time.

My 2015 was a year of recovering from a serious injury and becoming a better engineer. I also gave four talks and published a chapter I’ve been working on for a while. I’m hopeful that 2016 will bring as many opportunities for growth and fewer broken bones.

TPF

I broke my knee – a tibial plateau fracture – at the beginning of February, 2015. It required surgery and several months on crutches. I absolutely do not recommend this.

Recovering from an injury like this requires a lot of determination and a lot of help. I’m grateful to have been able to temporarily use all of the lazy-techie apps (groceries, laundry, ridesharing, etc. etc.). But there aren’t apps for everything, and I’m incredibly thankful for the many friends and family who helped with this process, from bringing me crutches when I first broke the knee, coming over with dinner, or keeping me company and taking out the trash on the way out.

It is really hard to be on crutches. Your triceps burn constantly. You can’t carry anything in your hands. I was very worried about falling. I live in a third-floor walkup, so the last thing I did every day was climb two flights of stairs. (How? One hand on the railing, one hand on the crutches, then jump.)

Body battles

Oddly, I felt more comfortable in some ways while on crutches than I do while healthy. On crutches, it was obvious what the current focus was at any given point: learn to use crutches; avoid falling down; manage medication; go to physical therapy. When healthy, I constantly think that I’m about to start working out harder than I currently do. (I think I’m not the only one who does this.) It was in some ways easier to think, “Nothing about my fitness routine is going to change for at least twelve weeks. All I need to do is work on my knee.”

Persistent questions

When I first started using the crutches, I was flabbergasted by some people who would ask about my injury very persistently. (“What happened? What kind of fracture? When’s your surgery? Where’d you have it? Do you have hardware?”) Then I realized that everyone with persistent questions had had knee surgery themselves. After realizing that, those people became very easy to deal with: I’d just ask them about their own surgeries and sit back while they told me all about it. Better yet, almost everyone I talked to had recovered fully and was now doing great, some number of years later.

If you’re on crutches, especially if you’re wearing the distinctive knee brace, the best piece of advice I can give you is this fact. Those with persistent questions have had their own surgeries. They would love to tell you about it.

In case you’re wondering – now that I’m recovered, I most certainly am one of those people. However, I always lead with my own surgery before asking any questions. :)

If you’re on crutches, I also recommend this marvel of human engineering and velcro, this detachable shower head, a travel coffee mug, a stylish backpack, and as much stubbornness as you can muster.

Talks

Notwithstanding the broken leg, I gave four conference talks in 2015. Two were at PyCon North America in Montreal while on crutches. One was at !!con in NYC, near the very end of the crutches era, and the fourth was at Kiwi PyCon in Christchurch, New Zealand, where I was fully ambulatory.

Exploring is Never Boring: Understanding CPython without reading the code & Bytes in the Machine: Inside the CPython interpreter (PyCon North America)

I gave these two talks at PyCon in Montreal back-to-back. “I think this’ll actually be easier for you,” said the organizer, and that turned out to be true, but not for reasons either of us predicted. As it turned out, the hardest part of PyCon 2015 for me was getting around on crutches, so the less of that I had to do the better off I was.

I was pleased with how these talks went, especially “Bytes in the Machine”, which I’d been working on in one form or another for more than a year. I originally proposed this talk for PyCon 2014 and it was rejected. I was able to propose and then give a substantially better version of it at PyCon 2015. One person told me that they’d never wanted to dig into CPython before and now they did, which was exactly what I was hoping for.

PyCon 2015 on crutches took an enormous amount of energy. The organizers were all very kind and helpful, but the convention center in Montreal was simply very large, and a ton of moving was required. My thanks to the organizers for their accomodations, and to the friends and strangers I pressed into carrying my lunches. (I also offer my apologies to anyone near enough to smell me on Friday, before my wayward luggage arrived. Crutching around is regrettably strenuous.)

This photo from Anja Boskovic shows me in the same body position I was in for almost three months: sitting on a chair with one leg up. I find this position to be quite masculine – asymmetric and unapologetically taking up space. There aren’t a lot of perks to breaking a leg, but I enjoyed taking up a lot of space while having a perfect excuse for doing so. There’s something delightful about having your foot up on the table during a meeting with someone who outranks you, or while presenting a talk at a conference. Interestingly, not everyone who attended the talk realized that I was using crutches during the conference, which means they can’t have properly attributed my body language to my injury.[^1]

Video and slides for Exploring is Never Boring Video and slides for Bytes in the Machine

If you’re catching up now and you prefer written material, consider reading the chapter version of this talk instead of watching the video.

Limitless and Recursion-free recursions limits! (!!con)

!!con (“bangbangcon”) is a conference about “the joy, excitement, and surprise in programming.” This was my second year speaking there, and it’s consistently one of my favorite conferences. I described the CfP as “an invitation to meditate on your delight,” and the whole conference felt like that. Things can be challenging and difficult and outright terrible in this industry, and there’s a lot of hard work to do, but it’s nice to spend a couple of days learning about the amazing, fascinating, and weird world we live in.

This year there were talks on how wifi keeps getting faster, how to program a knitting machine, lightpainting with robots, making a cell phone, quines, and roller derby. It was a truly delightful lineup, and I’m honored to have been a part of it.

My talk covered how to hit the recursion limit in Python without doing any recursion and how to implement the world’s jankiest tail call optimization in Python. This talk features the following: – Me saying “Any day we can segfault Python is a good day in my book.” – Me saying “Remember, our beef today is with the compiler.” – An audible “Oh no” from the audience on seeing a slide with Python code and GOTOs

Unfortunately, the sound quality’s not great on this video.

[sketch by Danielle]

Video and slides for Limitless and Recursion-free recursion limits!

Learning Strategies for Programmers (Kiwi PyCon keynote)

My final talk this year was a keynote at the Kiwi PyCon conference in Christchurch, New Zealand. I loved this trip. The organizers were hospitable from start to finish. Marek Kuziel was kind enough to meet me at the airport (and kind enough to depart before I attempted to drive my rental car on the left side of the road). I also want to give Marek credit for effective enforcement of a Code of Conduct. On one occassion in particular he gently redirected some mildly-dirty humor before anyone got uncomfortable. This can be tricky to do and he did it well.

I wrote a blog post in October that captures the best parts of this talk. There is also video available and slides.

Architecture of Open Source Applications chapter

After many rounds of writing and procrastinating, I finished and published my chapter for the Architecture of Open Source Applications 4th edition, on Byterun, a Python interpreter written in Python with Ned Batchelder. This version of the AOSA book is themed “500 lines or less,” and it features real software that does something significant in under 500 lines. It was a fun challenge to trim Byterun down to that size, and an even better challenge to try to explain the resulting code clearly.

My thanks to the editors enough for their patience and grit in this process, especially Mike DiBernardo and the talented copy editor Amy Brown.

A Python Interpreter written in Python

Dropbox

I’ve now been at Dropbox for slightly over a year. Most of what I’m most excited about I can’t talk about publically. What I can say is that I feel like I’ve matured as an engineer This means things like getting better at skills like living with my decisions, thinking farther ahead, architecting software, gathering consensus, getting and giving technical input, and other skills beyond pure programming.

I’m on the desktop client team, and desktop software in particular presents interesting challenges that I hadn’t thought much about before I joined Dropbox. For example, you generally can’t roll back a desktop release – once it’s out there, it’s out there. It’s also nontrivial to make sure we can get enough data to debug when something goes wrong. With a badly-behaving server, you might be able to ssh in and poke around. This is obviously not possible with someone else’s desktop.

[1] One person even congratulated Jessica McKellar for my talk, thinking she was me. I was obviously thrilled to be mistaken for her.

In early September I gave a keynote at Kiwi PyCon in New Zealand on effective learning for programmers. There were two pieces to the talk: one about mindset, and one about particular strategies we can use. The text below is an aspirational and lightly edited transcript of the mindset piece of that talk. There’s also a video available if you’d like to see the strategies piece.

Recurse Center

Before I joined Dropbox last year, I spent two years working at a company in NYC called the Recurse Center. The Recurse Center is like a writers’ retreat for programmers. Participants spend 3 months working on whatever is most interesting to them. So someone who’d been writing Java for ten years might come to RC to learn a new language like Clojure, or someone who just graduated with a CS degree might come work on their web development skills, or someone who’d been learning programming in their spare time might come to turbo-charge their learning. There’s almost no structure to the program – no deadlines, no assignments, no teaching. It’s an experiment in unstructured learning for adults.

My role as a facilitator was to help people make the most of that disorienting amount of freedom that they had at RC. People who come out of traditional educational experiences or traditional jobs very often don’t know what to do with that. So I’d help them with goal-setting and help them make the most of the experience. One of the things we thought a lot about was how to have the most effective learning experience possible for programmers. Today I’ll talk about some of the research into how to be an effective learner, and how we can apply that research to our daily lives as programmers and engineers.

What to get out of this post

Take a minute and consider what you’d like to get out of this post. You might want to learn something new about how to be as efficient and effective in your job as possible. You might want to hear about how you can be a better teacher or mentor to junior engineers. Or you might want to hear about how you can make institutional change in your organization to set up a better environment for these kinds of things.

All of these are useful goals, and I’ll touch on material relevant to all of them. However, I want to challenge you to consider the strategies mostly for yourself. When I hear about these strategies, very often it seems obvious to me that other people should be following them, but not necessarily obvious that I myself should. I’ll come back to that tension a little bit later on.

Growth mindset: Carol Dweck

Let’s talk about the first key to effective learning. The sociologist Carol Dweck has done a ton of interesting research about how people think about intelligence. She’s found that there are two different frameworks for thinking about intelligence. The first, which she calls the fixed mindset, holds that intelligence is a fixed trait, and people can’t change how much of it they have. The other mindset is a growth mindset. Under a growth mindset, people believe that intelligence is malleable and can increase with effort.

Dweck found that a person’s theory of intelligence – whether they hold a fixed or growth mindset – can significantly influence the way they select tasks to work on, the way they respond to challenges, their cognitive performance, and even their honesty. I’m going to run through a couple of the most interesting results from her work here.

These mindsets cause differences in effort

The first interesting result is that this framing impacts how people view effort. If you have a fixed mindset – you believe that people are either smart or they’re not, and they can’t really change that – then you also tend to believe that if you’re good at something, it should be easy for you, and if something is hard for you than you must not be good at it. That’s a fixed-mindset view. People who have a growth mindset believe that you need to exert effort and work hard at something to become better at it.

Several studies found that people with a fixed mindset can be reluctant to really exert effort, because they believe it means they’re not good at the thing they’re working hard on. Dweck notes, “It would be hard to maintain confidence in your ability if every time a task requires effort, your intelligence is called into question.”

“Praise that backfires”

The second interesting result is probably the most famous. Dweck and her collaborators showed that giving students subtly different kinds of praise significantly impacted their performance.

In this study, Dweck and her collaborators gave a students a series of problems. After the first set of problems, all of the students did pretty well. Then half of the students were told “Wow, you did really well on those problems – you must be very smart.” and the other “Wow, you did really well on those problems – you must have worked very hard.” Then they got a second set of problems, much harder, where everyone did badly. Then they got a third set of problems that were like the first set – back to the easier level.

Here, they’re creating a fixed mindset in the first group of students (your performance shows that you’re smart) and a growth mindset in the second set of students (your effort drives your success).

They found a bunch of interesting things from this. The first aspect of the experiment is that in between the first and second problem sets they asked the students if they’d like to do an easier exercise or a harder one next. (In practice, everyone got the harder set next.) Dweck et al. wanted to see if there would be a difference between the students who got different kinds of praise. And sure enough, there was: 90% of the students praised for effort chose to do a harder set of problems next, compared to only a third of the group praised for intelligence. The kids praised for effort were much more interested in a challenge.

The second thing that they looked at was how student performed on the third set of problems. They found that students who’d been praised for their intelligence did significantly worse on the third problem set than they had on the first, but students who’d been praised for effort did slightly better. Students who got intelligence praise weren’t able to recover effectively from hitting a wall on the second set of problems, while students who got effort praise could bounce back.

After this, they had the students write letters to pen pals about the study, saying “We did this study at school, and here’s the score that I got.” They found that almost half of the students praised for intelligence lied about their scores, and almost no one who was praised for working hard was dishonest.

So there are three implications here: a growth mindset made students more likely to choose a challenge instead of something easy, more likely to persist after a setback, and more honest about their performance, compared to the students with a fixed mindset.

What’s fascinating about this is how subtle the difference in praise is. Being told you’re smart leads to all of these attempts to preserve the appearance of smartness, by only doing easy things you know you can perform well on and by hiding your poor performance. Being told that you work hard leads to attempts to preserve the appearance of working hard – and the best way to do that is to actually work hard.

Response to confusion

Another study looked at what happened when students faced a temporary period of confusion. Dweck and her collaborators designed a short course on psychology to give to elementary school students. The course was a booklet on psychology followed by a quiz. Some of the booklets had a confusing passage in them, and others didn’t. The confusing part wasn’t on the quiz, so students could master the material if they just completely ignored the confusing bit. The researchers wanted to see whether students would be able to recover from being totally bewildered in the middle of this booklet.

They found that students with a growth mindset mastered the material about 70% of the time, regardless of whether there was a confusing passage in it. Among students with a fixed mindset, if they read the booklet without the confusing passage, again about 70% of them mastered the material. But the fixed-mindset students who encountered the confusing passage saw their mastery drop to 30%. Students with a fixed mindset were pretty bad at recovering from being confused.

“How can one best describe the nature of people who will most of all be that way which will make the imitating of others happen most often? Is it that these are the people we want to be like because they are fine or is it that these are the people we want to be liked by?”

I wanted to put up a section of the confusing passage because this really resonated with me. Hands up if you’ve ever started using a new tool and run into documentation that sounded like this. [Roughly 100% of hands go up.] It happens all the time – you get domain experts writing docs aimed at beginners, or out-of-date docs, or some other issue. It’s a critical skill for programmers to push past this kind of confusion and be able to successfully retain the rest of the information in the document we’re reading.

Programmers need a growth mindset

Programmers need a growth mindset! Key skills for programmers – like responding to confusion, recovering from setbacks, and being willing to take on new challenges – are all much easier with a growth mindset, and much harder with a fixed mindset.

Does anyone believe in a fixed mindset?

Now sometimes when people hear this idea of the fixed mindset, it almost sounds like a straw man. Like, does anyone in the tech industry actually believe this? I think that absolutely a fixed mindset is a widespread belief. Here are a couple of examples.

10x engineers

Start with the idea of the 10x engineer. This is the idea that some engineers are an order of magnitude more effective than others, for some definition of effective. And there’s lots of critiques of this framing, but we’ll set that aside for a moment. If you believe in the idea of the 10x engineer, do you think that engineer was born as a super effective engineer? Or did they get to be 10x one x at a time?

I think very often in the popular framing of this, the 10x engineer is set up on a pedestal, as someone that other people cannot become. Very often this is approached from a fixed-mindset perspective.

Hero worship

Another case where we see evidence of a fixed mindset is with hero worship. So Julie Pagano did a great talk at PyCon 2014 about impostor syndrome, and one of her suggestions for a way to combat impostor syndrome was “kill your heroes.” Don’t put other programmers on a pedestal, don’t say “that person is so different from me.” Fixed/growth mindset is a really useful framing for this too. If you have programming heroes, do you consider them to be totally different from you? Could you become more like the kind of person you admire? If you don’t think so, that’s some evidence of a fixed mindset.

So I’d argue that yes, a fixed mindset is quite prevalent in the tech industry.

Can you change a fixed mindset? Heck yes

Hopefully by now you’re convinced that a growth mindset is better for you than a fixed mindset. So the next question is: is this malleable? Can you take a fixed mindset and turn it into a growth mindset? And the answer is heck yes, you absolutely can change a fixed mindset into a growth one.

In fact, in many of Dweck’s studies they experimentally induce a fixed or growth mindset, often in really subtle ways. The praise study is one example: one sentence of praise changes the students’ behavior. In other studies they have students read a paragraph about a famous person’s success, and at the end it says “because they worked very hard,” or “because it was in their DNA.” This is absolutely a malleable thing.

So how do you change a fixed mindset? Sometimes the challenge is mostly in actually identifying the fixed mindset, and once you hear yourself say the words, “I could never learn physics,” it’s already obvious that that’s probably not true. But other times it’s harder to root out the fixed mindset. So here are a couple of flags you can use to identify fixed mindsets so you can root them out.

How do you identify a fixed mindset?

“I am ..”

“Some people are just …”

If you’re on the lookout for places where your mindset might be fixed, you should be listening for sentences that start like this. Things like “I’ve never been good at CSS” or “I’m not a people person” or “Some programmers are just faster than others.” Anything that starts with “I am …” is a candidate. The word “just” is often present.

Now, obviously, you can say sentences with “I am” that aren’t indicators of a fixed mindset. Instead, the point here is to treat sentences like this as a little bit of a yellow flag for yourself, to notice and then to examine your mindset more closely.

Just as an aside, the example “I’m not a people person” is supported by the research – Dweck and collaborators did a study on making friends and social situations, and this research holds there too. [See the Q&A for more about this.]

How do you change a fixed mindset?

Reframe praise & success

Ok, so once you’ve identified a fixed mindset, how can you go about changing it? Here are four strategies.

The first is to reframe praise and success. By reframe praise I mean that when you get the wrong kind of compliments, turn them into growth-mindset compliments. So if someone says “wow, great job on that project, you’re so smart,” translate it to “yeah, it was great, I worked really hard on that project.” You don’t necessarily have to do this out loud! But this reframing reinforces for yourself that you gain mastery by seeking out challenges and by exerting effort.

And you can use the same techniques for successes and accomplishments. When something goes well, don’t think, “Of course that went well because I’m awesome.” Instead think, “I used an effective strategy on that project! I should do that more often.”

Reframe failure

Of course the flip side of this dynamic is also really effective. A huge part of a fixed or growth mindset is how you respond to failure. What’s your self-talk when you face a setback or don’t get what you wanted? If you’re saying, “Maybe I’m not cut out for this job after all,” treat that as a red flag. Instead, ask what you learned from your unsuccessful attempt or what strategies you could have used instead. It sounds cheesy, but it really works.

Celebrate challenges

The third way that you can change a fixed mindset is to celebrate challenges. How do you respond when you have to struggle? Try explicitly celebrating. This is something that I was really consistent about when I was facilitating at the Recurse Center. Someone would sit down next to me and say, “[sigh] I think I’ve got a weird Python bug,” and I’d say, “Awesome, I love weird Python bugs!” First of all, this is definitely true – if you have a weird Python bug, let’s discuss – but more importantly, it emphasized to the participant that finding something where they struggled an accomplishment, it was intentional, and it was a good thing for them to have done that day.

As I mentioned, at the Recurse Center there are no deadlines and no assignments, so this attitude is pretty much free. I’d say, “You get to spend a day chasing down this weird bug in Flask, how exciting!” Now, at Dropbox, where we have a product to ship, and deadlines, and users, I’m not always uniformly delighted about spending a day on a weird bug. So I’m sympathetic to the reality of the world where there are deadlines. However, if I have a bug to fix, I have to fix it, and being grumbly about the existence of the bug isn’t going to help me fix it faster. I think that even in a world where deadlines loom, you can still apply this attitude.

Ask about processes

The last strategy for changing a fixed mindset is to ask about processes. Like many of you, I work with some great engineers. Sometimes, I’ll try to fix a tricky bug and won’t be able to, and then one of them will be able to fix it right away. In these situations I’ve tried to be really disciplined about asking how they did it. Particularly when I was new at Dropbox, the answers would be really illuminating. Sometimes the information had come from a source I didn’t know existed. Now that I’ve been there longer, it’s usually a technique or strategy difference, or a detail about why my strategy had not succeeded.

This is a much more useful strategy in the long term than saying “Oh, of course, that person got the bug because they are a wizard.”

Confidence & imposter syndrome

Dweck’s research is really interesting in the context of the discussion around impostor syndrome. Impostor syndrome is the feeling that you’re secretly an unqualified fraud who will be uncovered any second now. Hands up if you’ve ever felt impostor syndrome in your career? [80% of hands in the room go up.] Yeah, that’s lots of you, and I definitely have as well. And it sucks! It’s so painful, and it’s really bad for your career, because you’re less likely to take chances or to look for new opportunities to grow if you’re worrying about getting fired from the job you already have.

The proposed solutions for impostor syndrome very often center around confidence. Like, “Oh, if you feel like you’re not qualified for the job you already have, you should be more confident, and then you’ll be fine.” This sometimes is as simple as, “Don’t feel that way,” which is not very helpful as advice goes. But even when it’s more nuanced than that, there’s a focus on confidence and past accomplishments.

Confidence doesn’t help you respond to challenges

Henderson & Dweck, 1990

But here’s the catch. Dweck’s research shows that confidence doesn’t actually predict your success at responding to new challenges or recovering from setbacks.

Henderson and Dweck did a study of students moving from elementary school to junior high in the U.S. They asked the students to assess their confidence when they were still in the younger grade, and they also measured whether the students held fixed or growth mindsets. Then they tracked the students’ academic performance in junior high.

They found that confident students with a fixed mindset suffered academically. By contrast, students with a growth mindset tended to thrive academically, regardless of whether their confidence was high or low. Confidence wasn’t a useful predictor of success at all.

Now, there’s lots of other research that shows confidence is correlated with success. Dweck argues that confidence is a good predictor of how well you can do things you’re already doing, but it’s not a good predictor of how you respond to new challenges and how you feel about failure.

The second, related point that Dweck has discovered is that a history of success also doesn’t impact how you respond to challenges and how you deal with failure.

So past successes don’t predict your response to new setbacks and failures, and your confidence level also doesn’t predict your response to failure. The thing that is a good predictor of resilience in the face of failure is having a growth mindset.

Break the framework

This is hugely exciting to me and I think it doesn’t come up nearly often enough in the discussions around impostor syndrome. This gives us a new and more useful framework for combating impostor syndrome. Basically, if you’re holding a fixed mindset, you’re going to be really stressed and afraid at any moment that you have to struggle. We’re programmers, so it’s mostly struggle, right? It’s struggle all the time. With a growth mindset, you can enjoy the struggling and enjoy working on something that’s really hard.

And guess what? When your identity isn’t being threatened by a particularly tricky bug, it’s a lot easier to stay focused on the bug. You’re not worried about also getting fired and being a fraud, so you can free up those cognitive resources to focus on the task at hand.

So, again: if you believe, for example, that “some people just aren’t cut out for programming,” you can spend a ton of time & energy trying to find evidence and validation and reassurance that you are one of the people who can make it. Instead, upend this framework. Break the idea of fixed levels of talent and move to the idea that everyone can increase their skill by exerting effort.

Self-theories: Their role in motivation, personality, and development

Having a growth mindset makes you more resilient to failure, makes it easier to exert effort, makes you more likely to take on challenges, all things that are very useful to programmers.

If you’d like to dig more into the details of this research, and also see some of the findings that I didn’t have time to cover today, I highly recommend a book Dweck wrote called Self-theories. Self-theories is a collection of short essays that summarize many major points of her research. It’s got detail about the studies but is accessible to a lay reader. She’s also got a book called Mindset that’s written for a pop-science audience, but if you want a little more nuance and detail about the particular studies, Self-theories is the way to go.

Q & A

A selection from the Q&A:

Q: Is there any research in growth and fixed mindsets at the team-level, and how teams approach problems?

A: I’m not aware of any, but that’s a fascinating question. I’d love to see that research if it exists.

Q: I read Mindset, and I’m a father to twin girls. I found that these strategies really changed their resilience and their approach to problem solving.

A: Yeah, this research is kind of terrifying. Like, do you tell your children that they’re smart? You’re ruining them! I didn’t have a chance to talk about this, but there is some research in this book about gender discrepancies, and findings that high-achieving girls are more likely to have a fixed mindset and less likely to risk failure when they hit something hard. Many of the women in the room in particular can probably relate to this.

Q: Is this binary, or a gray scale?

A: I think it probably is a spectrum, yes. For this research it’s classified into a binary model. I’m not precisely sure where the line gets drawn. And some of these cases with experimental induction of a fixed or growth mindset, if someone has one mindset going in and has the other induced, they’ll probably end up in a moderate place.

Q: Is it possible to have a fixed mindset in one area and a growth mindset in another?

A: Absolutely. One that is common for programmers is to have a growth mindset about programming and a fixed mindset about social skills.

Q (from a CS lecturer/TA): For our new students, is there a way we can create a growth mindset in the students? A lot of people come in from school with a fixed one, and it can be damaging in those early courses.

A: If you’re a lecturer or have a chance to get up in front of the auditorium, you can say it explicitly: “Programming is a skill that you can get better at with effort,” and even though it doesn’t sound like it’d convince people, the research shows that it does make a difference.

The other thing that’s really interesting is a study on a values exercise. This shows that having women in particular write down their values before they enter into a context where they’d experience stereotype threat can significantly improve their performance. The basic idea here is if you’re identifying as a programmer, and your programmer identity is threatened, that’s very painful and difficult. But if you have these other things that you value about yourself, then that mitigates the threat. The results are really dramatic for people who are marginalized in tech (and it doesn’t hurt those who aren’t). For more, see this worksheet by Leigh Honeywell.

Q: So this is nature versus nurture all over again, isn’t it?

A: I wouldn’t characterize it that way, in part because I think both of those remove agency from the individual. Your mindset is something that you can control to a significant extent. That’s why I think it’s so important to think about this research from the context of ourselves, and not just our children or our students.

Q: It’s easy to think of lots of ways to apply this in programming, but can you talk more about ways to apply this in social situations?

A: Sure. In the study covered in a Self-theories, Dweck had children write letters applying to the pen pal club (which was a real pen pal club – they did eventually match people up). Then all the children got rejected from the pen pal club. [Audience laughter] Before writing the letter, they’d told half the children, “This is to see how good you are at making friends,” and the other half, “This is a chance to practice and improve your ways of making friends.” The children who heard the fixed-mindset framing sometimes wrote the same letter or sometimes wrote a shorter and less detailed letter. The kids who got the growth framing were much more likely to write longer things, to be more inviting, to say, “Oh, I love talking to you” even though it’s a first letter to a pen pal. [Audience makes sympathetic noises.] Yeah, throughout this book Dweck and her collaborators were pretty careful to not traumatize any students, not to leave them thinking that they’re stupid or bad at making friends.

If you’re interested in particular strategies for social situations, I highly recommend the blog Captain Awkward. Captain Awkward has some constructions of social challenges, like “I’ll go to a party and talk to three people, and award myself ten points for each person I talk to and learn a fact about.” There’s a lot of interesting stuff on the internet about strategies for coping with social anxiety that I think you can apply whether or not that’s something that you struggle with.

Thanks

My thanks to Maggie Zhou, Amy Hanlon, Alyssa Frazee, and Julia Evans for feedback on early versions of this talk.

Thanks to Sasha Laundy, who invited people to consider what they wanted to get out of her PyCon talk on giving and getting help, and inspired me to use the same construction.

Thanks to the Kiwi PyCon organizers, particularly Marek Kuziel, for hosting me.

I spend a lot of time flipping back and forth between Python 2.x and 3.x: I use different versions for different projects, talk to people about different versions, explore differences between the two, and paste the output of REPL sessions into chat windows. I also like to keep long-running REPL sessions. These two activities in combination became quite confusing, and I’d often forget which version I was using.

1
2
3
4
5
6

>>> print some_var
  File "<stdin>", line 1
    print some_var
                 ^
SyntaxError: invalid syntax
>>> # *swears*

After the hundredth time I made this mistake, I decided to modify my prompt to make it always obvious which version was which, even in long-running REPL sessions. You can do this by creating a file to be run when Python starts up. Add this line to your .bashrc:

1

export PYTHONSTARTUP=~/mystartupscript.py

Then in mystartupscript.py:

1
2
3
4

import sys
if sys.version_info.major == 3:
    sys.ps1 = "PY3 >>> "
    sys.ps2 = "PY3 ... "

This makes it obvious when you’re about to slip up:

1
2
3

PY3 >>> for value in giant_collection:
PY3 ...     print(value)
PY3 ...

I’ve also add this line to mystartupscript.py to bite the bullet and start using print as a function everywhere:

1

from __future__ import print_function

This has no effect in Python3.x, but will move 2.x to the new syntax.

Big news for me: I’m leaving Hacker School and going to work for Dropbox in San Francisco, joining Jessica McKellar’s team. I met Jessica when she was part of the first round of residents at Hacker School in fall 2012, and I’ve had tremendous respect for her work and leadership ever since. Dropbox has an impressive crop of Pythonistas (including Guido van Rossum, of course), and I couldn’t be more excited to join. I’ll be moving to San Francisco at the end of October. If you have recommendations for people to meet, places to go, or things to do, let me know!

This means I’m leaving Hacker School, after more than two years facilitating. My last day will be October 24th. I love Hacker School, and I know I’m going to miss it. Hacker School is entirely responsible for the fact that I’m a programmer at all. I was working in a finance job and contemplating new careers when my brother saw this post about Hacker School’s experiment with Etsy to get more qualified women into the summer 2012 batch. I read the post and the thoughtful, welcoming FAQ, then went home and picked up a Python book. Two months later, I started Hacker School.

Hacker School is about becoming a better programmer, and there’s no doubt that it’s worked for me. For two years, I’ve had total freedom to chase down whatever weird thing catches my eye; I’ve worked with creative, hilarious, brilliant Hacker Schoolers and residents on a dizzying variety of projects; and I’ve been delighted to help build a more inclusive environment at Hacker School, although there’s always more work to be done. (If you’re a curious, sharp, and self-directed programmer, I can’t recommend Hacker School enough.)

I’m thankful that leaving my job at Hacker School doesn’t mean leaving the Hacker School community. I’m trading in my faculty status and becoming one of hundreds of alumni around the world. I’ll still be on Zulip, Community, and everywhere else Hacker Schoolers can be found, and I’ll still have my cape. I may be leaving, but I’ll never graduate.

I hit a very fun bug yesterday while trying to run a script that sends emails to certain subsets of Hacker Schoolers. When I tried to test the script locally, I discovered that one of the tables of the database, Batch, was missing from my local version. After briefly panicking and making sure that the actual site was still up, I could dig in.

It turns out that my local version of psql was way out of date, and as of a few days ago we’d started using a data type that wasn’t present in my old version. Because of that, creating that particular table failed when I pulled from the production database the night before. The failure was logged, but the output is so verbose that I didn’t notice the problem. Both the diagnosis and the fix here were easy – I went back and read the logs, googled the data type that was raising an error, and then upgraded Postgres.app and psql. That’s when the real trouble started.

The new version of Postgres.app was placed in a new spot on the $PATH, as you’d expect, and the upgrade prompted me to change my .bashrc, which I did. But the rake tasks we use to manage local copies of the database errored out with this message:

1
2

$ pg_restore --verbose --clean --no-acl --no-owner -h localhost -U `whoami` -d hackerschool latest.dump
sh: pg_restore: command not found

This was pretty clearly a $PATH problem. I tried the usual things first, like sourcing my .bashrc in the terminal I was using, closing the terminal and opening a new one, etc. None of that worked.

One thing that jumped out to me was the sh in the error message. That was an indicator that rake wasn’t using bash as a shell – it was using sh – which means my .bashrc wasn’t setting the environment. Reading the rake task showed that it was a thin wrapper around lots of system calls via Ruby’s system("cmd here"). I added the line system("echo $PATH") and verified that the new location of pg_restore wasn’t in it.

At this point I found I had lots of questions about the execution context of the rake task. Since I was making system calls and could easily edit the rakefile, I added in the line system("sh") to drop me into a shell mid-execution. This turned out to be an efficient way to figure out what was going on (and made me feel like a badass hacker).

From within in that shell, I could do $$ to get that process’s PID, then repeatedly do ps -ef | grep [PID] to find the parent process.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

sh-3.2$ $$
sh: 34652: command not found
sh-3.2$ ps -ef | grep 34652
  501 34652 34639   0  4:18PM ??         0:00.04 sh
    0 34881 34652   0  4:26PM ??         0:00.01 ps -ef
  501 34882 34652   0  4:26PM ??         0:00.01 grep 34652
sh-3.2$ ps -ef | grep 34639
  501 34639  2914   0  4:18PM ??         0:00.41 rake db:drop db:create db:pull
  501 34652 34639   0  4:18PM ??         0:00.04 sh
  501 34885 34652   0  4:28PM ??         0:00.00 grep 34639
sh-3.2$ ps -ef | grep 2914
  501  2914  2913   0 10Sep14 ??        27:11.72 spring app    | hackerschool | started 244 hours ago | development mode
  501 34639  2914   0  4:18PM ??         0:00.41 rake db:drop db:create db:pull
  501 34889 34652   0  4:28PM ??         0:00.01 grep 2914
sh-3.2$ ps -ef | grep 2913
  501  2914  2913   0 10Sep14 ??        27:11.98 spring app    | hackerschool | started 244 hours ago | development mode
  501 34892 34652   0  4:29PM ??         0:00.00 grep 2913
  501  2913     1   0 10Sep14 ttys001    0:00.94 spring server | hackerschool | started 244 hours ago

Aha! The parent process of the rake task I was running is the spring server, which starts on boot – several days ago, at the time – and doesn’t have the new and updated $PATH information.¹ A kick to the spring server (with kill 2913) forced the server process to restart with the new environment.

It turns out there’s a handy utility called pstree² (brew installable) to visualize the tree of processes. This would have saved me a couple of steps of grepping. For example:

1
2
3
4
5

hackerschool [master] $ pstree -p 35351
-+= 00001 root /sbin/launchd
 \-+- 35129 afk spring server | hackerschool | started 25 hours ago
   \-+= 35130 afk spring app    | hackerschool | started 25 hours ago | development mode
     \--- 35351 afk rails_console

This bug and some related ones have gotten me more interested in operating systems, and I’ve started reading the book Operating Systems: Three Easy Pieces. I’m only a few chapters in, but so far it’s readable, clear, and entertaining. I look forward to building up my mental model of processes and environments as I keep reading it.

“All accepted proposals are alike, but each rejected proposal is rejected in its own way” – Tolstoy, if he were on the PyCon talk review committee

I’m building a collection of old PyCon talk proposals, particularly rejected ones. I think rejected proposals are more interesting than accepted ones, for a couple of reasons:

See examples of anti-patterns

Flipping through these proposals, you can see concrete examples of the talk committee’s suggestions for what to avoid. There is an example of a “state of our project” talk and one of “here’s some code I hope to have written by the time the conference rolls around.”

“I can do better than that”

Being a great or famous programmer doesn’t mean you’ll give a great talk or submit a great proposal. You’ll notice that you can write a better proposal than some of the ones from people you’ve heard of. (This fits with the Kill your heroes theme from Julie Pagano’s great talk on impostor syndrome at PyCon 2014.)

Empathize with the talk committee

Any application is an exercise in empathy – you need to imagine what the people who will be reading your submission are thinking. What do they care about? Where are they coming from? You can read past proposals and decide if you’d make the same decision the committee did. When submitters have shared the feedback they received, you can see exactly what the committee members thought. This helps you write a proposal that will address their concerns.

The deadline for submitting a proposal is Monday, September 15th. I encourage you to browse through the collection of past proposals to get inspiration or to improve your proposal. Once you’ve submitted a proposal, please add it to the collection!

I talk to a lot of people at Hacker School and elsewhere who have been programming Python for some time and want to get a better mental model of what’s happening under the hood. The words “really” or “why” often features in these questions – “What’s really happening when I write a list comprehension?” “Why are function calls considered expensive?” If you’ve seen any of the rest of this blog, you know I love digging around in Python internals, and I’m always happy to share that with others.

Why do this?

First off, I reject the idea that you have to understand the internals of Python to be a good Python developer. Many of the things you’ll learn about Python won’t help you write better Python. The “under the hood” construction is specious, too – why stop at Python internals? Do you also need to know C perfectly, and the C compiler, and the assembly, and …

That said, I think you should dig around in Python – it sometimes will help you write better Python, you’ll be more prepared to contribute to Python if you want to, and most importantly, it’s often really interesting and fun.

Setup

Follow the instructions in the Python dev guide under “Version Control Setup” and “Getting the Source Code”. You now have a Python that you can play with.

Strategies

1. Naturalism

Peter Seibel has a great blog post about reading code. He thinks that “reading” isn’t how most people interact with code – instead, they dissect it. From the post:

But then it hit me. Code is not literature and we are not readers. Rather, interesting pieces of code are specimens and we are naturalists. So instead of trying to pick out a piece of code and reading it and then discussing it like a bunch of Comp Lit. grad students, I think a better model is for one of us to play the role of a 19th century naturalist returning from a trip to some exotic island to present to the local scientific society a discussion of the crazy beetles they found: “Look at the antenna on this monster! They look incredibly ungainly but the male of the species can use these to kill small frogs in whose carcass the females lay their eggs.”

The point of such a presentation is to take a piece of code that the presenter has understood deeply and for them to help the audience understand the core ideas by pointing them out amidst the layers of evolutionary detritus (a.k.a. kluges) that are also part of almost all code. One reasonable approach might be to show the real code and then to show a stripped down reimplementation of just the key bits, kind of like a biologist staining a specimen to make various features easier to discern.

2. Science!

I’m a big fan of hypothesis-driven debugging, and that also applies in exploring Python. I think you should not just sit down and read CPython at random. Instead, enter the codebase with (1) a question and (2) a hypothesis. For each thing you’re wondering about, make a guess for how it might be implemented, then try to confirm or refute your guess.

3. Guided tours

Follow a step-by-step guide to changing something in Python. I like Amy Hanlon’s post on changing a keyword in Python and Eli Bendersky’s on adding a keyword.

4. Reading recommendations.

I don’t think you should sit down and read CPython at random, but I do have some suggestions for my favorite modules that are implemented in Python. I think you should read the implementation of

1. timeit in Lib/timeit.py

2. namedtuple in Lib/collections.py.

If you have a favorite module implemented in Python, tweet at me and I’ll add it to this list.

5. Blog & talk

Did you learn something interesting? Write it up and share it, or present at your local meetup group! It’s easy to feel like everyone else already knows everything you know, but trust me, they don’t.

6. Rewrite

Try to write your own implementation of timeit or namedtuple before reading the implementation. Or read a bit of C and rewrite the logic in Python. Byterun is an example of the latter strategy.

Tools

I sometimes hesitate to recommend tooling because it’s so easy to get stuck on installation problems. If you’re having trouble installing something, get assistance (IRC, StackOverflow, a Meetup, etc.) These problems are challenging to fix if you haven’t seen them before, but often straightforward once you know what you’re looking for. If you don’t believe me, this thread features Guido van Rossum totally misunderstanding a problem he’s having with a module that turns out to be related to upgrading to OS X Mavericks. This stuff is hard.

1. Ack

I’d been using grep in the CPython codebase, which was noticeably slow. (It’s especially slow when you forget to add the . at the end of the command and grep patiently waits on stdin, a mistake I manage to make all the time.) I started using ack a few months ago and really like it.

If you’re on a Mac and use homebrew, you can brew install ack, which takes only a few seconds. Then do ack string_youre_looking_for and you get a nicely-formatted output. I imagine you could get the same result with grep if you knew the right options to pass it, but I find ack fast and simple.

Try using ack on the text of an error message or a mysterious constant. You may be surprised how often this leads you directly to the relevant source code.

2. timeit

Timing & efficiency questions are a great place to use the “Science!” strategy. You may have a question like “Which is faster, X or Y?” For example, is it faster to do two assignment statements in a row, or do both in one tuple-unpacking assignment? I’m guessing that the tuple-unpacking will take longer because of the unpacking step. Let’s find out!

1
2
3
4

~ ⚲ python -m timeit "x = 1; y = 2"
10000000 loops, best of 3: 0.0631 usec per loop
~ ⚲ python -m timeit "x, y = 1, 2"
10000000 loops, best of 3: 0.0456 usec per loop

I’m wrong! Interesting! I wonder why that is. What if instead of unpacking a tuple, we did …

A lot of people I talk to like using IPython for timing. IPython is pip-installable, and it usually installs smoothly into a virtual environment. In the IPython REPL, you can use %timeit for timing questions. There are also other magic functions available in IPython.

1
2
3
4
5

In [1]: %timeit x = 1; y = 2;
10000000 loops, best of 3: 82.3 ns per loop

In [2]: %timeit x, y = 1, 2
10000000 loops, best of 3: 47.3 ns per loop

One caveat on timing stuff – use timeit to enhance your understanding, but unless you have real speed problems, you should write code for clarity, not for miniscule speed gains like this one.

3. Disassembling

Python compiles down to bytecode, an intermediate representation of your Python code used by the Python virtual machine. It’s sometimes enlightening and often fun to look at that bytecode using the built-in dis module.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

>>> def one():
...     x = 1
...     y = 2
...
>>> def two():
...     x, y = 1, 2
...
>>> import dis
>>> dis.dis(one)
  2           0 LOAD_CONST               1 (1)
              3 STORE_FAST               0 (x)

  3           6 LOAD_CONST               2 (2)
              9 STORE_FAST               1 (y)
             12 LOAD_CONST               0 (None)
             15 RETURN_VALUE
>>> dis.dis(two)
  2           0 LOAD_CONST               3 ((1, 2))
              3 UNPACK_SEQUENCE          2
              6 STORE_FAST               0 (x)
              9 STORE_FAST               1 (y)
             12 LOAD_CONST               0 (None)
             15 RETURN_VALUE

The implementation of the various operations are in Python/ceval.c.

4. Inspect/cinspect

You can get into the habit of trying to call inspect on anything you’re curious about to see the source code.

1
2
3
4
5
6

>>> import inspect
>>> import collections
>>> print inspect.getsource(collections.namedtuple)
def namedtuple(typename, field_names, verbose=False, rename=False):
    """Returns a new subclass of tuple with named fields.
    ...

However, inspect will only show the source code of things that are implemented in Python, which can be frustrating.

1
2
3
4
5

>>> print inspect.getsource(collections.defaultdict)
Traceback (most recent call last):
   [... snip ...]
IOError: could not find class definition
>>> :(

To get around this, Puneeth Chaganti wrote a tool called cinspect that extends inspect to work reasonably consistently with C-implemented code as well.

5. K&R

I think C is about a hundred times easier to read than it is to write, so I encourage you to read C code even if you don’t totally know what’s going on. That said, I think an afternoon spent with the first few chapters of K&R would take you pretty far. Hacking: The Art of Exploitation is another fun, if less direct, way to learn C.

Get started!

CPython is a huge codebase, and you should expect that building a mental model of it will be a long process. Download the source code now and begin poking around, spending five or ten minutes when you’re curious about something. Over time, you’ll get faster and more rigorous, and the process will get easier.

Do you have recommended strategies and tools that don’t appear here? Let me know!

Last Thursday I gave a lightning talk at Hacker School about the peephole optimizer in Python. A “peephole optimization” is a compiler optimization that looks at a small chunk of code at a time and optimizes in that little spot. This post explains one surprising side-effect of an optimization in CPython.

Writing a test coverage tool

Suppose that we’re setting out to write a test coverage tool. Python provides an easy way to trace execution using sys.settrace, so a simple version of a coverage analyzer isn’t too hard.

Our code to test is one simple function:

1
2
3
4
5

def iffer(condition):
    if condition:
        return 3
    else:
        return 10

Then we’ll write the world’s simplest testing framework:

1
2
3
4
5
6
7
8

from example import iffer

def test_iffer():
    assert iffer(True) == 3
    assert iffer(False) == 10

def run_tests():
    test_iffer()

Now for the simplest possible coverage tool. We can pass sys.settrace any tracing function, and it’ll be called with the arguments frame, event, and arg every time an event happens in the execution. Lines of code being executed, function calls, function returns, and exceptions are all events. We’ll filter out everything but line and call events, then keep track of what line of code was executing.¹ Then we run the tests while the trace function is tracing, and finally report which (non-empty lines) failed to execute.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45

import sys
import tests
import inspect

class TinyCoverage(object):
    def __init__(self, file_to_watch):
        self.source_file = file_to_watch
        self.source_code = open(file_to_watch).readlines()
        self.executed_code = []

    def trace(self, frame, event, arg):
        current_file = inspect.getframeinfo(frame).filename

        if self.source_file in current_file and \
            (event == "line" or event == "call"):

            self.executed_code.append(frame.f_lineno)

        return self.trace

    def unexecuted_code(self):
        skipped = []
        for line_num in range(1, len(self.source_code)+1):
            if line_num not in self.executed_code:
                src = self.source_code[line_num - 1]
                if src != "\n":
                    skipped.append(line_num)
        return skipped

    def report(self):
        skipped = self.unexecuted_code()
        percent_skipped = float(len(skipped)) / len(self.source_code)
        if skipped:
            print "{} line(s) did not execute ({:.0%})".format(len(skipped), percent_skipped)
            for line_num in skipped:
                print line_num, self.source_code[line_num - 1]
        else:
            print "100% coverage, go you!"

if __name__ == '__main__':
    t = TinyCoverage('example.py')
    sys.settrace(t.trace)
    tests.run_tests()
    sys.settrace(None)
    t.report()

Let’s try it. We’re pretty confident in our test coverage – there are only two branches in the code, and we’ve tested both of them.

1
2
3

peephole [master *] ⚲ python coverage.py
1 line(s) did not execute (9%)
4     else:

Why didn’t the else line execute? To answer this, we’ll run our function through the disassembler.²

1
2
3
4
5
6
7
8
9
10
11
12
13

>>> from example import iffer
>>> import dis
>>> dis.dis(iffer)
  2           0 LOAD_FAST                0 (condition)
              3 POP_JUMP_IF_FALSE       10

  3           6 LOAD_CONST               1 (3)
              9 RETURN_VALUE

  5     >>   10 LOAD_CONST               2 (10)
             13 RETURN_VALUE
             14 LOAD_CONST               0 (None)
             17 RETURN_VALUE

You don’t need to follow exactly what’s going on in this bytecode, but note that the first column is the line numbers of source code and line 4, the one containing the else, doesn’t appear. Why not? Well, there’s nothing to do with an else statement – it’s just a separator between two branches of an if statement. The second line in the disassembly, POP_JUMP_IF_FALSE 10, means that the interpreter will pop the top thing off of the virtual machine stack, jump to bytecode index ten if that thing is false, or continue with the next instruction if it’s true.

From the bytecode’s perspective, there’s no difference at all between writing this:

1
2
3
4
5
6
7

if a:
    ...
else:
    if b:
       ...
    else:
        ...

and this:

1
2
3
4
5
6

if a:
    ...
elif b:
   ...
else:
    ...

(even though the second is better style).

We’ve learned we need to special-case else statements in our code coverage tool. Since there’s no logic in them, let’s just drop lines that only contain else:. We can revise our unexecuted_code method accordingly:

1
2
3
4
5
6
7
8

def unexecuted_code(self):
    skipped = []
    for line_num in range(1, len(self.source_code)+1):
        if line_num not in self.executed_code:
            src = self.source_code[line_num - 1]
            if src != "\n" and "else:\n" not in src:  # Add "else" dropping
                skipped.append(line_num)
    return skipped

Then run it again:

1
2

peephole [master *] ⚲ python coverage.py
100% coverage, go you!

Success!

Complications arise

Our previous example was really simple. Let’s add a more complex one.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

def iffer(condition):
    if condition:
        return 3
    else:
        return 10

def continuer():
    a = b = c = 0
    for n in range(100):
        if n % 2:
            if n % 4:
                a += 1
            continue
        else:
            b += 1
        c += 1

    return a, b, c

continuer will increment a on all odd numbers and increment b and c for all even numbers. Don’t forget to add a test:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

import sys
import inspect
from example2 import iffer, continuer

def test_iffer():
    assert iffer(True) == 3
    assert iffer(False) == 10

def test_continuer():
    assert continuer() == (50, 50, 50)

def run_tests():
    test_iffer()
    test_continuer()

1
2
3

peephole [master *] ⚲ python coverage2.py
1 line(s) did not execute (4%)
13             continue

Hmm. The test we wrote certainly did involve the continue statement – if the interpreter hadn’t skipped the bottom half of the loop, the test wouldn’t have passed. Let’s use the strategy we used before to understand what’s happening: examining the output of the disassembler.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52

>>> dis.dis(continuer)
  8           0 LOAD_CONST               1 (0)
              3 DUP_TOP
              4 STORE_FAST               0 (a)
              7 DUP_TOP
              8 STORE_FAST               1 (b)
             11 STORE_FAST               2 (c)

  9          14 SETUP_LOOP              79 (to 96)
             17 LOAD_GLOBAL              0 (range)
             20 LOAD_CONST               2 (100)
             23 CALL_FUNCTION            1
             26 GET_ITER
        >>   27 FOR_ITER                65 (to 95)
             30 STORE_FAST               3 (n)

 10          33 LOAD_FAST                3 (n)
             36 LOAD_CONST               3 (2)
             39 BINARY_MODULO
             40 POP_JUMP_IF_FALSE       72

 11          43 LOAD_FAST                3 (n)
             46 LOAD_CONST               4 (4)
             49 BINARY_MODULO
             50 POP_JUMP_IF_FALSE       27

 12          53 LOAD_FAST                0 (a)
             56 LOAD_CONST               5 (1)
             59 INPLACE_ADD
             60 STORE_FAST               0 (a)
             63 JUMP_ABSOLUTE           27

 13          66 JUMP_ABSOLUTE           27
             69 JUMP_FORWARD            10 (to 82)

 15     >>   72 LOAD_FAST                1 (b)
             75 LOAD_CONST               5 (1)
             78 INPLACE_ADD
             79 STORE_FAST               1 (b)

 16     >>   82 LOAD_FAST                2 (c)
             85 LOAD_CONST               5 (1)
             88 INPLACE_ADD
             89 STORE_FAST               2 (c)
             92 JUMP_ABSOLUTE           27
        >>   95 POP_BLOCK

 18     >>   96 LOAD_FAST                0 (a)
             99 LOAD_FAST                1 (b)
            102 LOAD_FAST                2 (c)
            105 BUILD_TUPLE              3
            108 RETURN_VALUE

There’s a lot more going on here, but you don’t need to understand all of it to proceed. Here are the things we need to know to make sense of this:

• The second column in the output is the index in the bytecode, the third is the byte name, and the fourth is the argument. The fifth, when present, is a hint about the meaning of the argument.
• POP_JUMP_IF_FALSE, POP_JUMP_IF_TRUE, and JUMP_ABSOLUTE have the jump target as their argument. So, e.g. POP_JUMP_IF_TRUE 27 means “if the popped expression is true, jump to position 27.”
• JUMP_FORWARD’s argument specifies the distance to jump forward in the bytecode, and the fifth column shows where the jump will end.
• When an iterator is done, FOR_ITER jumps forward the number of bytes specified in its argument.

Unlike the else case, the line containing the continue does appear in the bytecode. But trace through the bytecode using what you know about jumps: no matter how hard you try, you can’t end up on bytes 66 or 69, the two that belong to line 13.

The continue is unreachable because of a compiler optimization. In this particular optimization, the compiler notices that two instructions in a row are jumps, and it combines these two hops into one larger jump. So, in a very real sense, the continue line didn’t execute – it was optimized out – even though the logic reflected in the continue is still reflected in the bytecode.

What would this bytecode have looked like without the optimizations? There’s not currently an option to disable the peephole bytecode optimizations, although there will be in a future version of Python (following an extensive debate on the python-dev list). For now, the only way to turn off optimizations is to comment out the relevant line in compile.c, the call to PyCode_Optimize, and recompile Python. Here’s the diff, if you’re playing along at home.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

cpython ⚲ hg diff
diff -r 77f36cdb71b0 Python/compile.c
--- a/Python/compile.c  Fri Aug 01 17:48:34 2014 +0200
+++ b/Python/compile.c  Sat Aug 02 15:43:45 2014 -0400
@@ -4256,10 +4256,6 @@
     if (flags < 0)
         goto error;

-    bytecode = PyCode_Optimize(a->a_bytecode, consts, names, a->a_lnotab);
-    if (!bytecode)
-        goto error;
-
     tmp = PyList_AsTuple(consts); /* PyCode_New requires a tuple */
     if (!tmp)
         goto error;
@@ -4270,7 +4266,7 @@
     kwonlyargcount = Py_SAFE_DOWNCAST(c->u->u_kwonlyargcount, Py_ssize_t, int);
     co = PyCode_New(argcount, kwonlyargcount,
                     nlocals_int, stackdepth(c), flags,
-                    bytecode, consts, names, varnames,
+                    a->a_bytecode, consts, names, varnames,
                     freevars, cellvars,
                     c->c_filename, c->u->u_name,
                     c->u->u_firstlineno,

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52

>>> dis.dis(continuer)
  8           0 LOAD_CONST               1 (0)
              3 DUP_TOP
              4 STORE_FAST               0 (a)
              7 DUP_TOP
              8 STORE_FAST               1 (b)
             11 STORE_FAST               2 (c)

  9          14 SETUP_LOOP              79 (to 96)
             17 LOAD_GLOBAL              0 (range)
             20 LOAD_CONST               2 (100)
             23 CALL_FUNCTION            1 (1 positional, 0 keyword pair)
             26 GET_ITER
        >>   27 FOR_ITER                65 (to 95)
             30 STORE_FAST               3 (n)

  10         33 LOAD_FAST                3 (n)
             36 LOAD_CONST               3 (2)
             39 BINARY_MODULO
             40 POP_JUMP_IF_FALSE       72

  11         43 LOAD_FAST                3 (n)
             46 LOAD_CONST               4 (4)
             49 BINARY_MODULO
             50 POP_JUMP_IF_FALSE       66

  12         53 LOAD_FAST                0 (a)
             56 LOAD_CONST               5 (1)
             59 INPLACE_ADD
             60 STORE_FAST               0 (a)
             63 JUMP_FORWARD             0 (to 66)

  13    >>   66 JUMP_ABSOLUTE           27
             69 JUMP_FORWARD            10 (to 82)

  14    >>   72 LOAD_FAST                1 (b)
             75 LOAD_CONST               5 (1)
             78 INPLACE_ADD
             79 STORE_FAST               1 (b)

  15    >>   82 LOAD_FAST                2 (c)
             85 LOAD_CONST               5 (1)
             88 INPLACE_ADD
             89 STORE_FAST               2 (c)
             92 JUMP_ABSOLUTE           27
        >>   95 POP_BLOCK

  16    >>   96 LOAD_FAST                0 (a)
             99 LOAD_FAST                1 (b)
            102 LOAD_FAST                2 (c)
            105 BUILD_TUPLE              3
            108 RETURN_VALUE

Just as we expected, the jump targets have changed. The instruction at position 50, POP_JUMP_IF_FALSE, now has 66 as its jump target – a previously unreachable instruction associated with the continue. Instruction 63, JUMP_FORWARD, is also targeting 66. In both cases, the only way to reach this instruction is to jump to it, and the instruction itself jumps away.³

Now we can run our coverage tool with the unoptimized Python:

1
2

peephole [master *+] ⚲ ../cpython/python.exe coverage2.py
100% coverage, go you!

Complete success!

So is this a good idea or not?

Compiler optimizations are often a straightforward win. If the compiler can apply simple rules that make my code faster without requiring work from me, that’s great. Almost nobody requires a strict mapping of code that they write to code that ends up executing. So, peephole optimization in general: yes! Great!

But “almost nobody” is not nobody, and one kind of people who do require strict reasoning about executed code are the authors of test coverage software. In the python-dev thread I linked to earlier, there was an extensive discussion over whether or not serving this demographic by providing an option to disable to optimizations was worth increasing the complexity of the codebase. Ultimately it was decided that it was worthwhile, but this is a fair question to ask.

Further reading

Beyond the interesting Python-dev thread linked above, my other suggestions are mostly code. CPython’s peephole.c is pretty readable C code, and I encourage you to take a look at it. (“Constant folding” is a great place to start.) There’s also a website compileroptimizations.com which has short examples and discussion of 45 different optimizations. If you’d like to play with these code examples, they’re all available on my github.