I’m currently a senior engineer at Pilot.com, working on automating bookkeeping for startups. Before that, I worked for Dropbox on the desktop client team, and I’ll have a few stories about my work there. Earlier, I was a facilitator at the Recurse Center, a writers retreat for programmers in NYC. I studied astrophysics in college and worked in finance for a few years before becoming an engineer.
But none of that is really important to remember – the only thing you need to know about me is that I love bugs. I love bugs because they’re entertaining. They’re dramatic. The investigation of a great bug can be full of twists and turns. A great bug is like a good joke or a riddle – you’re expecting one outcome, but the result veers off in another direction.
Over the course of this talk I’m going to tell you about some bugs that I have loved, explain why I love bugs so much, and then convince you that you should love bugs too.
Ok, straight into bug #1. This is a bug that I encountered while working at Dropbox. As you may know, Dropbox is a utility that syncs your files from one computer to the cloud and to your other computers.
1 2 3 4 5 6 7 8 9 10 11 12 |
|
Here’s a vastly simplified diagram of Dropbox’s architecture. The desktop client runs on your local computer listening for changes in the file system. When it notices a changed file, it reads the file, then hashes the contents in 4MB blocks. These blocks are stored in the backend in a giant key-value store that we call blockserver. The key is the digest of the hashed contents, and the values are the contents themselves.
Of course, we want to avoid uploading the same block multiple times. You can imagine that if you’re writing a document, you’re probably mostly changing the end – we don’t want to upload the beginning over and over. So before uploading a block to the blockserver the client talks to a different server that’s responsible for managing metadata and permissions, among other things. The client asks metaserver whether it needs the block or has seen it before. The “metaserver” responds with whether or not each block needs to be uploaded.
So the request and response look roughly like this: The client says, “I have a changed file made up of blocks with hashes 'abcd,deef,efgh'
”. The server responds, “I have those first two, but upload the third.” Then the client sends the block up to the blockserver.
1 2 3 4 5 6 7 8 9 10 11 12 |
|
That’s the setup. So here’s the bug.
1 2 3 4 5 6 7 8 9 10 11 12 |
|
Sometimes the client would make a weird request: each hash value should have been sixteen characters long, but instead it was thirty-three characters long – twice as many plus one. The server wouldn’t know what to do with this and would throw an exception. We’d see this exception get reported, and we’d go look at the log files from the desktop client, and really weird stuff would be going on – the client’s local database had gotten corrupted, or python would be throwing MemoryErrors, and none of it would make sense.
If you’ve never seen this problem before, it’s totally mystifying. But once you’d seen it once, you can recognize it every time thereafter. Here’s a hint: the middle character of each 33-character string that we’d often see instead of a comma was l
. These are the other characters we’d see in the middle position:
1
|
|
The ordinal value for an ascii comma – ,
– is 44. The ordinal value for l
is 108. In binary, here’s how those two are represented:
1 2 |
|
You’ll notice that an l
is exactly one bit away from a comma. And herein lies your problem: a bitflip. One bit of memory that the desktop client is using has gotten corrupted, and now the desktop client is sending a request to the server that is garbage.
And here are the other characters we’d frequently see instead of the comma when a different bit had been flipped.
1 2 3 4 5 6 7 8 |
|
I love this bug because it shows that bitflips are a real thing that can happen, not just a theoretical concern. In fact, there are some domains where they’re more common than others. One such domain is if you’re getting requests from users with low-end or old hardware, which is true for a lot of laptops running Dropbox. Another domain with lots of bitflips is outer space – there’s no atmosphere in space to protect your memory from energetic particles and radiation, so bitflips are pretty common.
You probably really care about correctness in space – your code might be keeping astronauts alive on the ISS, for example, but even if it’s not mission-critical, it’s hard to do software updates to space. If you really need your application to defend against bitflips, there are a variety of hardware & software approaches you can take, and there’s a very interesting talk by Katie Betchold about this.
Dropbox in this context doesn’t really need to protect against bitflips. The machine that is corrupting memory is a user’s machine, so we can detect if the bitflip happens to fall in the comma – but if it’s in a different character we don’t necessarily know it, and if the bitflip is in the actual file data read off of disk, then we have no idea. There’s a pretty limited set of places where we could address this, and instead we decide to basically silence the exception and move on. Often this kind of bug resolves after the client restarts.
This is one of my favorite bugs for a couple of reasons. The first is that it’s a reminder of the difference between unlikely and impossible. At sufficient scale, unlikely events start to happen at a noticable rate.
My second favorite thing about this bug is that it’s a tremendously social one. This bug can crop up anywhere that the desktop client talks to the server, which is a lot of different endpoints and components in the system. This meant that a lot of different engineers at Dropbox would see versions of the bug. The first time you see it, you can really scratch your head, but after that it’s easy to diagnose, and the investigation is really quick: you look at the middle character and see if it’s an l
.
One interesting side-effect of this bug was that it exposed a cultural difference between the server and client teams. Occasionally this bug would be spotted by a member of the server team and investigated from there. If one of your servers is flipping bits, that’s probably not random chance – it’s probably memory corruption, and you need to find the affected machine and get it out of the pool as fast as possible or you risk corrupting a lot of user data. That’s an incident, and you need to respond quickly. But if the user’s machine is corrupting data, there’s not a lot you can do.
So if you’re investigating a confusing bug, especially one in a big system, don’t forget to talk to people about it. Maybe your colleagues have seen a bug shaped like this one before. If they have, you might save a lot of time. And if they haven’t, don’t forget to tell people about the solution once you’ve figured it out – write it up or tell the story in your team meeting. Then the next time your teams hits something similar, you’ll all be more prepared.
Before I joined Dropbox, I worked for the Recurse Center. The idea behind RC is that it’s a community of self-directed learners spending time together getting better as programmers. That is the full extent of the structure of RC: there’s no curriculum or assignments or deadlines. The only scoping is a shared goal of getting better as a programmer. We’d see people come to participate in the program who had gotten CS degrees but didn’t feel like they had a solid handle on practical programming, or people who had been writing Java for ten years and wanted to learn Clojure or Haskell, and many other profiles as well.
My job there was as a facilitator, helping people make the most of the lack of structure and providing guidance based on what we’d learned from earlier participants. So my colleagues and I were very interested in the best techniques for learning for self-motivated adults.
There’s a lot of different research in this space, and one of the ones I think is most interesting is the idea of deliberate practice. Deliberate practice is an attempt to explain the difference in performance between experts & amateurs. And the guiding principle here is that if you look just at innate characteristics – genetic or otherwise – they don’t go very far towards explaining the difference in performance. So the researchers, originally Ericsson, Krampe, and Tesch-Romer, set out to discover what did explain the difference. And what they settled on was time spent in deliberate practice.
Deliberate practice is pretty narrow in their definition: it’s not work for pay, and it’s not playing for fun. You have to be operating on the edge of your ability, doing a project appropriate for your skill level (not so easy that you don’t learn anything and not so hard that you don’t make any progress). You also have to get immediate feedback on whether or not you’ve done the thing correctly.
This is really exciting, because it’s a framework for how to build expertise. But the challenge is that as programmers this is really hard advice to apply. It’s hard to know whether you’re operating at the edge of your ability. Immediate corrective feedback is very rare – in some cases you’re lucky to get feedback ever, and in other cases maybe it takes months. You can get quick feedback on small things in the REPL and so on, but if you’re making a design decision or picking a technology, you’re not going to get feedback on those things for quite a long time.
But one category of programming where deliberate practice is a useful model is debugging. If you wrote code, then you had a mental model of how it worked when you wrote it. But your code has a bug, so your mental model isn’t quite right. By definition you’re on the boundary of your understanding – so, great! You’re about to learn something new. And if you can reproduce the bug, that’s a rare case where you can get immediate feedback on whether or not your fix is correct.
A bug like this might teach you something small about your program, or you might learn something larger about the system your code is running in. Now I’ve got a story for you about a bug like that.
This bug also one that I encountered at Dropbox. At the time, I was investigating why some desktop client weren’t sending logs as consistently as we expected. I’d started digging into the client logging system and discovered a bunch of interesting bugs. I’ll tell you only the subset of those bugs that is relevant to this story.
Again here’s a very simplified architecture of the system.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
|
The desktop client would generate logs. Those logs were compress, encrypted, and written to disk. Then every so often the client would send them up to the server. The client would read a log off of disk and send it to the log server. The server would decrypt it and store it, then respond with a 200.
If the client couldn’t reach the log server, it wouldn’t let the log directory grow unbounded. After a certain point it would start deleting logs to keep the directory under a maximum size.
The first two bugs were not a big deal on their own. The first one was that the desktop client sent logs up to the server starting with the oldest one instead of starting with the newest. This isn’t really what you want – for example, the server would tell the client to send logs if the client reported an exception, so probably you care about the logs that just happened and not the oldest logs that happen to be on disk.
The second bug was similar to the first: if the log directory hit its maximum size, the client would delete the logs starting with the newest instead of starting with the oldest. Again, you lose log files either way, but you probably care less about the older ones.
The third bug had to do with the encryption. Sometimes, the server would be unable to decrypt a log file. (We generally didn’t figure out why – maybe it was a bitflip.) We weren’t handling this error correctly on the backend, so the server would reply with a 500. The client would behave reasonably in the face of a 500: it would assume that the server was down. So it would stop sending log files and not try to send up any of the others.
Returning a 500 on a corrupted log file is clearly not the right behavior. You could consider returning a 400, since it’s a problem with the client request. But the client also can’t fix the problem – if the log file can’t be decrypted now, we’ll never be able to decrypt it in the future. What you really want the client to do is just delete the log and move on. In fact, that’s the default behavior when the client gets a 200 back from the server for a log file that was successfully stored. So we said, ok – if the log file can’t be decrypted, just return a 200.
All of these bugs were straightforward to fix. The first two bugs were on the client, so we’d fixed them on the alpha build but they hadn’t gone out to the majority of clients. The third bug we fixed on the server and deployed.
Suddenly traffic to the log cluster spikes. The serving team reaches out to us to ask if we know what’s going on. It takes me a minute to put all the pieces together.
Before these fixes, there were four things going on:
A client with a corrupted log file would try to send it, the server would 500, the client would give up sending logs. On its next run, it would try to send the same file again, fail again, and give up again. Eventually the log directory would get full, at which point the client would start deleting its newest files, leaving the corrupted one on disk.
The upshot of these three bugs: if a client ever had a corrupted log file, we would never see logs from that client again.
The problem is that there were a lot more clients in this state than we thought. Any client with a single corrupted file had been dammed up from sending logs to the server. Now that dam was cleared, and all of them were sending up the rest of the contents of their log directories.
Ok, there’s a huge flood of traffic coming from machines around the world. What can we do? (This is a fun thing about working at a company with Dropbox’s scale, and particularly Dropbox’s scale of desktop clients: you can trigger a self-DDOS very easily.)
The first option when you do a deploy and things start going sideways is to rollback. Totally reasonable choice, but in this case, it wouldn’t have helped us. The state that we’d transformed wasn’t the state on the server but the state on the client – we’d deleted those files. Rolling back the server would prevent additional clients from entering this state but it wouldn’t solve the problem.
What about increasing the size of the logging cluster? We did that – and started getting even more requests, now that we’d increased our capacity. We increased it again, but you can’t do that forever. Why not? This cluster isn’t isolated. It’s making requests into another cluster, in this case to handle exceptions. If you have a DDOS pointed at one cluster, and you keep scaling that cluster, you’re going to knock over its depedencies too, and now you have two problems.
Another option we considered was shedding load – you don’t need every single log file, so can we just drop requests. One of the challenges here was that we didn’t have an easy way to tell good traffic from bad. We couldn’t quickly differentiate which log files were old and which were new.
The solution we hit on is one that’s been used at Dropbox on a number of different occassions: we have a custom header, chillout
, which every client in the world respects. If the client gets a response with this header, then it doesn’t make any requests for the provided number of seconds. Someone very wise added this to the Dropbox client very early on, and it’s come in handy more than once over the years. The logging server didn’t have the ability to set that header, but that’s an easy problem to solve. So two of my colleagues, Isaac Goldberg and John Lai, implemented support for it. We set the logging cluster chillout to two minutes initially and then managed it down as the deluge subsided over the next couple of days.
The first lesson from this bug is to know your system. I had a good mental model of the interaction between the client and the server, but I wasn’t thinking about what would happen when the server was interacting with all the clients at once. There was a level of complexity that I hadn’t thought all the way through.
The second lesson is to know your tools. If things go sideways, what options do you have? Can you reverse your migration? How will you know if things are going sideways and how can you discover more? All of those things are great to know before a crisis – but if you don’t, you’ll learn them during a crisis and then never forget.
The third lesson is for you if you’re writing a mobile or a desktop application: You need server-side feature gating and server-side flags. When you discover a problem and you don’t have server-side controls, the resolution might take days or weeks as you push out a new release or submit a new version to the app store. That’s a bad situation to be in. The Dropbox desktop client isn’t going through an app store review process, but just pushing out a build to tens of millions of clients takes time. Compare that to hitting a problem in your feature and flipping a switch on the server: ten minutes later your problem is resolved.
This strategy is not without its costs. Having a bunch of feature flags in your code adds to the complexity dramatically. You get a combinatoric problem with your testing: what if feature A is enabled and feature B, or just one, or neither – multiplied across N features. It’s extremely difficult to get engineers to clean up their feature flags after the fact (and I was also guilty of this). Then for the desktop client there’s multiple versions in the wild at the same time, so it gets pretty hard to reason about.
But the benefit – man, when you need it, you really need it.
I’ve talked about some bugs that I love and I’ve talked about why to love bugs. Now I want to tell you how to love bugs. If you don’t love bugs yet, I know of exactly one way to learn, and that’s to have a growth mindset.
The sociologist Carol Dweck has done a ton of interesting research about how people think about intelligence. She’s found that there are two different frameworks for thinking about intelligence. The first, which she calls the fixed mindset, holds that intelligence is a fixed trait, and people can’t change how much of it they have. The other mindset is a growth mindset. Under a growth mindset, people believe that intelligence is malleable and can increase with effort.
Dweck found that a person’s theory of intelligence – whether they hold a fixed or growth mindset – can significantly influence the way they select tasks to work on, the way they respond to challenges, their cognitive performance, and even their honesty.
[I also talked about a growth mindset in my Kiwi PyCon keynote, so here are just a few excerpts. You can read the full transcript here.]
Findings about honesty:
After this, they had the students write letters to pen pals about the study, saying “We did this study at school, and here’s the score that I got.” They found that almost half of the students praised for intelligence lied about their scores, and almost no one who was praised for working hard was dishonest.
On effort:
Several studies found that people with a fixed mindset can be reluctant to really exert effort, because they believe it means they’re not good at the thing they’re working hard on. Dweck notes, “It would be hard to maintain confidence in your ability if every time a task requires effort, your intelligence is called into question.”
On responding to confusion:
They found that students with a growth mindset mastered the material about 70% of the time, regardless of whether there was a confusing passage in it. Among students with a fixed mindset, if they read the booklet without the confusing passage, again about 70% of them mastered the material. But the fixed-mindset students who encountered the confusing passage saw their mastery drop to 30%. Students with a fixed mindset were pretty bad at recovering from being confused.
These findings show that a growth mindset is critical while debugging. We have to recover from confusion, be candid about the limitations of our understanding, and at times really struggle on the way to finding solutions – all of which is easier and less painful with a growth mindset.
I learned to love bugs by explicitly celebrating challenges while working at the Recurse Center. A participant would sit down next to me and say, “[sigh] I think I’ve got a weird Python bug,” and I’d say, “Awesome, I love weird Python bugs!” First of all, this is definitely true, but more importantly, it emphasized to the participant that finding something where they struggled an accomplishment, and it was a good thing for them to have done that day.
As I mentioned, at the Recurse Center there are no deadlines and no assignments, so this attitude is pretty much free. I’d say, “You get to spend a day chasing down this weird bug in Flask, how exciting!” At Dropbox and later at Pilot, where we have a product to ship, deadlines, and users, I’m not always uniformly delighted about spending a day on a weird bug. So I’m sympathetic to the reality of the world where there are deadlines. However, if I have a bug to fix, I have to fix it, and being grumbly about the existence of the bug isn’t going to help me fix it faster. I think that even in a world where deadlines loom, you can still apply this attitude.
If you love your bugs, you can have more fun while you’re working on a tough problem. You can be less worried and more focused, and end up learning more from them. Finally, you can share a bug with your friends and colleagues, which helps you and your teammates.
My thanks to folks who gave me feedback on this talk and otherwise contributed to my being there:
There’s lots of good advice for managers, mentors, and senior engineers about giving specific, actionable feedback – but it’s important to know what kind of feedback someone needs. I’ve found that people asking for more feedback are generally looking for one of two very different things:
The first kind of feedback is strategic feedback. The engineer asking for strategic feedback means something like this: “I feel like my work is going okay, and I’m wondering if I can be more effective. Are there strategies I can change to be even better?” The person asking these questions probably feels open, secure, and calm. They’re eager to grow and want to know if there’s anything they’re missing. Ideally, this engineer is asking specific questions for the feedback they’re seeking, like “Did my architecture doc clearly explain our project? Was my last pull request the right size and scope? What are the most important problems that our team is facing?” Feedback for this person should certainly be specific and actionable.
The other kind of feedback is less often dicussed – belonging feedback. An engineer seeking belonging feedback might be asking “Do you have any feedback for me?” but means something like, “Are things going ok? Do people like me? Am I making dumb mistakes?” In this state, they probably feel vulnerable. They might not yet feel comfortable with their coworkers. They might even be worried they’re going to get fired.
As a feedback-seeker (whether you’re a new engineer or an experienced hand), the more clear you can be about what you’re looking for, the more likely you are to get it. I once sat down for a one-on-one with my manager to discuss an incident, and said, “My agenda for this meeting is how it happened, what our plan for remediation is, and my feelings.” For my manager and me, this worked great. Being clear about what you want also helps you determine whom to ask for it – different problems might go to your direct manager, a more senior engineer, someone who joined the company at the same time you did, or a friend outside of work.
As a manager, mentor, or senior colleague, the best thing you can do is understand what kind of feedback is being requested. You don’t want to tell someone seeking strategic feedback, “Don’t worry! Everything’s great!” Similarly, you don’t want to give a list of ten areas for improvement to someone who’s already being too hard on themselves. What kind of feedback is being sought isn’t always obvious, and probably requires some follow-up questions.
Differentiating between these two types of request allows everyone to have a more pleasant experience and get better feedback at the same time.
]]>I broke my knee – a tibial plateau fracture – at the beginning of February, 2015. It required surgery and several months on crutches. I absolutely do not recommend this.
Recovering from an injury like this requires a lot of determination and a lot of help. I’m grateful to have been able to temporarily use all of the lazy-techie apps (groceries, laundry, ridesharing, etc. etc.). But there aren’t apps for everything, and I’m incredibly thankful for the many friends and family who helped with this process, from bringing me crutches when I first broke the knee, coming over with dinner, or keeping me company and taking out the trash on the way out.
It is really hard to be on crutches. Your triceps burn constantly. You can’t carry anything in your hands. I was very worried about falling. I live in a third-floor walkup, so the last thing I did every day was climb two flights of stairs. (How? One hand on the railing, one hand on the crutches, then jump.)
Oddly, I felt more comfortable in some ways while on crutches than I do while healthy. On crutches, it was obvious what the current focus was at any given point: learn to use crutches; avoid falling down; manage medication; go to physical therapy. When healthy, I constantly think that I’m about to start working out harder than I currently do. (I think I’m not the only one who does this.) It was in some ways easier to think, “Nothing about my fitness routine is going to change for at least twelve weeks. All I need to do is work on my knee.”
When I first started using the crutches, I was flabbergasted by some people who would ask about my injury very persistently. (“What happened? What kind of fracture? When’s your surgery? Where’d you have it? Do you have hardware?”) Then I realized that everyone with persistent questions had had knee surgery themselves. After realizing that, those people became very easy to deal with: I’d just ask them about their own surgeries and sit back while they told me all about it. Better yet, almost everyone I talked to had recovered fully and was now doing great, some number of years later.
If you’re on crutches, especially if you’re wearing the distinctive knee brace, the best piece of advice I can give you is this fact. Those with persistent questions have had their own surgeries. They would love to tell you about it.
In case you’re wondering – now that I’m recovered, I most certainly am one of those people. However, I always lead with my own surgery before asking any questions. :)
If you’re on crutches, I also recommend this marvel of human engineering and velcro, this detachable shower head, a travel coffee mug, a stylish backpack, and as much stubbornness as you can muster.
Notwithstanding the broken leg, I gave four conference talks in 2015. Two were at PyCon North America in Montreal while on crutches. One was at !!con in NYC, near the very end of the crutches era, and the fourth was at Kiwi PyCon in Christchurch, New Zealand, where I was fully ambulatory.
I gave these two talks at PyCon in Montreal back-to-back. “I think this’ll actually be easier for you,” said the organizer, and that turned out to be true, but not for reasons either of us predicted. As it turned out, the hardest part of PyCon 2015 for me was getting around on crutches, so the less of that I had to do the better off I was.
I was pleased with how these talks went, especially “Bytes in the Machine”, which I’d been working on in one form or another for more than a year. I originally proposed this talk for PyCon 2014 and it was rejected. I was able to propose and then give a substantially better version of it at PyCon 2015. One person told me that they’d never wanted to dig into CPython before and now they did, which was exactly what I was hoping for.
PyCon 2015 on crutches took an enormous amount of energy. The organizers were all very kind and helpful, but the convention center in Montreal was simply very large, and a ton of moving was required. My thanks to the organizers for their accomodations, and to the friends and strangers I pressed into carrying my lunches. (I also offer my apologies to anyone near enough to smell me on Friday, before my wayward luggage arrived. Crutching around is regrettably strenuous.)
This photo from Anja Boskovic shows me in the same body position I was in for almost three months: sitting on a chair with one leg up. I find this position to be quite masculine – asymmetric and unapologetically taking up space. There aren’t a lot of perks to breaking a leg, but I enjoyed taking up a lot of space while having a perfect excuse for doing so. There’s something delightful about having your foot up on the table during a meeting with someone who outranks you, or while presenting a talk at a conference. Interestingly, not everyone who attended the talk realized that I was using crutches during the conference, which means they can’t have properly attributed my body language to my injury.[^1]
Video and slides for Exploring is Never Boring Video and slides for Bytes in the Machine
If you’re catching up now and you prefer written material, consider reading the chapter version of this talk instead of watching the video.
!!con (“bangbangcon”) is a conference about “the joy, excitement, and surprise in programming.” This was my second year speaking there, and it’s consistently one of my favorite conferences. I described the CfP as “an invitation to meditate on your delight,” and the whole conference felt like that. Things can be challenging and difficult and outright terrible in this industry, and there’s a lot of hard work to do, but it’s nice to spend a couple of days learning about the amazing, fascinating, and weird world we live in.
This year there were talks on how wifi keeps getting faster, how to program a knitting machine, lightpainting with robots, making a cell phone, quines, and roller derby. It was a truly delightful lineup, and I’m honored to have been a part of it.
My talk covered how to hit the recursion limit in Python without doing any recursion and how to implement the world’s jankiest tail call optimization in Python. This talk features the following: – Me saying “Any day we can segfault Python is a good day in my book.” – Me saying “Remember, our beef today is with the compiler.” – An audible “Oh no” from the audience on seeing a slide with Python code and GOTOs
Unfortunately, the sound quality’s not great on this video.
[sketch by Danielle]
Video and slides for Limitless and Recursion-free recursion limits!
My final talk this year was a keynote at the Kiwi PyCon conference in Christchurch, New Zealand. I loved this trip. The organizers were hospitable from start to finish. Marek Kuziel was kind enough to meet me at the airport (and kind enough to depart before I attempted to drive my rental car on the left side of the road). I also want to give Marek credit for effective enforcement of a Code of Conduct. On one occassion in particular he gently redirected some mildly-dirty humor before anyone got uncomfortable. This can be tricky to do and he did it well.
I wrote a blog post in October that captures the best parts of this talk. There is also video available and slides.
After many rounds of writing and procrastinating, I finished and published my chapter for the Architecture of Open Source Applications 4th edition, on Byterun, a Python interpreter written in Python with Ned Batchelder. This version of the AOSA book is themed “500 lines or less,” and it features real software that does something significant in under 500 lines. It was a fun challenge to trim Byterun down to that size, and an even better challenge to try to explain the resulting code clearly.
My thanks to the editors enough for their patience and grit in this process, especially Mike DiBernardo and the talented copy editor Amy Brown.
A Python Interpreter written in Python
I’ve now been at Dropbox for slightly over a year. Most of what I’m most excited about I can’t talk about publically. What I can say is that I feel like I’ve matured as an engineer This means things like getting better at skills like living with my decisions, thinking farther ahead, architecting software, gathering consensus, getting and giving technical input, and other skills beyond pure programming.
I’m on the desktop client team, and desktop software in particular presents interesting challenges that I hadn’t thought much about before I joined Dropbox. For example, you generally can’t roll back a desktop release – once it’s out there, it’s out there. It’s also nontrivial to make sure we can get enough data to debug when something goes wrong. With a badly-behaving server, you might be able to ssh in and poke around. This is obviously not possible with someone else’s desktop.
[1] One person even congratulated Jessica McKellar for my talk, thinking she was me. I was obviously thrilled to be mistaken for her.
]]>Before I joined Dropbox last year, I spent two years working at a company in NYC called the Recurse Center. The Recurse Center is like a writers’ retreat for programmers. Participants spend 3 months working on whatever is most interesting to them. So someone who’d been writing Java for ten years might come to RC to learn a new language like Clojure, or someone who just graduated with a CS degree might come work on their web development skills, or someone who’d been learning programming in their spare time might come to turbo-charge their learning. There’s almost no structure to the program – no deadlines, no assignments, no teaching. It’s an experiment in unstructured learning for adults.
My role as a facilitator was to help people make the most of that disorienting amount of freedom that they had at RC. People who come out of traditional educational experiences or traditional jobs very often don’t know what to do with that. So I’d help them with goal-setting and help them make the most of the experience. One of the things we thought a lot about was how to have the most effective learning experience possible for programmers. Today I’ll talk about some of the research into how to be an effective learner, and how we can apply that research to our daily lives as programmers and engineers.
Take a minute and consider what you’d like to get out of this post. You might want to learn something new about how to be as efficient and effective in your job as possible. You might want to hear about how you can be a better teacher or mentor to junior engineers. Or you might want to hear about how you can make institutional change in your organization to set up a better environment for these kinds of things.
All of these are useful goals, and I’ll touch on material relevant to all of them. However, I want to challenge you to consider the strategies mostly for yourself. When I hear about these strategies, very often it seems obvious to me that other people should be following them, but not necessarily obvious that I myself should. I’ll come back to that tension a little bit later on.
Let’s talk about the first key to effective learning. The sociologist Carol Dweck has done a ton of interesting research about how people think about intelligence. She’s found that there are two different frameworks for thinking about intelligence. The first, which she calls the fixed mindset, holds that intelligence is a fixed trait, and people can’t change how much of it they have. The other mindset is a growth mindset. Under a growth mindset, people believe that intelligence is malleable and can increase with effort.
Dweck found that a person’s theory of intelligence – whether they hold a fixed or growth mindset – can significantly influence the way they select tasks to work on, the way they respond to challenges, their cognitive performance, and even their honesty. I’m going to run through a couple of the most interesting results from her work here.
The first interesting result is that this framing impacts how people view effort. If you have a fixed mindset – you believe that people are either smart or they’re not, and they can’t really change that – then you also tend to believe that if you’re good at something, it should be easy for you, and if something is hard for you than you must not be good at it. That’s a fixed-mindset view. People who have a growth mindset believe that you need to exert effort and work hard at something to become better at it.
Several studies found that people with a fixed mindset can be reluctant to really exert effort, because they believe it means they’re not good at the thing they’re working hard on. Dweck notes, “It would be hard to maintain confidence in your ability if every time a task requires effort, your intelligence is called into question.”
The second interesting result is probably the most famous. Dweck and her collaborators showed that giving students subtly different kinds of praise significantly impacted their performance.
In this study, Dweck and her collaborators gave a students a series of problems. After the first set of problems, all of the students did pretty well. Then half of the students were told “Wow, you did really well on those problems – you must be very smart.” and the other “Wow, you did really well on those problems – you must have worked very hard.” Then they got a second set of problems, much harder, where everyone did badly. Then they got a third set of problems that were like the first set – back to the easier level.
Here, they’re creating a fixed mindset in the first group of students (your performance shows that you’re smart) and a growth mindset in the second set of students (your effort drives your success).
They found a bunch of interesting things from this. The first aspect of the experiment is that in between the first and second problem sets they asked the students if they’d like to do an easier exercise or a harder one next. (In practice, everyone got the harder set next.) Dweck et al. wanted to see if there would be a difference between the students who got different kinds of praise. And sure enough, there was: 90% of the students praised for effort chose to do a harder set of problems next, compared to only a third of the group praised for intelligence. The kids praised for effort were much more interested in a challenge.
The second thing that they looked at was how student performed on the third set of problems. They found that students who’d been praised for their intelligence did significantly worse on the third problem set than they had on the first, but students who’d been praised for effort did slightly better. Students who got intelligence praise weren’t able to recover effectively from hitting a wall on the second set of problems, while students who got effort praise could bounce back.
After this, they had the students write letters to pen pals about the study, saying “We did this study at school, and here’s the score that I got.” They found that almost half of the students praised for intelligence lied about their scores, and almost no one who was praised for working hard was dishonest.
So there are three implications here: a growth mindset made students more likely to choose a challenge instead of something easy, more likely to persist after a setback, and more honest about their performance, compared to the students with a fixed mindset.
What’s fascinating about this is how subtle the difference in praise is. Being told you’re smart leads to all of these attempts to preserve the appearance of smartness, by only doing easy things you know you can perform well on and by hiding your poor performance. Being told that you work hard leads to attempts to preserve the appearance of working hard – and the best way to do that is to actually work hard.
Another study looked at what happened when students faced a temporary period of confusion. Dweck and her collaborators designed a short course on psychology to give to elementary school students. The course was a booklet on psychology followed by a quiz. Some of the booklets had a confusing passage in them, and others didn’t. The confusing part wasn’t on the quiz, so students could master the material if they just completely ignored the confusing bit. The researchers wanted to see whether students would be able to recover from being totally bewildered in the middle of this booklet.
They found that students with a growth mindset mastered the material about 70% of the time, regardless of whether there was a confusing passage in it. Among students with a fixed mindset, if they read the booklet without the confusing passage, again about 70% of them mastered the material. But the fixed-mindset students who encountered the confusing passage saw their mastery drop to 30%. Students with a fixed mindset were pretty bad at recovering from being confused.
“How can one best describe the nature of people who will most of all be that way which will make the imitating of others happen most often? Is it that these are the people we want to be like because they are fine or is it that these are the people we want to be liked by?”
I wanted to put up a section of the confusing passage because this really resonated with me. Hands up if you’ve ever started using a new tool and run into documentation that sounded like this. [Roughly 100% of hands go up.] It happens all the time – you get domain experts writing docs aimed at beginners, or out-of-date docs, or some other issue. It’s a critical skill for programmers to push past this kind of confusion and be able to successfully retain the rest of the information in the document we’re reading.
Programmers need a growth mindset! Key skills for programmers – like responding to confusion, recovering from setbacks, and being willing to take on new challenges – are all much easier with a growth mindset, and much harder with a fixed mindset.
Now sometimes when people hear this idea of the fixed mindset, it almost sounds like a straw man. Like, does anyone in the tech industry actually believe this? I think that absolutely a fixed mindset is a widespread belief. Here are a couple of examples.
Start with the idea of the 10x engineer. This is the idea that some engineers are an order of magnitude more effective than others, for some definition of effective. And there’s lots of critiques of this framing, but we’ll set that aside for a moment. If you believe in the idea of the 10x engineer, do you think that engineer was born as a super effective engineer? Or did they get to be 10x one x at a time?
I think very often in the popular framing of this, the 10x engineer is set up on a pedestal, as someone that other people cannot become. Very often this is approached from a fixed-mindset perspective.
Another case where we see evidence of a fixed mindset is with hero worship. So Julie Pagano did a great talk at PyCon 2014 about impostor syndrome, and one of her suggestions for a way to combat impostor syndrome was “kill your heroes.” Don’t put other programmers on a pedestal, don’t say “that person is so different from me.” Fixed/growth mindset is a really useful framing for this too. If you have programming heroes, do you consider them to be totally different from you? Could you become more like the kind of person you admire? If you don’t think so, that’s some evidence of a fixed mindset.
So I’d argue that yes, a fixed mindset is quite prevalent in the tech industry.
Hopefully by now you’re convinced that a growth mindset is better for you than a fixed mindset. So the next question is: is this malleable? Can you take a fixed mindset and turn it into a growth mindset? And the answer is heck yes, you absolutely can change a fixed mindset into a growth one.
In fact, in many of Dweck’s studies they experimentally induce a fixed or growth mindset, often in really subtle ways. The praise study is one example: one sentence of praise changes the students’ behavior. In other studies they have students read a paragraph about a famous person’s success, and at the end it says “because they worked very hard,” or “because it was in their DNA.” This is absolutely a malleable thing.
So how do you change a fixed mindset? Sometimes the challenge is mostly in actually identifying the fixed mindset, and once you hear yourself say the words, “I could never learn physics,” it’s already obvious that that’s probably not true. But other times it’s harder to root out the fixed mindset. So here are a couple of flags you can use to identify fixed mindsets so you can root them out.
If you’re on the lookout for places where your mindset might be fixed, you should be listening for sentences that start like this. Things like “I’ve never been good at CSS” or “I’m not a people person” or “Some programmers are just faster than others.” Anything that starts with “I am …” is a candidate. The word “just” is often present.
Now, obviously, you can say sentences with “I am” that aren’t indicators of a fixed mindset. Instead, the point here is to treat sentences like this as a little bit of a yellow flag for yourself, to notice and then to examine your mindset more closely.
Just as an aside, the example “I’m not a people person” is supported by the research – Dweck and collaborators did a study on making friends and social situations, and this research holds there too. [See the Q&A for more about this.]
Ok, so once you’ve identified a fixed mindset, how can you go about changing it? Here are four strategies.
The first is to reframe praise and success. By reframe praise I mean that when you get the wrong kind of compliments, turn them into growth-mindset compliments. So if someone says “wow, great job on that project, you’re so smart,” translate it to “yeah, it was great, I worked really hard on that project.” You don’t necessarily have to do this out loud! But this reframing reinforces for yourself that you gain mastery by seeking out challenges and by exerting effort.
And you can use the same techniques for successes and accomplishments. When something goes well, don’t think, “Of course that went well because I’m awesome.” Instead think, “I used an effective strategy on that project! I should do that more often.”
Of course the flip side of this dynamic is also really effective. A huge part of a fixed or growth mindset is how you respond to failure. What’s your self-talk when you face a setback or don’t get what you wanted? If you’re saying, “Maybe I’m not cut out for this job after all,” treat that as a red flag. Instead, ask what you learned from your unsuccessful attempt or what strategies you could have used instead. It sounds cheesy, but it really works.
The third way that you can change a fixed mindset is to celebrate challenges. How do you respond when you have to struggle? Try explicitly celebrating. This is something that I was really consistent about when I was facilitating at the Recurse Center. Someone would sit down next to me and say, “[sigh] I think I’ve got a weird Python bug,” and I’d say, “Awesome, I love weird Python bugs!” First of all, this is definitely true – if you have a weird Python bug, let’s discuss – but more importantly, it emphasized to the participant that finding something where they struggled an accomplishment, it was intentional, and it was a good thing for them to have done that day.
As I mentioned, at the Recurse Center there are no deadlines and no assignments, so this attitude is pretty much free. I’d say, “You get to spend a day chasing down this weird bug in Flask, how exciting!” Now, at Dropbox, where we have a product to ship, and deadlines, and users, I’m not always uniformly delighted about spending a day on a weird bug. So I’m sympathetic to the reality of the world where there are deadlines. However, if I have a bug to fix, I have to fix it, and being grumbly about the existence of the bug isn’t going to help me fix it faster. I think that even in a world where deadlines loom, you can still apply this attitude.
The last strategy for changing a fixed mindset is to ask about processes. Like many of you, I work with some great engineers. Sometimes, I’ll try to fix a tricky bug and won’t be able to, and then one of them will be able to fix it right away. In these situations I’ve tried to be really disciplined about asking how they did it. Particularly when I was new at Dropbox, the answers would be really illuminating. Sometimes the information had come from a source I didn’t know existed. Now that I’ve been there longer, it’s usually a technique or strategy difference, or a detail about why my strategy had not succeeded.
This is a much more useful strategy in the long term than saying “Oh, of course, that person got the bug because they are a wizard.”
Dweck’s research is really interesting in the context of the discussion around impostor syndrome. Impostor syndrome is the feeling that you’re secretly an unqualified fraud who will be uncovered any second now. Hands up if you’ve ever felt impostor syndrome in your career? [80% of hands in the room go up.] Yeah, that’s lots of you, and I definitely have as well. And it sucks! It’s so painful, and it’s really bad for your career, because you’re less likely to take chances or to look for new opportunities to grow if you’re worrying about getting fired from the job you already have.
The proposed solutions for impostor syndrome very often center around confidence. Like, “Oh, if you feel like you’re not qualified for the job you already have, you should be more confident, and then you’ll be fine.” This sometimes is as simple as, “Don’t feel that way,” which is not very helpful as advice goes. But even when it’s more nuanced than that, there’s a focus on confidence and past accomplishments.
But here’s the catch. Dweck’s research shows that confidence doesn’t actually predict your success at responding to new challenges or recovering from setbacks.
Henderson and Dweck did a study of students moving from elementary school to junior high in the U.S. They asked the students to assess their confidence when they were still in the younger grade, and they also measured whether the students held fixed or growth mindsets. Then they tracked the students’ academic performance in junior high.
They found that confident students with a fixed mindset suffered academically. By contrast, students with a growth mindset tended to thrive academically, regardless of whether their confidence was high or low. Confidence wasn’t a useful predictor of success at all.
Now, there’s lots of other research that shows confidence is correlated with success. Dweck argues that confidence is a good predictor of how well you can do things you’re already doing, but it’s not a good predictor of how you respond to new challenges and how you feel about failure.
The second, related point that Dweck has discovered is that a history of success also doesn’t impact how you respond to challenges and how you deal with failure.
So past successes don’t predict your response to new setbacks and failures, and your confidence level also doesn’t predict your response to failure. The thing that is a good predictor of resilience in the face of failure is having a growth mindset.
This is hugely exciting to me and I think it doesn’t come up nearly often enough in the discussions around impostor syndrome. This gives us a new and more useful framework for combating impostor syndrome. Basically, if you’re holding a fixed mindset, you’re going to be really stressed and afraid at any moment that you have to struggle. We’re programmers, so it’s mostly struggle, right? It’s struggle all the time. With a growth mindset, you can enjoy the struggling and enjoy working on something that’s really hard.
And guess what? When your identity isn’t being threatened by a particularly tricky bug, it’s a lot easier to stay focused on the bug. You’re not worried about also getting fired and being a fraud, so you can free up those cognitive resources to focus on the task at hand.
So, again: if you believe, for example, that “some people just aren’t cut out for programming,” you can spend a ton of time & energy trying to find evidence and validation and reassurance that you are one of the people who can make it. Instead, upend this framework. Break the idea of fixed levels of talent and move to the idea that everyone can increase their skill by exerting effort.
Having a growth mindset makes you more resilient to failure, makes it easier to exert effort, makes you more likely to take on challenges, all things that are very useful to programmers.
If you’d like to dig more into the details of this research, and also see some of the findings that I didn’t have time to cover today, I highly recommend a book Dweck wrote called Self-theories. Self-theories is a collection of short essays that summarize many major points of her research. It’s got detail about the studies but is accessible to a lay reader. She’s also got a book called Mindset that’s written for a pop-science audience, but if you want a little more nuance and detail about the particular studies, Self-theories is the way to go.
A selection from the Q&A:
Q: Is there any research in growth and fixed mindsets at the team-level, and how teams approach problems?
A: I’m not aware of any, but that’s a fascinating question. I’d love to see that research if it exists.
Q: I read Mindset, and I’m a father to twin girls. I found that these strategies really changed their resilience and their approach to problem solving.
A: Yeah, this research is kind of terrifying. Like, do you tell your children that they’re smart? You’re ruining them! I didn’t have a chance to talk about this, but there is some research in this book about gender discrepancies, and findings that high-achieving girls are more likely to have a fixed mindset and less likely to risk failure when they hit something hard. Many of the women in the room in particular can probably relate to this.
Q: Is this binary, or a gray scale?
A: I think it probably is a spectrum, yes. For this research it’s classified into a binary model. I’m not precisely sure where the line gets drawn. And some of these cases with experimental induction of a fixed or growth mindset, if someone has one mindset going in and has the other induced, they’ll probably end up in a moderate place.
Q: Is it possible to have a fixed mindset in one area and a growth mindset in another?
A: Absolutely. One that is common for programmers is to have a growth mindset about programming and a fixed mindset about social skills.
Q (from a CS lecturer/TA): For our new students, is there a way we can create a growth mindset in the students? A lot of people come in from school with a fixed one, and it can be damaging in those early courses.
A: If you’re a lecturer or have a chance to get up in front of the auditorium, you can say it explicitly: “Programming is a skill that you can get better at with effort,” and even though it doesn’t sound like it’d convince people, the research shows that it does make a difference.
The other thing that’s really interesting is a study on a values exercise. This shows that having women in particular write down their values before they enter into a context where they’d experience stereotype threat can significantly improve their performance. The basic idea here is if you’re identifying as a programmer, and your programmer identity is threatened, that’s very painful and difficult. But if you have these other things that you value about yourself, then that mitigates the threat. The results are really dramatic for people who are marginalized in tech (and it doesn’t hurt those who aren’t). For more, see this worksheet by Leigh Honeywell.
Q: So this is nature versus nurture all over again, isn’t it?
A: I wouldn’t characterize it that way, in part because I think both of those remove agency from the individual. Your mindset is something that you can control to a significant extent. That’s why I think it’s so important to think about this research from the context of ourselves, and not just our children or our students.
Q: It’s easy to think of lots of ways to apply this in programming, but can you talk more about ways to apply this in social situations?
A: Sure. In the study covered in a Self-theories, Dweck had children write letters applying to the pen pal club (which was a real pen pal club – they did eventually match people up). Then all the children got rejected from the pen pal club. [Audience laughter] Before writing the letter, they’d told half the children, “This is to see how good you are at making friends,” and the other half, “This is a chance to practice and improve your ways of making friends.” The children who heard the fixed-mindset framing sometimes wrote the same letter or sometimes wrote a shorter and less detailed letter. The kids who got the growth framing were much more likely to write longer things, to be more inviting, to say, “Oh, I love talking to you” even though it’s a first letter to a pen pal. [Audience makes sympathetic noises.] Yeah, throughout this book Dweck and her collaborators were pretty careful to not traumatize any students, not to leave them thinking that they’re stupid or bad at making friends.
If you’re interested in particular strategies for social situations, I highly recommend the blog Captain Awkward. Captain Awkward has some constructions of social challenges, like “I’ll go to a party and talk to three people, and award myself ten points for each person I talk to and learn a fact about.” There’s a lot of interesting stuff on the internet about strategies for coping with social anxiety that I think you can apply whether or not that’s something that you struggle with.
My thanks to Maggie Zhou, Amy Hanlon, Alyssa Frazee, and Julia Evans for feedback on early versions of this talk.
Thanks to Sasha Laundy, who invited people to consider what they wanted to get out of her PyCon talk on giving and getting help, and inspired me to use the same construction.
Thanks to the Kiwi PyCon organizers, particularly Marek Kuziel, for hosting me.
]]>1 2 3 4 5 6 |
|
After the hundredth time I made this mistake, I decided to modify my prompt to make it always obvious which version was which, even in long-running REPL sessions. You can do this by creating a file to be run when Python starts up. Add this line to your .bashrc
:
1
|
|
Then in mystartupscript.py
:
1 2 3 4 |
|
This makes it obvious when you’re about to slip up:
1 2 3 |
|
I’ve also add this line to mystartupscript.py
to bite the bullet and start using print as a function everywhere:
1
|
|
This has no effect in Python3.x, but will move 2.x to the new syntax.
]]>This means I’m leaving Hacker School, after more than two years facilitating. My last day will be October 24th. I love Hacker School, and I know I’m going to miss it. Hacker School is entirely responsible for the fact that I’m a programmer at all. I was working in a finance job and contemplating new careers when my brother saw this post about Hacker School’s experiment with Etsy to get more qualified women into the summer 2012 batch. I read the post and the thoughtful, welcoming FAQ, then went home and picked up a Python book. Two months later, I started Hacker School.
Hacker School is about becoming a better programmer, and there’s no doubt that it’s worked for me. For two years, I’ve had total freedom to chase down whatever weird thing catches my eye; I’ve worked with creative, hilarious, brilliant Hacker Schoolers and residents on a dizzying variety of projects; and I’ve been delighted to help build a more inclusive environment at Hacker School, although there’s always more work to be done. (If you’re a curious, sharp, and self-directed programmer, I can’t recommend Hacker School enough.)
I’m thankful that leaving my job at Hacker School doesn’t mean leaving the Hacker School community. I’m trading in my faculty status and becoming one of hundreds of alumni around the world. I’ll still be on Zulip, Community, and everywhere else Hacker Schoolers can be found, and I’ll still have my cape. I may be leaving, but I’ll never graduate.
]]>Batch
, was missing from my local version. After briefly panicking and making sure that the actual site was still up, I could dig in.
It turns out that my local version of psql was way out of date, and as of a few days ago we’d started using a data type that wasn’t present in my old version. Because of that, creating that particular table failed when I pulled from the production database the night before. The failure was logged, but the output is so verbose that I didn’t notice the problem. Both the diagnosis and the fix here were easy – I went back and read the logs, googled the data type that was raising an error, and then upgraded Postgres.app and psql. That’s when the real trouble started.
The new version of Postgres.app was placed in a new spot on the $PATH, as you’d expect, and the upgrade prompted me to change my .bashrc
, which I did. But the rake tasks we use to manage local copies of the database errored out with this message:
1 2 |
|
This was pretty clearly a $PATH problem. I tried the usual things first, like sourcing my .bashrc
in the terminal I was using, closing the terminal and opening a new one, etc. None of that worked.
One thing that jumped out to me was the sh
in the error message. That was an indicator that rake wasn’t using bash as a shell – it was using sh
– which means my .bashrc
wasn’t setting the environment. Reading the rake task showed that it was a thin wrapper around lots of system calls via Ruby’s system("cmd here")
. I added the line system("echo $PATH")
and verified that the new location of pg_restore
wasn’t in it.
At this point I found I had lots of questions about the execution context of the rake task. Since I was making system calls and could easily edit the rakefile, I added in the line system("sh")
to drop me into a shell mid-execution. This turned out to be an efficient way to figure out what was going on (and made me feel like a badass hacker).
From within in that shell, I could do $$
to get that process’s PID, then repeatedly do ps -ef | grep [PID]
to find the parent process.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
|
Aha! The parent process of the rake task I was running is the spring server, which starts on boot – several days ago, at the time – and doesn’t have the new and updated $PATH information.1 A kick to the spring server (with kill 2913
) forced the server process to restart with the new environment.
It turns out there’s a handy utility called pstree
2 (brew installable) to visualize the tree of processes. This would have saved me a couple of steps of grepping. For example:
1 2 3 4 5 |
|
This bug and some related ones have gotten me more interested in operating systems, and I’ve started reading the book Operating Systems: Three Easy Pieces. I’m only a few chapters in, but so far it’s readable, clear, and entertaining. I look forward to building up my mental model of processes and environments as I keep reading it.
]]>“All accepted proposals are alike, but each rejected proposal is rejected in its own way” – Tolstoy, if he were on the PyCon talk review committee
I’m building a collection of old PyCon talk proposals, particularly rejected ones. I think rejected proposals are more interesting than accepted ones, for a couple of reasons:
Flipping through these proposals, you can see concrete examples of the talk committee’s suggestions for what to avoid. There is an example of a “state of our project” talk and one of “here’s some code I hope to have written by the time the conference rolls around.”
Being a great or famous programmer doesn’t mean you’ll give a great talk or submit a great proposal. You’ll notice that you can write a better proposal than some of the ones from people you’ve heard of. (This fits with the Kill your heroes theme from Julie Pagano’s great talk on impostor syndrome at PyCon 2014.)
Any application is an exercise in empathy – you need to imagine what the people who will be reading your submission are thinking. What do they care about? Where are they coming from? You can read past proposals and decide if you’d make the same decision the committee did. When submitters have shared the feedback they received, you can see exactly what the committee members thought. This helps you write a proposal that will address their concerns.
The deadline for submitting a proposal is Monday, September 15th. I encourage you to browse through the collection of past proposals to get inspiration or to improve your proposal. Once you’ve submitted a proposal, please add it to the collection!
]]>First off, I reject the idea that you have to understand the internals of Python to be a good Python developer. Many of the things you’ll learn about Python won’t help you write better Python. The “under the hood” construction is specious, too – why stop at Python internals? Do you also need to know C perfectly, and the C compiler, and the assembly, and …
That said, I think you should dig around in Python – it sometimes will help you write better Python, you’ll be more prepared to contribute to Python if you want to, and most importantly, it’s often really interesting and fun.
Follow the instructions in the Python dev guide under “Version Control Setup” and “Getting the Source Code”. You now have a Python that you can play with.
Peter Seibel has a great blog post about reading code. He thinks that “reading” isn’t how most people interact with code – instead, they dissect it. From the post:
But then it hit me. Code is not literature and we are not readers. Rather, interesting pieces of code are specimens and we are naturalists. So instead of trying to pick out a piece of code and reading it and then discussing it like a bunch of Comp Lit. grad students, I think a better model is for one of us to play the role of a 19th century naturalist returning from a trip to some exotic island to present to the local scientific society a discussion of the crazy beetles they found: “Look at the antenna on this monster! They look incredibly ungainly but the male of the species can use these to kill small frogs in whose carcass the females lay their eggs.”
The point of such a presentation is to take a piece of code that the presenter has understood deeply and for them to help the audience understand the core ideas by pointing them out amidst the layers of evolutionary detritus (a.k.a. kluges) that are also part of almost all code. One reasonable approach might be to show the real code and then to show a stripped down reimplementation of just the key bits, kind of like a biologist staining a specimen to make various features easier to discern.
I’m a big fan of hypothesis-driven debugging, and that also applies in exploring Python. I think you should not just sit down and read CPython at random. Instead, enter the codebase with (1) a question and (2) a hypothesis. For each thing you’re wondering about, make a guess for how it might be implemented, then try to confirm or refute your guess.
Follow a step-by-step guide to changing something in Python. I like Amy Hanlon’s post on changing a keyword in Python and Eli Bendersky’s on adding a keyword.
I don’t think you should sit down and read CPython at random, but I do have some suggestions for my favorite modules that are implemented in Python. I think you should read the implementation of
timeit
in Lib/timeit.py
namedtuple
in Lib/collections.py
.If you have a favorite module implemented in Python, tweet at me and I’ll add it to this list.
Did you learn something interesting? Write it up and share it, or present at your local meetup group! It’s easy to feel like everyone else already knows everything you know, but trust me, they don’t.
Try to write your own implementation of timeit
or namedtuple
before reading the implementation. Or read a bit of C and rewrite the logic in Python. Byterun is an example of the latter strategy.
I sometimes hesitate to recommend tooling because it’s so easy to get stuck on installation problems. If you’re having trouble installing something, get assistance (IRC, StackOverflow, a Meetup, etc.) These problems are challenging to fix if you haven’t seen them before, but often straightforward once you know what you’re looking for. If you don’t believe me, this thread features Guido van Rossum totally misunderstanding a problem he’s having with a module that turns out to be related to upgrading to OS X Mavericks. This stuff is hard.
I’d been using grep
in the CPython codebase, which was noticeably slow. (It’s especially slow when you forget to add the .
at the end of the command and grep patiently waits on stdin, a mistake I manage to make all the time.) I started using ack a few months ago and really like it.
If you’re on a Mac and use homebrew, you can brew install ack
, which takes only a few seconds. Then do ack string_youre_looking_for
and you get a nicely-formatted output. I imagine you could get the same result with grep
if you knew the right options to pass it, but I find ack fast and simple.
Try using ack
on the text of an error message or a mysterious constant. You may be surprised how often this leads you directly to the relevant source code.
Timing & efficiency questions are a great place to use the “Science!” strategy. You may have a question like “Which is faster, X or Y?” For example, is it faster to do two assignment statements in a row, or do both in one tuple-unpacking assignment? I’m guessing that the tuple-unpacking will take longer because of the unpacking step. Let’s find out!
1 2 3 4 |
|
I’m wrong! Interesting! I wonder why that is. What if instead of unpacking a tuple, we did …
A lot of people I talk to like using IPython for timing. IPython is pip-installable, and it usually installs smoothly into a virtual environment. In the IPython REPL, you can use %timeit
for timing questions. There are also other magic functions available in IPython.
1 2 3 4 5 |
|
One caveat on timing stuff – use timeit
to enhance your understanding, but unless you have real speed problems, you should write code for clarity, not for miniscule speed gains like this one.
Python compiles down to bytecode, an intermediate representation of your Python code used by the Python virtual machine. It’s sometimes enlightening and often fun to look at that bytecode using the built-in dis
module.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 |
|
The implementation of the various operations are in Python/ceval.c
.
You can get into the habit of trying to call inspect
on anything you’re curious about to see the source code.
1 2 3 4 5 6 |
|
However, inspect
will only show the source code of things that are implemented in Python, which can be frustrating.
1 2 3 4 5 |
|
To get around this, Puneeth Chaganti wrote a tool called cinspect that extends inspect
to work reasonably consistently with C-implemented code as well.
I think C is about a hundred times easier to read than it is to write, so I encourage you to read C code even if you don’t totally know what’s going on. That said, I think an afternoon spent with the first few chapters of K&R would take you pretty far. Hacking: The Art of Exploitation is another fun, if less direct, way to learn C.
CPython is a huge codebase, and you should expect that building a mental model of it will be a long process. Download the source code now and begin poking around, spending five or ten minutes when you’re curious about something. Over time, you’ll get faster and more rigorous, and the process will get easier.
Do you have recommended strategies and tools that don’t appear here? Let me know!
]]>Suppose that we’re setting out to write a test coverage tool. Python provides an easy way to trace execution using sys.settrace
, so a simple version of a coverage analyzer isn’t too hard.
Our code to test is one simple function:
1 2 3 4 5 |
|
Then we’ll write the world’s simplest testing framework:
1 2 3 4 5 6 7 8 |
|
Now for the simplest possible coverage tool. We can pass sys.settrace
any tracing function, and it’ll be called with the arguments frame
, event
, and arg
every time an event happens in the execution. Lines of code being executed, function calls, function returns, and exceptions are all events. We’ll filter out everything but line
and call
events, then keep track of what line of code was executing.1 Then we run the tests while the trace function is tracing, and finally report which (non-empty lines) failed to execute.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 |
|
Let’s try it. We’re pretty confident in our test coverage – there are only two branches in the code, and we’ve tested both of them.
1 2 3 |
|
Why didn’t the else
line execute? To answer this, we’ll run our function through the disassembler.2
1 2 3 4 5 6 7 8 9 10 11 12 13 |
|
You don’t need to follow exactly what’s going on in this bytecode, but note that the first column is the line numbers of source code and line 4, the one containing the else
, doesn’t appear. Why not? Well, there’s nothing to do with an else statement – it’s just a separator between two branches of an if
statement. The second line in the disassembly, POP_JUMP_IF_FALSE 10
, means that the interpreter will pop the top thing off of the virtual machine stack, jump to bytecode index ten if that thing is false, or continue with the next instruction if it’s true.
From the bytecode’s perspective, there’s no difference at all between writing this:
1 2 3 4 5 6 7 |
|
and this:
1 2 3 4 5 6 |
|
(even though the second is better style).
We’ve learned we need to special-case else
statements in our code coverage tool. Since there’s no logic in them, let’s just drop lines that only contain else:
. We can revise our unexecuted_code
method accordingly:
1 2 3 4 5 6 7 8 |
|
Then run it again:
1 2 |
|
Success!
Our previous example was really simple. Let’s add a more complex one.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
|
continuer
will increment a
on all odd numbers and increment b
and c
for all even numbers. Don’t forget to add a test:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
|
1 2 3 |
|
Hmm. The test we wrote certainly did involve the continue
statement – if the interpreter hadn’t skipped the bottom half of the loop, the test wouldn’t have passed. Let’s use the strategy we used before to understand what’s happening: examining the output of the disassembler.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 |
|
There’s a lot more going on here, but you don’t need to understand all of it to proceed. Here are the things we need to know to make sense of this:
POP_JUMP_IF_FALSE
, POP_JUMP_IF_TRUE
, and JUMP_ABSOLUTE
have the jump target as their argument. So, e.g. POP_JUMP_IF_TRUE 27
means “if the popped expression is true, jump to position 27.”JUMP_FORWARD
’s argument specifies the distance to jump forward in the bytecode, and the fifth column shows where the jump will end.FOR_ITER
jumps forward the number of bytes specified in its argument.Unlike the else
case, the line containing the continue
does appear in the bytecode. But trace through the bytecode using what you know about jumps: no matter how hard you try, you can’t end up on bytes 66 or 69, the two that belong to line 13.
The continue
is unreachable because of a compiler optimization. In this particular optimization, the compiler notices that two instructions in a row are jumps, and it combines these two hops into one larger jump. So, in a very real sense, the continue
line didn’t execute – it was optimized out – even though the logic reflected in the continue
is still reflected in the bytecode.
What would this bytecode have looked like without the optimizations? There’s not currently an option to disable the peephole bytecode optimizations, although there will be in a future version of Python (following an extensive debate on the python-dev list). For now, the only way to turn off optimizations is to comment out the relevant line in compile.c
, the call to PyCode_Optimize
, and recompile Python. Here’s the diff, if you’re playing along at home.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 |
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 |
|
Just as we expected, the jump targets have changed. The instruction at position 50, POP_JUMP_IF_FALSE
, now has 66 as its jump target – a previously unreachable instruction associated with the continue
. Instruction 63, JUMP_FORWARD
, is also targeting 66. In both cases, the only way to reach this instruction is to jump to it, and the instruction itself jumps away.3
Now we can run our coverage tool with the unoptimized Python:
1 2 |
|
Complete success!
Compiler optimizations are often a straightforward win. If the compiler can apply simple rules that make my code faster without requiring work from me, that’s great. Almost nobody requires a strict mapping of code that they write to code that ends up executing. So, peephole optimization in general: yes! Great!
But “almost nobody” is not nobody, and one kind of people who do require strict reasoning about executed code are the authors of test coverage software. In the python-dev thread I linked to earlier, there was an extensive discussion over whether or not serving this demographic by providing an option to disable to optimizations was worth increasing the complexity of the codebase. Ultimately it was decided that it was worthwhile, but this is a fair question to ask.
Beyond the interesting Python-dev thread linked above, my other suggestions are mostly code. CPython’s peephole.c
is pretty readable C code, and I encourage you to take a look at it. (“Constant folding” is a great place to start.) There’s also a website compileroptimizations.com which has short examples and discussion of 45 different optimizations. If you’d like to play with these code examples, they’re all available on my github.
We need to include call
events to capture the first line of a function declaration, def fn(...):
↩
I’ve previously written an introduction to the disassembler here.↩
You may be wondering what the JUMP_ABSOLUTE
instruction at position 66 is doing. This instruction does nothing unless a particular compiler optimization is turned on. The optimization support faster loops, but creates restrictions on what those loops can do. See ceval.c
for more. Edit: This footnote previously incorrectly referenced JUMP_FORWARD
.↩
SyntaxWarning: import * only allowed at module level
. I had never seen a SyntaxWarning
before, so I decided to dig in.
The wording of the warning is strange: it says that star-import is only allowed at the module level, but it’s not a syntax error, just a warning. In fact, you can use a star-import in a scope that isn’t a module (in Python 2):
1 2 3 4 5 6 7 |
|
The Python spec gives some more details:
The from form with * may only occur in a module scope. If the wild card form of import — import * — is used in a function and the function contains or is a nested block with free variables, the compiler will raise a SyntaxError.
Just having import *
in a function isn’t enough to raise a syntax error – we also need free variables. The Python execution model refers to three kinds of variables, ‘local,’ ‘global,’ and ‘free’, defined as follows:
If a name is bound in a block, it is a local variable of that block. If a name is bound at the module level, it is a global variable. (The variables of the module code block are local and global.) If a variable is used in a code block but not defined there, it is a free variable.
Now we can see how to trigger a syntax error from our syntax warning:
1 2 3 4 5 6 7 8 9 |
|
and similarly,
1 2 3 4 5 6 7 8 9 |
|
As Python programmers, we’re used to our lovely dynamic language, and it’s unusual to hit compile-time constraints. As Amy Hanlon points out, it’s particularly weird to hit a compile-time error for code that wouldn’t raise a NameError when it ran – randint
would indeed be in one
’s namespace if the import-star had executed.
But we can’t run code that doesn’t compile, and in this case the compiler doesn’t have enough information to determine what bytecode to emit. There are different opcodes for loading and storing each of global, free, and local variables. A variable’s status as global, free, or local must be determined at compile time and then stored in the symbol table.
To investigate this, let’s look at minor variations on this code snippet and disassemble them.
1 2 3 4 5 6 7 8 9 10 11 |
|
First, when x
is local, the compiler emits STORE_FAST
in the assignment statement and LOAD_FAST
to load it, marked with arrows below.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 |
|
When x
is global, the compiler emits LOAD_GLOBAL
to load it. I think the assignment is STORE_FAST
again, but it’s not pictured here because the assignment is outside the function and thus not disassembled.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 |
|
Finally, when x
is nonlocal, the compiler notices that we’ll need a closure, and emits the opcodes LOAD_CLOSURE
, MAKE_CLOSURE
, and later LOAD_DEREF
.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 |
|
Let’s now return to a case that throws a syntax error.
1 2 3 4 5 6 7 8 9 |
|
I’d love to show what the disassembled bytecode for this one looks like, but we can’t do that because there is no bytecode! We got a compile-time error, so there’s nothing here.
Everything I know about symbol tables I learned from Eli Bendersky’s blog. I’ve skipped some complexity in the implementation that Eli covers.
ack
ing through the source code of CPython for the text of the error message leads us right to symtable.c
, which is exactly where we’d expect this message to be emitted. The function check_unoptimized
shows where the syntax error gets thrown (and shows another illegal construct, too – but we’ll leave that one as an exercise for the reader).
p.s. In Python 3, import *
anywhere other than a module is just an unqualified syntax error – none of this messing around with the symbol table.
quit
into avada_kedavra
, and many analogous jokes.
Amy also had the idea to replace import
with accio
! Replacing import
is a much harder problem than renaming a builtin. Python doesn’t prevent you from overwriting builtins, whereas to change keywords you have to edit the grammar and recompile Python. You should go read Amy’s post on making this work.
This brings us to an interesting question: why is import
a keyword, anyway? There’s a function, __import__
, that does (mostly) the same thing:
1 2 |
|
The function form requires the programmer to assign the return value – the module – to a name, but once we’ve done that it works just like a normal module:
1 2 3 |
|
The __import__
function can handle all the forms of import, including from foo import bar
and from baz import *
(although it never modifies the calling namespace). There’s no technical reason why __import__
couldn’t be the regular way to do imports.1
As far as I can tell, the main argument against an import
function is aesthetic. Compare:
1 2 3 4 5 6 7 |
|
The first way certainly feels much cleaner and more readable.2
Part of my goal in my upcoming PyCon talk is to invite Pythonistas to consider decisions they might not have thought about before. import
is a great vehicle for this, because everyone learns it very early on in their programming development, but most people don’t ever think about it again. Here’s another variation on that theme: import
doesn’t have to be a keyword!
I think all keywords could be expressed as functions, except those used for flow control (which I loosely define as keywords that generate any JUMP
instructions when compiled). For example, between Python 2 and 3, two keywords did become functions – print
and exec
.↩
I realize this is a slightly circular argument – if the function strategy were the regular way to import, it probably wouldn’t be so ugly.↩
Modifying Python’s grammar starts in the Grammar/Grammar file. I’ve recently learned how to read this (which, for me, mostly means learning how to pronounce the punctuation), so I want to walk through the import
example in some detail. The syntax here is Extended Backus-Naur Form, or EBNF. You read it like a tree, and your primary verb is “consists of”:
import_stmt
consists of one of two forms, import_name
or import_from
.import_name
consists of the literal word import
followed by dotted_as_names
.dotted_as_names
consists of a dotted_as_name
(note the singular), optionally followed by one or more pairs of a comma and another dotted_as_name
.dotted_as_name
consists of a dotted_name
, optionally followed by the literal word ‘as’ and a NAME.dotted_name
consists of a NAME
, maybe followed by pairs of a dot and another NAME
.You can walk the other branches in a similar way.
1 2 3 4 5 6 7 8 9 10 |
|
To accio
-ify Python, we had to replace the occurences of 'import'
with 'accio'
. There are only two – we were only interested in the literal string import
, not all the other names. import_as_name
and so on are just nodes in the tree, and only matter to the parser and compiler.
Every other keyword and symbol that has special meaning to the Python parser also appears in Grammar as a string.
Perusing the grammar is (goofy) way to learn about corner cases of Python syntax, too! For example, did you know that with
can take more than one context manager? It’s right there in the grammar:
1 2 |
|
1 2 3 4 5 6 7 8 |
|
Now go ahead and add your favorite keyword into Python!
Like Eli, I’m not advocating for Python’s actual grammar to change – it’s just a fun exercise.↩
import
at PyCon in April. In the talk, we’ll imagine that there is no import
and will reinvent it from scratch. I hope this will give everyone (including me!) a deeper understanding of the choices import
makes and the ways it could have been different. Ideally, the structure will be a couple of sections of the form “We could have made [decisions]. That would mean [effects]. Surprise – that’s how it works in [language]!”1
This is the first of (probably) several posts with notes of things I’m learning as I prepare my talk. Feedback is welcome.
Today I’m looking into Ruby’s require
and require_relative
2 to see if aspects of them would be interesting to Python programmers. So far, here’s what I think is most relevant:
Unlike Python, require
won’t load all objects in the required file. There’s a concept of local versus global variables in the file scope that doesn’t exist in Python.
Unlike Python, one file does not map to one module. Modules are created by using the keyword module
.
Unlike Python, namespace collisions are completely possible. Consider the following simple files:
1 2 3 4 5 |
|
1 2 3 4 5 |
|
1 2 3 4 |
|
And the output from running main.rb
:
1 2 3 |
|
import
, require
will only load a file once. This can interact interestingly with namespace collisions – to take a contrived example:1 2 3 4 5 |
|
Because one.rb
isn’t reloaded, foo
is still 'world'
:
1 2 3 |
|
My talk should not convince people that Python is Right and other languages are Wrong. I’m trying to overcome my bias towards the system I’m most used to. (I think I’ve written roughly equal amounts of Python and Ruby, but the vast majority of the Ruby I’ve written is Rails, where all the require
ing and namespacing happens by magic.) Here are some questions I’d like to research more.
Python’s namespacing feels much better to me, although I’m sure that’s partly because I’m used to it. What’s the advantage to doing namespacing this way?
Why have both require
and require_relative
? Why not have require
check the relative path as well before raising a LoadError
?
What’s the advantage of uncoupling a module from a file?
I asked on twitter for suggestions of languages that make interesting decisions about import
equivalents. So far the suggestions are R, Go, Rust, Ruby, JavaScript, and Clojure. If you have others, let me know.↩
As far as I can tell, the only difference between require
and require_relative
is the load path searched.↩
This is Part 4 in a series on the Python interpreter. Read Part 1, Part 2, and Part 3. If you’re enjoying this series, consider applying to Hacker School, where I work as a facilitator.
One of the things I was confused about when I started digging into python internals was how python could be “dynamic” if it was also “compiled.” Often, in casual coversation, those two words are used as antonyms – there are “dynamic languages,”1 like Python, Ruby, and Javascript, and “compiled languages,” like C, Java, and Haskell.
Most of the time, when people talk about a “compiled” language, they mean one that compiles down to native x86/ARM/etc instructions2 – instructions for an actual machine made of metal. An “interpreted” language either doesn’t have any compilation at all3, or compiles to an intermediate representation, like bytecode. Bytecode is instructions for a virtual machine, not a piece of hardware. Python falls into this latter category: the Python compiler’s job is to generate bytecode for the Python interpreter.4
The Python interpreter’s job is to make sense of the bytecode via the virtual machine, which turns out to be a lot of work. We’ll dig in to the virtual machine in Part 5.
So far our discussion of compiling versus interpretation has been abstract. These ideas become more clear with an example.
1 2 3 4 5 6 7 8 9 10 |
|
Here’s a function, its bytecode, and its bytecode run through the disassembler. By the time we get the prompt back after the function definition, the modulus
function has been compiled and a code object generated. That code object will never be modified.
This seems pretty easy to reason about. Unsurprisingly, typing a modulus (%
) causes the compiler to emit the instruction BINARY_MODULO
. It looks like this function will be useful if we need to calculate a remainder.
1 2 |
|
So far, so good. But what if we don’t pass it numbers?
1 2 |
|
Uh-oh, what happened there? You’ve probably seen this before, but it usually looks like this:
1 2 |
|
Somehow, when BINARY_MODULO
is faced with two strings, it does string interpolation instead of taking a remainder. This situation is a great example of dynamic typing. When the compiler built our code object for modulus
, it had no idea whether x
and y
would be strings, numbers, or something else entirely. It just emitted some instructions: load one name, load another, BINARY_MODULO
the two objects, and return the result. It’s the interpreter’s job to figure out what BINARY_MODULO
actually means.
I’d like to reflect on the depth of our ignorance for a moment. Our function modulus
can calculate remainders, or it can do string formatting … what else? If we define a custom object that responds to __mod__
, then we can do anything.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
|
The same function modulus
, with the same bytecode, has wildly different effects when passed different kinds of objects. It’s also possible for modulus
to raise an error – for example, a TypeError
if we called it on objects that didn’t implement __mod__
. Heck, we could even write a custom object that raises a SystemExit
when __mod__
is invoked. Our __mod__
function could have written to a file, or changed a global variable, or deleted another attribute of the object. We have near-total freedom.
This ignorance is one of the reasons that it’s hard to optimize Python: you don’t know when you’re compiling the code object and generating the bytecode what it’s going to end up doing. The compiler has no idea what’s going to happen. As Russell Power and Alex Rubinsteyn wrote in “How fast can we make interpreted Python?”, “In the general absence of type information, almost every instruction must be treated as INVOKE_ARBITRARY_METHOD.”
While a general definition of “compiling” and “interpreting” can be difficult to nail down, in the context of Python it’s fairly straightforward. Compiling is generating the code objects, including the bytecode. Interpreting is making sense of the bytecode in order to actually make things happen. One of the ways in which Python is “dynamic” is that the same bytecode doesn’t always have the same effect. More generally, in Python the compiler does relatively little work, and the intrepreter relatively more.
In Part 5, we’ll look at the actual virtual machine and interpreter.
You sometimes hear “interpreted language” instead of “dynamic language,” which is usually, mostly, synonymous.↩
Thanks to David Nolen for this definition. The lines between “parsing,” “compiling,” and “interpreting” are not always clear. ↩
Some languages that are usually not compiled at all include R, Scheme, and binary, depending on the implementation and your definition of “compile.”↩
As always in this series, I’m talking about CPython and Python 2.7, although most of this content is true across implementations.↩
This is Part 3 in a series on the Python interpreter. Part 1 here, Part 2 here. If you’re enjoying this series, consider applying to Hacker School, where I work as a facilitator.
When we left our heroes, they had come across some odd-looking output:
1 2 |
|
This is python bytecode.
You recall from Part 2 that “python bytecode” and “a python code object” are not the same thing: the bytecode is an attribute of the code object, among many other attributes. Bytecode is found in the co_code
attribute of the code object, and contains instructions for the interpreter.
So what is bytecode? Well, it’s just a series of bytes. They look wacky when we print them because some bytes are printable and others aren’t, so let’s take the ord
of each byte to see that they’re just numbers.
1 2 |
|
Here are the bytes that make up python bytecode. The interpreter will loop through each byte, look up what it should do for each one, and then do that thing. Notice that the bytecode itself doesn’t include any python objects, or references to objects, or anything like that.
One way to understand python bytecode would be to find the CPython interpreter file (it’s ceval.c
), and flip through it looking up what 100
means, then 1
, then 0
, and so on. We’ll do this later in the series! For now, there’s a simpler way: the dis
module.
Disassembling bytecode means taking this series of bytes and printing out something we humans can understand. It’s not a step in python execution; the dis
module just helps us understand an intermediate state of python internals. I can’t think of a reason why you’d ever want to use dis
in production code – it’s for humans, not for machines.
Today, however, taking some bytecode and making it human-readable is exactly what we’re trying to do, so dis
is a great tool. We’ll use the function dis.dis
to analyze the code object of our function foo
.
1 2 3 4 5 6 7 8 9 10 11 12 13 |
|
(You usually see this called as dis.dis(foo)
, directly on the function object. That’s just a convenience: dis
is really analyzing the code object. If it’s passed a function, it just gets its code object.)
The numbers in the left-hand column are line numbers in the original source code. The second column is the offset into the bytecode: LOAD_CONST
appears at position 0, STORE_FAST
at position 3, and so on. The middle column shows the names of bytes. These names are just for our (human) benefit – the interpreter doesn’t need the names.
The last two columns give details about the instructions’s argument, if there is an argument. The fourth column shows the argument itself, which represents an index into other attributes of the code object. In the example, LOAD_CONST
’s argument is an index into the list co_consts
, and STORE_FAST
’s argument is an index into co_varnames
. Finally, in the fifth column, dis
has looked up the constants or names in the place the fourth column specified and told us what it found there. We can easily verify this:
1 2 3 4 |
|
This also explains why the second instruction, STORE_FAST
, is found at bytecode position 3. If a bytecode has an argument, the next two bytes are that argument. It’s the interpreter’s job to handle this correctly.
(You may be surprised that BINARY_ADD
doesn’t have arguments. We’ll come back to this in a future installment, when we get to the interpreter itself.)
People often say that dis
is a disassembler of python bytecode. This is true enough – the dis
module’s docs say it – but dis
knows about more than just the bytecode, too: it uses the whole code object to give us an understandable printout. The middle three columns show information actually encoded in the bytecode, while the first and the last columns show other information. Again, the bytecode itself is really limited: it’s just a series of numbers, and things like names and constants are not a part of it.
How does the dis
module get from bytes like 100
to names like LOAD_CONST
and back? Try to think of a way you’d do it. If you thought “Well, you could have a list that has the byte names in the right order,” or you thought, “I guess you could have a dictionary where the names are the keys and the byte values are the values,” then congratulations! That’s exactly what’s going on. The file opcode.py
defines the list and the dictionary. It’s full of lines like these (def_op
inserts the mapping in both the list and the dictionary):
1 2 3 4 |
|
There’s even a friendly comment telling us what each byte’s argument means.
Ok, now we understand what python bytecode is (and isn’t), and how to use dis
to make sense of it. In Part 4, we’ll look at another example to see how Python can compile down to bytecode but still be a dynamic language.
This is part of a series on the python interpreter. Part 1 here.
When we left our heroes, they were examining a simple function object. Let’s now dive a level deeper, and look at this function’s code object.
1 2 3 4 5 6 7 8 |
|
As you can see in the code above, the code object is an attribute of the function object. (There are lots of other attributes on the function object, too. They’re mostly not interesting because foo
is so simple.)
A code object is generated by the Python compiler and intepreted by the interpreter. It contains information that this interpreter needs to do its job. Let’s look at the attributes of the code object.
1 2 3 4 5 6 7 |
|
There’s a bunch of stuff going on here, much of which we’re not going to worry about today. Let’s take a look at three attributes that are interesting to us for our code object on foo
.
1 2 3 4 5 6 |
|
Here are some intelligible-looking things: the names of the variables and the constants that our function knows about and the number of arguments the function takes. But so far, we haven’t seen anything that looks like instructions for how to execute the code object. These instructions are called bytecode. Bytecode is an attribute of the code object:
1 2 |
|
So much for our intelligible-looking things. What’s going on here? We’ll dive in to bytecode in Part 3.
]]>Over the last three months, I’ve spent a lot of time working with Ned Batchelder on byterun, a python bytecode interpreter written in python. Working on byterun has been tremendously educational and a lot of fun for me. At the end of this series, I’m going to attempt to convince you that it would be interesting and fun for you to play with byterun, too. But before we do that, we need a bit of a warm-up: an overview of how python’s internals work, so that we can understand what an interpreter is, what it does, and what it doesn’t do.
This series assumes that you’re in a similar position to where I was three months ago: you know python, but you don’t know anything about the internals.
One quick note: I’m going to work in and talk about Python 2.7 in this post. The interpreter in Python 3 is mostly pretty similar. There are also some syntax and naming differences, which I’m going to ignore, but everything we do here is possible in Python 3 as well.
We’ll start out with a really (really) high-level view of python’s internals. What happens when you execute a line of code in your python REPL?
1 2 3 4 5 |
|
There are four steps that python takes when you hit return: lexing, parsing, compiling, and interpreting. Lexing is breaking the line of code you just typed into tokens. The parser takes those tokens and generates a structure that shows their relationship to each other (in this case, an Abstract Syntax Tree). The compiler then takes the AST and turns it into one (or more) code objects. Finally, the interpreter takes each code object executes the code it represents.
I’m not going to talk about lexing, parsing, or compiling at all today, mainly because I don’t know anything about these steps yet. Instead, we’ll suppose that all that went just fine, and we’ll have a proper python code object for the interpreter to interpret.
Before we get to code objects, let me clear up some common confusion. In this series, we’re going to talk about function objects, code objects, and bytecode. They’re all different things. Let’s start with function objects. We don’t really have to understand function objects to get to the interpreter, but I want to stress that function objects and code objects are not the same – and besides, function objects are cool.
You might have heard of “function objects.” These are the things people are talking about when they say things like “Functions are first-class objects,” or “Python has first-class functions.” Let’s take a look at one.
1 2 3 4 5 6 |
|
“Functions are first-class objects,” means that function are objects, like a list is an object or an instance of MyObject
is an object. Since foo
is an object, we can talk about it without invoking it (that is, there’s a difference between foo
and foo()
). We can pass foo
into another function as an argument, or we could bind it to a new name (other_function = foo
). With first-class functions, all sorts of possibilities are open to us!
In Part 2, we’ll dive down a level and look at the code object.
]]>First, clarifications. (These weren’t always clear in the problem statement, particularly if you got the problem off of twitter, so award yourself full marks as desired.)
“Order doesn’t matter” means that the three-line version always returns False
, and the semicolon version always returns True
.
Several people, including Pepijn De Vos, David Wolever, and diarmuidbourke suggested something like the following:
1 2 3 4 5 6 |
|
I’m being pedantic here, but I rule this cheating, since (a) each line has to be a valid python expression or statement, and a multi-line string literal is only one expression, and (b) the string """a; b; c"""
is not the same as the string """a\nb\nc"""
.
Solutions appear below the fold.
Jessica suggests the following solution:
1 2 3 4 5 6 7 8 9 10 11 12 13 |
|
However, Jessica’s solution fails the “order doesn’t matter” test, and it is stateful:
1 2 3 4 |
|
Edit: As Jessica points out, I’m wrong here: her solution does pass the order test. She also notes that the restriction against state wasn’t present in the blog post (and she didn’t see the original tweet). Full credit to Jessica, then!
Javier suggests a solution for Python 2 that fails the order test:
1 2 3 4 5 6 |
|
Again, this depends on order and is stateful.
1 2 3 4 |
|
Alex suggests using sys._getframe(0)
. I might quibble that sys._getframe
constitutes introspection, but Alex is on to something. He notices that each line of code executed in the interpreter gets its own frame. (foreshadowing)
Alex gets full credit because I just discovered that the behavior underlying the original puzzle works in CPython, but is different in PyPy. Sorry, Alex!
Edit: Never mind – there is a simpler solution in PyPy. See below.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
|
For sheer inventiveness and creativity I have to hand it to Anton, working in the iPython REPL:
1 2 3 4 5 6 |
|
He’s exploiting the fact that when executed on a single line, all three lines are considered part of the for
loop. When the lines are broken up, only the x=n or x-1
part belongs to the loop. (This pretty much works in the standard CPython REPL too, but you have to throw in an extra line break.)
Alexey was the only person outside of Hacker School to hit on my solution.
1 2 3 4 5 6 |
|
Nicely done, Alexey!
(For the pedantic among us, like me: this was the version I had in mind when I described my solution as 14 non-whitespace characters. Of course, to make the function version work too, we have to add print
to the last line, taking us up to 19. I’ve edited the original post.)
So what’s going on here, and why is this interesting?
I described Alex Gaynor’s solution as foreshadowing earlier. Alex appears to have started from a good question: what’s different about executing three lines of code versus executing three statements as one line of code?
One thing to note is that each time you get an interpreter prompt back (>>>
), your python has compiled and executed some code. Every prompt corresponds to at least one code object.
How can we use the difference in code objects to generate this behavior? One way is to use constants: in CPython, two instances of the same number (larger than 256) are different objects. This explains why a is b
returns False
in the REPL (in the extended version, without semicolons). In the semicolon version, all three statements are part of one code object. We’re giving the CPython compiler a chance to optimize. It notices that we have two constants with the same value, and only hangs on to one copy.
Let’s take a look. It’s easy to get a handle on the code object corresponding to a function – use f.func_code
in Python 2, and f.__code__
in Python 3. Getting a handle on a code object corresponding to a line in the REPL is a little bit trickier, but we can do it if we compile them by hand. In Python 2.x:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 |
|
The compiler’s being a little smart here, and we only get one occurence of the constant 257
per code object. When we execute the assignment statements on two different lines, we get two different code objects with two different instances of the integer 257
. This explains why is
, which is comparing object identity, sees one as the same and the others as different.
This came up originally at Hacker School when Steve Katz stumbled across that thing with small integers and is
versus ==
in python. (If you’re not familiar with it, you can read many questions on Stack Overflow about it. As a general rule, don’t use is
for integer comparisons.) Steve went on to notice that the behavior changes when run from a script rather than in the REPL.
I didn’t realize when posing this problem that it was an implementation detail of CPython, and PyPy behaves differently (and arguably correctly). That’s why Alex Gaynor had to hook into the frame object.
More cleverness in my twitter feed!
Ned Batchelder suggests using -9
instead of 257
to shave off a few characters.
Zev came up with my favorite so far, in 14 characters including the print. He exploits the intricacies of floating-point arithmetic
1 2 3 4 5 |
|
If you’ve never dug around with floating-point math, do yourself a favor: it’s really interesting stuff. The Julia language suggests some good background reading.
Edit 2: Zev emails that floating-point precision isn’t involved here. Instead, he’s exploiting the fact that floating-point numbers – even those that can be precisely represented – are not interned. We’d get the same results using 1.
instead of .1
. This suggests a broader point: many solutions here could have used a = thing; a is thing
, omitting LINE_B.
Nick has a fix to the PyPy integer handling: use a string instead of an int.
1 2 3 4 5 6 |
|
This was fun! If I missed your solution and you want it to be included, ping me on twitter.
]]>1 2 3 4 5 6 7 8 9 10 11 12 |
|
What are the lines?
Some ground rules:
__foo__
) methods allowed.return
unless you add it everywhere.For bonus points, code golf! My solution to this is 14 19 characters long, not counting whitespace.