Allison Kaptur

An occasional blog on programming

2015 in Review

My 2015 was a year of recovering from a serious injury and becoming a better engineer. I also gave four talks and published a chapter I’ve been working on for a while. Here’s a quick summary of the year.

TPF

I broke my knee – a tibial plateau fracture – at the beginning of February, 2015. It required surgery and several months on crutches. I absolutely do not recommend this.

Recovering from an injury like this requires a lot of determination and a lot of help. I’m grateful to have been able to temporarily use all of the SF-lazy-techie apps (grocery delivery, laundry pickup, ridesharing, etc. etc.). But there aren’t apps for everything, and I’m incredibly thankful for the many friends and family who helped with this process, from bringing me crutches when I first broke the knee, coming over with dinner, or keeping me company and then taking out the trash on the way out.

It is really hard to use crutches. Your triceps burn constantly. You can’t carry anything in your hands. I was very worried about falling. I live in a third-floor walkup, so the last thing I did every day was climb two flights of stairs. (How? One hand on the railing, one hand on the crutches, then jump.)

Picking your battles

Oddly, I felt more comfortable in some ways while on crutches than I do while healthy. On crutches, it was obvious what the current focus was at any given point: learn to use crutches; avoid falling down; manage medication; go to physical therapy. When healthy, I constantly think that I’m about to start working out harder than I currently do. (I think I’m not the only one who does this.) It was in some ways simpler to think, “Nothing about my fitness routine is going to change for at least twelve weeks. All I need to do is work on my knee.”

Persistent questions

When I first started using the crutches, I was flabbergasted by some people who would ask about my injury very persistently. (“What happened? What kind of fracture? When’s your surgery? Where’d you have it? Do you have hardware?”) Then I realized that everyone with persistent questions had had knee surgery themselves. After realizing that, those people became very easy to deal with: I’d just ask them about their own surgeries and sit back while they told me all about it. Better yet, almost everyone I talked to had recovered fully and was now doing great, which was reassuring to hear.

If you’re on crutches, especially if you’re wearing the distinctive knee brace, the best piece of advice I can give you is this fact. Those with persistent questions have had their own surgeries. They would love to tell you about it.

In case you’re wondering – now that I’m recovered, I most certainly am one of those people. However, I always lead with my own surgery before asking any questions. :)

If you’re on crutches, I also recommend this marvel of human engineering and velcro, this detachable shower head, a travel coffee mug, a stylish backpack, and as much stubbornness as you can muster.

Talks

Notwithstanding the broken leg, I gave four conference talks in 2015. Two were at PyCon North America in Montreal while on crutches. One was at !!con in NYC, near the very end of the crutches era, and the fourth was at Kiwi PyCon in Christchurch, New Zealand, where I was fully ambulatory.

Exploring is Never Boring: Understanding CPython without reading the code

Bytes in the Machine: Inside the CPython interpreter

I gave these two talks at PyCon in Montreal back-to-back. “I think this’ll actually be easier for you,” said the organizer, and that turned out to be true, but not for reasons either of us predicted. As it turned out, the hardest part of PyCon 2015 for me was getting around on crutches, so the less of that I had to do the better off I was.

I was pleased with how these talks went, especially “Bytes in the Machine”, which I’d been working on in one form or another for more than a year. I originally proposed this talk for PyCon 2014, when it was rejected; I was able to propose and then give a substantially better version of it at PyCon 2015. One person told me that they’d never wanted to dig into CPython before and now they did, which was exactly what I was hoping for.

PyCon 2015 on crutches took an enormous amount of energy. The organizers were all very kind and helpful, but the convention center in Montreal was simply very large, and a ton of moving was required. My thanks to the organizers for their accommodations, and to the friends and strangers I pressed into carrying my lunches. (I also offer my apologies to anyone near enough to smell me on Friday, before my wayward luggage arrived. Crutching around is regrettably strenuous and my clean clothes were stuck in Chicago.)

This photo of my talk from Anja Boskovic shows me in the same body position I was in for almost three months: sitting on a chair with one leg up. I find this position to be quite masculine – asymmetric and unapologetically taking up space. There aren’t a lot of perks to breaking a leg, but I enjoyed taking up a lot of space while having a perfect excuse for doing so. There’s something delightful about having your foot up on the table during a meeting with someone who outranks you, or while presenting a talk at a conference. Interestingly, not everyone who attended the talk realized that I was using crutches during the conference, which means they can’t have properly attributed my body language to my injury. 1

Video and slides for Exploring is Never Boring

Video and slides for Bytes in the Machine

If you’re catching up now and you prefer written material, consider reading the chapter version of this talk instead of watching the video.

Limitless and Recursion-free recursions limits!

!!con (“bangbangcon”) is a conference about “the joy, excitement, and surprise in programming.” This was my second year speaking there, and it’s consistently one of my favorite conferences. I described the CfP as “an invitation to meditate on your delight,” and the whole conference felt like that. It can sometimes be challenging and difficult and outright terrible in this industry, and there’s a lot of hard work to do, but it’s nice to spend a couple of days learning about the amazing, fascinating, and weird world we live in.

This year there were talks on how wifi keeps getting faster, how to program a knitting machine, lightpainting with robots, making a cell phone, quines, and roller derby. It was a truly delightful lineup, and I’m honored to have been a part of it.

My talk covered how to hit the recursion limit in Python without doing any recursion and how to implement the world’s jankiest tail call optimization in Python. This talk features the following:

  • Me saying “Any day we can segfault Python is a good day in my book.”

  • Me saying “Remember, our beef today is with the compiler.”

  • An audible “Oh no” from the audience on seeing a slide with Python code and GOTOs

Unfortunately, the sound quality’s not great on this video – sorry about that.

The talented Danielle Sucher did four ten-minute sketches of speakers during four ten-minute talks, and I loved the result. (That’s me in the green.)

Video and slides for Limitless and Recursion-free recursion limits!

Learning Strategies for Programmers

My final talk this year was a keynote at the Kiwi PyCon conference in Christchurch, New Zealand. I loved this trip. The organizers were hospitable from start to finish, and I welcome any excuse to visit New Zealand. Marek Kuziel was kind enough to meet me at the airport (and kind enough to depart before I attempted to drive my rental car on the left side of the road).

I also want to give Marek credit for effective enforcement of a Code of Conduct. On one occasion in particular he gently redirected some mildly-dirty humor before anyone got uncomfortable. This can be tricky to do and he did it well.

I wrote a blog post in October that captures the best parts of this talk. There is also video available and slides.

Architecture of Open Source Applications chapter

After many rounds of writing and procrastinating, I finished and published my chapter for the Architecture of Open Source Applications 4th edition, on Byterun, a Python interpreter written in Python with Ned Batchelder. This version of the AOSA book is themed “500 lines or less,” and it features real software that does something significant in under 500 lines. It was a fun challenge to trim Byterun down to that size, and an even better challenge to try to explain the resulting code clearly.

This chapter covers similar ground to “Bytes in the Machine,” but at a somewhat more relaxed pace.

My thanks to the editors for their patience and grit in this process, especially Mike DiBernardo and the talented copy editor Amy Brown.

A Python Interpreter written in Python

Dropbox

I’ve now been at Dropbox for slightly over a year. Most of what I’m most excited about I can’t talk about publicly. What I can say is that I feel like I’ve matured as an engineer. This means getting better at skills like living with my decisions, thinking farther ahead, architecting software, gathering consensus, getting and giving technical input, and other skills beyond pure programming.

I’m on the desktop client team, and desktop software in particular presents interesting challenges that I hadn’t thought much about before I joined Dropbox. For example, you generally can’t roll back a desktop release – once it’s out there, it’s out there. It’s also nontrivial to make sure we can get enough data to debug when something goes wrong. With a badly-behaving server, you might be able to ssh in and poke around. This is obviously not possible with someone else’s desktop.

2016

I’m hopeful that 2016 will bring as many opportunities for growth and fewer broken bones.

A few plugs: If you’re a new or experienced programmer, consider spending some time at the Recurse Center. If you’re interested in joining Dropbox, please drop me a line on twitter.


  1. One person even congratulated Jessica McKellar for my talk, thinking she was me. I was thrilled to be mistaken for her.

Effective Learning Strategies for Programmers

In early September I gave a keynote at Kiwi PyCon in New Zealand on effective learning for programmers. There were two pieces to the talk: one about mindset, and one about particular strategies we can use. The text below is an aspirational and lightly edited transcript of the mindset piece of that talk. There’s also a video available if you’d like to see the strategies piece.

Recurse Center

Before I joined Dropbox last year, I spent two years working at a company in NYC called the Recurse Center. The Recurse Center is like a writers’ retreat for programmers. Participants spend 3 months working on whatever is most interesting to them. So someone who’d been writing Java for ten years might come to RC to learn a new language like Clojure, or someone who just graduated with a CS degree might come work on their web development skills, or someone who’d been learning programming in their spare time might come to turbo-charge their learning. There’s almost no structure to the program – no deadlines, no assignments, no teaching. It’s an experiment in unstructured learning for adults.

My role as a facilitator was to help people make the most of that disorienting amount of freedom that they had at RC. People who come out of traditional educational experiences or traditional jobs very often don’t know what to do with that. So I’d help them with goal-setting and help them make the most of the experience. One of the things we thought a lot about was how to have the most effective learning experience possible for programmers. Today I’ll talk about some of the research into how to be an effective learner, and how we can apply that research to our daily lives as programmers and engineers.

What to get out of this post

Take a minute and consider what you’d like to get out of this post. You might want to learn something new about how to be as efficient and effective in your job as possible. You might want to hear about how you can be a better teacher or mentor to junior engineers. Or you might want to hear about how you can make institutional change in your organization to set up a better environment for these kinds of things.

All of these are useful goals, and I’ll touch on material relevant to all of them. However, I want to challenge you to consider the strategies mostly for yourself. When I hear about these strategies, very often it seems obvious to me that other people should be following them, but not necessarily obvious that I myself should. I’ll come back to that tension a little bit later on.

Growth mindset: Carol Dweck

Let’s talk about the first key to effective learning. The sociologist Carol Dweck has done a ton of interesting research about how people think about intelligence. She’s found that there are two different frameworks for thinking about intelligence. The first, which she calls the fixed mindset, holds that intelligence is a fixed trait, and people can’t change how much of it they have. The other mindset is a growth mindset. Under a growth mindset, people believe that intelligence is malleable and can increase with effort.

Dweck found that a person’s theory of intelligence – whether they hold a fixed or growth mindset – can significantly influence the way they select tasks to work on, the way they respond to challenges, their cognitive performance, and even their honesty. I’m going to run through a couple of the most interesting results from her work here.

These mindsets cause differences in effort

The first interesting result is that this framing impacts how people view effort. If you have a fixed mindset – you believe that people are either smart or they’re not, and they can’t really change that – then you also tend to believe that if you’re good at something, it should be easy for you, and if something is hard for you than you must not be good at it. That’s a fixed-mindset view. People who have a growth mindset believe that you need to exert effort and work hard at something to become better at it.

Several studies found that people with a fixed mindset can be reluctant to really exert effort, because they believe it means they’re not good at the thing they’re working hard on. Dweck notes, “It would be hard to maintain confidence in your ability if every time a task requires effort, your intelligence is called into question.”

“Praise that backfires”

The second interesting result is probably the most famous. Dweck and her collaborators showed that giving students subtly different kinds of praise significantly impacted their performance.

In this study, Dweck and her collaborators gave a students a series of problems. After the first set of problems, all of the students did pretty well. Then half of the students were told “Wow, you did really well on those problems – you must be very smart.” and the other “Wow, you did really well on those problems – you must have worked very hard.” Then they got a second set of problems, much harder, where everyone did badly. Then they got a third set of problems that were like the first set – back to the easier level.

Here, they’re creating a fixed mindset in the first group of students (your performance shows that you’re smart) and a growth mindset in the second set of students (your effort drives your success).

They found a bunch of interesting things from this. The first aspect of the experiment is that in between the first and second problem sets they asked the students if they’d like to do an easier exercise or a harder one next. (In practice, everyone got the harder set next.) Dweck et al. wanted to see if there would be a difference between the students who got different kinds of praise. And sure enough, there was: 90% of the students praised for effort chose to do a harder set of problems next, compared to only a third of the group praised for intelligence. The kids praised for effort were much more interested in a challenge.

The second thing that they looked at was how student performed on the third set of problems. They found that students who’d been praised for their intelligence did significantly worse on the third problem set than they had on the first, but students who’d been praised for effort did slightly better. Students who got intelligence praise weren’t able to recover effectively from hitting a wall on the second set of problems, while students who got effort praise could bounce back.

After this, they had the students write letters to pen pals about the study, saying “We did this study at school, and here’s the score that I got.” They found that almost half of the students praised for intelligence lied about their scores, and almost no one who was praised for working hard was dishonest.

So there are three implications here: a growth mindset made students more likely to choose a challenge instead of something easy, more likely to persist after a setback, and more honest about their performance, compared to the students with a fixed mindset.

What’s fascinating about this is how subtle the difference in praise is. Being told you’re smart leads to all of these attempts to preserve the appearance of smartness, by only doing easy things you know you can perform well on and by hiding your poor performance. Being told that you work hard leads to attempts to preserve the appearance of working hard – and the best way to do that is to actually work hard.

Response to confusion

Another study looked at what happened when students faced a temporary period of confusion. Dweck and her collaborators designed a short course on psychology to give to elementary school students. The course was a booklet on psychology followed by a quiz. Some of the booklets had a confusing passage in them, and others didn’t. The confusing part wasn’t on the quiz, so students could master the material if they just completely ignored the confusing bit. The researchers wanted to see whether students would be able to recover from being totally bewildered in the middle of this booklet.

They found that students with a growth mindset mastered the material about 70% of the time, regardless of whether there was a confusing passage in it. Among students with a fixed mindset, if they read the booklet without the confusing passage, again about 70% of them mastered the material. But the fixed-mindset students who encountered the confusing passage saw their mastery drop to 30%. Students with a fixed mindset were pretty bad at recovering from being confused.

“How can one best describe the nature of people who will most of all be that way which will make the imitating of others happen most often? Is it that these are the people we want to be like because they are fine or is it that these are the people we want to be liked by?”

I wanted to put up a section of the confusing passage because this really resonated with me. Hands up if you’ve ever started using a new tool and run into documentation that sounded like this. [Roughly 100% of hands go up.] It happens all the time – you get domain experts writing docs aimed at beginners, or out-of-date docs, or some other issue. It’s a critical skill for programmers to push past this kind of confusion and be able to successfully retain the rest of the information in the document we’re reading.

Programmers need a growth mindset

Programmers need a growth mindset! Key skills for programmers – like responding to confusion, recovering from setbacks, and being willing to take on new challenges – are all much easier with a growth mindset, and much harder with a fixed mindset.

Does anyone believe in a fixed mindset?

Now sometimes when people hear this idea of the fixed mindset, it almost sounds like a straw man. Like, does anyone in the tech industry actually believe this? I think that absolutely a fixed mindset is a widespread belief. Here are a couple of examples.

10x engineers

Start with the idea of the 10x engineer. This is the idea that some engineers are an order of magnitude more effective than others, for some definition of effective. And there’s lots of critiques of this framing, but we’ll set that aside for a moment. If you believe in the idea of the 10x engineer, do you think that engineer was born as a super effective engineer? Or did they get to be 10x one x at a time?

I think very often in the popular framing of this, the 10x engineer is set up on a pedestal, as someone that other people cannot become. Very often this is approached from a fixed-mindset perspective.

Hero worship

Another case where we see evidence of a fixed mindset is with hero worship. So Julie Pagano did a great talk at PyCon 2014 about impostor syndrome, and one of her suggestions for a way to combat impostor syndrome was “kill your heroes.” Don’t put other programmers on a pedestal, don’t say “that person is so different from me.” Fixed/growth mindset is a really useful framing for this too. If you have programming heroes, do you consider them to be totally different from you? Could you become more like the kind of person you admire? If you don’t think so, that’s some evidence of a fixed mindset.

So I’d argue that yes, a fixed mindset is quite prevalent in the tech industry.

Can you change a fixed mindset? Heck yes

Hopefully by now you’re convinced that a growth mindset is better for you than a fixed mindset. So the next question is: is this malleable? Can you take a fixed mindset and turn it into a growth mindset? And the answer is heck yes, you absolutely can change a fixed mindset into a growth one.

In fact, in many of Dweck’s studies they experimentally induce a fixed or growth mindset, often in really subtle ways. The praise study is one example: one sentence of praise changes the students’ behavior. In other studies they have students read a paragraph about a famous person’s success, and at the end it says “because they worked very hard,” or “because it was in their DNA.” This is absolutely a malleable thing.

So how do you change a fixed mindset? Sometimes the challenge is mostly in actually identifying the fixed mindset, and once you hear yourself say the words, “I could never learn physics,” it’s already obvious that that’s probably not true. But other times it’s harder to root out the fixed mindset. So here are a couple of flags you can use to identify fixed mindsets so you can root them out.

How do you identify a fixed mindset?

“I am ..”

“Some people are just …”

If you’re on the lookout for places where your mindset might be fixed, you should be listening for sentences that start like this. Things like “I’ve never been good at CSS” or “I’m not a people person” or “Some programmers are just faster than others.” Anything that starts with “I am …” is a candidate. The word “just” is often present.

Now, obviously, you can say sentences with “I am” that aren’t indicators of a fixed mindset. Instead, the point here is to treat sentences like this as a little bit of a yellow flag for yourself, to notice and then to examine your mindset more closely.

Just as an aside, the example “I’m not a people person” is supported by the research – Dweck and collaborators did a study on making friends and social situations, and this research holds there too. [See the Q&A for more about this.]

How do you change a fixed mindset?

Reframe praise & success

Ok, so once you’ve identified a fixed mindset, how can you go about changing it? Here are four strategies.

The first is to reframe praise and success. By reframe praise I mean that when you get the wrong kind of compliments, turn them into growth-mindset compliments. So if someone says “wow, great job on that project, you’re so smart,” translate it to “yeah, it was great, I worked really hard on that project.” You don’t necessarily have to do this out loud! But this reframing reinforces for yourself that you gain mastery by seeking out challenges and by exerting effort.

And you can use the same techniques for successes and accomplishments. When something goes well, don’t think, “Of course that went well because I’m awesome.” Instead think, “I used an effective strategy on that project! I should do that more often.”

Reframe failure

Of course the flip side of this dynamic is also really effective. A huge part of a fixed or growth mindset is how you respond to failure. What’s your self-talk when you face a setback or don’t get what you wanted? If you’re saying, “Maybe I’m not cut out for this job after all,” treat that as a red flag. Instead, ask what you learned from your unsuccessful attempt or what strategies you could have used instead. It sounds cheesy, but it really works.

Celebrate challenges

The third way that you can change a fixed mindset is to celebrate challenges. How do you respond when you have to struggle? Try explicitly celebrating. This is something that I was really consistent about when I was facilitating at the Recurse Center. Someone would sit down next to me and say, “[sigh] I think I’ve got a weird Python bug,” and I’d say, “Awesome, I love weird Python bugs!” First of all, this is definitely true – if you have a weird Python bug, let’s discuss – but more importantly, it emphasized to the participant that finding something where they struggled an accomplishment, it was intentional, and it was a good thing for them to have done that day.

As I mentioned, at the Recurse Center there are no deadlines and no assignments, so this attitude is pretty much free. I’d say, “You get to spend a day chasing down this weird bug in Flask, how exciting!” Now, at Dropbox, where we have a product to ship, and deadlines, and users, I’m not always uniformly delighted about spending a day on a weird bug. So I’m sympathetic to the reality of the world where there are deadlines. However, if I have a bug to fix, I have to fix it, and being grumbly about the existence of the bug isn’t going to help me fix it faster. I think that even in a world where deadlines loom, you can still apply this attitude.

Ask about processes

The last strategy for changing a fixed mindset is to ask about processes. Like many of you, I work with some great engineers. Sometimes, I’ll try to fix a tricky bug and won’t be able to, and then one of them will be able to fix it right away. In these situations I’ve tried to be really disciplined about asking how they did it. Particularly when I was new at Dropbox, the answers would be really illuminating. Sometimes the information had come from a source I didn’t know existed. Now that I’ve been there longer, it’s usually a technique or strategy difference, or a detail about why my strategy had not succeeded.

This is a much more useful strategy in the long term than saying “Oh, of course, that person got the bug because they are a wizard.”

Confidence & imposter syndrome

Dweck’s research is really interesting in the context of the discussion around impostor syndrome. Impostor syndrome is the feeling that you’re secretly an unqualified fraud who will be uncovered any second now. Hands up if you’ve ever felt impostor syndrome in your career? [80% of hands in the room go up.] Yeah, that’s lots of you, and I definitely have as well. And it sucks! It’s so painful, and it’s really bad for your career, because you’re less likely to take chances or to look for new opportunities to grow if you’re worrying about getting fired from the job you already have.

The proposed solutions for impostor syndrome very often center around confidence. Like, “Oh, if you feel like you’re not qualified for the job you already have, you should be more confident, and then you’ll be fine.” This sometimes is as simple as, “Don’t feel that way,” which is not very helpful as advice goes. But even when it’s more nuanced than that, there’s a focus on confidence and past accomplishments.

Confidence doesn’t help you respond to challenges

Henderson & Dweck, 1990

But here’s the catch. Dweck’s research shows that confidence doesn’t actually predict your success at responding to new challenges or recovering from setbacks.

Henderson and Dweck did a study of students moving from elementary school to junior high in the U.S. They asked the students to assess their confidence when they were still in the younger grade, and they also measured whether the students held fixed or growth mindsets. Then they tracked the students’ academic performance in junior high.

They found that confident students with a fixed mindset suffered academically. By contrast, students with a growth mindset tended to thrive academically, regardless of whether their confidence was high or low. Confidence wasn’t a useful predictor of success at all.

Now, there’s lots of other research that shows confidence is correlated with success. Dweck argues that confidence is a good predictor of how well you can do things you’re already doing, but it’s not a good predictor of how you respond to new challenges and how you feel about failure.

The second, related point that Dweck has discovered is that a history of success also doesn’t impact how you respond to challenges and how you deal with failure.

So past successes don’t predict your response to new setbacks and failures, and your confidence level also doesn’t predict your response to failure. The thing that is a good predictor of resilience in the face of failure is having a growth mindset.

Break the framework

This is hugely exciting to me and I think it doesn’t come up nearly often enough in the discussions around impostor syndrome. This gives us a new and more useful framework for combating impostor syndrome. Basically, if you’re holding a fixed mindset, you’re going to be really stressed and afraid at any moment that you have to struggle. We’re programmers, so it’s mostly struggle, right? It’s struggle all the time. With a growth mindset, you can enjoy the struggling and enjoy working on something that’s really hard.

And guess what? When your identity isn’t being threatened by a particularly tricky bug, it’s a lot easier to stay focused on the bug. You’re not worried about also getting fired and being a fraud, so you can free up those cognitive resources to focus on the task at hand.

So, again: if you believe, for example, that “some people just aren’t cut out for programming,” you can spend a ton of time & energy trying to find evidence and validation and reassurance that you are one of the people who can make it. Instead, upend this framework. Break the idea of fixed levels of talent and move to the idea that everyone can increase their skill by exerting effort.

Self-theories: Their role in motivation, personality, and development

Having a growth mindset makes you more resilient to failure, makes it easier to exert effort, makes you more likely to take on challenges, all things that are very useful to programmers.

If you’d like to dig more into the details of this research, and also see some of the findings that I didn’t have time to cover today, I highly recommend a book Dweck wrote called Self-theories. Self-theories is a collection of short essays that summarize many major points of her research. It’s got detail about the studies but is accessible to a lay reader. She’s also got a book called Mindset that’s written for a pop-science audience, but if you want a little more nuance and detail about the particular studies, Self-theories is the way to go.

Q & A

A selection from the Q&A:

Q: Is there any research in growth and fixed mindsets at the team-level, and how teams approach problems?

A: I’m not aware of any, but that’s a fascinating question. I’d love to see that research if it exists.

Q: I read Mindset, and I’m a father to twin girls. I found that these strategies really changed their resilience and their approach to problem solving.

A: Yeah, this research is kind of terrifying. Like, do you tell your children that they’re smart? You’re ruining them! I didn’t have a chance to talk about this, but there is some research in this book about gender discrepancies, and findings that high-achieving girls are more likely to have a fixed mindset and less likely to risk failure when they hit something hard. Many of the women in the room in particular can probably relate to this.

Q: Is this binary, or a gray scale?

A: I think it probably is a spectrum, yes. For this research it’s classified into a binary model. I’m not precisely sure where the line gets drawn. And some of these cases with experimental induction of a fixed or growth mindset, if someone has one mindset going in and has the other induced, they’ll probably end up in a moderate place.

Q: Is it possible to have a fixed mindset in one area and a growth mindset in another?

A: Absolutely. One that is common for programmers is to have a growth mindset about programming and a fixed mindset about social skills.

Q (from a CS lecturer/TA): For our new students, is there a way we can create a growth mindset in the students? A lot of people come in from school with a fixed one, and it can be damaging in those early courses.

A: If you’re a lecturer or have a chance to get up in front of the auditorium, you can say it explicitly: “Programming is a skill that you can get better at with effort,” and even though it doesn’t sound like it’d convince people, the research shows that it does make a difference.

The other thing that’s really interesting is a study on a values exercise. This shows that having women in particular write down their values before they enter into a context where they’d experience stereotype threat can significantly improve their performance. The basic idea here is if you’re identifying as a programmer, and your programmer identity is threatened, that’s very painful and difficult. But if you have these other things that you value about yourself, then that mitigates the threat. The results are really dramatic for people who are marginalized in tech (and it doesn’t hurt those who aren’t). For more, see this worksheet by Leigh Honeywell.

Q: So this is nature versus nurture all over again, isn’t it?

A: I wouldn’t characterize it that way, in part because I think both of those remove agency from the individual. Your mindset is something that you can control to a significant extent. That’s why I think it’s so important to think about this research from the context of ourselves, and not just our children or our students.

Q: It’s easy to think of lots of ways to apply this in programming, but can you talk more about ways to apply this in social situations?

A: Sure. In the study covered in a Self-theories, Dweck had children write letters applying to the pen pal club (which was a real pen pal club – they did eventually match people up). Then all the children got rejected from the pen pal club. [Audience laughter] Before writing the letter, they’d told half the children, “This is to see how good you are at making friends,” and the other half, “This is a chance to practice and improve your ways of making friends.” The children who heard the fixed-mindset framing sometimes wrote the same letter or sometimes wrote a shorter and less detailed letter. The kids who got the growth framing were much more likely to write longer things, to be more inviting, to say, “Oh, I love talking to you” even though it’s a first letter to a pen pal. [Audience makes sympathetic noises.] Yeah, throughout this book Dweck and her collaborators were pretty careful to not traumatize any students, not to leave them thinking that they’re stupid or bad at making friends.

If you’re interested in particular strategies for social situations, I highly recommend the blog Captain Awkward. Captain Awkward has some constructions of social challenges, like “I’ll go to a party and talk to three people, and award myself ten points for each person I talk to and learn a fact about.” There’s a lot of interesting stuff on the internet about strategies for coping with social anxiety that I think you can apply whether or not that’s something that you struggle with.

Thanks

My thanks to Maggie Zhou, Amy Hanlon, Alyssa Frazee, and Julia Evans for feedback on early versions of this talk.

Thanks to Sasha Laundy, who invited people to consider what they wanted to get out of her PyCon talk on giving and getting help, and inspired me to use the same construction.

Thanks to the Kiwi PyCon organizers, particularly Marek Kuziel, for hosting me.

PS1 for Python3

I spend a lot of time flipping back and forth between Python 2.x and 3.x: I use different versions for different projects, talk to people about different versions, explore differences between the two, and paste the output of REPL sessions into chat windows. I also like to keep long-running REPL sessions. These two activities in combination became quite confusing, and I’d often forget which version I was using.

1
2
3
4
5
6
>>> print some_var
  File "<stdin>", line 1
    print some_var
                 ^
SyntaxError: invalid syntax
>>> # *swears*

After the hundredth time I made this mistake, I decided to modify my prompt to make it always obvious which version was which, even in long-running REPL sessions. You can do this by creating a file to be run when Python starts up. Add this line to your .bashrc:

1
export PYTHONSTARTUP=~/mystartupscript.py

Then in mystartupscript.py:

mystartupscript.py
1
2
3
4
import sys
if sys.version_info.major == 3:
    sys.ps1 = "PY3 >>> "
    sys.ps2 = "PY3 ... "

This makes it obvious when you’re about to slip up:

1
2
3
PY3 >>> for value in giant_collection:
PY3 ...     print(value)
PY3 ...

I’ve also add this line to mystartupscript.py to bite the bullet and start using print as a function everywhere:

mystartupscript.py
1
from __future__ import print_function

This has no effect in Python3.x, but will move 2.x to the new syntax.

I’m Joining Dropbox

Big news for me: I’m leaving Hacker School and going to work for Dropbox in San Francisco, joining Jessica McKellar’s team. I met Jessica when she was part of the first round of residents at Hacker School in fall 2012, and I’ve had tremendous respect for her work and leadership ever since. Dropbox has an impressive crop of Pythonistas (including Guido van Rossum, of course), and I couldn’t be more excited to join. I’ll be moving to San Francisco at the end of October. If you have recommendations for people to meet, places to go, or things to do, let me know!

This means I’m leaving Hacker School, after more than two years facilitating. My last day will be October 24th. I love Hacker School, and I know I’m going to miss it. Hacker School is entirely responsible for the fact that I’m a programmer at all. I was working in a finance job and contemplating new careers when my brother saw this post about Hacker School’s experiment with Etsy to get more qualified women into the summer 2012 batch. I read the post and the thoughtful, welcoming FAQ, then went home and picked up a Python book. Two months later, I started Hacker School.

Hacker School is about becoming a better programmer, and there’s no doubt that it’s worked for me. For two years, I’ve had total freedom to chase down whatever weird thing catches my eye; I’ve worked with creative, hilarious, brilliant Hacker Schoolers and residents on a dizzying variety of projects; and I’ve been delighted to help build a more inclusive environment at Hacker School, although there’s always more work to be done. (If you’re a curious, sharp, and self-directed programmer, I can’t recommend Hacker School enough.)

I’m thankful that leaving my job at Hacker School doesn’t mean leaving the Hacker School community. I’m trading in my faculty status and becoming one of hundreds of alumni around the world. I’ll still be on Zulip, Community, and everywhere else Hacker Schoolers can be found, and I’ll still have my cape. I may be leaving, but I’ll never graduate.

Debugging With Pstree

I hit a very fun bug yesterday while trying to run a script that sends emails to certain subsets of Hacker Schoolers. When I tried to test the script locally, I discovered that one of the tables of the database, Batch, was missing from my local version. After briefly panicking and making sure that the actual site was still up, I could dig in.

It turns out that my local version of psql was way out of date, and as of a few days ago we’d started using a data type that wasn’t present in my old version. Because of that, creating that particular table failed when I pulled from the production database the night before. The failure was logged, but the output is so verbose that I didn’t notice the problem. Both the diagnosis and the fix here were easy – I went back and read the logs, googled the data type that was raising an error, and then upgraded Postgres.app and psql. That’s when the real trouble started.

The new version of Postgres.app was placed in a new spot on the $PATH, as you’d expect, and the upgrade prompted me to change my .bashrc, which I did. But the rake tasks we use to manage local copies of the database errored out with this message:

1
2
$ pg_restore --verbose --clean --no-acl --no-owner -h localhost -U `whoami` -d hackerschool latest.dump
sh: pg_restore: command not found

This was pretty clearly a $PATH problem. I tried the usual things first, like sourcing my .bashrc in the terminal I was using, closing the terminal and opening a new one, etc. None of that worked.

One thing that jumped out to me was the sh in the error message. That was an indicator that rake wasn’t using bash as a shell – it was using sh – which means my .bashrc wasn’t setting the environment. Reading the rake task showed that it was a thin wrapper around lots of system calls via Ruby’s system("cmd here"). I added the line system("echo $PATH") and verified that the new location of pg_restore wasn’t in it.

At this point I found I had lots of questions about the execution context of the rake task. Since I was making system calls and could easily edit the rakefile, I added in the line system("sh") to drop me into a shell mid-execution. This turned out to be an efficient way to figure out what was going on (and made me feel like a badass hacker).

From within in that shell, I could do $$ to get that process’s PID, then repeatedly do ps -ef | grep [PID] to find the parent process.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
sh-3.2$ $$
sh: 34652: command not found
sh-3.2$ ps -ef | grep 34652
  501 34652 34639   0  4:18PM ??         0:00.04 sh
    0 34881 34652   0  4:26PM ??         0:00.01 ps -ef
  501 34882 34652   0  4:26PM ??         0:00.01 grep 34652
sh-3.2$ ps -ef | grep 34639
  501 34639  2914   0  4:18PM ??         0:00.41 rake db:drop db:create db:pull
  501 34652 34639   0  4:18PM ??         0:00.04 sh
  501 34885 34652   0  4:28PM ??         0:00.00 grep 34639
sh-3.2$ ps -ef | grep 2914
  501  2914  2913   0 10Sep14 ??        27:11.72 spring app    | hackerschool | started 244 hours ago | development mode
  501 34639  2914   0  4:18PM ??         0:00.41 rake db:drop db:create db:pull
  501 34889 34652   0  4:28PM ??         0:00.01 grep 2914
sh-3.2$ ps -ef | grep 2913
  501  2914  2913   0 10Sep14 ??        27:11.98 spring app    | hackerschool | started 244 hours ago | development mode
  501 34892 34652   0  4:29PM ??         0:00.00 grep 2913
  501  2913     1   0 10Sep14 ttys001    0:00.94 spring server | hackerschool | started 244 hours ago

Aha! The parent process of the rake task I was running is the spring server, which starts on boot – several days ago, at the time – and doesn’t have the new and updated $PATH information.1 A kick to the spring server (with kill 2913) forced the server process to restart with the new environment.

It turns out there’s a handy utility called pstree2 (brew installable) to visualize the tree of processes. This would have saved me a couple of steps of grepping. For example:

1
2
3
4
5
hackerschool [master] $ pstree -p 35351
-+= 00001 root /sbin/launchd
 \-+- 35129 afk spring server | hackerschool | started 25 hours ago
   \-+= 35130 afk spring app    | hackerschool | started 25 hours ago | development mode
     \--- 35351 afk rails_console

This bug and some related ones have gotten me more interested in operating systems, and I’ve started reading the book Operating Systems: Three Easy Pieces. I’m only a few chapters in, but so far it’s readable, clear, and entertaining. I look forward to building up my mental model of processes and environments as I keep reading it.


  1. We can tell it (probably) starts on boot because the parent process ID is 1. This means that rebooting my computer would have solved the problem.

  2. Thanks to Paul Tag for the pointer to pstree.

Rejected PyCon Proposals

“All accepted proposals are alike, but each rejected proposal is rejected in its own way” – Tolstoy, if he were on the PyCon talk review committee

I’m building a collection of old PyCon talk proposals, particularly rejected ones. I think rejected proposals are more interesting than accepted ones, for a couple of reasons:

See examples of anti-patterns

Flipping through these proposals, you can see concrete examples of the talk committee’s suggestions for what to avoid. There is an example of a “state of our project” talk and one of “here’s some code I hope to have written by the time the conference rolls around.”

“I can do better than that”

Being a great or famous programmer doesn’t mean you’ll give a great talk or submit a great proposal. You’ll notice that you can write a better proposal than some of the ones from people you’ve heard of. (This fits with the Kill your heroes theme from Julie Pagano’s great talk on impostor syndrome at PyCon 2014.)

Empathize with the talk committee

Any application is an exercise in empathy – you need to imagine what the people who will be reading your submission are thinking. What do they care about? Where are they coming from? You can read past proposals and decide if you’d make the same decision the committee did. When submitters have shared the feedback they received, you can see exactly what the committee members thought. This helps you write a proposal that will address their concerns.

The deadline for submitting a proposal is Monday, September 15th. I encourage you to browse through the collection of past proposals to get inspiration or to improve your proposal. Once you’ve submitted a proposal, please add it to the collection!

Getting Started With Python Internals

I talk to a lot of people at Hacker School and elsewhere who have been programming Python for some time and want to get a better mental model of what’s happening under the hood. The words “really” or “why” often features in these questions – “What’s really happening when I write a list comprehension?” “Why are function calls considered expensive?” If you’ve seen any of the rest of this blog, you know I love digging around in Python internals, and I’m always happy to share that with others.

Why do this?

First off, I reject the idea that you have to understand the internals of Python to be a good Python developer. Many of the things you’ll learn about Python won’t help you write better Python. The “under the hood” construction is specious, too – why stop at Python internals? Do you also need to know C perfectly, and the C compiler, and the assembly, and …

That said, I think you should dig around in Python – it sometimes will help you write better Python, you’ll be more prepared to contribute to Python if you want to, and most importantly, it’s often really interesting and fun.

Setup

Follow the instructions in the Python dev guide under “Version Control Setup” and “Getting the Source Code”. You now have a Python that you can play with.

Strategies

1. Naturalism

Peter Seibel has a great blog post about reading code. He thinks that “reading” isn’t how most people interact with code – instead, they dissect it. From the post:

But then it hit me. Code is not literature and we are not readers. Rather, interesting pieces of code are specimens and we are naturalists. So instead of trying to pick out a piece of code and reading it and then discussing it like a bunch of Comp Lit. grad students, I think a better model is for one of us to play the role of a 19th century naturalist returning from a trip to some exotic island to present to the local scientific society a discussion of the crazy beetles they found: “Look at the antenna on this monster! They look incredibly ungainly but the male of the species can use these to kill small frogs in whose carcass the females lay their eggs.”

The point of such a presentation is to take a piece of code that the presenter has understood deeply and for them to help the audience understand the core ideas by pointing them out amidst the layers of evolutionary detritus (a.k.a. kluges) that are also part of almost all code. One reasonable approach might be to show the real code and then to show a stripped down reimplementation of just the key bits, kind of like a biologist staining a specimen to make various features easier to discern.

2. Science!

I’m a big fan of hypothesis-driven debugging, and that also applies in exploring Python. I think you should not just sit down and read CPython at random. Instead, enter the codebase with (1) a question and (2) a hypothesis. For each thing you’re wondering about, make a guess for how it might be implemented, then try to confirm or refute your guess.

3. Guided tours

Follow a step-by-step guide to changing something in Python. I like Amy Hanlon’s post on changing a keyword in Python and Eli Bendersky’s on adding a keyword.

4. Reading recommendations.

I don’t think you should sit down and read CPython at random, but I do have some suggestions for my favorite modules that are implemented in Python. I think you should read the implementation of

1. timeit in Lib/timeit.py
2. namedtuple in Lib/collections.py.

If you have a favorite module implemented in Python, tweet at me and I’ll add it to this list.

5. Blog & talk

Did you learn something interesting? Write it up and share it, or present at your local meetup group! It’s easy to feel like everyone else already knows everything you know, but trust me, they don’t.

6. Rewrite

Try to write your own implementation of timeit or namedtuple before reading the implementation. Or read a bit of C and rewrite the logic in Python. Byterun is an example of the latter strategy.

Tools

I sometimes hesitate to recommend tooling because it’s so easy to get stuck on installation problems. If you’re having trouble installing something, get assistance (IRC, StackOverflow, a Meetup, etc.) These problems are challenging to fix if you haven’t seen them before, but often straightforward once you know what you’re looking for. If you don’t believe me, this thread features Guido van Rossum totally misunderstanding a problem he’s having with a module that turns out to be related to upgrading to OS X Mavericks. This stuff is hard.

1. Ack

I’d been using grep in the CPython codebase, which was noticeably slow. (It’s especially slow when you forget to add the . at the end of the command and grep patiently waits on stdin, a mistake I manage to make all the time.) I started using ack a few months ago and really like it.

If you’re on a Mac and use homebrew, you can brew install ack, which takes only a few seconds. Then do ack string_youre_looking_for and you get a nicely-formatted output. I imagine you could get the same result with grep if you knew the right options to pass it, but I find ack fast and simple.

Try using ack on the text of an error message or a mysterious constant. You may be surprised how often this leads you directly to the relevant source code.

2. timeit

Timing & efficiency questions are a great place to use the “Science!” strategy. You may have a question like “Which is faster, X or Y?” For example, is it faster to do two assignment statements in a row, or do both in one tuple-unpacking assignment? I’m guessing that the tuple-unpacking will take longer because of the unpacking step. Let’s find out!

1
2
3
4
~ ⚲ python -m timeit "x = 1; y = 2"
10000000 loops, best of 3: 0.0631 usec per loop
~ ⚲ python -m timeit "x, y = 1, 2"
10000000 loops, best of 3: 0.0456 usec per loop

I’m wrong! Interesting! I wonder why that is. What if instead of unpacking a tuple, we did …

A lot of people I talk to like using IPython for timing. IPython is pip-installable, and it usually installs smoothly into a virtual environment. In the IPython REPL, you can use %timeit for timing questions. There are also other magic functions available in IPython.

1
2
3
4
5
In [1]: %timeit x = 1; y = 2;
10000000 loops, best of 3: 82.3 ns per loop

In [2]: %timeit x, y = 1, 2
10000000 loops, best of 3: 47.3 ns per loop

One caveat on timing stuff – use timeit to enhance your understanding, but unless you have real speed problems, you should write code for clarity, not for miniscule speed gains like this one.

3. Disassembling

Python compiles down to bytecode, an intermediate representation of your Python code used by the Python virtual machine. It’s sometimes enlightening and often fun to look at that bytecode using the built-in dis module.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
>>> def one():
...     x = 1
...     y = 2
...
>>> def two():
...     x, y = 1, 2
...
>>> import dis
>>> dis.dis(one)
  2           0 LOAD_CONST               1 (1)
              3 STORE_FAST               0 (x)

  3           6 LOAD_CONST               2 (2)
              9 STORE_FAST               1 (y)
             12 LOAD_CONST               0 (None)
             15 RETURN_VALUE
>>> dis.dis(two)
  2           0 LOAD_CONST               3 ((1, 2))
              3 UNPACK_SEQUENCE          2
              6 STORE_FAST               0 (x)
              9 STORE_FAST               1 (y)
             12 LOAD_CONST               0 (None)
             15 RETURN_VALUE

The implementation of the various operations are in Python/ceval.c.

4. Inspect/cinspect

You can get into the habit of trying to call inspect on anything you’re curious about to see the source code.

1
2
3
4
5
6
>>> import inspect
>>> import collections
>>> print inspect.getsource(collections.namedtuple)
def namedtuple(typename, field_names, verbose=False, rename=False):
    """Returns a new subclass of tuple with named fields.
    ...

However, inspect will only show the source code of things that are implemented in Python, which can be frustrating.

1
2
3
4
5
>>> print inspect.getsource(collections.defaultdict)
Traceback (most recent call last):
   [... snip ...]
IOError: could not find class definition
>>> :(

To get around this, Puneeth Chaganti wrote a tool called cinspect that extends inspect to work reasonably consistently with C-implemented code as well.

5. K&R

I think C is about a hundred times easier to read than it is to write, so I encourage you to read C code even if you don’t totally know what’s going on. That said, I think an afternoon spent with the first few chapters of K&R would take you pretty far. Hacking: The Art of Exploitation is another fun, if less direct, way to learn C.

Get started!

CPython is a huge codebase, and you should expect that building a mental model of it will be a long process. Download the source code now and begin poking around, spending five or ten minutes when you’re curious about something. Over time, you’ll get faster and more rigorous, and the process will get easier.

Do you have recommended strategies and tools that don’t appear here? Let me know!

The CPython Peephole Optimizer and You

Last Thursday I gave a lightning talk at Hacker School about the peephole optimizer in Python. A “peephole optimization” is a compiler optimization that looks at a small chunk of code at a time and optimizes in that little spot. This post explains one surprising side-effect of an optimization in CPython.

Writing a test coverage tool

Suppose that we’re setting out to write a test coverage tool. Python provides an easy way to trace execution using sys.settrace, so a simple version of a coverage analyzer isn’t too hard.

Our code to test is one simple function:

example.py
1
2
3
4
5
def iffer(condition):
    if condition:
        return 3
    else:
        return 10

Then we’ll write the world’s simplest testing framework:

tests.py
1
2
3
4
5
6
7
8
from example import iffer

def test_iffer():
    assert iffer(True) == 3
    assert iffer(False) == 10

def run_tests():
    test_iffer()

Now for the simplest possible coverage tool. We can pass sys.settrace any tracing function, and it’ll be called with the arguments frame, event, and arg every time an event happens in the execution. Lines of code being executed, function calls, function returns, and exceptions are all events. We’ll filter out everything but line and call events, then keep track of what line of code was executing.1 Then we run the tests while the trace function is tracing, and finally report which (non-empty lines) failed to execute.

coverage.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
import sys
import tests
import inspect

class TinyCoverage(object):
    def __init__(self, file_to_watch):
        self.source_file = file_to_watch
        self.source_code = open(file_to_watch).readlines()
        self.executed_code = []

    def trace(self, frame, event, arg):
        current_file = inspect.getframeinfo(frame).filename

        if self.source_file in current_file and \
            (event == "line" or event == "call"):

            self.executed_code.append(frame.f_lineno)

        return self.trace

    def unexecuted_code(self):
        skipped = []
        for line_num in range(1, len(self.source_code)+1):
            if line_num not in self.executed_code:
                src = self.source_code[line_num - 1]
                if src != "\n":
                    skipped.append(line_num)
        return skipped

    def report(self):
        skipped = self.unexecuted_code()
        percent_skipped = float(len(skipped)) / len(self.source_code)
        if skipped:
            print "{} line(s) did not execute ({:.0%})".format(len(skipped), percent_skipped)
            for line_num in skipped:
                print line_num, self.source_code[line_num - 1]
        else:
            print "100% coverage, go you!"

if __name__ == '__main__':
    t = TinyCoverage('example.py')
    sys.settrace(t.trace)
    tests.run_tests()
    sys.settrace(None)
    t.report()

Let’s try it. We’re pretty confident in our test coverage – there are only two branches in the code, and we’ve tested both of them.

1
2
3
peephole [master *] ⚲ python coverage.py
1 line(s) did not execute (9%)
4     else:

Why didn’t the else line execute? To answer this, we’ll run our function through the disassembler.2

1
2
3
4
5
6
7
8
9
10
11
12
13
>>> from example import iffer
>>> import dis
>>> dis.dis(iffer)
  2           0 LOAD_FAST                0 (condition)
              3 POP_JUMP_IF_FALSE       10

  3           6 LOAD_CONST               1 (3)
              9 RETURN_VALUE

  5     >>   10 LOAD_CONST               2 (10)
             13 RETURN_VALUE
             14 LOAD_CONST               0 (None)
             17 RETURN_VALUE

You don’t need to follow exactly what’s going on in this bytecode, but note that the first column is the line numbers of source code and line 4, the one containing the else, doesn’t appear. Why not? Well, there’s nothing to do with an else statement – it’s just a separator between two branches of an if statement. The second line in the disassembly, POP_JUMP_IF_FALSE 10, means that the interpreter will pop the top thing off of the virtual machine stack, jump to bytecode index ten if that thing is false, or continue with the next instruction if it’s true.

From the bytecode’s perspective, there’s no difference at all between writing this:

1
2
3
4
5
6
7
if a:
    ...
else:
    if b:
       ...
    else:
        ...

and this:

1
2
3
4
5
6
if a:
    ...
elif b:
   ...
else:
    ...

(even though the second is better style).

We’ve learned we need to special-case else statements in our code coverage tool. Since there’s no logic in them, let’s just drop lines that only contain else:. We can revise our unexecuted_code method accordingly:

coverage.py
1
2
3
4
5
6
7
8
def unexecuted_code(self):
    skipped = []
    for line_num in range(1, len(self.source_code)+1):
        if line_num not in self.executed_code:
            src = self.source_code[line_num - 1]
            if src != "\n" and "else:\n" not in src:  # Add "else" dropping
                skipped.append(line_num)
    return skipped

Then run it again:

1
2
peephole [master *] ⚲ python coverage.py
100% coverage, go you!

Success!

Complications arise

Our previous example was really simple. Let’s add a more complex one.

example.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
def iffer(condition):
    if condition:
        return 3
    else:
        return 10

def continuer():
    a = b = c = 0
    for n in range(100):
        if n % 2:
            if n % 4:
                a += 1
            continue
        else:
            b += 1
        c += 1

    return a, b, c

continuer will increment a on all odd numbers and increment b and c for all even numbers. Don’t forget to add a test:

tests.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
import sys
import inspect
from example2 import iffer, continuer

def test_iffer():
    assert iffer(True) == 3
    assert iffer(False) == 10

def test_continuer():
    assert continuer() == (50, 50, 50)

def run_tests():
    test_iffer()
    test_continuer()
1
2
3
peephole [master *] ⚲ python coverage2.py
1 line(s) did not execute (4%)
13             continue

Hmm. The test we wrote certainly did involve the continue statement – if the interpreter hadn’t skipped the bottom half of the loop, the test wouldn’t have passed. Let’s use the strategy we used before to understand what’s happening: examining the output of the disassembler.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
>>> dis.dis(continuer)
  8           0 LOAD_CONST               1 (0)
              3 DUP_TOP
              4 STORE_FAST               0 (a)
              7 DUP_TOP
              8 STORE_FAST               1 (b)
             11 STORE_FAST               2 (c)

  9          14 SETUP_LOOP              79 (to 96)
             17 LOAD_GLOBAL              0 (range)
             20 LOAD_CONST               2 (100)
             23 CALL_FUNCTION            1
             26 GET_ITER
        >>   27 FOR_ITER                65 (to 95)
             30 STORE_FAST               3 (n)

 10          33 LOAD_FAST                3 (n)
             36 LOAD_CONST               3 (2)
             39 BINARY_MODULO
             40 POP_JUMP_IF_FALSE       72

 11          43 LOAD_FAST                3 (n)
             46 LOAD_CONST               4 (4)
             49 BINARY_MODULO
             50 POP_JUMP_IF_FALSE       27

 12          53 LOAD_FAST                0 (a)
             56 LOAD_CONST               5 (1)
             59 INPLACE_ADD
             60 STORE_FAST               0 (a)
             63 JUMP_ABSOLUTE           27

 13          66 JUMP_ABSOLUTE           27
             69 JUMP_FORWARD            10 (to 82)

 15     >>   72 LOAD_FAST                1 (b)
             75 LOAD_CONST               5 (1)
             78 INPLACE_ADD
             79 STORE_FAST               1 (b)

 16     >>   82 LOAD_FAST                2 (c)
             85 LOAD_CONST               5 (1)
             88 INPLACE_ADD
             89 STORE_FAST               2 (c)
             92 JUMP_ABSOLUTE           27
        >>   95 POP_BLOCK

 18     >>   96 LOAD_FAST                0 (a)
             99 LOAD_FAST                1 (b)
            102 LOAD_FAST                2 (c)
            105 BUILD_TUPLE              3
            108 RETURN_VALUE

There’s a lot more going on here, but you don’t need to understand all of it to proceed. Here are the things we need to know to make sense of this:

  • The second column in the output is the index in the bytecode, the third is the byte name, and the fourth is the argument. The fifth, when present, is a hint about the meaning of the argument.
  • POP_JUMP_IF_FALSE, POP_JUMP_IF_TRUE, and JUMP_ABSOLUTE have the jump target as their argument. So, e.g. POP_JUMP_IF_TRUE 27 means “if the popped expression is true, jump to position 27.”
  • JUMP_FORWARD’s argument specifies the distance to jump forward in the bytecode, and the fifth column shows where the jump will end.
  • When an iterator is done, FOR_ITER jumps forward the number of bytes specified in its argument.

Unlike the else case, the line containing the continue does appear in the bytecode. But trace through the bytecode using what you know about jumps: no matter how hard you try, you can’t end up on bytes 66 or 69, the two that belong to line 13.

The continue is unreachable because of a compiler optimization. In this particular optimization, the compiler notices that two instructions in a row are jumps, and it combines these two hops into one larger jump. So, in a very real sense, the continue line didn’t execute – it was optimized out – even though the logic reflected in the continue is still reflected in the bytecode.

What would this bytecode have looked like without the optimizations? There’s not currently an option to disable the peephole bytecode optimizations, although there will be in a future version of Python (following an extensive debate on the python-dev list). For now, the only way to turn off optimizations is to comment out the relevant line in compile.c, the call to PyCode_Optimize, and recompile Python. Here’s the diff, if you’re playing along at home.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
cpython ⚲ hg diff
diff -r 77f36cdb71b0 Python/compile.c
--- a/Python/compile.c  Fri Aug 01 17:48:34 2014 +0200
+++ b/Python/compile.c  Sat Aug 02 15:43:45 2014 -0400
@@ -4256,10 +4256,6 @@
     if (flags < 0)
         goto error;

-    bytecode = PyCode_Optimize(a->a_bytecode, consts, names, a->a_lnotab);
-    if (!bytecode)
-        goto error;
-
     tmp = PyList_AsTuple(consts); /* PyCode_New requires a tuple */
     if (!tmp)
         goto error;
@@ -4270,7 +4266,7 @@
     kwonlyargcount = Py_SAFE_DOWNCAST(c->u->u_kwonlyargcount, Py_ssize_t, int);
     co = PyCode_New(argcount, kwonlyargcount,
                     nlocals_int, stackdepth(c), flags,
-                    bytecode, consts, names, varnames,
+                    a->a_bytecode, consts, names, varnames,
                     freevars, cellvars,
                     c->c_filename, c->u->u_name,
                     c->u->u_firstlineno,
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
>>> dis.dis(continuer)
  8           0 LOAD_CONST               1 (0)
              3 DUP_TOP
              4 STORE_FAST               0 (a)
              7 DUP_TOP
              8 STORE_FAST               1 (b)
             11 STORE_FAST               2 (c)

  9          14 SETUP_LOOP              79 (to 96)
             17 LOAD_GLOBAL              0 (range)
             20 LOAD_CONST               2 (100)
             23 CALL_FUNCTION            1 (1 positional, 0 keyword pair)
             26 GET_ITER
        >>   27 FOR_ITER                65 (to 95)
             30 STORE_FAST               3 (n)

  10         33 LOAD_FAST                3 (n)
             36 LOAD_CONST               3 (2)
             39 BINARY_MODULO
             40 POP_JUMP_IF_FALSE       72

  11         43 LOAD_FAST                3 (n)
             46 LOAD_CONST               4 (4)
             49 BINARY_MODULO
             50 POP_JUMP_IF_FALSE       66

  12         53 LOAD_FAST                0 (a)
             56 LOAD_CONST               5 (1)
             59 INPLACE_ADD
             60 STORE_FAST               0 (a)
             63 JUMP_FORWARD             0 (to 66)

  13    >>   66 JUMP_ABSOLUTE           27
             69 JUMP_FORWARD            10 (to 82)

  14    >>   72 LOAD_FAST                1 (b)
             75 LOAD_CONST               5 (1)
             78 INPLACE_ADD
             79 STORE_FAST               1 (b)

  15    >>   82 LOAD_FAST                2 (c)
             85 LOAD_CONST               5 (1)
             88 INPLACE_ADD
             89 STORE_FAST               2 (c)
             92 JUMP_ABSOLUTE           27
        >>   95 POP_BLOCK

  16    >>   96 LOAD_FAST                0 (a)
             99 LOAD_FAST                1 (b)
            102 LOAD_FAST                2 (c)
            105 BUILD_TUPLE              3
            108 RETURN_VALUE

Just as we expected, the jump targets have changed. The instruction at position 50, POP_JUMP_IF_FALSE, now has 66 as its jump target – a previously unreachable instruction associated with the continue. Instruction 63, JUMP_FORWARD, is also targeting 66. In both cases, the only way to reach this instruction is to jump to it, and the instruction itself jumps away.3

Now we can run our coverage tool with the unoptimized Python:

1
2
peephole [master *+] ⚲ ../cpython/python.exe coverage2.py
100% coverage, go you!

Complete success!

So is this a good idea or not?

Compiler optimizations are often a straightforward win. If the compiler can apply simple rules that make my code faster without requiring work from me, that’s great. Almost nobody requires a strict mapping of code that they write to code that ends up executing. So, peephole optimization in general: yes! Great!

But “almost nobody” is not nobody, and one kind of people who do require strict reasoning about executed code are the authors of test coverage software. In the python-dev thread I linked to earlier, there was an extensive discussion over whether or not serving this demographic by providing an option to disable to optimizations was worth increasing the complexity of the codebase. Ultimately it was decided that it was worthwhile, but this is a fair question to ask.

Further reading

Beyond the interesting Python-dev thread linked above, my other suggestions are mostly code. CPython’s peephole.c is pretty readable C code, and I encourage you to take a look at it. (“Constant folding” is a great place to start.) There’s also a website compileroptimizations.com which has short examples and discussion of 45 different optimizations. If you’d like to play with these code examples, they’re all available on my github.


  1. We need to include call events to capture the first line of a function declaration, def fn(...):

  2. I’ve previously written an introduction to the disassembler here.

  3. You may be wondering what the JUMP_ABSOLUTE instruction at position 66 is doing. This instruction does nothing unless a particular compiler optimization is turned on. The optimization support faster loops, but creates restrictions on what those loops can do. See ceval.c for more. Edit: This footnote previously incorrectly referenced JUMP_FORWARD.

Of Syntax Warnings and Symbol Tables

A Hacker Schooler hit an interesting bug today: her program would sometimes emit the message SyntaxWarning: import * only allowed at module level. I had never seen a SyntaxWarning before, so I decided to dig in.

The wording of the warning is strange: it says that star-import is only allowed at the module level, but it’s not a syntax error, just a warning. In fact, you can use a star-import in a scope that isn’t a module (in Python 2):

1
2
3
4
5
6
7
>>> def nope():
...     from random import *
...     print randint(1,10)
...
<stdin>:1: SyntaxWarning: import * only allowed at module level
>>> nope()
7

The Python spec gives some more details:

The from form with * may only occur in a module scope. If the wild card form of import — import * — is used in a function and the function contains or is a nested block with free variables, the compiler will raise a SyntaxError.

Just having import * in a function isn’t enough to raise a syntax error – we also need free variables. The Python execution model refers to three kinds of variables, ‘local,’ ‘global,’ and ‘free’, defined as follows:

If a name is bound in a block, it is a local variable of that block. If a name is bound at the module level, it is a global variable. (The variables of the module code block are local and global.) If a variable is used in a code block but not defined there, it is a free variable.

Now we can see how to trigger a syntax error from our syntax warning:

1
2
3
4
5
6
7
8
9
>>> def one():
...     def two():
...         from random import *
...         print randint(1,10)
...     two()
...
<stdin>:2: SyntaxWarning: import * only allowed at module level
  File "<stdin>", line 3
SyntaxError: import * is not allowed in function 'two' because it is a nested function

and similarly,

1
2
3
4
5
6
7
8
9
>>> def one():
...     from random import *
...     def two():
...         print randint(1,10)
...     two()
...
<stdin>:1: SyntaxWarning: import * only allowed at module level
  File "<stdin>", line 2
SyntaxError: import * is not allowed in function 'one' because it contains a nested function with free variables

As Python programmers, we’re used to our lovely dynamic language, and it’s unusual to hit compile-time constraints. As Amy Hanlon points out, it’s particularly weird to hit a compile-time error for code that wouldn’t raise a NameError when it ran – randint would indeed be in one’s namespace if the import-star had executed.

But we can’t run code that doesn’t compile, and in this case the compiler doesn’t have enough information to determine what bytecode to emit. There are different opcodes for loading and storing each of global, free, and local variables. A variable’s status as global, free, or local must be determined at compile time and then stored in the symbol table.

To investigate this, let’s look at minor variations on this code snippet and disassemble them.

1
2
3
4
5
6
7
8
9
10
11
>>> code_type = type((lambda: 1).__code__) # just a handle on the code type, which isn't exposed as a builtin
>>> # A helper function to disassemble nested functions
>>> def recur_dis(fn):
...     print(fn.__code__)
...     dis.dis(fn)
...     for const in fn.__code__.co_consts:
...         if isinstance(const, code_type):
...             print
...             print
...             print(const)
...             dis.dis(const)

First, when x is local, the compiler emits STORE_FAST in the assignment statement and LOAD_FAST to load it, marked with arrows below.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
>>> def one():
...     def two():
...         x = 1
...         print x
...     two()
>>> recur_dis(one)
<code object one at 0x10e246730, file "<stdin>", line 1>
  2           0 LOAD_CONST               1 (<code object two at 0x10e246030, file "<stdin>", line 2>)
              3 MAKE_FUNCTION            0
              6 STORE_FAST               0 (two)

  5           9 LOAD_FAST                0 (two)
             12 CALL_FUNCTION            0
             15 POP_TOP
             16 LOAD_CONST               0 (None)
             19 RETURN_VALUE

<code object two at 0x10e246030, file "<stdin>", line 2>
  3           0 LOAD_CONST               1 (1)
              3 STORE_FAST               0 (x)          <---- STORE_FAST

  4           6 LOAD_FAST                0 (x)          <----- LOAD_FAST
              9 PRINT_ITEM
             10 PRINT_NEWLINE
             11 LOAD_CONST               0 (None)
             14 RETURN_VALUE

When x is global, the compiler emits LOAD_GLOBAL to load it. I think the assignment is STORE_FAST again, but it’s not pictured here because the assignment is outside the function and thus not disassembled.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
>>> x = 1
>>> def one():
...     def two():
...         print x
...     two()
...
>>> recur_dis(one)
<code object one at 0x10e246730, file "<stdin>", line 1>
  2           0 LOAD_CONST               1 (<code object two at 0x10e2464b0, file "<stdin>", line 2>)
              3 MAKE_FUNCTION            0
              6 STORE_FAST               0 (two)

  4           9 LOAD_FAST                0 (two)
             12 CALL_FUNCTION            0
             15 POP_TOP
             16 LOAD_CONST               0 (None)
             19 RETURN_VALUE

<code object two at 0x10e2464b0, file "<stdin>", line 2>
  3           0 LOAD_GLOBAL              0 (x)          <----- LOAD_GLOBAL
              3 PRINT_ITEM
              4 PRINT_NEWLINE
              5 LOAD_CONST               0 (None)
              8 RETURN_VALUE

Finally, when x is nonlocal, the compiler notices that we’ll need a closure, and emits the opcodes LOAD_CLOSURE, MAKE_CLOSURE, and later LOAD_DEREF.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
>>> def one():
...     x = 1
...     def two():
...        print x
...     two()
...
>>> recur_dis(one)
<code object one at 0x10e246e30, file "<stdin>", line 1>
  2           0 LOAD_CONST               1 (1)
              3 STORE_DEREF              0 (x)          <----- STORE_DEREF

  3           6 LOAD_CLOSURE             0 (x)          <----- LOAD_CLOSURE
              9 BUILD_TUPLE              1
             12 LOAD_CONST               2 (<code object two at 0x10e246d30, file "<stdin>", line 3>)
             15 MAKE_CLOSURE             0
             18 STORE_FAST               0 (two)

  5          21 LOAD_FAST                0 (two)
             24 CALL_FUNCTION            0
             27 POP_TOP
             28 LOAD_CONST               0 (None)
             31 RETURN_VALUE

<code object two at 0x10e246d30, file "<stdin>", line 3>
  4           0 LOAD_DEREF               0 (x)          <----- LOAD_DEREF
              3 PRINT_ITEM
              4 PRINT_NEWLINE
              5 LOAD_CONST               0 (None)
              8 RETURN_VALUE

Let’s now return to a case that throws a syntax error.

1
2
3
4
5
6
7
8
9
>>> def one():
...      from random import *
...      def two():
...          print x
...      two()
...
<stdin>:1: SyntaxWarning: import * only allowed at module level
  File "<stdin>", line 2
SyntaxError: import * is not allowed in function 'one' because it contains a nested function with free variables

I’d love to show what the disassembled bytecode for this one looks like, but we can’t do that because there is no bytecode! We got a compile-time error, so there’s nothing here.

Further reading

Everything I know about symbol tables I learned from Eli Bendersky’s blog. I’ve skipped some complexity in the implementation that Eli covers.

acking through the source code of CPython for the text of the error message leads us right to symtable.c, which is exactly where we’d expect this message to be emitted. The function check_unoptimized shows where the syntax error gets thrown (and shows another illegal construct, too – but we’ll leave that one as an exercise for the reader).

p.s. In Python 3, import * anywhere other than a module is just an unqualified syntax error – none of this messing around with the symbol table.

PyCon Prep: Import Is a Keyword

Last week I had a ton of fun working with Amy Hanlon on her Harry Potter themed fork of Python, called Nagini. Nagini is full of magic and surprises. It implements the things you’d hope for out of a Harry Potter Python, like making quit into avada_kedavra, and many analogous jokes.

Amy also had the idea to replace import with accio! Replacing import is a much harder problem than renaming a builtin. Python doesn’t prevent you from overwriting builtins, whereas to change keywords you have to edit the grammar and recompile Python. You should go read Amy’s post on making this work.

This brings us to an interesting question: why is import a keyword, anyway? There’s a function, __import__, that does (mostly) the same thing:

1
2
>>> __import__('random')
<module 'random' from '/path/to/random.pyc'>

The function form requires the programmer to assign the return value – the module – to a name, but once we’ve done that it works just like a normal module:

1
2
3
>>> random = __import__('random')
>>> random.random()
0.32574174955668145

The __import__ function can handle all the forms of import, including from foo import bar and from baz import * (although it never modifies the calling namespace). There’s no technical reason why __import__ couldn’t be the regular way to do imports.1

As far as I can tell, the main argument against an import function is aesthetic. Compare:

1
2
3
4
5
6
7
import foo
from foo import bar
import longmodulename as short

foo = __import__('foo')
bar = __import__('random').bar
short = __import__('longmodulename')

The first way certainly feels much cleaner and more readable.2

Part of my goal in my upcoming PyCon talk is to invite Pythonistas to consider decisions they might not have thought about before. import is a great vehicle for this, because everyone learns it very early on in their programming development, but most people don’t ever think about it again. Here’s another variation on that theme: import doesn’t have to be a keyword!


  1. I think all keywords could be expressed as functions, except those used for flow control (which I loosely define as keywords that generate any JUMP instructions when compiled). For example, between Python 2 and 3, two keywords did become functions – print and exec.

  2. I realize this is a slightly circular argument – if the function strategy were the regular way to import, it probably wouldn’t be so ugly.