Darkest Before the Dawn: Bayesian Impostor Syndrome

Jan 12 JDN 2458860

At the time of writing, I have just returned from my second Allied Social Sciences Association Annual Meeting, the AEA’s annual conference (or AEA and friends, I suppose, since there several other, much smaller economics and finance associations are represented as well). This one was in San Diego, which made it considerably cheaper for me to attend than last year’s. Alas, next year’s conference will be in Chicago. At least flights to Chicago tend to be cheap because it’s a major hub.

My biggest accomplishment of the conference was getting some face-time and career advice from Colin Camerer, the Caltech economist who literally wrote the book on behavioral game theory. Otherwise I would call the conference successful, but not spectacular. Some of the talks were much better than others; I think I liked the one by Emmanuel Saez best, and I also really liked the one on procrastination by Matthew Gibson. I was mildly disappointed by Ben Bernanke’s keynote address; maybe I would have found it more compelling if I were more focused on macroeconomics.

But while sitting through one of the less-interesting seminars I had a clever little idea, which may help explain why Impostor Syndrome seems to occur so frequently even among highly competent, intelligent people. This post is going to be more technical than most, so be warned: Here There Be Bayes. If you fear yon algebra and wish to skip it, I have marked below a good place for you to jump back in.

Suppose there are two types of people, high talent H and low talent L. (In reality there is of course a wide range of talents, so I could assign a distribution over that range, but it would complicate the model without really changing the conclusions.) You don’t know which one you are; all you know is a prior probability h that you are high-talent. It doesn’t matter too much what h is, but for concreteness let’s say h = 0.50; you’ve got to be in the top 50% to be considered “high-talent”.

You are engaged in some sort of activity that comes with a high risk of failure. Many creative endeavors fit this pattern: Perhaps you are a musician looking for a producer, an actor looking for a gig, an author trying to secure an agent, or a scientist trying to publish in a journal. Or maybe you’re a high school student applying to college, or a unemployed worker submitting job applications.

If you are high-talent, you’re more likely to succeed—but still very likely to fail. And even low-talent people don’t always fail; sometimes you just get lucky. Let’s say the probability of success if you are high-talent is p, and if you are low-talent, the probability of success is q. The precise value depends on the domain; but perhaps p = 0.10 and q = 0.02.

Finally, let’s suppose you are highly rational, a good and proper Bayesian. You update all your probabilities based on your observations, precisely as you should.

How will you feel about your talent, after a series of failures?

More precisely, what posterior probability will you assign to being a high-talent individual, after a series of n+k attempts, of which k met with success and n met with failure?

Since failure is likely even if you are high-talent, you shouldn’t update your probability too much on a failurebut each failure should, in fact, lead to revising your probability downward.

Conversely, since success is rare, it should cause you to revise your probability upward—and, as will become important, your revisions upon success should be much larger than your revisions upon failure.

We begin as any good Bayesian does, with Bayes’ Law:

P[H|(~S)^n (S)^k] = P[(~S)^n (S)^k|H] P[H] / P[(~S)^n (S)^k]

In words, this reads: The posterior probability of being high-talent, given that you have observed k successes and n failures, is equal to the probability of observing such an outcome, given that you are high-talent, times the prior probability of being high-skill, divided by the prior probability of observing such an outcome.

We can compute the probabilities on the right-hand side using the binomial distribution:

P[H] = h

P[(~S)^n (S)^k|H] = (n+k C k) p^k (1-p)^n

P[(~S)^n (S)^k] = (n+k C k) p^k (1-p)^n h + (n+k C k) q^k (1-q)^n (1-h)

Plugging all this back in and canceling like terms yields:

P[H|(~S)^n (S)^k] = 1/(1 + [1-h/h] [q/p]^k [(1-q)/(1-p)]^n)

This turns out to be particularly convenient in log-odds form:

L[X] = ln [ P(X)/P(~X) ]

L[(~S)^n) (S)^k|H] = ln [h/(1-h)] + k ln [p/q] + n ln [(1-p)/(1-q)]

Since p > q, ln[p/q] is a positive number, while ln[(1-p)/(1-q)] is a negative number. This corresponds to the fact that you will increase your posterior when you observe a success (k increases by 1) and decrease your posterior when you observe a failure (n increases by 1).

But when p and q are small, it turns out that ln[p/q] is much larger in magnitude than ln[(1-p)/(1-q)]. For the numbers I gave above, p = 0.10 and q = 0.02, ln[p/q] = 1.609 while ln[(1-p)/(1-q)] = -0.085. You will therefore update substantially more upon a success than on a failure.

Yet successes are rare! This means that any given success will most likely be first preceded by a sequence of failures. This results in what I will call the darkest-before-dawn effect: Your opinion of your own talent will tend to be at its very worst in the moments just preceding a major success.

I’ve graphed the results of a few simulations illustrating this: On the X-axis is the number of overall attempts made thus far, and on the Y-axis is the posterior probability of being high-talent. The simulated individual undergoes randomized successes and failures with the probabilities I chose above.

Bayesian_Impostor_full

There are 10 simulations on that one graph, which may make it a bit confusing. So let’s focus in on two runs in particular, which turned out to be run 6 and run 10:

[If you skipped over the math, here’s a good place to come back. Welcome!]

Bayesian_Impostor_focus

Run 6 is a lucky little devil. They had an immediate success, followed by another success in their fourth attempt. As a result, they quickly update their posterior to conclude that they are almost certainly a high-talent individual, and even after a string of failures beyond that they never lose faith.

Run 10, on the other hand, probably has Impostor Syndrome. Failure after failure after failure slowly eroded their self-esteem, leading them to conclude that they are probably a low-talent individual. And then, suddenly, a miracle occurs: On their 20th attempt, at last they succeed, and their whole outlook changes; perhaps they are high-talent after all.

Note that all the simulations are of high-talent individuals. Run 6 and run 10 are equally competent. Ex ante, the probability of success for run 6 and run 10 was exactly the same. Moreover, both individuals are completely rational, in the sense that they are doing perfect Bayesian updating.

And yet, if you compare their self-evaluations after the 19th attempt, they could hardly look more different: Run 6 is 85% sure that they are high-talent, even though they’ve been in a slump for the last 13 attempts. Run 10, on the other hand, is 83% sure that they are low-talent, because they’ve never succeeded at all.

It is darkest just before the dawn: Run 10’s self-evaluation is at its very lowest right before they finally have a success, at which point their self-esteem surges upward, almost to baseline. With just one more success, their opinion of themselves would in fact converge to the same as Run 6’s.

This may explain, at least in part, why Impostor Syndrome is so common. When successes are few and far between—even for the very best and brightest—then a string of failures is the most likely outcome for almost everyone, and it can be difficult to tell whether you are so bright after all. Failure after failure will slowly erode your self-esteem (and should, in some sense; you’re being a good Bayesian!). You’ll observe a few lucky individuals who get their big break right away, and it will only reinforce your fear that you’re not cut out for this (whatever this is) after all.

Of course, this model is far too simple: People don’t just come in “talented” and “untalented” varieties, but have a wide range of skills that lie on a continuum. There are degrees of success and failure as well: You could get published in some obscure field journal hardly anybody reads, or in the top journal in your discipline. You could get into the University of Northwestern Ohio, or into Harvard. And people face different barriers to success that may have nothing to do with talent—perhaps why marginalized people such as women, racial minorities, LGBT people, and people with disabilities tend to have the highest rates of Impostor Syndrome. But I think the overall pattern is right: People feel like impostors when they’ve experienced a long string of failures, even when that is likely to occur for everyone.

What can be done with this information? Well, it leads me to three pieces of advice:

1. When success is rare, find other evidence. If truly “succeeding” (whatever that means in your case) is unlikely on any given attempt, don’t try to evaluate your own competence based on that extremely noisy signal. Instead, look for other sources of data: Do you seem to have the kinds of skills that people who succeed in your endeavors have—preferably based on the most objective measures you can find? Do others who know you or your work have a high opinion of your abilities and your potential? This, perhaps is the greatest mistake we make when falling prey to Impostor Syndrome: We imagine that we have somehow “fooled” people into thinking we are competent, rather than realizing that other people’s opinions of us are actually evidence that we are in fact competent. Use this evidence. Update your posterior on that.

2. Don’t over-update your posterior on failures—and don’t under-update on successes. Very few living humans (if any) are true and proper Bayesians. We use a variety of heuristics when judging probability, most notably the representative and availability heuristics. These will cause you to over-respond to failures, because this string of failures makes you “look like” the kind of person who would continue to fail (representative), and you can’t conjure to mind any clear examples of success (availability). Keeping this in mind, your update upon experiencing failure should be small, probably as small as you can make it. Conversely, when you do actually succeed, even in a small way, don’t dismiss it. Don’t look for reasons why it was just luck—it’s always luck, at least in part, for everyone. Try to update your self-evaluation more when you succeed, precisely because success is rare for everyone.

3. Don’t lose hope. The next one really could be your big break. While astronomically baffling (no, it’s darkest at midnight, in between dusk and dawn!), “it is always darkest before the dawn” really does apply here. You are likely to feel the worst about yourself at the very point where you are about to finally succeed. The lowest self-esteem you ever feel will be just before you finally achieve a major success. Of course, you can’t know if the next one will be it—or if it will take five, or ten, or twenty more tries. And yes, each new failure will hurt a little bit more, make you doubt yourself a little bit more. But if you are properly grounded by what others think of your talents, you can stand firm, until that one glorious day comes and you finally make it.

Now, if I could only manage to take my own advice….

Why did we ever privatize prisons?

JDN 2457103 EDT 10:24.

Since the Reagan administration (it’s always Reagan), the United States has undergone a spree of privatization of public services, in which services that are ordinarily performed by government agencies are instead contracted out to private companies. Enormous damage to our society has been done by this sort of privatization, from healthcare to parking meters.

This process can vary in magnitude.

The weakest form, which is relatively benign, is for the government to buy specific services like food service or equipment manufacturing from companies that already provide them to consumers. There’s no particular reason for the government to make their own toothpaste or wrenches rather than buy them from corporations like Proctor & Gamble and Sears. Toothpaste is toothpaste and wrenches are wrenches.

The moderate form is for the government to contract services to specific companies that may involve government-specific features like security clearances or powerful military weapons. This is already raising a lot of problems: When Northrop-Grumman makes our stealth bombers, and Boeing builds our nuclear ICBMs, these are publicly-traded, for-profit corporations manufacturing some of the deadliest weapons ever created—weapons that could literally destroy human civilization in a matter of minutes. Markets don’t work well in the presence of externalities, and weapons by definition are almost nothing but externalities; their entire function is to cause harm—typically, death—to people without their consent. While this violence may sometimes be justified, it must never be taken lightly; and we are right to be uncomfortable with the military-industrial complex whose shareholders profit from death and destruction. (Eisenhower tried to warn us!) Still, there are some good arguments to be made for this sort of privatization, since many of these corporations already have high-tech factories and skilled engineers that they can easily repurpose, and competitive bids between different corporations can keep the price down. (Of course, with no-bid contracts that no longer applies; and it certainly hasn’t stopped us from spending nearly as much on the military as the rest of the world combined.)

What I’d really like to focus on today is the strongest form of privatization, in which basic government services are contracted out to private companies. This is what happens when you attempt to privatize soldiers, SWAT teams, and prisons—all of which the United States has done since Reagan.

I say “attempt” to privatize because in a very real sense the privatization of these services is incoherent—they are functions so basic to government that simply to do them makes you, de facto, part of the government. (Or, if done without government orders, it would be organized crime.) All you’ve really done by “privatizing” these services is reduced their transparency and accountability, as well as siphoning off a portion of the taxpayer money in the form of profits for shareholders.

The benefits of privatization, when they exist, are due to competition and consumer freedom. The foundation of a capitalist economy is the ability to say “I’ll take my business elsewhere.” (This is why the notion that a bank can sell your loan to someone else is the opposite of a free market; forcing you to write a check to someone you never made a contract with is antithetical to everything the free market stands for.) Actually the closest thing to a successful example of privatized government services is the United States Postal Service, which collects absolutely no tax income. They do borrow from the government and receive subsidies for some of their services—but so does General Motors. Frankly I think the Postal Service has a better claim to privatization than GM, which you may recall only exists today because of a massive government bailout with a net cost to the US government of $11 billion. All the Postal Service does differently is act as a tightly-regulated monopoly that provides high-quality service to everyone at low prices and pays good wages and pensions, all without siphoning profits to shareholders. (They really screwed up my mail forwarding lately, but they are still one of the best postal systems in the world.) It is in many ways the best of both worlds, the efficiency of capitalism with the humanity of socialism.

The Corrections Corporation of America, on the other hand, is the exact opposite, the worst of both worlds, the inefficiency of socialism with the inhumanity of capitalism. It is not simply corrupt but frankly inherently corrupt—there is simply no way you can have a for-profit prison system that isn’t corrupt. Maybe it can be made less corrupt or more corrupt, but the mere fact that shareholders are earning profits from incarcerating prisoners is fundamentally antithetical to a free and just society.

I really can’t stress this enough: Privatizing soldiers and prisons makes no sense at all. It doesn’t even make sense in a world of infinite identical psychopaths; nothing in neoclassical economic theory in any way supports these privatizations. Neoclassical theory is based upon the presumption of a stable government that enforces property rights, a government that provides as much service as necessary exactly at cost and is not attempting to maximize any notion of its own “profit”.

That’s ridiculous, of course—much like the neoclassical rational agent—and more recent work has been done in public choice theory about the various interest groups that act against each other in government, including lobbyists for private corporations—but public choice theory is above all a theory of government failure. It is a theory of why governments don’t work as well as we would like them to—the main question is how we can suppress the influence of special interest groups to advance the public good. Privatization of prisons means creating special interest groups where none existed, making the government less directed at the public good.

Privatizing government services is often described as “reducing the size of government”, usually interpreted in the most narrow sense to mean the tax burden. But Big Government doesn’t mean you pay 22% of GDP instead of 18% of GDP; Big Government means you can be arrested and imprisoned without trial. Even using the Heritage Foundation’s metrics, the correlation between tax burden and overall freedom is positive. Tyrannical societies don’t bother with taxes; they own the oil refineries directly (Venezuela), or print money whenever they want (Zimbabwe), or build the whole society around doing what they want (North Korea).

The incarceration rate is a much better measure of a society’s freedom than the tax rate will ever be—and the US isn’t doing so well in that regard; indeed we have by some measures the highest incarceration rate in the world. Fortunately we do considerably better when it comes to things like free speech and freedom of religion—indeed we are still above average in overall freedom. Though we do imprison more of our people than China, I’m not suggesting that China has a freer society. But why do we imprison so many people?

Well, it seems to have something to do with privatization of prisons. Indeed, there is a strong correlation between the privatization of US prisons and the enormous explosion of incarceration in the United States. In fact privatized prisons don’t even reduce the tax burden, because privatization does not decrease demand and “privatized” prisons must still be funded by taxes. Prisons do not have customers who choose between different competing companies and shop for the highest quality and lowest price—prisoners go to the prison they are assigned to and they can’t leave (which is really the whole point). Even competition at the purchase end doesn’t make much sense, since the government can’t easily transfer all the prisoners to a new company. Maybe they could transfer ownership of the prison to a different company, but even then the transition costs would be substantial, and besides, there are only a handful of prison corporations that corner most of the (so-called) market.

There is simply no economic basis for privatization of prisons. Nothing in either neoclassical theory or more modern cognitive science in any way supports the idea. So the real question is: Why did we ever privatize prisons?

Basically there is only one reason: Ideology. The post-Reagan privatization spree was not actually based on economics—it was based on economic ideology. Either because they actually believed it, or by the Upton Sinclair Principle, a large number of economists adopted a radical far-right ideology that government basically should not exist—that the more we give more power to corporations and less power to elected officials the better off we will be.

They defended this ideology on vaguely neoclassical grounds, mumbling something about markets being more efficient; but this isn’t even like cutting off the wings of the airplane because we’re assuming frictionless vacuum—it’s like cutting off the engines of the airplane because we simply hate engines and are looking for any excuse to get rid of them. There is absolutely nothing in neoclassical economic theory that says it would be efficient or really beneficial in any way to privatize prisons. It was all about taking power away from the elected government and handing it over to for-profit corporations.

This is a bit of consciousness-raising I’m trying to do: Any time you hear someone say that something should be apolitical, I want you to substitute the word undemocratic. When they say that judges shouldn’t be elected so that they can be apolitical—they mean undemocratic. When they say that the Federal Reserve should be independent of politics—they mean independent of voting. They want to take decision power away from the public at large and concentrate it more in the hands of an elite. People who say this sort of thing literally do not believe in democracy.

To be fair, there may actually be good reasons to not believe in democracy, or at least to believe that democracy should be constrained by a constitution and a system of representation. Certain rights are inalienable, regardless of what the voting public may say, which is why we need a constitution that protects those rights above all else. (In theory… there’s always the PATRIOT ACT, speaking of imprisoning people without trial.) Moreover, most people are simply not interested enough—or informed enough—to vote on every single important decision the government makes. It makes sense for us to place this daily decision-making power in the hands of an elite—but it must be an elite we choose.

And yes, people often vote irrationally. One of the central problems in the United States today is that almost half the population consistently votes against rational government and their own self-interest on the basis of a misguided obsession with banning abortion, combined with a totally nonsensical folk theory of economics in which poor people are poor because they are lazy, the government inherently destroys whatever wealth it touches, and private-sector “job creators” simply hand out jobs to other people because they have extra money lying around. Then of course there’s—let’s face it—deep-seated bigotry toward women, racial minorities, and LGBT people. (The extreme hatred toward Obama and suspicion that he isn’t really born in the US really can’t be explained any other way.) In such circumstances it may be tempting to say that we should give up on democracy and let expert technocrats take charge; but in the absence of democratic safeguards, technocracy is little more than another name for oligarchy. Maybe it’s enough that the President appoints the Federal Reserve chair and the Supreme Court? I’m not so sure. Ben Bernanke definitely handled the Second Depression better than Congress did, I’ll admit; but I’m not sure Alan Greenspan would have in his place, and given his babbling lately about returning to Bretton Woods I’m pretty sure Paul Volcker wouldn’t have. (If you don’t see what’s wrong with going back to Bretton Woods, which was basically a variant of the gold standard, you should read what Krugman has to say about the gold standard.) So basically we got lucky and our monetary quasi-tyrant was relatively benevolent and wise. (Or maybe Bernanke was better because Obama appointed him, while Reagan appointed Greenspan. Carter appointed Volcker, oddly enough; but Reagan reappointed him. It’s always Reagan.) And if you could indeed ensure that tyrants would always be benevolent and wise, tyranny would be a great system—but you can’t.

Democracy doesn’t always lead to the best outcomes, but that’s really not what it’s for. Rather, democracy is for preventing the worst outcomes—no large-scale famine has ever occurred under a mature democracy, nor has any full-scale genocide. Democracies do sometimes forcibly “relocate” populations (particularly indigenous populations, as the US did under Andrew Jackson), and we should not sugar-coat that; people are forced out of their homes and many die. It could even be considered something close to genocide. But no direct and explicit mass murder of millions has ever occurred under a democratic government—no, the Nazis were not democratically elected—and that by itself is a fully sufficient argument for democracy. It could be true that democracies are economically inefficient (they are economically efficient), unbearably corrupt (they are less corrupt), and full of ignorant idiotic hicks (they have higher average educational attainment), and democracy would still be better simply because it prevents famine and genocide. As Churchill said, “Democracy is the worst system, except for all the others.”

Indeed, I think the central reason why American democracy isn’t working well right now is that it’s not very democratic; a two-party system with a plurality “first-past-the-post” vote is literally the worst possible voting system that can still technically be considered democracy. Any worse than that and you only have one party. If we had a range voting system (which is mathematically optimal) and say a dozen parties (they have about a dozen parties in France), people would be able to express their opinions more clearly and in more detail, with less incentive for strategic voting. We probably wouldn’t have such awful turnout at that point, and after realizing that they actually had such a strong voice, maybe people would even start educating themselves about politics in order to make better decisions.

Privatizing prisons and soldiers takes us in exactly the opposite direction: It makes our government deeply less democratic, fundamentally less accountable to voters. It hands off the power of life and death to institutions whose sole purpose for existence is their own monetary gain. We should never have done it—and we must undo it as soon as we possibly can.