On the quality of matches

Apr 11 JDN 2459316

Many situations in the real world involve matching people to other people: Dating, job hunting, college admissions, publishing, organ donation.

Alvin Roth won his Nobel Prize for his work on matching algorithms. I have nothing to contribute to improving his algorithm; what baffles me is that we don’t use it more often. It would probably feel too impersonal to use it for dating; but why don’t we use it for job hunting or college admissions? (We do use it for organ donation, and that has saved thousands of lives.)

In this post I will be looking at matching in a somewhat different way. Using a simple model, I’m going to illustrate some of the reasons why it is so painful and frustrating to try to match and keep getting rejected.

Suppose we have two sets of people on either side of a matching market: X and Y. I’ll denote an arbitrarily chosen person in X as x, and an arbitrarily chosen person in Y as y. There’s no reason the two sets can’t have overlap or even be the same set, but making them different sets makes the model as general as possible.

Each person in X wants to match with a person in Y, and vice-versa. But they don’t merely want to accept any possible match; they have preferences over which matches would be better or worse.

In general, we could say that people have some kind of utility function: Ux:Y->R and Uy:X->R that maps from possible match partners to the utility of such a match. But that gets very complicated very fast, because it raises the question of when you should keep searching, and when you should stop searching and accept what you have. (There’s a whole literature of search theory on this.)

For now let’s take the simplest possible case, and just say that there are some matches each person will accept, and some they will reject. This can be seen as a special case where the utility functions Ux and Uy always yield a result of 1 (accept) or 0 (reject).

This defines a set of acceptable partners for each person: A(x) is the set of partners x will accept: {y in Y|Ux(y) = 1} and A(y) is the set of partners y will accept: {x in X|Uy(x) = 1}

Then, the set of mutual matches than x can actually get is the set of ys that x wants, which also want x back: M(x) = {y in A(x)|x in A(y)}

Whereas, the set of mutual matches that y can actually get is the set of xs that y wants, which also want y back: M(y) = {x in A(y)|y in A(x)}

This relation is mutual by construction: If x is in M(y), then y is in M(x).

But this does not mean that the sets must be the same size.

For instance, suppose that there are three people in X, x1, x2, x3, and three people in Y, y1, y2, y3.

Let’s say that the acceptable matches are as follows:

A(x1) = {y1, y2, y3}

A(x2) = {y2, y3}

A(x3) = {y2, y3}

A(y1) = {x1,x2,x3}

A(y2) = {x1,x2}

A(y3) = {x1}

This results in the following mutual matches:

M(x1) = {y1, y2, y3}

M(y1) = {x1}

M(x2) = {y2}

M(y2) = {x1, x2}

M(x3) = {}

M(y3) = {x1}

x1 can match with whoever they like; everyone wants to match with them. x2 can match with y2. But x3, despite having the same preferences as x2, and being desired by y3, can’t find any mutual matches at all, because the one person who wants them is a person they don’t want.

y1 can only match with x1, but the same is true of y3. So they will be fighting over x1. As long as y2 doesn’t also try to fight over x1, x2 and y2 will be happy together. Yet x3 will remain alone.

Note that the number of mutual matches has no obvious relation with the number of individually acceptable partners. x2 and x3 had the same number of acceptable partners, but x2 found a mutual match and x3 didn’t. y1 was willing to accept more potential partners than y3, but got the same lone mutual match in the end. y3 was only willing to accept one partner, but will get a shot at x1, the one that everyone wants.

One thing is true: Adding another acceptable partner will never reduce your number of mutual matches, and removing one will never increase it. But often changing your acceptable partners doesn’t have any effect on your mutual matches at all.

Now let’s consider what it must feel like to be x1 versus x3.

For x1, the world is their oyster; they can choose whoever they want and be guaranteed to get a match. Life is easy and simple for them; all they have to do is decide who they want most and that will be it.

For x3, life is an endless string of rejection and despair. Every time they try to reach out to suggest a match with someone, they are rebuffed. They feel hopeless and alone. They feel as though no one would ever actually want them—even though in fact there is someone who wants them, it’s just not someone they were willing to consider.

This is of course a very simple and small-scale model; there are only six people in it, and they each only say yes or no. Yet already I’ve got x1 who feels like a rock star and x3 who feels utterly hopeless if not worthless.

In the real world, there are so many more people in the system that the odds that no one is in your mutual match set are negligible. Almost everyone has someone they can match with. But some people have many more matches than others, and that makes life much easier for the ones with many matches and much harder for the ones with fewer.

Moreover, search costs then become a major problem: Even knowing that in all probability there is a match for you somewhere out there, how do you actually find that person? (And that’s not even getting into the difficulty of recognizing a good match when you see it; in this simple model you know immediately, but in the real world it can take a remarkably long time.)

If we think of the acceptable partner sets as preferences, they may not be within anyone’s control; you want what you want. But if we instead characterize them as decisions, the results are quite differentand I think it’s easy to see them, if nothing else, as the decision of how high to set your standards.

This raises a question: When we are searching and not getting matches, should we lower our standards and add more people to our list of acceptable partners?

This simple model would seem to say that we should always do that—there’s no downside, since the worst that can happen is nothing. And x3 for instance would be much happier if they were willing to lower their standards and accept y1. (Indeed, if they did so, there would be a way to pair everyone off happily: x1 with y3, x2 with y2, and x3 with y1.)

But in the real world, searching is often costly: There is at least the involved, and often a literal application or submission fee; but perhaps worst of all is the crushing pain of rejection. Under those circumstances, adding another acceptable partner who is not a mutual match will actually make you worse off.

That’s pretty much what the job market has been for me for the last six months. I started out with the really good matches: GiveWell, the Oxford Global Priorities Institute, Purdue, Wesleyan, Eastern Michigan University. And after investing considerable effort into getting those applications right, I made it as far as an interview at all those places—but no further.

So I extended my search, applying to dozens more places. I’ve now applied to over 100 positions. I knew that most of them were not good matches, because there simply weren’t that many good matches to be found. And the result of all those 100 applications has been precisely 0 interviews. Lowering my standards accomplished absolutely nothing. I knew going in that these places were not a good fit for me—and it looks like they all agreed.

It’s possible that lowering my standards in some different way might have worked, but even this is not clear: I’ve already been willing to accept much lower salaries than a PhD in economics ought to entitle, and included positions in my search that are only for a year or two with no job security, and applied to far-flung locales across the globe that I don’t know if I’d really be willing to move to.

Honestly at this point I’ve only been using the following criteria: (1) At least vaguely related to my field (otherwise they wouldn’t want me anyway), (2) a higher salary than I currently get as a grad student (otherwise why bother?), (3) a geographic location where homosexuality is not literally illegal and an institution that doesn’t actively discriminate against LGBT employees (this rules out more than you’d think—there are at least three good postings I didn’t apply to on these grounds), (4) in a region that speaks a language I have at least some basic knowledge of (i.e. preferably English, but also allowing Spanish, French, German, or Japanese) (5) working conditions that don’t involve working more than 40 hours per week (which has severely detrimental health effects, even ignoring my disability which would compound the effects), and (6) not working for a company that is implicated in large-scale criminal activity (as a remarkable number of major banks have in fact been implicated). I don’t feel like these are unreasonably high standards, and yet so far I have failed to land a match.

What’s more, the entire process has been emotionally devastating. While others seem to be suffering from pandemic burnout, I don’t think I’ve made it that far; I think I’d be just as burnt out even if there were no pandemic, simply from how brutal the job market has been.

Why does rejection hurt so much? Why does being turned down for a date, or a job, or a publication feel so utterly soul-crushing? When I started putting together this model I had hoped that thinking of it in terms of match-sets might actually help reduce that feeling, but instead what happened is that it offered me a way of partly explaining that feeling (much as I did in my post on Bayesian Impostor Syndrome).

What is the feeling of rejection? It is the feeling of expending search effort to find someone in your acceptable partner set—and then learning that you were not in their acceptable partner set, and thus you have failed to make a mutual match.

I said earlier that x1 feels like a rock star and x3 feels hopeless. This is because being present in someone else’s acceptable partner set is a sign of status—the more people who consider you an acceptable partner, the more you are “worth” in some sense. And when it’s something as important as a romantic partner or a career, that sense of “worth” is difficult to circumscribe into a particular domain; it begins to bleed outward into a sense of your overall self-worth as a human being.

Being wanted by someone you don’t want makes you feel superior, like they are “beneath” you; but wanting someone who doesn’t want you makes you feel inferior, like they are “above” you. And when you are applying for jobs in a market with a Beveridge Curve as skewed as ours, or trying to get a paper or a book published in a world flooded with submissions, you end up with a lot more cases of feeling inferior than cases of feeling superior. In fact, I even applied for a few jobs that I felt were “beneath” my level—they didn’t take me either, perhaps because they felt I was overqualified.

In such circumstances, it’s hard not to feel like I am the problem, like there is something wrong with me. Sometimes I can convince myself that I’m not doing anything wrong and the market is just exceptionally brutal this year. But I really have no clear way of distinguishing that hypothesis from the much darker possibility that I have done something terribly wrong that I cannot correct and will continue in this miserable and soul-crushing fruitless search for months or even years to come. Indeed, I’m not even sure it’s actually any better to know that you did everything right and still failed; that just makes you helpless instead of defective. It might be good for my self-worth to know that I did everything right; but it wouldn’t change the fact that I’m in a miserable situation I can’t get out of. If I knew I were doing something wrong, maybe I could actually fix that mistake in the future and get a better outcome.

As it is, I guess all I can do is wait for more opportunities and keep trying.

Men and violence

Apr4 JDN 2459302

Content warning: In this post, I’m going to be talking about violence, including sexual violence. April is Sexual Assault Awareness and Prevention Month. I won’t go into any explicit detail, but I understand that discussion of such topics can still be very upsetting for many people.

After short posts for the past two weeks, get ready for a fairly long post. This is a difficult and complicated topic, and I want to make sure that I state things very clearly and with all necessary nuance.

While the overall level of violence between human societies varies tremendously, one thing is astonishingly consistent: Violence is usually committed by men.

In fact, violence is usually suffered by men as well—with the quite glaring exception of sexual violence. This is why I am particularly offended by claims like “All men benefit from male violence”; no, men who were murdered by other men did not benefit from male violence, and it is frankly appalling to say otherwise. Most men would be better off if male violence were somehow eliminated from the world. (Most women would also be much better off as well, of course.)

I therefore consider it both a matter of both moral obligation and self-interest to endeavor to reduce the amount of male violence in the world, which is almost coextensive with reducing the amount of violence in general.

On the other hand, ought implies can, and despite significant efforts I have made to seek out recommendations for concrete actions I could be taking… I haven’t been able to find very many.

The good news is that we appear to be doing something right—overall rates of violent crime have declined by nearly half since 1990. The decline in rape has been slower, only about 25% since 1990, though this is a bit misleading since the legal definition of rape has been expanded during that interval. The causes of this decline in violence are unclear: Some of the most important factors seem to be changes in policing, economic growth, and reductions in lead pollution. For whatever reason, Millennials just don’t seem to commit crimes at the same rates that Gen-X-ers or Boomers did. We are also substantially more feminist, so maybe that’s an important factor too; the truth is, we really don’t know.

But all of this still leaves me asking: What should I be doing?

When I searched for an answer to this question, a significant fraction of the answers I got from various feminist sources were some variation on “ruminate on your own complicity in male violence”. I tried it; it was painful, difficult—and basically useless. I think this is particularly bad advice for someone like me who has a history of depression.

When you ruminate on your own life, it’s easy to find mistakes; but how important were those mistakes? How harmful were they? I can’t say that I’ve never done anything in my whole life that hurt anyone emotionally (can anyone?), but I can only think of a few times I’ve harmed someone physically (mostly by accident, once in self-defense). I’ve definitely never raped or murdered anyone, and as far as I can tell I’ve never done anything that would have meaningfully contributed to anyone getting raped or murdered. If you were to somehow replace every other man in the world with a copy of me, maybe that wouldn’t immediately bring about a utopian paradise—but I’m pretty sure that rates of violence would be a lot lower. (And in this world ruled by my clones, we’d have more progressive taxes! Less military spending! A basic income! A global democratic federation! Greater investment in space travel! Hey, this sounds pretty good, actually… though inbreeding would be a definite concern.) So, okay, I’m no angel; but I don’t think it’s really fair to say that I’m complicit in something that would radically decrease if everyone behaved as I do.

The really interesting thing is, I think this is true of most men. A typical man commits less than the average amount of violence—because there is great skew in the distribution, with most men committing little or no violence and a small number of men committing lots of violence. Truly staggering amounts of violence are committed by those at the very top of the distribution—that would be mass murderers like Hitler and Stalin. It sounds strange, but if all men in the world were replaced by a typical man, the world would surely be better off. The loss of the very best men would be more than compensated by the removal of the very worst. In fact, since most men are not rapists or murderers, replacing every man in the world with the median man would automatically bring the rates of rape and murder to zero. I know that feminists don’t like to hear #NotAllMen; but it’s not even most men. Maybe the reason that the “not all men” argument keeps coming up is… it’s actually kind of true? Maybe it’s not so unreasonable for men to resent the implication that we are complicit in acts we abhor that we have never done and would never do? Maybe this whole concept that an entire sex of people, literally almost half the human race, can share responsibility for violent crimes—is wrong?

I know that most women face a nearly constant bombardment of sexual harassment, and feel pressured to remain constantly vigilant in order to protect themselves against being raped. I know that victims of sexual violence are often blamed for their victimization (though this happens in a lot of crimes, not just sex crimes). I know that #YesAllWomen is true—basically all women have been in some way harmed or threatened by sexual violence. But the fact remains that most men are already not committing sexual violence. Many people seem to confuse the fact that most women are harmed by men with the claim that most men harm women; these are not at all equivalent. As long as one man can harm many women, there don’t need to be very many harmful men for all women to be affected.

Plausible guesses would be that about 20-25% of women suffer sexual assault, committed by about 4% or 5% of men, each of whom commits an average of 4 to 6 assaults—and some of whom commit far more. If these figures are right, then 95% of men are not guilty of sexual assault. The highest plausible estimate I’ve seen is from a study which found that 11% of men had committed rape. Since it’s only one study and its sample size was pretty small, I’m actually inclined to think that this is an overestimate which got excessive attention because it was so shocking. Larger studies rarely find a number above 5%.

But even if we suppose that it’s really 11%, that leaves 89%; in what sense is 89% not “most men”? I saw some feminist sites responding to this result by saying things like “We can’t imprison 11% of men!” but, uh, we almost do already. About 9% of American men will go to prison in their lifetimes. This is probably higher than it should be—it’s definitely higher than any other country—but if those convictions were all for rape, I’d honestly have trouble seeing the problem. (In fact only about 10% of US prisoners are incarcerated for rape.) If the US were the incarceration capital of the world simply because we investigated and prosecuted rape more reliably, that would be a point of national pride, not shame. In fact, the American conservatives who don’t see the problem with our high incarceration rate probably do think that we’re mostly incarcerating people for things like rape and murder—when in fact large portions of our inmates are incarcerated for drug possession, “public order” crimes, or pretrial detention.

Even if that 11% figure is right, “If you know 10 men, one is probably a rapist” is wrong. The people you know are not a random sample. If you don’t know any men who have been to prison, then you likely don’t know any men who are rapists. 37% of prosecuted rapists have prior criminal convictions, and 60% will be convicted of another crime within 5 years. (Of course, most rapes are never even reported; but where would we get statistics on those rapists?) Rapists are not typical men. They may seem like typical men—it may be hard to tell the difference at a glance, or even after knowing someone for a long time. But the fact that narcissists and psychopaths may hide among us does not mean that all of us are complicit in the crimes of narcissists and psychopaths. If you can’t tell who is a psychopath, you may have no choice but to be wary; but telling every man to search his heart is worthless, because the only ones who will listen are the ones who aren’t psychopaths.

That, I think, is the key disagreement here: Where the standard feminist line is “any man could be a rapist, and every man should search his heart”, I believe the truth is much more like, “monsters hide among us, and we should do everything in our power to stop them”. The monsters may look like us, they may often act like us—but they are not us. Maybe there are some men who would commit rapes but can be persuaded out of it—but this is not at all the typical case. Most rapes are committed by hardened, violent criminals and all we can really do is lock them up. (And for the love of all that is good in the world, test all the rape kits!)

It may be that sexual harassment of various degrees is more spread throughout the male population; perhaps the median man indeed commits some harassment at some point in his life. But even then, I think it’s pretty clear that the really awful kinds of harassment are largely committed by a small fraction of serial offenders. Indeed, there is a strong correlation between propensity toward sexual harassment and various measures of narcissism and psychopathy. So, if most men look closely enough, maybe they can think of a few things that they do occasionally that might make women uncomfortable; okay, stop doing those things. (Hint: Do not send unsolicited dick pics. Ever. Just don’t. Anyone who wants to see your genitals will ask first.) But it isn’t going to make a huge difference in anyone’s life. As long as the serial offenders continue, women will still feel utterly bombarded.

There are other kinds of sexual violations that more men commit—being too aggressive, or persisting too much after the first rejection, or sending unsolicited sexual messages or images. I’ve had people—mostly, but not only, men—do things like that to me; but it would be obviously unfair to both these people and actual rape victims to say I’d ever been raped. I’ve been groped a few times, but it seems like quite a stretch to call it “sexual assault”. I’ve had experiences that were uncomfortable, awkward, frustrating, annoying, occasionally creepy—but never traumatic. Never violence. Teaching men (and women! There is evidence that women are not much less likely than men to commit this sort of non-violent sexual violation) not to do these things is worthwhile and valuable in itself—but it’s not going to do much to prevent rape or murder.

Thus, whatever responsibility men have in reducing sexual violence, it isn’t simply to stop; you can’t stop doing what you already aren’t doing.

After pushing through all that noise, at last I found a feminist site making a more concrete suggestion: They recommended that I read a book by Jackson Katz on the subject entitled The Macho Paradox: Why Some Men Hurt Women and How All Men Can Help.

First of all, I must say I can’t remember any other time I’ve read a book that was so poorly titled. The only mention of the phrase “macho paradox” is a brief preface that was added to the most recent edition explaining what the term was meant to mean; it occurs nowhere else in the book. And in all its nearly 300 pages, the book has almost nothing that seriously addresses either the motivations underlying sexual violence or concrete actions that most men could take in order to reduce it.

As far as concrete actions (“How all men can help”), the clearest, most consistent advice the book seems to offer that would apply to most men is “stop consuming pornography” (something like 90% of men and 60% of women regularly consume porn), when in fact there is a strong negative correlation between consumption of pornography and real-world sexual violence. (Perhaps Millennials are less likely to commit rape and murder because we are so into porn and video games!) This advice is literally worse than nothing.

The sex industry exists on a continuum from the adult-only but otherwise innocuous (smutty drawings and erotic novels), through the legal but often problematic (mainstream porn, stripping), to the usually illegal but defensible (consensual sex work), all the way to the utterly horrific and appalling (the sexual exploitation of children). I am well aware that there are many deep problems with the mainstream porn industry, but I confess I’ve never quite seen how these problems are specific to porn rather than endemic to media or even capitalism more generally. Particularly with regard to the above-board sex industry in places like Nevada or the Netherlands, it’s not obvious to me that a prostitute is more exploited than a coal miner, a sweatshop worker, or a sharecropper—indeed, given the choice between those four careers, I’d without hesitation choose to be a prostitute in Amsterdam. Many sex workers resent the paternalistic insistence by anti-porn feminists that their work is inherently degrading and exploitative. Overall, sex workers report job satisfaction not statistically different than the average for all jobs. There are a multitude of misleading statistics often reported about the sex industry that often make matters seem far worse than they are.

Katz (all-too) vividly describes the depiction of various violent or degrading sex acts in mainstream porn, but he seems unwilling to admit that any other forms of porn do or even could exist—and worse, like far too many anti-porn feminists, he seems to willfully elide vital distinctions, effectively equating fantasy depiction with genuine violence and consensual kinks with sexual abuse. I like to watch action movies and play FPS video games; does that mean I believe it’s okay to shoot people with machine guns? I know the sophisticated claim is that it somehow “desensitizes” us (whatever that means), but there’s not much evidence of that either. Given that porn and video games are negatively correlated with actual violence, it may in fact be that depicting the fantasy provides an outlet for such urges and helps prevent them from becoming reality. Or, it may simply be that keeping a bunch of young men at home in front of their computers keeps them from going out and getting into trouble. (Then again, homicides actually increased during the COVID pandemic—though most other forms of crime decreased.) But whatever the cause, the evidence is clear that porn and video games don’t increase actual violence—they decrease them.

At the very end of the book, Katz hints at a few other things men might be able to do, or at least certain groups of men: Challenge sexism in sports, the military, and similar male-dominated spaces (you know, if you have clout in such spaces, which I really don’t—I’m an effete liberal intellectual, a paradigmatic “soy boy”; do you think football players or soldiers are likely to listen to me?); educate boys with more positive concepts of masculinity (if you are in a position to do so, e.g. as a teacher or parent); or, the very best advice in the entire book, worth more than the rest of the book combined: Donate to charities that support survivors of sexual violence. Katz doesn’t give any specific recommendations, but here are a few for you: RAINN, NAESV and NSVRC.

Honestly, I’m more impressed by Upworthy’s bulleted list of things men can do, though they’re mostly things that conscientious men do anyway, and even if 90% of men did them, it probably wouldn’t greatly reduce actual violence.

As far as motivations (“Why some men hurt women”), the book does at least manage to avoid the mindless slogan “rape is about power, not sex” (there is considerable evidence that this slogan is false or at least greatly overstated). Still, Katz insists upon collective responsibility, attributing what are in fact typically individual crimes, committed mainly by psychopaths, motivated primarily by anger or sexual desire, to some kind of institutionalized system of patriarchal control that somehow permeates all of society. The fact that violence is ubiquitous does not imply that it is coordinated. It’s very much the same cognitive error as “murderism”.

I agree that sexism exists, is harmful, and may contribute to the prevalence of rape. I agree that there are many widespread misconceptions about rape. I also agree that reducing sexism and toxic masculinity are worthwhile endeavors in themselves, with numerous benefits for both women and men. But I’m just not convinced that reducing sexism or toxic masculinity would do very much to reduce the rates of rape or other forms of violence. In fact, despite widely reported success of campaigns like the “Don’t Be That Guy” campaign, the best empirical research on the subject suggests that such campaigns actually tend to do more harm than good. The few programs that seem to work are those that focus on bystander interventions—getting men who are not rapists to recognize rapists and stop them. Basically nothing has ever been shown to convince actual rapists; all we can do is deny them opportunities—and while bystander intervention can do that, the most reliable method is probably incarceration. Trying to change their sexist attitudes may be worse than useless.

Indeed, I am increasingly convinced that much—not all, but much—of what is called “sexism” is actually toxic expressions of heterosexuality. Why do most creepy male bosses only ever hit on their female secretaries? Well, maybe because they’re straight? This is not hard to explain. It’s a fair question why there are so many creepy male bosses, but one need not posit any particular misogyny to explain why their targets would usually be women. I guess it’s a bit hard to disentangle; if an incel hates women because he perceives them as univocally refusing to sleep with him, is that sexism? What if he’s a gay incel (yes they exist) and this drives him to hate men instead?

In fact, I happen to know of a particular gay boss who has quite a few rumors surrounding him regarding his sexual harassment of male employees. Or you could look at Kevin Spacey, who (allegedly) sexually abused teenage boys. You could tell a complicated story about how this is some kind of projection of misogynistic attitudes onto other men (perhaps for being too “femme” or something)—or you could tell a really simple story about how this man is only sexually abusive toward other men because that’s the gender of people he’s sexually attracted to. Occam’s Razor strongly favors the latter.

Indeed, what are we to make of the occasional sexual harasser who targets men and women equally? On the theory that abuse is caused by patriarchy, that seems pretty hard to explain. On the theory that abusive people sometimes happen to be bisexual, it’s not much of a mystery. (Though I would like to take a moment to debunk the stereotype of the “depraved bisexual”: Bisexuals are no more likely to commit sexual violence, but are far more likely to suffer it—more likely than either straight or gay people, independently of gender. Trans people face even higher risk; the acronym LGBT is in increasing order of danger of violence.)

Does this excuse such behavior? Absolutely not. Sexual harassment and sexual assault are definitely wrong, definitely harmful, and rightfully illegal. But when trying to explain why the victims are overwhelmingly female, the fact that roughly 90% of people are heterosexual is surely relevant. The key explanandum here is not why the victims are usually female, but rather why the perpetrators are usually male.

That, indeed, requires explanation; but such an explanation is really not so hard to come by. Why is it that, in nearly every human society, for nearly every form of violence, the vast majority of that violence is committed by men? It sure looks genetic to me.

Indeed, in anyother context aside from gender or race, we would almost certainly reject any explanation other than genetics for such a consistent pattern. Why is it that, in nearly every human society, about 10% of people are LGBT? Probably genetics. Why is it that, in near every human society, about 10% of people are left-handed? Genetics. Why, in nearly every human society, do smiles indicate happiness, children fear loud noises, and adults fear snakes? Genetics. Why, in nearly every human society, are men on average much taller and stronger than women? Genetics. Why, in nearly every human society, is about 90% of violence, including sexual violence, committed by men? Clearly, it’s patriarchy.

A massive body of scientific evidence from multiple sources shows a clear casual relationship between increased testosterone and increased aggression. The correlation is moderate, only about 0.38—but it’s definitely real. And men have a lot more testosterone than women: While testosterone varies a frankly astonishing amount between men and over time—including up to a 2-fold difference even over the same day—a typical adult man has about 250 to 950 ng/dL of blood testosterone, while a typical adult woman has only 8 to 60 ng/dL. (An adolescent boy can have as much as 1200 ng/dL!) This is a difference ranging from a minimum of 4-fold to a maximum of over 100-fold, with a typical value of about 20-fold. It would be astonishing if that didn’t have some effect on behavior.

This is of course far from a complete explanation: With a correlation of 0.38, we’ve only explained about 14% of the variance, so what’s the other 86%? Well, first of all, testosterone isn’t the only biological difference between men and women. It’s difficult to identify any particular genes with strong effects on aggression—but the same is true of height, and nobody disputes that the height difference between men and women is genetic.

Clearly societal factors do matter a great deal, or we couldn’t possibly explain why homicide rates vary between countries from less than 3 per million per year in Japan to nearly 400 per million per year in Hondurasa full 2 orders of magnitude! But gender inequality does not appear to strongly predict homicide rates. Japan is not a very feminist place (in fact, surveys suggest that, after Spain, Japan is second-worst highly-developed country for women). Sweden is quite feminist, and their homicide rate is relatively low; but it’s still 4 times as high as Japan’s. The US doesn’t strike me as much more sexist than Canada (admittedly subjective—surveys do suggest at least some difference, and in the expected direction), and yet our homicide rate is nearly 3 times as high. Also, I think it’s worth noting that while overall homicide rates vary enormously across societies, the fact that roughly 90% of homicides are committed by men does not. Through some combination of culture and policy, societies can greatly reduce the overall level of violence—but no society has yet managed to change the fact that men are more violent than women.

I would like to do a similar analysis of sexual assault rates across countries, but unfortunately I really can’t, because different countries have such different laws and different rates of reporting that the figures really aren’t comparable. Sweden infamously has a very high rate of reported sex crimes, but this is largely because they have very broad definitions of sex crimes and very high rates of reporting. The best I can really say for now is there is no obvious pattern of more feminist countries having lower rates of sex crimes. Maybe there really is such a pattern; but the data isn’t clear.

Yet if biology contributes anything to the causation of violence—and at this point I think the evidence for that is utterly overwhelming—then mainstream feminism has done the world a grave disservice by insisting upon only social and cultural causes. Maybe it’s the case that our best options for intervention are social or cultural, but that doesn’t mean we can simply ignore biology. And then again, maybe it’s not the case at all:A neurological treatment to cure psychopathy could cut almost all forms of violence in half.

I want to be completely clear that a biological cause is not a justification or an excuse: literally billions of men manage to have high testosterone levels, and experience plenty of anger and sexual desire, without ever raping or murdering anyone. The fact that men appear to be innately predisposed toward violence does not excuse actual violence, and the fact that rape is typically motivated at least in part by sexual desire is no excuse for committing rape.

In fact, I’m quite worried about the opposite: that the notion that sexual violence is always motivated by a desire to oppress and subjugate women will be used to excuse rape, because men who know that their motivation was not oppression will therefore be convinced that what they did wasn’t rape. If rape is always motivated by a desire to oppress women, and his desire was only to get laid, then clearly, what he did can’t be rape, right? The logic here actually makes sense. If we are to reject this argument—as we must—then we must reject the first premise, that all rape is motivated by a desire to oppress and subjugate women. I’m not saying that’s never a motivation—I’m simply saying we can’t assume it is always.

The truth is, I don’t know how to end violence, and sexual violence may be the most difficult form of violence to eliminate. I’m not even sure what most of us can do to make any difference at all. For now, the best thing to do is probably to donate money to organizations like RAINN, NAESV and NSVRC. Even $10 to one of these organizations will do more to help survivors of sexual violence than hours of ruminating on your own complicity—and cost you a lot less.

Love in a time of quarantine

Feb 14JDN 2459260

This is our first Valentine’s Day of quarantine—and hopefully our last. With Biden now already taking action and the vaccine rollout proceeding more or less on schedule, there is good reason to think that this pandemic will be behind us by the end of this year.

Yet for now we remain isolated from one another, attempting to substitute superficial digital interactions for the authentic comforts of real face-to-face contact. And anyone who is single, or forced to live away from their loved ones, during quarantine is surely having an especially hard time right now.

I have been quite fortunate in this regard: My fiancé and I have lived together for several years, and during this long period of isolation we’ve at least had each other—if basically no one else.

But even I have felt a strong difference, considerably stronger than I expected it would be: Despite many of my interactions already being conducted via the Internet, needing to do so with all interactions feels deeply constraining. Nearly all of my work can be done remotely—but not quite all, and even what can be done remotely doesn’t always work as well remotely. I am moderately introverted, and I still feel substantially deprived; I can only imagine how awful it must be for the strongly extraverted.

As awkward as face-to-face interactions can be, and as much as I hate making phone calls, somehow Zoom video calls are even worse than either. Being unable to visit someone’s house for dinner and games, or go out to dinner and actually sit inside a restaurant, leaves a surprisingly large emotional void. Nothing in particular feels radically different, but the sum of so many small differences adds up to a rather large one. I think I felt it the most when we were forced to cancel our usual travel back to Michigan over the holiday season.

Make no mistake: Social interaction is not simply something humans enjoy, or are good at. Social interaction is a human need. We need social interaction in much the same way that we need food or sleep. The United Nations considers solitary confinement for more than two weeks to be torture. Long periods in solitary confinement are strongly correlated with suicide—so in that sense, isolation can kill you. Think about the incredibly poor quality of social interactions that goes on in most prisons: Endless conflict, abuse, racism, frequent violence—and then consider that the one thing that inmates find most frightening is to be deprived of that social contact. This is not unlike being fed nothing but stale bread and water, and then suddenly having even that taken away from you.

Even less extreme forms of social isolation—like most of us are feeling right now—have as detrimental an effect on health as smoking or alcoholism, and considerably worse than obesity. Long-term social isolation increases overall mortality risk by more than one-fourth. Robust social interaction is critical for long-term health, both physically and mentally.

This does not mean that the quarantines were a bad idea—on the contrary, we should have enforced them more aggressively, so as to contain the pandemic faster and ultimately need less time in quarantine. Timing is critical here: Successfully containing the pandemic early is much easier than trying to bring it back under control once it has already spread. When the pandemic began, lockdown might have been able to stop the spread. At this point, vaccines are really our only hope of containment.

But it does mean that if you feel terrible lately, there is a very good reason for this, and you are not alone. Due to forces much larger than any of us can control, forces that even the world’s most powerful governments are struggling to contain, you are currently being deprived of a basic human need.

And especially if you are on your own this Valentine’s Day, remember that there are people who love you, even if they can’t be there with you right now.

A new chapter in my life, hopefully

Jan 17 JDN 2459232

My birthday is coming up soon, and each year around this time I try to step back and reflect on how the previous year has gone and what I can expect from the next one.

Needless to say, 2020 was not a great year for me. The pandemic and its consequences made this quite a bad year for almost everyone. Months of isolation and fear have made us all stressed and miserable, and even with the vaccines coming out the end is still all too far away. Honestly I think I was luckier than most: My work could be almost entirely done remotely, and my income is a fixed stipend, so financially I faced no hardship at all. But isolation still wreaks its toll.

Most of my energy this past year has been spent on the job market. I applied to over 70 different job postings, and from that I received 6 interviews, all but one of which I’ve already finished. Then, if they liked how I did in those interviews, I will be invited to another phase, which in normal times would be a flyout where candidates visit the campus; but due to COVID it’s all being done remotely now. And then, finally, I may actually get some job offers. Statistically I think I will probably get some kind of offer at this point, but I can’t be sure—and that uncertainty is quite nerve-wracking. I may get a job and move somewhere new, or I may not and have to stay here for another year and try again. Both outcomes are still quite probable, and I really can’t plan on either one.

If I do actually get a job, this will open a new chapter in my life—and perhaps I will finally be able to settle down with a permanent career, buy a house, start a family. One downside of graduate school I hadn’t really anticipated is how it delays adulthood: You don’t really feel like you are a proper adult, because you are still in the role of a student for several additional years. I am all too ready to be done with being a student. I feel as though I’ve spent all my life preparing to do things instead of actually doing them, and I am now so very tired of preparing.

I don’t even know for sure what I want to do—I feel disillusioned with academia, I haven’t been able to snare any opportunities in government or nonprofits, and I need more financial security than I could get if I leapt headlong into full-time writing. But I am quite certain that I want to actually do something, and no longer simply be trained and prepared (and continually evaluated on that training and preparation).

I’m even reluctant to do a postdoc, because that also likely means packing up and moving again in a few year (though I would prefer it to remaining here another year).

I have to keep reminding myself that all of this is temporary: The pandemic will eventually be quelled by vaccines, and quarantine procedures will end, and life for most of us will return to normal. Even if I don’t get a job I like this year, I probably will next year; and then I can finally tie off my education with a bow and move on. Even if the first job isn’t permanent, eventually one will be, and at last I’ll be able to settle into a stable adult life.

Much of this has already dragged on longer than I thought it would. Not the job market, which has gone more or less as expected. (More accurately, my level of optimism has jumped up and down like a roller coaster, and on average what I thought would happen has been something like what actually happened so far.) But the pandemic certainly has; the early attempts at lockdown were ineffective, the virus kept spreading worse and worse, and now there are more COVID cases in the US than ever before. Southern California in particular has been hit especially hard, and hospitals here are now overwhelmed just as we feared they might be.

Even the removal of Trump has been far more arduous than I expected. First there was the slow counting of ballots because so many people had (wisely) voted absentee. Then there were the frivolous challenges to the counts—and yes, I mean frivolous in a legal sense, as 61 out of 62 lawsuits were thrown out immediately and the 1 that made it through was a minor technical issue.

And then there was an event so extreme I can barely even fathom that it actually happened: An armed mob stormed the Capitol building, forced Congress to evacuate, and made it inside with minimal resistance from the police. The stark difference in how the police reacted to this attempted insurrection and how they have responded to the Black Lives Matter protests underscores the message of Black Lives Matter better than they ever could have by themselves.

In one sense it feels like so much has happened: We have borne witness to historic events in real-time. But in another sense it feels like so little has happened: Staying home all the time under lockdown has meant that days are alway much the same, and each day blends into the next. I feel somehow unhinged frrom time, at once marveling that a year has passed already, and marveling that so much happened in only a year.

I should soon hear back from these job interviews and have a better idea what the next chapter of my life will be. But I know for sure that I’ll be relieved once this one is over.

I dislike overstatement

Jan 10 JDN 2459225

I was originally planning on titling this post “I hate overstatement”, but I thought that might be itself an overstatement; then I considered leaning into the irony with something like “Overstatement is the worst thing ever”. But no, I think my point best comes across if I exemplify it, rather than present it ironically.

It’s a familiar formula: “[Widespread belief] is wrong! [Extreme alternative view] is true! [Obvious exception]. [Further qualifications]. [Revised, nuanced view that is only slightly different from the widespread belief].”

Here are some examples of the formula (these are not direct quotes but paraphrases of their general views). Note that these are all people I basically agree with, and yet I still find their overstatement annoying:

Bernie Sanders: “Capitalism is wrong! Socialism is better! Well, not authoritarian socialism like the Soviet Union. And some industries clearly function better when privatized. Scandinavian social democracy seems to be the best system.”

Richard Dawkins: “Religion is a delusion! Only atheists are rational! Well, some atheists are also pretty irrational. And most religious people are rational about most things most of the time, and don’t let their religious beliefs interfere too greatly with their overall behavior. Really, what I mean to say that is that God doesn’t exist and organized religion is often harmful.”

Black Lives Matter: “Abolish the police! All cops are bastards! Well, we obviously still need some kind of law enforcement system for dealing with major crimes; we can’t just let serial killers go free. In fact, while there are deep-seated flaws in police culture, we could solve a lot of the most serious problems with a few simple reforms like changing the rules of engagement.”

Sam Harris is particularly fond of this formula, so here is a direct quote that follows the pattern precisely:

“The link between belief and behavior raises the stakes considerably. Some propositions are so dangerous that it may even be ethical to kill people for believing them. This may seem an extraordinary claim, but it merely enunciates an ordinary fact about the world in which we live. Certain beliefs place their adherents beyond the reach of every peaceful means of persuasion, while inspiring them to commit acts of extraordinary violence against others. There is, in fact, no talking to some people. If they cannot be captured, and they often cannot, otherwise tolerant people may be justified in killing them in self-defense. This is what the United States attempted in Afghanistan, and it is what we and other Western powers are bound to attempt, at an even greater cost to ourselves and to innocents abroad, elsewhere in the Muslim world. We will continue to spill blood in what is, at bottom, a war of ideas.”

Somehow in a single paragraph he started with the assertion “It is permissible to punish thoughtcrime with death” and managed to qualify it down to “The Afghanistan War was largely justified”. This is literally the difference between a proposition fundamentally antithetical to everything America stands for, and an utterly uncontroversial statement most Americans agree with. Harris often complains that people misrepresent his views, and to some extent this is true, but honestly I think he does this on purpose because he knows that controversy sells. There’s taking things out of context—and then there’s intentionally writing in a style that will maximize opportunities to take you out of context.

I think the idea behind overstating your case is that you can then “compromise” toward your actual view, and thereby seem more reasonable.

If there is some variable X that we want to know the true value of, and I currently believe that it is some value x1 while you believe that it is some larger value x2, and I ask you what you think, you may not want to tell me x2. Intead you might want to give some number even larger than x2 that you choose to try to make me adjust all the way into adopting your new belief.

For instance, suppose I think the probability of your view being right is p and the probability of my view being right is 1-p. But you think that the probability of your view being right is q > p and the probability of my view being right is 1-q < 1-p.

I tell you that my view is x1. Then I ask you what your view is. What answer should you give?


Well, you can expect that I’ll revise my belief to a new value px + (1-p)x1, where x is whatever answer you give me. The belief you want me to hold is qx2 + (1-q)x1. So your optimal choice is as follows:

qx2 + (1-q)x1 = px + (1-p)x1

x = x1 + q/p(x2-x1)

Since q > p, q/p > 1 and the x you report to me will be larger than your true value x2. You will overstate your case to try to get me to adjust my beliefs more. (Interestingly, if you were less confident in your own beliefs, you’d report a smaller difference. But this seems like a rare case.)

In a simple negotiation over dividing some resource (e.g. over a raise or a price), this is quite reasonable. When you’re a buyer and I’m a seller, our intentions are obvious enough: I want to sell high and you want to buy low. Indeed, the Nash Equilibrium of this game seems to be that we both make extreme offers then compromise on a reasonable offer, all the while knowing that this is exactly what we’re doing.

But when it comes to beliefs about the world, things aren’t quite so simple.

In particular, we have reasons for our beliefs. (Or at least, we’re supposed to!) And evidence isn’t linear. Even when propositions can be placed on a one-dimensional continuum in this way (and quite frankly we shoehorn far too many complex issues onto a simple “left/right” continuum!), evidence that X = x isn’t partial evidence that X = 2x. A strong argument that the speed of light is 3*108 m/s isn’t a weak argument that the speed of light is 3*109 m/s. A compelling reason to think that taxes should be over 30% isn’t even a slight reason to think that taxes should be over 90%.

To return to my specific examples: Seeing that Norway is a very prosperous country doesn’t give us reasons to like the Soviet Union. Recognizing that religion is empirically false doesn’t justify calling all religious people delusional. Reforming the police is obviously necessary, and diverting funds to other social services is surely a worthwhile goal; but law enforcement is necessary and cannot simply be abolished. And defending against the real threat of Islamist terrorism in no way requires us to institute the death penalty for thoughtcrime.

I don’t know how most people response to overstatement. Maybe it really does cause them to over-adjust their beliefs. Hyperbole is a very common rhetorical tactic, and for all I know perhaps it is effective on many people.

But personally, here is my reaction: At the very start, you stated something implausible. That has reduced your overall credibility.

If I continue reading and you then deal with various exceptions and qualifications, resulting in a more reasonable view, I do give you some credit for that; but now I am faced with a dilemma: Either (1) you were misrepresenting your view initially, or (2) you are engaging in a motte-and-bailey doctrine, trying to get me to believe the strong statement while you can only defend the weak statement. Either way I feel like you are being dishonest and manipulative. I trust you less. I am less interested in hearing whatever else you have to say. I am in fact less likely to adopt your nuanced view than I would have been if you’d simply presented it in the first place.

And that’s assuming I have the opportunity to hear your full nuanced version. If all I hear is the sound-byte overstatement, I will come away with an inaccurate assessment of your beliefs. I will have been presented with an implausible claim and evidence that doesn’t support that claim. I will reject your view out of hand, without ever actually knowing what your view truly was.

Furthermore, I know that many others who are listening are not as thoughtful as I am about seeking out detailed context, so even if I know the nuanced version I know—and I think you know—that some people are going to only hear the extreme version.

Maybe what it really comes down to is a moral question: Is this a good-faith discussion where we are trying to reach the truth together? Or is this a psychological manipulation to try to get me to believe what you believe? Am I a fellow rational agent seeking knowledge with you? Or am I a behavior machine that you want to control by pushing the right buttons?

I won’t say that overstatement is always wrong—because that would be an overstatement. But please, make an effort to avoid it whenever you can.

Signaling and the Curse of Knowledge

Jan 3 JDN 2459218

I received several books for Christmas this year, and the one I was most excited to read first was The Sense of Style by Steven Pinker. Pinker is exactly the right person to write such a book: He is both a brilliant linguist and cognitive scientist and also an eloquent and highly successful writer. There are two other books on writing that I rate at the same tier: On Writing by Stephen King, and The Art of Fiction by John Gardner. Don’t bother with style manuals from people who only write style manuals; if you want to learn how to write, learn from people who are actually successful at writing.

Indeed, I knew I’d love The Sense of Style as soon as I read its preface, containing some truly hilarious takedowns of Strunk & White. And honestly Strunk & White are among the best standard style manuals; they at least actually manage to offer some useful advice while also being stuffy, pedantic, and often outright inaccurate. Most style manuals only do the second part.

One of Pinker’s central focuses in The Sense of Style is on The Curse of Knowledge, an all-too-common bias in which knowing things makes us unable to appreciate the fact that other people don’t already know it. I think I succumbed to this failing most greatly in my first book, Special Relativity from the Ground Up, in which my concept of “the ground” was above most people’s ceilings. I was trying to write for high school physics students, and I think the book ended up mostly being read by college physics professors.

The problem is surely a real one: After years of gaining expertise in a subject, we are all liable to forget the difficulty of reaching our current summit and automatically deploy concepts and jargon that only a small group of experts actually understand. But I think Pinker underestimates the difficulty of escaping this problem, because it’s not just a cognitive bias that we all suffer from time to time. It’s also something that our society strongly incentivizes.

Pinker points out that a small but nontrivial proportion of published academic papers are genuinely well written, using this to argue that obscurantist jargon-laden writing isn’t necessary for publication; but he didn’t seem to even consider the fact that nearly all of those well-written papers were published by authors who already had tenure or even distinction in the field. I challenge you to find a single paper written by a lowly grad student that could actually get published without being full of needlessly technical terminology and awkward passive constructions: “A murian model was utilized for the experiment, in an acoustically sealed environment” rather than “I tested using mice and rats in a quiet room”. This is not because grad students are more thoroughly entrenched in the jargon than tenured professors (quite the contrary), nor that grad students are worse writers in general (that one could really go either way), but because grad students have more to prove. We need to signal our membership in the tribe, whereas once you’ve got tenure—or especially once you’ve got an endowed chair or something—you have already proven yourself.

Pinker seems to briefly touch this insight (p. 69), without fully appreciating its significance: “Even when we have an inlkling that we are speaking in a specialized lingo, we may be reluctant to slip back into plain speech. It could betray to our peers the awful truth that we are still greenhorns, tenderfoots, newbies. And if our readers do know the lingo, we might be insulting their intelligence while spelling it out. We would rather run the risk of confusing them while at least appearing to be soophisticated than take a chance at belaboring the obvious while striking them as naive or condescending.”

What we are dealing with here is a signaling problem. The fact that one can write better once one is well-established is the phenomenon of countersignaling, where one who has already established their status stops investing in signaling.

Here’s a simple model for you. Suppose each person has a level of knowledge x, which they are trying to demonstrate. They know their own level of knowledge, but nobody else does.

Suppose that when we observe someone’s knowledge, we get two pieces of information: We have an imperfect observation of their true knowledge which is x+e, the real value of x plus some amount of error e. Nobody knows exactly what the error is. To keep the model as simple as possible I’ll assume that e is drawn from a uniform distribution between -1 and 1.

Finally, assume that we are trying to select people above a certain threshold: Perhaps we are publishing in a journal, or hiring candidates for a job. Let’s call that threshold z. If x < z-1, then since e can never be larger than 1, we will immediately observe that they are below the threshold and reject them. If x > z+1, then since e can never be smaller than -1, we will immediately observe that they are above the threshold and accept them.

But when z-1 < x < z+1, we may think they are above the threshold when they actually are not (if e is positive), or think they are not above the threshold when they actually are (if e is negative).

So then let’s say that they can invest in signaling by putting some amount of visible work in y (like citing obscure papers or using complex jargon). This additional work may be costly and provide no real value in itself, but it can still be useful so long as one simple condition is met: It’s easier to do if your true knowledge x is high.

In fact, for this very simple model, let’s say that you are strictly limited by the constraint that y <= x. You can’t show off what you don’t know.

If your true value x > z, then you should choose y = x. Then, upon observing your signal, we know immediately that you must be above the threshold.

But if your true value x < z, then you should choose y = 0, because there’s no point in signaling that you were almost at the threshold. You’ll still get rejected.

Yet remember before that only those with z-1 < x < z+1 actually need to bother signaling at all. Those with x > z+1 can actually countersignal, by also choosing y = 0. Since you already have tenure, nobody doubts that you belong in the club.

This means we’ll end up with three groups: Those with x < z, who don’t signal and don’t get accepted; those with z < x < z+1, who signal and get accepted; and those with x > z+1, who don’t signal but get accepted. Then life will be hardest for those who are just above the threshold, who have to spend enormous effort signaling in order to get accepted—and that sure does sound like grad school.

You can make the model more sophisticated if you like: Perhaps the error isn’t uniformly distributed, but some other distribution with wider support (like a normal distribution, or a logistic distribution); perhaps the signalling isn’t perfect, but itself has some error; and so on. With such additions, you can get a result where the least-qualified still signal a little bit so they get some chance, and the most-qualified still signal a little bit to avoid a small risk of being rejected. But it’s a fairly general phenomenon that those closest to the threshold will be the ones who have to spend the most effort in signaling.

This reveals a disturbing overlap between the Curse of Knowledge and Impostor Syndrome: We write in impenetrable obfuscationist jargon because we are trying to conceal our own insecurity about our knowledge and our status in the profession. We’d rather you not know what we’re talking about than have you realize that we don’t know what we’re talking about.

For the truth is, we don’t know what we’re talking about. And neither do you, and neither does anyone else. This is the agonizing truth of research that nearly everyone doing research knows, but one must be either very brave, very foolish, or very well-established to admit out loud: It is in the nature of doing research on the frontier of human knowledge that there is always far more that we don’t understand about our subject than that we do understand.

I would like to be more open about that. I would like to write papers saying things like “I have no idea why it turned out this way; it doesn’t make sense to me; I can’t explain it.” But to say that the profession disincentivizes speaking this way would be a grave understatement. It’s more accurate to say that the profession punishes speaking this way to the full extent of its power. You’re supposed to have a theory, and it’s supposed to work. If it doesn’t actually work, well, maybe you can massage the numbers until it seems to, or maybe you can retroactively change the theory into something that does work. Or maybe you can just not publish that paper and write a different one.

Here is a graph of one million published z-scores in academic journals:

It looks like a bell curve, except that almost all the values between -2 and 2 are mysteriously missing.

If we were actually publishing all the good science that gets done, it would in fact be a very nice bell curve. All those missing values are papers that never got published, or results that were excluded from papers, or statistical analyses that were massaged, in order to get a p-value less than the magical threshold for publication of 0.05. (For the statistically uninitiated, a z-score less than -2 or greater than +2 generally corresponds to a p-value less than 0.05, so these are effectively the same constraint.)

I have literally never read a single paper published in an academic journal in the last 50 years that said in plain language, “I have no idea what’s going on here.” And yet I have read many papers—probably most of them, in fact—where that would have been an appropriate thing to say. It’s actually quite a rare paper, at least in the social sciences, that actually has a theory good enough to really precisely fit the data and not require any special pleading or retroactive changes. (Often the bar for a theory’s success is lowered to “the effect is usually in the right direction”.) Typically results from behavioral experiments are bizarre and baffling, because people are a little screwy. It’s just that nobody is willing to stake their career on being that honest about the depth of our ignorance.

This is a deep shame, for the greatest advances in human knowledge have almost always come from people recognizing the depth of their ignorance. Paradigms never shift until people recognize that the one they are using is defective.

This is why it’s so hard to beat the Curse of Knowledge: You need to signal that you know what you’re talking about, and the truth is you probably don’t, because nobody does. So you need to sound like you know what you’re talking about in order to get people to listen to you. You may be doing nothing more than educated guesses based on extremely limited data, but that’s actually the best anyone can do; those other people saying they have it all figured out are either doing the same thing, or they’re doing something even less reliable than that. So you’d better sound like you have it all figured out, and that’s a lot more convincing when you “utilize a murian model” than when you “use rats and mice”.

Perhaps we can at least push a little bit toward plainer language. It helps to be addressing a broader audience: it is both blessing and curse that whatever I put on this blog is what you will read, without any gatekeepers in my path. I can use plainer language here if I so choose, because no one can stop me. But of course there’s a signaling risk here as well: The Internet is a public place, and potential employers can read this as well, and perhaps decide they don’t like me speaking so plainly about the deep flaws in the academic system. Maybe I’d be better off keeping my mouth shut, at least for awhile. I’ve never been very good at keeping my mouth shut.

Once we get established in the system, perhaps we can switch to countersignaling, though even this doesn’t always happen. I think there are two reasons this can fail: First, you can almost always try to climb higher. Once you have tenure, aim for an endowed chair. Once you have that, try to win a Nobel. Second, once you’ve spent years of your life learning to write in a particular stilted, obscurantist, jargon-ridden way, it can be very difficult to change that habit. People have been rewarding you all your life for writing in ways that make your work unreadable; why would you want to take the risk of suddenly making it readable?

I don’t have a simple solution to this problem, because it is so deeply embedded. It’s not something that one person or even a small number of people can really fix. Ultimately we will need to, as a society, start actually rewarding people for speaking plainly about what they don’t know. Admitting that you have no clue will need to be seen as a sign of wisdom and honesty rather than a sign of foolishness and ignorance. And perhaps even that won’t be enough: Because the fact will still remain that knowing what you know that other people don’t know is a very difficult thing to do.

2020 is almost over

Dec27 JDN 2459211

I don’t think there are many people who would say that 2020 was their favorite year. Even if everything else had gone right, the 1.7 million deaths from the COVID pandemic would already make this a very bad year.

As if that weren’t bad enough, shutdowns in response to the pandemic, resulting unemployment, and inadequate fiscal policy responses have in a single year thrown nearly 150 million people back into extreme poverty. Unemployment in the US this year spiked to nearly 15%, its highest level since World War 2. Things haven’t been this bad for the US economy since the Great Depression.

And this Christmas season certainly felt quite different, with most of us unable to safely travel and forced to interact with our families only via video calls. New Year’s this year won’t feel like a celebration of a successful year so much as relief that we finally made it through.

Many of us have lost loved ones. Fortunately none of my immediate friends and family have died of COVID, but I can now count half a dozen acquaintances, friends-of-friends or distant relatives who are no longer with us. And I’ve been relatively lucky overall; both I and my partner work in jobs that are easy to do remotely, so our lives haven’t had to change all that much.

Yet 2020 is nearly over, and already there are signs that things really will get better in 2021. There are many good reasons for hope.


Joe Biden won the election by a substantial margin in both the popular vote and the Electoral College.

There are now multiple vaccines for COVID that have been successfully fast-tracked, and they are proving to be remarkably effective. Current forecasts suggest that we’ll have most of the US population vaccinated by the end of next summer.

Maybe the success of this vaccine will finally convince some of the folks who have been doubting the safety and effectiveness of vaccines in general. (Or maybe not; it’s too soon to tell.)

Perhaps the greatest reason to be hopeful about the future is the fact that 2020 is a sharp deviation from the long-term trend toward a better world. That 150 million people thrown back into extreme poverty needs to be compared against the over 1 billion people who have been lifted out of extreme poverty in just the last 30 years.

Those 1.7 million deaths need to be compared against the fact that global life expectancy has increased from 45 to 73 since 1950. The world population is 7.8 billion people. The global death rate has fallen from over 20 deaths per 1000 people per year to only 7.6 deaths per 1000 people per year. Multiplied over 7.8 billion people, that’s nearly 100 million lives saved every single year by advances in medicine and overall economic development. Indeed, if we were to sustain our current death rate indefinitely, our life expectancy would rise to over 130. There are various reasons to think that probably won’t happen, mostly related to age demographics, but in fact there are medical breakthroughs we might make that would make it possible. Even according to current forecasts, world life expectancy is expected to exceed 80 years by the end of the 21st century.

There have also been some significant environmental milestones this year: Global carbon emissions fell an astonishing 7% in 2020, though much of that was from reduced economic activity in response to the pandemic. (If we could sustain that, we’d cut global emissions in half each decade!) But many other milestones were the product of hard work, not silver linings of a global disaster: Whales returned to the Hudson river, Sweden officially terminated their last coal power plant, and the Great Barrier Reef is showing signs of recovery.

Yes, it’s been a bad year for most of us—most of the world, in fact. But there are many reasons to think that next year will be much better.

Hyper-competition

Dec13 JDN 2459197

This phenomenon has been particularly salient for me the last few months, but I think it’s a common experience for most people in my generation: Getting a job takes an awful lot of work.

Over the past six months, I’ve applied to over 70 different positions and so far gone through 4 interviews (2 by video, 2 by phone). I’ve done about 10 hours of test work. That so far has gotten me no offers, though I have yet to hear from 50 employers. Ahead of me I probably have about another 10 interviews, then perhaps 4 of what would have been flyouts and in-person presentations but instead will be “comprehensive interviews” and presentations conducted online, likely several more hours of test work, and then finally, maybe, if I’m lucky, I’ll get a good offer or two. If I’m unlucky, I won’t, and I’ll have to stick around for another year and do all this over again next year.

Aside from the limitations imposed by the pandemic, this is basically standard practice for PhD graduates. And this is only the most extreme end of a continuum of intensive job search efforts, for which even applying to be a cashier at Target requires a formal application, references, and a personality test.

This wasn’t how things used to be. Just a couple of generations ago, low-wage employers would more or less hire you on the spot, with perhaps a resume or a cursory interview. More prestigious employers would almost always require a CV with references and an interview, but it more or less stopped there. I discussed in an earlier post how much of the difference actually seems to come from our chronic labor surplus.

Is all of this extra effort worthwhile? Are we actually fitting people to better jobs this way? Even if the matches are better, are they enough better to justify all this effort?

It is a commonly-held notion among economists that competition in markets is good, that it increases efficiency and improves outcomes. I think that this is often, perhaps usually, the case. But the labor market has become so intensely competitive, particularly for high-paying positions, that the costs of this competitive effort likely outweigh the benefits.

How could this happen? Shouldn’t the free market correct for such an imbalance? Not necessarily. Here is a simple formal model of how this sort of intensive competition can result in significant waste.

Note that this post is about a formal mathematical model, so it’s going to use a lot of algebra. If you are uninterested in such things, you can read the next two paragraphs and then skip to the conclusions at the end.

The overall argument is straightforward: If candidates are similar in skill level, a complicated application process can make sense from a firm’s perspective, but be harmful from society’s perspective, due to the great cost to the applicants. This can happen because the difficult application process imposes an externality on the workers who don’t get the job.

All right, here is where the algebra begins.

I’ve included each equation as both formatted text and LaTeX.

Consider a competition between two applicants, X and Z.

They are each asked to complete a series of tasks in an application process. The amount of effort X puts into the application is x, and the amount of effort Z puts into the application is z. Let’s say each additional bit of effort has a fixed cost, normalized to 1.

Let’s say that their skills are similar, but not identical; this seems quite realistic. X has skill level hx, and Z has skill level hz.

Getting hired has a payoff for each worker of V. This includes all the expected benefits of the salary, benefits, and working conditions. I’ll assume that these are essentially the same for both workers, which also seems realistic.

The benefit to the employer is proportional to the worker’s skill, so letting h be the skill level of the actually hired worker, the benefit of hiring that worker is hY. The reason they are requiring this application process is precisely because they want to get the worker with the highest h. Let’s say that this application process has a cost to implement, c.

Who will get hired? Well, presumably whoever does better on the application. The skill level will amplify the quality of their output, let’s say proportionally to the effort they put in; so X’s expected quality will be hxx and Z’s expected output will be hzz.

Let’s also say there’s a certain amount of error in the process; maybe the more-qualified candidate will sleep badly the day of the interview, or make a glaring and embarrassing typo on their CV. And quite likely the quality of application output isn’t perfectly correlated with the quality of actual output once hired. To capture all this, let’s say that having more skill and putting in more effort only increases your probability of getting the job, rather than actually guaranteeing it.

In particular, let’s say that the probability of X getting hired is P[X] = hxx/(hxx + hzz).

\[ P[X] = \frac{h_x}{h_x x + h_z z} \]

This results in a contest function, a type of model that I’ve discussed in some earlier posts in a rather different context.


The expected payoff for worker X is:

E[Ux] = hxx/(hxx + hzz) V – x

\[ E[U_x] = \frac{h_x x}{h_x x + h_z z} V – x \]

Maximizing this with respect to the choice of effort x (which is all that X can control at this point) yields:

hxhzz V = (hxx + hzz)2

\[ h_x h_z x V = (h_x x + h_z z)^2 \]

A similar maximization for worker Z yields:

hxhzx V = (hxx + hzz)2

\[ h_x h_z z V = (h_x x + h_z z)^2 \]

It follows that x=z, i.e. X and Z will exert equal efforts in Nash equilibrium. Their probability of success will then be contingent entirely on their skill levels:

P[X] = hx/(hx + hz).

\[ P[X] = \frac{h_x}{h_x + h_y} \]

Substituting that back in, we can solve for the actual amount of effort:

hxhzx V = (hx + hz)2x2

\[h_x h_z x V = (h_x + h_z)^2 x^2 \]

x = hxhzV/(hx + hz)2

\[ x = \frac{h_x h_z}{h_x + h_z} V \]

Now let’s see what that gives for the expected payoffs of the firm and the workers. This is worker X’s expected payoff:

E[Ux] = hx/(hx + hz) V – hxhzV/(hx + hz)2 = (hx/(hx + hz))2 V

\[ E[U_x] = \frac{h_x}{h_x + h_z} V – \frac{h_x h_z}{(h_x + h_z)^2} V = \left( \frac{h_x}{h_x + h_z}\right)^2 V \]

Worker Z’s expected payoff is the same, with hx and hz exchanged:

E[Uz] = (hz/(hx + hz))2 V

\[ E[U_z] = \left( \frac{h_z}{h_x + h_z}\right)^2 V \]

What about the firm? Their expected payoff is the the probability of hiring X, times the value of hiring X, plus the probability of hiring Z, times the value of hiring Z, all minus the cost c:

E[Uf] = hx/(hx + hz) hx Y + hz/(hx + hz) hz Y – c= (hx2 + hz2)/(hx + hz) Y – c

\[ E[U_f] = \frac{h_x}{h_x + h_z} h_x Y + \frac{h_z}{h_x + h_z} h_z Y – c = \frac{h_x^2 + h_z^2}{h_x + h_z} Y – c\]

To see whether the application process was worthwhile, let’s compare against the alternative of simply flipping a coin and hiring X or Z at random. The probability of getting hired is then 1/2 for each candidate.

Expected payoffs for X and Z are now equal:

E[Ux] = E[Uz] = V/2

\[ E[U_x] = E[U_z] = \frac{V}{2} \]

The expected payoff for the firm can be computed the same as before, but now without the cost c:

E[Uf] = 1/2 hx Y + 1/2 hz Y = (hx + hz)/2 Y

\[ E[U_f] = \frac{1}{2} h_x Y + \frac{1}{2} h_z Y = \frac{h_x + h_z}{2} Y \]

This has a very simple interpretation: The expected value to the firm is just the average quality of the two workers, times the overall value of the job.

Which of these two outcomes is better? Well, that depends on the parameters, of course. But in particular, it depends on the difference between hx and hz.

Consider two extremes: In one case, the two workers are indistinguishable, and hx = hz = h. In that case, the payoffs for the hiring process reduce to the following:

E[Ux] = E[Uz] = V/4

\[ E[U_x] = E[U_z] = \frac{V}{4} \]

E[Uf] = h Y – c

\[ E[U_f] = h Y – c \]

Compare this against the payoffs for hiring randomly:

E[Ux] = E[Uz] = V/2

\[ E[U_x] = E[U_z] = \frac{V}{2} \]

E[Uf] = h Y

\[ E[U_f] = h Y \]

Both the workers and the firm are strictly better off if the firm just hires at random. This makes sense, since the workers have identical skill levels.

Now consider the other extreme, where one worker is far better than the other; in fact, one is nearly worthless, so hz ~ 0. (I can’t do exactly zero because I’d be dividing by zero, but let’s say one is 100 times better or something.)

In that case, the payoffs for the hiring process reduce to the following:

E[Ux] = V

E[Uz] = 0

\[ E[U_x] = V \]

\[ E[U_z] = 0 \]

X will definitely get the job, so X is much better off.

E[Uf] = hx Y – c

\[ E[U_f] = h_x Y – c \]

If the firm had hired randomly, this would have happened instead:

E[Ux] = E[Uz] = V/2

\[ E[U_x] = E[U_z] = \frac{V}{2} \]

E[Uf] = hY/2

\[ E[U_f] = \frac{h}{2} Y \]

As long as c < hY/2, both the firm and the higher-skill worker are better off in this scenario. (The lower-skill worker is worse off, but that’s not surprising.) The total expected benefit for everyone is also higher in this scenario.


Thus, the difference in skill level between the applicants is vital. If candidates are very different in skill level, in a way that the application process can accurately measure, then a long and costly application process can be beneficial, not only for the firm but also for society as a whole.

In these extreme examples, it was either not worth it for the firm, or worth it for everyone. But there is an intermediate case worth looking at, where the long and costly process can be worth it for the firm, but not for society as a whole. I will call this case hyper-competition—a system that is so competitive it makes society overall worse off.

This inefficient result occurs precisely when:
c < (hx2 + hz2)/(hx + hz) Y – (hx + hz)/2 Y < c + (hx/(hx + hz))2 V + (hz/(hx + hz))2 V

\[ c < \frac{h_x^2 + h_z^2}{h_x + h_z} Y – \frac{h_x + h_z}{2} Y < c + \left( \frac{h_x}{h_x + h_z}\right)^2 V + \left( \frac{h_z}{h_x + h_z}\right)^2 V \]

This simplifies to:

c < (hx – hz)2/(2hx + 2hz) Y < c + (hx2 + hz2)/(hx + hz)2 V

\[ c < \frac{(h_x – h_z)^2}{2 (h_x + h_z)} Y < c + \frac{(h_x^2 + h_z^2)}{(h_x+h_z)^2} V \]

If c is small, then we are interested in the case where:

(hx – hz)2 Y/2 < (hx2 + hz2)/(hx + hz) V

\[ \frac{(h_x – h_z)^2}{2} Y < \frac{h_x^2 + h_z^2}{h_x + h_z} V \]

This is true precisely when the difference hx – hz is small compared to the overall size of hx or hz—that is, precisely when candidates are highly skilled but similar. This is pretty clearly the typical case in the real world. If the candidates were obviously different, you wouldn’t need a competitive process.

For instance, suppose that hx = 10 and hz = 8, while V = 180, Y = 20 and c = 1.

Then, if we hire randomly, these are the expected payoffs:

E[Uf] = (hx + hz)/2 Y = 180

E[Ux] = E[Uz] = V/2 = 90

If we use the complicated hiring process, these are the expected payoffs:

E[Ux] = (hx/(hx + hz))2 V = 55.5

E[Uz] = (hz/(hx + hz))2 V = 35.5

E[Uf] = (hx2 + hz2)/(hx + hz) Y – c = 181

The firm gets a net benefit of 1, quite small; while the workers face a far larger total expected loss of 90. And these candidates aren’t that similar: One is 25% better than the other. Yet because the effort expended in applying was so large, even this improvement in quality wasn’t worth it from society’s perspective.

This conclude’s the algebra for today, if you’ve been skipping it.

In this model I’ve only considered the case of exactly two applicants, but this can be generalized to more applicants, and the effect only gets stronger: Seemingly-large differences in each worker’s skill level can be outweighed by the massive cost of making so many people work so hard to apply and get nothing to show for it.

Thus, hyper-competition can exist despite apparently large differences in skill. Indeed, it is precisely the typical real-world scenario with many applicants who are similar that we expect to see the greatest inefficiencies. In the absence of intervention, we should expect markets to get this wrong.

Of course, we don’t actually want employers to hire randomly, right? We want people who are actually qualified for their jobs. Yes, of course; but you can probably assess that with nothing more than a resume and maybe a short interview. Most employers are not actually trying to find qualified candidates; they are trying to sift through a long list of qualified candidates to find the one that they think is best qualified. And my suspicion is that most of them honestly don’t have good methods of determining that.

This means that it could be an improvement for society to simply ban long hiring processes like these—indeed, perhaps ban job interviews altogether, as I can hardly think of a more efficient mechanism for allowing employers to discriminate based on race, gender, age, or disability than a job interview. Just collect a resume from each applicant, remove the ones that are unqualified, and then roll a die to decide which one you hire.

This would probably make the fit of workers to their jobs somewhat worse than the current system. But most jobs are learned primarily through experience anyway, so once someone has been in a job for a few years it may not matter much who was hired originally. And whatever cost we might pay in less efficient job matches could be made up several times over by the much faster, cheaper, easier, and less stressful process of applying for jobs.

Indeed, think for a moment of how much worse it feels being turned down for a job after a lengthy and costly application process that is designed to assess your merit (but may or may not actually do so particularly well), as opposed to simply finding out that you lost a high-stakes die roll. Employers could even send out letters saying one of two things: “You were rejected as unqualifed for this position.” versus “You were qualified, but you did not have the highest die roll.” Applying for jobs already feels like a crapshoot; maybe it should literally be one.

People would still have to apply for a lot of jobs—actually, they’d probably end up applying for more, because the lower cost of applying would attract more applicants. But since the cost is so much lower, it would still almost certainly be easier to do a job search than it is in the current system. In fact, it could largely be automated: simply post your resume on a central server and the system matches you with employers’ requirements and then randomly generates offers. Employers and prospective employees could fill out a series of forms just once indicating what they were looking for, and then the system could do the rest.

What I find most interesting about this policy idea is that it is in an important sense anti-meritocratic. We are in fact reducing the rewards for high levels of skill—at least a little bit—in order to improve society overall and especially for those with less skill. This is exactly the kind of policy proposal that I had hoped to see from a book like The Meritocracy Trap, but never found there. Perhaps it’s too radical? But the book was all about how we need fundamental, radical change—and then its actual suggestions were simple, obvious, and almost uncontroversial.

Note that this simplified process would not eliminate the incentives to get major, verifiable qualifications like college degrees or years of work experience. In fact, it would focus the incentives so that only those things matter, instead of whatever idiosyncratic or even capricious preferences HR agents might have. There would be no more talk of “culture fit” or “feeling right for the job”, just: “What is their highest degree? How many years have they worked in this industry?” I suppose this is credentialism, but in a world of asymmetric information, I think credentialism may be our only viable alternative to nepotism.

Of course, it’s too late for me. But perhaps future generations may benefit from this wisdom.

Adversity is not a gift

Nov 29 JDN 2459183

For the last several weeks I’ve been participating in a program called “positive intelligence” (which they abbreviate “PQ” even though that doesn’t make sense); it’s basically a self-help program that is designed to improve mood and increase productivity. I am generally skeptical of such things, and I could tell from the start that it was being massively oversold, but I had the opportunity to participate for free, and I looked into the techniques involved and most of them seem to be borrowed from cognitive-behavioral therapy and mindfulness meditation.

Overall, I would say that the program has had small but genuine benefits for me. I think the most helpful part was actually getting the chance to participate in group sessions (via Zoom of course) with others also going through the program. That kind of mutual social support can make a big difference. The group I joined was all comprised of fellow economists (some other grad students, some faculty), so we had a lot of shared experiences.

Some of the techniques feel very foolish, and others just don’t seem to work for me; but I did find at least some of the meditation techniques (which they annoyingly insist on calling by the silly name “PQ reps”) have helped me relax.

But there’s one part of the PQ program in particular that I just can’t buy into, and this is the idea that adversity is a gift and an opportunity.

They call it the “Sage perspective”: You observe the world without judging what is good or bad, and any time you think something is bad, you find a way to transform it into a gift and an opportunity. The claim is that everything—or nearly everything—that happens to you can make you better off. There’s a lot of overlap here with the attitude “Everything happens for a reason”.

I don’t doubt that sincerely believing this would make you happier. Nevertheless, it is obviously false.

If indeed adversity were a gift, we would seek it out. If getting fired or going bankrupt or getting sick were a gift and an opportunity, we’d work to make these things happen.

Yes, it’s true that sometimes an event which seems bad at the time can turn out to have good consequences in the long run. This is simply because we are unable to foresee all future ramifications. Sometimes things turn out differently than you think they will. But most of the time, when something seems bad, it is actually bad.

There might be some small amount of discomfort or risk that would be preferable to a life of complete safety and complacency; but we are perfectly capable of seeking out whatever discomfort or risk we choose. Most of us live with far more discomfort and risk than we would prefer, and simply have no choice in the matter.

If adversity were a gift, people would thank you for giving it to them. “Thanks for dumping me!” “Thanks for firing me!” “Thanks for punching me!” These aren’t the sort of thing we hear very often (at least not sincerely).

I think this is fairly obvious, honestly, so I won’t belabor it any further. But it raises a question: Is there a way to salvage the mental health benefits of this attitude while abandoning its obvious falsehood?

“Everything happens for a reason” doesn’t work; we live in a universe of deep randomness, ruled by the blind idiot gods of natural law.

“Every cloud has a silver lining” is better; but clearly not every bad thing has an upside, or if it does the upside can be so small as to be utterly negligible. (What was the upside of Rwandan genocide?) Restricted to ordinary events like getting fired this one works pretty well; but it obviously fails for the most extreme traumas, and doesn’t seem particularly helpful for the death of a loved one either.

“What doesn’t kill me makes me stronger” is better still, but clearly not true in every case; some bad events that don’t actually kill us can traumatize us and make the rest of our lives harder. Perhaps “What doesn’t permanently damage me makes me stronger”?

I think the version of this attitude that I have found closest to the truth is “Everything is raw material”. Sometimes bad things just happen: Bad luck, or bad actions, can harm just about anyone at just about any time. But it is within our power to decide how we will respond to what happens to us, and wallowing in despair is almost never the best response.

Thus, while it is foolish to see adversity as a gift, it is not so foolish to see it as an opportunity. Don’t try to pretend that bad things aren’t bad. There’s no sense in denying that we would prefer some outcomes over others, and we feel hurt or disappointed when things don’t turn out how we wanted. Yet even what is bad can still contain within it chances to learn or make things better.

What’s wrong with “should”?

Nov 8 JDN 2459162

I have been a patient in cognitive behavioral therapy (CBT) for many years now. The central premise that thoughts can influence emotions is well-founded, and the results of CBT are empirically well supported.

One of the central concepts in CBT is cognitive distortions: There are certain systematic patterns in how we tend to think, which often results in beliefs and emotions that are disproportionate with reality.

Most of the cognitive distortions CBT deals with make sense to me—and I am well aware that my mind applies them frequently: All-or-nothing, jumping to conclusions, overgeneralization, magnification and minimization, mental filtering, discounting the positive, personalization, emotional reasoning, and labeling are all clearly distorted modes of thinking that nevertheless are extremely common.

But there’s one “distortion” on CBT lists that always bothers me: “should statements”.

Listen to this definition of what is allegedly a cognitive distortion:

Another particularly damaging distortion is the tendency to make “should” statements. Should statements are statements that you make to yourself about what you “should” do, what you “ought” to do, or what you “must” do. They can also be applied to others, imposing a set of expectations that will likely not be met.

When we hang on too tightly to our “should” statements about ourselves, the result is often guilt that we cannot live up to them. When we cling to our “should” statements about others, we are generally disappointed by their failure to meet our expectations, leading to anger and resentment.

So any time we use “should”, “ought”, or “must”, we are guilty of distorted thinking? In other words, all of ethics is a cognitive distortion? The entire concept of obligation is a symptom of a mental disorder?

Different sources on CBT will define “should statements” differently, and sometimes they offer a more nuanced definition that doesn’t have such extreme implications:

Individuals thinking in ‘shoulds’, ‘oughts; or ‘musts’ have an ironclad view of how they and others ‘should’ and ‘ought’ to be. These rigid views or rules can generate feels of anger, frustration, resentment, disappointment and guilt if not followed.

Example: You don’t like playing tennis but take lessons as you feel you ‘should’, and that you ‘shouldn’t’ make so many mistakes on the court, and that your coach ‘ought to’ be stricter on you. You also feel that you ‘must’ please him by trying harder.

This is particularly problematic, I think, because of the All-or-Nothing distortion which does genuinely seem to be common among people with depression: Unless you are very clear from the start about where to draw the line, our minds will leap to saying that all statements involving the word “should” are wrong.

I think what therapists are trying to capture with this concept is something like having unrealistic expectations, or focusing too much on what could or should have happened instead of dealing with the actual situation you are in. But many seem to be unable to articulate that clearly, and instead end up asserting that entire concept of moral obligation is a cognitive distortion.

There may be a deeper error here as well: The way we study mental illness doesn’t involve enough comparison with the control group. Psychologists are accustomed to asking the question, “How do people with depression think?”; but they are not accustomed to asking the question, “How do people with depression think compared to people who don’t?” If you want to establish that A causes B, it’s not enough to show that those with B have A; you must also show that those who don’t have B also don’t have A.

This is an extreme example for illustration, but suppose someone became convinced that depression is caused by having a liver. They studied a bunch of people with depression, and found that they all had livers; hypothesis confirmed! Clearly, we need to remove the livers, and that will cure the depression.

The best example I can find of a study that actually asked that question compared nursing students and found that cognitive distortions explain about 20% of the variance in depression. This is a significant amount—but still leaves a lot unexplained. And most of the research on depression doesn’t even seem to think to compare against people without depression.

My impression is that some cognitive distortions are genuinely more common among people with depression—but not all of them. There is an ongoing controversy over what’s called the depressive realism effect, which is the finding that in at least some circumstances the beliefs of people with mild depression seem to be more accurate than the beliefs of people with no depression at all. The result is controversial both because it seems to threaten the paradigm that depression is caused by distortions, and because it seems to be very dependent on context; sometimes depression makes people more accurate in their beliefs, other times it makes them less accurate.

Overall, I am inclined to think that most people have a variety of cognitive distortions, but we only tend to notice when those distortions begin causing distress—such when are they involved in depression. Human thinking in general seems to be a muddled mess of heuristics, and the wonder is that we function as well as we do.

Does this mean that we should stop trying to remove cognitive distortions? Not at all. Distorted thinking can be harmful even if it doesn’t cause you distress: The obvious example is a fanatical religious or political belief that leads you to harm others. And indeed, recognizing and challenging cognitive distortions is a highly effective treatment for depression.

Actually I created a simple cognitive distortion worksheet based on the TEAM-CBT approach developed by David Burns that has helped me a great deal in a remarkably short time. You can download the worksheet yourself and try it out. Start with a blank page and write down as many negative thoughts as you can, and then pick 3-5 that seem particularly extreme or unlikely. Then make a copy of the cognitive distortion worksheet for each of those thoughts and follow through it step by step. Particularly do not ignore the step “This thought shows the following good things about me and my core values:”; that often feels the strangest, but it’s a critical part of what makes the TEAM-CBT approach better than conventional CBT.

So yes, we should try to challenge our cognitive distortions. But the mere fact that a thought is distressing doesn’t imply that it is wrong, and giving up on the entire concept of “should” and “ought” is throwing out a lot of babies with that bathwater.

We should be careful about labeling any thoughts that depressed people have as cognitive distortions—and “should statements” is a clear example where many psychologists have overreached in what they characterize as a distortion.