Noah Smith

Sheepskin effect doesn’t prove much

pnrj education, public policy , , , , , , , , ,

Sep 20 JDN 2459113

The sheepskin effect is the observation that the increase in income from graduating from college after four years, relative going through college for three years, is much higher than the increase in income from simply going through college for three years instead of two.

It has been suggested that this provides strong evidence that education is primarily due to signaling, and doesn’t provide any actual value. In this post I’m going to show why this view is mistaken. The sheepskin effect in fact tells us very little about the true value of college. (Noah Smith actually made a pretty decent argument that it provides evidence against signaling!)

To see this, consider two very simple models.

In both models, we’ll assume that markets are competitive but productivity is not directly observable, so employers sort you based on your education level and then pay a wage equal to the average productivity of people at your education level, compensated for the cost of getting that education.

Model 1:

In this model, people all start with the same productivity, and are randomly assigned by their life circumstances to go to either 0, 1, 2, 3, or 4 years of college. College itself has no long-term cost.

The first year of college you learn a lot, the next couple of years you don’t learn much because you’re trying to find your way, and then in the last year of college you learn a lot of specialized skills that directly increase your productivity.

So this is your productivity after x years of college:

Years of collegeProductivity
010
117
222
325
431

We assumed that you’d get paid your productivity, so these are also your wages.

The increase in income each year goes from +7, to +5, to +3, then jumps up to +6. So if you compare the 4-year-minus-3-year gap (+6) with the 3-year-minus-2-year gap (+3), you get a sheepskin effect.

Model 2:

In this model, college is useless and provides no actual benefits. People vary in their intrinsic productivity, which is also directly correlated with the difficulty of making it through college.

In particular, there are five types of people:

TypeProductivityCost per year of college
0108
1116
2144
3193
4310

The wages for different levels of college education are as follows:

Years of collegeWage
010
117
222
325
431

Notice that these are exactly the same wages as in scenario 1. This is of course entirely intentional. In a moment I’ll show why this is a Nash equilibrium.

Consider the choice of how many years of college to attend. You know your type, so you know the cost of college to you. You want to maximize your net benefit, which is the wage you’ll get minus the total cost of going to college.

Let’s assume that if a given year of college isn’t worth it, you won’t try to continue past it and see if more would be.

For a type-0 person, they could get 10 by not going to college at all, or 17-(1)(8) = 9 by going for 1 year, so they stop.

For a type-1 person, they could get 10 by not going to college at all, or 17-(1)(6) = 11 by going for 1 year, or 22-(2)(6) = 10 by going for 2 years, so they stop.

Filling out all the possibilities yields this table:

Years \ Type01234
01010101010
1911131417
2
10141622
3

131925
4


1930

I’d actually like to point out that it was much harder to find numbers that allowed me to make the sheepskin effect work in the second model, where education was all signaling. In the model where education provides genuine benefit, all I need to do is posit that the last year of college is particularly valuable (perhaps because high-level specialized courses are more beneficial to productivity). I could pretty much vary that parameter however I wanted, and get whatever magnitude of sheepskin effect I chose.

For the signaling model, I had to carefully calibrate the parameters so that the costs and benefits lined up just right to make sure that each type chose exactly the amount of college I wanted them to choose while still getting the desired sheepskin effect. It took me about two hours of very frustrating fiddling just to get numbers that worked. And that’s with the assumption that someone who finds 2 years of college not worth it won’t consider trying for 4 years of college (which, given the numbers above, they actually might want to), as well as the assumption that when type-3 individuals are indifferent between staying and dropping out they drop out.

And yet the sheepskin effect is supposed to be evidence that the world works like the signaling model?

I’m sure a more sophisticated model could make the signaling explanation a little more robust. The biggest limitation of these models is that once you observe someone’s education level, you immediately know their true productivity, whether it came from college or not. Realistically we should be allowing for unobserved variation that can’t be sorted out by years of college.

Maybe it seems implausible that the last year of college is actually more beneficial to your productivity than the previous years. This is probably the intuition behind the idea that sheepskin effects are evidence of signaling rather than genuine learning.

So how about this model?

Model 3:

As in the second model, there are four types of people, types 0, 1, 2, 3, and 4. They all start with the same level of productivity, and they have the same cost of going to college; but they get different benefits from going to college.

The problem is, people don’t start out knowing what type they are. Nor can they observe their productivity directly. All they can do is observe their experience of going to college and then try to figure out what type they must be.

Type 0s don’t benefit from college at all, and they know they are type 0; so they don’t go to college.

Type 1s benefit a tiny amount from college (+1 productivity per year), but don’t realize they are type 1s until after one year of college.

Type 2s benefit a little from college (+2 productivity per year), but don’t realize they are type 2s until after two years of college.

Type 3s benefit a moderate amount from college (+3 productivity per year), but don’t realize they are type 3s until after three years of college.

Type 4s benefit a great deal from college (+5 productivity per year), but don’t realize they are type 4s until after three years of college.

What then will happen? Type 0s will not go to college. Type 1s will go one year and then drop out. Type 2s will go two years and then drop out. Type 3s will go three years and then drop out. And type 4s will actually graduate.

That results in the following before-and-after productivity:

TypeProductivity before collegeYears of collegeProductivity after college
010010
110111
210214
310319
410430

If each person is paid a wage equal to their productivity, there will be a huge sheepskin effect; wages only go up +1 for 1 year, +3 for 2 years, +5 for 3 years, but then they jump up to +11 for graduation. It appears that the benefit of that last year of college is more than the other three combined. But in fact it’s not; for any given individual, the benefits of college are the same each year. It’s just that college is more beneficial to the people who decided to stay longer.

And I could of course change that assumption too, making the early years more beneficial, or varying the distribution of types, or adding more uncertainty—and so on. But it’s really not hard at all to make a model where college is beneficial and you observe a large sheepskin effect.

In reality, I am confident that some of the observed benefit of college is due to sorting—not the same thing as signaling—rather than the direct benefits of education. The earnings advantage of going to a top-tier school may be as much about the selection of students as they are the actual quality of the education, since once you control for measures of student ability like GPA and test scores those benefits drop dramatically.

Moreover, I agree that it’s worth looking at this: Insofar as college is about sorting or signaling, it’s wasteful from a societal perspective, and we should be trying to find more efficient sorting mechanisms.

But I highly doubt that all the benefits of college are due to sorting or signaling; there definitely are a lot of important things that people learn in college, not just conventional academic knowledge like how to do calculus, but also broader skills like how to manage time, how to work in groups, and how to present ideas to others. Colleges also cultivate friendships and provide opportunities for networking and exposure to a diverse community. Judging by voting patterns, I’m going to go out on a limb and say that college also makes you a better citizen, which would be well worth it by itself.

The truth is, we don’t know exactly why college is beneficial. We certainly know that it is beneficial: Unemployment rates and median earnings are directly sorted by education level. Yes, even PhDs in philosophy and sociology have lower unemployment and higher incomes (on average) than the general population. (And of course PhDs in economics do better still.)

What would a better job market look like?

pnrj personal, public policy , , , , , , , , ,

Sep 13 JDN 2459106

I probably don’t need to tell you this, but getting a job is really hard. Indeed, much harder than it seems like it ought to be.

Having all but completed my PhD, I am now entering the job market. The job market for economists is quite different from the job market most people deal with, and these differences highlight some potential opportunities for improving job matching in our whole economy—which, since employment is such a large part of our lives, could have wide-ranging benefits for our society.

The most obvious difference is that the job market for economists is centralized: Job postings are made through the American Economic Association listing of Job Openings for Economists (often abbrievated AEA JOE); in a typical year about 4,000 jobs are posted there. All of them have approximately the same application deadline, near the end of the year. Then, after applying to various positions, applicants get interviewed in rapid succession, all at the annual AEA conference. Then there is a matching system, where applicants get to send two “signals” indicating their top choices and then offers are made.

This year of course is different, because of COVID-19. The conference has been canceled, with all of its presentations moved online; interviews will also be conducted online. Perhaps more worrying, the number of postings has been greatly reduced, and based on past trends may be less than half of the usual number. (The number of applicants may also be reduced, but it seems unlikely to drop as much as the number of postings does.)

There are a number of flaws in even this system. First, it’s too focused on academia; very few private-sector positions use the AEA JOE system, and almost no government positions do. So those of us who are not so sure we want to stay in academia forever end up needing to deal with both this system and the conventional system in parallel. Second, I don’t understand why they use this signaling system and not a deferred-acceptance matching algorithm. I should be able to indicate more about my preferences than simply what my top two choices are—particularly when most applicants apply to over 100 positions. Third, it isn’t quite standardized enough—some positions do have earlier deadlines or different application materials, so you can’t simply put together one application packet and send it to everyone at once.

Still, it’s quite obvious that this system is superior to the decentralized job market that most people deal with. Indeed, this becomes particularly obvious when one is participating in both markets at once, as I am. The decentralized market has a wide range of deadlines, where upon seeing an application you may need to submit to it within that week, or you may have several months to respond. Nearly all applications require a resume, but different institutions will expect different content on it. Different applications may require different materials: Cover letters, references, writing samples, and transcripts are all things that some firms will want and others won’t.

Also, this is just my impression from a relatively small sample, but I feel like the AEA JOE listings are more realistic, in the following sense: They don’t all demand huge amounts of prior experience, and those that do ask for prior experience are either high-level positions where that’s totally reasonable, or are willing to substitute education for experience. For private-sector job openings you basically have to subtract three years from whatever amount of experience they say they require, because otherwise you’d never have anywhere you could apply to. (Federal government jobs are a weird case here; they all say they require a lot of experience at a specific government pay grade, but from talking with those who have dealt with the system before, they are apparently willing to make lots of substitutions—private-sector jobs, education, and even hobbies can sometimes substitute.)

I think this may be because the decentralized market has to some extent unraveled. The job market is the epitome of a matching market; unraveling in a matching market occurs when there is fierce competition for a small number of good candidates or, conversely, a small number of good openings. Each firm has the incentive to make a binding offer earlier than the others, with a short deadline so that candidates don’t have time to shop around. As firms compete with each other, they start making deadlines earlier and earlier until candidates feel like they are in a complete crapshoot: An offer made on Monday might be gone by Friday, and you have no way of knowing if you should accept it now or wait for a better one to come along. This is a Tragedy of the Commons: Given what other firms are doing, each firm benefits from making an earlier binding offer. But once they all make early offers, that benefit disappears and the result just makes the whole system less efficient.

The centralization of the AEA JOE market prevents this from happening: Everyone has common deadlines and does their interviews at the same time. Each institution may be tempted to try to break out of the constraints of the centralized market, but they know that if they do, they will be punished by receiving fewer applicants.

The fact that the centralized market is more efficient is likely a large part of why economics PhDs have the lowest unemployment rate of any PhD graduates and nearly the lowest unemployment rate of any job sector whatsoever. In some sense we should expect this: If anyone understands how to make employment work, it should be economists. Noah Smith wrote in 2013 (and I suppose I took it to heart): “If you get a PhD, get an economics PhD.” I think PhD graduates are the right comparison group here: If we looked at the population as a whole, employment rates and salaries for economists look amazing, but that isn’t really fair since it’s so much harder to become an economist than it is to get most other jobs. But I don’t think it’s particularly easier to get a PhD in physics or biochemistry than to get one in economics, and yet economists still have a lower unemployment rate than physicists or biochemists. (Though it’s worth noting that any PhD—yes, even in the humanities—will give you a far lower risk of unemployment than the general population.) The fact that we have AEA JOE and they don’t may be a major factor here.


So, here’s my question: Why don’t we do this in more job markets? It would be straightforward enough to do this for all PhD graduates, at least—actually my understanding is that some other disciplines do have centralized markets similar to the one in economics, but I’m not sure how common this is.

The federal government could relatively easily centralize its own job market as well; maybe not for positions that need to be urgently filled, but anything that can wait several months would be worth putting into a centralized system that has deadlines once or twice a year.

But what about the private sector, which after all is where most people work? Could we centralize that system as well?

It’s worth noting the additional challenges that immediately arise: Many positions need to be filled immediately, and centralization would make that impossible. There are thousands of firms that would need to be coordinated (there are at least 100,000 firms in the US with 100 or more employees). There are millions of different jobs to be filled, requiring a variety of different skills. In an average month over 5 million jobs are filled in the United States.

Most people want a job near where they live, so part of the solution might be to centralize only jobs within a certain region, such as a particular metro area. But if we are limited to open positions of a particular type within a particular city, there might not be enough openings at any given time to be worth centralizing. And what about applicants who don’t care so much about geography? Should they be applying separately to each regional market?

Yet even with all this in mind, I think some degree of centralization would be feasible and worthwhile. If nothing else, I think standardizing deadlines and application materials could make a significant difference—it’s far easier to apply to many places if they all use the same application and accept them at the same time.

Another option would be to institute widespread active labor market policies, which are a big part of why #ScandinaviaIsBetter. Denmark especially invests heavily in such programs, which provide training and job matching for unemployed citizens. It is no coincidence that Denmark has kept their unemployment rate under 7% even through the worst of the Great Recession. The US unemployment rate fluctuates wildly with the business cycle, while most of Europe has steadier but higher unemployment. Indeed, the lowest unemployment rates in France over the last 30 years have exceeded the highest rates in Denmark over the same period. Denmark spends a lot on their active labor market programs, but I think they’re getting their money’s worth.

Such a change would make our labor markets more efficient, matching people to jobs that fit them better, increasing productivity and likely decreasing turnover. Wages probably wouldn’t change much, but working in a better job for the same wage is still a major improvement in your life. Indeed, job satisfaction is one of the strongest predictors of life satisfaction, which isn’t too surprising given how much of our lives we spend at work.