Tips for Scrutinizing Data on College Value

Speaking at the Education Writers Association’s 2021 National Seminar, the researchers urged reporters to be careful about drawing sweeping conclusions when using what are still very limited and imperfect datasets. But journalists can use data to try to assess some aspects of college value as long as they’re honest and up front in their reporting about what exactly the data is, who it reflects, and who it excludes, they advised.

“College value is a big deal,” Baker said.

Think broadly about “value,” both speakers urged journalists.” Kelchen, for example, warned against ‘overreliance on post-college earnings. “You’ll also want to take debt and repayment into account, and there are fields of study where you don’t care about earnings,” he said.

While individuals might value their college experience because “it helped me get a job, or I met my best friend or partner… it is not just about individuals,” Baker said.  Colleges add value to society by broadening the tax base, improving public health, improving local economies and providing other civic benefits that are often “really difficult to measure,” she added.

In the session, which was moderated by Chris Quintana, an education reporter for USA Today, the researchers provided some general advice on using college value data, and dove into specifics for a few of the most important datasets.

First: Ask who and what the college data covers

Whatever dataset you use can be biased because there are likely some important people missing. Some college value surveys are based, for example, on students who filled out the Free Application for Federal Student Aid (FAFSA).

So those sets exclude “students who are unaware and lack resources; students who are undocumented; students who are pinching pennies at a community college to strategically save their financial aid eligibility for a more expensive four-year college, or students who are wealthy enough to pay without federal aid,” Baker said.

Additionally, different student debt datasets can produce very different numbers because some separate out undergraduate and graduate debt, while others combine them. People with graduate degrees tend to have much higher cumulative debt loads than those who only hold a bachelor’s, but they also tend to earn higher incomes.

And broad numbers, such as averages or medians, can hide variations caused by factors such as race and gender discrimination. Average earnings data, for example, often don’t account for structural barriers like racism, Baker said.

The time period covered by the datasets can also affect the conclusions you draw. Many datasets only provide earnings for those who recently graduated or left college.

“That’s a concern in fields of study where life paths may change,” Kelchen said. One example: physicians: Recent medical school graduates have among the worst short-term earning statistics. “But that is because they are in residencies. They will make plenty of money in a couple years, but right now, it looks terrible.”

Kelchen lamented a shortage of good data on “medium-to-longer-term outcomes” for colleges and programs of study.

Using broad national trend data

To get averages for the nation (and, in some cases, states or smaller jurisdictions) Baker recommended using nationally representative datasets, including the  National Center for Education Statistics, the Census Bureau’s American Community Survey, the Bureau of Labor Statistics, Strada Education Center and the Pew Research Center.

Most of these databases sample a representative portion of the U.S. population to provide insight on attitudes toward higher education, or correlations between, say, a parent’s level of income and a child’s eventual educational attainment.

She recommended the U.S. Department of Education’s Data Lab, which offers QuickStats and PowerStats.  These are free tools that allow you to fairly easily look up data and build charts on a wide variety of studies and datasets. You can, for example, quickly  generate charts that compare, say, marital status and educational attainment, or the college Grade Point Average of students who had high GPAs in high school.

Analyzing colleges and programs

To investigate any particular college, or even a college major’s value, there are two free U.S. Department of Education datasets: the Integrated Postsecondary Education Data System (IPEDS) and the College Scorecard.

IPEDS data reflects all institutions that participate in the federal financial aid program and reports on college-level data, such as a college’s admission statistics, graduation rates, financial aid awards, spending on instructors, etc. (Here’s a free quick video tutorial on how to download and use IPEDS data.)

The College Scorecard reports on a myriad of data points for students who received any federal financial aid, such as student loans or Pell Grants. (Here are free quick video tutorials on using the College Scorecard tool and downloading the entire database.) The  College Scorecard provides data on students’ debt loads, loan repayment rates and earnings for each college major, allowing you to, for example, compare the financial outcomes for accounting majors at a college against, say, the biology majors at the same college, or accounting majors at other colleges. The biggest caveat about the College Scorecard is the latest data is limited to the results for those who graduated. That means the data doesn’t reflect what happens to all the students who drop or transfer out.

To reach beyond the conventional statistics when producing the Washington Monthly college rankings, Kelchen also analyzes social impact factors. This includes how much the college promotes civic engagement activities, such as voting, what percentage of a college’s students join the military or the Peace Corps, or go on to medical schools.

And Baker suggested journalists look at other societal benefits, such as a school’s impact on the local economy.

“What type of companies are attracted to a community” because of a college or group of colleges? “Apple is moving into the Research Triangle. If Duke, the University of North Carolina and North Carolina State weren’t there, the Research Triangle wouldn’t be there,” she noted.

Beware of college rankings

One of the most common uses of college-specific data is to build college rankings.

Although Kelchen does the number crunching for the Washington Monthly college rankings, he warned journalists to be skeptical of rankings. Rankings collapse a lot of information into a single number. And they are based on averages, so often aren’t a good measure of which school will be the best fit for a particular student.

If a college moves up in the rankings, journalists considering writing about the move should investigate the reasons for the move. Was it because the school rejected more applicants? Or did student outcomes improve? And if the latter: Which students – those who came in with lots of advantages? Or did the school give boosts to students who  faced barriers?

Hope for better data in the future

To better examine the impact of race on college affordability and outcomes, the federal government will soon start asking aid applicants for race information.

And a few states are doing a much better job of tracking how all students are faring. Texas, for example, set up systems that track every high school student and monitor their progress through colleges and the workforce. (The data  doesn’t release personally identifiable information about the students.)

Baker said  her “ideal dream” as a researcher would be a comprehensive national system. Because of opposition from colleges and privacy advocates, efforts to create such a national database have been debated for years in Washington.