It may be a new year, but the Obama Administration’s proposed Postsecondary Institution Ratings System (PIRS) is still a hot topic. Most observers in the higher education policy and research communities (myself included) were less than overwhelmed by the proposed metrics released on December 19—sixteen months after the idea of ratings was first floated. My first take on the metrics can be found here, and there are too many good pieces about the metrics to mention them all.
The U.S. Department of Education has invited the public to provide additional feedback about the metrics used in PIRS (as well as the ratings system itself). You can submit your comments here before February 17. Below are my comments that I will submit to ED.
—————
January 5, 2015
My name is Robert Kelchen and I am an assistant professor in the Department of Education Leadership, Management and Policy at Seton Hall University as well as the methodologist for Washington Monthly magazine’s annual college rankings. (All opinions are my own.) After carefully examining the draft metrics proposed for potential inclusion in the Postsecondary Institution Ratings System (PIRS), I have the following comments and suggestions:
First, I am encouraged by the decision to exclude nondegree-granting colleges (mainly small for-profit colleges) from PIRS, as they are already subject to gainful employment. Holding them accountable for two different metrics is unreasonable. But in the two-year sector, it is essential to rate colleges that primarily grant associate’s degrees separately from those that primarily grant certificates due to the different lengths of those programs. The Department must divide two-year colleges up by their program emphasis (degree or certificate) in order for those ratings to be viewed as reasonable.
While I am glad to see discussions of multiple data sources in the draft metrics, I think the focus in the short term has to be using IPEDS data and previously-collected National Student Loan Data System (NSLDS) data for student loan repayment or default rates. Using NSLDS data for student background characteristics (such as first-generation status) is nice for the future, but is unlikely to be ready by this fall—particularly if colleges wish to dispute those data. I encourage the Department to focus on two sets of measures: refining readily available metrics from IPEDS and NSLDS for the draft ratings and continuing to develop new metrics for 2018 and beyond.
Most of the metrics proposed seem reasonable, although I am thoroughly confused by the “EFC gap” metric due to the lack of details provided. Would this be a measure of unmet need, of the percentage of FAFSA filers below a certain EFC, or something else? The Department should consider how strongly correlated the EFC gap measure may be with existing net price or family income data already in IPEDS—and also issue additional guidance on what the metric might be so the public can provide more informed comments.
I was disappointed not to see a technical discussion of potential weights that could be used in the system, and there were no mentions of the possibility of using multiple years of data in developing PIRS. It is important that the ratings be reasonably robust to a number of model specifications, including variables selected and weights used. I encourage the Department to continue working in this area and consulting with statisticians and education researchers.
While I do not expect PIRS to be tied to any federal financial aid dollars—and it is quite possible that draft ratings are never released to the public—the Department has a tremendous opportunity to improve data collection. Overturning the ban on student unit record data would significantly improve the quality of the data, but this is a great time to have a conversation about what information should be collected and processed for both public-sector and private-sector accountability systems. I am happy to provide assistance to the Department if desired and I wish them the best of luck in this difficult endeavor.
————-
I encourage everyone with an interest in PIRS to submit comments on the ratings, and to leave a copy of your comments in the comments section of this blog post.
[Cross-posted at Kelchen on Education]
