ITHAKA Sustainable Scholarship 2012: Peer Review and Other Quality Questions

The theme of this year’s ITHAKA Sustainable Scholarship focused on a central theme of the information age: “The Question of Quality: New Forms of Grading, Credentialing, and Peer Review in the Digital World.” The annual conference of publishing, academia, technology, and information science professionals, held on October 16 at the Hilton New York, drew close to twenty AAUP presses, signifying how vital these meetings have become to the university press community. And, as expected, each panel was fresh and thought-provoking, exploring the latest transformations and future trends of scholarship.

Throughout the day, the idea of sustainable scholarship was viewed through the lens of discerning, judging, and promoting high-quality content, at a time when scholarship continues to explore new formats and produce exponentially. The quality question is something that university presses confront constantly—both in longstanding practices like looking for the best manuscripts and proposals and like shaping and improving projects through peer review, but also in new ways, like how to select the best projects for apps and digital shorts, or like experimenting with new online review systems.

The last—online review—was the focus of the first morning panel, “Next Generation Peer Review,” with panelists representing the Modern Language Association and online journal publishers PeerJ and PLoS One. Kathleen Fitzpatrick, Director of Scholarly Communication at MLA, began by discussing open review systems like those she explored with New York University Press and MediaCommons, thanks to a recent grant from the Mellon Foundation.

Fitzpatrick weighed the merits of an open peer review system against those of traditional review. At its heart, peer review is a tool for feedback and selection, intended to further the best of scholarly values without bias; how could a digital platform do this better? These and many other questions are what have been explored in the collaboration. For example, weighing reviewer anonymity, Fitzpatrick considered that an open digital platform could equalize the power balance between author and reviewer, encouraging thorough, accountable reviews while at the same time encouraging more scholars to become, and receive credit as, active reviewers.

Fitzpatrick claims that innovative review processes that experiment with variables like anonymity, among others, have the power to elevate scholarly discourse and conversation. The group’s report concludes that while the possibilities for scholarly dialogue on open review systems like MediaCommons are certainly inspiring, current challenges for adoption lean toward the social rather than the technological. It’s no longer just about creating the right platform; the platform’s creators will have to work to encourage its integration into existing institutions and workflows. For a more thorough review of the group’s conclusions, the project report can be reviewed online on MediaCommons.

Peter Binfield, representing PLoS One and his current venture, PeerJ, next compared the review processes of various online journal platforms, along with their successes. While the review processes of STEM journals and of most university press manuscripts have some clear differences, Binfield described some new platforms experimenting in ways that university presses may find increasingly relevant.

Binfield’s previous project, PLoS One, a popular, mostly medical, STEM platform, has a high submission rate along with a high acceptance rate; an average of two peer reviewers approve articles for revision and publication based solely on the content’s scientific validity. But this basic review process also incorporates room online for post-publication comments. Currents, another PLoS platform, publishes more quickly and without peer review; articles are submitted directly into XML and approved by an editorial board for rapid publication, and can later be revised. Unsurprisingly, Binfield judged Currents not as successful as PLoS One. F1000, Rubriq, and PeerJ (Binfield’s latest) are three more platforms launching in 2012 or shortly thereafter; one factor common to most, echoing MediaCommons, is the inclusion of shared, post-publication commentary. But all also include varieties of peer review: F1000 will publish articles first, and solicit reviews and commentary afterward; Rubriq will employ a standardized, numerical scorecard for review; and while PeerJ requires peer review, they encourage it to be open and credited, with an option to publish papers’ “audit trails.”

In two following panels, ITHAKA opened up the quality question to scholarship as a whole. Sandra Cook, representing Kentucky distance learning, discussed the validity of new forms of academic credentials with Sheryl Grant of HASTAC and “un-schooler” Dale Stephens. Daphne Koller, co-founder of the trailblazing MOOC program Coursera, spoke with former Tufts president Larry Bacow and Vanderbilt computer science professor Douglas Fisher on the possibilities and early observations of online courses.

These panels and the rest of this year’s Sustainable Scholarship conference, including closing thoughts in the final panel from University of California Press’s Alison Mudditt, can now be viewed in full, online, on the ITHAKA website.

Regan Colestock
Communications Coordinator, AAUP