Just as the rise of electronic publishing in the early 2000’s prompted reconsideration of how to evaluate publications in cases of reappointment, promotion, and tenure, the growing prominence of creative and scholarly works in Digital Humanities (DH) and Digital Media (DM) occasion a similar reassessment. Below are recommendations for selecting outside reviewers and for assessing the scholarly significance of DH and DM work.

Guidelines for Selection of Reviewers and Orientation of Faculty Discussion

Qualities that set DH and DM projects apart from more traditional scholarship are their novelty of form and delivery but also a level of intellectual engagement and labor that extends to choices about content and technical details of its delivery.

To ensure fair consideration is given to all aspects of intellectual work, outside reviewers should be selected in consideration of their domain expertise on both the content and technical details of a given body of DH and DM work. The following assumptions should guide selection and solicitation of external reviews, candidate self-reflection, and faculty discussions:

Respect Experimentation and Emerging Genres: recognize that the forms scholarly work might take may not resemble traditional genres of academic work. Some of the intellectual work behind a project may be in inventing an appropriate vehicle for delivery.

Appreciate Process: recognize and value the work that goes into the construction of a scholarly object. Whether the work includes planning an information architecture, coding, developing a usable interface, optimizing scholarship for discoverability and distribution, or something else, recognize that much of the decision making and development work that is otherwise handled by publishing houses and journals falls to individual scholars.

Appreciate Interdisciplinary and Collaborative Approaches: recognize that DH and DM projects are more often interdisciplinary and collaborative than not. These projects often require the input of people with a range of expertise that one person is unlikely to posses.

Appreciate New Models of Citation and Peer Review: recognize that traditional peer review might not be the norm or practicable for DH and DM projects.

Appreciate the Value of DH and DM Projects as Tools: recognize that part of the value of a DH or DM project might reside in its value as a tool for other scholars. Just as part of the value of an article is that it contributes to an ongoing scholarly conversation and promotes continuance of it, similar value should be considered for DH and DM projects.

These guidelines are adapted from Anderson and McPherson (2011) and from the MLA Guidelines for Evaluating Work in Digital Humanities and Digital Media (2012).

Assessment Categories and Criteria

The following assessment categories and criteria should be considered by the external reviewers when evaluating DH and DM work, addressed by the RPT candidate in his/her self-assessment, and discussed by the faculty when examining the contribution of a DH or DM work to a complete RPT dossier. It is the RPT candidate’s responsibility to suggest the appropriateness and prioritization of the following categories and criteria[1]:  

Quality of the Topical Content: criteria regarding the quality of the content in a scholarly work. The assessment of content should come from expert reviewers, faculty presenters, and from the RPT candidates. Criteria may include:

·      The novelty and creativity of the scholarship

·      The rigor and depth of scholarship on display

·      The skill employed in the execution of the project

Quality of the Artifact: criteria concerning the technical sophistication and usability of the scholarly artifact. These criteria should be assessed by reviewers with relevant expertise and by the RPT candidates. The purpose is to acknowledge the intellectual and technological decisions made at various visible and invisible levels in a DH and DM project to engage the intended audience in a particular interaction. All assessment should be made after viewing the DH or DM project in the medium for which it was intended (e.g., sound, web, film, etc.). Criteria may include:

·      Usability of the interface

·      Robustness of the underlying information architecture

·      Suitability of metadata and other optimization characteristics

·      Standards compliance for accessibility and longevity

Quality of the Review and Publication Medium: criteria concerning the manner in which a DH or DM project was reviewed prior to publication as well as criteria assessing the degree and quality of distribution/publication. Not all DH and DM projects are practicable to review prior to distribution/publication and not all will be distributed via 3^rd parties. As applicable, criteria may include:

·      Quality and extent of peer review, including consideration of who edits the source of publication and who serves on the review board

·      The collaboration or community out of which a DH or DM project develops or is incubated (e.g., NEH workshops).

·      Grant funding (i.e., successful proposals for grant funding should be considered equivalent to a favorable peer review)

·      The project host (e.g., university sponsored, commercial, etc.)

·      Project availability and reach (i.e., what audiences does the project reach and how effectively?)

·      Project affiliations (i.e., connections to other projects and conversations)

Significance, Distribution, and Uptake: criteria concerning the broader impact and influence of a DH or DM work post-publication. Reviewers and RPT candidates should specify which criteria are appropriate for measuring significant:

·      Secondary publications and project (e.g., grants, articles, workshops, lectures, products)

·      Post-publication review

·      Derivative works (e.g., use of source code)

·      Citations of the project

·      Uses of project as a scholarly tool

·      Long term sustainability and accessibility

·      Potential for future development

Anderson, S., & McPherson, T. (2011). Engaging Digital Scholarship: Thoughts on Evaluating Multimedia Scholarship. Profession, 2011(1), 136-151. doi:10.1632/prof.2011.2011.1.136

Della, C. D., Harris, J. A., Jewell, A., Martin, M., Pasanek, B., & Wythoff, G. (2011). Guidelines for Promotion and Tenure Committees in Judging Digital Work. NINES/NEH Summer Institute 2011-2012. Retrieved from http://institutes.nines.org/docs/2011-documents/guidelines-for-promotion-and-tenure-committees-in-judging-digital-work/

Modern Language Association. (2012). Guidelines for Evaluating Work in Digital Humanities and Digital Media. Retrieved from https://www.mla.org/About-Us/Governance/Committees/Committee-Listings/Professional-Issues/Committee-on-Information-Technology/Guidelines-for-Evaluating-Work-in-Digital-Humanities-and-Digital-Media

---

[1] Portions adapted from the NINES/NEH Summer Institute (Della et al., 2011).