On June 11, at the Association for the Assessment of Learning in Higher Education’s (AALHE) Ninth Annual Assessment Conference, Paul Edelblut of Vantage Labs with Rebecca Wolf and Andrew Wolf of the University of Rochester’s School of Nursing (URSON) co-presented a session entitled, “Artificial Intelligence to Enhance Formative Assessment of Student Writing.”  The session was well attended and resulted in an active discussion afterward.

The Session Abstract:

Effective assessment of student writing demands considerable faculty time. Additionally, learning scholarly writing is dependent on formative assessment on multiple drafts. We will review how the University of Rochester School of Nursing Writing Center addresses this challenge by incorporating artificial intelligence (AI) scored writing for diagnostic and formative assessment. In a pilot project, Master’s students complete a short essay in response to a specific prompt. The AI engine provides an overall score plus scores aligned with  focus and meaning; content development; organization; language use, voice, and style; and mechanics and conventions.  These immediate, detailed scores allow faculty to provide differentiated instruction. We will share our experience in this pilot and discuss plans to use more advanced AI systems providing adaptive and independent learning. AI provides faculty the value of a writing assessment with little or no impact on time and at a minimal cost. Attendees will learn how the AI works, its limitations, and how it can enhance student learning.

Rebecca started off the session by describing how the program managers at School of Nursing recognized a vulnerability because of writing.  School of Nursing faculty are experts in nursing and healthcare, not writing, and correcting papers increasingly meant correcting basic grammar and scholarly conventions like citation, problems with content development and clarity.  Outcomes were being affected as students struggled in their comprehensive exams due to widespread weakness in writing and put in question students’ ability to think critically.

To tackle the problem, a task force was established.  Among the goals that the committee establish was determining that students needed clear expectations (rubrics),meaningful, relevant context for writing, ample revision opportunities and well-designed assignment guidance.  Faculty needed a way to assess and track writing progress across the program, support for the revision of assignments, creating rubrics, and providing feedback.  To deliver on these needs, Rebecca, who was trained as a K-12 writing teacher, joined the School of Nursing and the Writing Center was established and a Writing Across the Curriculum approach was undertaken.

All involved in the process agreed that writing promotes critical thinking and with a writing expert on staff, writing is taught throughout the program and is the responsibility of ALL faculty members.  Students are expected to practice the academic conventions of a discipline to effectively communicate within that discipline.  To implement Writing Across the Curriculum, courses were updated and Program Learning Outcomes for written communication were created along with standardized rubric elements for assignments.  Assessments were created looking at both direct and indirect measures and feedback formats were derived using HoWC for faculty and writing coaches.

After considerable investigation, URSON chose IntelliMetric to score student writing through CollegeSuccess™ from McCann Associates as the delivery platform.  With a clear set of desired outcomes, IntelliMetric’s ability to use advanced Artificial Intelligence scoring of free text responses proved to be highly effective.  Scoring provided a holistic total and detail aligned with 5 domains of writing.  Scores from IntelliMetric proved to be at or above human-level performance and URSON liked this low cost, and the highly modifiable platform for delivering the assessments.  Data collected served immediate needs as well as creating a mechanism to track improvement over time and by cohort.

Andrew Wolf described how (now that this project is in its second year) they can start to take the data and address specific writing challenges.  Data collected identified where students struggled most and further, identified a hierarchy of concerns.  The effectiveness of individual courses can now be determined as well as the impact of the entire Writing Across the Curriculum initiative and remediation can start to target specific components of the program.  Survey data is also being collected in the assessments allowing a chance to collect anecdotal evidence along with that gleaned from the formal assessments.

There is a growing crisis in higher education today with incoming students not being able to write effectively.  What should have been mastered by students in high school is now left to institutions of higher education.  The University of Rochester’s School of Nursing has made bold steps forward in addressing these skills gap and is a pioneer in focusing on writing in non-writing focused majors.  AALHE Session – AI to Enhance Formative Assessment of Student Writing and feel free to contact the following:

Rebecca Wolf:  Rebecca_Wolf@urmc.Rochester.edu

Andrew Wolf:   Andrew_Wolf@urmc.Rochester.edu

Paul Edelblut:   Pedelblut@vantage.com