Meeting the challenges of item writing with assistance from artificial intelligence
Writing test items for nursing students is now quicker, easier
Most nurses who move from the clinical setting to academia are motivated by a desire to develop and mentor new nurses. But the actual demands of being a nurse educator can limit one-on-one time with students, leading to frustrations that can result in attrition. One of the most time-intensive faculty demands is item writing and assessment creation.
The struggles associated with assessment creation are widely recognized. A 2022 survey of 300 nurse educators from 44 states found that half did not feel confident in their ability to write NCLEX-style items — even if they had received training in this process (Moran et al, 2022).
Respondents who received formal education or training in item writing reported being only somewhat confident (51%), and 49% of respondents who had not received training reported being ambivalent or not confident.
SPEED UP YOUR ITEM WRITING
The authors of this study, published in Nurse Educator, stated that writing test questions is a complex and time-consuming process that is particularly challenging for new faculty who haven’t had prior academic training. They recommended that nursing schools develop programs, teams, and resources to support faculty in developing item-writing skills.
The authors’ experience and literature review identified concerns about the time required to write quality test items and the limited bandwidth nursing faculty have to do so. They noted that:
- Faculty spend considerable time planning lectures and course materials, often leaving insufficient time for test creation (Naeem, 2012).
- Writing a single high-quality multiple-choice question can take as long as an hour (Hijji, 2012).
- When writing test items, some nurse educators rely on test banks that accompany a course textbook, or they modify existing items rather than writing new questions. Both strategies make an assumption that the existing question or text is valid when it may not be, the authors said.
Other research has identified a lack of item-writing resources as a potential factor in faculty burnout and turnover.
In an unpublished 2023 survey of 795 nurse educators who left academia, 43.3% (n = 345) cited inadequate teaching resources as a contributing factor. In a follow-up question to 208 faculty, 55.8% (n =116) of respondents identified the creation of assessments as a top concern. The survey was conducted by Ascend Learning, the parent company of ATI Nursing Education.
New technologies can reduce item writing demands, freeing more time for teaching
As a well-known innovator of solutions for nursing education, ATI works closely with faculty throughout the country to identify their concerns and challenges. This knowledge contributes to the development of resources such as Custom Assessment Builder, the analytical platform that helps faculty write test questions, construct assessments, build case studies, and score and track results.
In March 2024, ATI took the scope and utility of Custom Assessment Builder (CAB) to new heights by integrating an artificial intelligence technology that will help educators save time when creating assessments.
Using the technology — known as Claire AI™️ — faculty enter simple prompts to generate an expansive variety of questions they can evaluate for inclusion in their assessments.
Importantly, the educator is always the final decision maker. After Claire AI generates each question, faculty review and edit it, request revision, or accept it and enter it into an assessment.
Premarket research shows that the integration of Claire AI into Custom Assessment Builder can significantly reduce the amount of time faculty spend writing test items.
During the pilot phase of CAB with Claire AI, 95% of faculty respondents reported that by using this technology to generate test items, the time they spent creating assessments dropped 50% or more.
Comments from the initial users included:
- “This resource is groundbreaking … Newly hired faculty will benefit greatly by having the ability to create assessment questions and bring them to the curriculum committee for approval. This will build experience and confidence.”
- “This is an excellent product … I feel it will make question writing easier for instructors.”
- “I love the new Claire AI. I tried it this morning with two of my tests and it was awesome!”
A screenshot of the greeting page for Claire AI in Custom Assessment Builder is shown below.
Research demonstrates the utility of AI in item writing when combined with faculty review and editing
As in most other industries, academia is determining appropriate uses for artificial intelligence in educational settings. The application of AI in select tasks performed by nursing faculty is evolving and a topic of research interest.
In a recently published study, Rachel Cox Simms, DNP, MS-HPEd, FNP-BC, and coauthors surveyed faculty in baccalaureate nursing programs to evaluate and compare the quality of item-writing by ChatGPT (free version) and by faculty (Cox, 2023). The human-created questions were written by the study authors and had scored well in actual use with students. The ChatGPT-generated questions were used as written, without editing.
The researchers asked faculty to evaluate the randomized NCLEX-style test items for clarity, grammar, and difficulty, and to assess them for clinical relevance.
The 92 respondents who completed the survey scored AI-created and human-created questions similarly in terms of clarity, grammar, difficulty, clinical relevance, use of complex terminology, and overall preference.
Of note, survey respondents perceived the human-generated questions as more difficult than the AI-generated questions. However, they found the AI-generated distractors to be clearer. “This suggests that AI has the capacity to generate questions that align with the standards of human-created examination materials,” the authors wrote.
They concluded that AI could assist nursing faculty with item generation but noted that faculty expertise is necessary to refine questions. In an interview with ATI, Dr. Cox Simms, the lead author for the study, emphasized the importance of balancing the strengths and limitations of AI.
“I think that ChatGPT’s real skill lies in its language capabilities,” said Dr. Cox Simms, who is an Assistant Professor in the School of Nursing at Massachusetts General Hospital Institute of Health Professions. “But a big weakness of all AI models is that they’re not always right. The expertise of faculty is really important for curating and editing. AI is not going to replace faculty. They’re still the gatekeeper.”
For Dr. Cox Simms, incorporating AI into item writing has been, in her words, “a massive time saver” that can lead to new approaches in question development.
“I know what I want to ask students, but I don’t want to spend forever wordsmithing,” she explained. “Although ChatGPT doesn’t always produce something that’s perfect, a lot of times it gives me a jumping off point or gets me to think about something in a new way. It might prompt me to adjust some distractors or shorten a stem. Overall, it is helping me produce better questions for students.”
Developing better nurses by supporting faculty and programs in tangible and innovative ways
The introduction of efficiency-generating resources for nursing programs has come at an opportune time. Data show that overall faculty vacancy rates, which were 8.8% in 2022 and 7.8% in 2023, are the highest they have been in a decade (AACN, 2022). By creating opportunities for more meaningful teaching experiences, the integration of Claire AI into Custom Assessment Builder has the potential to increase faculty satisfaction and keep more instructors in the classroom.
“As a former nurse educator, I know how difficult and time-consuming it can be to write fresh exam questions,” said Patty Knecht, PhD, RN, ANEF, Vice President and Chief Nursing Officer for Ascend Learning Healthcare.
“It’s exciting that Claire will now help nurse educators using Custom Assessment Builder focus more on developing one-on-one relationships with students and guiding them through challenging topics. This will result in a stronger nursing workforce and greater retention of nurse faculty.”
References
American Association of Colleges of Nursing. (2022). Special survey on vacant faculty positions for academic year 2022–2023. https://www.aacnnursing.org/Portals/0/PDFs/Data/Vacancy23.pdf
Cox, R. L., Hunt, K. L., Hill, R. R. (2023). Comparative analysis of NCLEX-RN questions: A duel between ChatGPT and human expertise. Journal of Nursing Education, 62 (12), 679-687. PMID: 38049305 DOI: 10.3928/01484834-20231006-07
Hijji, B. M. (2017). Flaws of multiple-choice questions in teacher-constructed nursing examinations: a pilot descriptive study. Journal of Nursing Education, 56 (8), 490-500. https://doi.org/10.3928/01484834-20170712-08
Moran, V., Wade, H., Moore, L., Israel, H., Bultas, M. (2022). Preparedness to write items for nursing education examinations: A national survey of nurse educators. Nurse Educator, 47 (2), 63-68. PMID: 34657106. DOI: 10.1097/NNE.0000000000001102
Naeem, N., van der Vleuten, C., Alfaris, E. A. (2012). Faculty development on item writing substantially improves item quality. Advances in Health Science Education, 17, 369-376. PMID: 21837548 DOI: 10.1007/s10459-011-9315-2
MEET CLAIRE AI, YOUR NEW ASSISTANT FOR ITEM WRITING