Welcome to the forefront of conversational AI as we explore the fascinating world of AI chatbots in our dedicated blog series. Discover the latest advancements, applications, and strategies that propel the evolution of chatbot technology. From enhancing customer interactions to streamlining business processes, these articles delve into the innovative ways artificial intelligence is shaping the landscape of automated conversational agents. Whether you’re a business owner, developer, or simply intrigued by the future of interactive technology, join us on this journey to unravel the transformative power and endless possibilities of AI chatbots.
Academic Medical Center Evaluates AI Chatbot for Faculty Promotion and Tenure Guidance
Navigating promotion and tenure policies within complex healthcare institutions presents a persistent administrative hurdle for academic faculty. In a proof-of-concept design-based research study conducted at the SingHealth Duke–NUS Academic Medical Center, investigators designed and evaluated an artificial intelligence chatbot named AskADD to guide clinical faculty and staff through institutional promotion and tenure documentation. The institutional environment oversees 3,000 clinical faculty members and processes approximately 400 academic appointments and promotions each year, yet the governing policies have historically remained dispersed across 30 separate intranet documents. The researchers established an iterative, two-phase development and evaluation framework to determine whether a conversational tool grounded directly in institutional policies could resolve faculty queries efficiently and accurately without requiring excessive manual document searching.
The study sought to address three core research questions focused on the iterative design process, faculty evaluations across specific usability and communication dimensions, and comparative effectiveness relative to traditional information-retrieval methods. The first research question examined how an artificial intelligence tool could be iteratively refined through consecutive development cycles to satisfy user-identified needs. The second evaluated self-perceptions among participating faculty regarding system usability, responsiveness, perceived empathy, and accessibility between successive iterations. The third investigated how participants perceived the tool’s practical utility compared to established traditional mechanisms, such as consulting administrative staff directly or manually browsing departmental intranet directories.
To guide the investigation, the research team adopted a design-based research methodology that coupled administrative efficiency principles with structured, iterative testing in an authentic operational environment. Theoretical foundations in generative artificial intelligence emphasize using retrieval-augmented architectures to ground language model outputs in verified local source repositories, minimizing fabricated assertions while streamlining organizational workflows. Across technology-enhanced learning environments, design-based research structures progress across sequential cycles of design, implementation, evaluation, and refinement, allowing direct participant feedback from an initial phase to inform technical configurations in the next.
The investigation was carried out across two distinct stages: an initial alpha phase and a subsequent beta phase, spanning August 2024 through April 2025, with user evaluations collected between January and April 2025. Following institutional review board approval from the National University of Singapore and the procurement of written informed consent, the research team deployed an initial alpha prototype configured using enterprise Microsoft Power Apps and Power Automate applications. This early build routed user inquiries to enterprise Azure OpenAI GPT-4o services, querying a dedicated Microsoft SharePoint repository housing the medical school’s promotion and tenure policy files before returning generated outputs to users through a Power Apps interface.
Recruitment relied on purposive sampling to represent diverse faculty tracks, institutional leadership positions, and administrative staff involved in appointment processes. From an initial pool of 105 faculty and staff contacted via email, 36 initially agreed to participate, and 35 completed both phases to form the final evaluative cohort. Within this group of 35 participants, medical personnel formed the largest single professional category at 60 percent (21 participants), followed by administrative personnel at 14 percent (5 participants), research staff at 11 percent (4 participants), dental staff at 9 percent (3 participants), one allied health professional (3 percent), and one registered nurse (3 percent). In terms of faculty tracks, the clinical track accounted for 40 percent (14 participants), the research track 17 percent (6 participants), the educator track 14 percent (5 participants), the tenure track 11 percent (4 participants), and the practice track 3 percent (1 participant), while 14 percent (5 participants) held non-track administrative appointments.
During the alpha evaluation, participant feedback revealed concrete functional requirements that guided extensive technical modifications for the beta release. Alpha users requested direct hyperlinks to underlying policy source files to facilitate independent document verification, greater handling of complex follow-up queries including routing to specific departmental contacts, and expanded coverage spanning wider human-resource and faculty-development information. Addressing these needs required migrating the architecture from low-code tools to a custom generative pre-trained transformer hosted on the university’s enterprise NUS AI-Know platform. The beta build integrated clickable source hyperlinks, added introductory conversation-starter prompts, broadened the source corpus to encompass public institutional websites, and incorporated structured prompt-engineering refinements.
The technical engineering process underscored practical challenges inherent in grounding language models within complex organizational records. Enabling automated source hyperlinking required approximately 60 development iterations, as early automated workflows frequently encountered inconsistent file-name formatting returned by the model in approximately 30 percent of trial runs during internal development testing. These formatting mismatches caused non-functional hyperlinks, which the team resolved through backend programming before beta deployment. System performance tracking demonstrated that query response latency averaged approximately 10 seconds in the alpha version and 8 seconds in the beta version across 20 standardized test queries. Additionally, model temperature was shifted from 0 in the alpha iteration to 0.3 in the beta iteration as an intentional design choice intended to sustain deterministic, repository-bounded outputs.
Evaluations collected through standardized Qualtrics surveys demonstrated marked descriptive shifts across self-perceived usability and communication metrics between development cycles. The proportion of faculty rating system responses as very relevant rose from 29 percent (10 of 35) in the alpha phase to 49 percent (17 of 35) in the beta phase, while ratings of very clear responses increased from 26 percent (9 of 35) to 54 percent (19 of 35). The proportion of participants reporting that they often had to rephrase queries dropped from initial levels down to zero percent in the beta phase. Perceptions of substantial time savings also grew, with users reporting savings exceeding 15 minutes rising from 31 percent (11 of 35) to 54 percent (19 of 35). Agreement that the chatbot simplified policy navigation increased from 94 percent to 97 percent, while perceived ease of understanding rose from 83 percent to 94 percent, culminating in 100 percent of beta participants reporting complete query resolution.
Measures addressing perceived responsiveness and accessibility showed consistently favorable baseline stability across both iterations. The proportion of respondents finding the tool straightforward to contact moved from 86 percent (30 of 35) to 89 percent (31 of 35), and agreement regarding interface accessibility rose from 94 percent to 97 percent. Conversely, perceived empathy registered substantial positive shifts, with participants reporting an enjoyable conversational interaction increasing from 60 percent (21 of 35) to 83 percent (29 of 35), alongside reductions in negative sentiment from 40 percent to 17 percent. When comparing the platform to traditional administrative methods, participants categorizing the chatbot as much more effective rose from 23 percent (8 of 35) to 37 percent (13 of 35), while approximately 15 percent judged it similarly or less effective across both phases. The corresponding Net Promoter Score improved from 12 percent to 42 percent, with promoters expanding from 23 percent to 51 percent.
The study authors emphasize several critical methodological and interpretive limitations that accompany these exploratory proof-of-concept findings. Because sampling was voluntary and limited to 35 individuals from a single academic medical center, selection bias may have elevated ratings among participants with higher baseline digital literacy or enthusiasm for emerging technologies. Furthermore, evaluating the same cohort across successive rounds introduced potential familiarity and expectation effects, while the survey instrument utilized unvalidated scales that captured self-reported perceptions rather than objective task completion times or error rates. System utility also remains tightly coupled to the currency of underlying institutional repositories, requiring ongoing administrative maintenance. Consequently, the authors conclude that while the platform demonstrates a transferable, platform-agnostic development framework, further validation using objective accuracy benchmarks and multi-institutional trials remains essential.
Subject of Research: Evaluation of an AI-assisted chatbot for academic medical center promotion and tenure guidance
Article Title: Developing and evaluating an AI-assisted chatbot for promotion and tenure guidance in an academic medical center
Article References: Yeo, M. M., Navedo, D. D., Casey, P. J., & Tan, B. C. Y. (2026). Developing and evaluating an AI-assisted chatbot for promotion and tenure guidance in an academic medical center. Frontiers in Artificial Intelligence, 9, Article 1919009. https://doi.org/10.3389/frai.2026.1919009
Image Credits: AI Generated
DOI: 10.3389/frai.2026.1919009
Keywords: artificial intelligence, academic medicine, chatbots, promotion and tenure, faculty development, generative AI, natural language processing, higher education administration, Developing, evaluating, AI-assisted, chatbot
News Source: Denise Maddox. (November 13, 2026). Academic Medical Center Evaluates AI Chatbot for Faculty Promotion and Tenure Guidance. Scienmag.
We bring you the latest biotechnology news from best research centers and universities around the world. Check our website.
Enter your email address to subscribe to this blog and receive notifications of new posts by email.
Bioengineer.org © Copyright 2023 All Rights Reserved.
Login to your account below
Please enter your username or email address to reset your password.
Bioengineer.org © Copyright 2023 All Rights Reserved.
