Presented By: Hartwig Hochmair, University of Florida

The recent surge in Generative AI, notably AI-powered chatbots, has sparked considerable interest in academic circles. This study delves into student evaluations of chatbot capabilities and limitations within an interdisciplinary graduate course titled "Digital Mapping" at UF, attracting students from diverse fields including natural resources, forestry, entomology and geomatics. The course, which covers theoretical and practical elements of digital spatial data mapping and analysis, introduced 22 graduate students to leading chatbots, i.e., ChatGPT, Bard, Claude-2, and Copilot. Instructed to apply these chatbots to five spatial tasks pertinent to the course, students were to assess each chatbot response on a 0-10 scale, highlighting any issues encountered. The tasks formulated by students primarily focused on map projections (19.8%), programming (16.5%), geo-computations (14.9%), and geographic literacy (14.9%). ChatGPT-3.5 emerged as the most favored tool (47.5% of tasks), with Bard following closely (36.1%). More than half of the students (54.5%) preferred using a single chatbot for all tasks, whereas only one student used four chatbots. Image-related tasks showed significant engagement (ChatGPT-4: 40%, Bard: 29%, Copilot: 43%). A Kruskal-Wallis Test indicated no significant difference in response quality across chatbots (Chi-square = 5.03, df = 4, p = 0.283), but it did reveal variations across task categories (Chi-square = 22.048, df = 8, p = 0.0048), with land cover analysis from aerial images receiving the highest scores. The study pinpointed various shortcomings in chatbot responses, including insufficient response detail (23.6%), stalled computations and analysis processes (18.1%), or incorrect computation results (13.9%). These insights suggest that integrating chatbot technology into natural science curricula could enhance educational outcomes and critical thinking skills, encouraging students to experiment with various chatbots, particularly for image analysis tasks, thereby broadening the scope of achievable assignments.