Scoping the use of AI to organise, synthesis and analyse a large scale community engagement and consultation exercise.

Introduction

Artificial Intelligence (AI) is emerging as a potentially transformative tool in qualitative research, offering opportunities to enhance the depth and scalability of a labour intensive and time consuming process¹. Qualitative research emphasises the richness of data over breadth or quantity, however, there is a growing need to accelerate qualitative research across various sectors to enable large-scale impact². Globally, AI is currently being used across a variety of public sector contexts. For example, in healthcare settings, it is used widely to analyse patient narratives and feedback; significantly reducing the time invested by clinicians to improve treatment protocols and prioritise patient care experiences^3,4. A six-month Australian Government trial found that Microsoft 365 Co-pilot helped departments efficiently summarise documents, draft meeting minutes, and reword reports. Most managers (64%) saw improved team efficiency and quality, while 40% of staff were able to spend more time on strategic tasks thanks to AI support⁵.

In Scotland, AI is currently being used by the NHS, Police Scotland and the Scottish Prison Service in a variety of ways, for example, assisting in effective communication with residents whose first language is not English and supporting those with visual or hearing impairments; enhancing resource allocation by analysing vast quantitative data to forecast demand and optimise logistics, such as reducing fuel costs and environmental impact⁶. Researchers from Edinburgh and Durham universities, in collaboration with Public Health Scotland and the Alan Turing Institute, have improved a tool for Scottish emergency departments using AI. This updated version better identifies individuals at high risk of urgent hospital care and helps healthcare providers manage resources efficiently. Emergency admissions, which make up about half of all hospital stays in Scotland, are now easier to predict and address⁶.

Automation in these sectors can clearly save time and reduce costs. However, qualitative data, such as that from qualitative interviews, focus groups or surveys is characterised by rich and deep insights into complex, real-world phenomena. Researchers in social and behavioural sciences adopt qualitative research methodologies, such as inductive coding, grounded theory, and data triangulation, to interpret text and capture the granularity and complexity of phenomena⁷. Human-led qualitative research excels at capturing nuanced emotions and complex social interactions, offering detailed insights. It provides contextual understanding, and allows researchers to adapt their techniques based on real-time feedback⁸. In their publication, ‘Scotland’s Artificial Intelligence Strategy’ the Scottish government encourages public sector organisations to utilise artificial intelligence for the analysis of qualitative data collected through surveys, interviews, and community feedback, suggesting that the implementation of AI could facilitate the rapid extraction of meaningful insights, thereby enabling more prompt evidence-based decision-making⁶.

The literature review reported in this paper was undertaken to inform recommendations around the suitability of employing AI software, specifically Microsoft 365 Co-pilot, to undertake the first-stage synthesis and analysis of qualitative data from a large scale local council community engagement consultation exercise.

Research context

In March 2025, Community Planning Aberdeen and Aberdeen City Council launched the Your Place, Your Plans, Your Future consultation⁹. This community engagement consultation aimed to inform a range of strategic plans and policies, including Housing, Health and Social Care, Community Learning and Development, the Visitor Levy Proposal, and the Local Development Plan Evidence Report. Multiple consultations were combined into a single process to streamline engagement, minimise duplication, and reduce the burden on residents and the resultant ‘consultation fatigue’¹⁰.

The consultation utilised the 14 themes of the Place Standard Tool, a nationally recognised Scottish framework for assessing the quality of places where people live, work, and interact¹¹. The tool encourages structured conversations by inviting residents to rate their local area across themes such as Housing, Public Transport, Safety, Social Interaction, Identity and Belonging, and Influence and Sense of Control. In a previous 2023 consultation using this tool, 367 residents participated. This generated over 10,000 responses. Due to the volume of responses, tight deadlines and limited staff capacity, a subset of this data was analysed using a framework analysis approach and a report of findings produced¹².

The current consultation anticipates over 2,000 responses, all of which will be organised and analysed. To address time and staffing constraints, Microsoft 365 Co-pilot was considered as an analysis tool since it was already included in Aberdeen City Council’s existing software package. This presented a potentially practical and cost-effective way to enhance qualitative analysis without requiring investment in additional software.

Aim

The aim is to compare the accuracy and consistency of AI-facilitated qualitative analysis, specifically using Microsoft 365 Co-pilot, with traditional human-led methods. Ultimately, the goal is to enable robust, evidence-based outcomes that respect citizens’ voices, and to assess whether integrating Co-pilot into routine local authority consultations offers a sustainable and cost-effective enhancement to qualitative research and analysis.

Methods

Inclusion and exclusion criteria

Studies were included if they reported a comparison between, or a dual-approach to, human-led qualitative analysis and AI-led analysis. Only studies that reported on data from interviews, focus groups or free text survey responses were included. Only those papers published in English were included. No limitations were put on publication date, although papers were excluded if they only reported findings from a COVID related study. Papers were included if they reported data from at least one of the following: patient satisfaction engagement; public health, local government, disadvantaged communities, community engagement, third sector or education research. Articles reporting on AI in market research, AI as a tool for students, the use of Chatbots in customer service or on health apps were excluded. Results of the search are shown below in Figure 1.

Figure 1. PRISMA diagram of search results¹³.

Results

Discussion

The studies in this review do not report on data at the scale of that expected to be generated by the Your Place, Your Plans, Your Future community engagement consultation exercise. In addition, no study could be found that has used Microsoft 365 Co-pilot, specifically, as a tool in qualitative analysis. Most of the studies reviewed used a version of ChatGPT, a Large Language Model like Microsoft 365 Co-pilot, thus they offer insight into the advantages and some of the challenges of using Microsoft 365 Co-pilot to assist in the analysis and synthesis of the qualitative data generated by a community engagement exercise.

All the studies included highlight the transformative potential of AI in qualitative research and raise questions around its capabilities. Across the studies the themes of efficiency, scalability, and AI’s skill at understanding nuance and context are discussed. Some studies^14,16 highlight ChatGPT's ability to quickly identify themes and provide valuable insights, significantly reducing analysis time with moderate to good concordance compared to traditional methods. Similarly, others²⁰ emphasise the rapid identification of themes by AI models, noting that ChatGPT identified ten themes in less than 30 seconds. These findings align with other academic research^24,25, which underscore AI's role in enhancing initial phases of qualitative data analysis while stressing the importance of human oversight to ensure relevance and specificity.

While AI models excel in efficiency, they often lack nuance and interpretive flexibility, i.e., the understanding that a text or narrative can be understood and interpreted in multiple ways depending on perspective and context. While nuance is inherent in human research, AI-generated content often misses subtleties that potentially alter the meanings of participants' views and experiences¹⁵. AI models such as ChatGPT are more successful at reproducing concrete, descriptive themes than at identifying subtle, interpretive ones²³. Not surprisingly, the need for human oversight to ensure accurate interpretation and comprehensive analysis is emphasised across the literature. For example, Viard et al.²⁶, discuss interpretive flexibility that human researchers provide, contrasting this with AI. They suggest that AI technology needs to ‘learn’ the complex relationships between social groups in order to fully interpret the complexities of qualitative data.

Also emphasised across the studies, is the need for transparent and replicable prompt engineering and the recognition that this requires training, practice and much trial and error^21,22,27. Khalid and Witmer²⁶ outline four key elements for prompt design: a clear task directive, relevant examples, contextual background, and specific response instructions. Microsoft 365 suggest Co-pilot prompts should include four parts: the goal, context, expectations, and source²⁸. Even with careful prompt design, using the same prompt multiple times can result in different responses. Large Language Models are built upon a neural network, which introduces some randomness, thus even a carefully designed input prompt, will produce slightly different results each time it is used²⁶. Collectively, these studies underscore the importance of a hybrid approach, where AI's efficiency is balanced with the nuanced understanding provided by human researchers, ensuring a thorough and accurate analysis of qualitative data.

The studies reviewed suggest that AI should be used as an adjunct to, rather than a replacement for, human-led qualitative analysis, emphasising that AI acts as an assistant in data exploration and evaluation, enhancing the effectiveness of established analytical methods without replacing them²⁵. These studies highlight the importance of a hybrid approach, where AI's capabilities are leveraged to enhance human-led qualitative analysis, ensuring a comprehensive and accurate understanding of complex data^1,18,20,21.

Ethical considerations

Our examination of the integration of AI in qualitative analysis highlights several ethical concerns that should be addressed. Confidentiality and privacy are paramount, requiring stringent protocols to protect sensitive participant data and prevent breaches of trust²⁹. Bias and fairness are also critical, as AI algorithms can introduce biases that skew research findings and lead to unfair representations. Researchers must scrutinise AI outputs for potential biases and implement mechanisms to ensure fair treatment of participants, transparency and accountability in order to maintain credibility and trust in research; as suggested above, researchers should document carefully how AI-derived conclusions are reached³⁰.

It has been noted that there are unique challenges to the adoption of AI in public sector and government settings, stemming from their capacity, structure, and operational methods³¹. The potential shortage of technical staff to implement and evaluate new technologies may result in inconsistency in, and of potential misuse of, AI, especially around security and privacy issues. The lack of resources and training around the use of AI may also hinder transparency and consistency in AI usage³². Addressing these ethical concerns is vital for the responsible integration of AI in qualitative analysis, especially in the public sector, to ensure that the benefits of AI are realised without compromising the integrity and ethical standards of qualitative research.

Future directions

The use of AI in qualitative research, and particularly in public sector contexts, is not well documented. Much of the literature on the use of AI-generated content or processes comes from the private sector where early adoption has taken place³³. Thus, a great deal can be learned from alternative, non-academic sources, about how to navigate the complexities of using AI moving forward. According to a 2024 McKinsey report³⁴, AI's role in policy development and public sector management is gaining significant interest largely driven by advancements in natural language processing and large language models. These technologies offer significant potential for enhancing qualitative data analysis through automated coding, thematic analysis, and even theory formulation. Researchers are encouraged to proceed with caution, balancing the benefits of AI augmentation with the need for rigorous ethical standards and human oversight³⁵.

Limitations

While this review provides valuable insights into the integration of AI, particularly large language models, in qualitative research, several limitations should be acknowledged. First, there is a notable scarcity of studies directly examining the use of Microsoft 365 Co-pilot in qualitative analysis. Most existing research focuses on related models such as ChatGPT, meaning our conclusions rely on the assumption that findings from these models can be extended to Co-pilot.

Second, the review highlights the lack of studies addressing qualitative data at the scale anticipated in large public sector initiatives like the Your Place, Your Plans, Your Future consultation. Much of the existing literature is based on smaller-scale datasets, which may limit the generalisability of findings to large-scale, community-driven consultations in the public sector.

The rapid evolution of AI technologies presents an inherent challenge. Studies included in this review may quickly become outdated as language models are updated and new features are introduced. As a result, some observations or limitations noted here may shift as the technology matures.

Finally, while ethical considerations are discussed, there is limited evidence in the reviewed literature about how ethical challenges are practically navigated or resolved in real-world applications, especially concerning issues of transparency, bias, and participant privacy within large-scale governmental projects. As AI continues to develop and its adoption widens, ongoing research will be essential to address these evolving ethical and methodological challenges.

Conclusion

The integration of AI in qualitative research presents a transformative opportunity to enhance the efficiency, scalability, and depth of analysis. The studies reviewed in this article highlight the significant advantages of AI-led qualitative analysis, particularly in terms of speed and consistency. Large language models, of which ChatGPT and Microsoft365 Co-pilot are only two, have demonstrated their ability to quickly identify themes and provide valuable insights, significantly reducing the time required for manual analysis. However, the studies also underscore the limitations of AI in capturing the nuanced understanding and interpretive flexibility inherent in human-led qualitative research. AI-generated content often misses subtle nuances, potentially altering the meanings of participants' views and experiences. This suggests that while AI can provide a useful first stage of analysis, additional human input is needed to ensure a comprehensive and accurate interpretation of qualitative data.

Moreover, the integration of AI in qualitative analysis brings several ethical concerns that must be addressed to ensure responsible use, particularly as there is a lack of AI strategies currently in place across the public sector. Confidentiality, privacy, bias, fairness, transparency, and accountability are paramount considerations. Researchers must implement stringent protocols to protect sensitive participant data, scrutinise AI outputs for potential biases, and document the AI-derived conclusions to maintain credibility and trust in scientific research.

While AI offers significant advantages in terms of efficiency, scalability, and consistency, it should be integrated with human expertise to ensure the richness and complexity of qualitative data are fully captured. The future of qualitative research lies in human researchers using human-led qualitative analysis and AI tools, leveraging the strengths of both to achieve a comprehensive and accurate understanding of complex data.