11297: Strategic Integration of Large Language Models: Challenges, Opportunities, and Organizational Impact

In recent years, the field of artificial intelligence (AI) has seen remarkable advancements, particularly in the development of large language models (LLMs). LLMs represent a significant subset of generative AI, focusing primarily on text-based interactions. These models are sophisticated neural networks designed to process sequential data, enabling them to predict the next word in a sequence based on the context of preceding words (Bubeck et al., 2023; Cornelissen, Höllerer, Boxenbaum, Faraj, & Gehman, 2024). They are based on massive amounts of text to understand and generate natural language and other types of content, such as images, models, or figures (Eloundou, Manning, Mishkin, & Rock, 2023). The advent of models, such as ChatGPT, has made these technologies publicly accessible, demonstrating their ability to produce relevant and coherent responses across various domains (Kietzmann & Park, 2024). Practitioners highlight the transformational potential of LLMs for work and businesses (Alavi & Westerman, 2023; Teutloff, Einsiedler, & Søndergaard Møller, 2024). 81% of companies anticipate a positive impact from generative AI technologies like LLMs, with 50% anticipating transformative effects on business processes (Foege et al., 2024). Research indicates that LLMs can enhance employee performance and handle increasingly open-ended tasks (Bowman, 2023; Dell’Acqua et al., 2023). Nonetheless, most organizations struggle to implement their LLM pilots into production. For example, 68% of the organization’s leaders indicated they implemented a third or less of their pilot projects throughout the organization (Rowan, Ammanath, Perricos, Sniderman, & Jarvis, 2024). The multitasking and multi-purpose character of LLM technologies offers the organization's management an immense variety of strategic opportunities to assess and evaluate their internal processes and practices (Bang et al., 2023; Retkowsky, Hafermalz, & Huysman, 2024; Zhang et al., 2023). Therefore, introducing LLMs into an organization’s technology portfolio poses severe challenges to the management. Ethan Mollick (2024) argues that ”[…] traditional software comes with an operating manual or a tutorial. AI, however, lacks such instruction. There’s no definitive guide on how to use AI in your organization” (p.66). Previously, organizations developed specialized AI applications that fit the respective task’s content and purpose. The application entirely automated or augmented the strategic tasks to support the human-driven decision-making process (Daugherty & Wilson, 2018; Davenport & Kirby, 2016; Raisch & Krakowski, 2021). In contrast to this task-driven approach to technology implementation, the introduction of LLMs was highly driven by the release and open access of ChatGPT, giving employees and top management an introduction to the potential capabilities of the technology and igniting discussions in organizations about potential strategic tasks that could be augmented or automated through this technology (Christian, 2023; Hu, 2023). Early explorations reveal that organizations use different strategic approaches to integrate LLM technologies into their practices and processes. For instance, the Danish company Topsoe, which specializes in developing technologies for energy transition, turned to its employees to identify potential use cases in their organization for integrating LLMs. Employees submitted 158 ideas, and the top management selected 15 for a subsequent pilot (Gam-Korg & Secher, 2024; Olesen, 2023b; Persson, 2024). Here, the employees were deeply involved in evaluating and deciding where to start with LLM technology in the organization. Other organizations focused on identifying highly routinized tasks that can easily be automatized through the use of LLMs. For example, the logistics company DSV developed an LLM-based application that reads the electronic and handwritten documents provided by customers required for customs clearance and inserts the information into the public authority system. After DSV developed their application based on an open-source LLM, they trained the model with internal data, prioritizing control over the technology rather than using models from external software providers (Olesen, 2023a). As outlined in those examples, LLMs can affect diverse strategic tasks and processes due to their specific characteristics and related side effects. Therefore, it is crucial to start uncovering the theories that can help us understand LLMs' impact and which methods and research techniques are suitable. In our panel symposium, we want to discuss how diverse theories and methods can help us understand the impact of introducing and implementing LLMs as a strategic technology in organizations. Building on interviews and document analysis, Madalina and Juliane will prepare two case studies of companies integrating LLMs into their work practices. The case studies will be provided to the panelists several weeks before the panel symposium. Each panelist will analyze the provided case studies based on their theoretical perspective.