This page contains automatically translated content.

M.Sc. Philipp Reinhard

Research assistant

Telephone: +49 561 804-6021
Email: philipp.reinhard[at]uni-kassel[dot]de

Location: Pfannkuchstraße 1

Room: ITeG, Room 1170, 1st floor

Consultation Hours: Wed, 13:00-14:00 Registration by e-mail required. Further appointments by arrangement.

Curriculum vitae

Education

(10/2019 - 05/2022) M.Sc. in Business Informatics at the Technical University of Darmstadt

(04/2021 - 05/2022) M.Sc. in "Entrepreneurship and Innovation Management" at the Technical University of Darmstadt

Topic of the master thesis: "Customer Education in Smart Grids: Facilitating the adoption of innovation for energy efficiency and flexibility"

(08/2020 - 06/2021) Erasmus program at the Chalmers University of Technology in Gothenburg, Sweden

(10/2015 - 03/2019) Studies B.Sc.in Business Informatics at the Technical University of Darmstadt

Topic of the bachelor thesis: "Analysis and mapping of hybrid approaches in project and program management using the example of cloud-based Microsoft solutions"

Work experience

(06/2016 - 03/2017) Working student in online marketing (SEO) at Re7Consulting UG (haftungsbeschränkt) in Fulda

(04/2016 - 03/2018) Student management consultant at Junior Comtec in Darmstadt with a focus on IT-driven process optimization (5 projects in total, 3 of them as project manager)

Projects:

Supplier Relationship Management (Procurement 4.0)
Re-engineering of business processes
Optimization and automation of the target agreement process

(04/2018 - 10/2018) Bachelor student in Enterprise Project Management (especially "hybrid project management approaches") at Campana & Schott in Frankfurt

(11/2018 - 11/2020) Working student in Enterprise Project Management at Campana & Schott in Frankfurt

(08/2021 - 04/2022) Student assistant at the IÖW in the research field Environmental Economics and Environmental Policy in Berlin Project: SustAIn - Sustainability Index for Artificial Intelligence

Research interests

Hybrid Intelligence (Human-AI Collaboration)
(Sustainable) Artificial Intelligence
Chatbots, voicebots and conversational AI
Service science, service engineering (especially IT service management)
Digital work and agile organization
IT project management & software development
Agentic AI and generative AI (in particular large language models)
Multi-agent systems and agentic workflows
Explainable Artificial Intelligence (XAI)
Process Mining

Publications

forthcoming[ to top ]

Reinhard, P.; Li, M. M.; Fina, M.; Peters, C.; & Leimeister, J. M. (forthcoming): Design Features for Explainable Generative AI (GenXAI) Systems in Knowledge-Intensive Service Work. In: Hawaii International Conference on System Sciences (HICSS) (Vol. Proceedings of the 59th Hawaii International Conference on System Sciences). Maui, Hawaii, USA.
- [ BibTeX ]
- [ EndNote ]
@inproceedings{ls_leimeister, abstract = {The use of generative AI (GenAI) and large language models (LLMs) in knowledge-intensive fields like customer support is rapidly growing. While GenAI responses often appear persuasive, they carry the risk of inaccuracies and hallucinations. Hence, users must critically evaluate responses to reach appropriate reliance and knowledge utilization. Despite technological advancements, design knowledge for enhancing human-GenAI interaction from an explainable AI (XAI) perspective remains lacking. Thus, this study applies the design science research (DSR) approach to develop explanations that aid human interaction with GenAI systems. Drawing from XAI literature and human reasoning theories, we built and evaluated seven design features and instantiated a prototype that contributes to the development of reliable explainable GenAI (GenXAI).}, address = {Maui, Hawaii, USA}, author = {Reinhard, Philipp and Li, Mahei Manhai and Fina, Matteo and Peters, Christoph and Leimeister, Jan Marco}, booktitle = {Hawaii International Conference on System Sciences (HICSS)}, keywords = {itegpub}, title = {Design Features for Explainable Generative AI (GenXAI) Systems in Knowledge-Intensive Service Work}, volume = {Proceedings of the 59th Hawaii International Conference on System Sciences}, year = {forthcoming} }
%0 Conference Paper %1 ls_leimeister %A Reinhard, Philipp %A Li, Mahei Manhai %A Fina, Matteo %A Peters, Christoph %A Leimeister, Jan Marco %B Hawaii International Conference on System Sciences (HICSS) %C Maui, Hawaii, USA %D forthcoming %T Design Features for Explainable Generative AI (GenXAI) Systems in Knowledge-Intensive Service Work %V Proceedings of the 59th Hawaii International Conference on System Sciences %X The use of generative AI (GenAI) and large language models (LLMs) in knowledge-intensive fields like customer support is rapidly growing. While GenAI responses often appear persuasive, they carry the risk of inaccuracies and hallucinations. Hence, users must critically evaluate responses to reach appropriate reliance and knowledge utilization. Despite technological advancements, design knowledge for enhancing human-GenAI interaction from an explainable AI (XAI) perspective remains lacking. Thus, this study applies the design science research (DSR) approach to develop explanations that aid human interaction with GenAI systems. Drawing from XAI literature and human reasoning theories, we built and evaluated seven design features and instantiated a prototype that contributes to the development of reliable explainable GenAI (GenXAI).
Reinhard, P.; Sajzev, V.; Li, M. M.; & Leimeister, J. M. (forthcoming): Evaluating and Improving Prompt Quality in LLM-Based Assistants: A Synthesis of Criteria and Indicators. In: Hawaii International Conference on System Sciences (HICSS) (Vol. Proceedings of the 59th Hawaii International Conference on System Sciences). Maui, Hawaii, USA.
- [ BibTeX ]
- [ EndNote ]
@inproceedings{ls_leimeister, abstract = {Generative AI (GenAI) assistants, particularly large language models (LLMs), are gaining increasing relevance across domains. The quality of outputs generated by these systems is highly contingent on the input prompts, giving rise to new professional roles such as prompt engineers. In this study, we systematically examine evaluation criteria and optimization methods that can improve prompt quality. Drawing on a systematic literature review, we identify key criteria, including clarity, accuracy, and precision, and initial measurement techniques. In addition, we synthesize common optimization methods such as iterative refinement and shot-based prompting. Our work contributes to the growing efforts to standardize the evaluation and improvement of prompts in interactions with LLM-based assistants, thereby fostering a more rigorous and coherent understanding of the prompt quality construct.}, address = {Maui, Hawaii, USA}, author = {Reinhard, Philipp and Sajzev, Vladimir and Li, Mahei Manhai and Leimeister, Jan Marco}, booktitle = {Hawaii International Conference on System Sciences (HICSS)}, keywords = {prompt_quality}, title = {Evaluating and Improving Prompt Quality in LLM-Based Assistants: A Synthesis of Criteria and Indicators}, volume = {Proceedings of the 59th Hawaii International Conference on System Sciences}, year = {forthcoming} }
%0 Conference Paper %1 ls_leimeister %A Reinhard, Philipp %A Sajzev, Vladimir %A Li, Mahei Manhai %A Leimeister, Jan Marco %B Hawaii International Conference on System Sciences (HICSS) %C Maui, Hawaii, USA %D forthcoming %T Evaluating and Improving Prompt Quality in LLM-Based Assistants: A Synthesis of Criteria and Indicators %V Proceedings of the 59th Hawaii International Conference on System Sciences %X Generative AI (GenAI) assistants, particularly large language models (LLMs), are gaining increasing relevance across domains. The quality of outputs generated by these systems is highly contingent on the input prompts, giving rise to new professional roles such as prompt engineers. In this study, we systematically examine evaluation criteria and optimization methods that can improve prompt quality. Drawing on a systematic literature review, we identify key criteria, including clarity, accuracy, and precision, and initial measurement techniques. In addition, we synthesize common optimization methods such as iterative refinement and shot-based prompting. Our work contributes to the growing efforts to standardize the evaluation and improvement of prompts in interactions with LLM-based assistants, thereby fostering a more rigorous and coherent understanding of the prompt quality construct.

2025[ to top ]

Reinhard, P.; Li, M. M.; Peters, C.; & Leimeister, J. M. (2025): Enhancing IT Service Management Through Process Mining – A Digital Analytics Perspective on Documented Customer Interactions. In: Forum Dienstleistungsmanagement, Vol. Digital Analytics im Dienstleistungsmanagement, Erscheinungsjahr/Year: 2025. pp. 327–360.
- [ BibTeX ]
- [ EndNote ]
- [ URL ]
@article{ls_leimeister, abstract = {Our study explores the integration of text mining and process mining to enhance the understanding of IT support agents' problem-solving activities documented in service tickets. Despite the rise of AI-based self-service systems, the pressure on IT support to deliver high-quality service remains significant, necessitating advanced analytical approaches. While text mining has been used for classifying customer requests or predicting satisfaction, it falls short in revealing the actual processes agents follow. By conducting a systematic literature review and a case study, this research outlines a novel approach combining text and process mining. The findings provide practical guidance for extracting activity catalogs and generating event logs from service documentation, offering valuable insights into service processes and highlighting challenges related to data quality in digital analytics.}, author = {Reinhard, Philipp and Li, Mahei Manhai and Peters, Christoph and Leimeister, Jan Marco}, editor = {Bruhn, Manfred and Hadwich, Karsten}, journal = {Forum Dienstleistungsmanagement}, keywords = {itegpub}, pages = {327-360}, title = {Enhancing IT Service Management Through Process Mining – A Digital Analytics Perspective on Documented Customer Interactions}, volume = {Digital Analytics im Dienstleistungsmanagement}, year = 2025 }
%0 Journal Article %1 ls_leimeister %A Reinhard, Philipp %A Li, Mahei Manhai %A Peters, Christoph %A Leimeister, Jan Marco %D 2025 %E Bruhn, Manfred %E Hadwich, Karsten %J Forum Dienstleistungsmanagement %P 327-360 %R 10.1007/978-3-658-48325-8_11 %T Enhancing IT Service Management Through Process Mining – A Digital Analytics Perspective on Documented Customer Interactions %U http://pubs.wi-kassel.de/wp-content/uploads/2025/06/JML_1028.pdf %V Digital Analytics im Dienstleistungsmanagement %X Our study explores the integration of text mining and process mining to enhance the understanding of IT support agents' problem-solving activities documented in service tickets. Despite the rise of AI-based self-service systems, the pressure on IT support to deliver high-quality service remains significant, necessitating advanced analytical approaches. While text mining has been used for classifying customer requests or predicting satisfaction, it falls short in revealing the actual processes agents follow. By conducting a systematic literature review and a case study, this research outlines a novel approach combining text and process mining. The findings provide practical guidance for extracting activity catalogs and generating event logs from service documentation, offering valuable insights into service processes and highlighting challenges related to data quality in digital analytics.
Karst, F.; Li, M. M.; Reinhard, P.; & Leimeister, J. M. (2025): Not Enough Data to Be Fair? Evaluating Fairness Implications of Data Scarcity Solutions. In: Proceedings of the 58th Hawaii International Conference on System Sciences (HICSS) (pp. 6886–6895). Hawaii, USA.
- [ BibTeX ]
- [ EndNote ]
- [ URL ]
@inproceedings{ls_leimeister, abstract = {This study explores the implications of the use of data scarcity solutions on fairness in machine learning, specifically in consumer credit interest rate prediction. We develop a comprehensive taxonomy of Data Scarcity Solutions (DSS) by analyzing academic literature, data science competitions, and practical implementations. We identify six distinct DSS clusters: Data Extension, Pre-Training, Public Data Inclusion, Data Sharing, Federated Learning, and Active Learning. Our evaluation shows that most DSS enhance both performance and fairness, with minimal negative correlation between the two. Notably, approaches incorporating external or synthetic data significantly improve fairness. This research contributes to understanding DSS beyond algorithmic performance, providing a framework for evaluating their societal impact. Furthermore, it offers practitioners a taxonomy to select the right method for tackling data scarcity and addresses fairness concerns in real-world scenarios.}, address = {Hawaii, USA}, author = {Karst, Fabian and Li, Mahei Manhai and Reinhard, Philipp and Leimeister, Jan Marco}, booktitle = {Proceedings of the 58th Hawaii International Conference on System Sciences (HICSS)}, keywords = {Fairness}, pages = {6886-6895}, title = {Not Enough Data to Be Fair? Evaluating Fairness Implications of Data Scarcity Solutions}, year = 2025 }
%0 Conference Paper %1 ls_leimeister %A Karst, Fabian %A Li, Mahei Manhai %A Reinhard, Philipp %A Leimeister, Jan Marco %B Proceedings of the 58th Hawaii International Conference on System Sciences (HICSS) %C Hawaii, USA %D 2025 %P 6886-6895 %T Not Enough Data to Be Fair? Evaluating Fairness Implications of Data Scarcity Solutions %U http://pubs.wi-kassel.de/wp-content/uploads/2025/04/JML_1014.pdf %X This study explores the implications of the use of data scarcity solutions on fairness in machine learning, specifically in consumer credit interest rate prediction. We develop a comprehensive taxonomy of Data Scarcity Solutions (DSS) by analyzing academic literature, data science competitions, and practical implementations. We identify six distinct DSS clusters: Data Extension, Pre-Training, Public Data Inclusion, Data Sharing, Federated Learning, and Active Learning. Our evaluation shows that most DSS enhance both performance and fairness, with minimal negative correlation between the two. Notably, approaches incorporating external or synthetic data significantly improve fairness. This research contributes to understanding DSS beyond algorithmic performance, providing a framework for evaluating their societal impact. Furthermore, it offers practitioners a taxonomy to select the right method for tackling data scarcity and addresses fairness concerns in real-world scenarios.
Reinhard, P.; Liessmann, A.; Weinzierl, S.; Zilker, S.; Li, M. M.; Matzner, M.; & Leimeister, J. M. (2025): EVENT LOG CONSTRUCTION FROM MULTIMODAL DATA – A REFERENCE ARCHITECTURE FOR EXPLOITING PROCESS MINING IN IT SERVICE MANAGEMENT. In: European Conference on Information Systems (ECIS). Amman, Jordan.
- [ BibTeX ]
- [ EndNote ]
- [ URL ]
@inproceedings{ls_leimeister, abstract = {Process mining holds substantial potential to discover and optimize processes utilizing event log data. However, current applications primarily rely on (semi-)structured data from process-aware information systems, limiting their capacity to incorporate multimodal data from diverse sources, particularly in domains like IT service management (ITSM). While existing stand-alone approaches can extract event log data from unstructured sources such as videos, documents, or bot logs, they fall short of leveraging the full range of real-world data available in ITSM. To address this gap, our research focuses on developing a reference architecture for constructing event logs from multimodal data. This architecture integrates diverse data types, construction functions, and process mining use cases. Following a design science research methodology, we aim to evaluate the architecture through a software artifact leveraging real-world ITSM data and incorporating state-of-the-art generative AI. In this study, we present the preliminary reference architecture and share early insights from expert evaluations.}, address = {Amman, Jordan}, author = {Reinhard, Philipp and Liessmann, Annina and Weinzierl, Sven and Zilker, Sandra and Li, Mahei Manhai and Matzner, Martin and Leimeister, Jan Marco}, booktitle = {European Conference on Information Systems (ECIS)}, keywords = {Event_Log_Construction}, title = {EVENT LOG CONSTRUCTION FROM MULTIMODAL DATA – A REFERENCE ARCHITECTURE FOR EXPLOITING PROCESS MINING IN IT SERVICE MANAGEMENT}, year = 2025 }
%0 Conference Paper %1 ls_leimeister %A Reinhard, Philipp %A Liessmann, Annina %A Weinzierl, Sven %A Zilker, Sandra %A Li, Mahei Manhai %A Matzner, Martin %A Leimeister, Jan Marco %B European Conference on Information Systems (ECIS) %C Amman, Jordan %D 2025 %T EVENT LOG CONSTRUCTION FROM MULTIMODAL DATA – A REFERENCE ARCHITECTURE FOR EXPLOITING PROCESS MINING IN IT SERVICE MANAGEMENT %U http://pubs.wi-kassel.de/wp-content/uploads/2025/09/JML_1015.pdf %X Process mining holds substantial potential to discover and optimize processes utilizing event log data. However, current applications primarily rely on (semi-)structured data from process-aware information systems, limiting their capacity to incorporate multimodal data from diverse sources, particularly in domains like IT service management (ITSM). While existing stand-alone approaches can extract event log data from unstructured sources such as videos, documents, or bot logs, they fall short of leveraging the full range of real-world data available in ITSM. To address this gap, our research focuses on developing a reference architecture for constructing event logs from multimodal data. This architecture integrates diverse data types, construction functions, and process mining use cases. Following a design science research methodology, we aim to evaluate the architecture through a software artifact leveraging real-world ITSM data and incorporating state-of-the-art generative AI. In this study, we present the preliminary reference architecture and share early insights from expert evaluations.
Reinhard, P.; Li, M. M.; Oeste-Reiß, S.; & Bretschneider, U. (2025): GenAI-CoP: A Reusable Co-Creation Process for Identifying Generative AI Agents. In: International Conference on Design Science Research in Information Systems and Technology (DESRIST). Montego Bay, Jamaica: Springer.
- [ BibTeX ]
- [ EndNote ]
- [ URL ]
@inproceedings{ls_leimeister, abstract = {Generative AI (GenAI) can enhance organizational processes and productivity. To realize these benefits, organizations must design GenAI agents that augment human work. A key challenge lies in making sense of the diverse forms of GenAI agents and aligning them with existing work processes. To address this, we propose a reusable co-creation process for identifying GenAI agents (GenAI-CoP) that enables organizations to involve domain experts in leveraging GenAI’s potential for their products and workflows. Grounded in action design research (ADR), our approach draws on collaboration engineering to develop GenAI-CoP. We iteratively refined and tested it through simulations, expert interviews, and pilot tests. Our research contributes to GenAI and collaboration engineering literature by introducing a reusable, bottom-up identification procedure for GenAI agents. GenAI-CoP packages facilitation expertise, allowing practitioners to execute it without prior training in tools or techniques. They gain actionable guidelines to identify augmentation potential and summarize it as GenAI agents.}, address = {Montego Bay, Jamaica}, author = {Reinhard, Philipp and Li, Mahei Manhai and Oeste-Reiß, Sarah and Bretschneider, Ulrich}, booktitle = {International Conference on Design Science Research in Information Systems and Technology (DESRIST)}, keywords = {itegpub}, publisher = {Springer}, title = {GenAI-CoP: A Reusable Co-Creation Process for Identifying Generative AI Agents}, year = 2025 }
%0 Conference Paper %1 ls_leimeister %A Reinhard, Philipp %A Li, Mahei Manhai %A Oeste-Reiß, Sarah %A Bretschneider, Ulrich %B International Conference on Design Science Research in Information Systems and Technology (DESRIST) %C Montego Bay, Jamaica %D 2025 %I Springer %T GenAI-CoP: A Reusable Co-Creation Process for Identifying Generative AI Agents %U http://pubs.wi-kassel.de/wp-content/uploads/2025/06/JML_1013.pdf %X Generative AI (GenAI) can enhance organizational processes and productivity. To realize these benefits, organizations must design GenAI agents that augment human work. A key challenge lies in making sense of the diverse forms of GenAI agents and aligning them with existing work processes. To address this, we propose a reusable co-creation process for identifying GenAI agents (GenAI-CoP) that enables organizations to involve domain experts in leveraging GenAI’s potential for their products and workflows. Grounded in action design research (ADR), our approach draws on collaboration engineering to develop GenAI-CoP. We iteratively refined and tested it through simulations, expert interviews, and pilot tests. Our research contributes to GenAI and collaboration engineering literature by introducing a reusable, bottom-up identification procedure for GenAI agents. GenAI-CoP packages facilitation expertise, allowing practitioners to execute it without prior training in tools or techniques. They gain actionable guidelines to identify augmentation potential and summarize it as GenAI agents.
Dittmar, A.; Reinhard, P.; Li, M. M.; Karst, F.; & Leimeister, J. M. (2025): Integrating Brand Identity Into AI-Based Conversational Agents: A Systematic Literature Review. In: Hawaii International Conference on System Sciences (HICSS). Big Island, Hawaii.
- [ BibTeX ]
- [ EndNote ]
- [ URL ]
@inproceedings{ls_leimeister, abstract = {A company's brand identity is vital for fostering meaningful interactions with customers in a digital economy. As generative artificial intelligence (GenAI) becomes increasingly integrated into our daily lives, AI-based assistants are evolving into the primary channel for representing a brand's identity. Consequently, brand departments face the challenge of translating their company's brand identity into highly interactive AI-based assistants effectively. While there is diverse research on single cues that influence user perceptions, the literature lacks a comprehensive understanding of conversational cues and their impact on brand-related outcomes. Hence, we conduct a systematic literature review to identify verbal and visual cues in the design of AI-based chatbots and examine how they influence the perceptions of the assistants and the brand. We contribute a conceptual framework that summarizes the independent and dependent variables as well as their interrelationships and provides researchers and practitioners with a basis for designing and studying conversational design in the age of GenAI.}, address = {Big Island, Hawaii}, author = {Dittmar, Angelia and Reinhard, Philipp and Li, Mahei Manhai and Karst, Fabian and Leimeister, Jan Marco}, booktitle = {Hawaii International Conference on System Sciences (HICSS)}, keywords = {itegpub}, title = {Integrating Brand Identity Into AI-Based Conversational Agents: A Systematic Literature Review}, year = 2025 }
%0 Conference Paper %1 ls_leimeister %A Dittmar, Angelia %A Reinhard, Philipp %A Li, Mahei Manhai %A Karst, Fabian %A Leimeister, Jan Marco %B Hawaii International Conference on System Sciences (HICSS) %C Big Island, Hawaii %D 2025 %T Integrating Brand Identity Into AI-Based Conversational Agents: A Systematic Literature Review %U http://pubs.wi-kassel.de/wp-content/uploads/2024/11/JML_982.pdf %X A company's brand identity is vital for fostering meaningful interactions with customers in a digital economy. As generative artificial intelligence (GenAI) becomes increasingly integrated into our daily lives, AI-based assistants are evolving into the primary channel for representing a brand's identity. Consequently, brand departments face the challenge of translating their company's brand identity into highly interactive AI-based assistants effectively. While there is diverse research on single cues that influence user perceptions, the literature lacks a comprehensive understanding of conversational cues and their impact on brand-related outcomes. Hence, we conduct a systematic literature review to identify verbal and visual cues in the design of AI-based chatbots and examine how they influence the perceptions of the assistants and the brand. We contribute a conceptual framework that summarizes the independent and dependent variables as well as their interrelationships and provides researchers and practitioners with a basis for designing and studying conversational design in the age of GenAI.
Grau, M.; Reinhard, P.; Bauer-Hänsel, I.; & Blohm, I. (2025): Rethinking AI Design Practices: A Reference Architecture for Generative AI Systems. In: European Conference on Information Systems (ECIS). Amman, Jordan: Aisel.
- [ BibTeX ]
- [ EndNote ]
- [ URL ]
@inproceedings{ls_leimeister, abstract = {Generative AI (GenAI) holds the potential to transform various facets of society. While research and practice increasingly explore its diverse use cases and assess its impact, many companies continue to face significant challenges in designing and developing advanced GenAI systems, hindering their ability to derive value from these technologies. To address these challenges and offer guidance on emerging GenAI development practices, we propose a reference architecture that highlights key design decisions. Using a design science research approach, we developed the reference architecture informed by consortium workshops, literature reviews, and expert interviews. The reference architecture is grounded in eight key design requirements and serves as a strategic guide to streamline system design and assist organisational decision-making and communication. Additionally, the architecture contributes to the academic discourse by offering a structured framework for future research on the design peculiarities of GenAI systems, helping bridge the gap between practical implementations and theoretical insights.}, address = {Amman, Jordan}, author = {Grau, Marc and Reinhard, Philipp and Bauer-Hänsel, Ingrid and Blohm, Ivo}, booktitle = {European Conference on Information Systems (ECIS)}, keywords = {itegpub}, publisher = {Aisel}, title = {Rethinking AI Design Practices: A Reference Architecture for Generative AI Systems}, year = 2025 }
%0 Conference Paper %1 ls_leimeister %A Grau, Marc %A Reinhard, Philipp %A Bauer-Hänsel, Ingrid %A Blohm, Ivo %B European Conference on Information Systems (ECIS) %C Amman, Jordan %D 2025 %I Aisel %T Rethinking AI Design Practices: A Reference Architecture for Generative AI Systems %U http://pubs.wi-kassel.de/wp-content/uploads/2025/06/JML_1032.pdf %X Generative AI (GenAI) holds the potential to transform various facets of society. While research and practice increasingly explore its diverse use cases and assess its impact, many companies continue to face significant challenges in designing and developing advanced GenAI systems, hindering their ability to derive value from these technologies. To address these challenges and offer guidance on emerging GenAI development practices, we propose a reference architecture that highlights key design decisions. Using a design science research approach, we developed the reference architecture informed by consortium workshops, literature reviews, and expert interviews. The reference architecture is grounded in eight key design requirements and serves as a strategic guide to streamline system design and assist organisational decision-making and communication. Additionally, the architecture contributes to the academic discourse by offering a structured framework for future research on the design peculiarities of GenAI systems, helping bridge the gap between practical implementations and theoretical insights.
Reinhard, P.; Moritz, J. M.; Wagner, S.; & Li, M. M. (2025): From Human-Human to Human-AI Delegation: A Leadership Theory Driven Investigation of Delegation. In: Poster presented at the Annual Meeting of the Academy of Management (AOM) (Vol. Poster). Copenhagen, Denmark.
- [ BibTeX ]
- [ EndNote ]
- [ URL ]
@inproceedings{ls_leimeister, abstract = {The emergence of generative AI (GenAI) has transformed work by enabling humans to delegate tasks like writing and coding to GenAI agents such as ChatGPT. While existing studies highlight AI capability awareness and perceived competence as drivers of delegation, they overlook parallels between human-AI and human-human delegation. Our ongoing research proposes that human-AI delegation can be understood through a leadership lens, with leadership experience and traits as key predictors. Hence, we investigate whether individuals with leadership experience demonstrate higher delegation levels than those without such experience. In an initial online experiment (n=48), participants were grouped by leadership experience and AI transparency to decide whether to delegate or personally perform image classification tasks. Preliminary findings indicate that under a low-transparency condition, leadership experience results in higher delegation rates. However, leadership alone does not significantly predict delegation. Transparency in GenAI consistently leads to higher delegation, while greater domain knowledge corresponds to lower delegation rates. Our ongoing research seeks to deepen understanding of delegation behavior and its predictors in the age of GenAI.}, address = {Copenhagen, Denmark}, author = {Reinhard, Philipp and Moritz, Josephine Mago and Wagner, Sophie and Li, Mahei Manhai}, booktitle = {Poster presented at the Annual Meeting of the Academy of Management (AOM)}, keywords = {itegpub}, title = {From Human-Human to Human-AI Delegation: A Leadership Theory Driven Investigation of Delegation}, volume = {Poster}, year = 2025 }
%0 Conference Paper %1 ls_leimeister %A Reinhard, Philipp %A Moritz, Josephine Mago %A Wagner, Sophie %A Li, Mahei Manhai %B Poster presented at the Annual Meeting of the Academy of Management (AOM) %C Copenhagen, Denmark %D 2025 %T From Human-Human to Human-AI Delegation: A Leadership Theory Driven Investigation of Delegation %U http://pubs.wi-kassel.de/wp-content/uploads/2025/04/JML_1016.pdf %V Poster %X The emergence of generative AI (GenAI) has transformed work by enabling humans to delegate tasks like writing and coding to GenAI agents such as ChatGPT. While existing studies highlight AI capability awareness and perceived competence as drivers of delegation, they overlook parallels between human-AI and human-human delegation. Our ongoing research proposes that human-AI delegation can be understood through a leadership lens, with leadership experience and traits as key predictors. Hence, we investigate whether individuals with leadership experience demonstrate higher delegation levels than those without such experience. In an initial online experiment (n=48), participants were grouped by leadership experience and AI transparency to decide whether to delegate or personally perform image classification tasks. Preliminary findings indicate that under a low-transparency condition, leadership experience results in higher delegation rates. However, leadership alone does not significantly predict delegation. Transparency in GenAI consistently leads to higher delegation, while greater domain knowledge corresponds to lower delegation rates. Our ongoing research seeks to deepen understanding of delegation behavior and its predictors in the age of GenAI.
Reinhard, P.; Li, M. M.; Fina, M.; & Leimeister, J. M. (2025): Fact or Fiction? Exploring Explanations to Identify Factual Confabulations in RAG-Based LLM Systems. In: Extended Abstracts of the CHI Conference on Human Factors in Computing Systems (CHI EA ’25). Yokohama, Japan.
- [ BibTeX ]
- [ EndNote ]
- [ URL ]
@inproceedings{ls_leimeister, abstract = {The adoption of generative artificial intelligence (GenAI) and large language models (LLMs) in society and business is growing rapidly. While these systems often generate convincing and coherent responses, they risk producing incorrect or non-factual information, known as confabulations or hallucinations. Consequently, users must critically assess the reliability of these outputs when interacting with LLM-based agents. Although advancements such as retrieval-augmented generation (RAG) have improved the technical performance of these systems, there is a lack of empirical models that explain how humans detect confabulations. Building on the explainable AI (XAI) literature, we examine the role of reasoning-based explanations in helping users identify confabulations in LLM systems. An online experiment (n = 97) reveals that analogical and factual explanations improve detection accuracy but require more time and cognitive effort than the no explanation baseline.}, address = {Yokohama, Japan}, author = {Reinhard, Philipp and Li, Mahei Manhai and Fina, Matteo and Leimeister, Jan Marco}, booktitle = {Extended Abstracts of the CHI Conference on Human Factors in Computing Systems (CHI EA ’25)}, keywords = {itegpub}, title = {Fact or Fiction? Exploring Explanations to Identify Factual Confabulations in RAG-Based LLM Systems}, year = 2025 }
%0 Conference Paper %1 ls_leimeister %A Reinhard, Philipp %A Li, Mahei Manhai %A Fina, Matteo %A Leimeister, Jan Marco %B Extended Abstracts of the CHI Conference on Human Factors in Computing Systems (CHI EA ’25) %C Yokohama, Japan %D 2025 %R 10.1145/3706599.3720249 %T Fact or Fiction? Exploring Explanations to Identify Factual Confabulations in RAG-Based LLM Systems %U http://pubs.wi-kassel.de/wp-content/uploads/2025/04/JML_1012.pdf %X The adoption of generative artificial intelligence (GenAI) and large language models (LLMs) in society and business is growing rapidly. While these systems often generate convincing and coherent responses, they risk producing incorrect or non-factual information, known as confabulations or hallucinations. Consequently, users must critically assess the reliability of these outputs when interacting with LLM-based agents. Although advancements such as retrieval-augmented generation (RAG) have improved the technical performance of these systems, there is a lack of empirical models that explain how humans detect confabulations. Building on the explainable AI (XAI) literature, we examine the role of reasoning-based explanations in helping users identify confabulations in LLM systems. An online experiment (n = 97) reveals that analogical and factual explanations improve detection accuracy but require more time and cognitive effort than the no explanation baseline.

2024[ to top ]

Reinhard, P.; Neis, N.; Kolb, L.; Wischer, D.; Li, M. M.; Winkelmann, A.; … Leimeister, J. M. (2024): Augmenting Frontline Service Employee Onboarding via Hybrid Intelligence: Examining the Effects of Different Degrees of Human-GenAI Interaction. In: Mandviwalla, M. Y., Söllner, M., & Tuunanen, T. (Eds.), International Conference on Design Science Research in Information Systems and Technology (DESRIST 2024). Trollhättan, Sweden.
- [ BibTeX ]
- [ EndNote ]
- [ URL ]
@inproceedings{ls_leimeister, abstract = {High turnover rates within help desks, caused by excessive workloads, make the efficient onboarding of novices a persistent and recurring challenge. Generative artificial intelligence (GenAI) possesses the potential to augment novice frontline service employees (FSE) during their onboarding phase. However, there is a lack of knowledge on how to design the interaction of FSE and GenAI. Thus, following design science research (DSR), we propose a conversational (co-)agent that leverages the capabilities of large language models and the concept of hybrid intelligence to augment novice FSE. To examine the interaction between novices and GenAI given various task difficulties, we derive and instantiate two hybrid intelligence meta-designs - a supporter that provides recommendations and a collaborator that allows for prompting the coagent. The results from an online experiment with 75 laypeople show that novices interact with GenAI more frequently and show a higher engagement, especially in difficult tasks. Overall, we uncover a paradox: Despite an increased interaction and a greater time investment, FSEs experience a lower perceived workload with a GenAI-based collaborator. From that, we derive implications for designing employee-facing co-agents in customer services.}, address = {Trollhättan, Sweden}, author = {Reinhard, Philipp and Neis, Nicolas and Kolb, Lisa and Wischer, Dennis and Li, Mahei Manhai and Winkelmann, Axel and Teuteberg, Frank and Lechner, Ulrike and Leimeister, Jan Marco}, booktitle = {International Conference on Design Science Research in Information Systems and Technology (DESRIST 2024)}, editor = {Mandviwalla, Munir Y. and Söllner, Matthias and Tuunanen, Tuure}, keywords = {itegpub}, title = {Augmenting Frontline Service Employee Onboarding via Hybrid Intelligence: Examining the Effects of Different Degrees of Human-GenAI Interaction}, year = 2024 }
%0 Conference Paper %1 ls_leimeister %A Reinhard, Philipp %A Neis, Nicolas %A Kolb, Lisa %A Wischer, Dennis %A Li, Mahei Manhai %A Winkelmann, Axel %A Teuteberg, Frank %A Lechner, Ulrike %A Leimeister, Jan Marco %B International Conference on Design Science Research in Information Systems and Technology (DESRIST 2024) %C Trollhättan, Sweden %D 2024 %E Mandviwalla, Munir Y. %E Söllner, Matthias %E Tuunanen, Tuure %T Augmenting Frontline Service Employee Onboarding via Hybrid Intelligence: Examining the Effects of Different Degrees of Human-GenAI Interaction %U http://pubs.wi-kassel.de/wp-content/uploads/2024/04/JML_969.pdf %X High turnover rates within help desks, caused by excessive workloads, make the efficient onboarding of novices a persistent and recurring challenge. Generative artificial intelligence (GenAI) possesses the potential to augment novice frontline service employees (FSE) during their onboarding phase. However, there is a lack of knowledge on how to design the interaction of FSE and GenAI. Thus, following design science research (DSR), we propose a conversational (co-)agent that leverages the capabilities of large language models and the concept of hybrid intelligence to augment novice FSE. To examine the interaction between novices and GenAI given various task difficulties, we derive and instantiate two hybrid intelligence meta-designs - a supporter that provides recommendations and a collaborator that allows for prompting the coagent. The results from an online experiment with 75 laypeople show that novices interact with GenAI more frequently and show a higher engagement, especially in difficult tasks. Overall, we uncover a paradox: Despite an increased interaction and a greater time investment, FSEs experience a lower perceived workload with a GenAI-based collaborator. From that, we derive implications for designing employee-facing co-agents in customer services.
Reinhard, P.; Li, M. M.; Peters, C.; & Leimeister, J. M. (2024): LET EMPLOYEES TRAIN THEIR OWN CHATBOTS: DESIGN OF GENERATIVE AI-ENABLED DELEGATION SYSTEMS. In: European Conference on Information Systems (ECIS). Paphos, Cyprus.
- [ BibTeX ]
- [ EndNote ]
- [ URL ]
@inproceedings{ls_leimeister, abstract = {While chatbots can be implemented with very little effort, scaling and maintaining chatbots remains a challenge. This is crucial in knowledge-intensive customer service like IT support, where domain knowledge must stay current with the evolving IT landscape. Following design science research, we derive design principles for a generative AI (GPT4) enabled textual training data creation and curation system (T²C²) as part of a new class of systems – bot delegation systems. For the design of T²C², chatbot and domain expert viewpoints are integrated. We evaluate two instances of T²C², each with distinct degrees of human-ai delegation where employees act both as creators and curators of training data. The paper’s theoretical contribution is two-fold: (1) we present a novel kernel theory that represents the material characteristics of bot delegation systems by contextualizing the IS delegation framework to the self-determination theory; (2) the design and evaluation of T²C² as the built-and-evaluated artifact.}, address = {Paphos, Cyprus}, author = {Reinhard, Philipp and Li, Mahei Manhai and Peters, Christoph and Leimeister, Jan Marco}, booktitle = {European Conference on Information Systems (ECIS)}, keywords = {itegpub}, title = {LET EMPLOYEES TRAIN THEIR OWN CHATBOTS: DESIGN OF GENERATIVE AI-ENABLED DELEGATION SYSTEMS}, year = 2024 }
%0 Conference Paper %1 ls_leimeister %A Reinhard, Philipp %A Li, Mahei Manhai %A Peters, Christoph %A Leimeister, Jan Marco %B European Conference on Information Systems (ECIS) %C Paphos, Cyprus %D 2024 %T LET EMPLOYEES TRAIN THEIR OWN CHATBOTS: DESIGN OF GENERATIVE AI-ENABLED DELEGATION SYSTEMS %U http://pubs.wi-kassel.de/wp-content/uploads/2024/04/JML_970.pdf %X While chatbots can be implemented with very little effort, scaling and maintaining chatbots remains a challenge. This is crucial in knowledge-intensive customer service like IT support, where domain knowledge must stay current with the evolving IT landscape. Following design science research, we derive design principles for a generative AI (GPT4) enabled textual training data creation and curation system (T²C²) as part of a new class of systems – bot delegation systems. For the design of T²C², chatbot and domain expert viewpoints are integrated. We evaluate two instances of T²C², each with distinct degrees of human-ai delegation where employees act both as creators and curators of training data. The paper’s theoretical contribution is two-fold: (1) we present a novel kernel theory that represents the material characteristics of bot delegation systems by contextualizing the IS delegation framework to the self-determination theory; (2) the design and evaluation of T²C² as the built-and-evaluated artifact.
Reinhard, P.; Li, M. M.; Peters, C.; & Leimeister, J. M. (2024): Augmentierung statt Automatisierung: Nachhaltigkeit in der Entwicklung von KI-Systemen aus der Mitarbeitendenperspektive. In: Forum Dienstleistungsmanagement, Vol. Sustainable Service Management, Erscheinungsjahr/Year: 2024. .
- [ BibTeX ]
- [ EndNote ]
- [ URL ]
@article{ls_leimeister, author = {Reinhard, Philipp and Li, Mahei Manhai and Peters, Christoph and Leimeister, Jan Marco}, editor = {Bruhn, Manfred and Hadwich, Karsten}, journal = {Forum Dienstleistungsmanagement}, keywords = {itegpub}, title = {Augmentierung statt Automatisierung: Nachhaltigkeit in der Entwicklung von KI-Systemen aus der Mitarbeitendenperspektive}, volume = {Sustainable Service Management}, year = 2024 }
%0 Journal Article %1 ls_leimeister %A Reinhard, Philipp %A Li, Mahei Manhai %A Peters, Christoph %A Leimeister, Jan Marco %D 2024 %E Bruhn, Manfred %E Hadwich, Karsten %J Forum Dienstleistungsmanagement %R 10.1007/978-3-658-45148-6_15 %T Augmentierung statt Automatisierung: Nachhaltigkeit in der Entwicklung von KI-Systemen aus der Mitarbeitendenperspektive %U http://pubs.wi-kassel.de/wp-content/uploads/2025/06/JML_1027.pdf %V Sustainable Service Management
Reinhard, P.; Li, M. M.; Peters, C.; & Leimeister, J. M. (2024): Generative AI in Customer Support Services: A Framework for Augmenting the Routines of Frontline Service Employees. In: Hawaii International Conference on System Sciences (HICSS). Waikiki, Hawaii, USA.
- [ BibTeX ]
- [ EndNote ]
- [ URL ]
@inproceedings{ls_leimeister, abstract = {Customer support service employees are facing an increased workload, while artificial intelligence (AI) appears to possess the potential to change the way we work. With the advent of modern types of generative AI, new opportunities to augment frontline service employees have emerged. However, little is known about how to integrate generative AI in customer support service organizations and purposefully change service employee work routines. Following multi-method qualitative research, we performed a literature review, conducted workshops, and interviewed IT support agents, managers, and AI experts. Thereby, we examine AI augmentation for frontline service employees in the context of IT support to carve out where and how GenAI can be leveraged to develop more efficient and higher-quality customer support. Our resulting framework reveals that especially adapting solutions and retaining knowledge is subject to a high degree of AI augmentation.}, address = {Waikiki, Hawaii, USA}, author = {Reinhard, Philipp and Li, Mahei Manhai and Peters, Christoph and Leimeister, Jan Marco}, booktitle = {Hawaii International Conference on System Sciences (HICSS)}, keywords = {itegpub}, title = {Generative AI in Customer Support Services: A Framework for Augmenting the Routines of Frontline Service Employees}, year = 2024 }
%0 Conference Paper %1 ls_leimeister %A Reinhard, Philipp %A Li, Mahei Manhai %A Peters, Christoph %A Leimeister, Jan Marco %B Hawaii International Conference on System Sciences (HICSS) %C Waikiki, Hawaii, USA %D 2024 %T Generative AI in Customer Support Services: A Framework for Augmenting the Routines of Frontline Service Employees %U https://pubs.wi-kassel.de/wp-content/uploads/2023/12/JML_956.pdf %X Customer support service employees are facing an increased workload, while artificial intelligence (AI) appears to possess the potential to change the way we work. With the advent of modern types of generative AI, new opportunities to augment frontline service employees have emerged. However, little is known about how to integrate generative AI in customer support service organizations and purposefully change service employee work routines. Following multi-method qualitative research, we performed a literature review, conducted workshops, and interviewed IT support agents, managers, and AI experts. Thereby, we examine AI augmentation for frontline service employees in the context of IT support to carve out where and how GenAI can be leveraged to develop more efficient and higher-quality customer support. Our resulting framework reveals that especially adapting solutions and retaining knowledge is subject to a high degree of AI augmentation.

2023[ to top ]

Li, M. M.; Reinhard, P.; Oeste-Reiss, S.; Peters, C.; & Leimeister, J. M. (2023): A Value Co-Creation Perspective on Data Labeling in Hybrid Intelligence Systems: A Design Study. In: Information Systems (IS), Erscheinungsjahr/Year: 2023. p. 102311.
- [ BibTeX ]
- [ EndNote ]
- [ URL ]
@article{ls_leimeister, abstract = {The adoption of innovative technologies confronts IT-Service-Management (ITSM) with an increasing volume and variety of requests. Artificial intelligence (AI) possesses the potential to augment customer service employees. However, the training data for AI systems are annotated by domain experts with little interest in labeling correctly due to their limited perceived value. Ultimately, insufficient labeled data leads to diminishing returns in AI performance. Following a design science research approach, we provide a novel human-in-the-loop (HIL) design for ITSM support ticket recommendations by incorporating a value co-creation perspective. The design incentivizes ITSM agents to provide labels during their everyday ticket-handling procedures. We develop a functional prototype based on 17,120 support tickets provided by a pilot partner as an instantiation and evaluate the design through accuracy metrics and user evaluations. Our evaluation revealed that recommendations after label improvement showed increased user ratings, and users are willing to contribute their domain knowledge. The improved labels can be utilized to continuously enhance the AI system as rewards showed increasing growth with decreasing marginal returns. Overall, our results emphasize agents' need for value-in-use by providing better results if they improve the labeling of support tickets pre-labeled by AI. Thus, we provide prescriptive knowledge of a novel HIL design that enables efficient and interactive labeling in the context of diverse applications of reinforcement learning systems.}, author = {Li, Mahei Manhai and Reinhard, Philipp and Oeste-Reiss, Sarah and Peters, Christoph and Leimeister, Jan Marco}, journal = {Information Systems (IS)}, keywords = {IT_support}, pages = 102311, title = {A Value Co-Creation Perspective on Data Labeling in Hybrid Intelligence Systems: A Design Study}, year = 2023 }
%0 Journal Article %1 ls_leimeister %A Li, Mahei Manhai %A Reinhard, Philipp %A Oeste-Reiss, Sarah %A Peters, Christoph %A Leimeister, Jan Marco %D 2023 %J Information Systems (IS) %P 102311 %R 10.1016/j.is.2023.102311 %T A Value Co-Creation Perspective on Data Labeling in Hybrid Intelligence Systems: A Design Study %U https://doi.org/10.1016/j.is.2023.102311 %X The adoption of innovative technologies confronts IT-Service-Management (ITSM) with an increasing volume and variety of requests. Artificial intelligence (AI) possesses the potential to augment customer service employees. However, the training data for AI systems are annotated by domain experts with little interest in labeling correctly due to their limited perceived value. Ultimately, insufficient labeled data leads to diminishing returns in AI performance. Following a design science research approach, we provide a novel human-in-the-loop (HIL) design for ITSM support ticket recommendations by incorporating a value co-creation perspective. The design incentivizes ITSM agents to provide labels during their everyday ticket-handling procedures. We develop a functional prototype based on 17,120 support tickets provided by a pilot partner as an instantiation and evaluate the design through accuracy metrics and user evaluations. Our evaluation revealed that recommendations after label improvement showed increased user ratings, and users are willing to contribute their domain knowledge. The improved labels can be utilized to continuously enhance the AI system as rewards showed increasing growth with decreasing marginal returns. Overall, our results emphasize agents' need for value-in-use by providing better results if they improve the labeling of support tickets pre-labeled by AI. Thus, we provide prescriptive knowledge of a novel HIL design that enables efficient and interactive labeling in the context of diverse applications of reinforcement learning systems.
Langes, B.; Peters, C.; Reinhard, P.; Li, M. M.; Hermann, S.; Katzenberger, R.; … Leimeister, J. M. (2023): Hybrid Intelligence Service Support. In: Langes, B., Peters, C., Boes, A., & Leimeister, J. M. (Eds.), Innovationen für datenbasierte Wertschöpfung. Mit internetbasierten Dienstleistungen die Zukunft der Wirtschaft gestalten (pp. 37–49). München: ISF München.
- [ BibTeX ]
- [ EndNote ]
- [ URL ]
@incollection{ls_leimeister, address = {München}, author = {Langes, Barbara and Peters, Christoph and Reinhard, Philipp and Li, Mahei Manhai and Hermann, Sophia and Katzenberger, Rebecca and Boes, Andreas and Leimeister, Jan Marco}, booktitle = {Innovationen für datenbasierte Wertschöpfung. Mit internetbasierten Dienstleistungen die Zukunft der Wirtschaft gestalten}, editor = {Langes, Barbara and Peters, Christoph and Boes, Andreas and Leimeister, Jan Marco}, keywords = {Hybride_Intelligenz}, pages = {37-49}, publisher = {ISF München}, title = {Hybrid Intelligence Service Support}, year = 2023 }
%0 Book Section %1 ls_leimeister %A Langes, Barbara %A Peters, Christoph %A Reinhard, Philipp %A Li, Mahei Manhai %A Hermann, Sophia %A Katzenberger, Rebecca %A Boes, Andreas %A Leimeister, Jan Marco %B Innovationen für datenbasierte Wertschöpfung. Mit internetbasierten Dienstleistungen die Zukunft der Wirtschaft gestalten %C München %D 2023 %E Langes, Barbara %E Peters, Christoph %E Boes, Andreas %E Leimeister, Jan Marco %I ISF München %P 37-49 %R 10.36194/ING_2023 %T Hybrid Intelligence Service Support %U https://pubs.wi-kassel.de/wp-content/uploads/2023/07/JML_941.pdf
Reinhard, P.; Li, M. M.; Dickhaut, E.; Peters, C.; & Leimeister, J. M. (2023): Empowering Recommender Systems in ITSM: A Pipeline Reference Model for AI-based Textual Data Quality Enrichment. In: Gerber, A., & Baskerville, R. (Eds.), International Conference on Design Science Research (DESRIST). Pretoria, South Africa.
- [ BibTeX ]
- [ EndNote ]
- [ URL ]
@inproceedings{ls_leimeister, abstract = {AI-based recommendation systems to augment working conditions in the field of IT service management (ITSM) have attracted new attention. However, many IT support organizations possess high volumes of tickets but are confronted with low quality, to which they train the underlying models of their AI systems. In particular, support tickets are documented insufficiently due to time pressure and lack of motivation. Following design science research, we design and evaluate an analytics pipeline to address the data quality issue. The pipeline can be applied to assess and extract high-quality support tickets for subsequent model training and operation. Based on a data set of 60.000 real-life support tickets from a manufacturing company, we develop the artifact, instantiate a recommender system and achieve a higher pre-diction performance in comparison to naïve enrichment methods. In terms of data management literature, we contribute to the understanding of as-sessing textual ticket data quality. By deriving a pipeline reference model, we move towards a general approach to designing machine learning-driven data quality analytics pipelines for attached recommender systems.}, address = {Pretoria, South Africa}, author = {Reinhard, Philipp and Li, Mahei Manhai and Dickhaut, Ernestine and Peters, Christoph and Leimeister, Jan Marco}, booktitle = {International Conference on Design Science Research (DESRIST)}, editor = {Gerber, Aurona and Baskerville, Richard}, keywords = {itegpub}, title = {Empowering Recommender Systems in ITSM: A Pipeline Reference Model for AI-based Textual Data Quality Enrichment}, year = 2023 }
%0 Conference Paper %1 ls_leimeister %A Reinhard, Philipp %A Li, Mahei Manhai %A Dickhaut, Ernestine %A Peters, Christoph %A Leimeister, Jan Marco %B International Conference on Design Science Research (DESRIST) %C Pretoria, South Africa %D 2023 %E Gerber, Aurona %E Baskerville, Richard %T Empowering Recommender Systems in ITSM: A Pipeline Reference Model for AI-based Textual Data Quality Enrichment %U https://pubs.wi-kassel.de/wp-content/uploads/2023/06/JML_932.pdf %X AI-based recommendation systems to augment working conditions in the field of IT service management (ITSM) have attracted new attention. However, many IT support organizations possess high volumes of tickets but are confronted with low quality, to which they train the underlying models of their AI systems. In particular, support tickets are documented insufficiently due to time pressure and lack of motivation. Following design science research, we design and evaluate an analytics pipeline to address the data quality issue. The pipeline can be applied to assess and extract high-quality support tickets for subsequent model training and operation. Based on a data set of 60.000 real-life support tickets from a manufacturing company, we develop the artifact, instantiate a recommender system and achieve a higher pre-diction performance in comparison to naïve enrichment methods. In terms of data management literature, we contribute to the understanding of as-sessing textual ticket data quality. By deriving a pipeline reference model, we move towards a general approach to designing machine learning-driven data quality analytics pipelines for attached recommender systems.
Reinhard, P.; Wischer, D.; Verlande, L.; Neis, N.; & Li, M. M. (2023): Towards designing an AI-based conversational agent for on-the-job training of customer support novices. In: International Conference on Design Science Research (DESRIST). Pretoria, South Africa.
- [ BibTeX ]
- [ EndNote ]
- [ URL ]
@inproceedings{ls_leimeister, abstract = {Due to the high drop-out rates in IT support desks, efficient onboarding of novices becomes a relevant and recurring challenge. Especially in the case of IT support, solving technical issues and service requests while the conversation with the customer is still ongoing imposes high demands on novice support agents. As artificial intelligence (AI) can already classify service requests and help find solutions, AIbased augmentation holds great potential for improving the onboarding phase and reducing time-to-performance. For this reason, we propose an AI-based conversational (co-)agent during the onboarding phase of customer support novices to reduce the time spent on service tasks and enable on-the-job training. Following action design research, we aim to develop an instantiation of an AI-based co-agent to reduce the job demand for the service center agent novices and augment problem-solving capabilities by considering cognitive load. The co-agent will be implemented with one development partner and evaluated with two different case partner organizations. In this research-in-progress project, we developed a low-fidelity prototype and derived a tentative architecture that allows for a generalized development of such conversational agents in customer service organizations.}, address = {Pretoria, South Africa}, author = {Reinhard, Philipp and Wischer, Dennis and Verlande, Lisa and Neis, Nicolas and Li, Mahei Manhai}, booktitle = {International Conference on Design Science Research (DESRIST)}, keywords = {itegpub}, title = {Towards designing an AI-based conversational agent for on-the-job training of customer support novices}, year = 2023 }
%0 Conference Paper %1 ls_leimeister %A Reinhard, Philipp %A Wischer, Dennis %A Verlande, Lisa %A Neis, Nicolas %A Li, Mahei Manhai %B International Conference on Design Science Research (DESRIST) %C Pretoria, South Africa %D 2023 %T Towards designing an AI-based conversational agent for on-the-job training of customer support novices %U https://pubs.wi-kassel.de/wp-content/uploads/2023/06/JML_931.pdf %X Due to the high drop-out rates in IT support desks, efficient onboarding of novices becomes a relevant and recurring challenge. Especially in the case of IT support, solving technical issues and service requests while the conversation with the customer is still ongoing imposes high demands on novice support agents. As artificial intelligence (AI) can already classify service requests and help find solutions, AIbased augmentation holds great potential for improving the onboarding phase and reducing time-to-performance. For this reason, we propose an AI-based conversational (co-)agent during the onboarding phase of customer support novices to reduce the time spent on service tasks and enable on-the-job training. Following action design research, we aim to develop an instantiation of an AI-based co-agent to reduce the job demand for the service center agent novices and augment problem-solving capabilities by considering cognitive load. The co-agent will be implemented with one development partner and evaluated with two different case partner organizations. In this research-in-progress project, we developed a low-fidelity prototype and derived a tentative architecture that allows for a generalized development of such conversational agents in customer service organizations.
Reinhard, P.; Li, M. M.; Dickhaut, E.; Reh, C.; Peters, C.; & Leimeister, J. M. (2023): A Conceptual Model for Labeling in Reinforcement Learning Systems: A Value Co-Creation Perspective. In: Gerber, A., & Baskerville, R. (Eds.), International Conference on Design Science Research (DESRIST). Pretoria, South Africa.
- [ BibTeX ]
- [ EndNote ]
- [ URL ]
@inproceedings{ls_leimeister, abstract = {Artificial intelligence (AI) possesses the potential to augment customer service employees e.g. via decision support or solution recommendations. Still, its underlying data for training and testing the AI systems is provided by human annotators through human-in-the-loop configurations. However, due to the high effort for annotators and lack of incentives, AI systems face low underlying data quality. That in turn results in low prediction performance and limited acceptance by the targeted user group. Faced with the enormous volume and increasing complexity of service requests, IT service management (ITSM) especially, relies on high data quality for AI systems and in-corporating domain-specific knowledge. By analyzing the existing labeling process in that specific case, we design a revised to-be process and develop a conceptual model from a value co-creation perspective. Finally, a functional prototype as an instantiation in the ITSM domain is implemented and evaluated through accuracy metrics and user evaluation. The results show that the new process increases the perceived value of both labeling quality and the perceived prediction quality. Thus, we contribute a conceptual model that supports the systematic design of efficient and interactive labeling processes in diverse applications of reinforcement learning systems.}, address = {Pretoria, South Africa}, author = {Reinhard, Philipp and Li, Mahei Manhai and Dickhaut, Ernestine and Reh, Cornelius and Peters, Christoph and Leimeister, Jan Marco}, booktitle = {International Conference on Design Science Research (DESRIST)}, editor = {Gerber, Aurona and Baskerville, Richard}, keywords = {Interactive_labeling}, title = {A Conceptual Model for Labeling in Reinforcement Learning Systems: A Value Co-Creation Perspective}, year = 2023 }
%0 Conference Paper %1 ls_leimeister %A Reinhard, Philipp %A Li, Mahei Manhai %A Dickhaut, Ernestine %A Reh, Cornelius %A Peters, Christoph %A Leimeister, Jan Marco %B International Conference on Design Science Research (DESRIST) %C Pretoria, South Africa %D 2023 %E Gerber, Aurona %E Baskerville, Richard %T A Conceptual Model for Labeling in Reinforcement Learning Systems: A Value Co-Creation Perspective %U https://pubs.wi-kassel.de/wp-content/uploads/2023/03/JML_923.pdf %X Artificial intelligence (AI) possesses the potential to augment customer service employees e.g. via decision support or solution recommendations. Still, its underlying data for training and testing the AI systems is provided by human annotators through human-in-the-loop configurations. However, due to the high effort for annotators and lack of incentives, AI systems face low underlying data quality. That in turn results in low prediction performance and limited acceptance by the targeted user group. Faced with the enormous volume and increasing complexity of service requests, IT service management (ITSM) especially, relies on high data quality for AI systems and in-corporating domain-specific knowledge. By analyzing the existing labeling process in that specific case, we design a revised to-be process and develop a conceptual model from a value co-creation perspective. Finally, a functional prototype as an instantiation in the ITSM domain is implemented and evaluated through accuracy metrics and user evaluation. The results show that the new process increases the perceived value of both labeling quality and the perceived prediction quality. Thus, we contribute a conceptual model that supports the systematic design of efficient and interactive labeling processes in diverse applications of reinforcement learning systems.

2022[ to top ]

Reinhard, P.; Li, M. M.; & Leimeister, J. M. (2022): Towards an Employee-Centered Design for Human-AI Collaboration: How Work Design Theory Informs the Design of AI Systems. In: Pre-ICIS Workshop on the Changing Nature of Work (SIG 11th CNoW). Copenhagen, Denmark.
- [ BibTeX ]
- [ EndNote ]
- [ URL ]
@inproceedings{ls_leimeister, abstract = {IT support is under growing pressure to ensure efficient, flexible, and scalable use of digital technologies (Kumbakara, 2008). As a result, technical support staff is affected by monotonous work and work overload (Schmidt et al., 2022). Our research aims to augment the precarious workplace of support agents with artificial intelligence (AI). To incorporate an employee-centered perspective a priori and ensure positive impacts, we propose a framework for combining work design theory (e.g. Demerouti et al., 2001) and design science theory (Peffers et al., 2007, Niehaves & Ortbach 2016). The advances in AI promise to leverage large potential in optimizing and enhancing service processes and workplaces (Huang & Rust, 2018, de Keyser, 2019). Introducing AI into service processes, imply effects on work characteristics (Larivière et al., 2017). By combining human and artificial intelligence we propose hybrid intelligence (Dellermann et al., 2019) as a suitable solution for mitigating the persistent issues of support workers and the possible negative impacts of AI. To a great extent, IS research emphasizes the implied impacts of AI use in workplaces (Verma & Singh 2022, Wang et al., 2020). As such, work design models are widely used to empirically evaluate the impacts of AI design (Sturm & Peters, 2020), but are rarely utilized to substantiate the design of AI-augmented work systems. Only Poser et al. (2022) and Zschech et al. (2021) recently applied such models. The goal of this paper is to overcome the lack of work design in design science research (DSR) for AI-based systems and to steer the development into desired socio-technological configurations. The here presented work is expected to answer : How can work design theory inform the design of AI-augmented workplaces? [RQ1] How should a hybrid intelligence system be designed to augment IT support agents’ workplaces by incorporating work design theory? [RQ2] To systematically design the integration of AI, we make use of the DSR paradigm (Peffers et al., 2007). We first interview support agents and utilize the organizing move theory of Pentland (1992) and the technical support work theory of Das (2003) to ensure relevance. Based on a review of the IS literature on work design theories, we then derive a preliminary theoretical framework (Paul & Benito, 2018) [RQ1]. The framework represents a kernel theory for the development of meta design requirements. Contributing to the second research question, we design and subsequently evaluate the augmentation based on work-related outcomes [RQ2].}, address = {Copenhagen, Denmark}, author = {Reinhard, Philipp and Li, Mahei Manhai and Leimeister, Jan Marco}, booktitle = {Pre-ICIS Workshop on the Changing Nature of Work (SIG 11th CNoW)}, keywords = {itegpub}, title = {Towards an Employee-Centered Design for Human-AI Collaboration: How Work Design Theory Informs the Design of AI Systems}, year = 2022 }
%0 Conference Paper %1 ls_leimeister %A Reinhard, Philipp %A Li, Mahei Manhai %A Leimeister, Jan Marco %B Pre-ICIS Workshop on the Changing Nature of Work (SIG 11th CNoW) %C Copenhagen, Denmark %D 2022 %T Towards an Employee-Centered Design for Human-AI Collaboration: How Work Design Theory Informs the Design of AI Systems %U https://pubs.wi-kassel.de/wp-content/uploads/2023/02/JML_921.pdf %X IT support is under growing pressure to ensure efficient, flexible, and scalable use of digital technologies (Kumbakara, 2008). As a result, technical support staff is affected by monotonous work and work overload (Schmidt et al., 2022). Our research aims to augment the precarious workplace of support agents with artificial intelligence (AI). To incorporate an employee-centered perspective a priori and ensure positive impacts, we propose a framework for combining work design theory (e.g. Demerouti et al., 2001) and design science theory (Peffers et al., 2007, Niehaves & Ortbach 2016). The advances in AI promise to leverage large potential in optimizing and enhancing service processes and workplaces (Huang & Rust, 2018, de Keyser, 2019). Introducing AI into service processes, imply effects on work characteristics (Larivière et al., 2017). By combining human and artificial intelligence we propose hybrid intelligence (Dellermann et al., 2019) as a suitable solution for mitigating the persistent issues of support workers and the possible negative impacts of AI. To a great extent, IS research emphasizes the implied impacts of AI use in workplaces (Verma & Singh 2022, Wang et al., 2020). As such, work design models are widely used to empirically evaluate the impacts of AI design (Sturm & Peters, 2020), but are rarely utilized to substantiate the design of AI-augmented work systems. Only Poser et al. (2022) and Zschech et al. (2021) recently applied such models. The goal of this paper is to overcome the lack of work design in design science research (DSR) for AI-based systems and to steer the development into desired socio-technological configurations. The here presented work is expected to answer : How can work design theory inform the design of AI-augmented workplaces? [RQ1] How should a hybrid intelligence system be designed to augment IT support agents’ workplaces by incorporating work design theory? [RQ2] To systematically design the integration of AI, we make use of the DSR paradigm (Peffers et al., 2007). We first interview support agents and utilize the organizing move theory of Pentland (1992) and the technical support work theory of Das (2003) to ensure relevance. Based on a review of the IS literature on work design theories, we then derive a preliminary theoretical framework (Paul & Benito, 2018) [RQ1]. The framework represents a kernel theory for the development of meta design requirements. Contributing to the second research question, we design and subsequently evaluate the augmentation based on work-related outcomes [RQ2].

2021[ to top ]

Hildebrand, A.; Jakopin, N.; Reinhard, P.; & Riegel, L. (2021): Der Smart-City-Markt in Deutschland: 2021-2026 (1st ed.).
- [ BibTeX ]
- [ EndNote ]
- [ URL ]
@book{ls_leimeister, abstract = {Die Corona-Pandemie, Klimaschutzziele, die zunehmende Urbanisierung und der internationale Wettbewerb um technologische Trends und Standortvorteile für Industrie stimulieren die Digitalisierung großer Teile der Gesellschaft. Städte und Kommunen stellt diese Entwicklung vor große Herausforderungen hinsichtlich der Priorisierung und Finanzierung von Projekten rund um das Internet der Dinge (englisch Internet of Things, kurz: IoT) und die Smart City. Das Konzept der Smart City hat sich in den letzten Jahren zu einem Treiber der Automatisierung von zentralen Säulen des gesellschaftlichen und wirtschaftlichen Zusammenlebens entwickelt. Mit der Studie „Der deutsche Smart-City-Markt 2021–2026“ haben eco – Verband der Internetwirtschaft e. V. – und Arthur D. Little anhand von Analysen und Experteninterviews untersucht, wie sich der Markt und dessen Segmente in den kommenden fünf Jahren entwickeln werden, welchen Effekt die Smart City auf Klimaziele haben wird und welche Implikationen sich daraus für Städte und Unternehmen ergeben.}, author = {Hildebrand, Anna and Jakopin, Nejc and Reinhard, Philipp and Riegel, Lars}, edition = 1, keywords = {Nachhaltigkeit}, title = {Der Smart-City-Markt in Deutschland: 2021-2026}, year = 2021 }
%0 Book %1 ls_leimeister %A Hildebrand, Anna %A Jakopin, Nejc %A Reinhard, Philipp %A Riegel, Lars %D 2021 %T Der Smart-City-Markt in Deutschland: 2021-2026 %U https://pubs.wi-kassel.de/wp-content/uploads/2022/08/JML_884.pdf %X Die Corona-Pandemie, Klimaschutzziele, die zunehmende Urbanisierung und der internationale Wettbewerb um technologische Trends und Standortvorteile für Industrie stimulieren die Digitalisierung großer Teile der Gesellschaft. Städte und Kommunen stellt diese Entwicklung vor große Herausforderungen hinsichtlich der Priorisierung und Finanzierung von Projekten rund um das Internet der Dinge (englisch Internet of Things, kurz: IoT) und die Smart City. Das Konzept der Smart City hat sich in den letzten Jahren zu einem Treiber der Automatisierung von zentralen Säulen des gesellschaftlichen und wirtschaftlichen Zusammenlebens entwickelt. Mit der Studie „Der deutsche Smart-City-Markt 2021–2026“ haben eco – Verband der Internetwirtschaft e. V. – und Arthur D. Little anhand von Analysen und Experteninterviews untersucht, wie sich der Markt und dessen Segmente in den kommenden fünf Jahren entwickeln werden, welchen Effekt die Smart City auf Klimaziele haben wird und welche Implikationen sich daraus für Städte und Unternehmen ergeben. %7 1 %@ 978-3-9821487-8-6
Rohde, F.; Wagner, J.; Reinhard, P.; Petschow, U.; Meyer, A.; Voß, M.; & Mollen, A. (2021): Nachhaltigkeitskriterien für künstliche Intelligenz - Entwicklung eines Kriterien- und Indikatorensets für die Nachhaltigkeitsbewertung von KI-Systemen entlang des LebenszyklusSchriftenreihe des IÖW. Berlin: Institut für ökologische Wirtschaftsforschung.
- [ BibTeX ]
- [ EndNote ]
- [ URL ]
@book{ls_leimeister, abstract = {Technische Dynamiken, die zunehmende Durchdringung vieler gesellschaftlicher und wirtschaftlicher Bereiche mit Informations- und Kommunikationstechnologien sowie die Verfügbarkeit großer Rechenkapazitäten zur Datenverarbeitung haben zur Entwicklung immer leistungsfähigerer Modelle des maschinellen Lernens (ML) geführt. Diese werden häufig mit dem Überbegriff „Künstliche Intelligenz“ bezeichnet, obwohl sie korrekterweise nur als schwache künstliche Intelligenz einzustufenen sind. Technologien basierend auf ML werden mittlerweile in vielen gesellschaftlichen Bereichen und Industriesektoren eingesetzt und sind mit großen Erwartungen in Bezug auf die Optimierung von Prozessen oder der Entscheidungsfindung verbunden. Weil die zugrundeliegenden Modelle für sehr unterschiedliche Optimierungsaufgaben eingesetzt werden können, werden sie auch als general purpose technology bezeichnet. Die zunehmende Nutzung dieser immer komplexer werdenden Systeme hat weltweit Debatten über diskriminierende Effekte, intransparente Entscheidungs- und Optimierungsprozesse oder die Reproduktion gesellschaftlicher Ungleichheiten aufgeworfen. Zunehmend werden auch die Energieverbräuche und Treibhausgasemissionen in der KI-Modellentwicklung und -anwendung diskutiert sowie weitreichendere Folgen auf Arbeitsmärkte, Konsummuster oder die Marktmacht großer Unternehmen. Dementsprechend sind Systeme künstlicher Intelligenz mit vielfältigen gesellschaftlichen, ökologischen und ökonomischen Herausforderungen verbunden. Dieses Diskussionspapier entwickelt eine übergreifende Nachhaltigkeitsperspektive auf künstliche Intelligenz und basiert auf den Arbeiten im Forschungsprojekt „SustAIn – Nachhaltigkeitsindex für Künstliche Intelligenz“. Das Ziel ist, aktuelle Diskussionen zu verantwortungsvoller KI aufzugreifen und zu einer übergreifenden Perspektive auf nachhaltige KI zu erweitern. Die Gestaltung dieser sozio-technischen Systeme ist dabei genauso relevant wie die Auswirkungen ihrer Anwendung. Betrachtet werden die sozialen, ökologischen und ökonomischen Auswirkungen entlang des gesamten KI-Lebenszyklus sowie die organisationale Einbettung dieser Systeme. Dafür werden zunächst die begrifflichen Grundlagen erläutert und eine Nachhaltigkeitsperspektive auf KI entwickelt. Basierend auf einer Analyse bestehender wissenschaftlicher und gesellschaftlicher Diskurse über die sozialen, ökologischen und ökonomischen Auswirkungen von KI werden dreizehn Nachhaltigkeitskriterien mit entsprechenden Nachhaltigkeitsindikatoren vorgeschlagen sowie Querschnittsindikatoren, die sich neben weiteren Indikatoren auf die organisationale Einbettung dieser Systeme beziehen. Abschließend wird skizziert, wie dieses Kriterien- und Indikatorenset im weiteren Forschungsprozess in Bewertungsinstrumente übersetzt wird. Ziel der hier vorgestellten Forschung ist es, den gesellschaftlichen Diskurs zur Nachhaltigkeit von KI-Systemen zu stärken sowie eine systematische Nachhaltigkeitsbewertung von KI-basierten Systemen zu ermöglichen, um deren Entwicklung und Nutzung im Sinne der Nachhaltigkeit zu stärken.}, address = {Berlin}, author = {Rohde, Friederike and Wagner, Josephin and Reinhard, Philipp and Petschow, Ulrich and Meyer, Andreas and Voß, Marcus and Mollen, Anne}, booktitle = {Schriftenreihe des IÖW}, keywords = {Nachhaltigkeit}, number = 220, publisher = {Institut für ökologische Wirtschaftsforschung}, series = {IÖW-Schriftenreihe}, title = {Nachhaltigkeitskriterien für künstliche Intelligenz - Entwicklung eines Kriterien- und Indikatorensets für die Nachhaltigkeitsbewertung von KI-Systemen entlang des Lebenszyklus}, year = 2021 }
%0 Book %1 ls_leimeister %A Rohde, Friederike %A Wagner, Josephin %A Reinhard, Philipp %A Petschow, Ulrich %A Meyer, Andreas %A Voß, Marcus %A Mollen, Anne %B Schriftenreihe des IÖW %C Berlin %D 2021 %I Institut für ökologische Wirtschaftsforschung %N 220 %T Nachhaltigkeitskriterien für künstliche Intelligenz - Entwicklung eines Kriterien- und Indikatorensets für die Nachhaltigkeitsbewertung von KI-Systemen entlang des Lebenszyklus %U https://pubs.wi-kassel.de/wp-content/uploads/2022/08/JML_883.pdf %X Technische Dynamiken, die zunehmende Durchdringung vieler gesellschaftlicher und wirtschaftlicher Bereiche mit Informations- und Kommunikationstechnologien sowie die Verfügbarkeit großer Rechenkapazitäten zur Datenverarbeitung haben zur Entwicklung immer leistungsfähigerer Modelle des maschinellen Lernens (ML) geführt. Diese werden häufig mit dem Überbegriff „Künstliche Intelligenz“ bezeichnet, obwohl sie korrekterweise nur als schwache künstliche Intelligenz einzustufenen sind. Technologien basierend auf ML werden mittlerweile in vielen gesellschaftlichen Bereichen und Industriesektoren eingesetzt und sind mit großen Erwartungen in Bezug auf die Optimierung von Prozessen oder der Entscheidungsfindung verbunden. Weil die zugrundeliegenden Modelle für sehr unterschiedliche Optimierungsaufgaben eingesetzt werden können, werden sie auch als general purpose technology bezeichnet. Die zunehmende Nutzung dieser immer komplexer werdenden Systeme hat weltweit Debatten über diskriminierende Effekte, intransparente Entscheidungs- und Optimierungsprozesse oder die Reproduktion gesellschaftlicher Ungleichheiten aufgeworfen. Zunehmend werden auch die Energieverbräuche und Treibhausgasemissionen in der KI-Modellentwicklung und -anwendung diskutiert sowie weitreichendere Folgen auf Arbeitsmärkte, Konsummuster oder die Marktmacht großer Unternehmen. Dementsprechend sind Systeme künstlicher Intelligenz mit vielfältigen gesellschaftlichen, ökologischen und ökonomischen Herausforderungen verbunden. Dieses Diskussionspapier entwickelt eine übergreifende Nachhaltigkeitsperspektive auf künstliche Intelligenz und basiert auf den Arbeiten im Forschungsprojekt „SustAIn – Nachhaltigkeitsindex für Künstliche Intelligenz“. Das Ziel ist, aktuelle Diskussionen zu verantwortungsvoller KI aufzugreifen und zu einer übergreifenden Perspektive auf nachhaltige KI zu erweitern. Die Gestaltung dieser sozio-technischen Systeme ist dabei genauso relevant wie die Auswirkungen ihrer Anwendung. Betrachtet werden die sozialen, ökologischen und ökonomischen Auswirkungen entlang des gesamten KI-Lebenszyklus sowie die organisationale Einbettung dieser Systeme. Dafür werden zunächst die begrifflichen Grundlagen erläutert und eine Nachhaltigkeitsperspektive auf KI entwickelt. Basierend auf einer Analyse bestehender wissenschaftlicher und gesellschaftlicher Diskurse über die sozialen, ökologischen und ökonomischen Auswirkungen von KI werden dreizehn Nachhaltigkeitskriterien mit entsprechenden Nachhaltigkeitsindikatoren vorgeschlagen sowie Querschnittsindikatoren, die sich neben weiteren Indikatoren auf die organisationale Einbettung dieser Systeme beziehen. Abschließend wird skizziert, wie dieses Kriterien- und Indikatorenset im weiteren Forschungsprozess in Bewertungsinstrumente übersetzt wird. Ziel der hier vorgestellten Forschung ist es, den gesellschaftlichen Diskurs zur Nachhaltigkeit von KI-Systemen zu stärken sowie eine systematische Nachhaltigkeitsbewertung von KI-basierten Systemen zu ermöglichen, um deren Entwicklung und Nutzung im Sinne der Nachhaltigkeit zu stärken. %@ 978-3-940920-24-9