M. Sc. Marek Herde

Collaborative Interactive Learning (CIL)

Telefon: +49 561 804-6311
Fax: +49 561 804-6022
E-Mail: marek.herde[at]uni-kassel[dot]de

Standort: Wilhelmshöher Allee 67
34121 Kassel

Publikationen

[ 2022 ]
[ 2021 ]
[ 2019 ]
[ 2018 ]

2022[ to top ]

A Concept for Automated Polarized Web Content Annotation based on Multimodal Active Learning. Herde, Marek; Huseljic, Denis; Mitrovic, Jelena; Granitzer, Michael; Sick, Bernhard. In Workshop on Interactive Adaptive Learning (IAL), ECML PKDD, bll 1–6. 2022.
- [ Abstract ]
- [ BibTeX ]
- [ EndNote ]
- [ URL ]
Active learning (AL) techniques hardly cope with complex annotations tasks, where, for example, annotations might express relationships across data modalities. As a use case, we consider the task of automatically detecting and reporting multimodal, polarized web content (PWC). Samples of this content type emerge dynamically, covering a broad spectrum of topics. Thus, training machine learning systems for detecting PWC is challenging, particularly if it needs to be done with minimum annotation cost. In this article, we propose the concept of multimodal AL for complex annotations in the context of PWC detection and formulate the resulting challenges as questions for future research.
@inproceedings{herde2022concept, abstract = {Active learning (AL) techniques hardly cope with complex annotations tasks, where, for example, annotations might express relationships across data modalities. As a use case, we consider the task of automatically detecting and reporting multimodal, polarized web content (PWC). Samples of this content type emerge dynamically, covering a broad spectrum of topics. Thus, training machine learning systems for detecting PWC is challenging, particularly if it needs to be done with minimum annotation cost. In this article, we propose the concept of multimodal AL for complex annotations in the context of PWC detection and formulate the resulting challenges as questions for future research.}, author = {Herde, Marek and Huseljic, Denis and Mitrovic, Jelena and Granitzer, Michael and Sick, Bernhard}, booktitle = {Workshop on Interactive Adaptive Learning (IAL), ECML PKDD}, keywords = {itegpub}, pages = {1–6}, title = {A Concept for Automated Polarized Web Content Annotation based on Multimodal Active Learning}, year = 2022 }
%0 Conference Paper %1 herde2022concept %A Herde, Marek %A Huseljic, Denis %A Mitrovic, Jelena %A Granitzer, Michael %A Sick, Bernhard %B Workshop on Interactive Adaptive Learning (IAL), ECML PKDD %D 2022 %P 1--6 %T A Concept for Automated Polarized Web Content Annotation based on Multimodal Active Learning %U http://ceur-ws.org/Vol-3259/ialatecml_paper1.pdf %X Active learning (AL) techniques hardly cope with complex annotations tasks, where, for example, annotations might express relationships across data modalities. As a use case, we consider the task of automatically detecting and reporting multimodal, polarized web content (PWC). Samples of this content type emerge dynamically, covering a broad spectrum of topics. Thus, training machine learning systems for detecting PWC is challenging, particularly if it needs to be done with minimum annotation cost. In this article, we propose the concept of multimodal AL for complex annotations in the context of PWC detection and formulate the resulting challenges as questions for future research.

2021[ to top ]

Toward optimal probabilistic active learning using a Bayesian approach. Kottke, Daniel; Herde, Marek; Sandrock, Christoph; Huseljic, Denis; Krempl, Georg; Sick, Bernhard. In Machine Learning, 110(6), bll 1199–1231. Springer, 2021.
- [ Abstract ]
- [ BibTeX ]
- [ EndNote ]
Gathering labeled data to train well-performing machine learning models is one of the critical challenges in many applications. Active learning aims at reducing the labeling costs by an efficient and effective allocation of costly labeling resources. In this article, we propose a decision-theoretic selection strategy that (1) directly optimizes the gain in misclassification error, and (2) uses a Bayesian approach by introducing a conjugate prior distribution to determine the class posterior to deal with uncertainties. By reformulating existing selection strategies within our proposed model, we can explain which aspects are not covered in current state-of-the-art and why this leads to the superior performance of our approach. Extensive experiments on a large variety of datasets and different kernels validate our claims.
@article{kottke2021optimal, abstract = {Gathering labeled data to train well-performing machine learning models is one of the critical challenges in many applications. Active learning aims at reducing the labeling costs by an efficient and effective allocation of costly labeling resources. In this article, we propose a decision-theoretic selection strategy that (1) directly optimizes the gain in misclassification error, and (2) uses a Bayesian approach by introducing a conjugate prior distribution to determine the class posterior to deal with uncertainties. By reformulating existing selection strategies within our proposed model, we can explain which aspects are not covered in current state-of-the-art and why this leads to the superior performance of our approach. Extensive experiments on a large variety of datasets and different kernels validate our claims.}, author = {Kottke, Daniel and Herde, Marek and Sandrock, Christoph and Huseljic, Denis and Krempl, Georg and Sick, Bernhard}, journal = {Machine Learning}, keywords = {itegpub}, number = 6, pages = {1199–1231}, publisher = {Springer}, title = {Toward optimal probabilistic active learning using a Bayesian approach}, volume = 110, year = 2021 }
%0 Journal Article %1 kottke2021optimal %A Kottke, Daniel %A Herde, Marek %A Sandrock, Christoph %A Huseljic, Denis %A Krempl, Georg %A Sick, Bernhard %D 2021 %I Springer %J Machine Learning %N 6 %P 1199--1231 %R 10.1007/s10994-021-05986-9 %T Toward optimal probabilistic active learning using a Bayesian approach %V 110 %X Gathering labeled data to train well-performing machine learning models is one of the critical challenges in many applications. Active learning aims at reducing the labeling costs by an efficient and effective allocation of costly labeling resources. In this article, we propose a decision-theoretic selection strategy that (1) directly optimizes the gain in misclassification error, and (2) uses a Bayesian approach by introducing a conjugate prior distribution to determine the class posterior to deal with uncertainties. By reformulating existing selection strategies within our proposed model, we can explain which aspects are not covered in current state-of-the-art and why this leads to the superior performance of our approach. Extensive experiments on a large variety of datasets and different kernels validate our claims.
Separation of Aleatoric and Epistemic Uncertainty in Deterministic Deep Neural Networks. Huseljic, Denis; Sick, Bernhard; Herde, Marek; Kottke, Daniel. In International Conference on Pattern Recognition (ICPR), bll 9172–9179. IEEE, 2021.
- [ BibTeX ]
- [ EndNote ]
@inproceedings{huseljic2021separation, author = {Huseljic, Denis and Sick, Bernhard and Herde, Marek and Kottke, Daniel}, booktitle = {International Conference on Pattern Recognition (ICPR)}, keywords = {itegpub}, pages = {9172–9179}, publisher = {IEEE}, title = {Separation of Aleatoric and Epistemic Uncertainty in Deterministic Deep Neural Networks}, year = 2021 }
%0 Conference Paper %1 huseljic2021separation %A Huseljic, Denis %A Sick, Bernhard %A Herde, Marek %A Kottke, Daniel %B International Conference on Pattern Recognition (ICPR) %D 2021 %I IEEE %P 9172--9179 %R 10.1109/icpr48806.2021.9412616 %T Separation of Aleatoric and Epistemic Uncertainty in Deterministic Deep Neural Networks
scikit-activeml: A Library and Toolbox for Active Learning Algorithms. Kottke, Daniel; Herde, Marek; Minh, Tuan Pham; Benz, Alexander; Mergard, Pascal; Roghman, Atal; Sandrock, Christoph; Sick, Bernhard. In Preprints, bl 2021030194. 2021.
- [ Abstract ]
- [ BibTeX ]
- [ EndNote ]
- [ URL ]
Machine learning applications often need large amounts of training data to perform well. Whereas unlabeled data can be easily gathered, the labeling process is difficult, time-consuming, or expensive in most applications. Active learning can help solve this problem by querying labels for those data points that will improve the performance the most. Thereby, the goal is that the learning algorithm performs sufficiently well with fewer labels. We provide a library called scikit-activeml that covers the most relevant query strategies and implements tools to work with partially labeled data. It is programmed in Python and builds on top of scikit-learn.
@article{kottke2021scikit, abstract = {Machine learning applications often need large amounts of training data to perform well. Whereas unlabeled data can be easily gathered, the labeling process is difficult, time-consuming, or expensive in most applications. Active learning can help solve this problem by querying labels for those data points that will improve the performance the most. Thereby, the goal is that the learning algorithm performs sufficiently well with fewer labels. We provide a library called scikit-activeml that covers the most relevant query strategies and implements tools to work with partially labeled data. It is programmed in Python and builds on top of scikit-learn.}, author = {Kottke, Daniel and Herde, Marek and Minh, Tuan Pham and Benz, Alexander and Mergard, Pascal and Roghman, Atal and Sandrock, Christoph and Sick, Bernhard}, journal = {Preprints}, keywords = {itegpub}, pages = 2021030194, title = {{scikit-activeml}: {A} Library and Toolbox for Active Learning Algorithms}, year = 2021 }
%0 Journal Article %1 kottke2021scikit %A Kottke, Daniel %A Herde, Marek %A Minh, Tuan Pham %A Benz, Alexander %A Mergard, Pascal %A Roghman, Atal %A Sandrock, Christoph %A Sick, Bernhard %D 2021 %J Preprints %P 2021030194 %T scikit-activeml: A Library and Toolbox for Active Learning Algorithms %U https://www.preprints.org/manuscript/202103.0194/ %X Machine learning applications often need large amounts of training data to perform well. Whereas unlabeled data can be easily gathered, the labeling process is difficult, time-consuming, or expensive in most applications. Active learning can help solve this problem by querying labels for those data points that will improve the performance the most. Thereby, the goal is that the learning algorithm performs sufficiently well with fewer labels. We provide a library called scikit-activeml that covers the most relevant query strategies and implements tools to work with partially labeled data. It is programmed in Python and builds on top of scikit-learn.
Multi-annotator Probabilistic Active Learning. Herde, Marek; Kottke, Daniel; Huseljic, Denis; Sick, Bernhard. In International Conference on Pattern Recognition (ICPR), bll 10281–10288. IEEE, 2021.
- [ BibTeX ]
- [ EndNote ]
@inproceedings{herde2021multi, author = {Herde, Marek and Kottke, Daniel and Huseljic, Denis and Sick, Bernhard}, booktitle = {International Conference on Pattern Recognition (ICPR)}, keywords = {itegpub}, pages = {10281–10288}, publisher = {IEEE}, title = {Multi-annotator Probabilistic Active Learning}, year = 2021 }
%0 Conference Paper %1 herde2021multi %A Herde, Marek %A Kottke, Daniel %A Huseljic, Denis %A Sick, Bernhard %B International Conference on Pattern Recognition (ICPR) %D 2021 %I IEEE %P 10281--10288 %R 10.1109/icpr48806.2021.9412298 %T Multi-annotator Probabilistic Active Learning
A Survey on Cost Types, Interaction Schemes, and Annotator Performance Models in Selection Algorithms for Active Learning in Classification. Herde, Marek; Huseljic, Denis; Sick, Bernhard; Calma, Adrian. In IEEE Access, 9, bll 166970–166989. IEEE, 2021.
- [ Abstract ]
- [ BibTeX ]
- [ EndNote ]
- [ URL ]
Pool-based active learning (AL) aims to optimize the annotation process (i.e., labeling) as the acquisition of annotations is often time-consuming and therefore expensive. For this purpose, an AL strategy queries annotations intelligently from annotators to train a high-performance classification model at a low annotation cost. Traditional AL strategies operate in an idealized framework. They assume a single, omniscient annotator who never gets tired and charges uniformly regardless of query difficulty. However, in real-world applications, we often face human annotators, e.g., crowd or in-house workers, who make annotation mistakes and can be reluctant to respond if tired or faced with complex queries. Recently, many novel AL strategies have been proposed to address these issues. They differ in at least one of the following three central aspects from traditional AL: 1) modeling of (multiple) human annotators whose performances can be affected by various factors, such as missing expertise; 2) generalization of the interaction with human annotators through different query and annotation types, such as asking an annotator for feedback on an inferred classification rule; 3) consideration of complex cost schemes regarding annotations and misclassification. This survey provides an overview of these AL strategies and refers to them as real-world AL. Therefore, we introduce a general real-world AL strategy as part of a learning cycle and use its elements, e.g., the query and annotator selection algorithm, to categorize about 60 real-world AL strategies. Finally, we outline possible directions for future research in the field of AL.
@article{herde2021survey, abstract = {Pool-based active learning (AL) aims to optimize the annotation process (i.e., labeling) as the acquisition of annotations is often time-consuming and therefore expensive. For this purpose, an AL strategy queries annotations intelligently from annotators to train a high-performance classification model at a low annotation cost. Traditional AL strategies operate in an idealized framework. They assume a single, omniscient annotator who never gets tired and charges uniformly regardless of query difficulty. However, in real-world applications, we often face human annotators, e.g., crowd or in-house workers, who make annotation mistakes and can be reluctant to respond if tired or faced with complex queries. Recently, many novel AL strategies have been proposed to address these issues. They differ in at least one of the following three central aspects from traditional AL: 1) modeling of (multiple) human annotators whose performances can be affected by various factors, such as missing expertise; 2) generalization of the interaction with human annotators through different query and annotation types, such as asking an annotator for feedback on an inferred classification rule; 3) consideration of complex cost schemes regarding annotations and misclassification. This survey provides an overview of these AL strategies and refers to them as real-world AL. Therefore, we introduce a general real-world AL strategy as part of a learning cycle and use its elements, e.g., the query and annotator selection algorithm, to categorize about 60 real-world AL strategies. Finally, we outline possible directions for future research in the field of AL.}, author = {Herde, Marek and Huseljic, Denis and Sick, Bernhard and Calma, Adrian}, journal = {IEEE Access}, keywords = {classification}, pages = {166970–166989}, publisher = {IEEE}, title = {A Survey on Cost Types, Interaction Schemes, and Annotator Performance Models in Selection Algorithms for Active Learning in Classification}, volume = 9, year = 2021 }
%0 Journal Article %1 herde2021survey %A Herde, Marek %A Huseljic, Denis %A Sick, Bernhard %A Calma, Adrian %D 2021 %I IEEE %J IEEE Access %P 166970--166989 %R 10.1109/ACCESS.2021.3135514 %T A Survey on Cost Types, Interaction Schemes, and Annotator Performance Models in Selection Algorithms for Active Learning in Classification %U https://ieeexplore.ieee.org/abstract/document/9650877 %V 9 %X Pool-based active learning (AL) aims to optimize the annotation process (i.e., labeling) as the acquisition of annotations is often time-consuming and therefore expensive. For this purpose, an AL strategy queries annotations intelligently from annotators to train a high-performance classification model at a low annotation cost. Traditional AL strategies operate in an idealized framework. They assume a single, omniscient annotator who never gets tired and charges uniformly regardless of query difficulty. However, in real-world applications, we often face human annotators, e.g., crowd or in-house workers, who make annotation mistakes and can be reluctant to respond if tired or faced with complex queries. Recently, many novel AL strategies have been proposed to address these issues. They differ in at least one of the following three central aspects from traditional AL: 1) modeling of (multiple) human annotators whose performances can be affected by various factors, such as missing expertise; 2) generalization of the interaction with human annotators through different query and annotation types, such as asking an annotator for feedback on an inferred classification rule; 3) consideration of complex cost schemes regarding annotations and misclassification. This survey provides an overview of these AL strategies and refers to them as real-world AL. Therefore, we introduce a general real-world AL strategy as part of a learning cycle and use its elements, e.g., the query and annotator selection algorithm, to categorize about 60 real-world AL strategies. Finally, we outline possible directions for future research in the field of AL.
A Concept for Highly Automated Pre-Labeling via Cross-Domain Label Transfer for Perception in Autonomous Driving. Bieshaar, Maarten; Herde, Marek; Huselijc, Denis; Sick, Bernhard. In Workshop on Interactive Adaptive Learning (IAL), ECML PKDD. 2021.
- [ BibTeX ]
- [ EndNote ]
@inproceedings{bieshaar2021concept, author = {Bieshaar, Maarten and Herde, Marek and Huselijc, Denis and Sick, Bernhard}, booktitle = {Workshop on Interactive Adaptive Learning (IAL), ECML PKDD}, keywords = {itegpub}, note = {(accepted)}, title = {A Concept for Highly Automated Pre-Labeling via Cross-Domain Label Transfer for Perception in Autonomous Driving}, year = 2021 }
%0 Conference Paper %1 bieshaar2021concept %A Bieshaar, Maarten %A Herde, Marek %A Huselijc, Denis %A Sick, Bernhard %B Workshop on Interactive Adaptive Learning (IAL), ECML PKDD %D 2021 %T A Concept for Highly Automated Pre-Labeling via Cross-Domain Label Transfer for Perception in Autonomous Driving

2019[ to top ]

Combining Self-reported Confidences from Uncertain Annotators to Improve Label Quality. Sandrock, C.; Herde, M.; Calma, A.; Kottke, D.; Sick, B. In International Joint Conference on Neural Networks (IJCNN), bll 1–8. IEEE, 2019.
- [ BibTeX ]
- [ EndNote ]
@inproceedings{sandrock2019combining, author = {{Sandrock}, C. and {Herde}, M. and {Calma}, A. and {Kottke}, D. and {Sick}, B.}, booktitle = {International Joint Conference on Neural Networks (IJCNN)}, keywords = {annotators;confidence}, pages = {1–8}, publisher = {IEEE}, title = {Combining Self-reported Confidences from Uncertain Annotators to Improve Label Quality}, year = 2019 }
%0 Conference Paper %1 sandrock2019combining %A Sandrock, C. %A Herde, M. %A Calma, A. %A Kottke, D. %A Sick, B. %B International Joint Conference on Neural Networks (IJCNN) %D 2019 %I IEEE %P 1--8 %R 10.1109/IJCNN.2019.8852456 %T Combining Self-reported Confidences from Uncertain Annotators to Improve Label Quality

2018[ to top ]

Active Sorting -- An Efficient Training of a Sorting Robot with Active Learning Techniques. Herde, Marek; Kottke, Daniel; Calma, Adrian; Bieshaar, Maarten; Deist, Stephan; Sick, Bernhard. In International Joint Conference on Neural Networks (IJCNN). IEEE, Rio de Janiero, Brazil, 2018.
- [ BibTeX ]
- [ EndNote ]
@inproceedings{herde2018active, address = {Rio de Janiero, Brazil}, author = {Herde, Marek and Kottke, Daniel and Calma, Adrian and Bieshaar, Maarten and Deist, Stephan and Sick, Bernhard}, booktitle = {International Joint Conference on Neural Networks (IJCNN)}, keywords = {imported}, publisher = {IEEE}, title = {Active Sorting – An Efficient Training of a Sorting Robot with Active Learning Techniques}, year = 2018 }
%0 Conference Paper %1 herde2018active %A Herde, Marek %A Kottke, Daniel %A Calma, Adrian %A Bieshaar, Maarten %A Deist, Stephan %A Sick, Bernhard %B International Joint Conference on Neural Networks (IJCNN) %C Rio de Janiero, Brazil %D 2018 %I IEEE %R 10.1109/ijcnn.2018.8489161 %T Active Sorting -- An Efficient Training of a Sorting Robot with Active Learning Techniques