M. Sc. Marek Herde

Collaborative Interactive Learning (CIL)

Telephone: +49 561 804-6311
Email: marek.herde[at]uni-kassel[dot]de

Location: Wilhelmshöher Allee 67
34121 Kassel

Publications

[ 2025 ]
[ 2024 ]
[ 2023 ]
[ 2022 ]
[ 2021 ]
[ 2020 ]
[ 2019 ]
[ 2018 ]

2025[ to top ]

crowd-hpo: Realistic Hyperparameter Optimization and Benchmarking for Learning from Crowds with Noisy Labels.Herde, Marek; Lührs, Lukas; Huseljic, Denis; Sick, Bernhard. In Transactions on Machine Learning Research. 2025.
- [ Abstract ]
- [ BibTeX ]
- [ EndNote ]
- [ URL ]
Crowdworking is a cost-efficient solution for acquiring class labels. Since these labels are subject to noise, various approaches to learning from crowds have been proposed. Typically, these approaches are evaluated using default hyperparameter configurations, which often result in unfair and suboptimal performance, or using hyperparameter configurations tuned via a validation set with ground truth class labels, which represents an often unrealistic scenario. Moreover, both setups can yield different approach rankings, complicating study comparisons. Therefore, we introduce crowd-hpo as a framework for evaluating approaches to learning from crowds, together with criteria for selecting well-performing hyperparameter configurations using only noisy crowd-labeled validation data. Extensive experiments with neural networks demonstrate that these criteria select hyperparameter configurations that improve the learning from crowds approaches' generalization performances, measured on separate test sets with ground truth labels. Hence, incorporating such criteria into experimental studies is essential for enabling fairer and more realistic benchmarking.
@article{herde2025crowd, abstract = {Crowdworking is a cost-efficient solution for acquiring class labels. Since these labels are subject to noise, various approaches to learning from crowds have been proposed. Typically, these approaches are evaluated using default hyperparameter configurations, which often result in unfair and suboptimal performance, or using hyperparameter configurations tuned via a validation set with ground truth class labels, which represents an often unrealistic scenario. Moreover, both setups can yield different approach rankings, complicating study comparisons. Therefore, we introduce crowd-hpo as a framework for evaluating approaches to learning from crowds, together with criteria for selecting well-performing hyperparameter configurations using only noisy crowd-labeled validation data. Extensive experiments with neural networks demonstrate that these criteria select hyperparameter configurations that improve the learning from crowds approaches' generalization performances, measured on separate test sets with ground truth labels. Hence, incorporating such criteria into experimental studies is essential for enabling fairer and more realistic benchmarking.}, author = {Herde, Marek and Lührs, Lukas and Huseljic, Denis and Sick, Bernhard}, journal = {Transactions on Machine Learning Research}, keywords = {itegpub}, title = {crowd-hpo: Realistic Hyperparameter Optimization and Benchmarking for Learning from Crowds with Noisy Labels}, year = 2025 }
%0 Journal Article %1 herde2025crowd %A Herde, Marek %A Lührs, Lukas %A Huseljic, Denis %A Sick, Bernhard %D 2025 %J Transactions on Machine Learning Research %T crowd-hpo: Realistic Hyperparameter Optimization and Benchmarking for Learning from Crowds with Noisy Labels %U https://openreview.net/forum?id=SaKfhylVLK %X Crowdworking is a cost-efficient solution for acquiring class labels. Since these labels are subject to noise, various approaches to learning from crowds have been proposed. Typically, these approaches are evaluated using default hyperparameter configurations, which often result in unfair and suboptimal performance, or using hyperparameter configurations tuned via a validation set with ground truth class labels, which represents an often unrealistic scenario. Moreover, both setups can yield different approach rankings, complicating study comparisons. Therefore, we introduce crowd-hpo as a framework for evaluating approaches to learning from crowds, together with criteria for selecting well-performing hyperparameter configurations using only noisy crowd-labeled validation data. Extensive experiments with neural networks demonstrate that these criteria select hyperparameter configurations that improve the learning from crowds approaches' generalization performances, measured on separate test sets with ground truth labels. Hence, incorporating such criteria into experimental studies is essential for enabling fairer and more realistic benchmarking.
Efficient Bayesian Updates for Deep Active Learning via Laplace Approximations.Huseljic, Denis; Herde, Marek; Rauch, Lukas; Hahn, Paul; Huang, Zhixin; Kottke, Daniel; Vogt, Stephan; Sick, Bernhard. In European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECMLPKDD), bll 20–35. Springer, 2025.
- [ Abstract ]
- [ BibTeX ]
- [ EndNote ]
- [ URL ]
Deep active learning (AL) selects batches of instances for annotation to avoid retraining deep neural networks (DNNs) after each new label. Employing a naive top-b selection can result in a batch of redundant (similar) instances. To address this, various AL strategies employ clustering techniques that ensure diversity within a batch. We approach this issue by substituting the costly retraining with an efficient Bayesian update. Our proposed update represents a second-order optimization step using the Gaussian posterior from a last-layer Laplace approximation. Thereby, we achieve low computational complexity by computing the inverse Hessian in closed form. We demonstrate that in typical AL settings, our update closely approximates retraining while being considerably faster. Leveraging our update, we introduce a new framework for batch selection through sequential construction, updating the DNN after each label acquisition. Furthermore, we incorporate our update into a look-ahead selection strategy as a feasible upper baseline approximating optimal batch selection. Our results highlight the potential of efficient updates to advance deep AL research.
@inproceedings{huseljic2025efficient, abstract = {Deep active learning (AL) selects batches of instances for annotation to avoid retraining deep neural networks (DNNs) after each new label. Employing a naive top-b selection can result in a batch of redundant (similar) instances. To address this, various AL strategies employ clustering techniques that ensure diversity within a batch. We approach this issue by substituting the costly retraining with an efficient Bayesian update. Our proposed update represents a second-order optimization step using the Gaussian posterior from a last-layer Laplace approximation. Thereby, we achieve low computational complexity by computing the inverse Hessian in closed form. We demonstrate that in typical AL settings, our update closely approximates retraining while being considerably faster. Leveraging our update, we introduce a new framework for batch selection through sequential construction, updating the DNN after each label acquisition. Furthermore, we incorporate our update into a look-ahead selection strategy as a feasible upper baseline approximating optimal batch selection. Our results highlight the potential of efficient updates to advance deep AL research.}, author = {Huseljic, Denis and Herde, Marek and Rauch, Lukas and Hahn, Paul and Huang, Zhixin and Kottke, Daniel and Vogt, Stephan and Sick, Bernhard}, booktitle = {European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECMLPKDD)}, keywords = {itegpub}, pages = {20–35}, publisher = {Springer}, title = {Efficient Bayesian Updates for Deep Active Learning via Laplace Approximations}, year = 2025 }
%0 Conference Paper %1 huseljic2025efficient %A Huseljic, Denis %A Herde, Marek %A Rauch, Lukas %A Hahn, Paul %A Huang, Zhixin %A Kottke, Daniel %A Vogt, Stephan %A Sick, Bernhard %B European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECMLPKDD) %D 2025 %I Springer %P 20--35 %R 10.1007/978-3-032-05981-9_2 %T Efficient Bayesian Updates for Deep Active Learning via Laplace Approximations %U https://link.springer.com/chapter/10.1007/978-3-032-05981-9_2 %X Deep active learning (AL) selects batches of instances for annotation to avoid retraining deep neural networks (DNNs) after each new label. Employing a naive top-b selection can result in a batch of redundant (similar) instances. To address this, various AL strategies employ clustering techniques that ensure diversity within a batch. We approach this issue by substituting the costly retraining with an efficient Bayesian update. Our proposed update represents a second-order optimization step using the Gaussian posterior from a last-layer Laplace approximation. Thereby, we achieve low computational complexity by computing the inverse Hessian in closed form. We demonstrate that in typical AL settings, our update closely approximates retraining while being considerably faster. Leveraging our update, we introduce a new framework for batch selection through sequential construction, updating the DNN after each label acquisition. Furthermore, we incorporate our update into a look-ahead selection strategy as a feasible upper baseline approximating optimal batch selection. Our results highlight the potential of efficient updates to advance deep AL research.

2024[ to top ]

Workshop on Interactive Adaptive Learning (IAL), ECML PKDD. Bunse, Mirko; Herde, Marek; Krempl, Georg; Lemaire, Vincent; Tharwat, Alaa; Pham, Minh Tuan; Saadallah, Amal. In Vol. 3770CEUR Workshop Proceedings. 2024.
- [ BibTeX ]
- [ EndNote ]
- [ URL ]
@proceedings{bunse2024workshop, editor = {Bunse, Mirko and Herde, Marek and Krempl, Georg and Lemaire, Vincent and Tharwat, Alaa and Pham, Minh Tuan and Saadallah, Amal}, keywords = {itegpub}, series = {{CEUR} Workshop Proceedings}, title = {Workshop on Interactive Adaptive Learning (IAL), ECML PKDD}, volume = 3770, year = 2024 }
%0 Conference Proceedings %1 bunse2024workshop %B CEUR Workshop Proceedings %D 2024 %E Bunse, Mirko %E Herde, Marek %E Krempl, Georg %E Lemaire, Vincent %E Tharwat, Alaa %E Pham, Minh Tuan %E Saadallah, Amal %T Workshop on Interactive Adaptive Learning (IAL), ECML PKDD %U http://ceur-ws.org/Vol-3770 %V 3770
Tutorial: Interactive Adaptive Learning.Herde, Marek; Pham, Minh Tuan; Tharwat, Alaa; Sick, Bernhard. In Workshop on Interactive Adapative Learning (IAL), ECML PKDD, bll 1–6. 2024.
- [ Abstract ]
- [ BibTeX ]
- [ EndNote ]
- [ URL ]
We summarize the contents of the tutorial we present as a part of the 8th Interactive Adaptive Learning workshop. This workshop is co-located with the ECML-PKDD conference, where it takes place on September 9th, 2024 in Vilnius, Lithuania.
@inproceedings{herde2024tutorial, abstract = {We summarize the contents of the tutorial we present as a part of the 8th Interactive Adaptive Learning workshop. This workshop is co-located with the ECML-PKDD conference, where it takes place on September 9th, 2024 in Vilnius, Lithuania.}, author = {Herde, Marek and Pham, Minh Tuan and Tharwat, Alaa and Sick, Bernhard}, booktitle = {Workshop on Interactive Adapative Learning (IAL), ECML PKDD}, keywords = {itegpub}, pages = {1–6}, title = {Tutorial: Interactive Adaptive Learning}, year = 2024 }
%0 Conference Paper %1 herde2024tutorial %A Herde, Marek %A Pham, Minh Tuan %A Tharwat, Alaa %A Sick, Bernhard %B Workshop on Interactive Adapative Learning (IAL), ECML PKDD %D 2024 %P 1--6 %T Tutorial: Interactive Adaptive Learning %U https://ceur-ws.org/Vol-3770/paper1.pdf %X We summarize the contents of the tutorial we present as a part of the 8th Interactive Adaptive Learning workshop. This workshop is co-located with the ECML-PKDD conference, where it takes place on September 9th, 2024 in Vilnius, Lithuania.
The Interplay of Uncertainty Modeling and Deep Active Learning: An Empirical Analysis in Image Classification.Huseljic, Denis; Herde, Marek; Nagel, Yannick; Rauch, Lukas; Strimaitis, Paulius; Sick, Bernhard. In Transactions on Machine Learning Research. 2024.
- [ Abstract ]
- [ BibTeX ]
- [ EndNote ]
- [ URL ]
Deep active learning (AL) seeks to reduce the annotation costs required for training deep neural networks (DNNs). Often, deep AL strategies focus on instances where the predictive uncertainty of a DNN is high. Furthermore, Bayesian concepts to model uncertainty are frequently adopted. Despite considerable research, a detailed analysis of the role of uncertainty in deep AL is still missing, especially regarding aleatoric and epistemic uncertainty, both related to predictive uncertainty. This article provides an in-depth empirical study analyzing the interplay of uncertainty and deep AL in image classification. Our study investigates four hypotheses that provide an intuitive understanding of the effects of accurately estimating aleatoric and epistemic uncertainty on existing uncertainty-based AL strategies but also, in the opposite direction, the impact of uncertainty-based AL on the quality of uncertainty estimates that are needed in many applications. By analyzing these hypotheses on synthetic and real-world data, we find that accurate aleatoric estimates can even impair instance selection, while accurate epistemic estimates have negligible effects. Moreover, we provide a publicly available toolbox for deep AL with various models and strategies to facilitate further research and practical applications. Code is available at https://github.com/dhuseljic/dal-toolbox.
@article{huseljic2024interplay, abstract = {Deep active learning (AL) seeks to reduce the annotation costs required for training deep neural networks (DNNs). Often, deep AL strategies focus on instances where the predictive uncertainty of a DNN is high. Furthermore, Bayesian concepts to model uncertainty are frequently adopted. Despite considerable research, a detailed analysis of the role of uncertainty in deep AL is still missing, especially regarding aleatoric and epistemic uncertainty, both related to predictive uncertainty. This article provides an in-depth empirical study analyzing the interplay of uncertainty and deep AL in image classification. Our study investigates four hypotheses that provide an intuitive understanding of the effects of accurately estimating aleatoric and epistemic uncertainty on existing uncertainty-based AL strategies but also, in the opposite direction, the impact of uncertainty-based AL on the quality of uncertainty estimates that are needed in many applications. By analyzing these hypotheses on synthetic and real-world data, we find that accurate aleatoric estimates can even impair instance selection, while accurate epistemic estimates have negligible effects. Moreover, we provide a publicly available toolbox for deep AL with various models and strategies to facilitate further research and practical applications. Code is available at https://github.com/dhuseljic/dal-toolbox.}, author = {Huseljic, Denis and Herde, Marek and Nagel, Yannick and Rauch, Lukas and Strimaitis, Paulius and Sick, Bernhard}, journal = {Transactions on Machine Learning Research}, keywords = {itegpub}, title = {The Interplay of Uncertainty Modeling and Deep Active Learning: An Empirical Analysis in Image Classification}, year = 2024 }
%0 Journal Article %1 huseljic2024interplay %A Huseljic, Denis %A Herde, Marek %A Nagel, Yannick %A Rauch, Lukas %A Strimaitis, Paulius %A Sick, Bernhard %D 2024 %J Transactions on Machine Learning Research %T The Interplay of Uncertainty Modeling and Deep Active Learning: An Empirical Analysis in Image Classification %U https://openreview.net/forum?id=KLBD13bsVl %X Deep active learning (AL) seeks to reduce the annotation costs required for training deep neural networks (DNNs). Often, deep AL strategies focus on instances where the predictive uncertainty of a DNN is high. Furthermore, Bayesian concepts to model uncertainty are frequently adopted. Despite considerable research, a detailed analysis of the role of uncertainty in deep AL is still missing, especially regarding aleatoric and epistemic uncertainty, both related to predictive uncertainty. This article provides an in-depth empirical study analyzing the interplay of uncertainty and deep AL in image classification. Our study investigates four hypotheses that provide an intuitive understanding of the effects of accurately estimating aleatoric and epistemic uncertainty on existing uncertainty-based AL strategies but also, in the opposite direction, the impact of uncertainty-based AL on the quality of uncertainty estimates that are needed in many applications. By analyzing these hypotheses on synthetic and real-world data, we find that accurate aleatoric estimates can even impair instance selection, while accurate epistemic estimates have negligible effects. Moreover, we provide a publicly available toolbox for deep AL with various models and strategies to facilitate further research and practical applications. Code is available at https://github.com/dhuseljic/dal-toolbox.
Systematic Evaluation of Uncertainty Calibration in Pretrained Object Detectors.Huseljic, Denis; Herde, Marek; Hahn, Paul; Müjde, Mehmet; Sick, Bernhard. In International Journal of Computer Vision, 133(3), bll 1033–1047. Springer, 2024.
- [ Abstract ]
- [ BibTeX ]
- [ EndNote ]
- [ URL ]
In the field of deep learning based computer vision, the development of deep object detection has led to unique paradigms (e.g., two-stage or set-based) and architectures (e.g., Faster-RCNN or DETR) which enable outstanding performance on challenging benchmark datasets. Despite this, the trained object detectors typically do not reliably assess uncertainty regarding their own knowledge, and the quality of their probabilistic predictions is usually poor. As these are often used to make subsequent decisions, such inaccurate probabilistic predictions must be avoided. In this work, we investigate the uncertainty calibration properties of different pretrained object detection architectures in a multi-class setting. We propose a framework to ensure a fair, unbiased, and repeatable evaluation and conduct detailed analyses assessing the calibration under distributional changes (e.g., distributional shift and application to out-of-distribution data). Furthermore, by investigating the influence of different detector paradigms, post-processing steps, and suitable choices of metrics, we deliver novel insights into why poor detector calibration emerges. Based on these insights, we are able to improve the calibration of a detector by simply finetuning its last layer.
@article{huseljic2024systematic, abstract = {In the field of deep learning based computer vision, the development of deep object detection has led to unique paradigms (e.g., two-stage or set-based) and architectures (e.g., Faster-RCNN or DETR) which enable outstanding performance on challenging benchmark datasets. Despite this, the trained object detectors typically do not reliably assess uncertainty regarding their own knowledge, and the quality of their probabilistic predictions is usually poor. As these are often used to make subsequent decisions, such inaccurate probabilistic predictions must be avoided. In this work, we investigate the uncertainty calibration properties of different pretrained object detection architectures in a multi-class setting. We propose a framework to ensure a fair, unbiased, and repeatable evaluation and conduct detailed analyses assessing the calibration under distributional changes (e.g., distributional shift and application to out-of-distribution data). Furthermore, by investigating the influence of different detector paradigms, post-processing steps, and suitable choices of metrics, we deliver novel insights into why poor detector calibration emerges. Based on these insights, we are able to improve the calibration of a detector by simply finetuning its last layer.}, author = {Huseljic, Denis and Herde, Marek and Hahn, Paul and Müjde, Mehmet and Sick, Bernhard}, journal = {International Journal of Computer Vision}, keywords = {itegpub}, number = 3, pages = {1033–1047}, publisher = {Springer}, title = {Systematic Evaluation of Uncertainty Calibration in Pretrained Object Detectors}, volume = 133, year = 2024 }
%0 Journal Article %1 huseljic2024systematic %A Huseljic, Denis %A Herde, Marek %A Hahn, Paul %A Müjde, Mehmet %A Sick, Bernhard %D 2024 %I Springer %J International Journal of Computer Vision %N 3 %P 1033--1047 %R 10.1007/s11263-024-02219-z %T Systematic Evaluation of Uncertainty Calibration in Pretrained Object Detectors %U https://link.springer.com/article/10.1007/s11263-024-02219-z %V 133 %X In the field of deep learning based computer vision, the development of deep object detection has led to unique paradigms (e.g., two-stage or set-based) and architectures (e.g., Faster-RCNN or DETR) which enable outstanding performance on challenging benchmark datasets. Despite this, the trained object detectors typically do not reliably assess uncertainty regarding their own knowledge, and the quality of their probabilistic predictions is usually poor. As these are often used to make subsequent decisions, such inaccurate probabilistic predictions must be avoided. In this work, we investigate the uncertainty calibration properties of different pretrained object detection architectures in a multi-class setting. We propose a framework to ensure a fair, unbiased, and repeatable evaluation and conduct detailed analyses assessing the calibration under distributional changes (e.g., distributional shift and application to out-of-distribution data). Furthermore, by investigating the influence of different detector paradigms, post-processing steps, and suitable choices of metrics, we deliver novel insights into why poor detector calibration emerges. Based on these insights, we are able to improve the calibration of a detector by simply finetuning its last layer.
Annot-Mix: Learning with Noisy Class Labels from Multiple Annotators via a Mixup Extension.Herde, Marek; Lührs, Lukas; Huseljic, Denis; Sick, Bernhard. In arXiv e-prints, bl arXiv:2405.0338. 2024.
- [ Abstract ]
- [ BibTeX ]
- [ EndNote ]
- [ URL ]
Training with noisy class labels impairs neural networks' generalization performance. In this context, mixup is a popular regularization technique to improve training robustness by making memorizing false class labels more difficult. However, mixup neglects that, typically, multiple annotators, e.g., crowdworkers, provide class labels. Therefore, we propose an extension of mixup, which handles multiple class labels per instance while considering which class label originates from which annotator. Integrated into our multi-annotator classification framework annot-mix, it performs superiorly to eight state-of-the-art approaches on eleven datasets with noisy class labels provided either by human or simulated annotators. Our code is publicly available through our repository at this https://github.com/ies-research/annot-mix..
@article{herde2024annot, abstract = {Training with noisy class labels impairs neural networks' generalization performance. In this context, mixup is a popular regularization technique to improve training robustness by making memorizing false class labels more difficult. However, mixup neglects that, typically, multiple annotators, e.g., crowdworkers, provide class labels. Therefore, we propose an extension of mixup, which handles multiple class labels per instance while considering which class label originates from which annotator. Integrated into our multi-annotator classification framework annot-mix, it performs superiorly to eight state-of-the-art approaches on eleven datasets with noisy class labels provided either by human or simulated annotators. Our code is publicly available through our repository at this https://github.com/ies-research/annot-mix..}, author = {Herde, Marek and Lührs, Lukas and Huseljic, Denis and Sick, Bernhard}, journal = {arXiv e-prints}, keywords = {itegpub}, pages = {arXiv:2405.0338}, title = {Annot-Mix: Learning with Noisy Class Labels from Multiple Annotators via a Mixup Extension}, year = 2024 }
%0 Journal Article %1 herde2024annot %A Herde, Marek %A Lührs, Lukas %A Huseljic, Denis %A Sick, Bernhard %D 2024 %J arXiv e-prints %P arXiv:2405.0338 %T Annot-Mix: Learning with Noisy Class Labels from Multiple Annotators via a Mixup Extension %U https://arxiv.org/abs/2405.03386 %X Training with noisy class labels impairs neural networks' generalization performance. In this context, mixup is a popular regularization technique to improve training robustness by making memorizing false class labels more difficult. However, mixup neglects that, typically, multiple annotators, e.g., crowdworkers, provide class labels. Therefore, we propose an extension of mixup, which handles multiple class labels per instance while considering which class label originates from which annotator. Integrated into our multi-annotator classification framework annot-mix, it performs superiorly to eight state-of-the-art approaches on eleven datasets with noisy class labels provided either by human or simulated annotators. Our code is publicly available through our repository at this https://github.com/ies-research/annot-mix..
Annot-Mix: Learning with Noisy Class Labels from Multiple Annotators via a Mixup Extension.Herde, Marek; Lührs, Lukas; Huseljic, Denis; Sick, Bernhard. In European Conference on Artificial Intelligence (ECAI), bll 2910–2918. IOS Press, 2024.
- [ Abstract ]
- [ BibTeX ]
- [ EndNote ]
- [ URL ]
Training with noisy class labels impairs neural networks' generalization performance. In this context, mixup is a popular regularization technique to improve training robustness by making memorizing false class labels more difficult. However, mixup neglects that multiple annotators, e.g., crowdworkers, typically provide class labels. Therefore, we propose an extension of mixup, which handles multiple class labels per instance while considering which class label originates from which annotator. Integrated into our multi-annotator classification framework annot-mix, it performs superiorly to eleven (mostly state-of-the-art) approaches in an evaluation study with eleven datasets comprising noisy class labels from either human or simulated annotators. Our code is publicly available through our GitHub repository at https://github.com/ies-research/multi-annotator-machine-learning/tree/annot-mix.
@inproceedings{herde2024annota, abstract = {Training with noisy class labels impairs neural networks' generalization performance. In this context, mixup is a popular regularization technique to improve training robustness by making memorizing false class labels more difficult. However, mixup neglects that multiple annotators, e.g., crowdworkers, typically provide class labels. Therefore, we propose an extension of mixup, which handles multiple class labels per instance while considering which class label originates from which annotator. Integrated into our multi-annotator classification framework annot-mix, it performs superiorly to eleven (mostly state-of-the-art) approaches in an evaluation study with eleven datasets comprising noisy class labels from either human or simulated annotators. Our code is publicly available through our GitHub repository at https://github.com/ies-research/multi-annotator-machine-learning/tree/annot-mix.}, author = {Herde, Marek and Lührs, Lukas and Huseljic, Denis and Sick, Bernhard}, booktitle = {European Conference on Artificial Intelligence (ECAI)}, keywords = {itegpub}, pages = {2910–2918}, publisher = {IOS Press}, title = {Annot-Mix: Learning with Noisy Class Labels from Multiple Annotators via a Mixup Extension}, year = 2024 }
%0 Conference Paper %1 herde2024annota %A Herde, Marek %A Lührs, Lukas %A Huseljic, Denis %A Sick, Bernhard %B European Conference on Artificial Intelligence (ECAI) %D 2024 %I IOS Press %P 2910--2918 %R 10.3233/FAIA240829 %T Annot-Mix: Learning with Noisy Class Labels from Multiple Annotators via a Mixup Extension %U https://ebooks.iospress.nl/doi/10.3233/FAIA240829 %X Training with noisy class labels impairs neural networks' generalization performance. In this context, mixup is a popular regularization technique to improve training robustness by making memorizing false class labels more difficult. However, mixup neglects that multiple annotators, e.g., crowdworkers, typically provide class labels. Therefore, we propose an extension of mixup, which handles multiple class labels per instance while considering which class label originates from which annotator. Integrated into our multi-annotator classification framework annot-mix, it performs superiorly to eleven (mostly state-of-the-art) approaches in an evaluation study with eleven datasets comprising noisy class labels from either human or simulated annotators. Our code is publicly available through our GitHub repository at https://github.com/ies-research/multi-annotator-machine-learning/tree/annot-mix.
dopanim: A Dataset of Doppelganger Animals with Noisy Annotations from Multiple Humans.Herde, Marek; Huseljic, Denis; Rauch, Lukas; Sick, Bernhard. In Conference on Neural Information Processing Systems (NeurIPS 2024) Track on Datasets and Benchmarks, bll 1–16. 2024.
- [ Abstract ]
- [ BibTeX ]
- [ EndNote ]
- [ URL ]
Human annotators typically provide annotated data for training machine learning models, such as neural networks. Yet, human annotations are subject to noise, impairing generalization performances. Methodological research on approaches counteracting noisy annotations requires corresponding datasets for a meaningful empirical evaluation. Consequently, we introduce a novel benchmark dataset, dopanim, consisting of about 15,750 animal images of 15 classes with ground truth labels. For approximately 10,500 of these images, 20 humans provided over 52,000 annotations with an accuracy of circa 67%. Its key attributes include (1) the challenging task of classifying doppelganger animals, (2) human-estimated likelihoods as annotations, and (3) annotator metadata. We benchmark well-known multi-annotator learning approaches using seven variants of this dataset and outline further evaluation use cases such as learning beyond hard class labels and active learning. Our dataset and a comprehensive codebase are publicly available to emulate the data collection process and to reproduce all empirical results.
@inproceedings{herde2024dopanim, abstract = {Human annotators typically provide annotated data for training machine learning models, such as neural networks. Yet, human annotations are subject to noise, impairing generalization performances. Methodological research on approaches counteracting noisy annotations requires corresponding datasets for a meaningful empirical evaluation. Consequently, we introduce a novel benchmark dataset, dopanim, consisting of about 15,750 animal images of 15 classes with ground truth labels. For approximately 10,500 of these images, 20 humans provided over 52,000 annotations with an accuracy of circa 67%. Its key attributes include (1) the challenging task of classifying doppelganger animals, (2) human-estimated likelihoods as annotations, and (3) annotator metadata. We benchmark well-known multi-annotator learning approaches using seven variants of this dataset and outline further evaluation use cases such as learning beyond hard class labels and active learning. Our dataset and a comprehensive codebase are publicly available to emulate the data collection process and to reproduce all empirical results.}, author = {Herde, Marek and Huseljic, Denis and Rauch, Lukas and Sick, Bernhard}, booktitle = {Conference on Neural Information Processing Systems (NeurIPS 2024) Track on Datasets and Benchmarks}, keywords = {itegpub}, pages = {1–16}, title = {dopanim: A Dataset of Doppelganger Animals with Noisy Annotations from Multiple Humans}, year = 2024 }
%0 Conference Paper %1 herde2024dopanim %A Herde, Marek %A Huseljic, Denis %A Rauch, Lukas %A Sick, Bernhard %B Conference on Neural Information Processing Systems (NeurIPS 2024) Track on Datasets and Benchmarks %D 2024 %P 1--16 %T dopanim: A Dataset of Doppelganger Animals with Noisy Annotations from Multiple Humans %U https://openreview.net/forum?id=XOGosbxLrz %X Human annotators typically provide annotated data for training machine learning models, such as neural networks. Yet, human annotations are subject to noise, impairing generalization performances. Methodological research on approaches counteracting noisy annotations requires corresponding datasets for a meaningful empirical evaluation. Consequently, we introduce a novel benchmark dataset, dopanim, consisting of about 15,750 animal images of 15 classes with ground truth labels. For approximately 10,500 of these images, 20 humans provided over 52,000 annotations with an accuracy of circa 67%. Its key attributes include (1) the challenging task of classifying doppelganger animals, (2) human-estimated likelihoods as annotations, and (3) annotator metadata. We benchmark well-known multi-annotator learning approaches using seven variants of this dataset and outline further evaluation use cases such as learning beyond hard class labels and active learning. Our dataset and a comprehensive codebase are publicly available to emulate the data collection process and to reproduce all empirical results.
General Reusability: Ensuring Long-Term Benefits of Deep Active Learning.Hahn, Paul; Huseljic, Denis; Herde, Marek; Sick, Bernhard. In Workshop on Interactive Adapative Learning (IAL), ECML PKDD, bll 33–46. 2024.
- [ Abstract ]
- [ BibTeX ]
- [ EndNote ]
- [ URL ]
Deep active learning (DAL) provides a promising framework for reducing labeling costs for data-hungry deep neural networks. In DAL research, experiments are evaluated based on the performance of the model that is responsible for querying data, which can be interpreted as a model-centric evaluation. However, the data queried by DAL should be model agnostic so that new models can be effectively trained on it. An example can be derived from the rapid progress in deep learning, where new architectures are proposed in quick succession. Naturally, we want to employ these new and more effective models while still using the queried data from DAL. This results in the need to ensure high performance for both current and future models, which we refer to as reusability. In line with this topic, we propose general reusability, a novel, data-centric evaluation metric that measures the long-term value of queried data. Our experiments demonstrate that the current evaluation protocol in DAL research does not assess the reusability of data and that GR can fill this gap.
@inproceedings{hahn2024general, abstract = {Deep active learning (DAL) provides a promising framework for reducing labeling costs for data-hungry deep neural networks. In DAL research, experiments are evaluated based on the performance of the model that is responsible for querying data, which can be interpreted as a model-centric evaluation. However, the data queried by DAL should be model agnostic so that new models can be effectively trained on it. An example can be derived from the rapid progress in deep learning, where new architectures are proposed in quick succession. Naturally, we want to employ these new and more effective models while still using the queried data from DAL. This results in the need to ensure high performance for both current and future models, which we refer to as reusability. In line with this topic, we propose general reusability, a novel, data-centric evaluation metric that measures the long-term value of queried data. Our experiments demonstrate that the current evaluation protocol in DAL research does not assess the reusability of data and that GR can fill this gap.}, author = {Hahn, Paul and Huseljic, Denis and Herde, Marek and Sick, Bernhard}, booktitle = {Workshop on Interactive Adapative Learning (IAL), ECML PKDD}, keywords = {itegpub}, pages = {33–46}, title = {General Reusability: Ensuring Long-Term Benefits of Deep Active Learning}, year = 2024 }
%0 Conference Paper %1 hahn2024general %A Hahn, Paul %A Huseljic, Denis %A Herde, Marek %A Sick, Bernhard %B Workshop on Interactive Adapative Learning (IAL), ECML PKDD %D 2024 %P 33--46 %T General Reusability: Ensuring Long-Term Benefits of Deep Active Learning %U https://ceur-ws.org/Vol-3770/paper5.pdf %X Deep active learning (DAL) provides a promising framework for reducing labeling costs for data-hungry deep neural networks. In DAL research, experiments are evaluated based on the performance of the model that is responsible for querying data, which can be interpreted as a model-centric evaluation. However, the data queried by DAL should be model agnostic so that new models can be effectively trained on it. An example can be derived from the rapid progress in deep learning, where new architectures are proposed in quick succession. Naturally, we want to employ these new and more effective models while still using the queried data from DAL. This results in the need to ensure high performance for both current and future models, which we refer to as reusability. In line with this topic, we propose general reusability, a novel, data-centric evaluation metric that measures the long-term value of queried data. Our experiments demonstrate that the current evaluation protocol in DAL research does not assess the reusability of data and that GR can fill this gap.
Fast Fishing: Approximating BAIT for Efficient and Scalable Deep Active Image Classification.Huseljic, Denis; Hahn, Paul; Herde, Marek; Rauch, Lukas; Sick, Bernhard. In European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECMLPKDD), bll 280–296. 2024.
- [ Abstract ]
- [ BibTeX ]
- [ EndNote ]
Deep active learning (AL) seeks to minimize the annotation costs for training deep neural networks. \\textsc{Bait, a recently proposed AL strategy based on the Fisher Information, has demonstrated impressive performance across various datasets. However, \\textsc{Bait's high computational and memory requirements hinder its applicability on large-scale classification tasks, resulting in current research neglecting \\textsc{Bait in their evaluation. This paper introduces two methods to enhance \\textsc{Bait's computational efficiency and scalability. Notably, we significantly reduce its time complexity by approximating the Fisher Information. In particular, we adapt the original formulation by i) taking the expectation over the most probable classes, and ii) constructing a binary classification task, leading to an alternative likelihood for gradient computations. Consequently, this allows the efficient use of \\textsc{Bait on large-scale datasets, including ImageNet. Our unified and comprehensive evaluation across a variety of datasets demonstrates that our approximations achieve strong performance with considerably reduced time complexity. Furthermore, we provide an extensive open-source toolbox that implements recent state-of-the-art AL strategies, available at \\url{https://github.com/dhuseljic/dal-toolbox.
@inproceedings{huseljic2024fast, abstract = {Deep active learning (AL) seeks to minimize the annotation costs for training deep neural networks. \\textsc{Bait}, a recently proposed AL strategy based on the Fisher Information, has demonstrated impressive performance across various datasets. However, \\textsc{Bait}'s high computational and memory requirements hinder its applicability on large-scale classification tasks, resulting in current research neglecting \\textsc{Bait} in their evaluation. This paper introduces two methods to enhance \\textsc{Bait}'s computational efficiency and scalability. Notably, we significantly reduce its time complexity by approximating the Fisher Information. In particular, we adapt the original formulation by i) taking the expectation over the most probable classes, and ii) constructing a binary classification task, leading to an alternative likelihood for gradient computations. Consequently, this allows the efficient use of \\textsc{Bait} on large-scale datasets, including ImageNet. Our unified and comprehensive evaluation across a variety of datasets demonstrates that our approximations achieve strong performance with considerably reduced time complexity. Furthermore, we provide an extensive open-source toolbox that implements recent state-of-the-art AL strategies, available at \\url{https://github.com/dhuseljic/dal-toolbox}.}, author = {Huseljic, Denis and Hahn, Paul and Herde, Marek and Rauch, Lukas and Sick, Bernhard}, booktitle = {European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECMLPKDD)}, keywords = {itegpub}, pages = {280–296}, title = {Fast Fishing: Approximating BAIT for Efficient and Scalable Deep Active Image Classification}, year = 2024 }
%0 Conference Paper %1 huseljic2024fast %A Huseljic, Denis %A Hahn, Paul %A Herde, Marek %A Rauch, Lukas %A Sick, Bernhard %B European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECMLPKDD) %D 2024 %P 280--296 %R 10.1007/978-3-031-70368-3_17 %T Fast Fishing: Approximating BAIT for Efficient and Scalable Deep Active Image Classification %X Deep active learning (AL) seeks to minimize the annotation costs for training deep neural networks. \\textsc{Bait, a recently proposed AL strategy based on the Fisher Information, has demonstrated impressive performance across various datasets. However, \\textsc{Bait's high computational and memory requirements hinder its applicability on large-scale classification tasks, resulting in current research neglecting \\textsc{Bait in their evaluation. This paper introduces two methods to enhance \\textsc{Bait's computational efficiency and scalability. Notably, we significantly reduce its time complexity by approximating the Fisher Information. In particular, we adapt the original formulation by i) taking the expectation over the most probable classes, and ii) constructing a binary classification task, leading to an alternative likelihood for gradient computations. Consequently, this allows the efficient use of \\textsc{Bait on large-scale datasets, including ImageNet. Our unified and comprehensive evaluation across a variety of datasets demonstrates that our approximations achieve strong performance with considerably reduced time complexity. Furthermore, we provide an extensive open-source toolbox that implements recent state-of-the-art AL strategies, available at \\url{https://github.com/dhuseljic/dal-toolbox.

2023[ to top ]

Who knows best? A Case Study on Intelligent Crowdworker Selection via Deep Learning.Herde, Marek; Huseljic, Denis; Sick, Bernhard; Bretschneider, Ulrich; Oeste-Reiß, Sarah. In Workshop on Interactive Adapative Learning (IAL), ECML PKDD, bll 14–18. 2023.
- [ Abstract ]
- [ BibTeX ]
- [ EndNote ]
- [ URL ]
Crowdworking is a popular approach for annotating large amounts of data to train deep neural networks. However, parts of the annotations are often erroneous. In a case study, we demonstrate how an intelligent crowdworker selection via deep learning reduces the number of erroneous annotations and, thus, the annotation costs of obtaining reliable data for training deep neural networks.
@inproceedings{herde2023who, abstract = {Crowdworking is a popular approach for annotating large amounts of data to train deep neural networks. However, parts of the annotations are often erroneous. In a case study, we demonstrate how an intelligent crowdworker selection via deep learning reduces the number of erroneous annotations and, thus, the annotation costs of obtaining reliable data for training deep neural networks.}, author = {Herde, Marek and Huseljic, Denis and Sick, Bernhard and Bretschneider, Ulrich and Oeste-Reiß, Sarah}, booktitle = {Workshop on Interactive Adapative Learning (IAL), ECML PKDD}, keywords = {itegpub}, pages = {14–18}, title = {Who knows best? A Case Study on Intelligent Crowdworker Selection via Deep Learning}, year = 2023 }
%0 Conference Paper %1 herde2023who %A Herde, Marek %A Huseljic, Denis %A Sick, Bernhard %A Bretschneider, Ulrich %A Oeste-Reiß, Sarah %B Workshop on Interactive Adapative Learning (IAL), ECML PKDD %D 2023 %P 14--18 %T Who knows best? A Case Study on Intelligent Crowdworker Selection via Deep Learning %U https://ceur-ws.org/Vol-3470/paper3.pdf %X Crowdworking is a popular approach for annotating large amounts of data to train deep neural networks. However, parts of the annotations are often erroneous. In a case study, we demonstrate how an intelligent crowdworker selection via deep learning reduces the number of erroneous annotations and, thus, the annotation costs of obtaining reliable data for training deep neural networks.
Active Label Refinement for Semantic Segmentation of Satellite Images.Pham, Minh Tuan; Wijesingha, Jayan; Kottke, Daniel; Herde, Marek; Huseljic, Denis; Sick, Bernhard; Wachendorf, Michael; Esch, Thomas. In arXiv e-prints, bl arXiv:2309.06159. 2023.
- [ Abstract ]
- [ BibTeX ]
- [ EndNote ]
- [ URL ]
Remote sensing through semantic segmentation of satellite images contributes to the understanding and utilisation of the earth's surface. For this purpose, semantic segmentation networks are typically trained on large sets of labelled satellite images. However, obtaining expert labels for these images is costly. Therefore, we propose to rely on a low-cost approach, e.g. crowdsourcing or pretrained networks, to label the images in the first step. Since these initial labels are partially erroneous, we use active learning strategies to cost-efficiently refine the labels in the second step. We evaluate the active learning strategies using satellite images of Bengaluru in India, labelled with land cover and land use labels. Our experimental results suggest that an active label refinement to improve the semantic segmentation network's performance is beneficial.
@article{pham2023active, abstract = {Remote sensing through semantic segmentation of satellite images contributes to the understanding and utilisation of the earth's surface. For this purpose, semantic segmentation networks are typically trained on large sets of labelled satellite images. However, obtaining expert labels for these images is costly. Therefore, we propose to rely on a low-cost approach, e.g. crowdsourcing or pretrained networks, to label the images in the first step. Since these initial labels are partially erroneous, we use active learning strategies to cost-efficiently refine the labels in the second step. We evaluate the active learning strategies using satellite images of Bengaluru in India, labelled with land cover and land use labels. Our experimental results suggest that an active label refinement to improve the semantic segmentation network's performance is beneficial.}, author = {Pham, Minh Tuan and Wijesingha, Jayan and Kottke, Daniel and Herde, Marek and Huseljic, Denis and Sick, Bernhard and Wachendorf, Michael and Esch, Thomas}, journal = {arXiv e-prints}, keywords = {itegpub}, pages = {arXiv:2309.06159}, title = {Active Label Refinement for Semantic Segmentation of Satellite Images}, year = 2023 }
%0 Journal Article %1 pham2023active %A Pham, Minh Tuan %A Wijesingha, Jayan %A Kottke, Daniel %A Herde, Marek %A Huseljic, Denis %A Sick, Bernhard %A Wachendorf, Michael %A Esch, Thomas %D 2023 %J arXiv e-prints %P arXiv:2309.06159 %T Active Label Refinement for Semantic Segmentation of Satellite Images %U https://arxiv.org/abs/2309.06159 %X Remote sensing through semantic segmentation of satellite images contributes to the understanding and utilisation of the earth's surface. For this purpose, semantic segmentation networks are typically trained on large sets of labelled satellite images. However, obtaining expert labels for these images is costly. Therefore, we propose to rely on a low-cost approach, e.g. crowdsourcing or pretrained networks, to label the images in the first step. Since these initial labels are partially erroneous, we use active learning strategies to cost-efficiently refine the labels in the second step. We evaluate the active learning strategies using satellite images of Bengaluru in India, labelled with land cover and land use labels. Our experimental results suggest that an active label refinement to improve the semantic segmentation network's performance is beneficial.
Active Learning with Fast Model Updates and Class-Balanced Selection for Imbalanced Datasets.Huang, Zhixin; He, Yujiang; Herde, Marek; Huseljic, Denis; Sick, Bernhard. In Workshop on Interactive Adapative Learning (IAL), ECML PKDD, bll 28–45. 2023.
- [ Abstract ]
- [ BibTeX ]
- [ EndNote ]
- [ URL ]
Active learning strategies aim to minimize the number of queried samples for model training. However, two challenges in pool-based deep active learning on imbalanced datasets are observed in experiments: (1) the declining performance of active learning strategies due to imbalanced class distribution; (2)~the lack of sample diversity in acquisition batches due to the absence of timely model updates. This paper proposes the AL-FaMoUS, a general solution combining fast model updates and class-balanced minibatch selection to the active learning process. Furthermore, a simplification of the AL-FaMoUS, which selects one single sample in each acquisition minibatch, is experimentally evaluated on four image and three time-series imbalanced datasets. The results demonstrate that the implemented AL-FaMoUS outperforms the other adopted AL strategies, including uncertainty sampling and BALD solely combined with either the fast model update or the class balance selection strategy, in terms of accuracy and Macro F1 score.
@inproceedings{huang2023active, abstract = {Active learning strategies aim to minimize the number of queried samples for model training. However, two challenges in pool-based deep active learning on imbalanced datasets are observed in experiments: (1) the declining performance of active learning strategies due to imbalanced class distribution; (2) the lack of sample diversity in acquisition batches due to the absence of timely model updates. This paper proposes the AL-FaMoUS, a general solution combining fast model updates and class-balanced minibatch selection to the active learning process. Furthermore, a simplification of the AL-FaMoUS, which selects one single sample in each acquisition minibatch, is experimentally evaluated on four image and three time-series imbalanced datasets. The results demonstrate that the implemented AL-FaMoUS outperforms the other adopted AL strategies, including uncertainty sampling and BALD solely combined with either the fast model update or the class balance selection strategy, in terms of accuracy and Macro F1 score.}, author = {Huang, Zhixin and He, Yujiang and Herde, Marek and Huseljic, Denis and Sick, Bernhard}, booktitle = {Workshop on Interactive Adapative Learning (IAL), ECML PKDD}, keywords = {itegpub}, pages = {28–45}, title = {Active Learning with Fast Model Updates and Class-Balanced Selection for Imbalanced Datasets}, year = 2023 }
%0 Conference Paper %1 huang2023active %A Huang, Zhixin %A He, Yujiang %A Herde, Marek %A Huseljic, Denis %A Sick, Bernhard %B Workshop on Interactive Adapative Learning (IAL), ECML PKDD %D 2023 %P 28--45 %T Active Learning with Fast Model Updates and Class-Balanced Selection for Imbalanced Datasets %U https://ceur-ws.org/Vol-3470/paper5.pdf %X Active learning strategies aim to minimize the number of queried samples for model training. However, two challenges in pool-based deep active learning on imbalanced datasets are observed in experiments: (1) the declining performance of active learning strategies due to imbalanced class distribution; (2)~the lack of sample diversity in acquisition batches due to the absence of timely model updates. This paper proposes the AL-FaMoUS, a general solution combining fast model updates and class-balanced minibatch selection to the active learning process. Furthermore, a simplification of the AL-FaMoUS, which selects one single sample in each acquisition minibatch, is experimentally evaluated on four image and three time-series imbalanced datasets. The results demonstrate that the implemented AL-FaMoUS outperforms the other adopted AL strategies, including uncertainty sampling and BALD solely combined with either the fast model update or the class balance selection strategy, in terms of accuracy and Macro F1 score.
Exploring the Potential of Optimal Active Learning via a Non-myopic Oracle Policy.Sandrock, Christoph; Herde, Marek; Kottke, Daniel; Sick, Bernhard. In Discovery Science (DS), bll 265–276. Springer, 2023.
- [ Abstract ]
- [ BibTeX ]
- [ EndNote ]
- [ URL ]
Active learning aims to reduce the amount of labeled data while maximizing machine learning models’ performances. Currently, there is sparse research on the potential of an optimal active learning strategy. Therefore, we propose a non-myopic oracle policy that accesses the true labels of the data pool to approximate an optimal active learning strategy. We evaluate how the hyperparameters of this oracle policy influence its performance and empirically demonstrate that it is an upper baseline for common active learning strategies while being faster than a state-of-the-art oracle policy. For the sake of reproducibility, all the code related to our research is publicly available on our GitHub repository at https://github.com/ies-research/non-myopic-oracle-policy.
@inproceedings{sandrock2023exploring, abstract = {Active learning aims to reduce the amount of labeled data while maximizing machine learning models’ performances. Currently, there is sparse research on the potential of an optimal active learning strategy. Therefore, we propose a non-myopic oracle policy that accesses the true labels of the data pool to approximate an optimal active learning strategy. We evaluate how the hyperparameters of this oracle policy influence its performance and empirically demonstrate that it is an upper baseline for common active learning strategies while being faster than a state-of-the-art oracle policy. For the sake of reproducibility, all the code related to our research is publicly available on our GitHub repository at https://github.com/ies-research/non-myopic-oracle-policy.}, author = {Sandrock, Christoph and Herde, Marek and Kottke, Daniel and Sick, Bernhard}, booktitle = {Discovery Science (DS)}, keywords = {itegpub}, pages = {265–276}, publisher = {Springer}, title = {Exploring the Potential of Optimal Active Learning via a Non-myopic Oracle Policy}, year = 2023 }
%0 Conference Paper %1 sandrock2023exploring %A Sandrock, Christoph %A Herde, Marek %A Kottke, Daniel %A Sick, Bernhard %B Discovery Science (DS) %D 2023 %I Springer %P 265--276 %R 10.1007/978-3-031-45275-8_18 %T Exploring the Potential of Optimal Active Learning via a Non-myopic Oracle Policy %U https://link.springer.com/chapter/10.1007/978-3-031-45275-8_18 %X Active learning aims to reduce the amount of labeled data while maximizing machine learning models’ performances. Currently, there is sparse research on the potential of an optimal active learning strategy. Therefore, we propose a non-myopic oracle policy that accesses the true labels of the data pool to approximate an optimal active learning strategy. We evaluate how the hyperparameters of this oracle policy influence its performance and empirically demonstrate that it is an upper baseline for common active learning strategies while being faster than a state-of-the-art oracle policy. For the sake of reproducibility, all the code related to our research is publicly available on our GitHub repository at https://github.com/ies-research/non-myopic-oracle-policy.
Multi-annotator Deep Learning: A Probabilistic Framework for Classification.Herde, Marek; Huseljic, Denis; Sick, Bernhard. In Transactions on Machine Learning Research. 2023.
- [ Abstract ]
- [ BibTeX ]
- [ EndNote ]
- [ URL ]
Solving complex classification tasks using deep neural networks typically requires large amounts of annotated data. However, corresponding class labels are noisy when provided by error-prone annotators, e.g., crowdworkers. Training standard deep neural networks leads to subpar performances in such multi-annotator supervised learning settings. We address this issue by presenting a probabilistic training framework named multi-annotator deep learning (MaDL). A downstream ground truth and an annotator performance model are jointly trained in an end-to-end learning approach. The ground truth model learns to predict instances' true class labels, while the annotator performance model infers probabilistic estimates of annotators' performances. A modular network architecture enables us to make varying assumptions regarding annotators' performances, e.g., an optional class or instance dependency. Further, we learn annotator embeddings to estimate annotators' densities within a latent space as proxies of their potentially correlated annotations. Together with a weighted loss function, we improve the learning from correlated annotation patterns. In a comprehensive evaluation, we examine three research questions about multi-annotator supervised learning. Our findings show MaDL's state-of-the-art performance and robustness against many correlated, spamming annotators.
@article{herde2023multi, abstract = {Solving complex classification tasks using deep neural networks typically requires large amounts of annotated data. However, corresponding class labels are noisy when provided by error-prone annotators, e.g., crowdworkers. Training standard deep neural networks leads to subpar performances in such multi-annotator supervised learning settings. We address this issue by presenting a probabilistic training framework named multi-annotator deep learning (MaDL). A downstream ground truth and an annotator performance model are jointly trained in an end-to-end learning approach. The ground truth model learns to predict instances' true class labels, while the annotator performance model infers probabilistic estimates of annotators' performances. A modular network architecture enables us to make varying assumptions regarding annotators' performances, e.g., an optional class or instance dependency. Further, we learn annotator embeddings to estimate annotators' densities within a latent space as proxies of their potentially correlated annotations. Together with a weighted loss function, we improve the learning from correlated annotation patterns. In a comprehensive evaluation, we examine three research questions about multi-annotator supervised learning. Our findings show MaDL's state-of-the-art performance and robustness against many correlated, spamming annotators.}, author = {Herde, Marek and Huseljic, Denis and Sick, Bernhard}, journal = {Transactions on Machine Learning Research}, keywords = {itegpub}, title = {Multi-annotator Deep Learning: A Probabilistic Framework for Classification}, year = 2023 }
%0 Journal Article %1 herde2023multi %A Herde, Marek %A Huseljic, Denis %A Sick, Bernhard %D 2023 %J Transactions on Machine Learning Research %T Multi-annotator Deep Learning: A Probabilistic Framework for Classification %U https://openreview.net/forum?id=MgdoxzImlK %X Solving complex classification tasks using deep neural networks typically requires large amounts of annotated data. However, corresponding class labels are noisy when provided by error-prone annotators, e.g., crowdworkers. Training standard deep neural networks leads to subpar performances in such multi-annotator supervised learning settings. We address this issue by presenting a probabilistic training framework named multi-annotator deep learning (MaDL). A downstream ground truth and an annotator performance model are jointly trained in an end-to-end learning approach. The ground truth model learns to predict instances' true class labels, while the annotator performance model infers probabilistic estimates of annotators' performances. A modular network architecture enables us to make varying assumptions regarding annotators' performances, e.g., an optional class or instance dependency. Further, we learn annotator embeddings to estimate annotators' densities within a latent space as proxies of their potentially correlated annotations. Together with a weighted loss function, we improve the learning from correlated annotation patterns. In a comprehensive evaluation, we examine three research questions about multi-annotator supervised learning. Our findings show MaDL's state-of-the-art performance and robustness against many correlated, spamming annotators.
Role of Hyperparameters in Deep Active Learning.Huseljic, Denis; Herde, Marek; Hahn, Paul; Sick, Bernhard. In Workshop on Interactive Adaptive Learning (IAL), ECML PKDD, bll 19–24. 2023.
- [ Abstract ]
- [ BibTeX ]
- [ EndNote ]
- [ URL ]
Despite considerable research on pool-based deep active learning (DAL), deploying DAL into real applications still has several challenges. A frequently neglected aspect is the choice of training hyperparameters (HPs), such as the learning rate. Since these HPs determine how the deep neural network learns in each cycle, they must be chosen carefully. In this article, we analyze the role of HPs in DAL. We find that optimizing HPs reduces the performance gap between DAL strategies. Conversely, we highlight challenges in finding optimal HPs when using datasets selected via DAL strategies.
@inproceedings{huseljic2023role, abstract = {Despite considerable research on pool-based deep active learning (DAL), deploying DAL into real applications still has several challenges. A frequently neglected aspect is the choice of training hyperparameters (HPs), such as the learning rate. Since these HPs determine how the deep neural network learns in each cycle, they must be chosen carefully. In this article, we analyze the role of HPs in DAL. We find that optimizing HPs reduces the performance gap between DAL strategies. Conversely, we highlight challenges in finding optimal HPs when using datasets selected via DAL strategies.}, author = {Huseljic, Denis and Herde, Marek and Hahn, Paul and Sick, Bernhard}, booktitle = {Workshop on Interactive Adaptive Learning (IAL), ECML PKDD}, keywords = {itegpub}, pages = {19–24}, title = {Role of Hyperparameters in Deep Active Learning}, year = 2023 }
%0 Conference Paper %1 huseljic2023role %A Huseljic, Denis %A Herde, Marek %A Hahn, Paul %A Sick, Bernhard %B Workshop on Interactive Adaptive Learning (IAL), ECML PKDD %D 2023 %P 19--24 %T Role of Hyperparameters in Deep Active Learning %U https://ceur-ws.org/Vol-3470/paper4.pdf %X Despite considerable research on pool-based deep active learning (DAL), deploying DAL into real applications still has several challenges. A frequently neglected aspect is the choice of training hyperparameters (HPs), such as the learning rate. Since these HPs determine how the deep neural network learns in each cycle, they must be chosen carefully. In this article, we analyze the role of HPs in DAL. We find that optimizing HPs reduces the performance gap between DAL strategies. Conversely, we highlight challenges in finding optimal HPs when using datasets selected via DAL strategies.

2022[ to top ]

Fast Bayesian Updates for Deep Learning with a Use Case in Active Learning.Herde, Marek; Huang, Zhixin; Huseljic, Denis; Kottke, Daniel; Vogt, Stephan; Sick, Bernhard. In arXiv e-prints, bl arXiv:2210.06112. 2022.
- [ Abstract ]
- [ BibTeX ]
- [ EndNote ]
- [ URL ]
Retraining deep neural networks when new data arrives is typically computationally expensive. Moreover, certain applications do not allow such costly retraining due to time or computational constraints. Fast Bayesian updates are a possible solution to this issue. Therefore, we propose a Bayesian update based on Monte-Carlo samples and a last-layer Laplace approximation for different Bayesian neural network types, i.e., Dropout, Ensemble, and Spectral Normalized Neural Gaussian Process (SNGP). In a large-scale evaluation study, we show that our updates combined with SNGP represent a fast and competitive alternative to costly retraining. As a use case, we combine the Bayesian updates for SNGP with different sequential query strategies to exemplarily demonstrate their improved selection performance in active learning.
@article{herde2022fast, abstract = {Retraining deep neural networks when new data arrives is typically computationally expensive. Moreover, certain applications do not allow such costly retraining due to time or computational constraints. Fast Bayesian updates are a possible solution to this issue. Therefore, we propose a Bayesian update based on Monte-Carlo samples and a last-layer Laplace approximation for different Bayesian neural network types, i.e., Dropout, Ensemble, and Spectral Normalized Neural Gaussian Process (SNGP). In a large-scale evaluation study, we show that our updates combined with SNGP represent a fast and competitive alternative to costly retraining. As a use case, we combine the Bayesian updates for SNGP with different sequential query strategies to exemplarily demonstrate their improved selection performance in active learning.}, author = {Herde, Marek and Huang, Zhixin and Huseljic, Denis and Kottke, Daniel and Vogt, Stephan and Sick, Bernhard}, journal = {arXiv e-prints}, keywords = {itegpub}, pages = {arXiv:2210.06112}, title = {Fast Bayesian Updates for Deep Learning with a Use Case in Active Learning}, year = 2022 }
%0 Journal Article %1 herde2022fast %A Herde, Marek %A Huang, Zhixin %A Huseljic, Denis %A Kottke, Daniel %A Vogt, Stephan %A Sick, Bernhard %D 2022 %J arXiv e-prints %P arXiv:2210.06112 %T Fast Bayesian Updates for Deep Learning with a Use Case in Active Learning %U https://arxiv.org/abs/2210.06112 %X Retraining deep neural networks when new data arrives is typically computationally expensive. Moreover, certain applications do not allow such costly retraining due to time or computational constraints. Fast Bayesian updates are a possible solution to this issue. Therefore, we propose a Bayesian update based on Monte-Carlo samples and a last-layer Laplace approximation for different Bayesian neural network types, i.e., Dropout, Ensemble, and Spectral Normalized Neural Gaussian Process (SNGP). In a large-scale evaluation study, we show that our updates combined with SNGP represent a fast and competitive alternative to costly retraining. As a use case, we combine the Bayesian updates for SNGP with different sequential query strategies to exemplarily demonstrate their improved selection performance in active learning.
A Review of Uncertainty Calibration in Pretrained Object Detectors.Huseljic, Denis; Herde, Marek; Muejde, Mehmet; Sick, Bernhard. In arXiv e-prints, bl arXiv:2210.02935. 2022.
- [ Abstract ]
- [ BibTeX ]
- [ EndNote ]
- [ URL ]
In the field of deep learning based computer vision, the development of deep object detection has led to unique paradigms (e.g., two-stage or set-based) and architectures (e.g., Faster-RCNN or DETR) which enable outstanding performance on challenging benchmark datasets. Despite this, the trained object detectors typically do not reliably assess uncertainty regarding their own knowledge, and the quality of their probabilistic predictions is usually poor. As these are often used to make subsequent decisions, such inaccurate probabilistic predictions must be avoided. In this work, we investigate the uncertainty calibration properties of different pretrained object detection architectures in a multi-class setting. We propose a framework to ensure a fair, unbiased, and repeatable evaluation and conduct detailed analyses assessing the calibration under distributional changes (e.g., distributional shift and application to out-of-distribution data). Furthermore, by investigating the influence of different detector paradigms, post-processing steps, and suitable choices of metrics, we deliver novel insights into why poor detector calibration emerges. Based on these insights, we are able to improve the calibration of a detector by simply finetuning its last layer.
@article{huseljic2022review, abstract = {In the field of deep learning based computer vision, the development of deep object detection has led to unique paradigms (e.g., two-stage or set-based) and architectures (e.g., Faster-RCNN or DETR) which enable outstanding performance on challenging benchmark datasets. Despite this, the trained object detectors typically do not reliably assess uncertainty regarding their own knowledge, and the quality of their probabilistic predictions is usually poor. As these are often used to make subsequent decisions, such inaccurate probabilistic predictions must be avoided. In this work, we investigate the uncertainty calibration properties of different pretrained object detection architectures in a multi-class setting. We propose a framework to ensure a fair, unbiased, and repeatable evaluation and conduct detailed analyses assessing the calibration under distributional changes (e.g., distributional shift and application to out-of-distribution data). Furthermore, by investigating the influence of different detector paradigms, post-processing steps, and suitable choices of metrics, we deliver novel insights into why poor detector calibration emerges. Based on these insights, we are able to improve the calibration of a detector by simply finetuning its last layer.}, author = {Huseljic, Denis and Herde, Marek and Muejde, Mehmet and Sick, Bernhard}, journal = {arXiv e-prints}, keywords = {itegpub}, pages = {arXiv:2210.02935}, title = {A Review of Uncertainty Calibration in Pretrained Object Detectors}, year = 2022 }
%0 Journal Article %1 huseljic2022review %A Huseljic, Denis %A Herde, Marek %A Muejde, Mehmet %A Sick, Bernhard %D 2022 %J arXiv e-prints %P arXiv:2210.02935 %T A Review of Uncertainty Calibration in Pretrained Object Detectors %U https://arxiv.org/abs/2210.02935 %X In the field of deep learning based computer vision, the development of deep object detection has led to unique paradigms (e.g., two-stage or set-based) and architectures (e.g., Faster-RCNN or DETR) which enable outstanding performance on challenging benchmark datasets. Despite this, the trained object detectors typically do not reliably assess uncertainty regarding their own knowledge, and the quality of their probabilistic predictions is usually poor. As these are often used to make subsequent decisions, such inaccurate probabilistic predictions must be avoided. In this work, we investigate the uncertainty calibration properties of different pretrained object detection architectures in a multi-class setting. We propose a framework to ensure a fair, unbiased, and repeatable evaluation and conduct detailed analyses assessing the calibration under distributional changes (e.g., distributional shift and application to out-of-distribution data). Furthermore, by investigating the influence of different detector paradigms, post-processing steps, and suitable choices of metrics, we deliver novel insights into why poor detector calibration emerges. Based on these insights, we are able to improve the calibration of a detector by simply finetuning its last layer.
A Concept for Automated Polarized Web Content Annotation based on Multimodal Active Learning.Herde, Marek; Huseljic, Denis; Mitrovic, Jelena; Granitzer, Michael; Sick, Bernhard. In Workshop on Interactive Adaptive Learning (IAL), ECML PKDD, bll 1–6. 2022.
- [ Abstract ]
- [ BibTeX ]
- [ EndNote ]
- [ URL ]
Active learning (AL) techniques hardly cope with complex annotations tasks, where, for example, annotations might express relationships across data modalities. As a use case, we consider the task of automatically detecting and reporting multimodal, polarized web content (PWC). Samples of this content type emerge dynamically, covering a broad spectrum of topics. Thus, training machine learning systems for detecting PWC is challenging, particularly if it needs to be done with minimum annotation cost. In this article, we propose the concept of multimodal AL for complex annotations in the context of PWC detection and formulate the resulting challenges as questions for future research.
@inproceedings{herde2022concept, abstract = {Active learning (AL) techniques hardly cope with complex annotations tasks, where, for example, annotations might express relationships across data modalities. As a use case, we consider the task of automatically detecting and reporting multimodal, polarized web content (PWC). Samples of this content type emerge dynamically, covering a broad spectrum of topics. Thus, training machine learning systems for detecting PWC is challenging, particularly if it needs to be done with minimum annotation cost. In this article, we propose the concept of multimodal AL for complex annotations in the context of PWC detection and formulate the resulting challenges as questions for future research.}, author = {Herde, Marek and Huseljic, Denis and Mitrovic, Jelena and Granitzer, Michael and Sick, Bernhard}, booktitle = {Workshop on Interactive Adaptive Learning (IAL), ECML PKDD}, keywords = {itegpub}, pages = {1–6}, title = {A Concept for Automated Polarized Web Content Annotation based on Multimodal Active Learning}, year = 2022 }
%0 Conference Paper %1 herde2022concept %A Herde, Marek %A Huseljic, Denis %A Mitrovic, Jelena %A Granitzer, Michael %A Sick, Bernhard %B Workshop on Interactive Adaptive Learning (IAL), ECML PKDD %D 2022 %P 1--6 %T A Concept for Automated Polarized Web Content Annotation based on Multimodal Active Learning %U http://ceur-ws.org/Vol-3259/ialatecml_paper1.pdf %X Active learning (AL) techniques hardly cope with complex annotations tasks, where, for example, annotations might express relationships across data modalities. As a use case, we consider the task of automatically detecting and reporting multimodal, polarized web content (PWC). Samples of this content type emerge dynamically, covering a broad spectrum of topics. Thus, training machine learning systems for detecting PWC is challenging, particularly if it needs to be done with minimum annotation cost. In this article, we propose the concept of multimodal AL for complex annotations in the context of PWC detection and formulate the resulting challenges as questions for future research.

2021[ to top ]

Multi-annotator Probabilistic Active Learning.Herde, Marek; Kottke, Daniel; Huseljic, Denis; Sick, Bernhard. In International Conference on Pattern Recognition (ICPR), bll 10281–10288. IEEE, 2021.
- [ BibTeX ]
- [ EndNote ]
@inproceedings{herde2021multi, author = {Herde, Marek and Kottke, Daniel and Huseljic, Denis and Sick, Bernhard}, booktitle = {International Conference on Pattern Recognition (ICPR)}, keywords = {itegpub}, pages = {10281–10288}, publisher = {IEEE}, title = {Multi-annotator Probabilistic Active Learning}, year = 2021 }
%0 Conference Paper %1 herde2021multi %A Herde, Marek %A Kottke, Daniel %A Huseljic, Denis %A Sick, Bernhard %B International Conference on Pattern Recognition (ICPR) %D 2021 %I IEEE %P 10281--10288 %R 10.1109/icpr48806.2021.9412298 %T Multi-annotator Probabilistic Active Learning
scikit-activeml: A Library and Toolbox for Active Learning Algorithms.Kottke, Daniel; Herde, Marek; Minh, Tuan Pham; Benz, Alexander; Mergard, Pascal; Roghman, Atal; Sandrock, Christoph; Sick, Bernhard. In Preprints, bl 2021030194. 2021.
- [ Abstract ]
- [ BibTeX ]
- [ EndNote ]
- [ URL ]
Machine learning applications often need large amounts of training data to perform well. Whereas unlabeled data can be easily gathered, the labeling process is difficult, time-consuming, or expensive in most applications. Active learning can help solve this problem by querying labels for those data points that will improve the performance the most. Thereby, the goal is that the learning algorithm performs sufficiently well with fewer labels. We provide a library called scikit-activeml that covers the most relevant query strategies and implements tools to work with partially labeled data. It is programmed in Python and builds on top of scikit-learn.
@article{kottke2021scikit, abstract = {Machine learning applications often need large amounts of training data to perform well. Whereas unlabeled data can be easily gathered, the labeling process is difficult, time-consuming, or expensive in most applications. Active learning can help solve this problem by querying labels for those data points that will improve the performance the most. Thereby, the goal is that the learning algorithm performs sufficiently well with fewer labels. We provide a library called scikit-activeml that covers the most relevant query strategies and implements tools to work with partially labeled data. It is programmed in Python and builds on top of scikit-learn.}, author = {Kottke, Daniel and Herde, Marek and Minh, Tuan Pham and Benz, Alexander and Mergard, Pascal and Roghman, Atal and Sandrock, Christoph and Sick, Bernhard}, journal = {Preprints}, keywords = {itegpub}, pages = 2021030194, title = {{scikit-activeml}: {A} Library and Toolbox for Active Learning Algorithms}, year = 2021 }
%0 Journal Article %1 kottke2021scikit %A Kottke, Daniel %A Herde, Marek %A Minh, Tuan Pham %A Benz, Alexander %A Mergard, Pascal %A Roghman, Atal %A Sandrock, Christoph %A Sick, Bernhard %D 2021 %J Preprints %P 2021030194 %T scikit-activeml: A Library and Toolbox for Active Learning Algorithms %U https://www.preprints.org/manuscript/202103.0194/ %X Machine learning applications often need large amounts of training data to perform well. Whereas unlabeled data can be easily gathered, the labeling process is difficult, time-consuming, or expensive in most applications. Active learning can help solve this problem by querying labels for those data points that will improve the performance the most. Thereby, the goal is that the learning algorithm performs sufficiently well with fewer labels. We provide a library called scikit-activeml that covers the most relevant query strategies and implements tools to work with partially labeled data. It is programmed in Python and builds on top of scikit-learn.
Separation of Aleatoric and Epistemic Uncertainty in Deterministic Deep Neural Networks.Huseljic, Denis; Sick, Bernhard; Herde, Marek; Kottke, Daniel. In International Conference on Pattern Recognition (ICPR), bll 9172–9179. IEEE, 2021.
- [ BibTeX ]
- [ EndNote ]
@inproceedings{huseljic2021separation, author = {Huseljic, Denis and Sick, Bernhard and Herde, Marek and Kottke, Daniel}, booktitle = {International Conference on Pattern Recognition (ICPR)}, keywords = {itegpub}, pages = {9172–9179}, publisher = {IEEE}, title = {Separation of Aleatoric and Epistemic Uncertainty in Deterministic Deep Neural Networks}, year = 2021 }
%0 Conference Paper %1 huseljic2021separation %A Huseljic, Denis %A Sick, Bernhard %A Herde, Marek %A Kottke, Daniel %B International Conference on Pattern Recognition (ICPR) %D 2021 %I IEEE %P 9172--9179 %R 10.1109/icpr48806.2021.9412616 %T Separation of Aleatoric and Epistemic Uncertainty in Deterministic Deep Neural Networks
Toward optimal probabilistic active learning using a Bayesian approach.Kottke, Daniel; Herde, Marek; Sandrock, Christoph; Huseljic, Denis; Krempl, Georg; Sick, Bernhard. In Machine Learning, 110(6), bll 1199–1231. Springer, 2021.
- [ Abstract ]
- [ BibTeX ]
- [ EndNote ]
Gathering labeled data to train well-performing machine learning models is one of the critical challenges in many applications. Active learning aims at reducing the labeling costs by an efficient and effective allocation of costly labeling resources. In this article, we propose a decision-theoretic selection strategy that (1) directly optimizes the gain in misclassification error, and (2) uses a Bayesian approach by introducing a conjugate prior distribution to determine the class posterior to deal with uncertainties. By reformulating existing selection strategies within our proposed model, we can explain which aspects are not covered in current state-of-the-art and why this leads to the superior performance of our approach. Extensive experiments on a large variety of datasets and different kernels validate our claims.
@article{kottke2021optimal, abstract = {Gathering labeled data to train well-performing machine learning models is one of the critical challenges in many applications. Active learning aims at reducing the labeling costs by an efficient and effective allocation of costly labeling resources. In this article, we propose a decision-theoretic selection strategy that (1) directly optimizes the gain in misclassification error, and (2) uses a Bayesian approach by introducing a conjugate prior distribution to determine the class posterior to deal with uncertainties. By reformulating existing selection strategies within our proposed model, we can explain which aspects are not covered in current state-of-the-art and why this leads to the superior performance of our approach. Extensive experiments on a large variety of datasets and different kernels validate our claims.}, author = {Kottke, Daniel and Herde, Marek and Sandrock, Christoph and Huseljic, Denis and Krempl, Georg and Sick, Bernhard}, journal = {Machine Learning}, keywords = {itegpub}, number = 6, pages = {1199–1231}, publisher = {Springer}, title = {Toward optimal probabilistic active learning using a Bayesian approach}, volume = 110, year = 2021 }
%0 Journal Article %1 kottke2021optimal %A Kottke, Daniel %A Herde, Marek %A Sandrock, Christoph %A Huseljic, Denis %A Krempl, Georg %A Sick, Bernhard %D 2021 %I Springer %J Machine Learning %N 6 %P 1199--1231 %R 10.1007/s10994-021-05986-9 %T Toward optimal probabilistic active learning using a Bayesian approach %V 110 %X Gathering labeled data to train well-performing machine learning models is one of the critical challenges in many applications. Active learning aims at reducing the labeling costs by an efficient and effective allocation of costly labeling resources. In this article, we propose a decision-theoretic selection strategy that (1) directly optimizes the gain in misclassification error, and (2) uses a Bayesian approach by introducing a conjugate prior distribution to determine the class posterior to deal with uncertainties. By reformulating existing selection strategies within our proposed model, we can explain which aspects are not covered in current state-of-the-art and why this leads to the superior performance of our approach. Extensive experiments on a large variety of datasets and different kernels validate our claims.
A Survey on Cost Types, Interaction Schemes, and Annotator Performance Models in Selection Algorithms for Active Learning in Classification.Herde, Marek; Huseljic, Denis; Sick, Bernhard; Calma, Adrian. In IEEE Access, 9, bll 166970–166989. IEEE, 2021.
- [ Abstract ]
- [ BibTeX ]
- [ EndNote ]
- [ URL ]
Pool-based active learning (AL) aims to optimize the annotation process (i.e., labeling) as the acquisition of annotations is often time-consuming and therefore expensive. For this purpose, an AL strategy queries annotations intelligently from annotators to train a high-performance classification model at a low annotation cost. Traditional AL strategies operate in an idealized framework. They assume a single, omniscient annotator who never gets tired and charges uniformly regardless of query difficulty. However, in real-world applications, we often face human annotators, e.g., crowd or in-house workers, who make annotation mistakes and can be reluctant to respond if tired or faced with complex queries. Recently, many novel AL strategies have been proposed to address these issues. They differ in at least one of the following three central aspects from traditional AL: 1) modeling of (multiple) human annotators whose performances can be affected by various factors, such as missing expertise; 2) generalization of the interaction with human annotators through different query and annotation types, such as asking an annotator for feedback on an inferred classification rule; 3) consideration of complex cost schemes regarding annotations and misclassification. This survey provides an overview of these AL strategies and refers to them as real-world AL. Therefore, we introduce a general real-world AL strategy as part of a learning cycle and use its elements, e.g., the query and annotator selection algorithm, to categorize about 60 real-world AL strategies. Finally, we outline possible directions for future research in the field of AL.
@article{herde2021survey, abstract = {Pool-based active learning (AL) aims to optimize the annotation process (i.e., labeling) as the acquisition of annotations is often time-consuming and therefore expensive. For this purpose, an AL strategy queries annotations intelligently from annotators to train a high-performance classification model at a low annotation cost. Traditional AL strategies operate in an idealized framework. They assume a single, omniscient annotator who never gets tired and charges uniformly regardless of query difficulty. However, in real-world applications, we often face human annotators, e.g., crowd or in-house workers, who make annotation mistakes and can be reluctant to respond if tired or faced with complex queries. Recently, many novel AL strategies have been proposed to address these issues. They differ in at least one of the following three central aspects from traditional AL: 1) modeling of (multiple) human annotators whose performances can be affected by various factors, such as missing expertise; 2) generalization of the interaction with human annotators through different query and annotation types, such as asking an annotator for feedback on an inferred classification rule; 3) consideration of complex cost schemes regarding annotations and misclassification. This survey provides an overview of these AL strategies and refers to them as real-world AL. Therefore, we introduce a general real-world AL strategy as part of a learning cycle and use its elements, e.g., the query and annotator selection algorithm, to categorize about 60 real-world AL strategies. Finally, we outline possible directions for future research in the field of AL.}, author = {Herde, Marek and Huseljic, Denis and Sick, Bernhard and Calma, Adrian}, journal = {IEEE Access}, keywords = {classification}, pages = {166970–166989}, publisher = {IEEE}, title = {A Survey on Cost Types, Interaction Schemes, and Annotator Performance Models in Selection Algorithms for Active Learning in Classification}, volume = 9, year = 2021 }
%0 Journal Article %1 herde2021survey %A Herde, Marek %A Huseljic, Denis %A Sick, Bernhard %A Calma, Adrian %D 2021 %I IEEE %J IEEE Access %P 166970--166989 %R 10.1109/ACCESS.2021.3135514 %T A Survey on Cost Types, Interaction Schemes, and Annotator Performance Models in Selection Algorithms for Active Learning in Classification %U https://ieeexplore.ieee.org/abstract/document/9650877 %V 9 %X Pool-based active learning (AL) aims to optimize the annotation process (i.e., labeling) as the acquisition of annotations is often time-consuming and therefore expensive. For this purpose, an AL strategy queries annotations intelligently from annotators to train a high-performance classification model at a low annotation cost. Traditional AL strategies operate in an idealized framework. They assume a single, omniscient annotator who never gets tired and charges uniformly regardless of query difficulty. However, in real-world applications, we often face human annotators, e.g., crowd or in-house workers, who make annotation mistakes and can be reluctant to respond if tired or faced with complex queries. Recently, many novel AL strategies have been proposed to address these issues. They differ in at least one of the following three central aspects from traditional AL: 1) modeling of (multiple) human annotators whose performances can be affected by various factors, such as missing expertise; 2) generalization of the interaction with human annotators through different query and annotation types, such as asking an annotator for feedback on an inferred classification rule; 3) consideration of complex cost schemes regarding annotations and misclassification. This survey provides an overview of these AL strategies and refers to them as real-world AL. Therefore, we introduce a general real-world AL strategy as part of a learning cycle and use its elements, e.g., the query and annotator selection algorithm, to categorize about 60 real-world AL strategies. Finally, we outline possible directions for future research in the field of AL.
A Concept for Highly Automated Pre-Labeling via Cross-Domain Label Transfer for Perception in Autonomous Driving.Bieshaar, Maarten; Herde, Marek; Huselijc, Denis; Sick, Bernhard. In Workshop on Interactive Adaptive Learning (IAL), ECML PKDD. 2021.
- [ BibTeX ]
- [ EndNote ]
@inproceedings{bieshaar2021concept, author = {Bieshaar, Maarten and Herde, Marek and Huselijc, Denis and Sick, Bernhard}, booktitle = {Workshop on Interactive Adaptive Learning (IAL), ECML PKDD}, keywords = {itegpub}, note = {(accepted)}, title = {A Concept for Highly Automated Pre-Labeling via Cross-Domain Label Transfer for Perception in Autonomous Driving}, year = 2021 }
%0 Conference Paper %1 bieshaar2021concept %A Bieshaar, Maarten %A Herde, Marek %A Huselijc, Denis %A Sick, Bernhard %B Workshop on Interactive Adaptive Learning (IAL), ECML PKDD %D 2021 %T A Concept for Highly Automated Pre-Labeling via Cross-Domain Label Transfer for Perception in Autonomous Driving

2020[ to top ]

Toward Optimal Probabilistic Active Learning Using a Bayesian Approach.Kottke, Daniel; Herde, Marek; Sandrock, Christoph; Huseljic, Denis; Krempl, Georg; Sick, Bernhard. In arXiv e-prints, bl arXiv:2006.01732. 2020.
- [ BibTeX ]
- [ EndNote ]
@article{kottke2020optimal, author = {Kottke, Daniel and Herde, Marek and Sandrock, Christoph and Huseljic, Denis and Krempl, Georg and Sick, Bernhard}, journal = {arXiv e-prints}, keywords = {Machine}, pages = {arXiv:2006.01732}, title = {{Toward Optimal Probabilistic Active Learning Using a Bayesian Approach}}, year = 2020 }
%0 Journal Article %1 kottke2020optimal %A Kottke, Daniel %A Herde, Marek %A Sandrock, Christoph %A Huseljic, Denis %A Krempl, Georg %A Sick, Bernhard %D 2020 %J arXiv e-prints %P arXiv:2006.01732 %T Toward Optimal Probabilistic Active Learning Using a Bayesian Approach

2019[ to top ]

Collaborative Interactive Learning -- A clarification of terms and a differentiation from other research fields.Hanika, Tom; Herde, Marek; Kuhn, Jochen; Leimeister, Jan Marco; Lukowicz, Paul; Oeste-Reiß, Sarah; Schmidt, Albrecht; Sick, Bernhard; Stumme, Gerd; Tomforde, Sven; Zweig, Katharina Anna. In arXiv e-prints, bl arXiv:1905.07264. 2019.
- [ BibTeX ]
- [ EndNote ]
@article{hanika2019collaborative, author = {{Hanika}, Tom and {Herde}, Marek and {Kuhn}, Jochen and {Leimeister}, Jan Marco and {Lukowicz}, Paul and {Oeste-Rei{\ss}}, Sarah and {Schmidt}, Albrecht and {Sick}, Bernhard and {Stumme}, Gerd and {Tomforde}, Sven and {Zweig}, Katharina Anna}, journal = {arXiv e-prints}, keywords = {Machine}, pages = {arXiv:1905.07264}, title = {{Collaborative Interactive Learning – A clarification of terms and a differentiation from other research fields}}, year = 2019 }
%0 Journal Article %1 hanika2019collaborative %A Hanika, Tom %A Herde, Marek %A Kuhn, Jochen %A Leimeister, Jan Marco %A Lukowicz, Paul %A Oeste-Reiß, Sarah %A Schmidt, Albrecht %A Sick, Bernhard %A Stumme, Gerd %A Tomforde, Sven %A Zweig, Katharina Anna %D 2019 %J arXiv e-prints %P arXiv:1905.07264 %T Collaborative Interactive Learning -- A clarification of terms and a differentiation from other research fields

2018[ to top ]

Automated Active Learning with a Robot.Scharei, Kristina; Herde, Marek; Bieshaar, Maarten; Calma, Adrian; Kottke, Daniel; Sick, Bernhard. In Archives of Data Science, Series A (Online First), 5(1), bl 16. KIT, 2018.
- [ BibTeX ]
- [ EndNote ]
@article{scharei2018automated, author = {Scharei, Kristina and Herde, Marek and Bieshaar, Maarten and Calma, Adrian and Kottke, Daniel and Sick, Bernhard}, journal = {Archives of Data Science, Series A (Online First)}, keywords = {imported}, number = 1, pages = 16, publisher = {KIT}, title = {Automated Active Learning with a Robot}, volume = 5, year = 2018 }
%0 Journal Article %1 scharei2018automated %A Scharei, Kristina %A Herde, Marek %A Bieshaar, Maarten %A Calma, Adrian %A Kottke, Daniel %A Sick, Bernhard %D 2018 %I KIT %J Archives of Data Science, Series A (Online First) %N 1 %P 16 %R 10.5445/KSP/1000087327/16 %T Automated Active Learning with a Robot %V 5
Active Sorting -- An Efficient Training of a Sorting Robot with Active Learning Techniques.Herde, Marek; Kottke, Daniel; Calma, Adrian; Bieshaar, Maarten; Deist, Stephan; Sick, Bernhard. In International Joint Conference on Neural Networks (IJCNN). IEEE, Rio de Janiero, Brazil, 2018.
- [ BibTeX ]
- [ EndNote ]
@inproceedings{herde2018active, address = {Rio de Janiero, Brazil}, author = {Herde, Marek and Kottke, Daniel and Calma, Adrian and Bieshaar, Maarten and Deist, Stephan and Sick, Bernhard}, booktitle = {International Joint Conference on Neural Networks (IJCNN)}, keywords = {imported}, publisher = {IEEE}, title = {Active Sorting – An Efficient Training of a Sorting Robot with Active Learning Techniques}, year = 2018 }
%0 Conference Paper %1 herde2018active %A Herde, Marek %A Kottke, Daniel %A Calma, Adrian %A Bieshaar, Maarten %A Deist, Stephan %A Sick, Bernhard %B International Joint Conference on Neural Networks (IJCNN) %C Rio de Janiero, Brazil %D 2018 %I IEEE %R 10.1109/ijcnn.2018.8489161 %T Active Sorting -- An Efficient Training of a Sorting Robot with Active Learning Techniques