M. Sc. Denis Huseljic

Collaborative Interactive Learning (CIL)

Telefon: +49 561 804-6199
Fax: +49 561 804-6022
E-Mail: dhuseljic[at]uni-kassel[dot]de

Publikationen

[ 2023 ]
[ 2022 ]
[ 2021 ]
[ 2020 ]
[ 2019 ]
[ 2018 ]
[ 2017 ]

2023[ to top ]

Who knows best? A Case Study on Intelligent Crowdworker Selection via Deep Learning. Herde, Marek; Huseljic, Denis; Sick, Bernhard; Bretschneider, Ulrich; Oeste-Reiß, Sarah. In Workshop on Interactive Adapative Learning (IAL), ECML PKDD, bll 14–18. 2023.
- [ Abstract ]
- [ BibTeX ]
- [ EndNote ]
- [ URL ]
Crowdworking is a popular approach for annotating large amounts of data to train deep neural networks. However, parts of the annotations are often erroneous. In a case study, we demonstrate how an intelligent crowdworker selection via deep learning reduces the number of erroneous annotations and, thus, the annotation costs of obtaining reliable data for training deep neural networks.
@inproceedings{herde2023who, abstract = {Crowdworking is a popular approach for annotating large amounts of data to train deep neural networks. However, parts of the annotations are often erroneous. In a case study, we demonstrate how an intelligent crowdworker selection via deep learning reduces the number of erroneous annotations and, thus, the annotation costs of obtaining reliable data for training deep neural networks.}, author = {Herde, Marek and Huseljic, Denis and Sick, Bernhard and Bretschneider, Ulrich and Oeste-Reiß, Sarah}, booktitle = {Workshop on Interactive Adapative Learning (IAL), ECML PKDD}, keywords = {itegpub}, pages = {14–18}, title = {Who knows best? A Case Study on Intelligent Crowdworker Selection via Deep Learning}, year = 2023 }
%0 Conference Paper %1 herde2023who %A Herde, Marek %A Huseljic, Denis %A Sick, Bernhard %A Bretschneider, Ulrich %A Oeste-Reiß, Sarah %B Workshop on Interactive Adapative Learning (IAL), ECML PKDD %D 2023 %P 14--18 %T Who knows best? A Case Study on Intelligent Crowdworker Selection via Deep Learning %U https://ceur-ws.org/Vol-3470/paper3.pdf %X Crowdworking is a popular approach for annotating large amounts of data to train deep neural networks. However, parts of the annotations are often erroneous. In a case study, we demonstrate how an intelligent crowdworker selection via deep learning reduces the number of erroneous annotations and, thus, the annotation costs of obtaining reliable data for training deep neural networks.
Role of Hyperparameters in Deep Active Learning. Huseljic, Denis; Herde, Marek; Hahn, Paul; Sick, Bernhard. In Workshop on Interactive Adaptive Learning (IAL), ECML PKDD, bll 19–24. 2023.
- [ Abstract ]
- [ BibTeX ]
- [ EndNote ]
- [ URL ]
Despite considerable research on pool-based deep active learning (DAL), deploying DAL into real applications still has several challenges. A frequently neglected aspect is the choice of training hyperparameters (HPs), such as the learning rate. Since these HPs determine how the deep neural network learns in each cycle, they must be chosen carefully. In this article, we analyze the role of HPs in DAL. We find that optimizing HPs reduces the performance gap between DAL strategies. Conversely, we highlight challenges in finding optimal HPs when using datasets selected via DAL strategies.
@inproceedings{huseljic2023role, abstract = {Despite considerable research on pool-based deep active learning (DAL), deploying DAL into real applications still has several challenges. A frequently neglected aspect is the choice of training hyperparameters (HPs), such as the learning rate. Since these HPs determine how the deep neural network learns in each cycle, they must be chosen carefully. In this article, we analyze the role of HPs in DAL. We find that optimizing HPs reduces the performance gap between DAL strategies. Conversely, we highlight challenges in finding optimal HPs when using datasets selected via DAL strategies.}, author = {Huseljic, Denis and Herde, Marek and Hahn, Paul and Sick, Bernhard}, booktitle = {Workshop on Interactive Adaptive Learning (IAL), ECML PKDD}, keywords = {itegpub}, pages = {19–24}, title = {Role of Hyperparameters in Deep Active Learning}, year = 2023 }
%0 Conference Paper %1 huseljic2023role %A Huseljic, Denis %A Herde, Marek %A Hahn, Paul %A Sick, Bernhard %B Workshop on Interactive Adaptive Learning (IAL), ECML PKDD %D 2023 %P 19--24 %T Role of Hyperparameters in Deep Active Learning %U https://ceur-ws.org/Vol-3470/paper4.pdf %X Despite considerable research on pool-based deep active learning (DAL), deploying DAL into real applications still has several challenges. A frequently neglected aspect is the choice of training hyperparameters (HPs), such as the learning rate. Since these HPs determine how the deep neural network learns in each cycle, they must be chosen carefully. In this article, we analyze the role of HPs in DAL. We find that optimizing HPs reduces the performance gap between DAL strategies. Conversely, we highlight challenges in finding optimal HPs when using datasets selected via DAL strategies.
Multi-annotator Deep Learning: A Probabilistic Framework for Classification. Herde, Marek; Huseljic, Denis; Sick, Bernhard. In Transactions on Machine Learning Research. 2023.
- [ Abstract ]
- [ BibTeX ]
- [ EndNote ]
- [ URL ]
Solving complex classification tasks using deep neural networks typically requires large amounts of annotated data. However, corresponding class labels are noisy when provided by error-prone annotators, e.g., crowdworkers. Training standard deep neural networks leads to subpar performances in such multi-annotator supervised learning settings. We address this issue by presenting a probabilistic training framework named multi-annotator deep learning (MaDL). A downstream ground truth and an annotator performance model are jointly trained in an end-to-end learning approach. The ground truth model learns to predict instances' true class labels, while the annotator performance model infers probabilistic estimates of annotators' performances. A modular network architecture enables us to make varying assumptions regarding annotators' performances, e.g., an optional class or instance dependency. Further, we learn annotator embeddings to estimate annotators' densities within a latent space as proxies of their potentially correlated annotations. Together with a weighted loss function, we improve the learning from correlated annotation patterns. In a comprehensive evaluation, we examine three research questions about multi-annotator supervised learning. Our findings show MaDL's state-of-the-art performance and robustness against many correlated, spamming annotators.
@article{herde2023multi, abstract = {Solving complex classification tasks using deep neural networks typically requires large amounts of annotated data. However, corresponding class labels are noisy when provided by error-prone annotators, e.g., crowdworkers. Training standard deep neural networks leads to subpar performances in such multi-annotator supervised learning settings. We address this issue by presenting a probabilistic training framework named multi-annotator deep learning (MaDL). A downstream ground truth and an annotator performance model are jointly trained in an end-to-end learning approach. The ground truth model learns to predict instances' true class labels, while the annotator performance model infers probabilistic estimates of annotators' performances. A modular network architecture enables us to make varying assumptions regarding annotators' performances, e.g., an optional class or instance dependency. Further, we learn annotator embeddings to estimate annotators' densities within a latent space as proxies of their potentially correlated annotations. Together with a weighted loss function, we improve the learning from correlated annotation patterns. In a comprehensive evaluation, we examine three research questions about multi-annotator supervised learning. Our findings show MaDL's state-of-the-art performance and robustness against many correlated, spamming annotators.}, author = {Herde, Marek and Huseljic, Denis and Sick, Bernhard}, journal = {Transactions on Machine Learning Research}, keywords = {itegpub}, title = {Multi-annotator Deep Learning: A Probabilistic Framework for Classification}, year = 2023 }
%0 Journal Article %1 herde2023multi %A Herde, Marek %A Huseljic, Denis %A Sick, Bernhard %D 2023 %J Transactions on Machine Learning Research %T Multi-annotator Deep Learning: A Probabilistic Framework for Classification %U https://openreview.net/forum?id=MgdoxzImlK %X Solving complex classification tasks using deep neural networks typically requires large amounts of annotated data. However, corresponding class labels are noisy when provided by error-prone annotators, e.g., crowdworkers. Training standard deep neural networks leads to subpar performances in such multi-annotator supervised learning settings. We address this issue by presenting a probabilistic training framework named multi-annotator deep learning (MaDL). A downstream ground truth and an annotator performance model are jointly trained in an end-to-end learning approach. The ground truth model learns to predict instances' true class labels, while the annotator performance model infers probabilistic estimates of annotators' performances. A modular network architecture enables us to make varying assumptions regarding annotators' performances, e.g., an optional class or instance dependency. Further, we learn annotator embeddings to estimate annotators' densities within a latent space as proxies of their potentially correlated annotations. Together with a weighted loss function, we improve the learning from correlated annotation patterns. In a comprehensive evaluation, we examine three research questions about multi-annotator supervised learning. Our findings show MaDL's state-of-the-art performance and robustness against many correlated, spamming annotators.
ActiveGLAE: A Benchmark for Deep Active Learning with Transformers. Rauch, Lukas; Aßenmacher, Matthias; Huseljic, Denis; Wirth, Moritz; Bischl, Bernd; Sick, Bernhard. In European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECMLPKDD), bll 55–74. Springer, 2023.
- [ Abstract ]
- [ BibTeX ]
- [ EndNote ]
- [ URL ]
Deep active learning (DAL) seeks to reduce annotation costs by enabling the model to actively query instance annotations from which it expects to learn the most. Despite extensive research, there is cur- rently no standardized evaluation protocol for transformer-based lan- guage models in the field of DAL. Diverse experimental settings lead to difficulties in comparing research and deriving recommendations for prac- titioners. To tackle this challenge, we propose the ActiveGLAE bench- mark, a comprehensive collection of data sets and evaluation guidelines for assessing DAL. Our benchmark aims to facilitate and streamline the evaluation process of novel DAL strategies. Additionally, we provide an extensive overview of current practice in DAL with transformer-based language models. We identify three key challenges - data set selection, model training, and DAL settings - that pose difficulties in comparing query strategies. We establish baseline results through an extensive set of experiments as a reference point for evaluating future work. Based on our findings, we provide guidelines for researchers and practitioners.
@inproceedings{rauch2023activeglae, abstract = {Deep active learning (DAL) seeks to reduce annotation costs by enabling the model to actively query instance annotations from which it expects to learn the most. Despite extensive research, there is cur- rently no standardized evaluation protocol for transformer-based lan- guage models in the field of DAL. Diverse experimental settings lead to difficulties in comparing research and deriving recommendations for prac- titioners. To tackle this challenge, we propose the ActiveGLAE bench- mark, a comprehensive collection of data sets and evaluation guidelines for assessing DAL. Our benchmark aims to facilitate and streamline the evaluation process of novel DAL strategies. Additionally, we provide an extensive overview of current practice in DAL with transformer-based language models. We identify three key challenges - data set selection, model training, and DAL settings - that pose difficulties in comparing query strategies. We establish baseline results through an extensive set of experiments as a reference point for evaluating future work. Based on our findings, we provide guidelines for researchers and practitioners.}, author = {Rauch, Lukas and Aßenmacher, Matthias and Huseljic, Denis and Wirth, Moritz and Bischl, Bernd and Sick, Bernhard}, booktitle = {European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECMLPKDD)}, keywords = {itegpub}, pages = {55–74}, publisher = {Springer}, title = {ActiveGLAE: A Benchmark for Deep Active Learning with Transformers}, year = 2023 }
%0 Conference Paper %1 rauch2023activeglae %A Rauch, Lukas %A Aßenmacher, Matthias %A Huseljic, Denis %A Wirth, Moritz %A Bischl, Bernd %A Sick, Bernhard %B European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECMLPKDD) %D 2023 %I Springer %P 55--74 %R 10.1007/978-3-031-43412-9_4 %T ActiveGLAE: A Benchmark for Deep Active Learning with Transformers %U https://link.springer.com/chapter/10.1007/978-3-031-43412-9_4 %X Deep active learning (DAL) seeks to reduce annotation costs by enabling the model to actively query instance annotations from which it expects to learn the most. Despite extensive research, there is cur- rently no standardized evaluation protocol for transformer-based lan- guage models in the field of DAL. Diverse experimental settings lead to difficulties in comparing research and deriving recommendations for prac- titioners. To tackle this challenge, we propose the ActiveGLAE bench- mark, a comprehensive collection of data sets and evaluation guidelines for assessing DAL. Our benchmark aims to facilitate and streamline the evaluation process of novel DAL strategies. Additionally, we provide an extensive overview of current practice in DAL with transformer-based language models. We identify three key challenges - data set selection, model training, and DAL settings - that pose difficulties in comparing query strategies. We establish baseline results through an extensive set of experiments as a reference point for evaluating future work. Based on our findings, we provide guidelines for researchers and practitioners.
Active Learning with Fast Model Updates and Class-Balanced Selection for Imbalanced Datasets. Huang, Zhixin; He, Yujiang; Herde, Marek; Huseljic, Denis; Sick, Bernhard. In Workshop on Interactive Adapative Learning (IAL), ECML PKDD, bll 28–45. 2023.
- [ Abstract ]
- [ BibTeX ]
- [ EndNote ]
- [ URL ]
Active learning strategies aim to minimize the number of queried samples for model training. However, two challenges in pool-based deep active learning on imbalanced datasets are observed in experiments: (1) the declining performance of active learning strategies due to imbalanced class distribution; (2)~the lack of sample diversity in acquisition batches due to the absence of timely model updates. This paper proposes the AL-FaMoUS, a general solution combining fast model updates and class-balanced minibatch selection to the active learning process. Furthermore, a simplification of the AL-FaMoUS, which selects one single sample in each acquisition minibatch, is experimentally evaluated on four image and three time-series imbalanced datasets. The results demonstrate that the implemented AL-FaMoUS outperforms the other adopted AL strategies, including uncertainty sampling and BALD solely combined with either the fast model update or the class balance selection strategy, in terms of accuracy and Macro F1 score.
@inproceedings{huang2023active, abstract = {Active learning strategies aim to minimize the number of queried samples for model training. However, two challenges in pool-based deep active learning on imbalanced datasets are observed in experiments: (1) the declining performance of active learning strategies due to imbalanced class distribution; (2) the lack of sample diversity in acquisition batches due to the absence of timely model updates. This paper proposes the AL-FaMoUS, a general solution combining fast model updates and class-balanced minibatch selection to the active learning process. Furthermore, a simplification of the AL-FaMoUS, which selects one single sample in each acquisition minibatch, is experimentally evaluated on four image and three time-series imbalanced datasets. The results demonstrate that the implemented AL-FaMoUS outperforms the other adopted AL strategies, including uncertainty sampling and BALD solely combined with either the fast model update or the class balance selection strategy, in terms of accuracy and Macro F1 score.}, author = {Huang, Zhixin and He, Yujiang and Herde, Marek and Huseljic, Denis and Sick, Bernhard}, booktitle = {Workshop on Interactive Adapative Learning (IAL), ECML PKDD}, keywords = {itegpub}, pages = {28–45}, title = {Active Learning with Fast Model Updates and Class-Balanced Selection for Imbalanced Datasets}, year = 2023 }
%0 Conference Paper %1 huang2023active %A Huang, Zhixin %A He, Yujiang %A Herde, Marek %A Huseljic, Denis %A Sick, Bernhard %B Workshop on Interactive Adapative Learning (IAL), ECML PKDD %D 2023 %P 28--45 %T Active Learning with Fast Model Updates and Class-Balanced Selection for Imbalanced Datasets %U https://ceur-ws.org/Vol-3470/paper5.pdf %X Active learning strategies aim to minimize the number of queried samples for model training. However, two challenges in pool-based deep active learning on imbalanced datasets are observed in experiments: (1) the declining performance of active learning strategies due to imbalanced class distribution; (2)~the lack of sample diversity in acquisition batches due to the absence of timely model updates. This paper proposes the AL-FaMoUS, a general solution combining fast model updates and class-balanced minibatch selection to the active learning process. Furthermore, a simplification of the AL-FaMoUS, which selects one single sample in each acquisition minibatch, is experimentally evaluated on four image and three time-series imbalanced datasets. The results demonstrate that the implemented AL-FaMoUS outperforms the other adopted AL strategies, including uncertainty sampling and BALD solely combined with either the fast model update or the class balance selection strategy, in terms of accuracy and Macro F1 score.
Active Label Refinement for Semantic Segmentation of Satellite Images. Pham, Minh Tuan; Wijesingha, Jayan; Kottke, Daniel; Herde, Marek; Huseljic, Denis; Sick, Bernhard; Wachendorf, Michael; Esch, Thomas. In arXiv e-prints, bl arXiv:2309.06159. 2023.
- [ Abstract ]
- [ BibTeX ]
- [ EndNote ]
- [ URL ]
Remote sensing through semantic segmentation of satellite images contributes to the understanding and utilisation of the earth's surface. For this purpose, semantic segmentation networks are typically trained on large sets of labelled satellite images. However, obtaining expert labels for these images is costly. Therefore, we propose to rely on a low-cost approach, e.g. crowdsourcing or pretrained networks, to label the images in the first step. Since these initial labels are partially erroneous, we use active learning strategies to cost-efficiently refine the labels in the second step. We evaluate the active learning strategies using satellite images of Bengaluru in India, labelled with land cover and land use labels. Our experimental results suggest that an active label refinement to improve the semantic segmentation network's performance is beneficial.
@article{pham2023active, abstract = {Remote sensing through semantic segmentation of satellite images contributes to the understanding and utilisation of the earth's surface. For this purpose, semantic segmentation networks are typically trained on large sets of labelled satellite images. However, obtaining expert labels for these images is costly. Therefore, we propose to rely on a low-cost approach, e.g. crowdsourcing or pretrained networks, to label the images in the first step. Since these initial labels are partially erroneous, we use active learning strategies to cost-efficiently refine the labels in the second step. We evaluate the active learning strategies using satellite images of Bengaluru in India, labelled with land cover and land use labels. Our experimental results suggest that an active label refinement to improve the semantic segmentation network's performance is beneficial.}, author = {Pham, Minh Tuan and Wijesingha, Jayan and Kottke, Daniel and Herde, Marek and Huseljic, Denis and Sick, Bernhard and Wachendorf, Michael and Esch, Thomas}, journal = {arXiv e-prints}, keywords = {itegpub}, pages = {arXiv:2309.06159}, title = {Active Label Refinement for Semantic Segmentation of Satellite Images}, year = 2023 }
%0 Journal Article %1 pham2023active %A Pham, Minh Tuan %A Wijesingha, Jayan %A Kottke, Daniel %A Herde, Marek %A Huseljic, Denis %A Sick, Bernhard %A Wachendorf, Michael %A Esch, Thomas %D 2023 %J arXiv e-prints %P arXiv:2309.06159 %T Active Label Refinement for Semantic Segmentation of Satellite Images %U https://arxiv.org/abs/2309.06159 %X Remote sensing through semantic segmentation of satellite images contributes to the understanding and utilisation of the earth's surface. For this purpose, semantic segmentation networks are typically trained on large sets of labelled satellite images. However, obtaining expert labels for these images is costly. Therefore, we propose to rely on a low-cost approach, e.g. crowdsourcing or pretrained networks, to label the images in the first step. Since these initial labels are partially erroneous, we use active learning strategies to cost-efficiently refine the labels in the second step. We evaluate the active learning strategies using satellite images of Bengaluru in India, labelled with land cover and land use labels. Our experimental results suggest that an active label refinement to improve the semantic segmentation network's performance is beneficial.

2022[ to top ]

Fast Bayesian Updates for Deep Learning with a Use Case in Active Learning. Herde, Marek; Huang, Zhixin; Huseljic, Denis; Kottke, Daniel; Vogt, Stephan; Sick, Bernhard. In arXiv e-prints, bl arXiv:2210.06112. 2022.
- [ Abstract ]
- [ BibTeX ]
- [ EndNote ]
- [ URL ]
Retraining deep neural networks when new data arrives is typically computationally expensive. Moreover, certain applications do not allow such costly retraining due to time or computational constraints. Fast Bayesian updates are a possible solution to this issue. Therefore, we propose a Bayesian update based on Monte-Carlo samples and a last-layer Laplace approximation for different Bayesian neural network types, i.e., Dropout, Ensemble, and Spectral Normalized Neural Gaussian Process (SNGP). In a large-scale evaluation study, we show that our updates combined with SNGP represent a fast and competitive alternative to costly retraining. As a use case, we combine the Bayesian updates for SNGP with different sequential query strategies to exemplarily demonstrate their improved selection performance in active learning.
@article{herde2022fast, abstract = {Retraining deep neural networks when new data arrives is typically computationally expensive. Moreover, certain applications do not allow such costly retraining due to time or computational constraints. Fast Bayesian updates are a possible solution to this issue. Therefore, we propose a Bayesian update based on Monte-Carlo samples and a last-layer Laplace approximation for different Bayesian neural network types, i.e., Dropout, Ensemble, and Spectral Normalized Neural Gaussian Process (SNGP). In a large-scale evaluation study, we show that our updates combined with SNGP represent a fast and competitive alternative to costly retraining. As a use case, we combine the Bayesian updates for SNGP with different sequential query strategies to exemplarily demonstrate their improved selection performance in active learning.}, author = {Herde, Marek and Huang, Zhixin and Huseljic, Denis and Kottke, Daniel and Vogt, Stephan and Sick, Bernhard}, journal = {arXiv e-prints}, keywords = {itegpub}, pages = {arXiv:2210.06112}, title = {Fast Bayesian Updates for Deep Learning with a Use Case in Active Learning}, year = 2022 }
%0 Journal Article %1 herde2022fast %A Herde, Marek %A Huang, Zhixin %A Huseljic, Denis %A Kottke, Daniel %A Vogt, Stephan %A Sick, Bernhard %D 2022 %J arXiv e-prints %P arXiv:2210.06112 %T Fast Bayesian Updates for Deep Learning with a Use Case in Active Learning %U https://arxiv.org/abs/2210.06112 %X Retraining deep neural networks when new data arrives is typically computationally expensive. Moreover, certain applications do not allow such costly retraining due to time or computational constraints. Fast Bayesian updates are a possible solution to this issue. Therefore, we propose a Bayesian update based on Monte-Carlo samples and a last-layer Laplace approximation for different Bayesian neural network types, i.e., Dropout, Ensemble, and Spectral Normalized Neural Gaussian Process (SNGP). In a large-scale evaluation study, we show that our updates combined with SNGP represent a fast and competitive alternative to costly retraining. As a use case, we combine the Bayesian updates for SNGP with different sequential query strategies to exemplarily demonstrate their improved selection performance in active learning.
Enhancing Active Learning with Weak Supervision and Transfer Learning by Leveraging Information and Knowledge Sources. Rauch, Lukas; Huseljic, Denis; Sick, Bernhard. In Workshop on Interactive Adaptive Learning (IAL), ECML PKDD, bll 27–42. 2022.
- [ Abstract ]
- [ BibTeX ]
- [ EndNote ]
- [ URL ]
One of the major limitations of deploying a machine learning model is the availability of labeled training data and the resulting expensive annotation process. Although active learning (AL) methods may reduce the annotation cost by actively selecting the most-useful instances, a costly human annotator usually provides the labels. Therefore, even with AL, we still consider the annotation process to be time-consuming and expensive. Besides human annotators, though, companies often have a vast amount of information and knowledge sources available that can generate low-cost labels (e.g., a black-box model) or improve the learning process (e.g., a pre-trained model). We present a novel approach that enhances AL with weak supervision (WS) and transfer learning (TL) to reduce the annotation cost by leveraging these sources. Specifically, we consider a black-box model like a rule-based system as an error-prone and weakly-supervised annotator that inexpensively provides labels. We estimate its performance with an annotator model to decide whether a human annotation is required. Additionally, we utilize unlabeled internal and external data by transferring knowledge from a pre-trained model to the AL cycle. We sequentially investigate the impact of WS and TL on annotation cost and model performance in an AL cycle through a use case. Our evaluation shows that our approach can reduce annotation cost by 51% while achieving nearly identical model performance compared to a traditional AL approach.
@inproceedings{rauch2022enhancing, abstract = {One of the major limitations of deploying a machine learning model is the availability of labeled training data and the resulting expensive annotation process. Although active learning (AL) methods may reduce the annotation cost by actively selecting the most-useful instances, a costly human annotator usually provides the labels. Therefore, even with AL, we still consider the annotation process to be time-consuming and expensive. Besides human annotators, though, companies often have a vast amount of information and knowledge sources available that can generate low-cost labels (e.g., a black-box model) or improve the learning process (e.g., a pre-trained model). We present a novel approach that enhances AL with weak supervision (WS) and transfer learning (TL) to reduce the annotation cost by leveraging these sources. Specifically, we consider a black-box model like a rule-based system as an error-prone and weakly-supervised annotator that inexpensively provides labels. We estimate its performance with an annotator model to decide whether a human annotation is required. Additionally, we utilize unlabeled internal and external data by transferring knowledge from a pre-trained model to the AL cycle. We sequentially investigate the impact of WS and TL on annotation cost and model performance in an AL cycle through a use case. Our evaluation shows that our approach can reduce annotation cost by 51% while achieving nearly identical model performance compared to a traditional AL approach.}, author = {Rauch, Lukas and Huseljic, Denis and Sick, Bernhard}, booktitle = {Workshop on Interactive Adaptive Learning (IAL), ECML PKDD}, keywords = {itegpub}, pages = {27–42}, title = {Enhancing Active Learning with Weak Supervision and Transfer Learning by Leveraging Information and Knowledge Sources}, year = 2022 }
%0 Conference Paper %1 rauch2022enhancing %A Rauch, Lukas %A Huseljic, Denis %A Sick, Bernhard %B Workshop on Interactive Adaptive Learning (IAL), ECML PKDD %D 2022 %P 27--42 %T Enhancing Active Learning with Weak Supervision and Transfer Learning by Leveraging Information and Knowledge Sources %U http://ceur-ws.org/Vol-3259/ialatecml_paper3.pdf %X One of the major limitations of deploying a machine learning model is the availability of labeled training data and the resulting expensive annotation process. Although active learning (AL) methods may reduce the annotation cost by actively selecting the most-useful instances, a costly human annotator usually provides the labels. Therefore, even with AL, we still consider the annotation process to be time-consuming and expensive. Besides human annotators, though, companies often have a vast amount of information and knowledge sources available that can generate low-cost labels (e.g., a black-box model) or improve the learning process (e.g., a pre-trained model). We present a novel approach that enhances AL with weak supervision (WS) and transfer learning (TL) to reduce the annotation cost by leveraging these sources. Specifically, we consider a black-box model like a rule-based system as an error-prone and weakly-supervised annotator that inexpensively provides labels. We estimate its performance with an annotator model to decide whether a human annotation is required. Additionally, we utilize unlabeled internal and external data by transferring knowledge from a pre-trained model to the AL cycle. We sequentially investigate the impact of WS and TL on annotation cost and model performance in an AL cycle through a use case. Our evaluation shows that our approach can reduce annotation cost by 51% while achieving nearly identical model performance compared to a traditional AL approach.
A Review of Uncertainty Calibration in Pretrained Object Detectors. Huseljic, Denis; Herde, Marek; Muejde, Mehmet; Sick, Bernhard. In arXiv e-prints, bl arXiv:2210.02935. 2022.
- [ Abstract ]
- [ BibTeX ]
- [ EndNote ]
- [ URL ]
In the field of deep learning based computer vision, the development of deep object detection has led to unique paradigms (e.g., two-stage or set-based) and architectures (e.g., Faster-RCNN or DETR) which enable outstanding performance on challenging benchmark datasets. Despite this, the trained object detectors typically do not reliably assess uncertainty regarding their own knowledge, and the quality of their probabilistic predictions is usually poor. As these are often used to make subsequent decisions, such inaccurate probabilistic predictions must be avoided. In this work, we investigate the uncertainty calibration properties of different pretrained object detection architectures in a multi-class setting. We propose a framework to ensure a fair, unbiased, and repeatable evaluation and conduct detailed analyses assessing the calibration under distributional changes (e.g., distributional shift and application to out-of-distribution data). Furthermore, by investigating the influence of different detector paradigms, post-processing steps, and suitable choices of metrics, we deliver novel insights into why poor detector calibration emerges. Based on these insights, we are able to improve the calibration of a detector by simply finetuning its last layer.
@article{huseljic2022review, abstract = {In the field of deep learning based computer vision, the development of deep object detection has led to unique paradigms (e.g., two-stage or set-based) and architectures (e.g., Faster-RCNN or DETR) which enable outstanding performance on challenging benchmark datasets. Despite this, the trained object detectors typically do not reliably assess uncertainty regarding their own knowledge, and the quality of their probabilistic predictions is usually poor. As these are often used to make subsequent decisions, such inaccurate probabilistic predictions must be avoided. In this work, we investigate the uncertainty calibration properties of different pretrained object detection architectures in a multi-class setting. We propose a framework to ensure a fair, unbiased, and repeatable evaluation and conduct detailed analyses assessing the calibration under distributional changes (e.g., distributional shift and application to out-of-distribution data). Furthermore, by investigating the influence of different detector paradigms, post-processing steps, and suitable choices of metrics, we deliver novel insights into why poor detector calibration emerges. Based on these insights, we are able to improve the calibration of a detector by simply finetuning its last layer.}, author = {Huseljic, Denis and Herde, Marek and Muejde, Mehmet and Sick, Bernhard}, journal = {arXiv e-prints}, keywords = {itegpub}, pages = {arXiv:2210.02935}, title = {A Review of Uncertainty Calibration in Pretrained Object Detectors}, year = 2022 }
%0 Journal Article %1 huseljic2022review %A Huseljic, Denis %A Herde, Marek %A Muejde, Mehmet %A Sick, Bernhard %D 2022 %J arXiv e-prints %P arXiv:2210.02935 %T A Review of Uncertainty Calibration in Pretrained Object Detectors %U https://arxiv.org/abs/2210.02935 %X In the field of deep learning based computer vision, the development of deep object detection has led to unique paradigms (e.g., two-stage or set-based) and architectures (e.g., Faster-RCNN or DETR) which enable outstanding performance on challenging benchmark datasets. Despite this, the trained object detectors typically do not reliably assess uncertainty regarding their own knowledge, and the quality of their probabilistic predictions is usually poor. As these are often used to make subsequent decisions, such inaccurate probabilistic predictions must be avoided. In this work, we investigate the uncertainty calibration properties of different pretrained object detection architectures in a multi-class setting. We propose a framework to ensure a fair, unbiased, and repeatable evaluation and conduct detailed analyses assessing the calibration under distributional changes (e.g., distributional shift and application to out-of-distribution data). Furthermore, by investigating the influence of different detector paradigms, post-processing steps, and suitable choices of metrics, we deliver novel insights into why poor detector calibration emerges. Based on these insights, we are able to improve the calibration of a detector by simply finetuning its last layer.
A Concept for Automated Polarized Web Content Annotation based on Multimodal Active Learning. Herde, Marek; Huseljic, Denis; Mitrovic, Jelena; Granitzer, Michael; Sick, Bernhard. In Workshop on Interactive Adaptive Learning (IAL), ECML PKDD, bll 1–6. 2022.
- [ Abstract ]
- [ BibTeX ]
- [ EndNote ]
- [ URL ]
Active learning (AL) techniques hardly cope with complex annotations tasks, where, for example, annotations might express relationships across data modalities. As a use case, we consider the task of automatically detecting and reporting multimodal, polarized web content (PWC). Samples of this content type emerge dynamically, covering a broad spectrum of topics. Thus, training machine learning systems for detecting PWC is challenging, particularly if it needs to be done with minimum annotation cost. In this article, we propose the concept of multimodal AL for complex annotations in the context of PWC detection and formulate the resulting challenges as questions for future research.
@inproceedings{herde2022concept, abstract = {Active learning (AL) techniques hardly cope with complex annotations tasks, where, for example, annotations might express relationships across data modalities. As a use case, we consider the task of automatically detecting and reporting multimodal, polarized web content (PWC). Samples of this content type emerge dynamically, covering a broad spectrum of topics. Thus, training machine learning systems for detecting PWC is challenging, particularly if it needs to be done with minimum annotation cost. In this article, we propose the concept of multimodal AL for complex annotations in the context of PWC detection and formulate the resulting challenges as questions for future research.}, author = {Herde, Marek and Huseljic, Denis and Mitrovic, Jelena and Granitzer, Michael and Sick, Bernhard}, booktitle = {Workshop on Interactive Adaptive Learning (IAL), ECML PKDD}, keywords = {itegpub}, pages = {1–6}, title = {A Concept for Automated Polarized Web Content Annotation based on Multimodal Active Learning}, year = 2022 }
%0 Conference Paper %1 herde2022concept %A Herde, Marek %A Huseljic, Denis %A Mitrovic, Jelena %A Granitzer, Michael %A Sick, Bernhard %B Workshop on Interactive Adaptive Learning (IAL), ECML PKDD %D 2022 %P 1--6 %T A Concept for Automated Polarized Web Content Annotation based on Multimodal Active Learning %U http://ceur-ws.org/Vol-3259/ialatecml_paper1.pdf %X Active learning (AL) techniques hardly cope with complex annotations tasks, where, for example, annotations might express relationships across data modalities. As a use case, we consider the task of automatically detecting and reporting multimodal, polarized web content (PWC). Samples of this content type emerge dynamically, covering a broad spectrum of topics. Thus, training machine learning systems for detecting PWC is challenging, particularly if it needs to be done with minimum annotation cost. In this article, we propose the concept of multimodal AL for complex annotations in the context of PWC detection and formulate the resulting challenges as questions for future research.

2021[ to top ]

Toward optimal probabilistic active learning using a Bayesian approach. Kottke, Daniel; Herde, Marek; Sandrock, Christoph; Huseljic, Denis; Krempl, Georg; Sick, Bernhard. In Machine Learning, 110(6), bll 1199–1231. Springer, 2021.
- [ Abstract ]
- [ BibTeX ]
- [ EndNote ]
Gathering labeled data to train well-performing machine learning models is one of the critical challenges in many applications. Active learning aims at reducing the labeling costs by an efficient and effective allocation of costly labeling resources. In this article, we propose a decision-theoretic selection strategy that (1) directly optimizes the gain in misclassification error, and (2) uses a Bayesian approach by introducing a conjugate prior distribution to determine the class posterior to deal with uncertainties. By reformulating existing selection strategies within our proposed model, we can explain which aspects are not covered in current state-of-the-art and why this leads to the superior performance of our approach. Extensive experiments on a large variety of datasets and different kernels validate our claims.
@article{kottke2021optimal, abstract = {Gathering labeled data to train well-performing machine learning models is one of the critical challenges in many applications. Active learning aims at reducing the labeling costs by an efficient and effective allocation of costly labeling resources. In this article, we propose a decision-theoretic selection strategy that (1) directly optimizes the gain in misclassification error, and (2) uses a Bayesian approach by introducing a conjugate prior distribution to determine the class posterior to deal with uncertainties. By reformulating existing selection strategies within our proposed model, we can explain which aspects are not covered in current state-of-the-art and why this leads to the superior performance of our approach. Extensive experiments on a large variety of datasets and different kernels validate our claims.}, author = {Kottke, Daniel and Herde, Marek and Sandrock, Christoph and Huseljic, Denis and Krempl, Georg and Sick, Bernhard}, journal = {Machine Learning}, keywords = {itegpub}, number = 6, pages = {1199–1231}, publisher = {Springer}, title = {Toward optimal probabilistic active learning using a Bayesian approach}, volume = 110, year = 2021 }
%0 Journal Article %1 kottke2021optimal %A Kottke, Daniel %A Herde, Marek %A Sandrock, Christoph %A Huseljic, Denis %A Krempl, Georg %A Sick, Bernhard %D 2021 %I Springer %J Machine Learning %N 6 %P 1199--1231 %R 10.1007/s10994-021-05986-9 %T Toward optimal probabilistic active learning using a Bayesian approach %V 110 %X Gathering labeled data to train well-performing machine learning models is one of the critical challenges in many applications. Active learning aims at reducing the labeling costs by an efficient and effective allocation of costly labeling resources. In this article, we propose a decision-theoretic selection strategy that (1) directly optimizes the gain in misclassification error, and (2) uses a Bayesian approach by introducing a conjugate prior distribution to determine the class posterior to deal with uncertainties. By reformulating existing selection strategies within our proposed model, we can explain which aspects are not covered in current state-of-the-art and why this leads to the superior performance of our approach. Extensive experiments on a large variety of datasets and different kernels validate our claims.
Separation of Aleatoric and Epistemic Uncertainty in Deterministic Deep Neural Networks. Huseljic, Denis; Sick, Bernhard; Herde, Marek; Kottke, Daniel. In International Conference on Pattern Recognition (ICPR), bll 9172–9179. IEEE, 2021.
- [ BibTeX ]
- [ EndNote ]
@inproceedings{huseljic2021separation, author = {Huseljic, Denis and Sick, Bernhard and Herde, Marek and Kottke, Daniel}, booktitle = {International Conference on Pattern Recognition (ICPR)}, keywords = {itegpub}, pages = {9172–9179}, publisher = {IEEE}, title = {Separation of Aleatoric and Epistemic Uncertainty in Deterministic Deep Neural Networks}, year = 2021 }
%0 Conference Paper %1 huseljic2021separation %A Huseljic, Denis %A Sick, Bernhard %A Herde, Marek %A Kottke, Daniel %B International Conference on Pattern Recognition (ICPR) %D 2021 %I IEEE %P 9172--9179 %R 10.1109/icpr48806.2021.9412616 %T Separation of Aleatoric and Epistemic Uncertainty in Deterministic Deep Neural Networks
Out-of-distribution Detection and Generation using Soft Brownian Offset Sampling and Autoencoders. Möller, Felix; Botache, Diego; Huseljic, Denis; Heidecker, Florian; Bieshaar, Maarten; Sick, Bernhard. In Workshop on Safe Artificial Intelligence for Automated Driving (SAIAD), CVPR, bll 1–10. 2021.
- [ BibTeX ]
- [ EndNote ]
- [ URL ]
@inproceedings{moeller2021out, author = {Möller, Felix and Botache, Diego and Huseljic, Denis and Heidecker, Florian and Bieshaar, Maarten and Sick, Bernhard}, booktitle = {Workshop on Safe Artificial Intelligence for Automated Driving (SAIAD), CVPR}, keywords = {itegpub}, pages = {1–10}, title = {{Out-of-distribution Detection and Generation using Soft Brownian Offset Sampling and Autoencoders}}, year = 2021 }
%0 Conference Paper %1 moeller2021out %A Möller, Felix %A Botache, Diego %A Huseljic, Denis %A Heidecker, Florian %A Bieshaar, Maarten %A Sick, Bernhard %B Workshop on Safe Artificial Intelligence for Automated Driving (SAIAD), CVPR %D 2021 %P 1--10 %T Out-of-distribution Detection and Generation using Soft Brownian Offset Sampling and Autoencoders %U https://openaccess.thecvf.com/content/CVPR2021W/SAIAD/papers/Moller_Out-of-Distribution_Detection_and_Generation_Using_Soft_Brownian_Offset_Sampling_and_CVPRW_2021_paper.pdf
Multi-annotator Probabilistic Active Learning. Herde, Marek; Kottke, Daniel; Huseljic, Denis; Sick, Bernhard. In International Conference on Pattern Recognition (ICPR), bll 10281–10288. IEEE, 2021.
- [ BibTeX ]
- [ EndNote ]
@inproceedings{herde2021multi, author = {Herde, Marek and Kottke, Daniel and Huseljic, Denis and Sick, Bernhard}, booktitle = {International Conference on Pattern Recognition (ICPR)}, keywords = {itegpub}, pages = {10281–10288}, publisher = {IEEE}, title = {Multi-annotator Probabilistic Active Learning}, year = 2021 }
%0 Conference Paper %1 herde2021multi %A Herde, Marek %A Kottke, Daniel %A Huseljic, Denis %A Sick, Bernhard %B International Conference on Pattern Recognition (ICPR) %D 2021 %I IEEE %P 10281--10288 %R 10.1109/icpr48806.2021.9412298 %T Multi-annotator Probabilistic Active Learning
A Survey on Cost Types, Interaction Schemes, and Annotator Performance Models in Selection Algorithms for Active Learning in Classification. Herde, Marek; Huseljic, Denis; Sick, Bernhard; Calma, Adrian. In IEEE Access, 9, bll 166970–166989. IEEE, 2021.
- [ Abstract ]
- [ BibTeX ]
- [ EndNote ]
- [ URL ]
Pool-based active learning (AL) aims to optimize the annotation process (i.e., labeling) as the acquisition of annotations is often time-consuming and therefore expensive. For this purpose, an AL strategy queries annotations intelligently from annotators to train a high-performance classification model at a low annotation cost. Traditional AL strategies operate in an idealized framework. They assume a single, omniscient annotator who never gets tired and charges uniformly regardless of query difficulty. However, in real-world applications, we often face human annotators, e.g., crowd or in-house workers, who make annotation mistakes and can be reluctant to respond if tired or faced with complex queries. Recently, many novel AL strategies have been proposed to address these issues. They differ in at least one of the following three central aspects from traditional AL: 1) modeling of (multiple) human annotators whose performances can be affected by various factors, such as missing expertise; 2) generalization of the interaction with human annotators through different query and annotation types, such as asking an annotator for feedback on an inferred classification rule; 3) consideration of complex cost schemes regarding annotations and misclassification. This survey provides an overview of these AL strategies and refers to them as real-world AL. Therefore, we introduce a general real-world AL strategy as part of a learning cycle and use its elements, e.g., the query and annotator selection algorithm, to categorize about 60 real-world AL strategies. Finally, we outline possible directions for future research in the field of AL.
@article{herde2021survey, abstract = {Pool-based active learning (AL) aims to optimize the annotation process (i.e., labeling) as the acquisition of annotations is often time-consuming and therefore expensive. For this purpose, an AL strategy queries annotations intelligently from annotators to train a high-performance classification model at a low annotation cost. Traditional AL strategies operate in an idealized framework. They assume a single, omniscient annotator who never gets tired and charges uniformly regardless of query difficulty. However, in real-world applications, we often face human annotators, e.g., crowd or in-house workers, who make annotation mistakes and can be reluctant to respond if tired or faced with complex queries. Recently, many novel AL strategies have been proposed to address these issues. They differ in at least one of the following three central aspects from traditional AL: 1) modeling of (multiple) human annotators whose performances can be affected by various factors, such as missing expertise; 2) generalization of the interaction with human annotators through different query and annotation types, such as asking an annotator for feedback on an inferred classification rule; 3) consideration of complex cost schemes regarding annotations and misclassification. This survey provides an overview of these AL strategies and refers to them as real-world AL. Therefore, we introduce a general real-world AL strategy as part of a learning cycle and use its elements, e.g., the query and annotator selection algorithm, to categorize about 60 real-world AL strategies. Finally, we outline possible directions for future research in the field of AL.}, author = {Herde, Marek and Huseljic, Denis and Sick, Bernhard and Calma, Adrian}, journal = {IEEE Access}, keywords = {classification}, pages = {166970–166989}, publisher = {IEEE}, title = {A Survey on Cost Types, Interaction Schemes, and Annotator Performance Models in Selection Algorithms for Active Learning in Classification}, volume = 9, year = 2021 }
%0 Journal Article %1 herde2021survey %A Herde, Marek %A Huseljic, Denis %A Sick, Bernhard %A Calma, Adrian %D 2021 %I IEEE %J IEEE Access %P 166970--166989 %R 10.1109/ACCESS.2021.3135514 %T A Survey on Cost Types, Interaction Schemes, and Annotator Performance Models in Selection Algorithms for Active Learning in Classification %U https://ieeexplore.ieee.org/abstract/document/9650877 %V 9 %X Pool-based active learning (AL) aims to optimize the annotation process (i.e., labeling) as the acquisition of annotations is often time-consuming and therefore expensive. For this purpose, an AL strategy queries annotations intelligently from annotators to train a high-performance classification model at a low annotation cost. Traditional AL strategies operate in an idealized framework. They assume a single, omniscient annotator who never gets tired and charges uniformly regardless of query difficulty. However, in real-world applications, we often face human annotators, e.g., crowd or in-house workers, who make annotation mistakes and can be reluctant to respond if tired or faced with complex queries. Recently, many novel AL strategies have been proposed to address these issues. They differ in at least one of the following three central aspects from traditional AL: 1) modeling of (multiple) human annotators whose performances can be affected by various factors, such as missing expertise; 2) generalization of the interaction with human annotators through different query and annotation types, such as asking an annotator for feedback on an inferred classification rule; 3) consideration of complex cost schemes regarding annotations and misclassification. This survey provides an overview of these AL strategies and refers to them as real-world AL. Therefore, we introduce a general real-world AL strategy as part of a learning cycle and use its elements, e.g., the query and annotator selection algorithm, to categorize about 60 real-world AL strategies. Finally, we outline possible directions for future research in the field of AL.
A Concept for Highly Automated Pre-Labeling via Cross-Domain Label Transfer for Perception in Autonomous Driving. Bieshaar, Maarten; Herde, Marek; Huselijc, Denis; Sick, Bernhard. In Workshop on Interactive Adaptive Learning (IAL), ECML PKDD. 2021.
- [ BibTeX ]
- [ EndNote ]
@inproceedings{bieshaar2021concept, author = {Bieshaar, Maarten and Herde, Marek and Huselijc, Denis and Sick, Bernhard}, booktitle = {Workshop on Interactive Adaptive Learning (IAL), ECML PKDD}, keywords = {itegpub}, note = {(accepted)}, title = {A Concept for Highly Automated Pre-Labeling via Cross-Domain Label Transfer for Perception in Autonomous Driving}, year = 2021 }
%0 Conference Paper %1 bieshaar2021concept %A Bieshaar, Maarten %A Herde, Marek %A Huselijc, Denis %A Sick, Bernhard %B Workshop on Interactive Adaptive Learning (IAL), ECML PKDD %D 2021 %T A Concept for Highly Automated Pre-Labeling via Cross-Domain Label Transfer for Perception in Autonomous Driving

2020[ to top ]

Toward Optimal Probabilistic Active Learning Using a Bayesian Approach. Kottke, Daniel; Herde, Marek; Sandrock, Christoph; Huseljic, Denis; Krempl, Georg; Sick, Bernhard. In arXiv e-prints, bl arXiv:2006.01732. 2020.
- [ BibTeX ]
- [ EndNote ]
@article{kottke2020optimal, author = {Kottke, Daniel and Herde, Marek and Sandrock, Christoph and Huseljic, Denis and Krempl, Georg and Sick, Bernhard}, journal = {arXiv e-prints}, keywords = {Machine}, pages = {arXiv:2006.01732}, title = {{Toward Optimal Probabilistic Active Learning Using a Bayesian Approach}}, year = 2020 }
%0 Journal Article %1 kottke2020optimal %A Kottke, Daniel %A Herde, Marek %A Sandrock, Christoph %A Huseljic, Denis %A Krempl, Georg %A Sick, Bernhard %D 2020 %J arXiv e-prints %P arXiv:2006.01732 %T Toward Optimal Probabilistic Active Learning Using a Bayesian Approach

2019[ to top ]

Limitations of Assessing Active Learning Performance at Runtime. Kottke, Daniel; Schellinger, Jim; Huseljic, Denis; Sick, Bernhard. In arXiv e-prints, bl arXiv:1901.10338. 2019.
- [ BibTeX ]
- [ EndNote ]
@article{kottke2019limitations, author = {{Kottke}, Daniel and {Schellinger}, Jim and {Huseljic}, Denis and {Sick}, Bernhard}, journal = {arXiv e-prints}, keywords = {Machine}, pages = {arXiv:1901.10338}, title = {{Limitations of Assessing Active Learning Performance at Runtime}}, year = 2019 }
%0 Journal Article %1 kottke2019limitations %A Kottke, Daniel %A Schellinger, Jim %A Huseljic, Denis %A Sick, Bernhard %D 2019 %J arXiv e-prints %P arXiv:1901.10338 %T Limitations of Assessing Active Learning Performance at Runtime

2018[ to top ]

Towards Proactive Health-enabling Living Environments: Simulation-based Study and Research Challenges. Tomforde, Sven; Dehling, Tobias; Haux, Reinhold; Huseljic, Denis; Kottke, Daniel; Scheerbaum, Jonas; Sick, Bernhard; Sunyaev, Ali; Wolf, Klaus-Hendrik. In International Workshop on Self-optimisation in Organic and Autonomic Computing Systems (SAOS), ARCS, bll 1–8. VDE, 2018.
- [ BibTeX ]
- [ EndNote ]
@inproceedings{tomforde2018towards, author = {Tomforde, Sven and Dehling, Tobias and Haux, Reinhold and Huseljic, Denis and Kottke, Daniel and Scheerbaum, Jonas and Sick, Bernhard and Sunyaev, Ali and Wolf, Klaus-Hendrik}, booktitle = {International Workshop on Self-optimisation in Organic and Autonomic Computing Systems (SAOS), ARCS}, keywords = {imported}, pages = {1–8}, publisher = {VDE}, title = {Towards Proactive Health-enabling Living Environments: Simulation-based Study and Research Challenges}, year = 2018 }
%0 Conference Paper %1 tomforde2018towards %A Tomforde, Sven %A Dehling, Tobias %A Haux, Reinhold %A Huseljic, Denis %A Kottke, Daniel %A Scheerbaum, Jonas %A Sick, Bernhard %A Sunyaev, Ali %A Wolf, Klaus-Hendrik %B International Workshop on Self-optimisation in Organic and Autonomic Computing Systems (SAOS), ARCS %D 2018 %I VDE %P 1--8 %T Towards Proactive Health-enabling Living Environments: Simulation-based Study and Research Challenges
The Other Human in The Loop -- A Pilot Study to Find Selection Strategies for Active Learning. Kottke, Daniel; Calma, Adrian; Huseljic, Denis; Sandrock, Christoph; Kachergis, George; Sick, Bernhard. In International Joint Conference on Neural Networks (IJCNN). IEEE, Rio de Janiero, Brazil, 2018.
- [ BibTeX ]
- [ EndNote ]
@inproceedings{kottke2018other, address = {Rio de Janiero, Brazil}, author = {Kottke, Daniel and Calma, Adrian and Huseljic, Denis and Sandrock, Christoph and Kachergis, George and Sick, Bernhard}, booktitle = {International Joint Conference on Neural Networks (IJCNN)}, keywords = {imported}, publisher = {IEEE}, title = {The Other Human in The Loop – A Pilot Study to Find Selection Strategies for Active Learning}, year = 2018 }
%0 Conference Paper %1 kottke2018other %A Kottke, Daniel %A Calma, Adrian %A Huseljic, Denis %A Sandrock, Christoph %A Kachergis, George %A Sick, Bernhard %B International Joint Conference on Neural Networks (IJCNN) %C Rio de Janiero, Brazil %D 2018 %I IEEE %R 10.1109/ijcnn.2018.8489637 %T The Other Human in The Loop -- A Pilot Study to Find Selection Strategies for Active Learning

2017[ to top ]

Challenges of Reliable, Realistic and Comparable Active Learning Evaluation. Kottke, Daniel; Calma, Adrian; Huseljic, Denis; Krempl, Georg; Sick, Bernhard. In Workshop on Interactive Adaptive Learning (IAL), ECML PKDD, CEUR Workshop Proceedings, bll 2–14. 2017.
- [ BibTeX ]
- [ EndNote ]
- [ URL ]
@inproceedings{kottke2017challenges, author = {Kottke, Daniel and Calma, Adrian and Huseljic, Denis and Krempl, Georg and Sick, Bernhard}, booktitle = {Workshop on Interactive Adaptive Learning (IAL), ECML PKDD}, keywords = {imported}, pages = {2–14}, series = {CEUR Workshop Proceedings}, title = {Challenges of Reliable, Realistic and Comparable Active Learning Evaluation}, year = 2017 }
%0 Conference Paper %1 kottke2017challenges %A Kottke, Daniel %A Calma, Adrian %A Huseljic, Denis %A Krempl, Georg %A Sick, Bernhard %B Workshop on Interactive Adaptive Learning (IAL), ECML PKDD %D 2017 %P 2--14 %T Challenges of Reliable, Realistic and Comparable Active Learning Evaluation %U http://ceur-ws.org/Vol-1924/ialatecml_paper0.pdf