Search

Training a deep neural network is computationally expensive, but achieving the same network performance with less computation is possible if the training data is carefully chosen. However, selecting input samples during training is challenging as their true importance for the optimization is unknown. Furthermore, evaluation of the importance of individual samples must be computationally efficient and unbiased. In this paper, we present a new input data importance sampling strategy for reducing the training time of deep neural networks. We investigate different importance metrics that can be efficiently retrieved as they are available during training, i.e., the training loss and gradient norm. We found that choosing only samples with large loss or gradient norm, which are hard for the network to learn, is not optimal for the network performance. Instead, we introduce an importance sampling strategy that selects samples based on the cumulative distribution function of the loss and gradient norm, thereby making it more likely to choose hard samples while still including easy ones. The behavior of the proposed strategy is first analyzed on a synthetic dataset, and then evaluated in the application of classification of malignant cancer in digital pathology image patches. As pathology images contain many repetitive patterns, there could be significant gains in focusing on features that contribute stronger to the optimization. Finally, we show how the importance sampling process can be used to gain insights about the input data through visualization of samples that are found most or least useful for the training., The fulltext is published under Creative Common license Attribution 4.0https://creativecommons.org/licenses/by/4.0/No changes have been made to the publication.

Artificial neural networks are networks made up of thousands and sometimes millions or more nodes also referred to as neurons. Due to the sheer scale of a network, the task of training the network can become very compute-intensive. This is because all samples need to be evaluated through the network during training, and the gradients need to be updated based on each sample`s loss. Like humans, neural networks find some samples more difficult to interpret correctly than others. By feeding the network with more difficult samples while avoiding samples it has already mastered the training process can be executed more efficiently. In the medical field neural networks are among other use cases used to identify malignant cancer in tissue samples. In such a use case being able to increase the performance of a model by 1-2 percentage units could have a huge impact on saving lives by correctly discovering malignant cancer. In this thesis project, different importance sampling methods are evaluated and tested on multiple networks and datasets. The results show how importance sampling can be utilized to faster reach a higher accuracy and save time. Not only are different importance sampling methods evaluated but also different thresholds and methods to determine when to start the importance sampling., Examensarbetet är utfört vid Institutionen för teknik och naturvetenskap (ITN) vid Tekniska fakulteten, Linköpings universitet

Artificial neural networks are networks made up of thousands and sometimes millions or more nodes also referred to as neurons. Due to the sheer scale of a network, the task of training the network can become very compute-intensive. This is because all samples need to be evaluated through the network during training, and the gradients need to be updated based on each sample`s loss. Like humans, neural networks find some samples more difficult to interpret correctly than others. By feeding the network with more difficult samples while avoiding samples it has already mastered the training process can be executed more efficiently. In the medical field neural networks are among other use cases used to identify malignant cancer in tissue samples. In such a use case being able to increase the performance of a model by 1-2 percentage units could have a huge impact on saving lives by correctly discovering malignant cancer. In this thesis project, different importance sampling methods are evaluated and tested on multiple networks and datasets. The results show how importance sampling can be utilized to faster reach a higher accuracy and save time. Not only are different importance sampling methods evaluated but also different thresholds and methods to determine when to start the importance sampling., Examensarbetet är utfört vid Institutionen för teknik och naturvetenskap (ITN) vid Tekniska fakulteten, Linköpings universitet

Training a deep neural network is computationally expensive, but achieving the same network performance with less computation is possible if the training data is carefully chosen. However, selecting input samples during training is challenging as their true importance for the optimization is unknown. Furthermore, evaluation of the importance of individual samples must be computationally efficient and unbiased. In this paper, we present a new input data importance sampling strategy for reducing the training time of deep neural networks. We investigate different importance metrics that can be efficiently retrieved as they are available during training, i.e., the training loss and gradient norm. We found that choosing only samples with large loss or gradient norm, which are hard for the network to learn, is not optimal for the network performance. Instead, we introduce an importance sampling strategy that selects samples based on the cumulative distribution function of the loss and gradient norm, thereby making it more likely to choose hard samples while still including easy ones. The behavior of the proposed strategy is first analyzed on a synthetic dataset, and then evaluated in the application of classification of malignant cancer in digital pathology image patches. As pathology images contain many repetitive patterns, there could be significant gains in focusing on features that contribute stronger to the optimization. Finally, we show how the importance sampling process can be used to gain insights about the input data through visualization of samples that are found most or least useful for the training., The fulltext is published under Creative Common license Attribution 4.0https://creativecommons.org/licenses/by/4.0/No changes have been made to the publication.

Training a deep neural network is computationally expensive, but achieving the same network performance with less computation is possible if the training data is carefully chosen. However, selecting input samples during training is challenging as their true importance for the optimization is unknown. Furthermore, evaluation of the importance of individual samples must be computationally efficient and unbiased. In this paper, we present a new input data importance sampling strategy for reducing the training time of deep neural networks. We investigate different importance metrics that can be efficiently retrieved as they are available during training, i.e., the training loss and gradient norm. We found that choosing only samples with large loss or gradient norm, which are hard for the network to learn, is not optimal for the network performance. Instead, we introduce an importance sampling strategy that selects samples based on the cumulative distribution function of the loss and gradient norm, thereby making it more likely to choose hard samples while still including easy ones. The behavior of the proposed strategy is first analyzed on a synthetic dataset, and then evaluated in the application of classification of malignant cancer in digital pathology image patches. As pathology images contain many repetitive patterns, there could be significant gains in focusing on features that contribute stronger to the optimization. Finally, we show how the importance sampling process can be used to gain insights about the input data through visualization of samples that are found most or least useful for the training., The fulltext is published under Creative Common license Attribution 4.0https://creativecommons.org/licenses/by/4.0/No changes have been made to the publication.

Training a deep neural network is computationally expensive, but achieving the same network performance with less computation is possible if the training data is carefully chosen. However, selecting input samples during training is challenging as their true importance for the optimization is unknown. Furthermore, evaluation of the importance of individual samples must be computationally efficient and unbiased. In this paper, we present a new input data importance sampling strategy for reducing the training time of deep neural networks. We investigate different importance metrics that can be efficiently retrieved as they are available during training, i.e., the training loss and gradient norm. We found that choosing only samples with large loss or gradient norm, which are hard for the network to learn, is not optimal for the network performance. Instead, we introduce an importance sampling strategy that selects samples based on the cumulative distribution function of the loss and gradient norm, thereby making it more likely to choose hard samples while still including easy ones. The behavior of the proposed strategy is first analyzed on a synthetic dataset, and then evaluated in the application of classification of malignant cancer in digital pathology image patches. As pathology images contain many repetitive patterns, there could be significant gains in focusing on features that contribute stronger to the optimization. Finally, we show how the importance sampling process can be used to gain insights about the input data through visualization of samples that are found most or least useful for the training., The fulltext is published under Creative Common license Attribution 4.0https://creativecommons.org/licenses/by/4.0/No changes have been made to the publication.

Training a deep neural network is computationally expensive, but achieving the same network performance with less computation is possible if the training data is carefully chosen. However, selecting input samples during training is challenging as their true importance for the optimization is unknown. Furthermore, evaluation of the importance of individual samples must be computationally efficient and unbiased. In this paper, we present a new input data importance sampling strategy for reducing the training time of deep neural networks. We investigate different importance metrics that can be efficiently retrieved as they are available during training, i.e., the training loss and gradient norm. We found that choosing only samples with large loss or gradient norm, which are hard for the network to learn, is not optimal for the network performance. Instead, we introduce an importance sampling strategy that selects samples based on the cumulative distribution function of the loss and gradient norm, thereby making it more likely to choose hard samples while still including easy ones. The behavior of the proposed strategy is first analyzed on a synthetic dataset, and then evaluated in the application of classification of malignant cancer in digital pathology image patches. As pathology images contain many repetitive patterns, there could be significant gains in focusing on features that contribute stronger to the optimization. Finally, we show how the importance sampling process can be used to gain insights about the input data through visualization of samples that are found most or least useful for the training., The fulltext is published under Creative Common license Attribution 4.0https://creativecommons.org/licenses/by/4.0/No changes have been made to the publication.