Search

Recently, end-to-end speech translation (ST) has gained significant attention as it avoids error propagation. However, the approach suffers from data scarcity. It heavily depends on direct ST data and is less efficient in making use of speech transcription and text translation data, which is often more easily available. In the related field of multilingual text translation, several techniques have been proposed for zero-shot translation. A main idea is to increase the similarity of semantically similar sentences in different languages. We investigate whether these ideas can be applied to speech translation, by building ST models trained on speech transcription and text translation data. We investigate the effects of data augmentation and auxiliary loss function. The techniques were successfully applied to few-shot ST using limited ST data, with improvements of up to +12.9 BLEU points compared to direct end-to-end ST and +3.1 BLEU points compared to ST models fine-tuned from ASR model., Comment: 6 pages, 5 figures, accepted to IEEE ICASSP 2022. arXiv admin note: text overlap with arXiv:2107.06010

With the increasing number of underwater pipeline investigation activities, the research on automatic pipeline detection is of great significance. At this stage, object detection algorithms based on Deep Learning (DL) are widely used due to their abilities to deal with various complex scenarios. However, DL algorithms require massive representative samples, which are difficult to obtain for pipeline detection with sub-bottom profiler (SBP) data. In this paper, a zero-shot pipeline detection method is proposed. First, an efficient sample synthesis method based on SBP imaging principles is proposed to generate samples. Then, the generated samples are used to train the YOLOv5s network and a pipeline detection strategy is developed to meet the real-time requirements. Finally, the trained model is tested with the measured data. In the experiment, the trained model achieved a mAP@0.5 of 0.962, and the mean deviation of the predicted pipeline position is 0.23 pixels with a standard deviation of 1.94 pixels in the horizontal direction and 0.34 pixels with a standard deviation of 2.69 pixels in the vertical direction. In addition, the object detection speed also met the real-time requirements. The above results show that the proposed method has the potential to completely replace the manual interpretation and has very high application value.

Data annotation is a major problem in all machine learning tasks. In the field of NLP, this problem is multiplied by the number of existing languages.Many languages do not have any annotations, and are therefore excluded from NLP systems. One possible solution to integrate these languages into the systems is to try to leverage the languages having many annotations, and to try to learn information about these resource-rich languages, and to transfer this knowledge to the low-resources languages.It is possible to rely on initiatives such as Universal Dependencies, which propose a universal annotation scheme between languages. The use of multilingual word embeddings and typological features from resources such as the WALS are solutions allowing knowledge sharing between languages.These tracks are studied in the framework of this thesis, through the prediction of parsing, morphology and parts of speech on 41 languages in total. We show that the impact of the WALS can be positive in a multilingual setting, but that its usefulness is not systematic in a zero-shot learning setting. Other language representations can be learned from the data, and perform better than the WALS, but have the downside of not working in a zero-shot setting. We also highlight the importance of the presence of a nearby language when learning patterns, as well as the problems associated with using a character pattern for isolated languages.; L'annotation de données est un problème majeur dans toutes les tâches d'apprentissage automatique. Dans le domaine du TAL, ce problème est multiplié par le nombre de langues existantes.De nombreuses langues se retrouvent sans annotations, et sont alors mises à l'écart des systèmes de TAL. Une solution possible pour intégrer ces langues dans les systèmes est de tenter d'exploiter les langues disposant de nombreuses annotations, d'apprendre des informations sur ces langues bien dotées, et de transférer ce savoir vers les langues peu dotées. Pour cela, il est possible de se reposer sur des initiatives comme les Universal Dependencies, qui proposent un schéma d'annotation universel entre les langues. L'utilisation de plongements de mots multilingues et de traits typologiques issus de ressources comme le WALS sont des solutions permettant un partage de connaissances entre les langues.Ces pistes sont étudiées dans le cadre de cette thèse, à travers la prédiction de l'analyse syntaxique, de la morphologie et des parties du discours sur 41 langues au total. Nous montrons que l'impact du WALS peut être positif dans un cadre multilingue, mais que son utilité n'est pas systématique dans une configuration d'apprentissage zero-shot. D'autres représentations des langues peuvent être apprises sur les données, et donnent de meilleurs résultats que le WALS, mais ont l'inconvénient de ne pas fonctionner dans un cadre de zero-shot. Nous mettons également en évidence l'importance de la présence d'une langue proche lors de l'apprentissage des modèles, ainsi que les problèmes liés à l'utilisation d'un modèle de caractère pour les langues isolées.

Conventional image annotation systems can only handle those images having labels within the exist library, but cannot recognize those novel labels. In order to learn new concepts, one has to gather large amount of labeled images and train the model from scratch. More importantly, it can come with a high price to collect those labeled images. For these reasons, we put forward a zero-shot image annotation model, to reduce the demand for the images with novel labels. In this paper, we focus on the two big challenges of zero-shot image annotation: polysemous words and a strong bias in the generalized zero-shot setting. For the first problem, instead of training on large corpus datasets as previous methods, we propose to adopt Node2Vec to obtain contextualized word embeddings, which can easily produce word vectors of the polysemous words. For the second problem, we alleviate the strong bias in two ways: on one hand, we utilize a model based on graph convolutional network (GCN) to make target images involved in the training process; on the other hand, we put forward a novel semantic coherent (SC) loss to capture the semantic relations of the source and target labels. The extensive experiments on NUSWIDE, COCO, IAPR TC-12, and Core15k datasets show the superiority of the proposed model and the annotation performance get improved by 4%-6% comparing with state-of-the-art methods.

Zero-shot learning aims to classify unseen class examples. It gained popularity in applications where examples for each category are limited. The main issue to consider is transferring information from seen classes to unseen classes via mapping image space to semantic space. Therefore, mapping from image space to semantic space is at the core of the learning process. In this work, Google’s Word2vec were used for semantic space. Total of 20 classes, 15 for training and 5 for zero-shot classes were chosen from Visual Gnome dataset. We have achieved 0.71 accuracy for top-5 classes.

Domain-agnostic data retrieval has lately become essential amidst the availability of large-scale data from different types of sensors. However, the unavailability of a sufficient amount of samples of certain classes during training curtails the utility of existing retrieval models in remote sensing (RS) applications. Here, we propose a novel framework for zero-shot intermodal data retrieval of RS data. Thereupon, we design an encoder-decoder structure that ensures enhanced overlapping among the two data domains utilizing cross-triplet and cross-projection loss functions. Furthermore, we propose a sketch-based representation of the RS database Earth on Canvas with diverse classes. We perform a thorough benchmarking of this data set and demonstrate that the proposed framework outperforms state-of-the-art methods for zero-shot sketch-based retrieval framework for RS data.

La traducció automàtica és la tasca de traduir automàticament unidioma a un altre. Aquest projecte avalua el rendiment dels últims siste-mes d'aprenentatge profund en la tasca de traducció d'idiomes similars.Avaluarem la traducció entre Català, Castella i Portuguès, que són llen-gües romàniques, per veure com l'arquitectura del Transformer realitzala tasca. També farem servir diverses tècniques per millorar la traduccióentre els idiomes. Primer, utilitzarem model multilingües que permetenfer transferència de coneixement entre idiomes i poder fer traduccionszero-shot. Després aplicarem backtranslation per poder fer ús dels textsmonolingües i millorar les traduccions del sistema. Per últim milloraremla traducció de domini específic fent ús de fine tuning. Machine translation is the task of automatically translating one lan-guage into another. This project aims to evaluate the performance ofstate-of-the-art Deep Learning systems on similar language translation.We will to evaluate the translation between Catalan, Spanish, and Por-tuguese, which are Romance languages, and see how the Transformer ar-chitecture performs in this task. We will additionally make use of differenttechniques to improve the translation between these languages. First ofall, we will make use of multilingual models that allow for transfer-learningas well as zero-shot translations. Secondly, we will apply the backtransla-tion technique to make use of the monolingual data and better the systemtranslations. Lastly, we will improve the specific domain data using finetuning.

Klassifikationsprobleme leiden oftmals unter einem Mangel an annotierten Trainingsdaten. Dies f��hrte zu der Entwicklung von Zero-shot Learning Modellen, welche mit Klassen trainiert werden f��r die ausreichend Trainingsdaten zur Verf��gung stehen, um dann unbekannte Klassen anhand von Beschreibungen zu erkennen. Oftmals sind diese Beschreibungen in der Form von Attributsvektoren, die allerdings ebenfalls selten zur Verf��gung stehen und aufw��ndig zu erstellen sind. Manche Ans��tze nutzen daher stattdessen Beschreibungen in nat��rlicher Sprache. In dieser Arbeit wird eine neue Methode zum Vergleich von Daten aus verschiedenen Dom��nen, die Autoencoder Distance (AD), vorgestellt und getestet in einer Zero-shot Anwendung mit Bilddaten und Beschreibungen in nat��rlicher Sprache. Die Distanzfunkion basiert auf der Normalised Compression Distance von Cilibrasi und Vit��nyi, ein Verfahren bei dem verlustfreie Komprimierungsalgorithmen genutzt werden um gemeinsame Muster zu erkennen, in dem die Gr����e von kombinierten Eingangsdaten nach Komprimierung gemessen werden. Die Messung wird normalisiert mit den Gr����en der Eingabedaten wenn sie unabh��ngig voneinander komprimiert werden. F��r die Methode die in dieser Arbeit vorgestellt wird ist statt eines verlustfreien Komprimierungsalgorithmus ein Autoencoder im Einsatz. Dieser wird zuerst darauf trainiert zusammengeh��rige Eingabedaten zu assoziieren, also Bilder und die Beschreibungen der Klassen denen sie angeh��ren. Die Distanz zwischen Eingabedaten wird dann approximiert indem die mittlere quadratische Abweichung zwischen der Beschreibung und der korrespondierenden Ausgabe berechnet wird. F��r die Normalisierung werden f��r alle Beschreibungen Durchschnitt und Standardabweichung dieser Abweichung f��r alle Bilder in einem festgelegten Set genutzt. Zur Klassifikationen eines Bildes werden alle Beschreibungen nach ihrem AD zu diesem Bild gereiht. Das Bild wird dann der Klasse die der erstgereihten Beschreibung entspricht zugeordnet. Evaluiert wird das Modell anhand einer Variation des Caltech-USCD Vogel-Datensets mit Klassenbeschreibungen von Reed et al. Des Weiteren werden Bildersets von diversen Tieren und Alltagsgenst��nden zur Normalisierung genutzt. Beim Klassifizieren mit 50 Beschreibungen die im Training nicht vorkamen konnte eine T1 Genauigkeit von 23,25% und eine T5 Genauigkeit von 57,14% erreicht werden, wobei Bilder von Seepferdchen zur Normalisierung genutzt wurden. Diese Werte sind geringer als Genauigkeiten die von anderen Werken auf den gleichen Daten erreicht werden konnte, aber durch die neuartige Methode werden viele bisher unerforschte Ans��tze f��r zuk��nftige Entwicklungen er��ffnet. Als ein Nebenziel wird zus��tzlich gezeigt, dass die Ausgabe des Autoencoders f��r Explainability genutzt werden kann., Many classification tasks suffer a lack of labelled data. This led to the development of zero-shot learning models, which are trained on classes with available data to recognise unknown classes from descriptions. Often this is done with descriptions in the form of attribute vectors, but those are again rarely available and expensive to produce. Some approaches therefore use descriptions in natural language instead. In this thesis a new method of comparing data from different domains, Autoencoder Distance (AD), is introduced and tested on a zero-shot application with image data and natural language descriptions. The distance function is based on the Normalised Compression Distance by Cilibrasi and Vit��nyi, a method that uses lossless compression algorithms to estimate shared patterns by measuring the size of combined inputs after compression, normalised by the compressed size of the inputs on their own. For the method introduced in this thesis an autoencoder is used instead of lossless compression. It is first trained to associate related inputs, i.e., images and the descriptions of their class. The distance between inputs is then approximated by calculating the mean squared error between the input description and its reconstruction. Normalisation for each description is done with the mean and standard deviation of this error over a shared set of images. For classification descriptions are ranked by their AD to a given image. The imaged is then placed in the class associated with the top ranked description. Evaluation is done on a variation of the Caltech-USCD bird dataset with descriptions provided by Reed et al. Further, image sets depicting various animals and commonplace items are used for normalisation. Classifying by ranking 50 descriptions not encountered in training, a T1 accuracy of 23.25% and a T5 accuracy of 57.14% could be achieved using pictures of sea horses for normalisation. This is lower than what was previously achieved on the same data, but the new method opens many novel avenues for future work. As a secondary objective it is also shown that the output of the autoencoder can be used for explainability.

Only a handful of the world's languages are abundant with the resources that enable practical applications of speech processing technologies. One of the methods to overcome this problem is to use the resources existing in other languages to train a multilingual automatic speech recognition (ASR) model, which, intuitively, should learn some universal phonetic representations. In this work, we focus on gaining a deeper understanding of how general these representations might be, and how individual phones are getting improved in a multilingual setting. To that end, we select a phonetically diverse set of languages, and perform a series of monolingual, multilingual and crosslingual (zero-shot) experiments. The ASR is trained to recognize the International Phonetic Alphabet (IPA) token sequences. We observe significant improvements across all languages in the multilingual setting, and stark degradation in the crosslingual setting, where the model, among other errors, considers Javanese as a tone language. Notably, as little as 10 hours of the target language training data tremendously reduces ASR error rates. Our analysis uncovered that even the phones that are unique to a single language can benefit greatly from adding training data from other languages - an encouraging result for the low-resource speech community., Comment: Submitted to Interspeech 2020. For some reason, the ArXiv Latex engine rendered it in more than 4 pages