Search

espanolEn este articulo se comparan tres sistemas de traduccion automatica poniendo especial atencion en la evaluacion humana. Los sistemas analizados son un sistema estadistico de traduccion automatica con adaptacion al dominio, un sistema neuronal de traduccion automatica con adaptacion al dominio y un sistema de traduccion automatica generico. La comparacion se lleva a cabo en una traduccion del espanol al aleman de documentacion industrial de componentes y procesos de maquina herramienta. El estudio se centra en la evaluacion humana de la traduccion automatica, en concreto en los siguientes aspectos: fluidez, adecuacion y ranquin a nivel de segmento; fluidez, adecuacion, necesidad de posedicion, facilidad de posedicion y esfuerzo mental requerido en la posedicion a nivel de documento; productividad (velocidad de posedicion y esfuerzo de posedicion) y actitudes. Se hace enfasis en los factores humanos del proceso de evaluacion EnglishThis article aims to compare three machine translation systems with a focus on human evaluation. The systems under analysis are a domain-adapted statistical machine translation system, a domain-adapted neural machine translation system and a generic machine translation system. The comparison is carried out on translation from Spanish into German with industrial documentation of machine tool components and processes. The focus is on the human evaluation of the machine translation output, specifically on: fluency, adequacy and ranking at the segment level; fluency, adequacy, need for post-editing, ease of post-editing, and mental effort required in post-editing at the document level; productivity (post-editing speed and post-editing effort) and attitudes. Emphasis is placed on human factors in the evaluation process

Raziskave so pokazale, da dosegajo nevronski strojni prevajalniki obetavne rezultate pri prevajanju literarnih besedil. Strojno prevajanje literature iz angleščine v slovenščino pa še ni bilo raziskano, zato je namen tega dela razvoj nevronskega strojnega prevajalnika v več različicah, prilagojenega za prevajanje literature, in primerjava lastnih prevajalnikov s splošnim prevajalnikom Google Translate. Da bi ugotovila, ali specializirani modeli dosežejo boljše rezultate kot splošni prevajalnik Google Translate in ali je prevajalnik, specializiran za prevajanje romanov ene avtorice, uspešnejši kot prevajalnik, specializiran na različnih literarnih delih, primerjam prevajalnike z metrikama BLEU in METEOR, analizo napak strojnih prevodov, ocenjevanjem berljivosti in ustreznosti ter merjenjem zahtevnosti popravljanja. Analiza je pokazala, da je popravljanje strojnih prevodov vseh treh prevajalnikov hitrejše od prevajanja od začetka. Vse metode evalvacije razen ocene berljivosti so potrdile, da je prevajalnik, specializiran za prevajanje romanov ene avtorice, uspešnejši kot prevajalnik, naučen na korpusu različnih literarnih del. Kljub vsemu pa pri vseh metodah evalvacije dosega najboljše rezultate prevajalnik Google Translate. Raziskava razkriva tudi velike nekonsistentnosti med udeleženci pri ocenjevanju berljivosti in ustreznosti ter pri merjenju zahtevnosti popravljanja. Rezultati nakazujejo na to, da morda te metode niso zanesljive, zato bi bilo dobro te ugotovitve nadaljnje raziskati. Neural Machine Translation (NMT) has shown a promising performance on literary texts. Since the literary machine translation has not yet been researched for the English-to-Slovene translation direction, this Master’s thesis aims to bridge this gap by presenting a comparison among bespoke NMT models, tailored to novels, and Google Translate. To determine whether models, adapted to literary texts, perform better than Google Translate, and to establish whether an adaptation to a specific author further improves the performance of the NMT system, models were evaluated by the BLEU and METEOR metrics, error analysis of machine translation output, assessment of fluency and adequacy, and measurement of the post-editing (PE) effort. The findings show that all evaluated NMT approaches resulted in increases in translation productivity. The model, tailored to a specific author, performs better than the model, trained on a literary corpus, based on all scores except the scores for fluency. However, Google Translate still outperforms all bespoke models. The evaluation reveals a very low inter-rater agreement on fluency and adequacy, based on the kappa coefficient values, and significant discrepancies between post-editors. This suggests that these methods are not reliable, which should be addressed in future studies.

This study aims to analyse how translation experts from the German department of the European Commission&rsquo, s Directorate-General for Translation (DGT) identify and correct different error categories in neural machine translated texts (NMT) and their post-edited versions (NMTPE). The term translation expert encompasses translator, post-editor as well as revisor. Even though we focus on neural machine-translated segments, translator and post-editor are used synonymously because of the combined workflow using CAT-Tools as well as machine translation. Only the distinction between post-editor, which refers to a DGT translation expert correcting the neural machine translation output, and revisor, which refers to a DGT translation expert correcting the post-edited version of the neural machine translation output, is important and made clear whenever relevant. Using an automatic error annotation tool and the more fine-grained manual error annotation framework to identify characteristic error categories in the DGT texts, a corpus analysis revealed that quality assurance measures by post-editors and revisors of the DGT are most often necessary for lexical errors. More specifically, the corpus analysis showed that, if post-editors correct mistranslations, terminology or stylistic errors in an NMT sentence, revisors are likely to correct the same error type in the same post-edited sentence, suggesting that the DGT experts were being primed by the NMT output. Subsequently, we designed a controlled eye-tracking and key-logging experiment to compare participants&rsquo, eye movements for test sentences containing the three identified error categories (mistranslations, terminology or stylistic errors) and for control sentences without errors. We examined the three error types&rsquo, effect on early (first fixation durations, first pass durations) and late eye movement measures (e.g., total reading time and regression path durations). Linear mixed-effects regression models predict what kind of behaviour of the DGT experts is associated with the correction of different error types during the post-editing process.

As more companies integrate machine translation (MT) systems into their translation workflows, it becomes increasingly relevant to accurately measure post-editing (PE) effort. In this paper we explore how different types of errors in the MT output may affect PE effort, and take a closer look at the techniques used to measure it. For our experiment we curated a test suite of 60 EN > ES sentence pairs controlling certain features (sentence length, error frequency, topic, etc.) and had a group of 7 translators post-edit them using the PET tool, which helped collect temporal, technical and cognitive effort metrics. The results seem to challenge some previous error difficulty rankings; they also imply that, once other sentence features are controlled, the type of error to be addressed might not be as influential on effort as previously assumed. The low correlation values between the metrics for the different effort aspects may indicate that they do not reliably account for the full PE effort if not used in combination of one another.

This study presents a series of translation process experiments carried out in a Master of Translation and Interpreting (MTI) course in China. The research was designed to facilitate the PE training purpose and to empirically test the impact of task types, text types, translation briefs, and post-editing (PE) guidelines on temporal, technical and cognitive efforts in the English-Chinese PE and human translation (HT) processes. The three efforts were gauged by production time per word, insertions, and deletions of keystrokes, and pause to word ratio (PWR) with a pause threshold of 1000ms, respectively. Thirty-one MTI students were assigned to translate/post-edit six texts according to specific translation briefs, which called for good enough quality for internal or publishable quality for external disseminations. Light and full TAUS PE guidelines were provided for corresponding PE tasks. Triangulated data from keystroke logging, screen recording, questionnaires, subjects’ guided interviews and written protocols were used for analysis. The preliminary results suggest that a) post-editing significantly reduces temporal, technical and cognitive efforts compared to translating manually; b) light post-editing takes more time than full post-editing; c) the students found PE different from human translation in many ways. They also reported various challenges in the post-editing process, mainly due to the influence of their previous translation training, lack of experience in PE and the ambiguous wording of the guidelines. The results also indicate that the direct use of process studies in the classroom can be effective both for producing quantitatively valid research findings and fulfilling pedagogical functions.

Human rating of predicted post-editing effort is a common activity and has been used to train confidence estimation models. However, the correlation between human ratings and actual post-editing effort is under-measured. Moreover, the impact of presenting effort indicators in a post-editing user interface on actual post-editing effort has hardly been researched. In this study, ratings of perceived post-editing effort are tested for correlations with actual temporal, technical and cognitive post-editing effort. In addition, the impact on post-editing effort of the presentation of post-editing effort indicators in the user interface is also tested. The language pair involved in this study is English-Brazilian Portuguese. Our findings, based on a small sample, suggest that there is little agreement between raters for predicted post-editing effort and that the correlations between actual post-editing effort and predicted effort are only moderate, and thus an inefficient basis for MT confidence estimation. Moreover, the presentation of post-editing effort indicators in the user interface appears not to impact on actual post-editing effort.

As the translation profession has become more technologized, translators increasingly work within an interface that combines translation from scratch, translation memory suggestions, machine translation post-editing, and terminological resources. This study analyses user activity data from one such interface, and measures temporal effort for English to Japanese translation at the segment level. Using previous studies of translation within the framework of relevance theory as a starting point, various features and edits were identified and annotated within the texts, in order to find whether there was a relationship between their prevalence and translation effort. Although this study is exploratory in nature, there was an expectation based on previous studies that procedurally encoded utterances would be associated with greater translation effort. This expectation was complicated by the choice of a language pair in which there has been little research applying relevance theory to translation, and by contemporary research that has made the distinction between procedural and conceptual encoding appear more fluid than previously believed. Our findings are that some features that lean more towards procedural encoding (such as prevalence of pronouns and manual addition of postpositions) are associated with increased temporal effort, although the small sample size makes it impossible to generalise. Segments translated with the aid of translation memory showed the least average temporal effort, and segments translated using machine translation appeared to require more effort than translation from scratch.

Translation Environment Tools make translators’ work easier by providing them with term lists, translation memories and machine translation output. Ideally, such tools automatically predict whether it is more effortful to post-edit than to translate from scratch, and determine whether or not to provide translators with machine translation output. Current machine translation quality estimation systems heavily rely on automatic metrics, even though they do not accurately capture actual post-editing effort. In addition, these systems do not take translator experience into account, even though novices’ translation processes are different from those of professional translators. In this paper, we report on the impact of machine translation errors on various types of post-editing effort indicators, for professional translators as well as student translators. We compare the impact of MT quality on a product effort indicator (HTER) with that on various process effort indicators. The translation and post-editing process of student translators and professional translators was logged with a combination of keystroke logging and eye-tracking, and the MT output was analyzed with a fine-grained translation quality assessment approach. We find that most post-editing effort indicators (product as well as process) are influenced by machine translation quality, but that different error types affect different post-editing effort indicators, confirming that a more fine-grained MT quality analysis is needed to correctly estimate actual post-editing effort. Coherence, meaning shifts, and structural issues are shown to be good indicators of post-editing effort. The additional impact of experience on these interactions between MT quality and post-editing effort is smaller than expected.

We propose a simple, linear-combination automatic evaluation measure (AEM) to approximate post-editing (PE) effort. Effort is measured both as PE time and as the number of PE operations performed. The ultimate goal is to define an AEM that can be used to optimize machine translation (MT) systems to minimize PE effort, but without having to perform unfeasible repeated PE during optimization. As PE effort is expected to be an extensive magnitude (i.e., one growing linearly with the sentence length and which may be simply added to represent the effort for a set of sentences), we use a linear combination of extensive and pseudo-extensive features. One such pseudo-extensive feature, 1–BLEU times the length of the reference, proves to be almost as good a predictor of PE effort as the best combination of extensive features. Surprisingly, effort predictors computed using independently obtained reference translations perform reasonably close to those using actual post-edited references. In the early stage of this research and given the inherent complexity of carrying out experiments with professional post-editors, we decided to carry out an automatic evaluation of the AEMs proposed rather than a manual evaluation to measure the effort needed to post-edit the output of an MT system tuned on these AEMs. The results obtained seem to support current tuning practice using BLEU, yet pointing at some limitations. Apart from this intrinsic evaluation, an extrinsic evaluation was also carried out in which the AEMs proposed were used to build synthetic training corpora for MT quality estimation, with results comparable to those obtained when training with measured PE efforts. Work supported by the Spanish government through project EFFORTUNE (TIN2015-69632-R) and through grant PRX16/00043 for Mikel L. Forcada, and by the European Commission through QT21 project (H2020 No. 645452).

Studies on Controlled Language (CL) suggest that by removing features that are known to be problematic for MT (termed here “negative translatability indicators”, or “NTIs”), the MT output can be improved. It is assumed that an improvement in the output will result in lower post-editing effort. This study tests that assumption by measuring the technical, temporal and cognitive post-editing effort (Krings 2001) for English sentences in a user manual that have been translated into German using an MT system and that have been subsequently postedited by nine professional translators. The post-editing effort for sentences containing known NTIs is compared with the post-editing effort for sentences where all known NTIs have been removed In addition, relative post-editing effort (Krings 2001) - a comparison of post-editing effort and translation effort - is also measured. The methodologies employed include the keyboard monitoring tool, Translog, and Choice Network Analysis Results indicate that temporal and technical effort is greater for those sentences that contain known NTIs. Other findings are that some NTIs result in higher post-editing effort than others and that post-editing effort can also be expended for sentences where NTIs have been removed.

Machine translation has been proved worthwhile, in terms of time saving and productivity gains, in technical and administrative translation domains. In order to examine whether this also applies to audio description, an experiment comparing the efforts of creating an audio description from scratch, of translating it manually from English into Catalan and of post-editing its machine translated version has shown that the objective post-editing effort is lower than creating it ex novo. However, the subjective effort is perceived to be higher.