Navigating the Nuances of AI Translation: A Technica Hackathon Reflection

Greetings, fellow tech aficionados and linguists!

I participated in the University Of Maryland’s Technica 2022 hackathon, specifically diving into the research track that delved deep into the intricacies of AI-based translation. I’m thrilled to share that our team emerged victoriously, and here’s a reflection on our journey, the challenges we faced, and the insights we gained.

The Inspiration Behind Our Project

In a world where AI-powered machine translation (MT) is increasingly ubiquitous, it becomes critical to question the reliability of such tools, especially in high-stakes environments. While MT has the potential to break down language barriers, it’s not infallible. Our project, “Metrics for Reliable AI-based Translation,” sought to explore this domain of uncertainty and develop methods to assess the quality of MT outputs critically.

The Mission of Our Analysis

Our project aimed to establish a framework for evaluating the acceptability of translations produced by AI. We focused on two distinct dataset types: TedX presentations, classified as low-risk, and COVID-19-related content, deemed high-risk. Our objective was to:

• Garner acceptability judgments from both bilingual and monolingual participants.
• Utilize translation metrics like Comet-Src and length-based heuristics to evaluate MT outputs.
• Produce visualizations that identify accuracy thresholds for French and Russian translations.
• Synthesize metrics to provide a recommendation on the translation’s acceptability.

Constructing the Solution

Our approach was grounded in collaboration and extensive research. Using Google Colab as our development platform, we delved into academic articles to inform our methodology and then cooperatively crafted our code.

Overcoming Technical Obstacles

One significant challenge was the similarity in scoring between machine and human translations, which muddied the waters in distinguishing between the two. We navigated this by recognizing that certain metrics performed better for specific languages, which informed our approach to the evaluation process.

Celebrating Our Achievements

We’re incredibly proud of constructing a high-quality dataset that facilitated machine translation acceptability for two language pairs across two risk conditions. The application of three distinct evaluation methods was a testament to our team’s dedication and synergy.

Lessons From Our Research

Our findings illuminated the nuanced nature of translation acceptability – highly dependent on the target language, the content domain, and the context of the application. We also discovered the potential pitfalls of relying on backtranslation, especially for monolingual users in high-risk scenarios.

Charting the Path Forward

The research horizon beckons with questions about language-specific acceptability metrics, the discovery of edge cases, and the refinement of evaluation methodologies. We also aim to enhance our survey instruments to minimize the discrepancy in acceptability judgments.

Inviting Further Exploration

For those interested in delving deeper into our research process or perhaps embarking on similar explorations, we’ve made our codebase and project presentation available, here. They serve as a comprehensive guide to our thought processes and methodological choices, as does our project summary on Devpost!

To those who share a passion for pushing the boundaries of what’s possible with AI in language translation, let’s connect and collaborate on future endeavors!