Developing a LLM-Driven Multi-Agent Framework for Multimodal Translation

Motivation

The increasing global demand for translated content, particularly in multimedia formats like manga, comic book, and literary works, has highlighted significant challenges in maintaining both efficiency and quality in translation processes. Traditional manual translation workflows, which rely heavily on human translators, editors, and typesetters, often struggle with scalability and consistency when handling large volumes of content that combines both textual and visual elements. This challenge becomes particularly acute in the context of manga translation, where cultural nuances, visual context, and textual accuracy must be carefully balanced by skilled professionals, leading to substantial time investments and increased production costs.

Our goal is to combine recent advancements in multi-agent translation systems and context-aware multimodal translation, leveraging the increasing power of LLMs and VLMs to create a manga translation pipeline that has both convenience of machine translation and the quality of human translation. In doing so, we aim to demonstrate the potential of modular multi-agent frameworks to flexibly accommodate a wide variety of media, which we believe can facilitate the globalization of translated content at affordable costs, allowing for the enrichment of people all around the world.

Dataset

Our primary baseline dataset comes from OpenMantra (Hinami et al.), an automatic manga translation project that provides 214 manually translated and annotated manga pages for use as a benchmark for future research.

OpenMantra text field annotations (excerpt), visualized

Notably, although we aim to match the general format for evaluation purposes, our identified text fields have a much tighter fit to facilitate typesetting. Additionally, while the English translation of the pictured text is used to evaluate the overall accuracy of our translation framework, we expect differences due to subjective measures of fluency and reader perception.

Methods

Our innovated method pipeline involves three stages with the collaboration of agents, beginning with page processing where text is recognized and clustered, then moving on to leverage multi-agent collaboration for translation, and at last the typesetting stage to return a translated version of the page with translations replacing the original texts.

Manga-Specialized Preprocessing

Original image

Text segmentation mask

Filtered clusters (color-coded)

Ordered text boxes

In the initial preprocessing stage, we implement a pipeline designed to identify the positions of text boxes and extract text from them.

• Text Segmentation: Creates a mask using a text segmentation model specialized for manga, robust against cases such as oddly shaped text fields and non-text elements within speech bubbles.
• Clustering: Groups potential text into clusters using the OPTICS algorithm.
• Speech Bubble Filter: Filters out text outside speech bubbles to match baseline data by analyzing the background color of the text.
• Page Element Ordering: Sorts clustered text boxes into reading order using the Magi model.
• Text Extraction: Extracts text using MangaOCR, designed to be robust against scenarios specific to manga, such as text overlaid over images or accompanied by furigana, wide varieties in font style and size, and different reading orientations.

Evaluation

The evaluation of the pipeline is separated into two major parts, one on the accuracy of page processing stage in capturing the correct position of texts and extracting the text accurately, the other part is in assessing the translation quality in context awareness and nuance. To effectively assess the accuracy, coherence, and context-awareness of the multi-agent translation system, we took a combination of automatic metrics such as BERTScore, as well as human evaluation to determine how well the system preserves the emotional nuance and narrative consistency.

Processing Stage Evaluation

Levenshtein distance, visualized

Green/blue boxes correspond to baseline/observed boxes, respectively

The preprocessing stage is evaluated on two primary elements:

• Text Extraction:
  • Uses Levenshtein distance ratio to penalize common errors (misidentified page elements) as deletions while accurately evaluating the accuracy of the rest of the extracted text.
  
  • Also used by comparable projects (Lippmann et al.), allowing for comparison in performance.
• Text Box Position:
  • Implemented box-center check to assess accuracy despite systematic discrepancies with baseline text boxes.
  • Major discrepancies are tighter box fit and split text boxes in our model (see figure above), which facilitate typesetting later in the pipeline.

BERTScore and Human Evaluation

To effectively assess the accuracy, coherence, and context-awareness of the multi-agent translation system, we took a combination of automatic metrics such as BERTScore, as well as human evaluation to determine how well the system preserves the emotional nuance and narrative consistency.

• BERTScore: Capture semantic similarity, focusing on contextual accuracy and nuance. Evaluated by Bert F1-score.
• Human Evaluation: Monolingual and bilingual experts rating translation for fluency, accuracy, proximity to released human translation and cultural appropriateness, choosing one best overall translation from the results of our model and one randomly selected baseline model.

Below is an example of our human evaluation survey question. We have a total of 4 different versions of the survey, each consists of ten randomly selected pages with one of the option being our model's output and the other being a randomly selected baseline's output to reduce bias. The final percentage of preference was taken as metrics.

Machine Evaluation

To systematically assess the quality of translations, we incorporate Machine Evaluation using the ChatGPT web application with the GPT-4o model as an automated evaluator. This approach allows us to efficiently compare different translation outputs at scale.

Results

Processing Stage Results

Distribution of accuracy metric results by page

Distribution of processing times by page

After processing the entire OpenMantra dataset and comparing our output to the baseline, we observed the following results:

• Text Extraction:
  • Achieves accuracy of 86.3%, with errors mostly coming from misidentified text fields that escape the filter.
  • Outperforms Lippmann et al. by almost 10%, primarily due to the implementation of the speech bubble filter.
  • Text extracted from actual dialogue fields is almost always perfectly extracted by the OCR system.
• Text Box Position:
  • 22% of pages contain at least one text box not included in the baseline data.
  • Almost all of these are misidentified sound effects, which do not appear to significantly affect translation quality.
• Latency:
  • Model requires an average of about six seconds of processing time per stage, primarily allotted to clustering.
  • Pages requiring extraordinarily long processing times usually have large amounts of sound effects, which add significant computational load to the clustering algorithm.
• Please note that these results come from processing using an A40 GPU; with increased computational power, the latency is expected to be significantly lower.

Latency breakdown by stage

Performance and BERTScore

We evaluate our pipeline against three baselines that are effective in I2I translation tasks:

• Comparison to mono-agent baseline: our multi-agent structure outperform the baseline with both models (llama3.1:8b and gpt-4-turbo)
• Comparison to Google Translate: out multi-agent structure using gpt-4-turbo outperforms google translate baseline.

This finding has been validated with human evaluation with results shown below.

Human Evaluation Result

As shown in the plot, our model achieved a 67% winning rate against mono-agent structure, and a 86% winning rate against google translate.

Machine Evaluation

The results from the GPT-4o-based Machine Evaluation indicate that our model performed favorably in comparison to baseline models:

• 90% preference over Google Translate, suggesting improvements in fluency and accuracy.
• 65% preference over the mono-agent baseline, reflecting notable enhancements in translation quality.

These findings suggest that GPT-4o's automated evaluation aligns well with human judgment and provides valuable insights into translation quality assessment.

Conclusion

Our multi-agent framework enhances manga translation by improving translation quality and aligning with human reading expectations. By distributing tasks—text detection, contextual translation, and typesetting—our system ensures more natural and readable translations.

We also developed a specialized typesetting system to seamlessly integrate translated text into manga, addressing formatting challenges often overlooked in automated approaches.

Challenges remain, including speech bubble resizing and processing latency. Future work will focus on real-time translation and expanding support for other media types.

This research sets the foundation for sophisticated multimodal translation systems that balance linguistic accuracy with visual presentation.

References