LLM-Driven AutoML and AI Agents for Multilingual Handwritten OCR

LLM-Driven AutoML for Handwritten Text Recognition

Handwritten text recognition (HTR), also known as handwritten OCR, remains a challenging problem in computer vision and document AI, particularly across different languages and writing systems.

Arabic, Persian, and English handwriting differ substantially in character structure, stroke patterns, connectivity, and visual appearance. These differences make multilingual and cross-lingual OCR difficult to address with manually designed neural network architectures.

In our paper, “LLM-Driven AutoML for Cross-Lingual Handwritten OCR: Closed-Loop Neural Architecture Search with GPT-5, GPT-4o, and Claude Sonnet 4,” we investigate whether large language models can act as autonomous AutoML agents for designing handwritten text recognition systems.

Large Language Models as AutoML Agents

Instead of manually selecting neural network architectures, preprocessing strategies, and model configurations, our framework uses large language models to propose, evaluate, and iteratively refine OCR architectures.

GPT-5, GPT-4o, and Claude Sonnet 4 independently participate in a closed-loop neural architecture search process, allowing the system to explore and improve candidate architectures with limited human intervention.

This connects several active research areas:

• large language model agents
• automated machine learning
• neural architecture search
• computer vision
• document intelligence
• handwritten text recognition
• multilingual AI

Multilingual and Cross-Lingual OCR

The framework is evaluated on Arabic, Persian, and English handwritten text recognition, making the work relevant to multilingual OCR and low-resource language technologies.

Arabic and Persian handwriting recognition are particularly important because many OCR systems and datasets remain concentrated on English and other high-resource languages.

The proposed approach therefore explores whether LLM-driven automated model design can generalize across substantially different writing systems.

Closed-Loop Neural Architecture Search

Traditional neural architecture search can require substantial manual engineering or large computational search spaces.

Our approach investigates an alternative in which LLM agents reason about experimental results and iteratively propose new neural architectures.

This creates a closed-loop process:

architecture generation → model training → evaluation → performance feedback → architecture refinement

The goal is to automate more of the model-development process while maintaining interpretable experimental iterations.

Applications in Document AI

The work is relevant beyond handwritten OCR. Similar LLM-driven AutoML approaches could potentially support:

• document understanding
• historical document recognition
• multilingual document processing
• digitization of handwritten archives
• low-resource language technologies
• automated computer vision model design
• intelligent document processing

This research contributes to LLM-driven AutoML, handwritten text recognition, multilingual OCR, neural architecture search, document AI, LLM agents, low-resource language processing, and automated neural network design.

AI Agents for Automated Machine Learning

A central idea of this work is to use large language models as AI agents for AutoML. Rather than treating an LLM only as a text generator, the framework gives the model an iterative role in neural architecture design.

Each LLM agent can propose an OCR architecture, observe training and evaluation results, reason about model performance, and generate a refined architecture for the next iteration.

This creates an agentic AI workflow:

AI agent → architecture proposal → training → evaluation → feedback → architecture refinement

The framework therefore connects AI agents, LLM agents, agentic AI, AutoML, neural architecture search, and multilingual handwritten OCR in a closed-loop model-design process.

Paper

LLM-Driven AutoML for Cross-Lingual Handwritten OCR: Closed-Loop Neural Architecture Search with GPT-5, GPT-4o, and Claude Sonnet 4

Mobina Kashaniyan, Amirhossein Ghassemi, Nasser Mozayani

Published in 2025 15th International Conference on Computer and Knowledge Engineering (ICCKE), IEEE.

DOI: https://doi.org/10.1109/ICCKE68588.2025.11273810

PDF: https://mobinakashaniyan.github.io/papers/llm-driven-automl-cross-lingual-handwritten-ocr.pdf

IEEE Xplore: https://ieeexplore.ieee.org/abstract/document/11273810

Keywords

handwritten text recognition, HTR, handwritten OCR, optical character recognition, multilingual OCR, Persian OCR, Arabic OCR, Arabic handwriting recognition, Persian handwriting recognition, LLM-driven AutoML, neural architecture search, AutoML, document AI, document intelligence, LLM agents, multilingual AI, low-resource languages.