跳转至

文章背景与核心概要

传统的基于图形用户界面（GUI）的软件操作智能体长期受限于“截图-点击”（screenshot-and-click）模式，表现为视觉状态不完整、GUI动作脆弱以及难以适应长程规划。尽管代码智能体在执行脚本和管理文件方面表现优异，但面对缺乏开放API的GUI应用时，这种视觉猜测的方法效率低下且极易出错。

为此，本文介绍了 ASIL (Agent-Software Interaction Layer)，这是一种专为智能体原生设计的新型交互层。ASIL通过结构化JSON观测和代码可执行的语义动作取代了视觉截图，实现了更高的准确率并大幅减少了操作步骤。该研究已被 Findings of EMNLP 2026 录用，展示了闭源与开源模型在监督微调（SFT）和强化学习（RL）加持下达到最先进（SOTA）水平的性能提升。

ASIL: Replacing Screenshot-and-Click with Structured State and Semantic Actions

📌 Executive Summary

ASIL (Agent-Software Interaction Layer) is a novel agent-native interface designed to overcome the core inefficiencies of traditional GUI-based "screenshot-and-click" software operation. While powerful code agents can easily execute scripts and manage files, interacting with graphical user interfaces remains bottlenecked by incomplete visual states, brittle GUI actions, and poor long-horizon planning compatibility.

ASIL replaces visual guessing with structured JSON observations and code-executable semantic actions, achieving high accuracy with drastically fewer steps. Accepted to Findings of EMNLP 2026, the research demonstrates state-of-the-art performance improvements across closed and open models through supervised fine-tuning (SFT) and reinforcement learning (RL).

📖 Paper Overview

Metadata Details
Title ASIL: Replacing Screenshot-and-Click with Structured State and Semantic Actions
Authors Rui Xie, Lu Chen
Primary Subject Artificial Intelligence (cs.AI)
Conference Findings of the Association for Computational Linguistics: EMNLP 2026
Submitted August 27, 2026
Identifiers arXiv: 2608.26991
Project Page Official Project Page

🔍 Abstract

Powerful code agents can execute scripts, call tools, and manage files, yet many important applications remain accessible primarily through graphical user interfaces. We argue that screenshot-and-click is an inefficient interface for software-operating agents: screenshots are state-incomplete, and GUI actions are brittle, semantically weak, and poorly matched to long-horizon planning.

We introduce ASIL (Agent-Software Interaction Layer), an agent-native interface that exposes software through structured JSON observations and code-executable semantic actions, realized through the deepest feasible access path for each application. We instantiate ASIL across 15 applications and a benchmark of 300 single-application and 80 multi-application tasks.

ASIL reaches above 80 with closed models while executing fewer than five actions per task. Under a repaired runtime and a 50-step screenshot budget, the same tasks yield 6.6 and 26.6 strict success under screenshot-and-click control, rising to 15.0 and 53.3 on an easier OSWorld-comparable band. Against application-native interfaces on matched tasks, ASIL exceeds LibreOffice's UNO API by 28-38 strict points but only matches draw.io's MCP content contract.

The structured modality also suits training: small-scale SFT raises Qwen3.5-2B from 58.0 to 72.1 and Qwen3.5-9B from 66.6 to 80.4, and resource-limited on-policy RL further raises them to 74.4 and 82.2.

📊 Key Results & Findings

  • Efficiency: Executes tasks with fewer than 5 actions per task using structured JSON states and semantic actions.
  • Performance Gain over Screenshot-and-Click: Outperforms traditional visual setups exponentially (rising from baseline single-digit/low-double-digit metrics up to >80% success rates).
  • Model Scaling & Training:
  • Qwen3.5-2B: SFT boosts accuracy from 58.0% \(\rightarrow\) 72.1%; RL further increases it to 74.4%.
  • Qwen3.5-9B: SFT boosts accuracy from 66.6% \(\rightarrow\) 80.4%; RL further increases it to 82.2%.

🔗 Full-Text & Resource Links