什么是 视觉 RPA 桌面自动化？

视觉 RPA 桌面自动化代表了 Openclaw Skills 库的重大进展，使用户能够通过纯视觉解释自动化任何桌面应用程序。通过利用通义千问视觉模型 (Qwen-VL)，该技能绕过了对传统 DOM 或辅助功能 API 的需求，使其能够与遗留软件和现代应用程序进行交互。它通过分析屏幕截图来识别 UI 元素，为标准编程访问受限的跨平台自动化提供了强大的解决方案。

该技能对于需要弥合 AI 推理与本地桌面执行之间差距的开发人员和高级用户特别有价值。无论是打开特定的即时通讯应用还是导航复杂的浏览器界面，Openclaw Skills 套件中的这一项都提供了一个类人交互层，既直观又能够通过自动化分解处理复合指令。

下载入口:https://github.com/openclaw/skills/tree/main/skills/neilhexiaoning-alt/visual-rpa-skill

安装与下载

1. ClawHub CLI

从源直接安装技能的最快方式。

2. 手动安装

将技能文件夹复制到以下位置之一

优先级：工作区 > 本地 > 内置

3. 提示词安装

将此提示词复制到 OpenClaw 即可自动安装。

视觉 RPA 桌面自动化 应用场景

• 在微信等应用中自动化消息工作流，包括搜索联系人和发送消息。
• 导航浏览器以执行搜索或与非标准 Web 元素交互。
• 通过右键菜单和图标管理桌面文件和文件夹。
• 向缺乏 API 支持或辅助功能挂钩的软件输入数据。
• 编排需要在窗口和浏览器之间切换的多应用工作流。

1. 技能捕获主显示器的屏幕截图，以识别目标元素的粗略缩略图位置。
2. 生成全分辨率裁剪图，允许视觉模型执行精确的坐标精细化。
3. 系统在精细化后的坐标处执行指定的鼠标或键盘动作。
4. 执行屏幕截图验证步骤，在进入下一条指令前确认动作成功。
5. 复杂的用户提示会自动分解为原子步骤进行顺序执行。

视觉 RPA 桌面自动化 配置指南

要开始使用此技能，请确保已设置 DASHSCOPE_API_KEY 环境变量。然后，您可以通过指向 Openclaw Skills 目录结构中的脚本路径来通过 CLI 执行任务。

# 设置您的 API 密钥
export DASHSCOPE_API_KEY="your_dashscope_api_key"

# 运行简单任务
python "$env:TAXBOT_ROOT/skills/visual-rpa/scripts/visual_rpa.py" --mode task --task "点击 Chrome 浏览器"

视觉 RPA 桌面自动化 数据架构与分类体系

visual-rpa 技能将其执行数据组织在专门的日志结构中，以方便排除故障。

name: visual-rpa
description: "Visual RPA desktop automation skill. Use when user asks to operate desktop apps, click icons, open applications, type text in input fields, click buttons, scroll pages, send messages via WeChat or other apps. Uses screen capture and Qwen vision model for pure visual positioning without DOM or accessibility APIs."

Visual RPA Desktop Automation

Auto-execute all steps without waiting for user confirmation between steps.

Desktop automation via screen capture + Qwen vision model (Qwen-VL). No DOM or accessibility API needed.

How it works

1. Capture screen -> thumbnail rough positioning
2. Full-resolution crop -> precise coordinate refinement
3. Execute mouse/keyboard action -> screenshot verification
4. Compound instructions automatically decomposed into atomic steps

Usage

Use exec tool to run commands. Script path: $env:TAXBOT_ROOT/skills/visual-rpa/scripts/visual_rpa.py

Requires DASHSCOPE_API_KEY environment variable to be set.

Single task

python "$env:TAXBOT_ROOT/skills/visual-rpa/scripts/visual_rpa.py" --mode task --task "click to open WeChat"

Compound task (auto-decomposed)

python "$env:TAXBOT_ROOT/skills/visual-rpa/scripts/visual_rpa.py" --mode task --task "open WeChat, open File Transfer ch@t, type hello in input box, click send"

Multi-step task (manually specified)

python "$env:TAXBOT_ROOT/skills/visual-rpa/scripts/visual_rpa.py" --mode task --task "click Chrome browser" "type baidu.com in address bar and press enter" "type weather in search box" "click search button"

Skip verification (faster)

python "$env:TAXBOT_ROOT/skills/visual-rpa/scripts/visual_rpa.py" --mode task --no-verify --task "click to open Calculator"

Parameters

Supported actions

Instruction tips

• Be specific: "click WeChat icon on taskbar" is better than "open WeChat"
• Instructions can be in Chinese or English, the model understands both
• Complex operations can be written as compound instructions, system auto-decomposes
• For text input: say "type XXX in YYY", system auto-detects as input action

Output format

  [OK] Step 0: click to open WeChat
       click @ (375,1591)
  [OK] Step 1: click File Transfer Assistant in WeChat
       click @ (154,97)
  [FAIL] Step 2: type hello in input box
       type @ (300,1364)
  2/3 succeeded

• OK = action succeeded and verified
• FAIL = action failed or verification failed, auto-retries up to 3 times

Common scenarios

Send WeChat message

python "$env:TAXBOT_ROOT/skills/visual-rpa/scripts/visual_rpa.py" --mode task --task "open WeChat, open File Transfer Assistant ch@t, type hello in input box, click send"

Open app and navigate

python "$env:TAXBOT_ROOT/skills/visual-rpa/scripts/visual_rpa.py" --mode task --task "click Chrome browser" "type https://www.baidu.com in address bar and press enter"

Desktop operations

python "$env:TAXBOT_ROOT/skills/visual-rpa/scripts/visual_rpa.py" --mode task --task "right click on desktop blank area" "click New Folder"

Notes

• Each step takes 3-8 seconds (screenshot + API calls + verification)
• Chinese text input uses clipboard paste, will overwrite current clipboard
• Only operates on primary screen
• Logs and screenshots saved in ./rpa_logs/ directory for debugging