One of the steepest barriers in enterprise Robotic Process Automation (RPA) and AI Agent development is manually writing and maintaining tool execution scripts for proprietary web dashboards and legacy desktop software.
Microsoft has open-sourced Skill Recorder, an end-to-end framework that watches a human engineer perform a task on screen once, and automatically generates a modular, self-healing SKILL.md and executable Python script that any autonomous agent can call.
1. How Trajectory-to-Skill Synthesis Works
- Multimodal Action Logging: Captures screen video frames paired synchronously with OS-level DOM events, keyboard inputs, and mouse coordinates.
- Heuristic Noise Reduction: LLM filters out unintended human actions (hesitations, accidental clicks, window resizing).
- Parameterized Code Synthesis: Converts concrete hard-coded values into dynamic arguments (e.g., converting a hard-coded invoice ID into
def process_invoice(invoice_id: str)). - Self-Healing Selectors: Generates hybrid selector strategies (combining visual template matching, ARIA labels, and relative spatial bounding boxes) to withstand future UI changes.