Software Engineer – Control Layer (CUA)

About the role

We are seeking a Software Engineer to develop the control layer that translates an AI model's intent into precise, reliable actions on a computer. This involves mouse movements, keyboard input, window management, UI element detection, and error recovery across macOS, Windows, and Linux.

Key responsibilities

• Work on low‑level computer control stack: mouse/keyboard injection, screen capture, coordinate mapping, input simulation.
• Implement UI element detection using accessibility APIs (AXUIElement, UI Automation), DOM/a11y trees, and visual grounding.
• Build abstraction layer enabling the agent to operate across macOS, Windows, Linux and various application types.
• Tackle reliability challenges such as element targeting under UI changes, window occlusion, resolution scaling, and cross‑app focus management.
• Develop feedback loops to verify actions succeeded and to recover from unexpected situations.
• Collaborate closely with model and planning teams on the interface between intent and execution.

Required profile

• Experience building OS‑level input automation (e.g., CGEvent, SendInput, xdotool).
• Familiarity with accessibility frameworks on macOS, Windows, and Linux.
• Proven ability to handle flaky element selectors, timing issues, and resolution‑dependent coordinates.
• Strong focus on reliability and edge‑case handling.
• Hands‑on work with automation tools such as Playwright, Appium, PyAutoGUI, or Hammerspoon.

Required skills

• CGEvent, SendInput, xdotool
• AXUIElement (macOS), UI Automation (Windows), AT‑SPI (Linux)
• Playwright, Appium, PyAutoGUI, Hammerspoon
• Screen reader internals, RDP/VNC, game automation
• LLM agent tool‑use systems, iOS UIAutomation/XCTest, Android UIAutomator/Accessibility