Gemini can now tap into your Mac for app automation

Posted

Macworld

Google announced on Thursday that its Gemini Mac app will soon get Computer Use and be able to automate Mac tasks with your installed apps without direct input. It can do all of this in the background while you do other work.

Gemini users will be able to enter a description of what needs to be done, and Gemini will do all the work in an isolated background window with a live preview in the app. You’ll be able to do other things on the Mac while Gemini is at work. Examples of the types of tasks that can be done, cited by Google, include batch color-grading photos, setting up complex 3D scenes, automated PDF form entries, data scraped from screenshots and then placed in a presentation, and batch analysis of items in a photo.

Google states that users will be able to control what apps Gemini can access, as well as pause or stop actions, and approve key actions before they are executed. While users have been able to use Apple’s Shortcuts or AppleScript to automate actions, this provides a straightforward, plain-language way for people who are intimidated by those tools.

The new automation support is being rolled out gradually as per Google’s usual updates. When I installed Gemini before writing this story, I downloaded the older version that does not have this new feature, so it kept failing when I tried to use Gemini to automate tasks in Pixelmator Pro, QuickTime Player, and the Finder. At one point, Gemini said that the version I have (version 1.147.5.1526) has “a built-in security boundary in Gemini’s Mac sandbox,” which is presumably being modified for the automation support in the new version. If you try to download the new version and keep getting the old one (like me), check back at a later time.

AI, MacOS, Personal Software