If I wanted to build a “grok bot”-esque application where I have each agent using a Sprite, how can I give my sprite a screen / GUI to work with that I can watch and see which actions the agent is taking
I’ve been thinking about doing something similar. My best guess is they’ve got some sort of custom desktop window manager which has maybe some extra hooks that they can talk to, but they’re also claiming it’s using vision, so that’s probably all driven via VNC, although it’s not impossible that they’ve written their own screen sharing stuff.
This is based off only having seen it in YouTube videos, I’ve not played with it myself yet.
I think for a lot of my proposed use, just using a headless Chrome via Playwright is probably going to get me most of the way.