Computer use

I ran a local computer-use model on OSWorld, a benchmark of real desktop tasks, on one RTX 3060. The model was GUI-Owl-1.5-8B at 4-bit. On 50 tasks, in one afternoon, it passed 26.

Of the 24 misses, 17 were the model saying it was done when it was not. I read seven of those by hand: the instruction, then the checker, then the model's own words step by step, then the screens. Three had the wrong plan. Two had the right plan and the wrong hand: it typed 100 next to a 5 and got 1005, and it dragged a slider right while saying left. One gave up. One was the benchmark's own setup failing before the first step, and the harness scored the model anyway. The checker was unfair in none of them.

The model also finishes a task and then spends the rest of its step budget checking its own work: clicking the desktop, searching for the folder it just renamed. At the harness's own default settings it stops one step after finishing.

Fixing what I found became four pull requests to OSWorld, one merged: a reply cut off before its closing code fence was being dropped as no action. The setup failure is another one, still open.

Next: the same model through two harnesses, several runs each, and a count of what moved. The write-up comes with it.