Whisk

Whisk was a nine-person Launchpad team at Berkeley with one goal: a pair of robot arms that could make a bowl of matcha. Printing the parts and bolting them together was the tractable half. Everything after that — getting the thing to actually pick up a whisk and use it — is where the semester went, and where we ended up throwing out our entire approach.

The rig is a leader/follower pair of SO-101 arms. The leader is geared low-friction so a human can push it around by hand; the follower is high-torque, mirrors whatever the leader does, and is the one that eventually runs the policy. You demonstrate a task by driving the leader, the follower copies you, and the joint states and camera frames get recorded as a dataset. I set up and calibrated one of the arms through its full range of motion and ran teleoperation sessions to collect demonstrations, which we pushed to the HuggingFace Hub through LeRobot. Calibration sounds like setup work and isn't — the policy learns in joint space, so an arm that's off teaches the model a coordinate system that doesn't exist.

Then we tried to train something on it. ACT first, the landmark action-chunking transformer that learns a single task from scratch on about fifty demonstrations and is the standard baseline everyone starts from. Then SmolVLA. Then Pi05. All three ran into the same three walls. Training runs long enough that you get one or two attempts a day, which kills iteration. Real money per run. And performance that tracked how much data we had far more than any decision we made — ACT fried without enough demonstrations, and Pi05 was simply too big for the dataset we could realistically collect by hand. We were spending the semester feeding a model instead of building a robot.

So we pivoted. Instead of training a policy to map pixels to joint angles, write the motions as tools and let an LLM decide when to call them. Every constraint inverts at once: simulation gives you unlimited iterations, the only cost is inference, and the training-data dependency disappears entirely because there is no training. But an agent reasoning about a scene needs coordinates, not vibes, so every object got an AprilTag — bowl, chawan, whisk, spoon, canister, kettle. A camera grabs an RGB frame, the detector finds tag corners and IDs, solvePnP recovers each tag's 6DoF pose relative to the camera, a camera-to-base calibration converts that into robot base coordinates, and per-object offsets walk from where the tag is to where you'd actually grab the thing. What comes out is a dictionary: object name to x, y, z, yaw.

The agent is Gemini 3.1 Flash Lite served through Fireworks, running a loop: observe, work out what's already done and what's next, pick a tool, execute, observe again. The tool surface is small on purpose. move_arm takes a pose and a mode — pick it up, put it down, hover above it — then plans a collision-free path and traverses it in chunks. open_gripper and close_gripper, where closing is force-limited, so it stops when it meets resistance and therefore knows it has hold of something. whisk rotates the wrist. pour does what it says. After every call the agent gets execution feedback back as its next observation: success or failure, joint states, end-effector pose, and the specific errors that matter — collisions, out of reach.

What actually shipped by the end of the semester was autonomous pick-and-place and cup stacking, driven end to end by the agent, in simulation and then on the real arm. The matcha itself is still ahead of it. But the thing worth taking away isn't the demo — it's that the fastest path to a robot doing something useful ran through writing better tools, not training a bigger model.