BRISA Reading Group 07 - Estimate, Don't Imitate: Reusing Differentiable State-Based Policies for Visuomotor Control

Today, we have a special one: Denis Shcherba presents his own paper, Estimate, Don’t Imitate: Reusing Differentiable State-Based Policies for Visuomotor Control (Shcherba et al., 2026), written right here in Berlin with Marc Toussaint (TU Berlin) and Wojciech Samek (Fraunhofer HHI). A rare chance to ask the author directly why things were done the way they were.

​In simulation a robot can cheat: it knows exactly where every object is. Policies trained with this privileged state learn contact-rich behaviour well, but a real robot only gets noisy camera images.

​The usual fix is teacher-student distillation, where a vision policy learns to copy the expert’s actions. That forces the student to do two jobs at once: work out the state from pixels, and relearn a mapping from state to action that the expert already has.

​This paper keeps the expert and only learns a visual state estimator.

​The catch is that an error in the estimate can matter a lot or not at all, depending on what the expert does with it. So the estimator is trained on two signals: plain state regression, and an action-consistency loss backpropagated through the frozen, differentiable expert.

​Across five goal-conditioned manipulation tasks, keeping the expert consistently beats direct pixel-to-action imitation trained on the same demonstrations. On a real Panda arm it reaches 76% success, without retraining the expert.

​No preparation needed.

​Come with questions for the author, disagreements or your own view on end-to-end versus modular.

​🎟️ There are only a limited number of spots, so we can keep the session personal and interactive. Please register early and only attend if you’ve received a ticket.

​📍 Hosted at the Merantix AI Campus on November 23rd. Doors open 18:30, we start at 18:45.

‍