SayCan

SayCan is an AI framework that grounds natural language instructions in real-world robotic actions by combining large language models with learned affordances. Its primary purpose is to enable robots to interpret high-level, open-ended commands and decide what they can and should do in a given environment, based on their actual capabilities and constraints. By integrating language understanding with physical feasibility, SayCan provides a principled way to translate human intent into executable robot behavior.

The system operates by decomposing a user’s natural language request into candidate sub-tasks and then scoring these options using both a language model and a learned value function over robot skills. SayCan leverages offline reinforcement learning to estimate which actions are possible and useful, given the robot’s prior experience and sensor inputs. This combination allows the robot to select actions that are both linguistically aligned with the instruction and physically achievable in context. The project page provides detailed methodology, experimental setups, videos of real-world deployments (e.g., household manipulation tasks), and links to the associated research paper and code resources.