Summary
I recently wrote about how frontier AI progress might generalize to robotics progress, leading to a rapid improvement in robot control capabilities. If frontier-AI progress substantially accelerated advances in robot control, true robot capabilities might be limited by robotic hardware – how precisely finger joints can move regardless of the digital inputs they receive, for example – rather than by software. It’d therefore be useful to know what capabilities robotic hardware could support if the model controlling the robot were maximally competent. Conversely, current robot evaluations are meant to test the ability of the controller software but don’t try to isolate hardware performance.
I propose that there should be dedicated evals for the purpose of estimating the capabilities supported by a robot’s hardware. These evals would do so by trying to elicit the best possible performance using any control method, which might include an autonomous model, teleoperation, or task-specific programming. If, for example, a teleoperator can make a robot complete a task that a general-purpose autonomous policy cannot, that would demonstrate a capability that the hardware supports. The results would therefore provide a lower bound on what capabilities improved models might unlock without any change to the hardware.
The need for measuring robotics hardware performance
There are some existing robotics evals. These evals usually hold a robot body and task suite fixed, then measure how successful a particular robot policy (the model that chooses a robot’s actions) is at completing tasks like stacking blocks. These sorts of evals are really valuable for understanding progress in robot models, and I’m excited to see more work on them.
However, robotics capabilities are determined by both the robot models and the robot’s hardware – the joints, end-effectors (robot “hands”), and other body components that allow a robot to interact with the physical world. Robot hardware varies a lot, with many different physical features and specifications. For example, even within humanoids, there is lots of variation in the number of fingers on a hand, the placement of sensors on the body, whether the robot moves on legs or wheels, how many joints there are, actuator gear ratios, the material the external casing is made of, whether the robot has a head or just a torso, and much more.
Evals designed to test robot policies generally don’t isolate hardware capabilities. Policy evals might test models on sub-frontier hardware (for example, because it’s cheaper, or the very best robot embodiments are not commercially available). And if the hardware is unchanged between evals, that wouldn’t give any signal on progress in robotics hardware between generations. Even if evals test robot policies on new, custom hardware (e.g., evaluating Figure’s humanoid models on Figure’s newest humanoid robots), capabilities are determined by both software and hardware. If software (i.e., robot policies) is the bottleneck, improvements in hardware might not show up at all.
Current autonomous performance may substantially understate what existing hardware can already do on many tasks. For example, general-purpose robotics models are probably not able to juggle, but custom-trained models are able to juggle. This is a demonstration that the hardware was probably not the barrier to juggling – we just didn’t have good enough autonomous models.
I think this means there is a substantial hardware-capability overhang for many tasks: a gap between the best performance that can be elicited from an existing robot body and what current autonomous policies can elicit. Better models could unlock some of this capability without any improvement in the hardware itself. Comparing autonomous performance with the best performance we can elicit from the same body would give us a lower bound on the size of that overhang. (To be clear, I think there are many tasks that have a software overhang, not a hardware overhang. As a simple example, the bottleneck to an industrial robot picking up more than the record 2,300 kg is not the intelligence of its control policy, but simply the hardware. Nonetheless, it’s useful to know how large the hardware overhang is for the tasks that do have an overhang.)
Trying to estimate the upper limit of hardware capabilities is important, because frontier AI progress could rapidly improve robot policies.
It’s unclear how feasible AI generalization to the physical world is, or how quickly it could happen. For example, even an AI capable of automating all remote work might need real-world data collection to build functional robot action policies, so frontier AI might not speed up, say, dexterous manipulation capabilities all that much.
However, it’s possible that there could be some threshold of AI capabilities beyond which AI progress rapidly translates into robotics models. This world seems especially significant, because the pace of progress in robotics could increase significantly, even leapfrogging existing progress in building robotics models.
The overhang is not unlimited. Once software becomes capable enough to use the hardware well, physical properties would become the binding constraints, instead of the software. Even with unbounded intelligence, a robot body that is too clunky to manipulate delicate items, too weak to move heavy things, or too slow to grab moving objects could not straightforwardly perform those tasks. The physical hardware is an upper bound on how capable robots could be at the beginning of an industrial explosion.
There are currently meaningful constraints on frontier robotics hardware that probably limit the capacity of robots to do certain tasks, especially dexterous manipulation capabilities. For example, there aren’t currently hand-like actuators that have both human-level dexterity and force. Putting actuators in the fingers gives you force but the bulk of the actuators in the joints messes with dexterity; putting actuators farther up the arm requires “tendons,” which are still in early stages of development in robots (engineering tendons turns out to be hard because they introduce friction and mess with wrist flexibility). Humans are much better at this because human tendons have really low friction.
Knowing, definitively, the upper limit of robotics capabilities given certain hardware is hard. With sufficient intelligence, there may be lots of ways to creatively get around the physical constraints of an embodiment. For example, advanced AIs might come up with clever ways to use their embodiments or available tools to complete a task, or figure out how to reengineer the task to avoid the embodiment constraints. Advanced AIs might eventually figure out how to design better robot embodiments altogether. However, at least initially, it seems plausible that they’d be constrained by how good the hardware is. The most advanced capabilities we can elicit from robot hardware at a given point in time are therefore a lower bound on just how capable today’s robots might become, even if it’s not definitive.
There are various ways to measure robotics hardware performance
The easiest method is just directly measuring specs like payload (how much weight a robot can hold), repeatability (how precisely a robotic arm can move to the same coordinates over and over again), or degrees of freedom (how many independent ways a robot can move). However, this probably doesn’t give that much information on overall performance, since it’s the integration of these attributes that determines capabilities.
Another method is teleoperation. Teleoperation involves a human operator remotely controlling the motions of a robot, either through controllers or a motion capture setup. Teleoperated robots can currently do more tasks than autonomous robots running on their own policies, so teleoperation tests establish a higher lower bound on robot capabilities. Teleoperation tests are more useful than just looking at specs, since they can directly measure performance on real-world tasks. Teleoperation is also already somewhat widely used for training robots (and in some cases, helping them in real-world deployments), so there’s already infrastructure for teleoperation tests.
Teleoperation tests are imperfect, though. The main limitation is that teleoperation tests are bounded by the abilities of humans, so teleoperation success on tasks is almost certainly below the level of an unboundedly intelligent robot model. Indeed, teleoperation abilities are often worse than human abilities on the same task, for a few reasons: controllers (like gloves) are often clunkier than human hands without controllers, remote teleoperation might be latency-constrained, and teleop setups often (though not always) have no haptic feedback (e.g., lack force or tactile feedback to the operator). See, for example, teleoperators at the World Humanoid Robot Games controlling robots to open bottles – they’re okay, but definitely a lot clunkier than a human doing it themselves. Teleoperation is also more difficult to measure on non-humanoid form factors.
In the future, evals could use other types of operation to get more signal on the maximum capabilities supported by the hardware. For example, you could specifically train models or carefully program the motions for one hardware task in order to elicit the best hardware capabilities. Boston Dynamics’ demo videos are a good example of one version of this: they carefully engineer the most impressive motion sequences they can get out of their robots, which include actions that are much more advanced than what their robots can do autonomously. Compare, for example, this demo video from 2021 of Atlas robots doing synchronized backflips to this recent video of Atlas robots autonomously trying to shelve various products – the heavily engineered demo, even from four years ago, looks more physically impressive than the recent autonomous capabilities. If we were only relying on the autonomous capabilities of Atlas robots, we’d severely underestimate the maximum capabilities supported by the hardware, so these heavily engineered demos are really useful.
Right now, we don’t really have the infrastructure to scale these types of bespoke tests the way we do for teleoperation – I imagine each 3 minute Boston Dynamics demo video took hundreds of hours to engineer, and would be difficult to coordinate and compare across different embodiments. But that infrastructure could be useful to develop, especially if teleoperation performance gets saturated (for example, if tasks become too difficult for human teleoperators but could be elicited through intentional engineering).
Whatever the method, these hardware tests could be integrated into evals (more on how these might work in the next section), and function as their own benchmark. Eval orgs could come up with a suite of hardware-relevant tasks, verify that the specified hardware was able to complete the task, and create a benchmark/leaderboard of hardware progress. If the possibility of rapid AGI generalization to robotics is to be taken seriously, these would be worthwhile evals to build.
How might hardware evals actually work?
One version would be for hardware evals to be done in-house by eval orgs. An eval org could purchase a new generation of robot hardware (e.g., Boston Dynamics’ most recent Atlas robot) or partner with the company to test it, bring it into their testing suite, and verify which tasks it was able to complete. How easy it is to test a robot on a task probably depends on the eval method. For teleoperation, it’s decently easy – have trained teleoperators attempt to complete the task (ideally controlling for operator ability, familiarity with the hardware, etc.) and determine whether they can complete it successfully within the allotted time. For other methods, it might be trickier. For example, if the method is to train a robot policy to complete a specific task using specific hardware (like training the Boston Dynamics robot to do a backflip), that would presumably be a lot more labor-intensive to do in-house each time, since each task would need to be engineered separately and the policy might need to vary between different embodiments.
Another version would be for eval orgs to function more like Guinness World Records Adjudicators. To verify a world record, Guinness sends out Adjudicators (or, when feasible, reviews video footage) to observe the record attempt and make sure it’s legitimate. Hardware eval orgs could function the same way, by creating a public list of tasks and reviewing submissions by robotics companies to verify whether those tasks have been successfully completed. The World Record model would be a more feasible setup for verifying bespoke methods of eliciting hardware capabilities (like carefully hard-coding motion sequences).
The World Humanoid Robot Games that happened last week might be the best existing version of this model. The Games, held in Beijing, evaluated humanoid abilities in a wide range of competitions. Some of the events mirror traditional athletic feats, like running, but several are meant to reflect real-world tasks, like unboxing or power-tool assembly. And the tasks the competition was designed around are physically difficult in different ways, variously testing features like hand dexterity, mobility, or strength. Some of the Games’ events ban remote control or penalize it, which limit the ability to measure pure hardware ability. But many of the events are specifically geared around teleoperation. For example, teams competed to teleoperate tasks like box cutting, cable connection, bottle-cap removal, and nail fixing.
The World Humanoid Robot Games are far from perfect. The games seemed more intended to showcase progress in Chinese humanoids and support the commercialization of Chinese humanoid companies than to accurately measure hardware progress. The competition included almost no non-Chinese teams, excluding a large swath of top robotics talent. But it’s probably the most comprehensive testbed for humanoid physical abilities that we’ve seen. I think better versions of this – especially ones that span more embodiment types, are more rigorous, and include a wider pool of robotics companies – would provide useful information.
Going forward
There are a number of institutions that could house robotics hardware evals. Existing robotics eval orgs that evaluate the capabilities of robot models could expand to incorporate hardware evals, or new eval orgs could be created for the purpose of facilitating hardware evals. Government bodies, too, could become the home of hardware evals. NIST, for example, already has several programs for evaluating robotic hardware, including a robotic grasping and manipulation competition and a bin-picking testbed. I could imagine any of these setups being a good fit for running hardware evals. Whatever the institution, hardware evals would be a step towards understanding just how capable today’s robots could be.
As a hardware engineer, I keep shouting from my little rooftop:
“Hardware capabilities are what’s limiting robotics”!
They will be limiting robotics always. Because hardware advances one build at a time, one failure at a time, there’s no way around this. And then - once you demonstrate an ability to do a task once, you have to test it a million times to see how it holds up.
Good post! I would be interested how we could design a biolab or a datacenter in a way that still requires a human. Making sure even if we max out the hardware capability, there are certain tasks that only a human would be physically able to do. Would not be surprised, if the solution ends up looking like some super weird design that leverages some trait of the human body for which there simply isnt enough use to actually implement it as part of the robot.