An analysis has found that existing benchmarks used to assess AI’s ability to perform work cover only a very small part of the real labor market. It also said it is difficult to judge that an AI that scores highly on certain tests can carry out workplace tasks overall.

Gigazine, a Japanese media outlet, reported on Oct. 19 local time that U.S. AI talent and evaluation firm Mercor said about 90 percent of U.S. labor hours correspond to tasks not sufficiently assessed by existing AI benchmarks. Mercor used the U.S. Department of Labor’s occupational information database O*NET to examine how well 29 AI benchmarks reflect real work.

O*NET divides the U.S. labor market into 923 occupations and 18,797 detailed tasks. The researchers linked individual benchmark questions to real tasks and judged a task to be within coverage if 5 or more questions corresponded to it.

The analysis found 924 tasks met that threshold, or about 4.9 percent of the total. Among the 923 occupations, only 208 had at least 1 task that was assessed.

Differences by field were also clear. Software development, law and financial analysis were assessed relatively often, but truck driving and gas plant operations were barely covered. Even within law, lawyers’ work was included relatively often, while legal secretaries and court clerks had a low share of assessment.

The researchers said benchmarks tend to focus on tasks that are easier to automate with AI and on occupations with a large economic scale. Tasks that are more likely to be affected by AI were about 6 times more likely to be included in assessments than tasks that are not.

Physical work was also cited as a blind spot in assessment. Jobs that can be done fully remotely in the United States are estimated at about 37 percent, and the rest may include on-site work and other tasks that are difficult to measure through digital tests.

The analysis, however, is an initial result based on 29 benchmarks selected by the researchers. It did not examine all AI evaluation systems.

Mercor stressed that this does not mean AI cannot perform 90 percent of U.S. work, but that current assessments make it difficult to fully understand real job performance.