Law firms have a new way to measure how good their lawyers are at AI

Law firms have a new way to measure how good their lawyers are at using AI, with a five-level framework designed to distinguish occasional users from those building sophisticated tools for entire practice areas.

Developed by legal AI consultant Peter Duffy in research commissioned by Harvey, the framework aims to put a clearer definition on what it means to be AI proficient.

Much like language qualifications distinguish between someone with basic French and someone who speaks it at C1 level, the framework gives firms a common way to describe lawyers' AI capabilities.

Rather than simply counting prompts or logins, it relies on usage data to assess how effectively lawyers apply AI to their work.

Levels of AI fluency

Duffy, founder and CEO of legal AI consultancy TITANS, previously worked in innovation roles at DWF and Deloitte. He developed the framework after years of helping law firms introduce AI tools and train thousands of lawyers.

The framework identifies five levels:

• Level 1 – Dormant: effectively a non-user who can't yet see where AI applies to their own work.
• Level 2 – Functional: uses AI occasionally for basic tasks like summarising, simple drafting and document querying.
• Level 3 – Proficient: uses AI regularly and confidently on substantive legal work, iterating and verifying for reliable output.
• Level 4 – Builder: builds reusable AI assets, such as agents and playbooks, for their own work and close colleagues.
• Level 5 – Catalyst: builds AI assets for a team, practice area or firm (or orchestrates the building) and sets the standards around them.

The biggest distinction comes between levels three and four, when lawyers move from using AI to building with it, and the levels have little to do with seniority.

“The number of years has nothing to do with the framework," Duffy told Non-Billable. "It's purely a measure of your ability to apply AI to legal work."

Beyond counting prompts

One of the framework's more interesting features is that firms can use data from their AI platforms to assess lawyers' proficiency without relying on self-assessments.

Firms can use AI to analyse their platform's usage data against the framework, automatically assigning lawyers to one of the five levels based on their activity. It also explains the evidence behind each classification, allowing firms to assess hundreds or thousands of lawyers without reviewing each person's activity manually.

"What people say they do and what they actually do can be quite different," Duffy explained, adding that "product data doesn't lie."

The framework is platform agnostic, meaning firms can apply it to data from different AI tools, provided they have access to the relevant usage analytics.

CMS is among the firms publicly discussing the framework. Bert Vries, director of innovation, knowledge and technology at CMS Netherlands, said that he had participated in the research and was becoming less interested in traditional AI adoption metrics.

"I've seen too many law firms celebrate thousands of prompts, hundreds of trained users, and near-perfect license uptake... only to discover that very little has actually changed in how people work," he wrote on LinkedIn.

Recruitment enters the chat

The framework is also beginning to find a place in recruitment.

Benelux law firm NautaDutilh has referenced the framework in an advert for two mid-level corporate M&A associates. "Want to know exactly what we are looking for?" the firm wrote, pointing candidates to the framework and saying it "captures it really well".

"Recruiting was not the initial objective behind the project, but it is actually really useful for that," Duffy said.

Rather than vaguely asking candidates to be "comfortable with AI tools", firms could specify that they want a "Proficient (L3)" associate or a "Builder (L4)", giving applicants a clearer idea of the skills expected.

The development comes as law firms grapple with AI-generated CV slop, making it harder to distinguish genuine expertise from polished applications. A defined fluency level could give employers a more concrete standard against which to assess candidates.

The framework also overlaps with the growing demand for legal engineers, who combine legal expertise with the ability to design AI tools and workflows. Levels four and five describe some of the skills associated with these increasingly sought-after roles.

The case for level three

For all the attention on advanced AI skills, Duffy is clear that most lawyers do not need to become builders.

"You don't need everybody to be building with AI," he said. "They should be focused on their work, which is serving clients."

The research suggests firms should aim to bring everyone to level three, while developing a smaller group of builders and catalysts according to their specific objectives.

Some firms are already creating space for this. Ropes & Gray allows lawyers to allocate 20% of their billable hours to be spent learning about and experimenting with AI.

Duffy also cautioned that AI fluency is not a permanent achievement. As platforms introduce new features, lawyers who fail to keep up risk slipping backwards.

"If you're not staying on top of things, your fluency can actually start to degrade and you can slide back down the fluency curve," he said.