For years, the goal of a technical assessment was to determine whether a candidate could solve programming problems independently. We designed coding challenges to measure technical knowledge, problem solving abilities, and coding style while trying to eliminate every possible external aid. That approach made sense when developers worked alone, but today, they don't.

‍

Modern software engineers work alongside AI every day. Whether they use ChatGPT, Claude, GitHub Copilot, Cursor, or another assistant, AI has become part of the development workflow. Ignoring that reality during the hiring process creates an artificial environment that doesn't reflect how software is actually built.

‍

It made us rethink an important question: If developers are already using AI as part of their everyday work, how should we evaluate how effectively they use it?

‍

As a software engineer and technical interviewer at Devlane, I've seen firsthand how AI has changed the way we evaluate candidates. Today, when I design a technical assessment, I'm not only interested in whether the candidate arrives at the correct solution. I'm equally interested in understanding how they collaborate with AI, how they validate its suggestions, and how they use it to improve their thinking rather than replace it.

‍

This is what AI fluency really means.

What is AI Fluency?

The term has become increasingly popular, but it's often misunderstood. Many people associate AI fluency with writing sophisticated prompts or knowing dozens of prompt engineering techniques. In my experience, that's only a very small part of the picture.

‍

A developer with strong AI fluency knows how to integrate AI into their engineering process without giving up ownership of the solution. They understand what information the assistant needs to produce useful answers. They know when AI is likely to make mistakes. They recognize incomplete or risky suggestions, verify assumptions, and continuously refine the conversation until they reach a solution they can confidently defend.

‍

On the other hand, developers with low AI fluency often treat AI as a search engine that happens to generate code. They ask vague questions, accept the first response they receive, copy code without validating it, and struggle when the assistant produces an incorrect answer.

‍

What's interesting is that both developers may eventually submit working code. If I only evaluated the final solution, I might conclude that they performed equally well, but their process tells a completely different story. That is why I believe every AI skills assessment should focus on how candidates think while using AI, not simply whether they used it.

Why traditional technical assessments no longer reflect modern software development

‍

Why traditional technical assessments coding assessments still have value, they remain one of the best ways to verify core programming knowledge, language proficiency, debugging skills, and software design fundamentals. The problem is that they were designed for a world where developers were expected to solve every problem alone. And that is no longer how engineering teams operate.

‍

In most organizations, developers regularly use AI to explain unfamiliar APIs, generate test cases, review code, write documentation, explore implementation alternatives, or simply speed up repetitive work. When the hiring process forbids AI completely, we're evaluating candidates under conditions they will never encounter after joining the company.

‍

Imagine interviewing a backend engineer and asking them to work without Stack Overflow ten years ago. Technically possible, but not representative of real life. The same thing is happening now with AI. As a tech partner focused on helping our clients build strong engineering teams, we're interested in understanding how candidates use AI to solve problems, validate ideas, and make technical decisions. That's becoming an increasingly important part of identifying the right talent for our clients.

AI fluency assessment methods for developers

At Devlane, we evaluate AI fluency through two complementary approaches. First, candidates participate in a live technical interview where they solve a coding challenge while sharing their screen and using tools such as Cursor, GitHub Copilot, or ChatGPT.

‍

During the interview, they explain their reasoning as they work. This allows the interviewer to observe how they provide context, question suggestions, validate generated code, identify errors, and incorporate AI into their solution. We complement the live interview with customized technical assessments conducted through HackerRank, an online platform used to create and deliver coding evaluations.

How we evaluate AI fluency in developers at Devlane

At Devlane, we evaluate AI fluency through two complementary approaches. We combine live technical interviews with customized technical assessments to understand not only whether a candidate reaches the right solution, but also how they use AI throughout the process.

Live Technical Interviews with AI Tools

The live technical interview gives us one of the clearest views of how a developer actually works with AI. Candidates solve a coding challenge while sharing their screen and using the AI tools they would normally work with, such as Cursor, GitHub Copilot, or ChatGPT. During the interview, they explain their reasoning as they work, which allows us to understand the decisions behind each step of the process.

‍

We pay close attention to the context they provide, the questions they ask, how they react when AI gives an incomplete or incorrect answer, whether they validate generated code, and how they decide what to incorporate into the final solution.

‍

Because the interview happens in real time, the interviewer can also ask follow-up questions and dig deeper into specific decisions. This helps us understand whether the candidate is using AI to support their engineering judgment or simply relying on the first answer they receive.

‍

The goal is to observe how naturally AI fits into their development workflow and how much technical ownership they maintain throughout the exercise. We complement the live interview with customized technical assessments conducted through HackerRank, an online platform used to create and deliver coding evaluations.

How we use HackerRank to evaluate AI Fluency

Every hiring process starts differently because every client is looking for different skills. Some need experienced React developers, others are hiring backend engineers working with Node.js or Python. And some require strong SQL knowledge, while others focus on cloud platforms or system design. Because of that, we don't rely on generic assessments.

‍

We build customized evaluations composed of multiple sections, where each section measures a different technology or competency based on the client's requirements. Candidates complete the assessment inside HackerRank's online IDE, which provides an environment very similar to what developers use every day.

‍

For selected sections of the assessment, we enable AI Fluency. Once enabled, candidates gain access to HackerRank's integrated AI assistant directly inside the IDE. They can interact with it naturally throughout the exercise, asking questions, requesting explanations, generating code suggestions, debugging problems, or refining their implementation exactly as they would during a normal workday.

‍

Instead of observing someone solving an isolated programming exercise, we're observing how they approach software development in an AI-assisted environment. After the assessment is completed, we prepare a detailed report that we share with the client together with the candidate's technical results.

That report doesn't simply indicate whether AI was used. It provides insight into how the candidate interacted with the assistant throughout the exercise. This distinction is extremely important because using AI is no longer the differentiator. Using it well is.

Interpreting AI fluency scores

HackerRank evaluates AI Fluency across three dimensions: content quality, critical thinking, and collaboration. While these scores provide valuable signals, I don't see them as the final verdict. Instead, I use them as a starting point for understanding how a candidate approaches AI-assisted development.

‍

The score tells me what happened and the interaction history tells me why it happened. That combination gives me far more confidence than looking at the submitted code alone.

Content quality

One of the biggest misconceptions about AI is that it can compensate for poor communication. In reality, AI is only as effective as the information it receives. This is why when I evaluate candidates I pay close attention to the context candidates provide when interacting with the assistant. Developers who consistently obtain useful responses rarely ask better questions simply because they're more experienced with prompting. They ask better questions because they understand the technical problem.

‍

For example, I often see prompts like this:

‍

Fix this function.

‍

Technically, the AI may still produce something useful. But compare it with this:

‍

The API returns duplicate records when pagination reaches the second page.

The repository uses offset based pagination and changing the interface isn't an option because it's shared across multiple services.

Can you identify possible causes and suggest a fix without modifying the repository contract?

‍

The difference isn't prompt engineering, the difference is engineering.

The candidate understands the problem, identifies the relevant constraints, and provides the context necessary for AI to become genuinely useful. Good developers don't ask AI to think instead of them, they give AI enough information to think with them.

Critical thinking

If I had to choose one dimension that best predicts long term success, it would be critical thinking. AI can generate impressive code in seconds but that doesn't mean the code is correct. One of the easiest patterns to identify is whether candidates immediately accept the assistant's first suggestion or whether they challenge it. Strong candidates constantly validate the AI's output, ask follow-up questions and compare different approaches.

‍

Sometimes they even reject perfectly valid suggestions because they know there is a cleaner solution that better fits the problem. I actually enjoy seeing candidates disagree with the AI, that usually tells me they understand the technology well enough to recognize tradeoffs. In contrast, candidates with weaker AI fluency often treat every response as authoritative. If the code compiles, they move on and that worries me much more than a small implementation mistake.

‍

Production software is full of situations where the first solution works but isn't the right one. The developers I want to hire are the ones who recognize that difference.

Collaboration

People sometimes assume collaboration simply means asking many questions but that's not what I look for. The best candidates build a conversation and refine the context over time.

‍

They explain what worked, what failed, narrow the problem and change direction when necessary. Watching these conversations feels surprisingly similar to watching two experienced engineers solve a problem together.

What the assessment process reveals about a developer

One of the biggest lessons I've learned is that the final code rarely tells the whole story. I've seen candidates produce elegant solutions while barely interacting with AI. I've also seen candidates rely heavily on AI and still demonstrate excellent engineering judgment because they continuously challenged the assistant, verified assumptions, rejected weak suggestions, and improved the generated code before submitting it.

‍

Then there are candidates who receive a perfectly reasonable answer from the assistant and immediately paste it into the editor without asking a single follow up question. The final solution might still compile but software engineering is not about producing code that works today, it's about making decisions you can justify tomorrow.

‍

That's why the interaction history becomes just as valuable as the implementation itself, it shows whether the candidate is thinking critically or simply outsourcing the thinking. In my experience as a software engineer and technical interviewer at Devlane, that's one of the strongest indicators of future performance when hiring engineers who will work in AI-enabled development environments

What Devlane looks for when evaluating AI skills

The AI Fluency report provides an excellent summary, but I never stop there. When I review a candidate, there are several behaviors that immediately stand out.

I pay attention to questions like these:

• Does the candidate improve their prompts as they learn more about the problem?
• Do they verify AI generated code before accepting it?
• Do they ask why something works instead of only asking how?
• Do they identify hallucinations or incorrect assumptions?
• Do they request alternatives and compare tradeoffs?
• Do they explain their reasoning before changing the implementation?

These behaviors reveal far more about a candidate's engineering judgment than the final solution alone.

AI fluency assessment in practice 

To show how this process works in practice, here's an example from one of our client assessments. One of our fintech clients was hiring senior backend engineers. The technical assessment covered several areas, including API design, SQL, and business logic.

‍

On paper, multiple candidates performed almost identically. Their implementations passed the required tests, their code quality was good, and there were no obvious differences in the final submissions. If we had stopped there, choosing between them would have been extremely difficult. The AI Fluency evaluation told a different story. One candidate used AI mostly to generate code snippets and accepted the first suggestions with very little validation. 

‍

Another candidate interacted with the assistant throughout the exercise in a completely different way, asking AI to explain tradeoffs between two approaches, questioning an optimization suggestion because it introduced unnecessary complexity, and requesting alternative implementations before deciding on one. At one point, they even rejected an AI recommendation after recognizing that it violated an important business requirement described earlier in the exercise.

‍

The final code submitted by both candidates looked equally strong, but the reasoning process behind it did not. That additional context gave both us and our client much greater confidence in the hiring decision.

AI Fluency doesn't replace technical knowledge

One concern I occasionally hear is whether AI fluency could replace traditional technical interviews. I don't think it should. At Devlane, we evaluate both technical fundamentals and AI fluency because each reveals a different part of how a developer works.

‍

Developers still need to understand algorithms, system design, architecture, debugging, testing, and software engineering principles. AI doesn't replace those skills, it amplifies them.

‍

A developer with weak technical foundations won't suddenly become a senior engineer because they know how to write prompts. Likewise, an experienced engineer who learns how to collaborate effectively with AI can often become dramatically more productive. The goal is to hire developers who know when to trust it, when to challenge it, and when to ignore it. 

Final thoughts

For years, hiring processes focused almost exclusively on what candidates could build on their own. While technical knowledge remains critical, that approach no longer reflects the reality of modern software development.

‍

At Devlane, we've adapted our technical assessments because the way developers work has changed with AI. Evaluating talent today means looking at both strong engineering fundamentals and how effectively candidates use AI to solve problems, validate ideas, and make technical decisions.

‍

That's why we intentionally create assessment environments where AI is available and provide our clients with insights that go far beyond whether the final solution passes the tests.

‍

If you're looking to build a high-performing engineering team, we'd be happy to show you how our technical assessment process works and how AI Fluency evaluations can help you make more confident hiring decisions.