Why traditional metrics fall short, and how modern frameworks like DORA and SPACE can guide better outcomes 

For years, engineering leaders have relied on familiar metrics to gauge developer performance: story points, bug counts, and lines of code. These measures offered a shared baseline, especially in Agile environments where estimation and output needed a common language. 

But in today’s AI-assisted world, those numbers no longer tell the full story. Performance isn’t just about volume or velocity. It’s about outcomes. Did the developer deliver the expected functionality, with the right quality, on time? That’s how we compensate today, and that’s still what matters. But how we measure those things must evolve.  

With tools like GitHub Copilot, Claude Code, and Cursor generating entire functions, tests, and documentation quickly, output is becoming less about what a developer types and more about what they model, validate, and evolve. 

The challenge for CIOs, CTOs, and SVPs of Engineering isn’t just adopting new tools. It’s rethinking how to measure effectiveness in a world where productivity is amplified by AI and complexity often hides behind automation. 

Why Traditional Metrics Break Down 

The future of measurement hinges on three categories: productivity, quality, and functionality. These have always been essential to evaluating engineering work. But in the AI era, we must measure them differently. That shift doesn’t mean abandoning objectivity; it means updating our tools. 

The problem isn’t that legacy metrics are useless. It’s that they’re easily gamed, misinterpreted, or disconnected from business value. 

• Story points were never meant to be performance metrics. They were team-based estimates but are often misused to compare individuals. 

• Lines of code favor verbosity over impact. In an AI-powered world, writing less can mean doing more. 

• Velocity becomes misleading when AI inflates the number of completed tasks, even if they lack strategic value. 

• Quality, or the number of bugs found during testing, can reflect poor understanding of story points and not poor coding. 

At best, these metrics create noise. At worst, they drive harmful incentives, like rewarding speed over safety, or activity over alignment. 

Today’s AI-assisted workflows lack mature solutions for tracking whether functionality requirements, like EPICs and user stories, have been fully met. But new approaches, like multi-domain linking (MDL), are emerging to close that gap. Measurement is getting smarter, and more connected, because it has to. 

The Rise of Directional Metrics 

Modern frameworks like DORA and SPACE were built to address these gaps. 

DORA (DevOps Research and Assessment) focuses on: 

• Deployment frequency 

• Lead time for changes 

• Change failure rate 

• Mean time to restore service 

These measure delivery health, not just effort. They’re useful for understanding how efficiently and safely value reaches users. 

SPACE (developed by Microsoft Research) considers: 

• Satisfaction and well-being 

• Performance 

• Activity 

• Communication and collaboration 

• Efficiency and flow 

SPACE offers a more holistic view, especially in cross-functional and AI-assisted teams. It acknowledges that psychological safety, cross-team communication, and real flow states often impact long-term output more than individual commits. 

AI Complicates the Picture 

AI tools don’t eliminate the need for metrics; they demand smarter ones. When an LLM can write 80% of the code for a feature, how do we credit the developer? By the number of keystrokes? Or by their judgment in prompting, curating, and validating what the tool produced? 

But here’s the deeper challenge: What if that feature doesn’t do what it was supposed to? 

In AI-assisted workflows: 

• Code volume no longer maps to effort 

• Execution time can be reduced, but review and validation time increases 

• Errors shift from logic bugs to alignment gaps—where delivered functionality doesn’t match requirements 

Productivity isn’t just about output; it’s about fitness to purpose. Without strong traceability between code, tests, user stories, and epics, it’s easy for teams to ship fast but fall short of the business goal. 

Many organizations today struggle to answer a basic question: Did this delivery actually fulfill the intended functionality? 

This is where multi-domain linking (MDL) and AI-powered traceability show promise. By connecting user stories, requirements, test cases, design artifacts, and even user feedback within a unified graph, teams can use LLMs to assess whether the output truly matches the input. 

And this capability unlocks more than just better alignment, it opens the door to innovation. AI-assisted development enables organizations to build more complex, interconnected, and adaptive systems than ever before. As those capabilities expand, so too must our ability to measure their economic value. What applications can we now build that we couldn’t before? And what is that worth to the business? 

That’s not a theoretical exercise. It’s the next frontier in engineering measurement. 

Productivity as a System, Not a Score 

The best engineering organizations treat productivity like instrumentation. No single number can tell you what’s working, but the right mix of signals can guide better decisions. That system must account for both delivery efficiency and functional alignment. High velocity is meaningless if the outcome doesn’t meet the requirements it was designed to fulfill. 

That means: 

• Creating dashboards that show patterns, not just totals 

• Blending technical metrics (DORA) with team dynamics (SPACE) 

• Tracking improvements over time, not absolutes per sprint 

• Using metrics as coaching tools, not judgment tools 

Most importantly, it means aligning measurement to what matters: Did the product deliver value? Did it meet its intended function? Was the effort worth the outcome? Those are the questions that still define success and the ones our measurement frameworks must help answer. 

How to Start Rethinking Measurement 

If your metrics haven’t evolved alongside your tooling, here’s how to get started: 

1. Audit your current metric stack. What are you measuring, and what are you missing? 

2. Align on outcomes. What does “good” look like for your business, not just your codebase? 

3. Pick 2–3 directional metrics from DORA or SPACE that reflect your actual goals. 

4. Baseline and benchmark. Don’t look for high scores; look for trends and improvement. 

5. Build measurement into your retros. Turn metrics into prompts for discussion, not weapons for comparison. 

AI is reshaping how software gets built. That doesn’t mean productivity can’t be measured. It means it must be measured differently. The leaders who shift from tracking motion to monitoring momentum will build faster, healthier, and more resilient engineering teams. 

Robots & Pencils: Measuring What Matters in an AI-Driven World 

At Robots & Pencils, we believe productivity isn’t a score; it’s a system. A system that must measure not just speed, but alignment. Did the output meet the requirements? Did it fulfill the epic? Was the intended functionality delivered? 

We help clients extend traditional measurement approaches to fit an AI-first world. That means combining DORA and SPACE metrics with functional traceability, such as linking code to requirements, outcomes to epics, and user stories to business results. 

Our secure, AWS-native platforms are already instrumented for this kind of visibility. And our teams are actively designing multi-domain models that give leaders better answers to the questions they care about most. 

As AI opens the door to applications we never thought were possible, our job is to help you measure what matters, including what’s newly possible. We don’t just help teams move faster. We help them build with confidence and prove it. 

How to responsibly explore tools like GitHub Copilot, Claude Code, and Cursor—without compromising privacy, security, or developer trust 

AI-assisted development isn’t a future state. It’s already here. Tools like GitHub Copilot, Claude Code, and Cursor are transforming how software gets built, accelerating boilerplate, surfacing better patterns, and enabling developers to focus on architecture and logic over syntax and scaffolding. 

The productivity upside is real. But so are the risks. 

For CIOs, CTOs, and senior engineering leaders, the challenge isn’t whether to adopt these tools—it’s how. Because without the right strategy, what starts as a quick productivity gain can turn into a long-term governance problem. 

Here’s how to think about piloting, protecting, and operationalizing AI code tools so you move fast, without breaking what matters. 

Why This Matters Now 

In a recent survey of more than 1,000 developers, 81% of engineers reported using AI assistance in some form, and 49% reported using AI-powered coding assistants daily. Adoption is happening organically, often before leadership even signs off. The longer organizations wait to establish usage policies, the more likely they are to lose visibility and control. 

On the other hand, overly restrictive mandates risk boxing teams into tools that may not deliver the best results and limit experimentation that could surface new ways of working. 

This isn’t just a tooling decision. It’s a cultural inflection point. 

Understand the Risk Landscape 

Before you scale any AI-assisted development program, it’s essential to map the risks: 

• Data leakage: Code snippets may contain proprietary logic or PII. With some tools, there’s a risk that these are logged, transmitted, or even used in model training. 

• Telemetry and usage tracking: Many tools send back usage metadata, which could raise compliance or IP concerns in regulated environments. 

• Model transparency: Enterprise IT teams often have limited visibility into how third-party LLMs are trained or updated. 

• Token costs: High-volume usage of external LLMs like Anthropic’s Claude or OpenAI’s GPT-4 can drive significant costs if left unmonitored. 

These aren’t reasons to avoid adoption. But they are reasons to move intentionally with the right boundaries in place. 

Protect First: Establish Clear Guardrails 

Protect First: Establish Clear Guardrails 

A successful AI coding tool rollout begins with protection, not just productivity. As developers begin experimenting with tools like Copilot, Claude, and Cursor, organizations must ensure that underlying architectures and usage policies are built for scale, compliance, and security. 

Consider: 

• Private repo isolation: Restrict tool access to non-sensitive codebases or open-source contributions during pilot phases. 

• In-house proxies or middle layers: Route prompt traffic through approved gateways that monitor or sanitize inputs. 

• Enterprise contracts over consumer logins: Ensure tools used by developers are under organizational agreements with clear data handling terms. 

• LLM containment strategies: For high-sensitivity environments, explore containerized models or fully managed options through secure platforms like Amazon Bedrock. Bedrock enables teams to use leading foundation models, including Anthropic’s Claude, within an enterprise-grade boundary, with no risk of model training leakage. 

For teams ready to push further, Bedrock AgentCore offers a secure, modular foundation for building scalable agents with memory, identity, sandboxed execution, and full observability, all inside AWS. Combined with S3 Vector Storage, which brings native embedding storage and cost-effective context management, these tools unlock a secure pathway to more advanced agentic systems. 

Most importantly, create an internal AI use policy tailored to software development. It should define tool approval workflows, prompt hygiene best practices, acceptable use policies, and escalation procedures when unexpected behavior occurs. 

These aren’t just technical recommendations, they’re prerequisites for building trust and control into your AI adoption journey. 

Pilot Intentionally 

Start with champion teams who can balance experimentation with critical evaluation. Identify low-risk use cases that reflect a variety of workflows: bug fixes, test generation, internal tooling, and documentation. 

Track results across three dimensions: 

• Developer experience: Does the tool actually help, or does it create new friction? 

• Code quality: Are generated suggestions valid, performant, and secure? 

• Team patterns: How do developers prompt? What guardrails do they naturally adopt or ignore? 

Encourage developers to contribute usage insights and prompt examples. This creates the foundation for internal education and tooling norms. 

Don’t Just Test—Teach 

AI coding tools don’t replace development skills; they shift where those skills are applied. Prompt engineering, semantic intent, and architectural awareness become more valuable than line-by-line syntax. 

That means education can’t stop with the pilot. To operationalize safely: 

• Embed coaching into code reviews (e.g., flagging unsafe prompt usage) 

• Create internal wikis or LLM-safe prompt libraries 

• Train tech leads on where generation helps and where it hurts 

• Build reusable workflows for common AI development scenarios 

When used well, these tools amplify good developers. When used poorly, they obscure problems and inflate false productivity. Training is what makes the difference. 

Produce with Confidence 

Once you’ve piloted responsibly and educated your teams, you’re ready to operationalize with confidence. That means: 

• Defining tool selection criteria for different project types 

• Monitoring token usage and LLM cost impact 

• Establishing a feedback loop between engineering, IT, and security 

• Treating AI-assisted development as an evolving discipline—not a one-time rollout 

Organizations that do this well won’t just accelerate development, they’ll build more resilient software teams. Teams that understand both what to build and how to orchestrate the right tools to do it. The best engineering leaders won’t mandate one AI tool or ban them altogether. They’ll create systems that empower teams to explore safely, evaluate critically, and build smarter together. 

Robots & Pencils: Secure by Design, Built to Scale 

At Robots & Pencils, we help enterprise engineering teams pilot AI-assisted development with the right mix of speed, structure, and security. Our preferred LLM, Anthropic, was chosen precisely because we prioritize data privacy, source integrity, and ethical model design; values we know matter to our clients as much as productivity gains. 

We’ve been building secure, AWS-native solutions for over a decade, earning recognition as an AWS Partner with a Qualified Software distinction. That means we meet AWS’s highest standards for reliability, security, and operational excellence while helping clients adopt tools like Copilot, Claude Code, and Cursor safely and strategically. 

We don’t just plug in AI; we help you govern it, contain it, and make it work in your world. From guardrails to guidance, we bring the technical and organizational design to ensure your AI tooling journey delivers impact without compromise.