Choosing Metrics to Report Engineering Activities to Management

Hello!

Recently, I have been working on how to explain an engineering organization’s activities and results to management.

As part of that effort, I looked at metrics organized in frameworks such as COBIT, ITIL, and ISO 27001 (ISMS). I wanted to identify metrics that would help connect engineering activities with management’s concerns when reporting on our work.

Engineering teams already quantify and document many of their activities, from measuring the Four Keys to writing incident postmortems. But how should we explain these activities when reporting to management?

I think using metrics from established, trusted frameworks is a useful approach: we can build on what already exists while providing a clear account of our work.

Table of Contents

About COBIT

COBIT (Control Objectives for Information and Related Technologies) is a framework for enterprise IT governance and management provided by ISACA.

Organizations can use it to assess and manage whether IT investments align with business strategy, risks are managed appropriately, and limited resources are used effectively. Publicly listed companies, in particular, may need to explain IT investments and risk management to shareholders and other stakeholders. COBIT looks useful as a shared framework for assessing and explaining the health of IT governance.

One feature of COBIT is that it can be used alongside other frameworks, including ITIL and ISO 27001. For example, an organization could use COBIT to structure its overall IT governance objectives and management practices, then use ITIL for IT service management and ISO 27001 for information security management.

COBIT 2019 defines 40 governance and management objectives, with deliverables and assessment metrics associated with each objective. The aim is to select and adopt objectives that fit the organization’s circumstances and goals, rather than introduce all of them uniformly. Before applying COBIT, I think it is important to understand the organization’s current situation and consider which objectives are appropriate.

I found the following article helpful (in Japanese): https://www.josis365.com/blog/cobit-it-governance-framework-guide/

About ITIL

ITIL (Information Technology Infrastructure Library) is a framework that organizes best practices for IT service management. Its purpose is to provide and manage IT services in line with business goals and customer needs. For example, it describes concepts and practices for incident management, including who responds to an outage, how service is restored, and how to work toward preventing recurrence.

ITIL defines 34 management practices across three categories:

• General management practices, such as risk management, change management, and continual improvement
• Service management practices, such as incident management, problem management, and service level management
• Technical management practices, such as software development and management, and infrastructure management

The following page provides more detail. One distinction I found particularly interesting was the separation between incident management and problem management.

ITIL 4 as a toolkit, not a rulebook

How I use ITIL 4 in infrastructure teams: the value system, value chain, 34 practices, CMDB, events and improvement, and what ITIL Version 5 changes.

punit.co.uk

Incident management aims to restore service quickly after an outage. Problem management, meanwhile, addresses underlying causes and the risk of recurrence to reduce the occurrence and impact of future incidents.

In other words, restoring service and working to prevent repeated outages are treated as activities with different purposes.

I thought this was a useful distinction to keep in mind when handling incidents in an engineering organization.

The latest version, ITIL 4, also incorporates ideas such as Agile, DevOps, and continual improvement.

This suggests that ITIL could help organizations adopt modern software development best practices as well as more traditional IT operations practices.

About ISO 27001 / ISMS

An ISMS (Information Security Management System) is a system for continuously managing information security within an organization. ISO/IEC 27001 is the international standard that defines the requirements an ISMS must meet.

Information security often brings technical measures to mind, such as vulnerability remediation, access controls, and encryption. An ISMS covers more than those measures.

Information security rests on three fundamental elements:

• Confidentiality: ensuring that information is accessible only to authorized people
• Integrity: ensuring that information is accurate and complete
• Availability: ensuring that information is available when needed

Protecting these elements matters, but eliminating every risk is unrealistic. We need to assess what risks exist and how much they could affect the business, then decide how much to invest in addressing them. That decision naturally requires a management perspective as well as a technical one.

An ISMS structures the processes for assessing information security risks and implementing appropriate measures. Third-party certification can also demonstrate externally that an organization manages information security appropriately.

ISMS certification does not guarantee that security incidents will never occur. It can, however, demonstrate that an organization has established a system for managing information security.

How Do the Three Frameworks Fit Together?

Looking at these frameworks, COBIT, ITIL, and ISO 27001 each approach IT management from a different perspective.

It seems important to choose and apply a framework based on what management wants to understand or assess.

COBIT’s Goals Cascade

When considering how to use a framework to explain engineering work to management, I found COBIT’s Goals Cascade helpful.

The Goals Cascade provides a way to translate enterprise goals into IT-related goals and connect them to governance and management objectives.

Broadly, the relationship looks like this:

flowchart LR
    A["Enterprise Goals"]
    B["Alignment Goals"]
    C["Governance and<br/>Management Objectives"]
    D["Specific Activities,<br/>Measurement, and Improvement"]

    A --> B --> C --> D

For example, suppose a company wants to bring new features to market quickly and accelerate business growth. An engineering organization might support that goal by shortening lead time for changes or increasing deployment frequency. Specific initiatives could include improving CI/CD, automating tests, or reviewing the development process.

The Goals Cascade helps position engineering activities in relation to business goals, rather than reporting them simply as technical improvements.

COBIT Objectives That Look Useful as a Starting Point

Of COBIT 2019’s 40 governance and management objectives, the following seemed particularly useful for explaining engineering activities to management.

For example, explaining improvements in development productivity could involve combining metrics associated with EDM02, APO05, and MEA01. For incident response and quality improvement, APO11, DSS02, and DSS03 could provide a useful combination.

Applying ITIL Practices

ITIL also includes several practices that can be connected to engineering activities.

What Would This Look Like in Practice?

Based on what I have learned so far, reporting to management should start with understanding management’s concerns, rather than simply listing items from a framework.

The metrics worth tracking depend on the goal. It seems important to clarify what management wants to know, then select appropriate metrics from COBIT, ITIL, and ISMS to support the explanation.

Conclusion

In this article, I explored COBIT, ITIL, and ISO 27001 as frameworks for explaining engineering activities to management.

Although they serve different purposes, I think they share a view of IT as a system for delivering business value, managing risk, and improving continuously, beyond the technical activities themselves.

COBIT’s Goals Cascade was particularly helpful for thinking about how to connect engineering activities with management goals.

At the same time, using a framework should not become a goal in itself. I think we need to consider which parts are useful based on management’s objectives.

Building on what I have learned, I would like to connect our actual engineering metrics and improvement initiatives with these frameworks so that we can better account for our work to management.