Observability is Part of Software Development Now: Allocate 30% for Long-Term Success - Part 2

Evgeny Potapov

This article is the second part of our blog series on effective observability strategies. If you missed the first part, you can find it here

In today's software development landscape, the understanding of observability and its necessity has grown clearer for many companies. However, despite this awareness, many still fall into the 80/20 rule, where 80% of organizations mistakenly consider regular monitoring as observability. In this article, I want to emphasize why implementing observability should be regarded as a continuous process and a crucial component of software development projects.

Although traditional DevOps, QA, and security silos are becoming less common, observability-related integrations frequently remain isolated from cross-functional processes and are often not allocated the necessary time in development schedules. Moreover, these practices remain largely unstandardized across developer teams. All too frequently, companies view tracing, metrics, and logging as tasks to be completed only after the code has been finished, creating significant challenges.

In 2024, everyone now understands that profiling, logging, and metrics form a critical triad in a successful observability strategy, each playing a significant role. Though, it's still often considered as something with which to wrap up the code after development and QA are done. This misconception results in a rushed implementation, treating observability as an afterthought rather than an integral part of the development process.

With this situation, organizations still rely on observability tools as something that provides observability, considering code instrumentation as something that supports the tool. Moreover, they often rely on the tool's background and features, assuming it will offer insights and help prevent bugs, degradations, and outages. This vision is flawed because it overlooks the importance of embedding observability practices into the development process itself. Treating observability as merely tool-based can lead to limited visibility, overlooking essential data and failing to address potential issues proactively.

To understand the scope and the amount of effort truly required to instrument observability, the OpenTelemetry Demo is an excellent resource. Being developed by the OpenTelemetry community, the OpenTelemetry Demo represents an ideal way of the right observability instrumentation. The application consists of a number of microservices written in Go, Java, .NET, C++, Python, and other programming languages. If you conduct a statistical code analysis, you'll find that about 30 to 45 percent of the code base is observability instrumentation related.

This amount of application logic is stunning. Of course, that doesn't represent the same amount of effort in terms of time for the application to be developed, but we see that it's a crucial part of the development process. It's clear that implementing observability as a part of the development process takes a big amount of time at all steps of application development:

Let's take a look at an example of an application development plan based on an application similar to the OpenTelemetry Demo, with time estimations both with and without observability instrumentation.

Development Plan Without Observability Instrumentation

  1. Architecture Design (40 hours)
  1. Development Phase (150 hours)
  1. Quality Assurance and Testing (40 hours)
  1. Security Implementation (20 hours)
  1. Infrastructure provisioning and configuration (15 hours)

Development Plan With Observability Instrumentation

  1. Architecture Design (50 hours)
  1. Development Phase (180 hours)
  1. Quality Assurance and Testing (60 hours)
  1. Security Implementation (25 hours)
  1. Deployment and Release Management (30 hours)

Total Estimated Time With Observability: 345 hours

The development plan with observability requires an estimated 75 additional hours compared to the plan without it, translating to nearly a 28% increase in effort. This additional time includes designing an observability architecture, implementing telemetry code across all services, performing dedicated observability QA, and configuring alerting and monitoring tools.

It is crucial to recognize that this additional effort is not a mere overhead but rather an investment in the long-term success and reliability of the application. By dedicating the necessary time and resources to observability instrumentation during the development phase, organizations can reap the benefits of improved monitoring, faster issue detection and resolution, and ultimately, a more stable and performant application. The proactive approach to observability helps prevent costly downtime, enhances user experience, and allows for more efficient resource allocation.

Moreover, the standardization of observability practices across development teams promotes consistency, facilitates collaboration, and enables a shared understanding of the application's behavior. By incorporating observability as an integral part of the development process, organizations can foster a culture of transparency, accountability, and continuous improvement. This shift in mindset, along with the proper allocation of time and effort, is essential for organizations to truly leverage the power of observability and stay competitive in today's rapidly evolving software development landscape.

So, what to do? If you're a business leader, you need to understand and ensure that observability implementation requires significant time and must be integrated into the continuous development process. If you're a manager or developer, allocate this time and communicate with the business to ensure it's prioritized. Business leaders need to recognize that observability is a long-term investment in quality and reliability, not a simple checklist item. Foster a culture that prioritizes monitoring and ensures collaboration across engineering, operations, and business teams to align observability goals with performance and customer experience.

Managers and developers should advocate for the time required to implement observability comprehensively and explain how this investment leads to faster debugging, reduced downtime, and a better user experience. Build observability into daily workflows, incorporating standards for logging, metrics, and tracing early in the development process. QA should treat observability data as first-class, verifying its accuracy and consistency. By embedding these practices, the organization will establish a culture of transparency, accountability, and continuous improvement, ultimately delivering resilient, high-performing applications that meet customer expectations. I hope this article will help.

At ApexData, our mission is to empower companies to embrace this purpose-oriented approach to observability. We're developing an observability product aimed at simplifying instrumentation by providing a developer-focused experience. We believe in creating a system that simplifies the complex, making it easier for teams to implement a robust observability framework that aligns with their operational goals and user expectations. Our solution ensures that teams can integrate observability seamlessly into their workflows, allowing them to monitor and improve system performance with ease.

Subscribe to our blog to get the latest updates