Observability is Part of Software Development Now: Allocate 30% for Long-Term Success - Part 2

This article is the second part of our blog series on effective observability strategies. If you missed the first part, you can find it here

In today's software development landscape, the understanding of observability and its necessity has grown clearer for many companies. However, despite this awareness, many still fall into the 80/20 rule, where 80% of organizations mistakenly consider regular monitoring as observability. In this article, I want to emphasize why implementing observability should be regarded as a continuous process and a crucial component of software development projects.

Although traditional DevOps, QA, and security silos are becoming less common, observability-related integrations frequently remain isolated from cross-functional processes and are often not allocated the necessary time in development schedules. Moreover, these practices remain largely unstandardized across developer teams. All too frequently, companies view tracing, metrics, and logging as tasks to be completed only after the code has been finished, creating significant challenges.

In 2024, everyone now understands that profiling, logging, and metrics form a critical triad in a successful observability strategy, each playing a significant role. Though, it's still often considered as something with which to wrap up the code after development and QA are done. This misconception results in a rushed implementation, treating observability as an afterthought rather than an integral part of the development process.

With this situation, organizations still rely on observability tools as something that provides observability, considering code instrumentation as something that supports the tool. Moreover, they often rely on the tool's background and features, assuming it will offer insights and help prevent bugs, degradations, and outages. This vision is flawed because it overlooks the importance of embedding observability practices into the development process itself. Treating observability as merely tool-based can lead to limited visibility, overlooking essential data and failing to address potential issues proactively.

To understand the scope and the amount of effort truly required to instrument observability, the OpenTelemetry Demo is an excellent resource. Being developed by the OpenTelemetry community, the OpenTelemetry Demo represents an ideal way of the right observability instrumentation. The application consists of a number of microservices written in Go, Java, .NET, C++, Python, and other programming languages. If you conduct a statistical code analysis, you'll find that about 30 to 45 percent of the code base is observability instrumentation related.

This amount of application logic is stunning. Of course, that doesn't represent the same amount of effort in terms of time for the application to be developed, but we see that it's a crucial part of the development process. It's clear that implementing observability as a part of the development process takes a big amount of time at all steps of application development:

• Onboarding, Standardizing, and Education: Apart from providing some standards across different teams on how the typical service architecture, data structures, and communication between services should look, development teams now need to standardize how metrics are provided, profiling is established, and log reporting is done.
• Development and QA Loop: As observability code now represents 30 to 45 percent of the codebase, many potential mistakes can be introduced, so the QA process must include the codebase with observability already implemented. Code reviewers should also have the necessary experience to review how metrics, tracing, and logging are implemented by developers.
• Observability QA as an Additional Process: In addition to QA of the application logic, the QA of data provided by the observability stack needs to ensure it appears correctly in observability tools. Feedback should be incorporated to make any necessary fixes.
• Security Considerations: Security teams must ensure no sensitive data is accidentally sent to logging and other systems, and if such data is needed in the logs, it complies with government standards like the DMCA.
• DevOps and SRE Integration: Alerting and response practices should be architected with the operations team. Critical service logic needs to be monitored via observability data, defining what the critical metrics are and what information developers require to locate issues rapidly during outages.

Let's take a look at an example of an application development plan based on an application similar to the OpenTelemetry Demo, with time estimations both with and without observability instrumentation.

Development Plan Without Observability Instrumentation

1. Architecture Design (40 hours)

• Service definitions and data flow (20 hours)
• API contracts between services (10 hours)
• Database schema design (10 hours)

2. Development Phase (150 hours)

• Microservices coding (120 hours)
• API gateway and communication setup (20 hours)
• Database integration (10 hours)

3. Quality Assurance and Testing (40 hours)

• Unit testing (15 hours)
• Integration testing (15 hours)
• Load testing and performance optimization (10 hours)

4. Security Implementation (20 hours)

• Implementing security protocols (15 hours)
• Vulnerability assessments (5 hours)
• Deployment and Release Management (20 hours)

5. Infrastructure provisioning and configuration (15 hours)

• Final deployment and documentation (5 hours)
• Total Estimated Time Without Observability: 270 hours

Development Plan With Observability Instrumentation

1. Architecture Design (50 hours)

• Service definitions and data flow (20 hours)
• API contracts between services (10 hours)
• Database schema design (10 hours)
• Observability architecture design (10 hours)

2. Development Phase (180 hours)

• Microservices coding (120 hours)
• API gateway and communication setup (20 hours)
• Database integration (10 hours)
• Observability instrumentation coding (30 hours)

3. Quality Assurance and Testing (60 hours)

• Unit testing (15 hours)
• Integration testing (15 hours)
• Load testing and performance optimization (10 hours)
• Observability QA and data validation (20 hours)

4. Security Implementation (25 hours)

• Implementing security protocols (15 hours)
• Vulnerability assessments (5 hours)
• Sensitive data audit for observability instrumentation (5 hours)

5. Deployment and Release Management (30 hours)

• Infrastructure provisioning and configuration (15 hours)
• Final deployment and documentation (5 hours)
• Observability tools configuration and alerting setup (10 hours)

Total Estimated Time With Observability: 345 hours

The development plan with observability requires an estimated 75 additional hours compared to the plan without it, translating to nearly a 28% increase in effort. This additional time includes designing an observability architecture, implementing telemetry code across all services, performing dedicated observability QA, and configuring alerting and monitoring tools.

It is crucial to recognize that this additional effort is not a mere overhead but rather an investment in the long-term success and reliability of the application. By dedicating the necessary time and resources to observability instrumentation during the development phase, organizations can reap the benefits of improved monitoring, faster issue detection and resolution, and ultimately, a more stable and performant application. The proactive approach to observability helps prevent costly downtime, enhances user experience, and allows for more efficient resource allocation.

Moreover, the standardization of observability practices across development teams promotes consistency, facilitates collaboration, and enables a shared understanding of the application's behavior. By incorporating observability as an integral part of the development process, organizations can foster a culture of transparency, accountability, and continuous improvement. This shift in mindset, along with the proper allocation of time and effort, is essential for organizations to truly leverage the power of observability and stay competitive in today's rapidly evolving software development landscape.

So, what to do? If you're a business leader, you need to understand and ensure that observability implementation requires significant time and must be integrated into the continuous development process. If you're a manager or developer, allocate this time and communicate with the business to ensure it's prioritized. Business leaders need to recognize that observability is a long-term investment in quality and reliability, not a simple checklist item. Foster a culture that prioritizes monitoring and ensures collaboration across engineering, operations, and business teams to align observability goals with performance and customer experience.

Managers and developers should advocate for the time required to implement observability comprehensively and explain how this investment leads to faster debugging, reduced downtime, and a better user experience. Build observability into daily workflows, incorporating standards for logging, metrics, and tracing early in the development process. QA should treat observability data as first-class, verifying its accuracy and consistency. By embedding these practices, the organization will establish a culture of transparency, accountability, and continuous improvement, ultimately delivering resilient, high-performing applications that meet customer expectations. I hope this article will help.

At ApexData, our mission is to empower companies to embrace this purpose-oriented approach to observability. We're developing an observability product aimed at simplifying instrumentation by providing a developer-focused experience. We believe in creating a system that simplifies the complex, making it easier for teams to implement a robust observability framework that aligns with their operational goals and user expectations. Our solution ensures that teams can integrate observability seamlessly into their workflows, allowing them to monitor and improve system performance with ease.