Maya is VP Product at Helios, a dev-first observability platform that helps dev and ops teams improve the reliability of distributed apps.
Most modern applications we use rely on a distributed architecture, but what separates the market leaders from the rest? In addition to having a great product, they also react quickly and proactively to production issues—before catastrophic failures are caused, and users are impacted.
Take the movie streaming market, for example. Recent data shows Netflix leads the market. Others are playing catch up. One reason is Netflix’s superior features that we all love—personalized movie suggestions, the ability to resume from where you left off and more.
But that’s not the only reason.
Netflix’s tech team implemented robust monitoring and distributed tracing architectures—Edgar and Telltale—which reduced their mean time to repair (MTTR) for issues encountered in production.
So how can you lower your MTTR and boost customer satisfaction?
Below, I’ll share five strategies—successfully adopted and validated by my own engineers—to remediate production issues quickly and keep your users happy and revenue flowing.
Applying Observability
Observability is being able to deduce what’s going on in a digital system from its outputs—telemetry signals like logs, metrics and traces—to forestall outages.
A key part of observability is to measure what matters. And that means going beyond traditional system metrics like CPU usage or memory usage.
Because although necessary, they don’t necessarily provide a complete overview of the health of how your customers are experiencing the application. Plus, based on your market, business objectives and KPIs, you might need to measure some custom metrics.
For example, Uber recognized the time it takes for their application to start up has a huge bearing on customer satisfaction. So, they created a metric called “startup latency,” and they tracked it.
Determine the metrics that matter most to your application and measure them. Use these questions as a guide.
• What is my product trying to achieve?
• What do I regard as an anomaly?
• Which metrics have been predictive of customer satisfaction in the past?
These answers can set you up for a solid baseline of what you’re trying to observe so that you can find the best framework and set it up.
Implementing State-Of-The-Art Monitoring Solutions
A critical step to reducing the time spent on troubleshooting is immediately knowing when there’s a problem. Otherwise, the problem could get lost in the noise, making troubleshooting hard and time-consuming. Before long, what started as a minor issue became a major emergency with a clear business impact.
Monitoring solutions track system performance and other business-specific rules to detect anomalies and alert on critical issues.
When picking out a state-of-the-art monitoring solution, ensure that:
• Developers can identify performance bottlenecks.
• It integrates seamlessly with developer workflows and supports deployment across services, environments and different components in the tech stack.
• It supports setting alerts and rules in a way that is aligned with your observability goals.
Being alerted accurately is an important step in your MTTR optimization journey.
Adopting Distributed Tracing
Distributed tracing is a method for analyzing traces across multiple services, using context propagation to identify performance bottlenecks and anomalies. It allows engineers to track the execution of a request from its origin to the destination.
Distributed tracing tools offer a wide range of capabilities to shorten MTTR, including:
• A visual representation of end-to-end requests flow and the various services they interact with.
• Reveal why a request is failing and the service responsible.
• Layers and controls to be compatible with each company’s privacy policy.
This way, engineers can quickly pinpoint the applicative flow that caused the issue.
For maximum visibility, implement the distributed tracing solution in every service, database, queue and cloud component that’s part of your distributed application.
Watch out for vendor lock! Prefer distributed tracing tools that rely on community-driven standards such as OpenTelemetry.
Providing The Right Data With The Right Context
The distributed nature of modern applications has significantly impacted organizational structures. It now takes multiple teams to develop and maintain applications—and often, those teams are siloed within the organization. Consequently, data points related to production issues are spread across various platforms, documents, communication channels, access authorities and teams.
This makes it difficult for many teams in the company, specifically for dev teams tasked with solving production issues, to investigate the issue and provide a root cause analysis in a timely manner.
Implementing observability, monitoring and distributed tracing solutions in a way that ensures developers have the right data with the right context at the right time is what unlocks the magic of slashing MTTR.
Everyone involved in the app’s development and maintenance can have a single source of truth describing the applicative flow; it’s a great way to improve collaboration and streamline the efforts of the different teams working to solve the issue.
Empowering Developers
Developer empowerment is critical to the success of any market-leading company. It’s about fostering a culture where developers are encouraged to take personal responsibility for products and display high levels of accountability.
When empowered, developers feel a sense of ownership and become invested in creating high-quality products and remediating issues when they happen.
Companies such as Google, Etsy, Figma and Airbnb have led the way when it comes to employee empowerment, resulting in massive business success. Initiatives include integrated infrastructure and internal platforms, access to data, relentlessly striving for autonomy, and focus on business outcome.
When the company’s culture aligns with personal values, employees do their best to fix issues fast when those evidently occur.
Optimize MTTR And Boost Customer Satisfaction
Having a great product is the baseline to ensuring user happiness. Swiftly addressing production issues before they impact the customer experience is just as crucial.
By following the five strategies mentioned above, you can minimize any hiccups in the way users gain value from using your product.
More importantly, a streamlined troubleshooting process can allow your developers to invest their skills and energy where they truly matter rather than in constant firefighting: building delightful products that users love.
Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?
Read the full article here
