Problem with Playing Demos

Walnut.io July 24-25, 2025 Incident Root Cause Analysis

Following the recent service disruption, we wanted to share the attached Root Cause Analysis (RCA), which outlines what occurred, how we resolved the issue, and the steps we're taking to prevent similar incidents in the future.

We know how important reliability is to your team, and we take incidents like this extremely seriously. While there is no formal SLA in place under our current agreement, our commitment is to minimize downtime and maintain platform availability well above industry standards. We've already implemented several improvements as a result of this incident, and we'll continue to strengthen our systems and processes based on the outcomes of our internal retrospective.

If you have any questions or would like to walk through the RCA together, we'd be happy to set up time. We also welcome your feedback on how we can continue to improve incident communications going forward.

Thank you again for your continued trust and partnership.

—The Walnut Team

Incident Overview

On June 30, 2025, a backend code change removed deprecated company data fields that were still referenced by existing demo configurations. This caused demo reconnection processes to fail, resulting in broken demo experiences for customers.

No customer data was lost or compromised during this incident. The issue was limited to demo configuration data and display logic.

Impact

101 demos across 59 customer accounts were affected. The affected demos would fail to load properly when accessed, displaying errors instead of the expected demo content. Demo availability and performance metrics were impacted during the incident period.

Root Cause

The incident resulted from multiple contributing factors:

• Deprecated company data fields were removed from the backend as part of a pricing project update
• Existing published demos still contained references to these removed fields
• When demos attempted to reconnect, the invalid data requests caused demo loading failures
• The system stored corrupted demo data instead of failing cleanly, leading to persistent demo issues

Timeline

• First identification: July 11, 2025 when a customer first noticed demo issues
• Mitigation began: July 18, 2025 with initial demo patches
• Full impact understood: July 21, 2025 when the complete scope was identified
• Full resolution achieved: July 25, 2025 when all affected demos were restored

Resolution

The engineering team identified that affected demos contained corrupted data with error messages. The issue was resolved by:

1. Updating all affected demos to use current data schema
2. Implementing fallback values for the removed data fields
3. Fixing error handling to prevent storing corrupted demo configurations

All affected demos were restored to working condition by July 25, 2025.

Improvements

• Implement backwards compatibility testing for deprecated fields before removal
• Add monitoring to detect when demo data contains errors for faster incident detection
• Improve error handling to fail operations cleanly rather than storing corrupted data
• Document demo reconnection processes for better incident response
• Review data dependencies in demos to eliminate unnecessary references