ViiBE Netsapiens Integration

Incident Report for Viirtue

Postmortem

On Thursday, February 1st at approximately 5:00AM EST Google upgraded our GKE Cluster. The updates were within the allotted maintenance window, however the upgrade caused incompatibilities with service mesh infrastructure which provides secured networking connectivity between our on-prem datacenters and our cloud provider.

The event affected various parts of ViiBE that rely on the Netsapiens integration. Most impacting was the inability to generate billing period invoices for that had associated voice plans.

We worked diligently to upgrade service mesh components and restore connectivity at which points ViiBE experienced 2 short windows of intermittent during the component updates. Once connectivity was restored, we triggered invoice creation and normal system function returned.

To prevent this from happening in the future, we have committed to regularly evaluating and adopting upstream upgrades of all major platform components on a quarterly basis.

Posted Feb 07, 2024 - 14:20 EST

Resolved

Our monitoring has indicated the integration is stable. We will follow up with a comprehensive post mortem early next week after we've received final updates / statements from CSI and Google.

Posted Feb 02, 2024 - 09:22 EST

Update

We are continuing to monitor for any further issues.

Posted Feb 01, 2024 - 16:51 EST

Monitoring

A fix has been implemented and we are monitoring the results.

Posted Feb 01, 2024 - 16:51 EST

Update

Due to the Netsapiens integration issue, accounts that have voice plans (Usage Rating) did not generate invoices. Because the invoices did not generate the billing items for February will not show until the invoices are generated.

We are continuing to restore functionality and will ensure all of the aforementioned are resolved today.

Posted Feb 01, 2024 - 15:07 EST

Update

We are continuing to work on a fix for this issue.

Posted Feb 01, 2024 - 14:48 EST

Identified

Yesterday's Netsapiens integration was restored but a mismatch in Istio (networking service mesh) versions is causing intermittent connection issues between our on-prem and cloud infrastructure. We are actively working to improve responsiveness and any connectivity issues.

Posted Feb 01, 2024 - 10:44 EST

This incident affected: ViiBE.