CSI Tax Engine Unavailable
Incident Report for Viirtue
Postmortem

On Saturday, January 27th at approximately 2AM EST CSI (Viirtue’s tax rating

engine partner) was the victim of a ransomware attack. The event caused

CSI’s infrastructure and services to become unavailable. ViiBE’s billing and

quoting capabilities started to experience error conditions as they directly

rely on CSI’s APIs.

To keep ViiBE operational, Viirtue implemented a kill switch that would allow

us to bypass tax rating until CSI recovered. This action restored the ability to

manage quotes, billing periods, and orders at the expense of taxes not being

computed due to the CSI APIs being down.

When CSI’s tax rating services were restored at approximately January 31th

9:30 EST, we proactively updated usage and re-rated any uninvoiced billing

periods. Subsequently, we generated invoices for any billing periods that

were slated to be invoiced. Once billing periods were invoiced, autopay and

other system functions were shown to be operating nominally.

To summarize a communication from CSI’s CEO: Despite having 6+ years of

successful SOC audits, bad actors found an attack vector they successfully

exploited. CSI engaged a third party to perform forensics and assist with

brining up a new and hardened environment. Additionally, they have

implemented a new disaster recovery process that would take hours and not

days to restore.

Posted Feb 07, 2024 - 14:19 EST

Resolved
This incident has been resolved.
Posted Feb 01, 2024 - 10:40 EST
Update
We are continuing to monitor for any further issues.
Posted Jan 31, 2024 - 08:36 EST
Monitoring
CSI has reported that the production environment is functional again. We have confirmed it is working with ViiBE and will continue to monitor the integrations health.
Posted Jan 30, 2024 - 21:55 EST
Update
CSI continues to remain down and the latest update from CSI is: "Still working to bring up the tax calc APIs at a minimum level for tax rating. Have faced several complications in this regard."
Posted Jan 30, 2024 - 07:20 EST
Update
The CSI team has communicated that they are anticipating having their environment functional late this evening. We will help facilitate testing and confirm that it's fully functional again, in partnership with CSI when they're ready. We will provide another update as soon as as we can.

Thank you for your continued patience.
Posted Jan 29, 2024 - 16:36 EST
Update
We continue to wait for a confirmation of resolution from our partner CSI. In the interim, we're keeping ViiBE "operational" by providing empty tax calculations for billing periods and quotes. We've also disabled the automated creation of invoices until this issue is resolved, at which point we'll recompute taxes for all open billing periods.

NOTE: Creation / Updates to billing periods, orders, quotes will not include taxes until the issue is resolved. However, you'll no longer receive errors when attempting the aforementioned.
Posted Jan 29, 2024 - 08:33 EST
Identified
The tax engine used to power ViiBE is currently unavailable. We've escalated the issue to the vendor (CSI) and will keep the community posted. Billing and Quoting features in ViiBE will not function correctly until this issue is rectified.
Posted Jan 28, 2024 - 09:51 EST
This incident affected: ViiBE.