NJ Datacenter

Incident Report for Viirtue

Postmortem

Incident Report: NJ Data Center Maintenance and Outage
Date of Incident: July 12, 2025
Prepared by: Alberto Fernandez, Director of Platform Engineering
Start Time: 7:53 AM EST
Resolution: 8:15 PM EST
Impacted Services: Apollo Services (NJ Region)
Customer Impact: Possible voice degradation, call setup delays, or dropped sessions during rerouting

Executive Summary

At approximately 7:53 AM EST, our monitoring systems detected that several locations were unable to reach our New Jersey data center.

Our team immediately initiated an investigation with both our ISP and the data center provider. We discovered that a scheduled maintenance procedure involving the Uninterruptible Power Supply (UPS) had not gone as planned. Although the maintenance was expected to have no impact, it resulted in full connectivity loss at the site.

To mitigate impact, we quickly rerouted traffic and relocated portal services to our Florida data center. This transition began at 8:05 AM, with the reseller portal fully redirected by 8:15 AM. While the data center had issued a notification about the maintenance, the notice included an incorrect date, August 11, and stated there would be no service interruption.

Root Cause

The outage was directly caused by the data center’s UPS maintenance. While intended to be non-disruptive, the maintenance led to a full outage of inbound and outbound connectivity. We are awaiting a detailed report from the data center.

Remediation and Preventative Measures

  • We are working closely with the data center provider to ensure future maintenance notices are timely, accurate, and clearly indicate any potential for impact.

  • All third-party maintenance affecting core infrastructure will now be reviewed under our internal change management protocols, even when the work is outside of our direct control.

Customer Communication

We understand that even short periods of service instability can disrupt your business. We remain committed to reliability, clear communication, and continuous improvement across our platform.

If you have any questions or concerns, or would like to discuss this incident in more detail, please reach out to our support team directly. We value your partnership and thank you for your continued trust.

Posted Jul 22, 2025 - 17:00 EDT

Resolved

All services have been restored and are operational.
Posted Jul 12, 2025 - 21:24 EDT

Update

All services have been fully restored in our NJ data center. We are currently in the process of normalizing the environment, including migrating DNS records back to NJ.
Posted Jul 12, 2025 - 20:46 EDT

Update

starting maintenance
Posted Jul 12, 2025 - 17:07 EDT

Update

We will be performing scheduled maintenance on our NJ infrastructure today at 5 PM EST. All services will remain fully operational and will continue running from our Florida data center during this time. The customer portal will also remain accessible and will continue to be served from our FL data center.
Posted Jul 12, 2025 - 14:45 EDT

Monitoring

All services are fully operational in our secondary data center.
Posted Jul 12, 2025 - 11:07 EDT

Update

Our engineers, in collaboration with on-site data center staff, are working to ensure full service restoration. All possible services have been migrated to our Florida data center, and certificate restoration for white-label partners is underway. All other services remain operational. The data center provider has provided an estimated time of resolution for NJ normalization by 1 PM EST.”
Posted Jul 12, 2025 - 10:21 EDT

Update

We are continuing to work on a fix for this issue.
Posted Jul 12, 2025 - 08:43 EDT

Identified

Our engineers have traced the issue to the data center and are coordinating with on-site staff for deeper diagnostics and resolution. The data center reported ongoing preventative UPS maintenance, though it’s not yet confirmed to be related. All services, including the portal, remain fully operational through our redundant Florida data center.
Posted Jul 12, 2025 - 08:42 EDT

Investigating

Our monitoring systems have identified an issue impacting the NJ data center. Our engineering team is actively investigating and working to resolve the incident.
Posted Jul 12, 2025 - 08:24 EDT
This incident affected: Apollo (NJ2 Core Server, Recording NJ).