Jan <var data-var='date'>20</var>, <var data-var='time'>20:53</var> PST Resolved - # Root Cause Analysis (RCA) Date: January 20, 2026 Status: Resolved ## Summary On January 20, 2026, we experienced a prolonged service disruption during a backend infrastructure upgrade intended to improve system capacity. While the upgrade was expected to cause only a brief interruption, an issue during the cloud provider’s upgrade process prevented the database from completing its automatic failover, extending the impact. No customer data was lost or corrupted, and full service has been restored. ## What Happened Earlier in the day, increased traffic caused elevated load on a backend database component, resulting in degraded performance. To address this, we initiated a capacity upgrade designed to improve stability and prevent further impact. During the upgrade, an internal failure occurred within the cloud provider’s managed database service. This failure prevented the database from completing its expected automatic failover. As a result, the system became unavailable for write operations longer than anticipated. The upgrade process became stuck mid-operation and required direct intervention from the cloud provider (AWS) to safely complete. ## Impact to Customers • Customers experienced service interruptions and were unable to perform write actions (such as updates or changes). • Candidates were not able to submit availability nor submit schedules • The disruption lasted longer than expected due to the failed upgrade process. • No customer data that was entered to the system was lost, corrupted, or exposed. ## Timeline (2026-01-20) • 9:00am PST - Replication lag cyclical pattern began on instance-3 • 12:04pm PST - Peak disk queue depth observed on instance-3 (114.34) • 12:13pm PST - Instance-2 began showing elevated disk queue depth • 12:57am PST - Root cause identified as instance class limitation • ~2:00pm PST - Remediation initiated: Instance class upgrade to db.r6gd.xlarge started • ~2:30 PST - Escalation: Database upgrade failed; site entered read-only mode • ~3:00 PST - Escalation: AWS Business Support engaged • ~6:30 PST - Service restored: Site returned to full read/write capability • ~19:30 PST - Full recovery: Backend workers scaled back to full capacity • ~20:00 PST - Incident closed; monitoring confirmed stable operations ## Root Cause The root cause was a combination of increased system load and a failure during a managed database upgrade performed by our cloud provider. While the upgrade was intended to be low-impact, the provider’s automation became stuck mid-process, preventing timely recovery. ## Resolution We worked directly with our cloud provider to diagnose and resolve the failed upgrade. Once the issue was corrected, database services resumed normal operation and system performance stabilized. ## Preventive Actions To reduce the likelihood and impact of similar incidents in the future, we are taking the following steps: • Increasing baseline capacity and operational headroom to better absorb traffic spikes. • Improving detection and alerting for early signs of resource saturation. • Enhancing safeguards and runbooks for infrastructure changes under load. • Working with our cloud provider to review failure scenarios and escalation procedures for managed service upgrades. ## Current Status All services are fully operational and performing normally. We continue to monitor the system closely. We take responsibility for the availability of our service and regret the extended disruption. While part of the incident involved a cloud provider failure, we are applying the lessons learned to further strengthen reliability going forward. Thank you for your patience.Jan <var data-var='date'>20</var>, <var data-var='time'>16:07</var> PST Update - Our infrastructure upgrade is taking longer than expected due to a backend synchronization step. We’re actively working with our cloud provider to complete the upgrade safely. No customer data is at risk.Jan <var data-var='date'>20</var>, <var data-var='time'>14:52</var> PST Identified - Increased demand caused performance degradation in a backend service. We’re scaling system capacity to resolve the issue and prevent recurrence. A short interruption may occur during the upgrade.Jan <var data-var='date'>20</var>, <var data-var='time'>11:01</var> PST Update - We are continuing to investigate this issue.Jan <var data-var='date'>20</var>, <var data-var='time'>10:41</var> PST Update - We are continuing to investigate this issue.Jan <var data-var='date'>20</var>, <var data-var='time'>10:40</var> PST Investigating - We are currently investigating this issue.

ModernLoop and Candidate Portal - Slow and/or Down

Aug <var data-var='date'>26</var>, <var data-var='time'>12:12</var> PDT Resolved - Root cause: Not enough servers to handle daily load. Latency would spike during the middle of the day (12pm-1pm PST). ModernLoop infra has been scaled up significantly. We expect this to be a permanent change. Apologies for the inconvenience as we diagnosed this issue.Aug <var data-var='date'>25</var>, <var data-var='time'>14:59</var> PDT Monitoring - A fix has been implemented and we are monitoring the results.Aug <var data-var='date'>25</var>, <var data-var='time'>14:53</var> PDT Investigating - We are currently investigating this issue.Aug <var data-var='date'>25</var>, <var data-var='time'>14:43</var> PDT Monitoring - A fix has been implemented and we are monitoring the results.Aug <var data-var='date'>25</var>, <var data-var='time'>14:33</var> PDT Investigating - We are currently investigating this issue.

ModernLoop Slowness

Aug <var data-var='date'>18</var>, <var data-var='time'>14:51</var> PDT Resolved - This incident has been resolved.Aug <var data-var='date'>18</var>, <var data-var='time'>13:01</var> PDT Investigating - We are currently investigating this issue.

Reports of ModernLoop Slowness

Aug <var data-var='date'>15</var>, <var data-var='time'>10:51</var> PDT Resolved - This incident has been resolved.Aug <var data-var='date'>15</var>, <var data-var='time'>10:39</var> PDT Investigating - We are currently investigating this issue.

ModernLoop Servers Not Responding

Aug <var data-var='date'>13</var>, <var data-var='time'>10:07</var> PDT Resolved - This incident has been resolved. Servers were erroneously restarting without a proper cooldown. The ModernLoop team has implemented a fix.Aug <var data-var='date'>13</var>, <var data-var='time'>10:01</var> PDT Update - We are continuing to investigate this issue.Aug <var data-var='date'>13</var>, <var data-var='time'>09:58</var> PDT Investigating - We are currently investigating this issue.

Slowness in the application

May <var data-var='date'>29</var>, <var data-var='time'>13:34</var> PDT Resolved - This incident has been resolved.May <var data-var='date'>29</var>, <var data-var='time'>13:16</var> PDT Investigating - We are currently investigating this issue.

Reports of slowness

Apr <var data-var='date'>16</var>, <var data-var='time'>19:51</var> PDT Resolved - https://status.zoom.us/incidents/pw9r9vnq5rvkApr <var data-var='date'>16</var>, <var data-var='time'>12:21</var> PDT Identified - Zoom Official Status Pages (may not load) https://status.zoom.us/ https://uptime.zoom.us/ ~50K people reporting Zoom is down for them in the last hour https://downdetector.com/status/zoom/

Zoom is down

Mar <var data-var='date'> 3</var>, <var data-var='time'>22:00</var> PST Resolved - Due to a version upgrade for our email service, some emails sent through gmail failed to deliver between 3/3 10pm PST - 3/4 7am PST

GMail email sends sporadically failing

Mar <var data-var='date'> 2</var>, <var data-var='time'>10:00</var> PST Completed - The scheduled maintenance has been completed.Mar <var data-var='date'> 2</var>, <var data-var='time'>03:00</var> PST In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.Feb <var data-var='date'>25</var>, <var data-var='time'>14:41</var> PST Scheduled - We will be migrating our search workloads to a new search cluster. Intermittent slowness and stale data is possible.

Search Infra Maintenance

Feb <var data-var='date'>26</var>, <var data-var='time'>17:16</var> PST Resolved - This incident has been resolved.Feb <var data-var='date'>26</var>, <var data-var='time'>17:15</var> PST Update - We are continuing to investigate this issue.Feb <var data-var='date'>26</var>, <var data-var='time'>08:18</var> PST Investigating - Slack is down: https://slack-status.com/2025-02/1b757d1d0f444c34

Slack is down

Feb <var data-var='date'>18</var>, <var data-var='time'>13:53</var> PST Resolved - See resolution here: https://status.lever.co/incidents/r94mjyqwbxtvFeb <var data-var='date'>18</var>, <var data-var='time'>09:27</var> PST Identified - ModernLoop team is following along here: https://status.lever.co/incidents/r94mjyqwbxtv This does not affect customers on other Applicant Tracking Systems.

Lever Hire Platform is returning 500 error

Feb <var data-var='date'>12</var>, <var data-var='time'>13:43</var> PST Resolved - This incident has been resolved.Feb <var data-var='date'>12</var>, <var data-var='time'>12:40</var> PST Update - This incident has been resolved. And we are continuing to monitor.Feb <var data-var='date'>12</var>, <var data-var='time'>12:32</var> PST Investigating - We are currently investigating this issue.

ML Slowness

Feb <var data-var='date'>12</var>, <var data-var='time'>10:03</var> PST Resolved - This incident has been resolved.Feb <var data-var='date'>12</var>, <var data-var='time'>09:54</var> PST Investigating - We are currently investigating this issue.

Site Not Loading

Feb <var data-var='date'> 5</var>, <var data-var='time'>04:19</var> PST Resolved - The issue was due to a combination of hitting our IP address limit for our Kubernetes deployment and a bad version number for our Datadog Java Agent. Datadog Java Agent is 1.46.0 on Maven but that version doesn't not exist on Google Cloud Console Artifact Registry. The incorrect version prevented new pods from starting up and the ip address allocation prevented proper visibility into this issue. Followups include: * Adding a new subnet to increase the number of IPs * Add a CI check to make sure the JAR artifact exists for the Datadog Agent Library we needFeb <var data-var='date'> 5</var>, <var data-var='time'>03:45</var> PST Monitoring - A fix has been implemented and we are monitoring the results.Feb <var data-var='date'> 5</var>, <var data-var='time'>03:22</var> PST Investigating - We are currently investigating this issue.

ModernLoop down

Jan <var data-var='date'>30</var>, <var data-var='time'>19:40</var> PST Resolved - This incident has been resolved.Jan <var data-var='date'>30</var>, <var data-var='time'>19:20</var> PST Investigating - We are currently investigating this issue.

Jan <var data-var='date'>29</var>, <var data-var='time'>20:32</var> PST Resolved - This incident has been resolved.Jan <var data-var='date'>29</var>, <var data-var='time'>17:58</var> PST Update - Load is back down. Site is performant.Jan <var data-var='date'>29</var>, <var data-var='time'>17:56</var> PST Monitoring - ModernLoop just received a high level of calendar sync events. Causing a high load.Jan <var data-var='date'>29</var>, <var data-var='time'>17:38</var> PST Investigating - Microsoft calendar sync experiencing high load. Investigating immediately.

Experienced High Load from Microsoft Outlook Webhooks

Jan <var data-var='date'>28</var>, <var data-var='time'>15:45</var> PST Resolved - We have reindexed customer's candidates to be searchable bringing full functionality to Candidates Tab and Command+K.Jan <var data-var='date'>28</var>, <var data-var='time'>10:38</var> PST Update - Currently all tabs should be populated except Candidates Tab. We are currently re-indexing our Candidate search. Candidates may not be searchable for a brief time from “Command+K” or the Candidates tab. To access candidates, please use the Tasks tab or the chrome extension until this is resolved. This process should be complete by 12pm PT. Candidate-facing flows are not impacted by this incident. Candidate-facing flows include: Candidate Portal, Candidate Self-Schedule, Candidate Submit AvailabilityJan <var data-var='date'>28</var>, <var data-var='time'>09:00</var> PST Identified - We have reindexed all objects besides Candidate searchJan <var data-var='date'>28</var>, <var data-var='time'>06:00</var> PST Investigating - ModernLoop loads with empty lists on Tasks, Candidates, Interview Modules and Employees tabs.

Empty Lists on Critical ModernLoop Tabs

Jan <var data-var='date'>27</var>, <var data-var='time'>18:29</var> PST Resolved - This incident has been resolved.Jan <var data-var='date'>27</var>, <var data-var='time'>16:24</var> PST Monitoring - A root cause analysis will be be conducted and posted as soon as possible.Jan <var data-var='date'>27</var>, <var data-var='time'>16:24</var> PST Identified - The issue has been identified and a fix is being implemented.Jan <var data-var='date'>27</var>, <var data-var='time'>14:10</var> PST Monitoring - A fix has been implemented and we are monitoring the results.Jan <var data-var='date'>27</var>, <var data-var='time'>13:05</var> PST Investigating - We are currently investigating this issue.

Jan <var data-var='date'>23</var>, <var data-var='time'>14:28</var> PST Resolved - Root Cause Analysis: Workday Integration Performance Issue Overview: On Jan 23 2023, customers experienced slower response times and intermittent service issues. We take these disruptions very seriously, and we want to share a transparent account of the issue, its root cause, and the steps we are taking to address it both in the short and long term. Root Cause: The issue was traced back to our Workday integration. Specifically: - The integration was fetching and processing an unexpectedly large volume of data. - This data processing occurred on our web servers, which caused high memory usage and led to out-of-memory errors and service outage. Impact: - Customers experienced slow application response times or outages. Fixes: Short-Term: We have optimized the processing of Workday data to reduce its memory usage and overall load on our web servers. This optimization has already been implemented and has mitigated the immediate issue. Medium-Term: To ensure long-term stability and scalability, we are in the process of moving this data processing workload off our web servers. Instead, it will be handled by dedicated worker nodes that are better equipped to process large datasets without impacting web server performance. This change is currently in progress and will be deployed as soon as possible. Next Steps: We will continue to monitor system performance closely to ensure the short-term optimizations are effective. Once the medium-term fix is deployed, we will communicate this update to all customers. We are also reviewing all other data integrations to proactively identify and mitigate similar risks. We sincerely apologize for the inconvenience caused by this issue and appreciate your patience as we work to improve our systems. Ensuring a reliable and high-performing service for our customers remains our top priority. If you have any further questions or concerns, please do not hesitate to reach out to our support team.Jan <var data-var='date'>23</var>, <var data-var='time'>10:45</var> PST Monitoring - A fix has been implemented and we are monitoring the results.Jan <var data-var='date'>23</var>, <var data-var='time'>10:23</var> PST Investigating - This incident has been resolved.

ModernLoop Service Outage

Jan <var data-var='date'>22</var>, <var data-var='time'>10:40</var> PST Resolved - This incident has been resolved.Jan <var data-var='date'>13</var>, <var data-var='time'>16:11</var> PST Monitoring - A fix has been implemented and we are monitoring the results.Jan <var data-var='date'>13</var>, <var data-var='time'>15:57</var> PST Investigating - We are currently investigating this issue.

ModernLoop Errors (502)

Dec <var data-var='date'>29</var>, <var data-var='time'>02:00</var> PST Completed - The scheduled maintenance has been completed.Dec <var data-var='date'>29</var>, <var data-var='time'>00:00</var> PST In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.Dec <var data-var='date'>21</var>, <var data-var='time'>07:05</var> PST Scheduled - We will be performing data partitioning to improve query response time for complex schedules.

Database Partitioning

Dec <var data-var='date'>28</var>, <var data-var='time'>20:50</var> PST Resolved - This incident has been resolved.Dec <var data-var='date'>16</var>, <var data-var='time'>13:03</var> PST Monitoring - The site has recovered, but we are continuing to investigate the root cause.Dec <var data-var='date'>16</var>, <var data-var='time'>12:01</var> PST Update - Site has recovered but issue is still being investigated.Dec <var data-var='date'>16</var>, <var data-var='time'>11:54</var> PST Update - We are continuing to investigate this issue.Dec <var data-var='date'>16</var>, <var data-var='time'>11:43</var> PST Investigating - We are currently investigating this issue.Dec <var data-var='date'>16</var>, <var data-var='time'>11:09</var> PST Monitoring - A fix has been implemented and we are monitoring the results.Dec <var data-var='date'>16</var>, <var data-var='time'>10:53</var> PST Investigating - We are currently investigating this issue.

Site not loading

Dec <var data-var='date'>12</var>, <var data-var='time'>09:00</var> PST Resolved - After the launch of One-click and Automated Scheduling, the site was unavailable from 8:51 am PST to 9:01

Site unresponsive after new feature launch

Oct <var data-var='date'>13</var>, <var data-var='time'>07:00</var> PDT Completed - The scheduled maintenance has been completed.Oct <var data-var='date'>13</var>, <var data-var='time'>04:00</var> PDT In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.Oct <var data-var='date'> 8</var>, <var data-var='time'>10:14</var> PDT Scheduled - We will upgrade our production Kubernetes cluster and migrating a portion of our database.

Kubernetes Upgrade & Database Migration

Sep <var data-var='date'>22</var>, <var data-var='time'>06:02</var> PDT Completed - The scheduled maintenance has been completed.Sep <var data-var='date'>22</var>, <var data-var='time'>04:00</var> PDT In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.Sep <var data-var='date'>17</var>, <var data-var='time'>10:37</var> PDT Scheduled - We will be updating our Kubernetes compute node versions.

Is ModernLoop down?

ModernLoop status is UP

Lastest published events

ModernLoop and Candidate Portal - Slow and/or Down

ModernLoop Slowness

Reports of ModernLoop Slowness

ModernLoop Services, Components and Regions

Recent ModernLoop events

ModernLoop and Candidate Portal - Slow and/or Down

ModernLoop Slowness

Reports of ModernLoop Slowness

ModernLoop Servers Not Responding

Slowness in the application

Reports of slowness

Get your FREE account now,
5 monitors included!

Is ModernLoop down?

ModernLoop status is UP

Lastest published events

ModernLoop and Candidate Portal - Slow and/or Down

ModernLoop Slowness

Reports of ModernLoop Slowness

ModernLoop Services, Components and Regions

Recent ModernLoop events

ModernLoop and Candidate Portal - Slow and/or Down

ModernLoop Slowness

Reports of ModernLoop Slowness

ModernLoop Servers Not Responding

Slowness in the application

Reports of slowness

Get your FREE account now,5 monitors included!

Get your FREE account now,
5 monitors included!