Sep <var data-var='date'>29</var>, <var data-var='time'>08:13</var> EDT Resolved - RTB is now Operational Sep <var data-var='date'>29</var>, <var data-var='time'>08:07</var> EDT Identified - Our developers have found the problem and are working on resolving it. Sep <var data-var='date'>29</var>, <var data-var='time'>08:03</var> EDT Investigating - A small subset of users are seeing errors when making RTB requests

Users are seeing degraded performance of RTB

Sep <var data-var='date'>28</var>, <var data-var='time'>02:05</var> EDT Completed - The scheduled maintenance has been completed. Sep <var data-var='date'>26</var>, <var data-var='time'>02:03</var> EDT In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Sep <var data-var='date'>26</var>, <var data-var='time'>02:01</var> EDT Scheduled - We will be restarting databases instances in order to activate necessary configuration changes. We expect very brief downtime of all Retreaver services. This maintenance period has been chosen due to historically low traffic in order to reduce impact on customers.

Database maintenance

Sep <var data-var='date'>23</var>, <var data-var='time'>19:13</var> EDT Resolved - At 21:00 UTC, a percentage of RTB traffic began returning timeouts. Overall traffic volume was unaffected, though some publishers may have received timeout responses during the incident. The issue was fully resolved by 22:30 UTC.

RTB reporting timeouts

Aug <var data-var='date'> 5</var>, <var data-var='time'>17:01</var> EDT Resolved - This incident has been resolved. Aug <var data-var='date'> 5</var>, <var data-var='time'>13:35</var> EDT Monitoring - A fix has been implemented and we are monitoring the results. Aug <var data-var='date'> 5</var>, <var data-var='time'>12:43</var> EDT Investigating - Calls and other services are operating fine, we are just seeing an elevated amount of errors on the account portal so marking as degraded.

Degraded performance on account portal

Aug <var data-var='date'> 3</var>, <var data-var='time'>13:29</var> EDT Resolved - This incident has been resolved. Aug <var data-var='date'> 3</var>, <var data-var='time'>10:26</var> EDT Investigating - RTB Inbound Request Delay Inbound RTB requests on the dashboard are currently experiencing higher-than-usual delays. The delay is usually under a minute, but it is currently taking significantly longer to update.

RTB reporting delay

Jul <var data-var='date'>14</var>, <var data-var='time'>14:04</var> EDT Resolved - This incident has been resolved. Jul <var data-var='date'>14</var>, <var data-var='time'>13:44</var> EDT Monitoring - API service is back to operational, we are continuing to monitor our servers before closing out this status issue. Jul <var data-var='date'>14</var>, <var data-var='time'>11:54</var> EDT Update - We are experience issues with our host and it is now impacting our account portal and JavaScript. Our engineers are working to bring everything back online and will provide an update soon Jul <var data-var='date'>14</var>, <var data-var='time'>11:20</var> EDT Investigating - Investigating some service issues with our API endpoints

API Issues

Jul <var data-var='date'>13</var>, <var data-var='time'>15:28</var> EDT Resolved - RTB service has been stable for the past 2hrs, marking as resolved. Jul <var data-var='date'>13</var>, <var data-var='time'>14:02</var> EDT Update - We are continuing to monitor for any further issues. Jul <var data-var='date'>13</var>, <var data-var='time'>13:29</var> EDT Monitoring - Service to RTB is improving and we are continuing to monitor it. Jul <var data-var='date'>13</var>, <var data-var='time'>13:06</var> EDT Update - Our engineering team is continuing to work on restoring RTB to full performance. Jul <var data-var='date'>13</var>, <var data-var='time'>12:38</var> EDT Identified - We are investigating an issue with our RTB servers.

RTB performance is degraded

Jul <var data-var='date'> 7</var>, <var data-var='time'>18:17</var> EDT Resolved - This incident has been resolved. Jul <var data-var='date'> 7</var>, <var data-var='time'>15:11</var> EDT Investigating - We are currently investigating this issue.

API requests return slower than normal

Jul <var data-var='date'> 2</var>, <var data-var='time'>19:07</var> EDT Resolved - This incident has been resolved. Jul <var data-var='date'> 2</var>, <var data-var='time'>18:14</var> EDT Monitoring - API service is back online and we are monitoring Jul <var data-var='date'> 2</var>, <var data-var='time'>18:02</var> EDT Investigating - After restoring service we detected another regression in API servers and are investigating. Jul <var data-var='date'> 2</var>, <var data-var='time'>17:57</var> EDT Update - We detected some our API servers went offline, our engineers responded, and service has been restored. We will continuing monitoring for a period of time before marking this as resolved. Jul <var data-var='date'> 2</var>, <var data-var='time'>17:53</var> EDT Monitoring - A fix has been implemented and we are monitoring the results. Jul <var data-var='date'> 2</var>, <var data-var='time'>17:43</var> EDT Investigating - We are currently investigating this issue.

Investigating issue with our API servers

Jun <var data-var='date'>28</var>, <var data-var='time'>07:15</var> EDT Completed - The scheduled maintenance has been completed. Jun <var data-var='date'>28</var>, <var data-var='time'>03:15</var> EDT In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Jun <var data-var='date'>28</var>, <var data-var='time'>03:02</var> EDT Scheduled - We will be restarting databases instances in order to activate necessary configuration changes. We expect very brief downtime of all Retreaver services. This maintenance period has been chosen due to historically low traffic in order to reduce impact on customers.

Jun <var data-var='date'>21</var>, <var data-var='time'>19:35</var> EDT Completed - The scheduled maintenance has been completed. Jun <var data-var='date'>21</var>, <var data-var='time'>15:35</var> EDT In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Jun <var data-var='date'>21</var>, <var data-var='time'>15:30</var> EDT Scheduled - We will be restarting databases instances in order to activate necessary configuration changes. We expect very brief downtime of all Retreaver services. This maintenance period has been chosen due to historically low traffic in order to reduce impact on customers.

May <var data-var='date'>20</var>, <var data-var='time'>02:10</var> EDT Resolved - This incident has been resolved. May <var data-var='date'>19</var>, <var data-var='time'>18:20</var> EDT Identified - We are working on resolving an issue with our AI model vendor. For the time being disposition analysis and AI chat features on learn.retreaver.com are currently offline. We will mark this as resolved once Retreaver AI services are restored.

AI Analyst temporarily offline

Apr <var data-var='date'>24</var>, <var data-var='time'>09:07</var> EDT Resolved - Improvements have been deployed and clients should not be experiencing this. Apr <var data-var='date'>24</var>, <var data-var='time'>04:55</var> EDT Investigating - Requests that were send through the night might have returned HTTP 504. If some ingest processes are dependent on this please be advised that these requests will have to be send again once the incident is resolved.

api.retreaver.com is degraded and might return results slower

Apr <var data-var='date'>23</var>, <var data-var='time'>09:09</var> EDT Resolved - This incident has been resolved. Apr <var data-var='date'>23</var>, <var data-var='time'>07:58</var> EDT Monitoring - Incident should be resolved. We are monitoring and confirming. Apr <var data-var='date'>23</var>, <var data-var='time'>07:19</var> EDT Update - A third party service involved in the reporting are experiencing issues. Working with them on a resolution. Apr <var data-var='date'>23</var>, <var data-var='time'>06:57</var> EDT Investigating - The issue is currently under investigation. Please use Call 1.0

Call/RTB Reporting dashboard 2.0 is not loaded

Mar <var data-var='date'>17</var>, <var data-var='time'>04:30</var> EDT Completed - The scheduled maintenance has been completed. Mar <var data-var='date'>17</var>, <var data-var='time'>02:30</var> EDT In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Mar <var data-var='date'>17</var>, <var data-var='time'>02:15</var> EDT Scheduled - We will be restarting databases instances in order to activate necessary configuration changes. We expect very brief downtime of all Retreaver services. This maintenance period has been chosen due to historically low traffic in order to reduce impact on customers.

Mar <var data-var='date'> 8</var>, <var data-var='time'>11:00</var> EDT Completed - The scheduled maintenance has been completed. Mar <var data-var='date'> 8</var>, <var data-var='time'>05:00</var> EDT In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Mar <var data-var='date'> 8</var>, <var data-var='time'>04:26</var> EDT Scheduled - We will be restarting databases instances in order to activate necessary configuration changes. We expect very brief downtime of all Retreaver services. This maintenance period has been chosen due to historically low traffic in order to reduce impact on customers.

Mar <var data-var='date'> 8</var>, <var data-var='time'>04:00</var> EDT Completed - The scheduled maintenance has been completed. Mar <var data-var='date'> 8</var>, <var data-var='time'>03:00</var> EDT In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Mar <var data-var='date'> 8</var>, <var data-var='time'>01:50</var> EST Scheduled - We will be restarting databases instances in order to activate necessary configuration changes. We expect very brief downtime of all Retreaver services. This maintenance period has been chosen due to historically low traffic in order to reduce impact on customers.

Feb <var data-var='date'>25</var>, <var data-var='time'>14:00</var> EST Resolved - The incident has been resolved. It is occurring for a limited amount of time. Feb <var data-var='date'>25</var>, <var data-var='time'>12:55</var> EST Investigating - Over the last three days, we have observed a recurring industry-wide performance dip between 11:30 AM and 12:30 PM EST. During this window, several external platforms and agencies are experiencing significant RTB latency. Because Retreaver processes these incoming requests and routes them to buyers, this upstream slowdown is resulting in occasional “heavy load” alerts or timeouts. To mitigate this, Ping Shield is actively scaling its protection to stabilize the system. These bursts typically last for 10–15 minutes before service returns to normal. Based on the data we will release a few additional improvements to Ping Shield to help protect against them.

RTB heavy load

Feb <var data-var='date'>20</var>, <var data-var='time'>17:44</var> EST Resolved - Carrier marked as operational Feb <var data-var='date'>20</var>, <var data-var='time'>14:04</var> EST Monitoring - A fix has been implemented and we are monitoring the results. Feb <var data-var='date'>20</var>, <var data-var='time'>13:36</var> EST Identified - A 'carrier' which is used by another 'call tracking provider' which isn’t Retreaver or Twilio is currently experiencing an incident.

Degraded performance on calls

Jan <var data-var='date'>22</var>, <var data-var='time'>08:27</var> EST Resolved - The incident has been resolved the benchmark report ran in a satisfactory amount of time Jan <var data-var='date'>22</var>, <var data-var='time'>08:06</var> EST Investigating - There has been and instance of a report with degraded performance.

Reports slowdown detected

Jan <var data-var='date'> 2</var>, <var data-var='time'>02:30</var> EST Completed - The scheduled maintenance has been completed. Jan <var data-var='date'> 2</var>, <var data-var='time'>01:30</var> EST In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Jan <var data-var='date'> 2</var>, <var data-var='time'>01:23</var> EST Scheduled - We will be restarting databases instances in order to activate necessary configuration changes. We expect very brief downtime of all Retreaver services. This maintenance period has been chosen due to historically low traffic in order to reduce impact on customers.

Dec <var data-var='date'>28</var>, <var data-var='time'>16:30</var> EST Completed - The scheduled maintenance has been completed. Dec <var data-var='date'>28</var>, <var data-var='time'>06:30</var> EST In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Dec <var data-var='date'>28</var>, <var data-var='time'>06:24</var> EST Scheduled - We will be restarting databases instances in order to activate necessary configuration changes. We expect very brief downtime of all Retreaver services. This maintenance period has been chosen due to historically low traffic in order to reduce impact on customers.

Nov <var data-var='date'> 2</var>, <var data-var='time'>02:41</var> EST Completed - The scheduled maintenance has been completed. Nov <var data-var='date'> 2</var>, <var data-var='time'>02:33</var> EST Verifying - Verification is currently underway Nov <var data-var='date'> 2</var>, <var data-var='time'>01:30</var> EST In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Nov <var data-var='date'> 2</var>, <var data-var='time'>01:24</var> EST Scheduled - We will be restarting databases instances in order to activate necessary configuration changes. We expect very brief downtime of all Retreaver services. This maintenance period has been chosen due to historically low traffic in order to reduce impact on customers.

Oct <var data-var='date'>21</var>, <var data-var='time'>02:39</var> EDT Resolved - This incident has been resolved. Oct <var data-var='date'>20</var>, <var data-var='time'>15:44</var> EDT Monitoring - We see no issues on our side. At the same time AWS reports that services are still impacted. Oct <var data-var='date'>20</var>, <var data-var='time'>14:12</var> EDT Update - We see calls not connected. We've identified that this is caused by the same issue experienced across the board with Twilio (https://status.twilio.com/) and AWS (https://status.aws.amazon.com/) Oct <var data-var='date'>20</var>, <var data-var='time'>12:44</var> EDT Identified - Monitoring the results. Oct <var data-var='date'>20</var>, <var data-var='time'>12:44</var> EDT Monitoring - Monitoring the results. Oct <var data-var='date'>20</var>, <var data-var='time'>12:35</var> EDT Investigating - We are currently experiencing service disruptions across several product lines due to an ongoing AWS outage in their US1 Region. You can track AWS’s official status here: https://health.aws.amazon.com/health/status This incident is impacting many of our connected services and telephony providers, resulting in disruptions across multiple areas of our platform. The issue stems from an ongoing AWS outage in their US1 Region and is the same as the one from a few hours ago.

AWS degraded

Oct <var data-var='date'>20</var>, <var data-var='time'>08:58</var> EDT Resolved - We see no issues on Retreaver at the moment despite the fact the that there are 64 services with degraded performance on AWS. Oct <var data-var='date'>20</var>, <var data-var='time'>07:24</var> EDT Monitoring - Call are processing again. Continuing monitoring Oct <var data-var='date'>20</var>, <var data-var='time'>04:24</var> EDT Update - We are continuing to investigate this issue. Oct <var data-var='date'>20</var>, <var data-var='time'>04:20</var> EDT Investigating - We are currently experiencing service disruptions across several product lines due to an ongoing AWS outage in their US1 Region. You can track AWS’s official status here: https://health.aws.amazon.com/health/status This incident is impacting many of our connected services and telephony providers, resulting in disruptions across multiple areas of our platform. The issue stems from an ongoing AWS outage in their US1 Region.

Is Retreaver down?

Retreaver status is UP

Lastest published events

Users are seeing degraded performance of RTB

Database maintenance

RTB reporting timeouts

Retreaver Services, Components and Regions

Recent Retreaver events

Users are seeing degraded performance of RTB

Database maintenance

RTB reporting timeouts

Degraded performance on account portal

RTB reporting delay

API Issues