Sep <var data-var='date'> 1</var>, <var data-var='time'>11:30</var> PDT Resolved - Phone carriers saw seeing elevated 502 errors when sending a message, which can appear in Avochato with the error: "Unable to create record Server error (502)." Our team has escalated this issue and it is now resolved: https://status.twilio.com

Phone carrier 502 errors

Sep <var data-var='date'> 1</var>, <var data-var='time'>09:27</var> PDT Resolved - This incident has been resolved Sep <var data-var='date'> 1</var>, <var data-var='time'>09:22</var> PDT Investigating - Avochato is currently investigating an issue where some users may be experiencing an issue logging in.

Latency and log in issues

Aug <var data-var='date'>28</var>, <var data-var='time'>20:17</var> PDT Completed - The scheduled maintenance has been completed. Aug <var data-var='date'>28</var>, <var data-var='time'>20:16</var> PDT Scheduled - Maintenance window

Maintenance

Jun <var data-var='date'>28</var>, <var data-var='time'>00:30</var> PDT Completed - The scheduled maintenance has been completed. Jun <var data-var='date'>27</var>, <var data-var='time'>23:00</var> PDT In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Jun <var data-var='date'>22</var>, <var data-var='time'>09:49</var> PDT Scheduled - Avochato will be undergoing a database maintenance procedure at this time. Users may experience a higher latency than normal, but no downtime is expected.

Database Maintenance

Oct <var data-var='date'>22</var>, <var data-var='time'>14:19</var> PDT Resolved - This incident has been resolved. Oct <var data-var='date'>20</var>, <var data-var='time'>16:42</var> PDT Monitoring - Amazon shared systems are recovered from the outage. Avochato is monitoring. Oct <var data-var='date'>20</var>, <var data-var='time'>08:15</var> PDT Identified - Amazon Web Services reported outages globally this morning which may impact services Avochato relies on for text messages and other services. Our team will be monitoring the status and providing updates here. https://health.aws.amazon.com/health/status https://www.cnn.com/business/live-news/amazon-tech-outage-10-20-25-intl

Global Amazon Outage

Jun <var data-var='date'>16</var>, <var data-var='time'>08:15</var> PDT Resolved - This incident has been resolved. Jun <var data-var='date'>16</var>, <var data-var='time'>08:00</var> PDT Monitoring - Avochato is monitoring the latency improvements as of Monday 8am PST. Jun <var data-var='date'>16</var>, <var data-var='time'>06:42</var> PDT Investigating - Latency in the product has been observed today. This may lead to longer than expected load times in inboxes across different organizations. Jun <var data-var='date'>10</var>, <var data-var='time'>08:05</var> PDT Monitoring - Our service provider Amazon Web Services has not yet identified the root of the issues on Friday. However, the Avochato team is continuing with the investigation while having performed several actions to remediate the latency issues. As of 10:55am PST on Monday June 9, latency improved for those organizations experiencing issues when refreshing inboxes. Avochato's team switched to new server instances on Monday, and will continue Tuesday with a rollout of a permanent solution. Jun <var data-var='date'> 9</var>, <var data-var='time'>10:11</var> PDT Update - The team is continuing to investigate this issue while working on options with Amazon Web Services. In the meantime, we recommend not refreshing an inbox. Once the inbox is loaded, messages will be received instantly and can be responded to in real-time. Refreshing the inbox may lead to longer delays. Jun <var data-var='date'> 9</var>, <var data-var='time'>09:18</var> PDT Investigating - Periodic latency has been observed starting at 8am PST. This may lead to longer than expected load times in inboxes across different organizations. Our team has been investigating the root cause with Amazon Web Services and will report back with findings.

Issues with Amazon Web Services

Apr <var data-var='date'>25</var>, <var data-var='time'>12:33</var> PDT Resolved - The issue was resolved at 10:40am PT. Avochato continues to monitor and investigate the root case for the latency. Apr <var data-var='date'>25</var>, <var data-var='time'>11:21</var> PDT Monitoring - Avochato is now monitoring the improvements across impacted inboxes with latency issues, which began 9:30am and resolved by 10:40am PT. Cloud provider resources are still under investigation but all impacted customers should now see normal inbox load times. Apr <var data-var='date'>25</var>, <var data-var='time'>10:32</var> PDT Update - Currently seeing improvements to load speeds, however the team is continuing to monitor and work with our cloud provider as the impacted servers may still be impacting certain organizations and inboxes. Apr <var data-var='date'>25</var>, <var data-var='time'>09:30</var> PDT Investigating - Avochato is investigating an issue with the AWS cloud provider, which may lead to latency on inbox loading. Please stay tuned for updates as the team resolves this issue asap.

Issues with cloud provider

Oct <var data-var='date'>24</var>, <var data-var='time'>15:19</var> PDT Resolved - This incident has been resolved. Oct <var data-var='date'>24</var>, <var data-var='time'>14:09</var> PDT Monitoring - A fix has been implemented and we are monitoring the results. Oct <var data-var='date'>24</var>, <var data-var='time'>04:45</var> PDT Identified - We are working on an issue with importing, saving, and creating new contacts in inboxes that use custom fields, as well as creating new fields in your inbox. As a temporary workaround, removing custom field columns from CSV uploads and/or API requests will resolve the issue. App load times may be slower than usual while we resolve the underlying issue.

Issue importing contacts with custom fields.

Sep <var data-var='date'> 4</var>, <var data-var='time'>20:49</var> PDT Resolved - Avochato continues to monitor the carrier resolution for calls across inboxes. Sep <var data-var='date'> 4</var>, <var data-var='time'>20:48</var> PDT Monitoring - Carrier call issues have been intermittent with resolutions rolling out across many numbers. Avochato is monitoring the resolution. Sep <var data-var='date'> 4</var>, <var data-var='time'>19:23</var> PDT Investigating - Avochato is investigating an issue with calls and texts that may be impacting your inbox.

Carrier issues with calls and texts

Jul <var data-var='date'>22</var>, <var data-var='time'>10:24</var> PDT Resolved - We have dug into the underlying connection issue with the Google team and patched the servers. Please contact our support team if you're still having issues accessing the analytics in your Avochato account. Jul <var data-var='date'>21</var>, <var data-var='time'>21:42</var> PDT Update - Avochato has had several conversations today with Google Cloud, and the issue has been escalated with the highest urgency. Google is still investigating the issue on their end. We will continue to share updates as soon as they become available. Jul <var data-var='date'>20</var>, <var data-var='time'>20:32</var> PDT Identified - We've identified a connectivity issue with a third-party vendor we use for visualizing inbox data. This may impact viewing graphs in the Avochato dashboard and the ability to download CSVs of those tables. Legacy analytics views are not impacted, we will share updates on this status page as a resolution is investigated by our third-party vendor.

Issues loading embedded analytics dashboards and related CSV downloads

Jul <var data-var='date'> 8</var>, <var data-var='time'>14:18</var> PDT Resolved - This issue is now resolved. Jul <var data-var='date'> 8</var>, <var data-var='time'>13:59</var> PDT Monitoring - Avochato is monitoring a fix released for select views in the inbox. Jul <var data-var='date'> 8</var>, <var data-var='time'>07:31</var> PDT Identified - Some Avochato desktop web users have observed screen-sizing issues on wide screens and in wide-mode inbox views, leading to message readability issues. Avochato is currently investigating the issue and recommends using Classic or Compact views if you are experiencing issues reading messages. Mobile web and app views are unaffected. Rest assured that all content sent and received to end-users via SMS is being delivered over SMS, as this issue is specifically a front-end web issue with specific screen widths.

Viewing issues on wide mode inbox view and on wide screen monitors

Apr <var data-var='date'>30</var>, <var data-var='time'>13:15</var> PDT Resolved - This incident has been resolved. Apr <var data-var='date'>30</var>, <var data-var='time'>10:39</var> PDT Update - We are continuing to monitor for any further issues. Apr <var data-var='date'>30</var>, <var data-var='time'>10:37</var> PDT Monitoring - Avochato is monitoring the results of resumed normal load times in the inbox. Apr <var data-var='date'>30</var>, <var data-var='time'>10:32</var> PDT Update - We are continuing to work on a fix for this issue. Apr <var data-var='date'>30</var>, <var data-var='time'>10:05</var> PDT Identified - Around 10:05am PT Avochato noticed delays in the loading of messages. This is being investigated with a resolution expected shortly.

Delays in loading inbox messages

Mar <var data-var='date'>29</var>, <var data-var='time'>20:30</var> PDT Completed - The scheduled maintenance has been completed. Mar <var data-var='date'>29</var>, <var data-var='time'>20:25</var> PDT Verifying - Verification is currently underway for the maintenance items. Mar <var data-var='date'>29</var>, <var data-var='time'>19:00</var> PDT In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Mar <var data-var='date'>25</var>, <var data-var='time'>16:17</var> PDT Scheduled - We will be undergoing scheduled maintenance during this time.

Database Update and Patch

Feb <var data-var='date'>23</var>, <var data-var='time'>08:14</var> PST Resolved - This incident has been resolved. Feb <var data-var='date'>22</var>, <var data-var='time'>07:01</var> PST Identified - AT&T’s network went down for many of its customers across the United States Thursday morning, leaving consumers unable to place calls, text or access the internet. Although Verizon and T-Mobile customers reported some network outages, too, they appeared far less widespread. T-Mobile and Verizon said their networks were unaffected by AT&T’s service outage and customers reporting outages may have been unable to reach customers who use AT&T. https://www.cnn.com/2024/02/22/tech/att-cell-service-outage/index.html

AT&T network outage

Dec <var data-var='date'>19</var>, <var data-var='time'>16:23</var> PST Resolved - On Friday 12/15, Avochato's cloud provider confirmed the issue has been identified on their end. A temporary solution was provided until the investigation is completed. Avochato will update this issue as additional information is received. Dec <var data-var='date'>14</var>, <var data-var='time'>11:45</var> PST Investigating - Delays have been observed in retrieving data from Avochato's cloud service provider. Engineering teams on both sides are actively investigating to identify the cause and a resolution.

Delays retrieving data from cloud service provider

Dec <var data-var='date'>14</var>, <var data-var='time'>11:48</var> PST Resolved - This incident has been resolved. Sep <var data-var='date'>25</var>, <var data-var='time'>15:33</var> PDT Update - Providers have identified that some Avochato phone numbers have been continuing to experience deliverability issues even with a fully verified A2P application. This is due to carrier issues with the phone number itself, not the A2P application nor an Avochato-specific issue. This remains escalated directly with carriers and Avochato will be sharing updates as they become available by carriers. Sep <var data-var='date'> 1</var>, <var data-var='time'>07:18</var> PDT Identified - Avochato has identified known issues with US carriers regarding A2P SMS/MMS traffic. Some Avochato customers that are fully approved and established on A2P may be experiencing deliverability errors due to unexpected carrier issues. Avochato is working with carriers on a solution and will post updates here.

Carrier A2P Deliverability Issues Detected

Sep <var data-var='date'>28</var>, <var data-var='time'>18:31</var> PDT Resolved - As of 1pm PT, providers were not experiencing MMS/SMS delivery delays when sending messages to Verizon and AT&T. Avochato has been monitoring results in the meantime and will share all known findings on the incident once that information is provided. This incident is now resolved. Sep <var data-var='date'>28</var>, <var data-var='time'>13:30</var> PDT Monitoring - Providers are observing successful MMS/SMS delivery to Verizon and AT&T networks in the US. Avochato will continue to monitor to ensure full service recovery and will provide additional updates. Sep <var data-var='date'>28</var>, <var data-var='time'>13:06</var> PDT Update - SMS/MMS providers are actively working on a resolution to delivery failures on Verizon and AT&T Networks in the US for a subset of Long Codes. Experts are on a bridge with a downstream telecom partner to work toward resolution. Customers may be experiencing delivery failures with a 30007 error for any traffic toward Verizon and AT&T. Avochato will continue to provide updates as they are received from providers. More details can also be found on: https://status.twilio.com. Sep <var data-var='date'>28</var>, <var data-var='time'>12:28</var> PDT Update - We have confirmed that MMS/SMS short code is not affected by the ongoing incident. We are experiencing MMS/SMS delivery failures to Verizon and AT&T networks in the US over a subset of long-codes. We are actively investigating the and will provide another update in 30 minutes or as soon as more information becomes available. Sep <var data-var='date'>28</var>, <var data-var='time'>12:20</var> PDT Investigating - We are experiencing MMS/SMS delivery failures to Verizon and AT&T networks in the US over a subset of long-codes. Our engineers are working with our carrier partner to resolve the issue. We expect to provide another update in 30 minutes or as soon as more information becomes available.

SMS/MMS Delivery Failures to Verizon and AT&T Networks in the US

Aug <var data-var='date'>18</var>, <var data-var='time'>10:45</var> PDT Resolved - This incident has been resolved. Aug <var data-var='date'>17</var>, <var data-var='time'>19:03</var> PDT Update - We are observing significantly reduced errors connecting to our carrier partner but are awaiting confirmation that the incident has been resolved. Aug <var data-var='date'>17</var>, <var data-var='time'>18:40</var> PDT Monitoring - Outbound messages for certain Avochato phone numbers are still experiencing issues sending outbound texts due to a network outage with our carrier partners. Aug <var data-var='date'>17</var>, <var data-var='time'>18:08</var> PDT Investigating - We are investigating an issue impacting outbound message delivery failures on the carrier network.

Outbound message failures

May <var data-var='date'> 5</var>, <var data-var='time'>14:52</var> PDT Resolved - Carriers have resolved the issues with MMS as of 13:41 PDT. May <var data-var='date'> 5</var>, <var data-var='time'>13:46</var> PDT Monitoring - US carriers have observed issues with MMS delivery towards US recipients using a US long code number. These issues may have impacted your Avochato number(s) on May 5. Our providers have released a patch to resolve this issue. We are in touch and expecting to receive updates on the status of the MMS issues.

MMS delivery issues (US long code numbers)

Apr <var data-var='date'>30</var>, <var data-var='time'>23:58</var> PDT Completed - The scheduled maintenance has been completed. Apr <var data-var='date'>30</var>, <var data-var='time'>23:51</var> PDT Verifying - Verification is currently underway for the maintenance items. Apr <var data-var='date'>30</var>, <var data-var='time'>23:00</var> PDT In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Apr <var data-var='date'>27</var>, <var data-var='time'>13:30</var> PDT Scheduled - We will be undergoing scheduled maintenance during this time.

Apr <var data-var='date'> 2</var>, <var data-var='time'>09:28</var> PDT Resolved - Avochato continues to investigate the incident.

Read-only Mode

Apr <var data-var='date'> 1</var>, <var data-var='time'>11:05</var> PDT Resolved - This incident has been resolved. Apr <var data-var='date'> 1</var>, <var data-var='time'>08:38</var> PDT Monitoring - A fix has been implemented and we are monitoring results. Our backlog of messages is being processed -- messages that were scheduled or sent during the impact period will deliver over the next hour. Thanks for your patience. Apr <var data-var='date'> 1</var>, <var data-var='time'>08:34</var> PDT Identified - The issue has been identified and a fix is being implemented. Apr <var data-var='date'> 1</var>, <var data-var='time'>08:02</var> PDT Investigating - We are experiencing a platform outage and Avochato is currently in read-only mode. We are working to resolve the issue as quickly as possible.

Mar <var data-var='date'>13</var>, <var data-var='time'>23:00</var> PDT Completed - The scheduled maintenance has been completed. Mar <var data-var='date'>13</var>, <var data-var='time'>22:01</var> PDT In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Mar <var data-var='date'>13</var>, <var data-var='time'>15:45</var> PDT Scheduled - We will be undergoing database maintenance and some parts of the dashboard (particularly the analytics) may experience latency.

Oct <var data-var='date'> 2</var>, <var data-var='time'>00:30</var> PDT Completed - The scheduled maintenance has been completed. Oct <var data-var='date'> 1</var>, <var data-var='time'>23:46</var> PDT In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Sep <var data-var='date'>30</var>, <var data-var='time'>11:44</var> PDT Scheduled - We will be undergoing scheduled maintenance during this time.

Database Migration

Sep <var data-var='date'>15</var>, <var data-var='time'>16:21</var> PDT Resolved - This incident has been resolved. Sep <var data-var='date'>14</var>, <var data-var='time'>16:04</var> PDT Monitoring - A fix has been implemented and we are monitoring the results. Sep <var data-var='date'>14</var>, <var data-var='time'>15:41</var> PDT Identified - We've noticed a significant spike in errors from our 3rd party service used to validate and choose new phone numbers. We're working to resolve this impact in your Avochato account.

Is Avochato down?

Avochato status is UP

Lastest published events

Phone carrier 502 errors

Latency and log in issues

Maintenance

Avochato Services, Components and Regions

Recent Avochato events

Phone carrier 502 errors

Latency and log in issues

Maintenance

Database Maintenance

Global Amazon Outage

Issues with Amazon Web Services