Mar <var data-var='date'>16</var>, <var data-var='time'>22:35</var> UTC Resolved - Around 22:00 UTC the US [Address] Autocomplete Pro API service in one of our datacenters went offline due to a misconfiguration. The problem was quickly identified and rectified. All other datacenters were unaffected.

US [Address] Autocomplete Pro API went offline briefly in one datacenter

Mar <var data-var='date'>10</var>, <var data-var='time'>15:58</var> UTC Resolved - The issue has been resolved. A small number of nodes experienced large memory consumption which resulted in a few 503 errors being returned. This has been resolved and measures have been put in place to mitigate this in the future. Additional alerts and monitors have also been put in place.Mar <var data-var='date'> 6</var>, <var data-var='time'>21:16</var> UTC Monitoring - The issue has been identified and we are currently monitoring overall system health.Mar <var data-var='date'> 6</var>, <var data-var='time'>17:59</var> UTC Investigating - We are currently investigating an increase in 503 Service Unavailable errors. We will provide an update as soon as more information is available.

Intermittent 503 Errors for US Street [Address] API

Feb <var data-var='date'>18</var>, <var data-var='time'>02:00</var> UTC Resolved - Around 19:15 UTC an incorrect configuration was deploy to our US Extract API service. The problem was quickly detected and reverted.

US Extract API briefly returned incorrect HTTP status code

Dec <var data-var='date'> 1</var>, <var data-var='time'>17:07</var> UTC Resolved - This incident has been resolved.Nov <var data-var='date'>27</var>, <var data-var='time'>00:32</var> UTC Monitoring - We've identified an issue affecting one of our redundant facilities. As a precaution, it has been taken out of active rotation while we investigate.

Intermittent HTTP 5xx Errors Returned by US Street API and US Zip Code API

Oct <var data-var='date'>31</var>, <var data-var='time'>15:35</var> UTC Resolved - This incident has been resolved.Oct <var data-var='date'>30</var>, <var data-var='time'>22:36</var> UTC Monitoring - During a routine data deployment, we noticed a brief spike in 5xx error responses. Systems have since stabilized, and we're investigating the root cause to confirm the issue is fully resolved and to prevent similar behavior in future deployments.

Jul <var data-var='date'>18</var>, <var data-var='time'>16:25</var> UTC Resolved - We have become aware that Google services have been exhibiting some intermittent failures. As a result, anyone using "Login with Google" to access their account dashboard on our website may also experience a failure to login. When Google is able to resolve the issues they are having, the login will continue to work as normal. In the meantime, if you have created a username and password, you should be able to use those to login.

Intermittent failures with Google login (oauth) integration

May <var data-var='date'>15</var>, <var data-var='time'>14:00</var> UTC Resolved - We discovered that the "API Keys" page in our Account dashboard web site was not able to retrieve the lists of keys from the backend service. This has now been fixed.

Account page not listing API keys

Mar <var data-var='date'>18</var>, <var data-var='time'>20:01</var> UTC Resolved - Around 19:10 UTC, increased traffic exposed resource limitations caused by a previously implemented algorithm update, leading to issues on several servers hosting some of our services (US Street API, US Extract API) in one of our data centers due to insufficient RAM. To ensure stability and service reliability, the affected servers have been decommissioned and replaced with upgraded hardware.

Misbehaving servers resulted in some 503 responses

Feb <var data-var='date'>26</var>, <var data-var='time'>17:00</var> UTC Resolved - The US Enrichment API intermittently returned HTTP 500 codes in a single data center on 2025 February 26 around 17:00 UTC. The issue has been resolved as of 17:35 UTC.

Intermittent HTTP 500 codes returned by US Enrichment API

Feb <var data-var='date'> 7</var>, <var data-var='time'>21:15</var> UTC Resolved - Around 18:00 UTC, one of our facilities experienced a network routing issue that lasted for about 30 minutes. The facility was removed from DNS rotation until the network issue had been resolved. If you have not yet read through our "Technical Requirements" and "Best Practices" documentation, this could be a good opportunity to become familiar with those important and helpful documents. - https://www.smarty.com/docs/cloud/requirements#retry - https://www.smarty.com/docs/cloud/best-practices

Network outage in one facility

Jan <var data-var='date'>10</var>, <var data-var='time'>21:12</var> UTC Resolved - From 01/08 12pm UTC to 01/09 9am UTC some requests to the US Street [Address] API instances experienced an intermittent issue with certain malformed addresses that caused some requests to not get a response, returning HTTP 503 errors to those requests. This affected a small number of requests and the error has since been resolved.

Intermittent HTTP 503 Errors

Dec <var data-var='date'>30</var>, <var data-var='time'>22:32</var> UTC Resolved - During a regular upgrade of our control plane, several controller nodes became unresponsive. This resulted in various routing issues which were manifest by intermittent HTTP 503 errors visible in the HTTP response to some callers. Our monitoring systems alerted our Operations Engineering Team of the anomaly and, as a result, the affected cluster, located in our US Central region (Chicago), was immediately removed from production rotation. The affected cluster has been restored to full health by provisioning new cloud resources using our IaC (infrastructure as code) systems. Following this, the cluster was confirmed to be was healthy via our automated monitoring and was subsequently re-introduced into active rotation.

Intermittent HTTP 503 Errors:

Dec <var data-var='date'>19</var>, <var data-var='time'>23:00</var> UTC Resolved - On 2024 December 19, around 22-24:00 UTC, some users may have received the HTTP 299 code in response to requests from one of our loadbalancers. The reason for this response has been identified and remediated. We would like to take this opportunity to remind our users to familiarize themselves with our best practices. https://www.smarty.com/docs/cloud/best-practices

HTTP 299 response code

Nov <var data-var='date'>28</var>, <var data-var='time'>17:00</var> UTC Resolved - Around 15:00 (3pm) UTC on 2024 November 28, a single IP address at one of our datacenters experienced a routing issue that caused some requests to be "null routed", which may have resulted in some clients experiencing 500 level errors during that time. This issue persisted until 17:00 (5pm) UTC on 2024 November 28, at which time our upstream provider resolved the issue. Our recommendation to clients to mitigate an issue such as this in the future is to maintain a pool of active TCP connections to the server nodes and to only send traffic over active, healthy TCP connections. Then a background "thread" (or programmatic equivalent) could attempt to connect to the suspect IPv4 address. Once that connection has determined to be healthy, regular traffic can be resumed over that TCP connection.

Single IP address caused some traffic to be null routed

May <var data-var='date'>31</var>, <var data-var='time'>21:41</var> UTC Resolved - As previously announced, JSONP support in US Autocomplete Pro API is being removed today. See previous announcements: - https://status.smarty.com/incidents/hftb2zdrfn3q - https://status.smarty.com/incidents/m8pc40fcq3c5

JSONP support removed from US Autocomplete Pro API

Apr <var data-var='date'> 5</var>, <var data-var='time'>18:08</var> UTC Resolved - No. We have evaluated our systems and determined that we are not at risk.

[INFO] Is Smarty affected by the CVE-2024-3094 (xz utils attack)?

Mar <var data-var='date'> 8</var>, <var data-var='time'>22:49</var> UTC Resolved - A configuration change to an older version of International Address Autocomplete API caused a few users to receive inaccurate 402 responses. The configuration was identified and rectified. Users of the older version of this API should no longer be receiving the erroneous response. Users of the new version of the API were unaffected.

International Address Autocomplete API: Configuration Fix Resolves 402 Response Issue

Mar <var data-var='date'> 1</var>, <var data-var='time'>23:11</var> UTC Resolved - Some modifications were made to TLS certification configurations on one of our load-balancers. Part of the modification adjusted the order in which certificates were presented to clients. Older applications that are unable to properly use Server Name Indication may have been affected adversely by this change. In order to continue supporting older clients, the certificate order was readjusted to maximize compatibility. We suggest updating any applications that call Smarty APIs to be compatible with modern TLS certificate usage. We also suggest ensuring that the hostnames being used in requests to our APIs are current. Hostnames can be found on the documentation pages for the products that you are using. (https://www.smarty.com/docs/cloud)

Older applications affected briefly by TLS modifications

Feb <var data-var='date'>29</var>, <var data-var='time'>17:30</var> UTC Resolved - During a regularly scheduled deployment of an update to our US Address Verification API and US Address Autocomplete API in one of our datacenters, a misconfiguration caused the services to not respond properly. The problem was quickly observed and remedied.

One datacenter returned 503 responses briefly

Jan <var data-var='date'>31</var>, <var data-var='time'>18:56</var> UTC Resolved - As previously announced, JSONP support in US Street Address API is being removed today.

JSONP support removed

Jan <var data-var='date'>22</var>, <var data-var='time'>19:00</var> UTC Resolved - On 2024 January 22 US Street Address API experienced some unexpected resource constraints that briefly caused a few intermittent 503 errors for a handful of customers. These restraints have been resolved.

Intermittent 503 responses from US Street Address API

Nov <var data-var='date'>15</var>, <var data-var='time'>19:57</var> UTC Resolved - A new endpoint with new capabilities has been released. As a result of this change, the older (existing) endpoint is being deprecated and will be removed from service. Please see our documentation regarding this mandatory and breaking changes: - https://get.smarty.com/intl-autocomplete-migrate - https://www.smarty.com/docs/cloud/international-address-autocomplete-api

BREAKING CHANGE: International [Address] Autocomplete API

Nov <var data-var='date'>14</var>, <var data-var='time'>17:22</var> UTC Resolved - JSONP is officially deprecated. Support will be removed for JSONP starting on 2024 January 31.

JSONP is officially deprecated

Nov <var data-var='date'> 6</var>, <var data-var='time'>21:39</var> UTC Resolved - During the last three days we have been observing a significant increase in CPU utilization of various application servers. As a result of this CPU utilization, we made some adjustments to the software running our proxy API servers to better watch for traffic causing the CPU spikes. As part of this remediation effort, at 2023-11-05 00:54 UTC, a new version of our proxy API with this new behavior was deployed to a single cluster. After the “canary” release appeared to be functioning correctly according to available metrics, we rolled the change out to a second cluster roughly 15 minutes later. Finally, roughly 21 hours later, at 2023-11-05 21:45 UTC nearly, we rolled out the change to the remaining third cluster. At 2023-11-06 01:40 UTC, we received notice from approximately four different customers of some intermittent issues wherein an HTTP request to our US Street API (https://us-street.api.smarty.com) would return an HTTP 400 response status. Within five minutes after that notice, at 2023-11-06 01:45 UTC, we rolled back the above changes to a working version of the proxy API across our entire fleet. Since that time, the underlying bug within the proxy API introduced and originally deployed at 2023-11-05 00:54 UTC was resolved and a new version of the software that is unaffected by this issue was deployed to a single cluster today at 2023-11-06 16:50 UTC. Additional tests and monitoring have been put in place to proactively watch for similar kinds of issues in the future.

US Street API - Intermittent HTTP 400 Responses

Jul <var data-var='date'>12</var>, <var data-var='time'>14:00</var> UTC Resolved - One of our IP addresses became unresponsive. Investigation showed that the IP was configured properly, however the connection of the IP, to the load balancer to which it was assigned, was incomplete. This situation started when one of our nodes went offline and the IP was automatically assigned to a different node (a common and normal occurrence). When the node that went offline came back online, the IP was reassigned to the original node (also normal). While the assignment was correctly executed, the IP provider did not properly complete the routing of traffic (abnormal and unusual). As we do not control the IP routing at the provider level, we are implementing additional monitoring checks that will allow us to detect and mitigate this unusual situation in the future.

Is Smarty down?

Smarty status is UP

Lastest published events

US [Address] Autocomplete Pro API went offline briefly in one datacenter

Intermittent 503 Errors for US Street [Address] API

US Extract API briefly returned incorrect HTTP status code

Smarty Services, Components and Regions

Recent Smarty events

US [Address] Autocomplete Pro API went offline briefly in one datacenter

Intermittent 503 Errors for US Street [Address] API

US Extract API briefly returned incorrect HTTP status code

Intermittent HTTP 5xx Errors Returned by US Street API and US Zip Code API

Intermittent HTTP 5xx Errors Returned by US Street API and US Zip Code API

Intermittent failures with Google login (oauth) integration

Get your FREE account now,
5 monitors included!

Is Smarty down?

Smarty status is UP

Lastest published events

US [Address] Autocomplete Pro API went offline briefly in one datacenter

Intermittent 503 Errors for US Street [Address] API

US Extract API briefly returned incorrect HTTP status code

Smarty Services, Components and Regions

Recent Smarty events

US [Address] Autocomplete Pro API went offline briefly in one datacenter

Intermittent 503 Errors for US Street [Address] API

US Extract API briefly returned incorrect HTTP status code

Intermittent HTTP 5xx Errors Returned by US Street API and US Zip Code API

Intermittent HTTP 5xx Errors Returned by US Street API and US Zip Code API

Intermittent failures with Google login (oauth) integration

Get your FREE account now,5 monitors included!

Get your FREE account now,
5 monitors included!