Jul <var data-var='date'>16</var>, <var data-var='time'>17:08</var> PDT Resolved - This incident has been resolved by our engineering team. Jul <var data-var='date'>16</var>, <var data-var='time'>17:03</var> PDT Investigating - We've received reports of errors when attempting to download the Doppler CLI. We're currently investigating the issue.

CLI Downloads failing

Nov <var data-var='date'>20</var>, <var data-var='time'>11:46</var> PST Resolved - This incident has been resolved. Nov <var data-var='date'>20</var>, <var data-var='time'>10:54</var> PST Monitoring - A fix has been implemented and we are monitoring the results. Nov <var data-var='date'>20</var>, <var data-var='time'>08:43</var> PST Update - We are continuing to investigate this issue. Nov <var data-var='date'>20</var>, <var data-var='time'>07:55</var> PST Investigating - We are currently investigating an issue involving delays impacting integration sync, webhook and email delivery.

Background job delays impacting sync, webhook and email delivery

Nov <var data-var='date'>18</var>, <var data-var='time'>06:59</var> PST Resolved - This incident is now resolved. Nov <var data-var='date'>18</var>, <var data-var='time'>05:49</var> PST Monitoring - Cloudflare identified the issue and has been working on implementing a fix. Our dashboard and API are currently available and we're continuing to monitor the situation. Nov <var data-var='date'>18</var>, <var data-var='time'>04:51</var> PST Identified - Services are still impacted. Our team is continuing to monitor the situation. Nov <var data-var='date'>18</var>, <var data-var='time'>03:48</var> PST Monitoring - Doppler is currently affected by an ongoing Cloudflare outage. This is impacting Dashboard and API access. Services accessing Doppler secrets through an integration (e.g., Kubernetes operator or syncs) should remain unaffected, but direct attempts to access the API will be unreliable until this issue resolves.

System Outage

Aug <var data-var='date'>20</var>, <var data-var='time'>12:58</var> PDT Resolved - This incident has been resolved. Aug <var data-var='date'>20</var>, <var data-var='time'>12:38</var> PDT Monitoring - A fix has been implemented and we're monitoring the situation. All unprocessed jobs have now been processed. Aug <var data-var='date'>20</var>, <var data-var='time'>12:36</var> PDT Identified - The issue has been identified and a fix is being implemented. Aug <var data-var='date'>20</var>, <var data-var='time'>12:15</var> PDT Update - We are continuing to investigate the issue. Aug <var data-var='date'>20</var>, <var data-var='time'>11:52</var> PDT Investigating - We're currently investigating an issue involving delays in background job processing that's impacting integration syncs, email delivery, activity log delivery, and webhook delivery (amongst other things). Impacted integration syncs will show as failed with "Out of sync". If expected emails aren't being received, it's likely related to this as well.

Background job delays impacting sync and email delivery

Aug <var data-var='date'> 5</var>, <var data-var='time'>22:40</var> PDT Resolved - The issue is resolved. Aug <var data-var='date'> 5</var>, <var data-var='time'>21:30</var> PDT Monitoring - We believe the issue is resolved and are monitoring for any additional issues. Aug <var data-var='date'> 5</var>, <var data-var='time'>20:36</var> PDT Investigating - We are currently investigating this issue.

Elevated error rates for API and Dashboard

Jun <var data-var='date'>12</var>, <var data-var='time'>14:07</var> PDT Resolved - This incident has been resolved. Jun <var data-var='date'>12</var>, <var data-var='time'>13:19</var> PDT Monitoring - GCP and Cloudflare have been gradually resolving their outages and Doppler services have been partially restored. Jun <var data-var='date'>12</var>, <var data-var='time'>11:29</var> PDT Investigating - We're currently investigating a system-wide outage related to disruptions with GCP and Cloudflare.

Apr <var data-var='date'>10</var>, <var data-var='time'>19:33</var> PDT Resolved - This incident has been resolved. Apr <var data-var='date'>10</var>, <var data-var='time'>19:15</var> PDT Investigating - We're currently experiencing downtime due to extended database maintenance.

Extended Maintenance

Apr <var data-var='date'>10</var>, <var data-var='time'>19:15</var> PDT Completed - The scheduled maintenance has been completed. Apr <var data-var='date'>10</var>, <var data-var='time'>18:15</var> PDT In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Mar <var data-var='date'>24</var>, <var data-var='time'>15:21</var> PDT Scheduled - We will be undergoing scheduled maintenance during this time.

Database Maintenance

Apr <var data-var='date'> 3</var>, <var data-var='time'>18:17</var> PDT Completed - The scheduled maintenance has been completed. Apr <var data-var='date'> 3</var>, <var data-var='time'>18:00</var> PDT In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Apr <var data-var='date'> 1</var>, <var data-var='time'>13:33</var> PDT Scheduled - We will be undergoing scheduled maintenance during this time.

Mar <var data-var='date'> 8</var>, <var data-var='time'>06:30</var> PST Completed - The scheduled maintenance has been completed. Mar <var data-var='date'> 8</var>, <var data-var='time'>06:00</var> PST In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Feb <var data-var='date'>27</var>, <var data-var='time'>08:06</var> PST Scheduled - We will be undergoing scheduled maintenance during this time.

Nov <var data-var='date'> 3</var>, <var data-var='time'>06:06</var> PST Completed - The scheduled maintenance has been completed. Nov <var data-var='date'> 3</var>, <var data-var='time'>06:00</var> PST In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Oct <var data-var='date'>18</var>, <var data-var='time'>08:29</var> PDT Scheduled - We will be undergoing scheduled maintenance during this time.

Oct <var data-var='date'>30</var>, <var data-var='time'>17:05</var> PDT Resolved - This incident has been resolved. Oct <var data-var='date'>30</var>, <var data-var='time'>16:53</var> PDT Update - Only users with access to personal configs via groups are affected. Users with direct access to personal configs are not affected. Oct <var data-var='date'>30</var>, <var data-var='time'>16:41</var> PDT Update - A fix has been identified and should be deployed soon. Oct <var data-var='date'>30</var>, <var data-var='time'>16:19</var> PDT Update - We are continuing to investigate this issue. Oct <var data-var='date'>30</var>, <var data-var='time'>15:10</var> PDT Investigating - Certain users may be unable to fetch from personal configs.

Partial Personal Config Outage

Jul <var data-var='date'>28</var>, <var data-var='time'>07:00</var> PDT Completed - The scheduled maintenance has been completed. Jul <var data-var='date'>28</var>, <var data-var='time'>06:00</var> PDT In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Jul <var data-var='date'>16</var>, <var data-var='time'>16:29</var> PDT Scheduled - We will be undergoing required database maintenance during this time.

May <var data-var='date'>31</var>, <var data-var='time'>12:51</var> PDT Resolved - A fix has been deployed and we are monitoring reconnections. We will backfill missed Activity Logs as needed after workplaces reconnect Splunk. May <var data-var='date'>31</var>, <var data-var='time'>11:05</var> PDT Update - We are continuing to work on a fix for this issue. May <var data-var='date'>31</var>, <var data-var='time'>11:04</var> PDT Identified - Activity Logs stopped publishing to Splunk as of May 30th at 19:55 UTC. The issue has been identified and a fix is in progress. After the issue has been resolved and the connections reconnected, missed activity logs will be republished.

Splunk Activity Logs Integration Outage

Apr <var data-var='date'>15</var>, <var data-var='time'>08:59</var> PDT Resolved - For a period of 18 minutes, secrets related API and dashboard actions saw increased error rates (503) and timeouts.

Elevated error rates for API and dashboard

Mar <var data-var='date'>31</var>, <var data-var='time'>06:12</var> PDT Completed - The scheduled maintenance has been completed. Mar <var data-var='date'>31</var>, <var data-var='time'>06:00</var> PDT In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Mar <var data-var='date'>19</var>, <var data-var='time'>09:58</var> PDT Scheduled - We will be undergoing required database maintenance during this time.

Feb <var data-var='date'>29</var>, <var data-var='time'>14:10</var> PST Resolved - This incident has been resolved. Feb <var data-var='date'>29</var>, <var data-var='time'>14:00</var> PST Monitoring - A fix has been implemented and we are monitoring the results. Feb <var data-var='date'>29</var>, <var data-var='time'>13:38</var> PST Update - We are continuing to work on a fix for this issue. Feb <var data-var='date'>29</var>, <var data-var='time'>13:29</var> PST Identified - We are working on a fix.

Syncs to Railway are intermittently failing

Oct <var data-var='date'> 3</var>, <var data-var='time'>14:39</var> PDT Completed - Enforcement of TLS v1.3 has been paused due to issues with the .NET framework. Testing will resume at a future date with advance notice. Oct <var data-var='date'> 3</var>, <var data-var='time'>12:59</var> PDT In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Oct <var data-var='date'> 3</var>, <var data-var='time'>12:55</var> PDT Update - We will be undergoing scheduled maintenance during this time. Oct <var data-var='date'> 3</var>, <var data-var='time'>12:29</var> PDT Scheduled - We will be testing enforcement of TLS v1.3 on the doppler.com domain and all subdomains. This should affect less than 0.2% of traffic. Testing will cease after 24 hours, at which point we will evaluate impact and determine if this change is safe to enforce permanently.

Testing enforcement of TLS v1.3

Aug <var data-var='date'> 4</var>, <var data-var='time'>15:50</var> PDT Resolved - This incident has been resolved. Aug <var data-var='date'> 4</var>, <var data-var='time'>15:46</var> PDT Update - We are continuing to investigate this issue. Aug <var data-var='date'> 4</var>, <var data-var='time'>15:28</var> PDT Investigating - We are currently investigating this issue.

Unable to fetch secrets from the Doppler dashboard and API

Jul <var data-var='date'>23</var>, <var data-var='time'>01:32</var> PDT Completed - The scheduled maintenance has been completed. Jul <var data-var='date'>23</var>, <var data-var='time'>01:00</var> PDT In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Jul <var data-var='date'> 6</var>, <var data-var='time'>21:34</var> PDT Scheduled - We will be undergoing required database maintenance during this time.

Jun <var data-var='date'> 3</var>, <var data-var='time'>20:33</var> PDT Completed - The scheduled maintenance has been completed. Jun <var data-var='date'> 3</var>, <var data-var='time'>20:18</var> PDT Verifying - Verification is currently underway for the maintenance items. Jun <var data-var='date'> 3</var>, <var data-var='time'>20:01</var> PDT In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. May <var data-var='date'>26</var>, <var data-var='time'>17:24</var> PDT Scheduled - We will be upgrading our Redis infrastructure during this time. Secret retrieval will continue to operate normally, however secret modification will be blocked.

Infrastructure Maintenance

Mar <var data-var='date'>22</var>, <var data-var='time'>16:13</var> PDT Resolved - An incident was identified at our provider for OS package registry downloads. They've fixed the issue and installs from OS package registries are available again. Impact: The incident affected all Doppler users attempting to install the Doppler CLI via their OS package registeries. This resulted in disruptions to their workflows relying on the Doppler CLI. Resolution: Cloudsmith resolved the incident after around an hour, and users were once again able to install the Doppler CLI from their OS package registries. Mar <var data-var='date'>22</var>, <var data-var='time'>15:30</var> PDT Investigating - We're currently investigating this issue. If you need immediate availability, you can install Doppler directly from our published release artifacts instead of via your OS's package registry by running: ``` (curl -Ls --tlsv1.2 --proto "=https" --retry 3 https://cli.doppler.com/install.sh || wget -t 3 -qO- https://cli.doppler.com/install.sh) | sh ```

Doppler CLI Failing to Install From OS Package Registries

Feb <var data-var='date'>27</var>, <var data-var='time'>12:45</var> PST Resolved - For several hours on February 27, 2023, users intermittently encountered errors when attempting to install the Doppler CLI. This issue was due to a related incident on GitHub, where Doppler hosts all CLI binaries and signatures. According to GitHub's incident report, GitHub experienced degraded performance and increased error rates for their packages service. Impact: The incident affected all Doppler users attempting to install the Doppler CLI. This resulted in disruptions to their workflows and GitHub actions relying on the Doppler CLI. Resolution: GitHub resolved the incident after a few hours, and users were once again able to install the Doppler CLI.

Doppler CLI Failing to Install Due to GitHub Incident

Jan <var data-var='date'>16</var>, <var data-var='time'>11:11</var> PST Resolved - During this incident, changes to secrets could not be synced to integrations or trigger webhook updates. Additionally, activity log notifications could not be posted to Slack, Microsoft Teams, Sumo Logic, Splunk, or Datadog.

Unable to sync secrets, webhooks, and activity log notifications

Jan <var data-var='date'> 5</var>, <var data-var='time'>10:23</var> PST Resolved - This issue is now resolved. Additional action is required to re-enable existing Heroku syncs. All workplaces will need to reconnect their Heroku sync integrations from the [workplace Settings page](https://dashboard.doppler.com/workplace/settings). Once integrations are reconnected, all syncs will automatically be triggered to sync any pending changes in Doppler. A postmortem will be available soon with more details regarding what happened. Jan <var data-var='date'> 4</var>, <var data-var='time'>17:06</var> PST Update - This incident may persist for up to 24 hours. In the meantime, if you have an urgent secret change that needs to be made, please update the secret in your Doppler dashboard and then manually update the secret in Heroku directly via `heroku config:set`. Updating the secret in Doppler will ensure that the value in Heroku is not overwritten once the incident is resolved. Jan <var data-var='date'> 4</var>, <var data-var='time'>16:48</var> PST Identified - New and existing Heroku sync integrations are currently not functioning. We have identified the issue and are working on resolving it. Secrets previously synced to Heroku apps will remain available but new secret changes in Doppler will not be synced until this is resolved.

Is Doppler down?

Doppler status is UP

Lastest published events

CLI Downloads failing

Background job delays impacting sync, webhook and email delivery

System Outage

Doppler Services, Components and Regions

Recent Doppler events

CLI Downloads failing

Background job delays impacting sync, webhook and email delivery

System Outage

Background job delays impacting sync and email delivery

Elevated error rates for API and Dashboard

System Outage