Sep <var data-var='date'>30</var>, <var data-var='time'>11:44</var> EDT Resolved - This issue is resolved. We found and fixed a code issue that caused the automated bi-weekly digest email to be sent more often than scheduled. Digest emails are back on their normal schedule. No other emails or platform features were affected. If you received extra digest emails, you can delete them, and you don't need to do anything on your account. We're sorry for the inconvenience. Sep <var data-var='date'>30</var>, <var data-var='time'>11:33</var> EDT Investigating - We're looking into reports that some admins are getting the automated bi-weekly digest email more often than scheduled. Only the digest email is affected. All other emails and platform features are working normally. You can delete any extra digest emails, and you don't need to do anything on your account. We'll post another update as soon as we have more information.

Bi-weekly digest emails sent repeatedly

Jun <var data-var='date'>18</var>, <var data-var='time'>20:50</var> EDT Resolved - This incident has been resolved. Jun <var data-var='date'>18</var>, <var data-var='time'>18:53</var> EDT Monitoring - Platform performance has improved after the earlier traffic spike subsided. Email queue processing has been re-enabled, and we are continuing to monitor database load and platform responsiveness. During the incident, elevated traffic caused increased database utilization, which resulted in slower query processing and intermittent delays across some platform requests. Current load levels are stable, and the platform is operating within acceptable thresholds. Our engineering team will continue monitoring the environment and optimizing the affected database queries and email processing flow. Jun <var data-var='date'>18</var>, <var data-var='time'>18:22</var> EDT Identified - We are currently investigating degraded performance affecting email sending and intermittent platform responsiveness. Some users may experience delays when sending emails, loading pages, or refreshing certain areas of the platform. In some cases, recently created email drafts or interface updates may take longer than expected to appear. Our engineering team has identified increased platform load and email queue processing as contributing factors. To stabilize the environment, we have temporarily paused part of the email sending daemon while we optimize the related database queries and processing flow. Event registration and core platform data continue to be processed. If an item does not appear immediately in the interface, we recommend waiting a few moments and refreshing the page. We are actively monitoring the platform and will provide another update as soon as additional information is available.

Degraded Performance on Email Sending and Intermittent Platform Delays

Nov <var data-var='date'>20</var>, <var data-var='time'>13:00</var> EST Resolved - Incident Summary We experienced a slowdown in the API’s ability to issue new tokens. This caused delays for some users attempting to generate tokens. Current Status The underlying issue has been identified and resolved. Changes have been applied, and we are actively monitoring the system to ensure stability. Users experiencing issues are advised to refresh their browsers, which should resolve the problem. Next Steps We will continue to monitor the performance and ensure that the service remains stable. A post-incident analysis will be conducted, and a full report will be made available. Thank you for your patience. If you experience further issues, please contact our support team.

Login delayed authentication

Oct <var data-var='date'> 5</var>, <var data-var='time'>23:00</var> EDT Completed - The scheduled maintenance has been completed. Oct <var data-var='date'> 5</var>, <var data-var='time'>22:00</var> EDT In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Oct <var data-var='date'> 3</var>, <var data-var='time'>13:20</var> EDT Scheduled - We will be undergoing scheduled maintenance during this time.

Server Maintenance

Oct <var data-var='date'> 5</var>, <var data-var='time'>22:00</var> EDT Completed - The scheduled maintenance has been completed. Oct <var data-var='date'> 5</var>, <var data-var='time'>21:00</var> EDT In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Oct <var data-var='date'> 3</var>, <var data-var='time'>13:19</var> EDT Scheduled - We will be undergoing scheduled maintenance during this time.

Sep <var data-var='date'>22</var>, <var data-var='time'>01:00</var> EDT Completed - The scheduled maintenance has been completed. Sep <var data-var='date'>21</var>, <var data-var='time'>23:00</var> EDT In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Jul <var data-var='date'>31</var>, <var data-var='time'>13:17</var> EDT Scheduled - On September 21st (Saturday), starting at 11pm EST, the InEvent Platform will go under Maintenance and it will not be accessible for two hours. Operations should resume on September 22nd (Sunday) at 1am EST.

Platform Maintenance

May <var data-var='date'> 9</var>, <var data-var='time'>18:28</var> EDT Resolved - We have identified an issue with our email provider when sending payment emails and we recently deployed a fix. New emails will be sent right away and emails from the past 72 hours that were not sent yet have been added to our queue and are being sent now.

Payment Emails not being sent properly

Apr <var data-var='date'> 2</var>, <var data-var='time'>05:34</var> EDT Resolved - We have identified slowness when opening certain modules of the UK Server Region and we have deployed a fix to the problem. A small percentage of users may have received a "Bad Gateway" response when connecting between 11:00 AM and 11:05 AM (CET) to the Virtual Lobby of the UK Server.

UK Server Region slowness

Mar <var data-var='date'>19</var>, <var data-var='time'>11:36</var> EDT Resolved - We had an issue with AWS auto scaling module that caused our servers to take longer than expected to reach the desired scaling point and that caused our servers to stop responding for a period of 5 minutes (the time that it took to reach the desired scaling point). This behavior is out of the ordinary and we have several scaling procedures running every day. Our team is investigating what caused this momentary issue and will apply a fix.

Servers non-responsive

Mar <var data-var='date'>29</var>, <var data-var='time'>05:21</var> EDT Resolved - This incident has been resolved. Mar <var data-var='date'>29</var>, <var data-var='time'>05:17</var> EDT Identified - Mux has identified the issue and deployed a fix. We will be monitoring the results and update this report soon. Mar <var data-var='date'>29</var>, <var data-var='time'>04:40</var> EDT Update - We are still waiting for Mux support to contact us. Mar <var data-var='date'>29</var>, <var data-var='time'>02:40</var> EDT Investigating - We are investigating an issue with Mux (mux.com) Streaming Platform that is rejecting new InEvent Live Studio streams due to an outage on their end. Our recommendation for now is to use Video Conferencing or RTMP modes for customers that need to stream content. Once we have updates from Mux, we will update this incident report.

Mux service is down and Live Studio customers might be affected

Mar <var data-var='date'> 9</var>, <var data-var='time'>05:25</var> EST Resolved - This incident has been resolved. Our European Servers are fully operational. Mar <var data-var='date'> 9</var>, <var data-var='time'>05:19</var> EST Update - We are continuing to monitor the servers. The API calls are now 100% responsive and without degraded performance. Mar <var data-var='date'> 9</var>, <var data-var='time'>04:46</var> EST Monitoring - The DNS servers are now responsive and our Load Balancer is now draining pending requests and flushing old requests in favor of new ones. New API calls will be handled normally and we will be monitoring the results. Mar <var data-var='date'> 9</var>, <var data-var='time'>04:29</var> EST Identified - We have identified a DNS issue that caused our server requests to queue and severely delay their responses on API requests. The API microservices had to wait more than anticipated for a DNS resolution and that caused timeouts with a Gateway Timeout HTTP error. We have deployed another DNS server to be used and this may take up to 30 minutes to take effect and requests starts to normalize. We will be monitoring the resolution development. Mar <var data-var='date'> 9</var>, <var data-var='time'>04:03</var> EST Investigating - We are investigating an issue with the European Servers that are causing a response delay and Gateway Timeouts on certain requests.

European Servers are not responsive

Jan <var data-var='date'>28</var>, <var data-var='time'>22:44</var> EST Completed - The scheduled maintenance has been completed. Jan <var data-var='date'>28</var>, <var data-var='time'>22:11</var> EST Update - Scheduled maintenance is currently in progress. We will provide updates as necessary. Jan <var data-var='date'>28</var>, <var data-var='time'>22:04</var> EST In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Jan <var data-var='date'>28</var>, <var data-var='time'>21:59</var> EST Scheduled - We will be undergoing scheduled maintenance during this time.

Database Migration and Optimization

Jan <var data-var='date'>17</var>, <var data-var='time'>13:15</var> EST Resolved - This issue has been resolved. Jan <var data-var='date'>17</var>, <var data-var='time'>04:30</var> EST Investigating - We have identified an issue on Mux's CDN that's affecting all Live Studio streams. Streams that lasts for more than an hour might have AV Synchronization issues due to an internal issue on Mux's servers.

Live Studio streams affected by Mux (mux.com) AV Sync issue

Dec <var data-var='date'>17</var>, <var data-var='time'>02:23</var> EST Resolved - This incident has been resolved. Dec <var data-var='date'>16</var>, <var data-var='time'>14:19</var> EST Update - We are continuing to investigate this issue. Dec <var data-var='date'>16</var>, <var data-var='time'>12:30</var> EST Investigating - This incident happened on Postmark's end. For further information, please go to https://status.postmarkapp.com/incidents/277086

Service Issue: Outbound email sending is delayed

Nov <var data-var='date'>13</var>, <var data-var='time'>01:30</var> EST Completed - The scheduled maintenance has been completed. Nov <var data-var='date'>12</var>, <var data-var='time'>22:00</var> EST In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Sep <var data-var='date'> 9</var>, <var data-var='time'>17:11</var> EDT Scheduled - On November 12, 2022 between 10:00 PM and 01:30 AM EST, internal resources will not be accessible. This includes the admin dashboard as well as the public pages. The mobile app will have restricted access to information. Should you have any questions or concerns, please contact the InEvent Support team via chat below or take a look on our status page [1], we will post live updates over there. [1]: https://trust.inevent.com

Sep <var data-var='date'>21</var>, <var data-var='time'>02:00</var> EDT Resolved - There was an issue with our Subnet Traffic Control policy in our public AWS VPC that was blocking HTTP and HTTPS traffic. Once we identified the issue and applied the fixes, there was a delay that lasted for around 30 minutes to apply the new policy to all our subnets.

Issue with AWS Subnet Traffic Control

Jul <var data-var='date'>28</var>, <var data-var='time'>15:31</var> EDT Resolved - This incident has been resolved. Jul <var data-var='date'>28</var>, <var data-var='time'>13:13</var> EDT Investigating - Our subprocessor Postmark is having issues delivering emails. Although delayed, all emails will still be delivered. We will update this incident when new information arises.

Emails are delayed

Feb <var data-var='date'>25</var>, <var data-var='time'>16:35</var> EST Resolved - We have identified the issue and fixed it. The Virtual Lobby should now open normally on Desktop and also on Mobile devices. Feb <var data-var='date'>25</var>, <var data-var='time'>15:40</var> EST Investigating - The Virtual Lobby is unresponsive and we are investigating the reason. The rest of the platform is working normally.

Virtual Lobby is unresponsive

Nov <var data-var='date'>16</var>, <var data-var='time'>13:23</var> EST Resolved - The issue has been mitigated and the service has been restored. https://status.firebase.google.com/incidents/Xed5fGf9USDWdzsqN4GX Nov <var data-var='date'>16</var>, <var data-var='time'>13:21</var> EST Update - We are continuing to monitor for any further issues. Nov <var data-var='date'>16</var>, <var data-var='time'>13:08</var> EST Update - Google Firebase issue can also be seen here on this page https://status.firebase.google.com/ Nov <var data-var='date'>16</var>, <var data-var='time'>13:06</var> EST Monitoring - We have implemented a fix which allows you to connect to the Virtual Lobby with limited connectivity. While Google Firebase is offline, we recommend to switch to InEvent native websockets under Settings > Tools. Nov <var data-var='date'>16</var>, <var data-var='time'>12:58</var> EST Update - We are continuing to investigate this issue. Nov <var data-var='date'>16</var>, <var data-var='time'>12:58</var> EST Investigating - We are checking with Google the status of the issue. For now, we recommend switching to the Native WebSockets option under Event > Tools.

Firebase is currently outputting closed connection errors

Sep <var data-var='date'>13</var>, <var data-var='time'>17:15</var> EDT Resolved - This incident has been resolved. Sep <var data-var='date'>13</var>, <var data-var='time'>13:07</var> EDT Monitoring - The team has concluded the implementation of all temporary solutions on the platform. This includes creating a timeout option on slow connecting sockets and also disabled Redis as a single connection. The web socket chat will remain with limited chat support on the firebase instance until we add support for per instance local connection, which should happen next week. The Redis cache team will be implementing a new permanent solution end of this week, Friday at the latest. Sep <var data-var='date'>13</var>, <var data-var='time'>11:45</var> EDT Update - We currently fixed an autoscaling Redis instance that was not able to redirect the load to a separate system. Redis is used by InEvent to quickly balance its write operations instead of relying only on a traditional SQL database. The following components are still affected: Firebase Web Sockets, which covers the live chat on the Virtual Lobby. Sep <var data-var='date'>13</var>, <var data-var='time'>10:55</var> EDT Identified - We have identified the slowness from the Firebase product. We are deploying an alternative with the Native websockets, under Company > Tools. We are currently deploying a fix for UI improvements on the Firebase UI console while Firebase is not responsive. Chat may be offline while the fix is being applied, but video and streaming should work normally. Sep <var data-var='date'>13</var>, <var data-var='time'>09:45</var> EDT Investigating - We are currently seeing a slow scalability response from the InEvent Firebase socket group. We are currently investigating the issue with Google Firebase team.

Slow web socket connection

Jul <var data-var='date'>27</var>, <var data-var='time'>23:00</var> EDT Resolved - AWS had an outage in our North America region and the failover and recovery mechanism took action. During 10:59:30 PM - 11:00:30 PM EDT, the website was slow and unresponsive. This issue has been fixed in 1 minute.

AWS Hardware Failure & Recovery (1 minute)

Jul <var data-var='date'>15</var>, <var data-var='time'>10:30</var> EDT Resolved - We encountered issues when using Low Latency with custom domains and we fixed it. The Low Latency Control Room option uses the Amazon AWS IVS technology for Low Latency Streaming (sub 5 seconds) and the 1.3.0 version of its IVS Video.js Player had an issue when working with cross-origin resource sharing (CORS) that resulted in a player with a frozen state (a pitch-black image with a blue play button that didn't work when pressing it). We have updated the player plugin to its newest version (1.3.1) and implemented fixes that will also circumvent CORS issues in the future -- a fallback to the regular iFrame using InEvent's domain in case the custom domain doesn't work (visually indistinguishable for regular users).

Low Latency Stream Issues

Feb <var data-var='date'>24</var>, <var data-var='time'>14:00</var> EST Resolved - This incident has been resolved. Feb <var data-var='date'>24</var>, <var data-var='time'>12:53</var> EST Investigating - Google is our socket hosting provider and their servers are offline. We are investigating this incident with Google engineers. https://status.firebase.google.com/incident/Console/21001

Socket performance is downgraded

Feb <var data-var='date'> 3</var>, <var data-var='time'>08:57</var> EST Resolved - This incident has been resolved. Feb <var data-var='date'> 3</var>, <var data-var='time'>08:57</var> EST Monitoring - We have identified unhealthy server replicas that were being treated as healthy servers, resulting in poor connectivity to our platform. We have deployed a fix for this issue. Feb <var data-var='date'> 3</var>, <var data-var='time'>06:45</var> EST Investigating - We are currently investigating this issue.

Issues on UK Region connectivity

Jan <var data-var='date'>26</var>, <var data-var='time'>03:04</var> EST Resolved - This incident has been resolved. Jan <var data-var='date'>26</var>, <var data-var='time'>02:54</var> EST Monitoring - We detected an issue on our auto scaling group in our AWS eu-west-1 region. A fix has been implemented and we are monitoring the results. Jan <var data-var='date'>26</var>, <var data-var='time'>01:30</var> EST Investigating - We are currently investigating this issue.

Is InEvent down?

InEvent status is UP

Lastest published events

Bi-weekly digest emails sent repeatedly

Degraded Performance on Email Sending and Intermittent Platform Delays

Login delayed authentication

InEvent Services, Components and Regions

Recent InEvent events

Bi-weekly digest emails sent repeatedly

Degraded Performance on Email Sending and Intermittent Platform Delays

Login delayed authentication

Server Maintenance

Server Maintenance

Platform Maintenance