Starting at ~2:30am EST today (May 14th), we experienced a backup in footage waiting to be processed. This was due to an issue with a database optimization.
No footage was lost during this period of time, however, due to the length of the delay in processing, Hotlist Alerts were not sent during this period and searches in Admin searches would have resulted in more unprocessed images than normal.
The issue was resolved at ~5:00pm EST and around ~6:30pm EST (May 14th) we were completely caught up on all images and back to normal processing time.
Posted May 14, 2019 - 18:38 EDT
Monitoring
Processing of images is trending towards normal. We will provide another update when service has returned to normal operation.
Posted May 14, 2019 - 17:31 EDT
Update
We are actively working to bring image processing back to normal operation. Again, no images have been lost and will be available once the system is back to normal.
Posted May 14, 2019 - 14:25 EDT
Update
We are continuing to work on a fix for this issue.
Posted May 14, 2019 - 09:47 EDT
Update
We are continuing to see a delay in the processing of new images, we are actively working to bring the system back to normal performance.
Posted May 14, 2019 - 09:46 EDT
Identified
Our image processing infrastructure is slightly behind, this may cause the processing of new images to take longer than normal.
No images has been lost and the system should be caught up shortly.
Posted May 14, 2019 - 08:20 EDT
This incident affected: Search, Insights, and Administration and Alerts & Map.