Incident Report: June 19, 2024
On June 19, 2024, around 12 AM Pacific Time, Knoword suffered an incident that resulted in the loss of some data. This post is going to talk about what happened, what we did to remediate the situation, and steps we've taken to ensure that something like this never happens again.
A breakdown of what happened
While developing tools to automate the migration of data from our current database to a new database, our current database was accidentally reset.
We have an automatic backup system that saves a daily snapshot of our database, so that it's easy to recover from incidents like these or roll back the database to a previous day for any reason. Unfortunately, the last few months of backups that had been captured were faulty and could not be used. This was an oversight on Knoword's part, and we did not realize that the backup system was not functioning properly because we've never had to use them until now.
These two factors culminated in the temporary loss of all data in our database. I want to stress that this incident was not perpetrated by any bad actors, and the data never leaked or changed hands. It was simply deleted.
How we fixed it
Luckily, as a part of the development of the previously metioned migration automation tools, a number of copies of critical database tables were made. We were able to restore those tables within a few hours of the incident taking place.
The copied data included users, packs, collections, and words—the important stuff. Since the users data was copied a day prior, any user accounts created between June 17 and June 19 were lost. That also means that any packs, collections, or words created by those users were also lost. Since the copies of packs, collections, and words data was made more recently, there was virtually no data lost there for any users created before June 17.
Unfortunately, fresh copies of the assignments and submissions database tables were not made, so that data had to be restored from the next most recent backups, captured on January 2023. That means that all assignments and student submissions made in the last 18 months were lost.
Learning from mistakes
This incident has certainly caused a lot of stress and turmoil, but it's also taught us some valuable lessons. The following is a list of measures that we're taking to ensure that something like this never happens again:
- Fix automatic database backups (done ✅)
- Monthly security check-up to verify that systems are working properly (scheduled 🗓️)
- Maintain a secondary "staging" environment that can be used to safely prototype and test projects like the migration script that resulted in this incident
- Implement restrictions on destructive actions that can be taken on our main "production" database
Moving forward
We deeply apologize for any inconvenience that our users have experienced as a result of their assignments or submissions being lost. We just want to be transparent about what happened, and let everyone know what we're doing to avoid mistakes like this in the future.
If you have any questions or want to reach out to us, please contact us and someone will be happy to chat.
—Trevor from Knoword