$25 million tech start-up in crisis after employee accidentally deletes wrong file


If you have ever had that horrible, horrible feeling immediately after deleting something in error, you will sympathise with this story.

A tech start-up with more than $25 million in funding was sent offline yesterday after an employee accidentally deleted a file containing important customer data.

Silicon Valley start-up GitLab, which provides a virtual workspace for programmers to test and work on projects, was driven into chaos as staff attempted to deal with a spam attack on their database, rendering it unusable.

However, in an attempt to restore the database from a backup, 300GB of live production data was accidentally deleted by a sys-admin, with just 4.5GB of data left before the delete command was cancelled.

The whole episode was played out in a series of sobering tweets on GitLab’s Twitter page.

“After a second or two he notices … terminates the removal, but it’s too late. Of around 300GB only about 4.5GB is left,” the company wrote on its blog.

Following the incident, GitLab took its site offline for emergency maintenance, while giving users an almost running commentary on social media and via a Google Doc of its progress.

The latest update is that despite noticing the error quickly, not all of the data can be recovered, however, GitLab’s services are now operational again, the Register reported.

“Out of five backup/replication techniques deployed, none are working reliably or set up in the first place. We ended up restoring a six hours old backup,” the company wrote.


Comments are closed.