A highly public data breach is one of the most commonly realized nightmare scenarios of a compromise. As you can imagine, this is a very stressful experience. But as with many of life's unpleasant surprises, there are lessons to be learned from them.
Setting the scene
Day 0, 8:00 AM
It all starts as just another morning in the office, as incidents often do. You're settling down for the day ahead, savoring your morning coffee, blissfully unaware today will begin one of the most challenging experiences of your career.
Day 0, 9:00AM
A familiar ding plays in your headphones as a new e-mail is delivered to your security@ inbox.
Subject: your database has many vulnerabilities!!
Dear Acme Inc,
greetings! your database is full of many vulnerabilities! i have copied all of your customer data to my machine. please send bug bounty payment to bitcoin address 1D9uKroHn8EWUNnra1Byb4vDj2LfpK2BwE payment and i will reveal these holes so you may patch them. if you dont respond i will post your customer data and it will be very bad for you!
regards,
hackerman
This doesn’t look all that legit, but in this house, we follow procedure. So you request that the "researcher" provide validation of the vulnerability. Still just another day at the office. It’s time for your second cup of coffee.
Day 0, 5:00pm
You’re getting ready to pack it up for the day when a new message from hackerman arrives. Attached, a copy of your database. It could be a current copy or it could be a backup from a couple of years ago. But one thing is for certain: At some point, your precious data left the confines of your infrastructure.
An incident is created and after an emergency sync you send a short message to hackerman letting them know that you are reviewing their report and will respond to them shortly. The bug bounty submission has been revealed to be a thin veil over extortion.
Your team has scheduled a meeting bright and early the next day to determine next steps. You missed your train home. It hasn’t been a good day.
Well, there’s been a breach, now what?
This scenario, while frightening, is not unfamiliar to companies across the globe. Uber, Sony, and many more have all been on the receiving end of an e-mail just like this, negotiating with a thinly-veiled extortionist posing as a bug bounty submitter.
This experience is challenging for everyone from the top down. Emotions are going to be running high. The good news is that with some good old fashion teamwork and an arsenal of deep breathing exercises, you will get through this just fine.
Without further exposition, let's get down to navigating this situation:
Tip #1: Don't Panic
It is very important that you do not panic. If you are going to respond at all, treat this "report" like any other bug bounty submission:
- Assert that bug bounties are only paid out after you have received a validated submission with steps to reproduce.
- Respond calmly, clearly, and professionally or do not respond at all.
- Try and respond as if this was a submission in good faith. This will be difficult, but it can lead to crucial information being given to your team that can help you have a leveled and targeted internal response.
Of course, you shouldn't feel pressured to respond at all. You and I both know this isn't an ordinary submission. However, it's been shown that sticking to the script can be very successful in coaxing more information from the attacker, which can help you make more informed decisions.
I've found that during these extraordinary events, a standard response is a comforting reprieve. You have a process. The process is comfortable. You know the process.
Tip #2: Trust the Process*
* But first, make sure there is a process
Hackerman is a bit of a wild card. You don't know if or when they are doing to publish this breach. What's important now is that you are prepared to deal with the fallout as early as possible. Having an Incident Response Plan in place before a breach occurs will help keep your team focused and on-task.
You'll want run-books for:
- Executives
- Security
- Public Relations / Marketing
- Legal
- Customer Support, Social Media Teams, Customer Success and anyone who needs to communicate directly with customers in regards to an incident
- Operations and Engineering
Remember to:
- Keep it RACI: These are stressful times. Clearly defined responsibilities the ship sailing true.
- Prepare: Any work you can do before an incident occurs should be done. This is especially true of public and customer communications.
- Practice: Run through fire drills to ensure that your team understands and can execute the process. Clear up ambiguities and make improvements after each run
Tip #3: Don't hesitate to bring in external resources
Your first instinct would rarely be to tell more people about the breach. That being said, you need to loop in these two groups asap:
The Feds
Truth be told, there's a good chance the FBI isn't going to swoop in and arrest the person. This isn't CSI. You may have folks who see this as a reason to delay communication with law enforcement. Here's the truth: You're the victim of a crime. Call the frickin' FBI.
The feds can often provide you with crucial context that's better to get sooner rather than later. They can let you know if this is a part of a larger pattern of crimes and if it matches the profile of any other recent breaches.
At the very least, you're doing your part to make the law enforcement aware of bad actors so that in the future, they have more information that can lead to the prevention of more breaches.
Legal
Every step of the way, you should have someone responsible for ensuring that you adhere to your compliance obligations. You don't want to add GDPR fines on top of this situation, do you?
Tip #4: Leadership: Be calm, transparent, and decisive.
Be Calm
Leadership plays a crucial role in these troubling times. Of course, you are human, it's alright to be stressed and upset but now is not the time to take it out on your team. In truth, there's never a good time for that. Everyone is on edge and they are looking to you more than ever to keep the ship sailing true.
Be Transparent
Uncertainty is the mother of fear and doubt. If you everyone isn't on the same page, mistakes will be made, tensions will rise, and nothing good has ever come of that. Make sure that what you are doing is something your team is not only aware of but is comfortable with.
This is another important reason to have a run book. When decisions are made adhoc, you will find a lot of room to debate the actions to be taken. Duress doesn't produce productive discourse. Prepare in advance when you can and make decisions transparently when you can't.
Be Decisive
Set clear expectations with your team every step of the way and try not to improvise or double back. Don't let yourself fall into the pattern of: "Ready! Aim! Aim! Aim!".
It's hard to feel good about decisions made from a place of panic. Delaying the call isn't going to make it easier. Give yourself a star to follow with a solid Incident Response Plan.
Tip #5: The truth is the easiest story to sell
When it comes to customer communication. Keep it short, focused, and to the point. Don't try and use vague language to minimize the situation.
Your customers are smarter than that and will think less of you for it. Even worse, they will begin to wonder what you're hiding from them.
Communicate early and often.
Customers want to know how they are impacted and how they can protect themselves and their businesses. Above all else, they need to know they can trust you moving forward. It helps to include the following in initial communication:
- What happened
- When it happened
- What data was breached
- What you have done so far to address the issue (password resets, invalidate tokens, etc.)
- What they can do to ensure the safety of their account (documentation on 2fa, auditing access tokens, etc.). Keep in mind if there's anything you can do for them automatically without interrupting service, you should do it.
- Assure them that you will continue to post updates as the investigation continues.
I personally try to keep apologies to a minimum. I don't think they do as much as visible action and risk sounding empty or hollow. How sorry you are shouldn't be the main focus of the communication. Let your actions be the star.
In the post-mortem, you'll want to include any plans you have that will prevent this from happening in the future. These plans should include tangible actions with realistic timelines. "We take security seriously are taking actions to prevent this in the future" is vague and does nothing to assure your customers that you are taking the appropriate steps to earn back their trust.
All actions don't need to be published immediately or even ahead of time. Following up even a year or two later with what you've done since the incident is a great way to show you've made good faith efforts to measurably improve your security posture.
Tip #6: Fixating on the past won't help you in the present
Obviously, you really wish this wasn't happening right now. You may be tempted to assign blame to others or yourself. Don't do that.
Breaches happen. They are unfortunate, but they happen. Your response to the breach should be your number one priority. A good response requires teamwork. You are all in this together. The company has a duty to its customers to respond as quickly and effectively as possible, which means remaining united and pressing on through the tasks ahead.
Tip #7: You will get hacked again
If your game-plan to prevent data breaches is wholly centered on "don't get hacked again," you have made a wrong turn and need to back up a bit.
Compromises come in many forms, and they usually are not due to a hot new zero-day. Phishing and malware are still the kings of breaching the perimeter. There is no way you can possibly conceive of all the ways your infrastructure can be compromised, let alone prevent it entirely.
But hope is not lost! First, clear your mind of unobtainable goals. Focus instead on improving all facets of your security program.
The Five Functions
Remember? From NIST? Those ones!
- Identify ← The more you understand your risk profile, the more effective you can be at mitigation.
- Protect ← Important! But not the only one!
- Detect ← often overlooked or under-developed. Give this attention! If you just suffered a breach, that means you weren't able to detect and respond effectively.
- Respond ← Detection without response isn't worth much. Run fire-drills. Get accurate data on how long it takes for your team to respond and mitigate a compromised service/device/etc. How much data was your red-team able to get away with? Partial breaches are infinitely better than total ones.
- Recover ← How do we deal with a particular machine/service being compromised? Can we blow it away and start from scratch? Recover from a known good? How long does that take?
Don't focus solely on preventing compromises. You should constantly work on minimizing their impact.
Review the state of your security program and then create real tasks, assign them to real projects, and give them real timelines just like any other engineering project.
Too often after a breach do companies shift gears to "security is our number one priority" mode for a few months without really forming a cohesive plan for long-term improvement.
Security is a marathon, not a one-time project you can knock out in a sprint. Don't pretend it is. Deal with the incident as best you can and iteratively improve your processes for long-term success.
Tip #8: Breathe
Surprise! Bonus tip!
Simple and yet so easy to forget: This is not the end of the world. You will get through this.
As hectic as everything is right now, I promise you it will pass. You're doing great!
Final Thoughts
I hope that no one reading this ever has to experience an incident like this. That being said, I hope that if you do, that you would have found this guide helpful and that you will come out of it with further wisdom that you can share.