A Guide to Cybersecurity Incident Simulation Options
What is the best way to prepare your team to respond to cybersecurity incidents?
While the first step is having the right plans, processes and tools without testing and practice it is impossible to know if they will work as expected. Even if everything is in place, if the team is unfamiliar with the tools and plan, they will not operate at peak performance.
Asking your responders to read the plan and manuals is a start, but you would not prepare for a marathon by reading a book on running. As with training for a marathon, the best way to prepare for incident response is to exercise! There are a number of exercise options available to you, including table top, capture-the-flag, cyber ranges, purple team and red team exercises.
In this blog post, I will share some lived experience working with each option and consider the costs and benefits of each. For the typical team, I usually recommend table top exercises based on their benefits for the time and cost investment.
My Experience With Simulations
Over the years, I have been involved in running variations of these exercises for a range of clients, from relatively small organizations with 10 or so people on the response team to large enterprises with a dedicated security team of 80 or more.
I have used a range of different types of exercise over the years, including:
- Table Top (TTX): A facilitator presents scenarios to the team for them to respond to. Sometimes there can be data sets to process (we optionally include Cyber Triage collections in ours).
- Training Data Sets: Data sets with attack artifacts are analyzed by the IR team.
- Cyber Range: A virtual environment that simulates the network being defended.
- Live Fire: The IR team or facilitator conducts attacker-like activity in your production environment that should trigger alerts and ideally some level of investigation and response.
- Purple Team: An extension of the live fire approach, but the ‘red team’ gets involved and runs a complete attack scenario, while collaborating with the IR team (blue team).
- Red Team: Similar to the Purple Team live fire approach, but without the collaboration. The IR team doesn’t know what the red team is doing and therefore this is a test of both the detection systems and the IR plan.
To better understand the different exercise types, see the section at the end of this post.
When it comes to return on investment, the humble table top is ahead of the rest by several orders of magnitude. While many organizations run these exercises, there are many variations on how they can be implemented and, to a certain extent, the outcomes.
Comparing the Costs
There are two easily measured costs associated with IR exercises: the direct financial cost of hiring consultants and facilitators and the time cost staff have to spend on the exercise.
The table below gives some idea of the costs of each exercise for a medium-sized organization. With the caveat that direct costs can vary depending on the complexity of the exercise and network.
| Exercise | Direct Cost 1 | Hours per participant 2 |
|---|---|---|
| Table Top | $5,000 to $20,000 | 2 to 4 |
| Live Fire | $12,000 to $50,000 | 8 to 16 |
| Training Data Sets | $500 to $8,000 | 10 |
| Cyber Range | $50,000 to $220,000 | 16 to 32 |
| Purple Team | $50,000 to $100,000+ | 8 to 32 |
| Red Team | $50,000 to $100,000+ | 16 to 40 |
1 This is assuming that an external third party is developing and facilitating the exercise. The prices are indicative of the average range, there are certainly outliers beyond these ranges, with associated varying levels of quality and complexity.
2 This time may not be contiguous, for example in a Red Team exercise participants are likely to be performing regular work functions in parallel with the exercise.
Recommendations
There are two primary benefits of conducting IR exercises: making the response team familiar with the response process and confirming that the process actually works! Which of these receives the higher focus will be determined by the maturity of your IR capabilities. For organizations in the early stages of IR capability development, Table Top Exercises provide an extremely high rate of return.
As with any form of exercise, the more frequently you practice, the better you will perform. If IR exercises are run as an annual event, while the benefits may be significant immediately following the exercise, the familiarity will decline.
It is better to run a two-hour table top four times a year rather than an eight-hour exercise once a year. Even better is to use a mix of exercise types to ensure that all the required skills are practiced.
I recommend that most teams start with one or more table top exercises and then progress to more advanced simulations as they are ready. TTX often identifies several areas that need work, and once changes have been made, it’s good to run another TTX to ensure the change worked as intended.
Once you are confident that the theory side of things is working, then it is time to test your systems using test data and live exercises. This may be a cyber range or using your production systems. This can then be followed by a purple team exercise, culminating in a red team exercise. This progression allows for maximum value from each exercise, as many issues will be identified by the simpler and less expensive exercises while at the same time familiarizing participants with the IR process.
Keys for a Successful IR Exercise
Proper Preparation
You are taking the right first step by reading this! Make sure you allow enough time to prepare the exercise, the standard rule of IT projects applies here, figure out how long you think it will take and triple it!
Clearly Defined Scope and Objectives
A critical part of planning a successful IR exercise is first identifying what you want to achieve from it. This will heavily influence the exercise design. Questions to ask include:
- Have we made any changes to our IR process or systems?
- Do we have new people on the team that need training?
- What issues were raised in recent post incident reviews?
Realistic Scenario
Designing a realistic scenario that addresses your objectives can be challenging. It is important to remember that you do not have to run through the entire IR lifecycle for the exercise but can focus on specific stages if that will better address your needs. For example, if one of your objectives is to test a new SIEM then a scenario built around the detection stage may be enough.
When designing a scenario, I draw from my experience and my team’s experience in recent investigations and will generally build the scenario using elements from three or four cases.
If you are not lucky (or unlucky enough) to be conducting regular investigations, case studies from sites like the DFIR Report, Recorded Future, and ReversingLabs are invaluable. If you use an external consultant to build and facilitate the exercise, ensure they have current IR experience.
Engagement Of All Participants (this also means limiting the number of participants)
A table top exercise can be rapidly derailed if even one participant is not engaged in the spirit of the exercise, and there are a number of things that can disengage participants.
One time we had been engaged by the PCI team to run a table top. Since it was PCI, the decision was made to build a scenario around a compromise of the website, resulting in compromise of credit card data. This exercise was run with around 20 participants, including a number of the website developers. As soon as the scenario was presented the developers became defensive about the security of the website, resulting in fifteen minutes or more spent discussing the scenario and why it would not work.This response was perfectly predictable and not unreasonable, as the developers felt their expertise was being criticized. This was a valuable learning experience for me, and following it and other more positive exercises.
I now use the following to help ensure that the participants are positively engaged in the exercise:
- Limit participant numbers — Group size will vary depending on the focus on identifying and resolving issues vs training. If the focus is on identifying issues and resolving them a smaller group is better, as the discussion will be more focused on training the number of participants can be larger, but the exercise will be more prescriptive.
- Communicate from the outset that this is a theoretical exercise — Focus on how the incident would be handled in a real situation. Emphasize that the scenario is not real nor a reflection on the teams involved.
- Seek input from all — Ensure that all teams to be involved in the exercise are consulted when developing the scenario (without giving the scenario away in advance).
- Manage discussion during the exercise — Ensure that everyone has a chance to be heard, but don’t be afraid to shut down discussions that are going way off-topic.
Facilitator with Real Incident Response Experience
Using a facilitator with Incident Response experience results in a scenario that is realistic and more importantly can provide real-life context to the responses from participants. They will know what containment actions are most likely to be effective and have a realistic understanding of analysis outcomes and timeframes.
Conclusion
There are a number of different ways of exercising your Incident Response capabilities. Like any form of exercise the more often it is repeated the better the performance. For a great example of a well-planned Incident Response exercise schedule, have a look at this post from Uber.
Different types of exercise require different levels of effort, and the humble table top can be an excellent low cost approach to regular exercising, backup up by training, test data sets and less frequent major exercises such as Cyber Range, Purple and Red teaming.
If you are interested in conducting a Table Top DFIR Scenario, please let us know. We offer that and it includes data sets to make it even more hands on.
Appendix: Types of IR Exercises
Table Top
A Table Top Exercise (TTX) is a purely theoretical exercise, with participants presented with a scenario and given the opportunity to discuss how they would respond. A good exercise will have several ‘injects’ along the way.
It starts out with the participants being presented with some basic information to start (for example: a SIEM alert for Mimikatz) and are provided with more information based on their responses to each inject. When I am running table tops, I normally build the scenario based on a recent incident response engagement with a similar organization. This helps to keep the scenario as realistic as possible.
I found table tops to be most effective with five to ten participants. More than that, it starts becoming challenging for everyone to be involved. For larger numbers of participants, it can help to break into teams, with each team being asked to discuss each inject amongst themselves and then present their conclusions to the group.
The beauty of a table top exercise is that it requires minimal effort to set up and run. Table top exercises can be run in as little time as an hour, and the only time participants are removed from their usual duties is the time taken to run the exercise.
One downside to a TTX is that it can reveal serious problems with your IR plan! One client I worked with wanted to run short hour-long table tops every two weeks. It was a great idea, but we had to stop for several months after the first exercise because they had identified so many things to work on that it would take months to address all the issues. This might sound bad, but it so much better to learn this during an exercise than during a real incident.
For a deeper discussion on what is involved in a table top exercise, see our previous post: Ready, Set, Defend: Cyber Tabletop Exercises.
Capture-the-flag (CTF) and Training Data Sets
While a table top exercise is a purely theoretical experience, training data and capture-the-flag exercises can be used to focus on the technical aspects of IR. There are a wealth of training courses, capture-the-flags, and training data sets available.
However, one limitation of these is that they do not provide an experience directly related to the environment under test. They tend to focus on a single technology with the context and complexity of multiple sources and systems encountered in a real response situation.
We have released capture-the-flag style training data sets for Cyber Triage as part of our Incident Response Workshop series, these are available for free on Thinkific.
Cyber Range
A Cyber Range is a virtual environment that can be used to simulate the network being defended. The advantage of this approach is that it tends to be far more realistic than table top exercises for the technical team. The technical team gets hands-on practice at incident handling without risking collateral damage to production systems.
Some teams try to create a full replication of a production network, but a cyber range does not have to be expensive or complicated. Sometimes a few virtual machines run on a single host can simulate the vast majority of systems that an investigator will likely see on their network.
The full replication option is expensive and really only practical in heavily virtualized environments. One other risk is that when duplicating production systems production data is also copied, breaking some fundamental security principles. So, if you go this route, be clear with the creators about where the line should be between being realistic and going too far.
It is also common to use a third-party Cyber Range. These will generally provide common network configurations, with the ability to recreate common security platforms (EDR, SIEM etc) without the overhead of a true duplication.
The cost of this approach will depend on the complexity of the duplication. Once again, the options range from simple configurations of a handful of endpoints and servers to complex multi-segment networks with a range of operating systems and applications.
Cyber Ranges are a good approach for training security teams on specific security software platforms and are often used as an extension of training courses. The downside is that the time cost to access or build them is significant, and often similar experiences can be created or simulated from the production environment.
Note that some vendors use the term cyber range to refer to a generic virtual training environment built around a particular technology or software product. For the purposes of this discussion, they are considered as training data sets\environments.
Live Fire Exercises
Often there is nothing quite like the real thing, and the same is true for cyber incident response. Live fire exercises involve creating activity in your production environment that should trigger alerts and ideally some level of investigation and response. These can be open or blind, and range from generating multiple failed logins to simulate a brute force attack to deploying fake malware. The beauty of this approach is that you can achieve a more targeted level of testing without the full expenses of a Red Team Exercise.
While no organization wants to take unnecessary risks, there are certain advantages to having your security team practice on the actual systems and network they will be operating on for their day-to-day jobs.
Note on Live Fire Exercises
Many years ago, I had a colleague running a live fire exercise for a local security team of a multinational organization. This involved launching some fake malware that had been created to trigger detections in the AV software in use.
Unfortunately, not all security team members had been informed of the exercise, and neither had members of senior leadership, including the CISO. When the malware was detected, the alert was received by a team not involved in the exercise, and it resulted in an escalation to the C-Suite and engagement of the incident response plan. While this provided as close to a real-life test of the IRP as you can get, the effort was not appreciated by senior management!
Purple Team (Red/Blue Team)
The key premise of a purple team exercise is that there is some communication between the attackers and defenders that allows each side to evaluate the effectiveness of their efforts.
As with TTX, there are several different approaches or interpretations that can be used for a purple team exercise. This can range from having a red teamer try a single exploit at a time and following up with the blue team to ensure that the attack has been detected and mitigated to a comprehensive day-long (or more) exercise.
One effective approach here is to run these exercises as a ‘live fire’ scenario where we run attacker tools on the network and see if the existing detection and containment systems are working as expected. More complex exercises involve the red team attacking the network while informing the blue team of their actions. Either as they are taking place or once they have been undetected for an agreed amount of time.
Purple team exercises have all the learning benefits of a cyber range exercise, with the downside of risking collateral damage to the production network. It is also important to ensure that all relevant parties have been notified and that the scope of the exercise is well-defined.
Red Team
While a Purple Team exercise should involve significant collaboration between the Red and Blue teams during a Red Team exercise, there will not be any communication between the teams. This provides a far more realistic test of detection and response capabilities but will have marginal value (from a IR perspective) if nothing is detected during the exercise. When you are engaging a Red Team, make sure that you request documentation and a debriefing session with your IR and detection engineering teams.