12 March 2018

Enforcing the Law at the Mid Atlantic Collegiate Cyber Defense Competition (MACCDC)

The MidAtlantic Collegiate Cyber Defense Competition (MACCDC) is one of the many regional CCDCs that includes a somewhat unique aspect: law enforcement and investigations. For those unfamiliar with CCDC's, they are live network security competitions where schools face off against each other, and a red cell of pentesters, to build and maintain a secure network. While fending off attacks the teams are responsible for creating new servers and services while performing business operations, such as running database queries for a business need. If the respective database is misconfigured, or hijacked by Red Team, then the query cannot be performed and teams suffer major score losses.

There are multiple regional CCDC competitions across the entire country as well as the National CCDC where the winners of each regional competition join to face off against each other. While each regional follows the same structure of competition, each can make slight adjustments to how they determine a winner. A law enforcement (LE) component was built into MACCDC years ago as a method to help expose competitors to the unique and frustrating challenges of fully documenting attacks.

In many competitions, schools practice on being extremely responsive to attacks and, in many cases, aggressive in their responses to remove an adversary as quickly as possible. While that effort is commendable, it does not translate into the actions taken by a real-world security team. In the event of a compromise, theft of data, or denial of service attack, corporate senior leadership will not be content with a message of "We were attacked, it's been fixed."

Questions will arise:

  • Who attacked us?
  • How did they get in?
  • What did they steal?
  • How long ago did they get in?
  • Are they still there?
  • Is this an attack also seen by industry partners?

Therefore, the role of MACCDC LE is to provide students the resources necessary to collect this evidence and document an incident that can be used to adequately describe the attack. Additionally, though infrequent, a well written incident report that clearly and accurately describes an attack, shows full logs of where the attack came from, and notes exact times of attack, can allow for Exercise Control to authorize a firewall block and a possible arrest of the Red Teamer involved.

Attacks are documented using a standard form that is supplied to the students at the start of the competition: the US Secret Service Incident Response Form 4017. This is a form that is used in actual investigations for law enforcement use, and was chosen as it contained an adequately wide number of scenarios to meet every attack used by Red Team.

In recent years, each team competiting in MACCDC has a requirement to complete at least two incident response forms by the end of the competition. While there is no limit to the number that can be submitted, each form does take time away from their other duties and each is additionally graded on the "worthiness" of law enforcement. They are then left to document any incident of their choosing with as much possible detail as they can.

Many teams who struggle often limit themselves with reports on suspicions of an attack. They see errant log entries, or information that doesn't look appropriate, and immediately jump into IR mode. After 30 minutes of detailing the log entries, they then learn the hard pressure of proving an attack to law enforcement.

There are many categories in which LE scores each team's submissions, with an ultimate value of 140 points given. These include 10 points in each of the following categories:

  • Attack clearly defined
  • Scope of attack clearly defined
  • Time window clearly defined
  • Source of attack identified
  • Delivery mechanism identified
  • Damage assessment provided
  • Remediation efforts explained
  • Appropriate evidence provided
  • Timeliness of report
  • Clarity of submission
  • Clear handwriting and presentation
  • Understandable grammar
  • Completeness
  • Accuracy of report

Additionally, each report has a subjective negative score of "Waste of Tax Payer Money". This last one allows for LE to weigh in on poorly written submissions. And, as LE can provide feedback to improve submissions, teams that overwhelmingly request help from LE to submit a report that ultimately has no value can face a negative score.

The grand prize that teams rarely see if an arrest made, rewarding 500 extra points. Given enough evidence, and indicators that can point to an actual hands-on-keyboard person, LE can make an arrest of a Red Teamer. The Red Teamer will be pulled from the team, their computers disconnected, and forced to sit in time-out for 20 minutes. This is often done within the competition pit itself to face public ridicule. In the past, arrests have been made based upon Red Teamers publicly documenting their attacks and the students positively identifying their systems in tweets. Students could also see unique names, handles, and identifying indicators in their attacks. Often, if a team can positively identify a very unique computer in use, via User-Agent value or similar, LE can do a probable cause search and see if they can spot the computer in the Red Team room. Most times students attempt to do this but are usually off slightly in their assessment. LE knows who the real attacker is, but if the evidence points to someone else, then the submission is of no use.

Based on these categories, here are some pitfalls and suggestions for teams:

Who attacked us?

It's easy to say "The Red Team over in the next room attacked us", but can you provide exactly who it was. Can you provide IP addresses that are unique to a certain attacker subset?

Can you prove the attack and actions taken?

Did they have the evidence of the attack stored? Are there logs or screenshots available? Are they in a method that can be given to law enforcement? If not, then they have no evidence of the attack. Some teams scramble to transcribe logs as quick as possible, while others take screenshots and copy log files to a separate system. In the case of the latter, we stop by with a USB drive, collect the data, and review it along with the incident report. Best action is that as soon as you see signs of an attack start collecting logs, especially before they're erased by Red Team. Toward the end of the competition, when Red Team becomes more brazen, it becomes even more important for Blue Teams to take screenshots. After all, how do you appropriately describe literal ghosts following your cursor on the screen or that your Exchange server was replaced with a flying nyan cat? That's also the time when Red Teamers often get lax in OPSEC, making connections from their raw systems, using their personal handles in their attacks, or bragging about them on Twitter.

Can they prove that the actions were malicious activity by an adversary and not the mistakes of an insider? "This is my team, we didn't do that." That's what they all say, prove it to me. Guess what? As LE, we can just go casually talk off-the-record to Red Team and verify if they did the attacks. Can they prove that it came from a very specific IP address, and that the IP is unique to an adversary and not to something like Scorebot?

Can you identify the source and delivery of an attack?

These are two very critical topics that many teams gloss over. What use is kicking an attacker out if you don't know how they got in? You're simply covering up the symptom but are still keeping any vulnerability wide open. By documenting the source of the attack, such as the offending IP address, a team can start creating a case of multiple attacks over a short period of time from the same source. Providing corresponding evidence from previous incident response forms can help set the foundation and allow easier points.

Additionally, the point of this exercise is to perform due diligence to identify threats and mitigate them. Many times the symptoms are reported upon, but teams don't put the extra five minutes in to determine how attacks started. Either via SMB connections, RDP, SSH, or bad passwords, documenting this shows not just a response but the building of an attacker profile. 

Evidence should be relevant and specific to the attack. If a Red Teamer got onto a Linux server, are there log entries for SSH brute forcing or does it show a direct login? There are different meanings, and different types of remediation, for each. Are there .bash_history files available to describe what commands were typed by the Red Teamer to show what their goals were? A brute force attempt will typically result in little actual activity, just the creation of an additional user account or application for back dooring. It will, however, create numerous log entries.

For Windows, can you find events within Event Logs to show signs of access, such as Type 10 logons (Remote Interactive Sessions) and Type 3 (connection to a shared CIFS/SMB folder). A Windows server that's come back from a reboot will have it's SYSTEM hive freshly stored to the hard drive, containing details from ShimCache to show commands executed.

Can we make our indicators shareable?

In recent years we have experimented with methods to implement threat intel into the competition, including methods for sharing indicators. As each team represents a business within the same industry as others, there's an expectation of shared attackers targeting their vertical. In the past we've made use of notifications similar to the FBI Flash Alerts. If a specific indicator or TTP is identified by at least two teams then a high level description of that TTP is provided to all teams.

For example, teams identify that a scheduled task is created for specific malware on all their systems. Multiple teams report the same method, naming conventions, and files on their system. A flash alert is created to notify that "malware has been seen being entrenched via Windows scheduled tasks." Teams are then on their own to use that information, disregard it, or figure out how to even relevant to their work (at times that indicator doesn't even exist on a particular team's network but they'll spend over an hour trying to find it).

Tools for analysis

CCDC events have great restrictions on the software that can be used in competition. However, most tools that are needed are already on the system or available as open-source. In the field, one's best tools are simply grep and awk. Grep will be used continually to find entries in log files, and works best with the -A, -B, and -C options which are, in order, show X lines after the match, show X lines before the match, and show X lines before and after the match. Even Windows has similar with findstr.exe!

Awk is a necessary skill to practice to take large amounts of logs and distill them down to usable data. If you're reviewing the messages log file on Linux, the output can be quite long. However, a grep for certain entries where you know the critical data is always in the first, second, and fifth columns, can reduce the display down to minimal information for quick response.

On Linux and Unix environments, many students block their own analysis by relying on standard commands, like `ls` to see times. Move beyond that. Use `stat` to see all meta for a given file. Use `find` in creative ways to watch for attacks. `find ~/ -type f -mtime -1` will quickly show any files modified within the last day while `find /etc -type f -cmin -10` will show every file from /etc created within the last 10 minutes. 

Learn your operating system! We're just taking existing tool sets and using the command line to chain data together, manipulating and sorting it, as a means to answer very simple questions.

When we create the Law Enforcement team for MACCDC we don't just bring together a few volunteers. We bring together experience, knowledge, skill, from those who have done the work. Some of us have done IR as part of commercial consulting, some for criminal law enforcement cases, others for detection and response within corporate environments. 

There is no magic here. There are no 'gotchas'. Attacks are realistic and so are the demands. The data exists to do investigations, and to even do them quickly. Some of the teams more experienced in this field have been able to document a full attack, from SSH brute force to privilege escalation to rootkit installation in real-time, providing a report mere minutes after removing the malware and patching their system. 

That is the necessary skill to learn.

No comments:

Post a Comment