[TERM PAPER - FALL 2024 - BERKELY - CYBER220 - MANAGING CYBER RISK]
More
and Faster: How the Advent of AI Cyberattack Agents Alters the Risk Management
Equation
Future Trend Identification and Discussion
Current State of the Concept
The announcement of Anthropic’s nascent “computer use”
ability[1], the results of the AIxCC Semifinals[2], the release of tools like
PentestGPT[3] and Nebula[4] along with a large body of research on how to skill
Large Language Model AI’s gives a good indication that the next 9 to 18 months we
will see a rise in the availability and use of highly capable adversarial hacking
agents with their widespread use in the next 3-5 years.
Currently AI Cyberattack Agents as a concept are in
alpha. The capabilities of the systems that pass for this kind of agent are
diverse from system to system, and these are pieced together in a patchwork
fashion. As Anthropic notes in their product announcement “Developing a
computer use model”: currently the tools are bent to fit the models with custom
environments and very specific tooling to complete tasks. They note that the proof
of concept for computer use indicates that the models can be built to fit the
tools.[1] This understanding helps us to see that the current state is more a
series of “frameworks” then true self-contained agents.
This underscores the main trait of this alpha state: the
parts are available, but the difficulty of re-targetability - the ability to
retool the agent, suite, or arrangement of resources to utilize a different
methodology, technical domain, or set of tools is the blocker to actual near
autonomous agency. Currently the process
of retooling takes time and effort on the part of humans, much like we see with
PentestGPT[3] and Nebula[4]. This does not preclude the possibility of a “true
agent” form of the concept existing, it is that the existence of one is not
publicly shared.
The beta phase is the journey to a more actualized autonomy – what this paper will call production. The beta phase for AI cyberattack agents begins with the ability to overcome this retooling barrier as we have seen with the Anthropic POCs. Right now, with the current tooling, we can tell an agent to do a thing we want it to do when we want it to do it, but only things that we have set up (“Earl grey, hot!”). Production level agents will have the ability to take the necessary steps to complete more generally defined goals (“prepare dinner for my vegan guest, surprise us, and make the after-dinner tea earl grey, hot!”). At this point the infrastructure used to access and interact with resources is the same for agents and humans, and this is the crux of the change for risk assessment and management.
Nature of Adversarial Capability Increase
As noted in the CrowdStrike blog [5] advanced
adversaries are seen to have different roles and specialties. These can take
the form of sub-teams that are part of a nation state actor, or as different
groups in the cybercrime ecosystem such as Initial Access Brokers (IABs), Big
Game Hunters (BGH) or ransomware-as-a-service (RasS). Currently these
specialties are leveraged via group – specialization offers advantages, but
specialization requires expertise is difficult to retool for. Between groups and
specialized sub-groups, we are seeing adversaries adopt methods that
structurally resemble the pack hunting tactics used by primates, wolves,
hyenas, lions and wild dogs – the collaboration between actors or elements of
actors with different roles.
In their study of chimpanzees and cooperative hunting
Boecsh and Boesch developed a scheme to measure complexity of the coordination
that we can borrow from here: Similarity, Synchrony, Coordination, and
Collaboration. [6]
Similarity: Separate
threat actors utilize similar actions against the same target but with no temporal
or spatial coordination. “Drive by” low resistance attacks would be a good
example of this phase.
Synchrony: Separate
threat actors initiate actions at the same time but do not operate in a
specific role or in any other coordinated way. This has been seen when two
separate nation state actors operating for the same nation state attack at the
same time but the escalation to coordination is limited.[7]
Coordination: Separate
threat actors or threat actor-subgroups initiate attacks and coordinate against
the same target using the same actions and capabilities.[1]
Collaboration: Separate
threat actors or threat actor sub-groups coordinate their attacks and operate in
different roles leveraging different capabilities to help the other groups. A
simple example would be the handoff from an IAB to a different specialist
group.
The current areas of “need based” adversarial
collaboration seems to be between separate state actor groups under the same
banner, within larger threat actor organizations (nation state again) or between
actors in the cybercrime ecosystem. What is central to all this collaboration
is that it is between groups. What AI Cyberattack Agents offer is the
diversification of capability within these group units. These agents will allow
these groups to diversify and will relieve some manpower issues. These agents
allow for the following adaptive advantages conferred from collaborative
hunting to be brought within the threat actor group:
·
Additional role fulfillment
·
Shift in success ratio via an enlarged
threat actor group
·
Agents can shift roles on a dime and need
no infrastructure tooling
These in turn allow for the following:
·
Attack layering through multi-wave attacks
(via a larger “virtual” group)
·
Increases the ability to leverage more
techniques simultaneously (“Combined Arms”)
·
More complex attacks given the
availability of expertise in multiple fields
Organizational Risk Response
Shifts in Risk Assessment
This change will affect risk assessment. The methods
used to determine what business critical assets need protection will remain the
same. The place where we will see the most change is in the calibration
exercises based on expectation of a particular event occurring within a given
time frame. Attacks utilizing this technology will come more frequently, will
be more intense and will be faster. Additionally, the scope of the attacks will
be able to quickly shift to allow for response. The nature of the agents is
that they have the ability to change what they are doing rapidly, and this will
allow an adversary that started an attack based on one technique to rapidly
shift to a methodology based on another. The shift here is really about
attention. Who with this capability is paying attention to your organization?
The saving graces in this area are several. There is a
good chance that cutting edge off the shelf “frontier model” capability is not
available to adversarial organizations. In
this case they will have to build their own. This requires funding, processing
power and know-how to create the agents. See the “Shifts in Response” section
for an exploration of what this might mean. There is also the issue of
adversary scope of vision. The past has shown adversaries to generally be very
linear in their thought and techniques. Threat actors tend to rely on a set of
techniques that they understand, or they are aware of. Once a threat actor has
a pattern that delivers results, they tend to reuse it as much as possible. This
is so much so that a lone actor using off the shelf tools can look like a
genius compared to the surprisingly simple techniques of a funded nation state
actor.[8] These operations are run like a business and so cost is a factor, and
so if a focus on low hanging fruit works why spend the development effort on
something that promises to be much more effective but might not pan out? These
factors may slow the adoption of agent capability to expand the breadth and
depth of an attack. It may take time for threat actors to tool up and shift
habits.
So how does an organization get their ability to
calibrate the expected timeline to a cyberincident that affects their business
goals? The answer is to tool up with AI attack agents for their red team. This
will give the engineers a better understanding of the time that it will take given
this new technology.
Shifts in Controls
The advent of AI Cyberattack Agents brings with it an
impact on our controls and how we make the decisions on what those controls
are. This paper utilizes a three-by-three matrix taxonomy to illustrate and
discuss controls. On one axis we have what the control does, the
action-oriented axis – Prevent, Detect, or Correct. The other axis is where the
domain where the control operates, the domain-oriented axis – the areas of the
Administrative, Behavioral, and Technical. For our discussion here we will
focus on each domain category and the action-oriented axis for each. Given the
shape of the impact that these agents will have we won’t be looking at all of
these controls as deeply as some since certain areas are less affected via this
change.
The administrative domain is perhaps the least affected.
In this domain controls are largely legal, procedural, financial, and cover
issues relevant to workforce and authority. However, it is the people that have
power in this domain whose identity adversaries are interested in. At the
administrative level the people who are making the various business decisions
are generally those that also approve changes and actions in the business. The
attack where an adversary used AI to create a fake multi-person video chat and
convince a finance employee to transfer millions would be an example of
administrative control failure.[9][2]
The rapid pace at which attack agents can work even at
the identity level will make it necessary for an organization’s administrative controls
to be more defined and corporate governance to tie these to technical controls.
Currently most have controls at this level to prevent issues. Agents working at
speed can get more time to operate given that most organizations’ detective
administrative controls are limited with little in the way of corrective
controls.
Behavioral controls can range from financial rewards that
encourage specific actions or kinds of behavior to training, codes of conduct,
and legal elements. These controls allow an organization to shift 90% of their constituent
members into a predictable pattern and allow the operation of the organization
to conform to a reasonable expectation. Most of this is not necessarily backed
up with any sort of technical tool, except for the detective aspects that are
focused on finding those behaviors that intentionally or unintentionally open
the organization to risk.
The advent of adversarial AI agents would make this an
area where you would want to expand the technical backup to the interpersonal day-to-day
business. Not necessarily countermeasures but rather systems that allow you to
track training, participation and buy in from various members of the
organization. The commitment to a record of an event happening has been shown
to be a powerful tool in helping to reinforce the needed behavior for most
people. This can help with targeting the systems that look to detect anomalous behaviors
that might expose insider threat detection, compromised accounts and other
abnormal activity. Increased and more granular behavioral monitoring can
shorten the time that an AI agent might have if it has accessed a user account
and is taking actions that are against the behavioral controls that the user
has agreed to. Another element of behavioral analysis is the speed at which an
individual appears to be taking actions. If the speed of the actions is superhuman
a behavioral analysis engine can detect this and sound an alarm.[10] Though as
we see there are ways to circumvent such simple detections which suggests that
having several technical tools to backup behavioral controls can be helpful.
This transitions us nicely into the realm of technical
controls. With the advent of AI agents, we need to make the mechanisms that
back up behavioral and administrative controls more granular in their
capability. This has the effect of narrowing time windows for review, ETL times,
and additional metrics that a team of AI attack agents might be able to
circumvent given the speed and capability that they will possess. The preventative
will have to be more thorough, the detective more detailed with an eye for
nuance, and the corrective implementations of these controls will have to be
faster.
In the technical controls specifically, we need to
address the infrastructure, compute, identity and data controls that are
present. For most organizations, the granularity and depth of the controls that
an organization will use need to be expanded to allow for the various issues
that speed and varied capability that the agents will have. More and faster
means wider and deeper.
In the end organizations will need to utilize an
expansion of control implementation and augmentation ability to back up the technical,
administrative and behavioral controls with AI and ML capabilities to make sure
that these items are followed, giving a technical early warning to
non-technical issues that might not get detected for some time, allowing them
to be reversed. This is the same for the implementation of technical control
monitoring – employ AI and ML to remediate the speed and tactical advantage
that AI attack agents possess.
Shifts in Architecture
Speed and know how can only get you so far if
everything is locked down properly. An effort to achieve this level of lockdown
will most likely miss some things. However, it will lock down most of the
environment and an environment that is mostly secured with this principle
provides enough resilience to allow defense to form in the kind of blitzkrieg situations
that we should expect to see with these technologies. This is a shift to hybrid
zero-trust/defense-in-depth architecture.[11]
An architecture that counters these agents needs to
have zero trust at all levels. Defense-In-Depth architecture allows for a
ringed deployment of zero trust principles leveraged through multiple security
tools and mechanisms used in unison. Such a ringed deployment enables an
organization to work from the inside out. The organizations can go from the assets
that are core to the business goals moving their way out to the assets and
infrastructure that may not be core to their operation. This allows smaller or lower funded
organizations to implement a powerful architecture that can protect the core
assets within budget.
We need to remember that these agents while equivalent
to super hackers are stopped by the same things that stop human hackers. The
issue is their speed, ability to attack en masse, and change tactics very
quickly. The concepts of defense in depth and zero-trust block changes in
tactics, and slow down the adversary.
Shifts in Response
The changes in roles in response to AI Cyberattack
Agents are minimal. The advent of these agents is simply all the things that
have happened before happening faster, and with more intelligence, variation
and completeness. Thus, the roles of those in charge of response and the
direction of the actions that should be taken will remain static. The main
element that these responders will need to keep available is the situational
awareness of what they are up against.
Sun Tzu said, “He will win who, prepared himself,
waits to take the enemy unprepared.” In the same way people in Security
Operations Centers prepare response plans and playbooks and the Organizations
they protect build out the tooling to support them plans for AI Cyberattack
Agents need to be developed. Countermeasures need to be devised that might
address the unique nature of these operations.
One aspect where the speed of agents works against an
attacker is that the agents may be so fast that the adversarial organization
may not have time to understand the landscape of their target before the tool
has gone too far. Unchecked speed leads to clumsiness and while highly capable,
it may be a long while before they learn the true meaning of “not clumsy.” This
is the area where planning and preparation can give the defenders home court
advantage. The trick is to not just plan for speed and change of tactics but
also for the error of moving too fast and other blunders that the highly
capable but inexperienced make. This preparation and planning means that the responders
and engineers in their orgs will need new tooling.
Shifts in the Tooling Landscape
For the most part the speed at which agents can attack
will be an asset, it can be expected that high speed fumbles will be rare. This
means that attacks will be fast, and they will iterate just as quickly, tactics
will shift potentially without pause. While there may not necessarily be more
data coming into the pipeline the data will come in faster. This assumes that
the detection data processing is timely enough to report the event in an
actionable time frame at all. Minutes may be too long. Part of the preparation
above is the tooling to handle event data in a timeframe that can create an
actionable alert. The tooling to reach this level is not trivial. What is
needed is the ability for products that protect the small organizations and
individuals. This would be a required product if this level of defense were
available to those who don’t have large development and IT budgets.
Additionally, we will need the ability to make
decisions rapidly based on security telemetry data. This might mean the ability
to process more locally rather than in aggregate to reduce the time it takes to
process a detection. If we have only minutes to detect, understand and respond
we will need tooling that can choose for us within the valid response timeframe
otherwise we would need to shift the roles of responders to the role of
recovery crew. A means to process faster is needed. Current systems may not be
lightweight enough. Agents will see paths through our systems that we never
understood could exist. LLMs are already experts at finding potential vuln
chains in NVD data. And this ability to receive alerts and telemetry is required
for the ability to make the necessary decisions for an effective response.
Effective decision-making capability within the
response timeframe is the item that will unlock the ability to defend at speed
and scale. Once the decision is made to take an action, we are in a race to
deploy that action before the situation or tactics of the adversary changes.
Configuration changes to systems and account locks and other defensive actions can’t
take several minutes to action. This means that we will need to have the ability
to communicate these without impediment or back up at a speed and scale we have
not seen before, possibly at speeds faster than human beings can keep up with.
Through this paper we have looked at what the
adversary can do with this technology. And clearly the last several paragraphs
have painted a bleak picture of a scale up few organizations have the
capability to carry out.[3] But they leave out one key
part of the response tooling that links it together and makes all of this
improved tooling exponentially more effective; and the thing that becomes
absolutely essential without the ability to deliver this tooling is the
utilization of AI Cyberdefense Agents. These are not SOC analyst replacements. Rather,
they are workers that can respond in fractions of a second, faster than any
person, to adversarial action. This includes action where the adversary is an
AI Cyberattck agent. AI Cyberdefense agents can be taught to analyze alert
data, follow playbooks, and seek assistance from humans with ambiguity
capability it was not trained for.
Development of such agents will follow the development
of the attack agents and will allow for the rapid response needed to block
attack agents. In fact, combined with the hardening techniques delivered through
the equally important but separate Cyber Reasoning Systems (CRS)[2], we might
just see the posture of the defense tilt to a position where attacks are more
difficult.
Conclusion
AI Cyberattack agents will become a reality in the
near future, and we must prepare. The steps to defend are several. 1) We must as
AI agents mature, build defense agents in tandem. 2) Push hard on full
production Cyber reasoning Systems to harden our deployed systems and code. 3)
Work to reduce the Cybersecurity telemetry processing time through distribution
or more powerful systems. Through these actions we can meet the threat of the AI
cybersecurity attack agent head on.
Sources Cited
[1] Developing
a computer use model \ Anthropic
[2] DARPA AI Cyber
Challenge Proves Promise of AI-Driven Cybersecurity
[3] GreyDGL/PentestGPT: A
GPT-empowered penetration testing tool
[4] berylliumsec/nebula:
AI-Powered Ethical Hacking Assistant
[5] 5
Holiday Tips to Secure Your Organization from Access Brokers
[6] Boesch, C., & Boesch, H. (1989). Hunting
behavior of wild chimpanzees in the Taï National Park. American Journal of
Physical Anthropology, 78(4), 547–573.
https://doi.org/10.1002/ajpa.1330780410
Crimson
Palace returns: New Tools, Tactics, and Targets – Sophos News
[8] Russian
Script Kiddie Assembles Massive DDoS Botnet
[9] Deepfake
scammer walks off with $25 million in first-of-its-kind AI heist - Ars Technica
[10] New
computer attack mimics user's keystroke characteristics and evades detection |
ScienceDaily
[11] Singh, K., Kumar, B., Saxena, R., & Lohani,
V. (2023). A Defense in Depth with Zero Trust Architecture for Securing 5G
Networks. 2023 31st Telecommunications Forum (TELFOR), 1–4.
https://doi.org/10.1109/TELFOR59449.2023.10372633
[1]
This stage is unlikely among cyber adversaries given that two actors with the
same capabilities are unlikely to need each other’s abilities. However, this may
be done to leverage scale in cases where the operation needs more man hours
than one actor has at their disposal.
[2] Indeed,
it is a failure all the way down – ideally there would have been checks and
required “paper trail” verifications at the administrative level, with behavioral
training and controls, with a final technical control on the financial
transaction that required a technical sign off to allow the transfer to go
through.
[3] Though this level
of capability is healthy, and they should no matter what.
No comments:
Post a Comment