Security professionals constantly get alerts, warning them of anomalies. These early warnings are imperative for effectively mitigating damages from a cyber attack – sometimes allowing infosec pros to halt attacks before they can breach sensitive data. False alarms do happen though, and over time, infosec pros become desensitized to the these alerts.
The key challenge facing enterprises is finding the right balance between false alarms and not enough alerts. How can we avoid tuning out the constant alerts and reduce the risk of ignoring the one alert that really matters? To gain some insight into how today's leading cybersecurity professionals are combating this pervasive issue, we reached out to a panel of infosec pros and asked them to answer this question:
"How can infosec pros avoid cyber alert fatigue?"
Find out what you should be doing to avoid cyber alert fatigue and see what our pros had to say below.
Gareth Marchant
Gareth Marchant has over 20 years’ experience in information technology leadership, infrastructure, systems engineering and cybersecurity. Gareth is an instructor at Learning Tree International, and teaches a variety of hands-on courses in information technology systems administration, cybersecurity and certification prep. He currently holds CISSP, CEH,GMON, Security+ and ITIL certifications.
"The appeal of security monitoring solutions is..."
Their ability to identify and alert on many, many events. While this feature may be appealing on the surface, it is actually a serious cause for concern. Once a monitoring system is put online, and it starts to generate alerts based on observed triggers, cyber alert fatigue can quickly follow. Here are three suggested approaches to minimize alert fatigue:
- Tune and optimize alerts. The central purpose of a monitoring system is to provide valuable operational awareness. If alerts are being ignored, filtered or missed in the deluge, this represents a huge objective failure. To combat this, it is critical to review the makeup of all alerts and identify the top talkers. It is common to identify that a single event trigger or system is the cause of the majority of alerts. Simply tuning the event trigger to more appropriate values or addressing problems on a single system can vastly improve the quality and validity of alerts.
- Include context to help determine importance. Single events by themselves can seem innocuous but included in the context of other events can quickly be identified as significant. Some of the most effective monitoring solutions include the capability to correlate events in meaningful ways which trigger alerts with actionable information, such as indicators of compromise based on a series of related events.
- Generate alerts in different ways. A single alerting channel is the simplest to set up but can lead to alert fatigue more quickly. Combining different alert types can offset fatigue. For example, some event types should be reported to a dashboard whereas others should generate an email or write to a log to be collected by a log consolidation system. To maintain the impact and effectiveness of a security monitoring system, it is essential to constantly tune and modify its operation to reduce false alerts, improve alert quality for effective intervention and reduce alert fatigue.
Jay Strickland
Jay is the CEO of Strategic Solutions of Virginia, an IT Managed Services Provider focused on the healthcare vertical market.
"We have spent a great deal of time thinking about ways to combat alert fatigue..."
We monitor and respond to operational and security alerts for our customer base 24x7x365. Many of our customers generate in excess of 1.2 million log entries each per month. Bringing awareness and insight into that volume of data CONSTANTLY is a daunting task. To combat alert fatigue an organization like ours MUST:
- Develop a culture of action - In a high volume Security Operations Center (SOC), there is always a risk of under responding. We try to instill a sense of ACTION in our team. Each member has the ability to individually alert management and customers any time they see something that isn't right. We also frequently review news stories of breaches to make the risk to our team more real. This leads us to all feel that it is far better to over-respond than to miss something.
- Develop an understanding of impact - By fostering a strong sense of customer (or business unit) awareness we make sure our team is very connected to the people, workflows and data that we are trying to protect. Our team members routinely do behind the scenes tours of our customer base to make sure they understand the impact to our customers' employees and our customers' customers. As an example, it is much easier to understand the risk of a breach of protected health information when you have walked through and spent time in a cancer treatment facility.
- Develop multiple layers of checks and balances - Alerts are monitored for action and in the event that an individual misses a response, we have workflow rules that cause escalations of the alerts and notifications of the management team. All of this information is visible to all team members on real-time dashboards in our SOC.
- Measure - We subscribe to the old adage that, what gets measured gets managed. We know that each alert response must be recorded and measured to be sure that we are meeting all service levels needed (in terms of speed and completeness) to protect our customers.
- Teach - We believe that we are better at responding to alerts when we have spent time educating our customers/partners and others in the ecosphere of the risks. We believe that teaching makes us better operationally.
- Reduce the noise - Organizations like ours must have an operational resource that is VERY focused on reducing non-actionable alerts. Any false-alert or informational item adds to the overall workload and increases the chance of a serious threat being missed. We believe it is important to have an Actionability Czar.
- Understand your opponent - Last and probably most importantly we work very hard to understand the threats and bad actors that we face. We know that teams that are engaged in researching new threats and identifying countermeasures are going to be much more action-oriented! We feel that any SOC needs to be involved in an Information Sharing and Analysis Center (ISAC).
Lenny Zeltser
Lenny Zeltser is a seasoned business and tech leader with extensive information security expertise. He builds innovative endpoint defense solutions as Vice President of Products at Minerva Labs. Beforehand, as a product portfolio owner at NCR, he delivered the financial success and expansion of the company's security services and SaaS products.
"The global shortage of IT security personnel results in..."
Understaff and overworked teams tasked with handling the alerts. This factor, combined with the overwhelming number of alerts that need to be handled on an ongoing basis, creates an imbalance. In turn, many important alerts go unnoticed or are disregarded even though they could be indicators of actual attacks.
Organizations need to assess whether the security architecture of the enterprise can be strengthened to stop adversarial actions earlier in the attack process. This might involve supplementing existing defenses with a layer that increases the effectiveness of preventative controls without overlapping with the tools already in place. If the enterprise can automatically stop more threats before they warrant a human's intervention and investigation, the organization will decrease the number of alerts and related events that IT personnel will need to handle. The result? Less noise in the alert stream, and more time for the team to dig into the alerts that truly warrant attention.
When evaluating security products, pay attention to the amount of noise they will contribute to the workload of your IT security staff. For instance, some products will typically generate more false positives than others and, therefore, will contribute toward alert fatigue. In addition, be sure to tune the product or the way in which you filter its output to avoid meaningless alerts that consume time and energy without providing useful and actionable information.
Kathie Miley
Kathie has 25 years of experience in the information technology and security field, and is currently serving as the Chief Operating Officer at Cybrary, Inc, the world’s first crowdsourced platform for cybersecurity and IT learning.
"A former colleague once told me..."
That analyzing security alerts is like looking for a needle in the haystack of needles. The problem is so severe that according to a 2017 survey from Cloud Security Alliance, only 23.2% of 2,500 alerts were real – meaning a staggering 76.8% were false positives. The result – SOC analysts suffering from False Positive PTSD. In the past, the only advice available would have been a balanced diet, 6 hours of sleep, say your prayers, and have one hell of a good security operations team. Today, technologies such as machine learning, behavior analysis, and AI have introduced nearly instantaneous detection, mitigation, and ongoing protection on the core of the endpoint. This dramatically reduces the volume of alerts, and allows SOC analysts and IR agents to focus on active threat hunting. Getting educated and starting to actively prevent intrusions truly is the only way to inoculate against alert fatigue.
Siobhan McNamara
Siobhan McNamara is a data scientist working on an email security solution for Agari.
"Threat alert fatigue comes about when there is..."
An abundance of false positives or benign content that security software mistakenly classifies as malicious. If you cast a wide net you can catch fraud, but inherently there will be a lot of additional false positives. In the case of targeted attacks, threat alert fatigue is particularly threatening because sophisticated socially engineered content will pass human filtering.
The best solution to avoiding threat alert fatigue is to build models into a security solution to learn patterns of false positives. What this means is security software that uses machine learning models to classify malicious from benign content is not limited to one model. You can build a hierarchy of models all of which are trained to learn different nuances of the problem at hand. While core models might generate a lot of false positives, you can adopt more granular models on top. If you scan your false positives regularly you will notice common patterns among them, characteristics of false positives that are recurring. From these real world examples you can build features. With that you can train a new model to learn this category of false positives. You can adapt your features and your model until you are happy the output is not allowing malicious content to pass but is reducing the false positives you have in mind. Then you can add this to the software and have it override output from your core models that it matches.
Threat alert fatigue is a given with overzealous security solutions. The best solution is to build machine learning models to learn false positives and to automate it. Over time, from concrete examples you see you can develop features and new models and rules and heuristics to bump a score up or down. Some false positives will always be there. There will never be no error but you want to get it low enough so you don't experience threat alert fatigue.
Robert Siciliano
Robert Siciliano, CSP, a #1 Best Selling Amazon.com author and a security expert with Hotspot Shield, is serious about security awareness training. Robert is a security expert and private investigator fiercely committed to informing, educating, and empowering people so they can protect themselves, both in their physical and virtual interactions.
"Preventing alert fatigue begins with..."
Filters and processes designed to create a framework that ignores the false positives and focuses on legitimate security alerts. This is done by utilizing security information and event management (SIEM) tools that are designed to assign priority and categorize alerts. When artificial intelligence is deployed, i.e., behavioral analysis, machine learning, and predictive analysis; security professionals will have a better handle on the basics of alert management. That said, one should never rely on tech by itself. By combining human and machine intelligence with constant tuning of the tech, administrators can reduce fatigue.
Gregory Morawietz
Gregory is an IT Security Specialist with Single Point of Contact with over twenty years' of network and security experience. He has worked with hundreds of firms on improving IT environments and consulting and integrating technology for the enterprise network.
"It is difficult to keep up with the large volume of alerts..."
The first thing to do is get the right SIEM. Then you need to make sure that your SIEM is current and well maintained. Secondly, you want to rely on products that have machine learning that can aggregate security alerts. Finally, you want to remediate security alerts so that they do not continue to replicate and flood the system. SIEM, Machine Learning, and Remediation will help combat alert fatigue.
Lindsey Havens
Lindsey Havens works at PhishLabs.
"Cybersecurity professionals are..."
Hit with having to come up with so many solutions to various security threat problems throughout their professional career which results in fatigue over time. One way to avoid this is to not always seek niche solutions, but instead look for multi-point solutions that can address the latest attacks without spreading resources too thin. A solution that solves multiple problems and performs respectably can reduce operational and expertise costs exponentially.
Amit Bareket
Amit Bareket is the Co-Founder and CEO of Perimeter 81 (powered by SaferVPN). He is an entrepreneur and cybersecurity expert with extensive experience in system architecture and software development. He graduated Cum Laude with a B.Sc. in Computer Science and Economics from Tel Aviv University.
"With the increasing adaptation of data science and machine learning..."
Many security systems involve some form of incident alerting. While many organizations implement multiple security tools, IT security teams are becoming overwhelmed with alert fatigue.
The Issue with Alert Systems
The average enterprise monitors 2 billion cloud-related events a month. A study by the Cloud Security Alliance found that 31.9% of respondents ignore alerts due to false positives while 40.4% indicate that the alerts they receive lack actionable intelligence to investigate. Even more concerning is that during actual incidents, 27.7% do not generate alerts at all.
High Alert Accuracy Is Essential
Alert fatigue has led to very real threats being ignored. For example, in 2014, one of Target’s security products detected a breach, but due to the high volume of alerts and false positives, the company’s IT security team did not take action. This neglect of important information cost the company $252 million and led to the resignation of its CIO and CEO.
An Effective Cloud Threat Protection Solution
Security solutions that deliver incessant false positives depreciate their purpose. To combat alert fatigue, endpoint protection platforms should keep false positives to a minimum. To achieve this, businesses should implement the Software Defined Perimeter (SDP) solution. According to The Cloud Security Alliance, the SDP security model has been shown to stop all forms of network attacks including DDoS, Man-in-the-Middle, Server Query (OWASP10) as well as Advanced Persistent Threats.
How SDP Solves Alert Fatigue
The Software Defined Network allows businesses to significantly enhance the level of network security of an organization. This eliminates the need for multiple security tools and replaces them with advanced auditing and alerting capabilities that continuously adjust and improve. Incorporating Software Defined Technologies provides ease of use along sophisticated, advanced and flexible platforms.
Nathan Wenzler
Nathan Wenzler is the Chief Security Strategist at AsTech, a leading information security consulting firm. Wenzler has more than two decades of experience designing, implementing and managing both technical and non-technical solutions for IT and information security organizations.
"As more organizations invest in their security programs..."
And deploy additional tools to bolster their layers of defense, they create more visibility into the security posture of their entire network. This, of course, is most commonly done through the huge volume of events and alerts which are generated by the activity detected from endpoints to network infrastructure devices to applications and user activities. Security Information and Event Management (SIEM) tools were meant to aggregate the millions upon millions of events generated and allow security professionals and administrators a way to filter through the noise to find the most important, high-priority events that needed attention. Even these tools can struggle to bring only the most pertinent items up to the attention of those who need it, leading to huge volumes of alerts that must be reviewed and dealt with almost constantly.
Of course, now a new wave of tools promises to fix this problem, too: behavioral analysis tools, machine learning tools, predictive analysis, and so on. However, at the end of the day, what administrators and security pros need to focus on is getting a better handle on the basics of managing their environment to create a stronger baseline for what they expect to happen. For example, properly managing your administrator credentials everywhere in your environment should create a scenario where these types of credentials are only used from authorized endpoints by authorized users and for specific activities. If controlled to that level, it means that any other activity by an unexpected credential or user is going to be a valid event that needs more immediate attention, and fewer alerts need to be generated for analysis by whatever tools are in use.
There are many other fundamental pieces of a security program that, if well-managed, can also reduce the number of less important or even truly false positive events that ultimately require humans to review and take action upon. Things like strong patch management programs, secure coding processes and security assessments for application vulnerabilities, and enforcing consistent least use privilege access concepts across the board will collectively decrease the number of potentially malicious events that would need to be reviewed. This makes any alerts triggered by your various monitoring systems much more likely to be a true positive event that requires attention, requiring less time and human resources to review, sort through and determine if it is a priority or not. As time goes on, many of the machine learning or behavioral analysis tools will get more sophisticated and find more accurate programmatic ways to filter through these events, but until then, dialing in the fundamentals that can reduce false positives is an ideal way to reduce the fatigue to admins and security personnel who have to sift through the events and alerts generated.
Stephen Tullos
Stephen Tullos is My IT's Cybersecurity Team Leader, a retired Army Ranger, and active Air Force Reserve in Cybersecurity.
"You have to tune out the noise by..."
Eliminating meaningless alerts, proactively stopping preventable issues from arising, and by utilizing automation. For our clients that need extra levels of cybersecurity, such as for a compliance like HIPAA, we use a SIEM (Security Information and Event Management) tool to parse through logs and to find the proverbial needle in the haystack. SIEM uses policies and AI (Artificial Intelligence) to distinguish the two or three concerns that need addressing while ignoring the thousands of alerts generated by an endpoint so we can focus on what matters without being flooded by false positives.
Isaac Kohen
Isaac Kohen is the founder and CEO of Teramind, an employee monitoring and insider threat prevention platform that detects, records, and prevents malicious user behavior.
"Given the value of data and the risk of data breach..."
Listening for warning signs caused by negligent or malicious insiders is one of the most impactful steps an infosec professional can take today. The challenge is to add the right listening technology so security teams aren’t over-run with more alerts that deliver little insight. Technology that incorporates machine learning and user behavior analysis provides intelligent alerting and minimizes alert fatigue. Ideally, the technology will also support configuration to enable teams to block, log out, or lock out a user when a serious threat is detected, bypassing the alert process completely.
Mark Wilcox
Mark is the Vice President of ICSynergy and has been working in cybersecurity since 1991 when he ran the computer lab at his university. He is considered one of the top experts in identity management and holds 2 patents in cybersecurity. When not helping customers solve their identity management problems, he likes to spend his time working on creating LEGO stop-motion movies.
"Here's how I avoid cyber-alert fatigue..."
- Focus on implementing best practices that work regardless of the threat environment.
- Develop patience. When a major alert comes out such as the recent Meltdown issue wait 72 hours before developing any opinion or concern. Then consult trusted sources for their opinion on the matters.
- Avoid TV shows (like Mr. Robot) which deal with using cybersecurity as the core of their plot. I understand that show is well done, but after 12 hours of dealing with cybersecurity, I don't see any benefit of getting engrossed even further in the subject.
Andrew S. Townley
Andrew is the founder and CEO of Archistry, a global business consultancy dedicated to turning strategy into reality. He is also a SABSA-certified Security Architect, and has over 20 years of security, IT, business and management consultancy experience creating client success on five continents.
"In principle, the answer is the same as for anything that gives you alerts..."
You need to know the context to avoid alert fatigue.
We faced this same problem working with a threat response team at a Global 300 petrochemical based in the Middle East. They have great sources of threat intelligence and a good team, but the issue was they didn’t really have any way to contextualize the information they were getting to their own environment.
If a threat intelligence vendor said the sky was falling, that intelligence either a) assumes you don’t have controls in place or b) that your existing controls will fail in the face of the threat. Basically, this is the same as conducting an inherent risk assessment. It’s valuable and interesting, but it doesn’t reflect the actual state of concern the organization should have.
To figure out whether it’s “Headless Chicken Time” or not, you need to compare the information you have about the threat against your control capabilities and the resources you’re protecting. The problem is, most organizations don’t have a good way to do this or a good way to measure the control capabilities they already have.
You can really only accomplish this if:
- You have a common language across the risk, threat, architecture and operations security teams to describe and rate threats and control capabilities.
- You have a way in your architecture method to understand how many layers of control you have in place that are relevant to the threat.
- You have consistency in the methods used by the risk and threat teams doing static and active risk assessments.
Without these things in place, it’s like lying in bed at night wondering if what set your alarm off was an untrimmed plant blowing in the wind or someone trying to break in and steal your TV. If you have no way of knowing, you have no choice but to react every time, and, eventually, you get tired, you get it wrong, and you wake up to someone you don’t know walking around your bedroom taking your stuff.
Jay Akin
Jay Akin is the CEO of Mushroom Networks, a networking company that builds advanced routers and firewall appliances, that have the capability of combining two or more Internet lines and providing intelligence about the network.
"To avoid cyber alert fatigue..."
It is important to have a filtering mechanism that is based on predetermined rules. Ideally, your monitoring system has the ability and intelligence to do this filtering for you, with perhaps minimal input from the user tightening or loosening up the thresholds. The newer filtering and intelligence systems are based on machine learning techniques, and can minimize false positives with high accuracy of detection.
Michelle Drolet
Michelle Drolet is the CEO of Towerwall, a data security services provider in Framingham, Mass.
"The only way to avoid cyber alert fatigue is to..."
Lay out plans with assigned roles among team members. Who's responsible for running regular network scans and doing the follow up? Whose job is it to mitigate after running a pen test? You must assign roles and responsibilities so everyone knows their relative position in the event of a serious breach. Otherwise you'll have many heads scrambling in panic. To avoid employee burn out have individuals cycle through the roles so that they receive a broad experience trying out new things.
Mario DiMarcantonio
Mario DiMarcantonio is the Owner of Progressive IT Solutions.
"It does get very noisy, very fast when..."
You are constantly receiving alerts about various issues, especially in the security realm. There are so many breaches, hacks, exploits and other cybercrimes that make the news. In addition, the various tools IT/Security professionals use for monitoring systems can generate a lot of alerts. In this industry, it's easy to feel overwhelmed. I personally feel that our brains tend to deal with cyber-alert overload the same way that it handles other stimulation, like email and spam. We tend to learn what is noise and what isn't by experience. On top of that, enabling filtering and rules to automatically process alerts accordingly helps. For example, you may learn over time which types of alerts are just noise. You can modify your alerting software to discontinue sending those, or have your email program file them in a special folder, etc. After a while, it's pretty easy to instantly identify the noise from those alerts with substance. The trick is to keep a sense of urgency and not fall into the trap of ignoring all of your alerts and then miss something important.
Peter Norman
Peter Norman is a Managing Member of Winnieware LLC and the Developer of ReplyToSome.
"In trying to understand alert fatigue..."
We reviewed research conducted by scholars who study human error, such as Don Norman and James Reason. Their research highlights the importance of what they call capture errors – cases where you are performing a relatively new activity that involves a sequence of steps very similar to that of a more common activity. Typically, the beginning of the sequences will be similar but the final steps will diverge. For example, the sequence of a deck of cards goes ... 6, 7, 8, 9, 10, Jack, Queen, King, Ace. But if you are not a frequent card player and are asked to list cards from memory while occupied with another task, you might count them out loud as 6, 7, 8, 9, 10, 11, 12, 13, 14.
Capture errors are closely related to alert fatigue. Often alerts will come at the end of a process. Take, for example, the normal sequence of actions for sending an email. First, you read a message, then press reply, then draft your response, then review it, and finally press send. Suppose you have a program that gives a warning after you press send, asking if you're sure you want to include an outside party. If you frequently intend to email outside parties and usually just disregard and click through the warning (i.e., because you intended to send the message to the outside party), then you are at risk of a capture error due to alert fatigue. Disregarding the warning becomes the most common activity and you are likely to respond to it by clicking through, even in cases where the warning is relevant.
Good designers can fight capture errors and thus many of the problems caused by alert fatigue in a few ways. First, they shorten the number of steps in a sequence – people are actually often more likely to pay attention to warnings on one click activities. Second, they provide immediate feedback on risks, not waiting until the end of a sequence. For example, if a user is completing a form, show errors as the user types rather than asking the user to confirm information after completing the entire form. Third, they reduce the number of false positives. This is often easier said than done, but targeted, infrequent warnings are much more likely to be effective.
Matt C. Pinsker
Matt C. Pinsker is a homeland security adjunct professor at Virginia Commonwealth University (VCU), published author in the field of cybersecurity, and former prosecutor and practicing criminal defense attorney experienced with computer crimes.
"The answer to reducing cybersecurity fatigue is simple..."
Do not send out alerts so frequently. Unfortunately, this is not entirely possible for reasons of legal liability where companies are obligated to notify of possible breaches. What companies can do is enhance their firewalls to decrease the number of breaches, which will decrease the number of notifications they do have to send. Second, companies can improve their speed and thoroughness of reviews of incidents to determine if anything actually happened, and if they need to send out a report.
Rodrigo Montagner
Rodrigo Montagner is the CEO and Founder of OM2 Tech Consulting Solutions.
"In order to be safe and sound from the alert fatigue of managing and thinking about risk possibilities in their environments..."
Cybersecurity pros need to learn how to rise above the chaos – and sometimes very harmful situations – without letting negative emotions take over them.
In other words, security pros should have or develop a personal repertoire of mental healthiness habits to avoid negative emotions whenever the going gets rough. Mental healthiness habits can include many different processes, such as a protocol of checking activities happening on a daily basis in your infrastructure, a silent office, light classical music in your headphones during crisis and, at the bottom of the drawer, the DR code and protocol that has been fully and continuously tested and documented.
Iurii Garasym
Iurii Garasym is the Head of Corporate Security at ELEKS and President of Cloud Security Alliance's Lviv Chapter. Iurii's professional goal is to make business survivable. He focuses on developments of the security program based on the emerging security solutions and integrates those into business goals, objectives, strategy and activities.
"The important questions to ask are..."
What can infosec pros do today to avoid cyber alert fatigue and what should the whole industry do tomorrow to understand the ROSI (return on security investments)?
Alerts are the results. Their numbers are caused by the quality of the internal processes and organisational context. What we have now: dozens of security tools that are efficient in their silos but not good enough as a whole system; tight budgets and no understanding of where to focus; increased usage of managed services and missing personal responsibility for the results; poor system integration. Involve your analysts and engineers in the market research to assess new opportunities, motivate them to automate routine work and focus on results and efficiency. Talk or at least listen to your organization. What is important to them: revenue growth, time to market, compliance, customer satisfaction? Clear and/or limit the scope of the security operation, understand their appetite to risk, and focus on the most critical points. Talk to your organization again to tune your “weapons” and produce the deliverables that they can use to drive the business forward. Soft skills matter. The industry is changing, and security should improve as well. You need to be on the same page with your business. While they innovate and transform, the security should keep up.
Chris Gonzales
Chris Gonzales is My IT's Chief Technology Officer (CTO) and has extensive knowledge designing complex networks and working in highly regulated industries including banking and oil & gas.
"Proactively preventing problems allows you to..."
Literally sleep better at night and reduces the unpredictable nature and burden that frequent alerts put on technology professionals. Staying ahead of the proverbial curve is easier to manage, less stressful, and better for a business' bottomline than always playing catch up, especially in regards to cybersecurity. You can spend the same amount of time fortifying your defenses to prevent a breach as you can remedying a breach, but the urgency is drastically reduced as is the downtime to the company, so the overall ROI (Return on Investment) of being proactive is a no- brainer.
