How anomaly detection can help today’s business world

Imagine a scenario where your credit card gets stolen. It is very likely that the thief will go on a shopping spree and spend large amounts of your money. Many payment processor companies like PayPal keep tabs on the usage patterns of your card. The usage pattern accounts for transaction amounts, the location of the transactions etc. In case there are any drastic changes in your credit card transactions, the company will inform you of it. This is an example where companies use the concept of anomalies to detect any abnormal transactions that may occur after the theft of a credit card.  

Anomaly detection is a technique that is used to identify items or events that do not conform to an expected pattern. These items are termed as anomalies or outliers. Nowadays, businesses need to handle rapidly changing data sets. They must send out prompt responses to them as well, especially in cases of cyber attacks. Anomaly detection is a commonly used solution for such threats as it immediately detects and identifies any abnormal behaviour. 

Anomalies can be broadly categorized as:

• Point anomalies: Here, an individual instance of data is considered as anomalous if it is very different from the rest. For example, high value transactions
• Contextual anomalies: This is a data instance that is anomalous in a particular context, but not otherwise. This type of anomaly is common in time-series data. For example, high temperatures in winter
• Collective anomalies:  If a collection of related data instances is anomalous with respect to the entire dataset, but not individual values. 

The demo video below shows how an Anomlay detection system can detect different typed of anomalies and execute predefined actions. 

How is anomaly detection carried out?

The algorithms to detect anomalies are derived from two types of machine learning techniques-

• Supervised machine learning

Supervised machine learning algorithms are trained by example. We provide them with datasets that have categorized examples, and this allows the algorithm to develop a predictive model of each category. Then, the algorithm processes the actual data and tries to place each item into one of the pre-learned categories. As a supervised algorithm can only recognize the categories that it has been trained with, a supervised machine learning algorithm cannot place an item into a category it has not seen an example of. This implies that an automated anomaly detection system built on such an algorithm would have to be given examples of every single possible type of anomaly on every possible data distribution, pattern and trend.The most common supervised algorithms are supervised neural networks, parameterization of training models, support vector machine learning, k-nearest neighbors, Bayesian networks and decision trees.

• Unsupervised machine learning

These techniques do not need training data. They depend on two primary assumptions. Firstly, they assume that the majority of the network connections are regular traffic and only a small percentage is anomalous. Secondly, they anticipate that abnormal traffic is statistically different from normal traffic. Based on these two assumptions, data groups of similar instances that appear frequently are assumed to be normal traffic and those data groups that are infrequent are considered to be malicious. The most common unsupervised algorithms are self-organizing maps (SOM), K-means, C-means, expectation-maximization meta-algorithm (EM), adaptive resonance theory (ART), and one-class support vector machine.

Where can we use anomaly detection?

• Intrusion detection systems

Many companies have to prepare themselves for cyber attacks. This can be done with intrusion detection systems that use anomaly detection algorithms. If unusual network traffic or abnormal user actions are detected, these algorithms can immediately alert the concerned authorities.

• Detecting fraudulent transactions 

This is one of the most important use cases for anomaly detection. In the case of credit card theft, abnormal usage patterns of the card can be detected. Consequently, companies can inform the cardholder of these suspicious activities. In telecommunication, the calling behavior of users can be scanned for abnormal behavior. Anomaly detection can also be used for insurance claim fraud detection. Generally, claim adjusters and investigators analyze claims for fraud. These manually verified cases are used as training data for supervised and semi-supervised techniques of anomaly detection. 

• Electronic sensor events 

When a sensor becomes dysfunctional, it fails to capture data properly and hence, produces anomalies.  Sometimes, there can be abnormal changes in the data sources as well. There are sensors in many of the IoT applications that we see today. When these sensors start to behave inconsistently the signals they convey get also uncanny, thereby causing unprecedented troubleshooting. Hence, systematic anomaly detection is a must here.

• Medical outlier detection

Medical records generally contain information like patient age, blood group, height, weight, etc. There may be anomalies in this data due to several reasons like abnormal patient condition or instrumentation errors or recording errors. The majority of the anomaly detection techniques that are seen in this domain aim at detecting anomalous records (point outliers). Generally, the labeled data is acquired from healthy patients, hence most of the techniques adopt a semi-supervised approach. Another type of data that can be seen in this field is time-series data, such as Electrocardiograms (ECG) and Electroencephalograms (EEG). Collective outlier detection techniques have been applied to detect anomalies in such data.

• Image processing

Here, anomaly detection aims to identify changes in an image over a certain period of time (motion detection) or in regions that appear abnormal on the static image. This domain includes digit recognition, satellite imagery, mammographic image, spectroscopy, and video surveillance. The anomalies arise from motion or insertion of foreign objects or instrumentation errors. The data has spatial as well as temporal characteristics. Each data point has a few continuous characteristics such as color, lightness, texture, etc. 

One of our clients was experiencing major challenges in managing warehouse operations. Unfortunate events like intruder activity, fires, leakages, etc. kept occurring. There were many instances where employee misconduct led to damages on the premises as well. However, the human resources that were employed for security surveillance were not efficient enough to take timely action to handle such mishaps. The client needed a software solution that could detect the occurrence of such problems so that the situation could be handled appropriately.

One of our products, Emotyx could deliver actionable insights on any abnormal events that occurred on the premises. Emotyx used anomaly detection techniques to detect and identify any threats or abnormal activities. If a threat is detected, an alert will be sent to the concerned authorities. This helped the company deal with problems like unauthorized entry, employee misconduct, etc. 

After implementing Emotyx, our client reduced losses caused by accidents by 80%. There was also a distinct improvement in warehouse management as the company was able to respond to threats promptly. An increase in the overall efficiency of the business was observed due to decreased employee misconduct and accidental damages.

Anomaly detection can be used in a wide range of industries like banking, telecommunication, manufacturing, and healthcare. Such techniques can identify threats and abnormalities much faster than human beings. This efficiency and speed are extremely important in cases like identity theft and the detection of illnesses. Nowadays, organizations must deal with large amounts of data that can only be handled by anomaly detection systems. These platforms are essential for reacting to changes in data immediately.