Unsupervised Threat Hunting using Continuous Bag of Terms and Time (CBoTT)

2023 | International Conference on Information Systems | Citations: 0

Authors: Kayhan, Varol; Shivendu, Shivendu; Behnia, Rouzbeh; Daniel, Clinton; Agrawal, Manish

Abstract: Threat hunting is sifting through system logs to detect malicious activities th ... Expand

Abstract: Threat hunting is sifting through system logs to detect malicious activities that might have bypassed existing security measures. It can be performed in several ways, one of which is based on detecting anomalies. We propose an unsupervised framework, called continuous bag-of-terms-and-time (CBoTT), and publish its application programming interface (API) to help researchers and cybersecurity analysts perform anomaly-based threat hunting among SIEM logs geared toward process auditing on endpoint devices. Analyses show that our framework consistently outperforms benchmark approaches. When logs are sorted by likelihood of being an anomaly (from most likely to least), our approach identifies anomalies at higher percentiles (between 1.82-6.46) while benchmark approaches identify the same anomalies at lower percentiles (between 3.25-80.92). This framework can be used by other researchers to conduct benchmark analyses and cybersecurity analysts to find anomalies in SIEM logs. Collapse

Topics: IT security application programming interface Hadoop Distributed File System website web analytics

Methods: natural language processing artificial neural network support vector machine term frequency–inverse document frequency DBSCAN

Application Massive Data Processing Platform for Smart Manufacturing Based on Optimization of Data Storage

2023 | ACM Transactions on Management Information Systems | Citations: 0

Authors: Ren, Bin; Chen, Yuquiang; Wang, Fujie

Abstract: The aim of smart manufacturing is to reduce manpower requirements of the product ... Expand

Abstract: The aim of smart manufacturing is to reduce manpower requirements of the production line by applying technology of huge amounts of data to the manufacturing industry. Smart manufacturing is also called Industry 4.0, and the platform for processing huge amounts of data has an indispensable role. The massive data processing platform is like the brain of the entire factory, receiving all data from production line sensors via edge computing, processing, and analyzing, and finally making feedback decisions. With the innovation of production technology, the data that the platform needs to process has become diverse and complex, and the amount has become increasingly large. As well, many precision manufacturing industries have begun to enter the field of Industry 4.0. In addition to the accuracy and availability of data processing, there is emphasis on the real-time nature of data processing. After the sensor receives the data, the platform must provide feedback within a short period of time. This article proposes a massive data processing platform based on the Lambda architecture, which has the coexistence of stream processing and batch processing to meet real-time feedback needs of high-precision manufacturing. To verify the effectiveness of the optimization, it is based on real data from the manufacturing industry. To generate a large amount of test data to confirm the optimization of the storage of pictures. The results show that it optimizes the storage and optimization of the image data generated by the Automated Optical Inspection technology used in manufacturing today and optimizes the query for data storage. It also reduces the consumption of a large amount of memory as expected, and the query for Hive reduced the time spent. Collapse

Topics: Apache HBase Apache Hadoop Hadoop Distributed File System stream processing Python

Methods: experiment

Cache-Based Multi-Query Optimization for Data-Intensive Scalable Computing Frameworks

2021 | Information Systems Frontiers | Citations: 0

Authors: Michiardi, Pietro; Carra, Damiano; Migliorini, Sara

Abstract: In modern large-scale distributed systems, analytics jobs submitted by various u ... Expand

Abstract: In modern large-scale distributed systems, analytics jobs submitted by various users often share similar work, for example scanning and processing the same subset of data. Instead of optimizing jobs independently, which may result in redundant and wasteful processing, multi-query optimization techniques can be employed to save a considerable amount of cluster resources. In this work, we introduce a novel method combining in-memory cache primitives and multi-query optimization, to improve the efficiency of data-intensive, scalable computing frameworks. By careful selection and exploitation of common (sub)expressions, while satisfying memory constraints, our method transforms a batch of queries into a new, more efficient one which avoids unnecessary recomputations. To find feasible and efficient execution plans, our method uses a cost-based optimization formulation akin to the multiple-choice knapsack problem. Extensive experiments on a prototype implementation of our system show significant benefits of worksharing for both TPC-DS workloads and detailed micro-benchmarks. Collapse

Topics: Apache Spark distributed computing application programming interface database system operating system

Methods: experiment computational algorithm design artifact optimization model knapsack problem

TextBenDS: a Generic Textual Data Benchmark for Distributed Systems

2021 | Information Systems Frontiers | Citations: 0

Authors: Truică, Ciprian-Octavian; Apostol, Elena-Simona; Darmont, Jérôme; Assent, Ira

Abstract: Extracting top-k keywords and documents using weighting schemes are popular tech ... Expand

Abstract: Extracting top-k keywords and documents using weighting schemes are popular techniques employed in text mining and machine learning for different analysis and retrieval tasks. The weights are usually computed in the data preprocessing step, as they are costly to update and keep track of all the modifications performed on the dataset. Furthermore, calculation errors are introduced when analyzing only subsets of the dataset, i.e., wrong weighting are computed as weighting schemes use the number of documents for scoring keywords and documents. Therefore, in a Big Data context, it is crucial to lower the runtime of computing weighting schemes, without hindering the analysis process and the accuracy of the machine learning algorithms. To address this requirement for the task of computing top-k keywords and documents (which largely relies on weighting schemes), it is customary to design benchmarks that compare weighting schemes within various configurations of distributedframeworks and database management systems. Thus, we propose TextBenDS - a generic document-oriented benchmark for storing textual data and constructing weighting schemes. Our benchmark offers a generic data model designed with a multidimensional approach for storing text documents. We also propose using aggregation queries with various complexities and selectivities for constructing term weighting schemes, that are utilized in extracting top-k keywords and documents. We evaluate the computing performance of the queries on several distributed environments set within the Apache Hadoop ecosystem. Our experimental results provide interesting insights. As an example, MongoDB shows the best overall performance, while Spark’s execution time remains almost constant regardless of weighting schemes. Collapse

Topics: information retrieval MongoDB database system distributed system MapReduce

Methods: term frequency–inverse document frequency Okapi BM25 experiment design science qualitative content analysis

LogGAN: a Log-level Generative Adversarial Network for Anomaly Detection using Permutation Event Modeling

2021 | Information Systems Frontiers | Citations: 0

Authors: Xia, Bin; Bai, Yuxuan; Yin, Junjie; Li, Yun; Xu, Jian

Abstract: System logs that trace system states and record valuable events comprise a signi ... Expand

Abstract: System logs that trace system states and record valuable events comprise a significant component of any computer system in our daily life. Each log contains sufficient information (i.e., normal and abnormal instances) that assist administrators in diagnosing and maintaining the operation of systems. If administrators cannot detect and eliminate diverse and complex anomalies (i.e., bugs and failures) efficiently, running workflows and transactions, even systems, would break down. Therefore, the technique of anomaly detection has become increasingly significant and attracted a lot of research attention. However, current approaches concentrate on the anomaly detection analyzing a high-level granularity of logs (i.e., session) instead of detecting log-level anomalies which weakens the efficiency of responding anomalies and the diagnosis of system failures. To overcome the limitation, we propose an LSTM-based generative adversarial network for anomaly detection based on system logs using permutation event modeling named LogGAN, which detects log-level anomalies based on patterns (i.e., combinations of latest logs). On the one hand, the permutation event modeling mitigates the strong sequential characteristics of LSTM for solving the out-of-order problem caused by the arrival delays of logs. On the other hand, the generative adversarial network-based model mitigates the impact of imbalance between normal and abnormal instances to improve the performance of detecting anomalies. To evaluate LogGAN, we conduct extensive experiments on two real-world datasets, and the experimental results show the effectiveness of our proposed approach on the task of log-level anomaly detection. Collapse

Topics: Hadoop Distributed File System high-performance computing source code

Methods: anomaly detection experiment generative adversarial network isolation forest computational algorithm

Real-time big data processing for anomaly detection: A Survey

2019 | International Journal of Information Management | Citations: 0

Authors: Ariyaluran Habeeb, Riyaz Ahamed; Nasaruddin, Fariza; Gani, Abdullah; Targio Hashem, Ibrahim Abaker; Ahmed, Ejaz; Imran, Muhammad

Abstract: The advent of connected devices and omnipresence of Internet have paved way for ... Expand

Abstract: The advent of connected devices and omnipresence of Internet have paved way for intruders to attack networks, which leads to cyber-attack, financial loss, information theft in healthcare, and cyber war. Hence, network security analytics has become an important area of concern and has gained intensive attention among researchers, off late, specifically in the domain of anomaly detection in network, which is considered crucial for network security. However, preliminary investigations have revealed that the existing approaches to detect anomalies in network are not effective enough, particularly to detect them in real time. The reason for the inefficacy of current approaches is mainly due the amassment of massive volumes of data though the connected devices. Therefore, it is crucial to propose a framework that effectively handles real time big data processing and detect anomalies in networks. In this regard, this paper attempts to address the issue of detecting anomalies in real time. Respectively, this paper has surveyed the state-of-the-art real-time big data processing technologies related to anomaly detection and the vital characteristics of associated machine learning algorithms. This paper begins with the explanation of essential contexts and taxonomy of real-time big data processing, anomalous detection, and machine learning algorithms, followed by the review of big data processing technologies. Finally, the identified research challenges of real-time big data processing in anomaly detection are discussed. Collapse

Topics: big data Apache Hadoop fraud detection fuzzy logic Hadoop Distributed File System

Methods: anomaly detection machine learning computational algorithm decision tree classification support vector machine

Theories: fuzzy set theory

Social Media for Nowcasting Flu Activity: Spatio-Temporal Big Data Analysis

2019 | Information Systems Frontiers | Citations: 0

Authors: Hassan Zadeh, Amir; Zolbanin, Hamed M.; Sharda, Ramesh; Delen, Dursun

Abstract: Contagious diseases pose significant challenges to public healthcare systems all ... Expand

Abstract: Contagious diseases pose significant challenges to public healthcare systems all over the world. The rise in emerging contagious and infectious diseases has led to calls for the use of new techniques and technologies capable of detecting, tracking, mapping and managing behavioral patterns in such diseases. In this study, we used Big Data technologies to analyze two sets of flu (influenza) activity data: Twitter data were used to extract behavioral patterns from a location-based social network and to monitor flu outbreaks (and their locations) in the US, and Cerner HealthFacts data warehouse was used to track real-world clinical encounters. We expected that the integration (mashing) of social media and real-world clinical encounters could be a valuable enhancement to the existing surveillance systems. Our results verified that flu-related traffic on social media is closely related with actual flu outbreaks. However, rather than using simple Pearson correlation, which assumes a zero lag between the online and real-world activities, we used a multi-method data analytics approach to obtain the spatio-temporal cross-correlation between the two flu trends and to explain behavioral patterns during the flu season. We found that clinical flu encounters lag behind online posts. Also, we identified several public locations from which a majority of posts initiated. These findings can help health authorities develop more effective interventions (behavioral and/or otherwise) during the outbreaks to reduce the spread and impact, and to inform individuals about the locations they should avoid during those periods. Collapse

Topics: Twitter social media healthcare data big data pandemic

Methods: descriptive statistic longitudinal research autocorrelation analysis spatial analysis machine learning

Theories: motivation theory

Evaluating Queries and Updates on Big XML Documents

2018 | Information Systems Frontiers | Citations: 2

Authors: Bidoit, Nicole; Colazzo, Dario; Malla, Noor; Sartiani, Carlo

Abstract: In this paper we present Andromeda, a system for processing queries and updates ... Expand

Abstract: In this paper we present Andromeda, a system for processing queries and updates on large XML documents. The system is based on the idea of statically and dynamically partitioning the input document, so as to distribute the computing load among the machines of a MapReduce cluster. Collapse

Topics: extensible markup language Apache Hadoop database system Java Hadoop Distributed File System

Methods: computational algorithm experiment case study literature study

A Process Pattern Model for Tackling and Improving Big Data Quality

2018 | Information Systems Frontiers | Citations: 9

Authors: Wahyudi, Agung; Kuk, George; Janssen, Marijn

Abstract: Data seldom create value by themselves. They need to be linked and combined from ... Expand

Abstract: Data seldom create value by themselves. They need to be linked and combined from multiple sources, which can often come with variable data quality. The task of improving data quality is a recurring challenge. In this paper, we use a case study of a large telecom company to develop a generic process pattern model for improving data quality. The process pattern model is defined as a proven series of activities, aimed at improving the data quality given a certain context, a particular objective, and a specific set of initial conditions. Four different patterns are derived to deal with the variations in data quality of datasets. Instead of having to find the way to improve the quality of big data for each situation, the process model provides data users with generic patterns, which can be used as a reference model to improve big data quality. Collapse

Topics: marketing management data quality relational database spreadsheet Apache Hadoop

Methods: case study longitudinal research machine learning time series analysis single case study

Benchmarking Big Data Technologies for Energy Procurement Efficiency

2017 | Americas Conference on Information Systems | Citations: 0

Authors: Fritz, Manuel; Albrecht, Simon; Ziekow, Holger; StrÃ, Jens

Abstract: The electrical power industry is undergoing radical change due to the push for r ... Expand

Abstract: The electrical power industry is undergoing radical change due to the push for renewable energy that makes energy supply less predictable. Smart meters along with analytics software can grant insights into customerspecific consumption and thereby enable a better match between the demand and supply side for an electric utility. However, the vast amount of allocatable smart metering data and complexity of analytics pose challenges to database system. We address the implementation of an analytics approach to optimize customer portfolios, eventually preventing excess energy procurement. Using real-world and simulated data, we test the suitability of big data approaches as well as traditional relational database technology. Furthermore, we present solutions based on big data platforms and demonstrate their cost effectiveness and performance. Our findings suggest economic feasibility of big data solutions for large utilities. Small and medium-sized utilities are advised to invest in more cost-effective solutions such as cluster-based systems. Collapse

Topics: big data PostgreSQL procurement management database system relational database

Methods: experiment descriptive statistic literature study design artifact design science