Mapreduce

HDT a is binary RDF serialization aiming at minimizing the space overheads of traditional RDF formats, while providing retrieval features in compressed space. Several HDT-based applications, such as the recent Linked Data Fragments proposal, leverage these features for diverse publication, interchange and consumption purposes. However, scalability issues emerge in HDT construction because the whole RDF dataset must be processed in a memory-consuming task. This is hindering the evolution of novel applications and techniques at Web scale. This paper introduces HDT-MR, a MapReduce-based technique to process huge RDF and build the HDT serialization. HDT-MR performs in linear time with the dataset size and has proven able to serialize datasets up to several billion triples, preserving HDT compression and retrieval features.

In this paper, we analytically derive, implement, and empirically evaluate a solution for maximizing the execution rate of Map-Reduce jobs subject to power constraints in data centers. Our solution is novel in that it takes into account the dependence of power consumption on temperature, attributed to temperature-induced changes in leakage current and fan speed. While this dependence is well-known, we are the first to consider it in the context of maximizing the throughput of Map-Reduce workdloads. Accordingly, we provide a new power model and optimization strategy for temperature-aware power allocation (TAPA), and modify Hadoop on a 13-machine cluster to implement our optimization algorithm. Our experimental results show that TAPA can not only limit the power consumption to the power budget but also achieves higher computational efficiency against static solutions and temperature oblivious DVFS solutions.

In last two decades continues increase of comput-ational power and recent advance in the
web technology cause to provide large amounts of data. That needs large scale data
processing mechanism to handle this volume of data. MapReduce is a programming model
for large scale distributed data processing in an efficient and transparent way. Due to its
excellent fault tolerance features, scalability and the ease of use. Currently, there are several
options for using MapReduce in cloud environments, such as using MapReduce as a service,
setting up one’s own MapReduce cluster on cloud instances, or using specialized cloud
MapReduce runtimes that take advantage of cloud infrastructure services. Cloud computing
has recently emerged as a new paradigm that provide computing infrastructure and large
scale data processing mechanism in the network. The cloud is on demand, scalable and high
availability so implement of MapReduce on the top of cloud services cause faster, scalable
and high available MapReduce framework for large scale data processing. In this paper we
explain how to implement MapReduce in the cloud and also have a comparison between
implementations of MapReduce on AzureCloud, Amazon Cloud and Hadoop at the end.

Pretty much every part of life now results in the generation of data. Logs are documentation of events or records of system activities and are created automatically through IT systems. Log data analysis is a process of making sense of these records. Log data often grows quickly and the conventional database solutions run short for dealing with a large volume of log files. Hadoop, having a wide area of applications for Big Data analysis, provides a solution for this problem. In this study, Hadoop was installed on two virtual machines. Log files generated by a Python script were analyzed in order to evaluate the system activities. The aim was to validate the importance of Hadoop in meeting the challenge of dealing with Big Data. The performed experiments show that analyzing logs with Hadoop MapReduce makes the data processing and detection of malfunctions and defects faster and simpler. Keywords— Hadoop, MapReduce, Big Data, log analysis, distributed file systems.

Twitter produces a massive amount of data due to its popularity that is one of the reasons underlying big data problems. One of those problems is the classification of tweets due to use of sophisticated and complex language, which makes the current tools insufficient. We present our framework HTwitt, built on top of the Hadoop ecosystem, which consists of a MapReduce algorithm and a set of machine learning techniques embedded within a big data analytics platform to efficiently address the following problems: (1) traditional data processing techniques are inadequate to handle big data; (2) data preprocessing needs substantial manual effort; (3) domain knowledge is required before the classification; (4) semantic explanation is ignored. In this work, these challenges are overcome by using different algorithms combined with a Naïve Bayes classifier to ensure reliability and highly precise recommendations in virtualization and cloud environments. These features make HTwitt different fro...

Apache Hadoop is a collection of open-source software utilities that facilitates using a network of many computers to solve problems involving massive amounts of data and computation. It provides a software framework for distributed storage and processing of big data using the MapReduce programming model. Hadoop was originally designed for computer clusters built from commodity hardware, which is still the common use. It has since also found use on clusters of higher-end hardware. All the modules in Hadoop are designed with a fundamental assumption that hardware failures are common occurrences and should be automatically handled by the framework.

The emergence of big data analytics as a way of deriving insights from data brought excitement to mathematicians, statisticians, computer scientists and other professionals. However, the absence of a mathematical foundation for analytics has become a real challenge amidst the flock of big data marketing activities, especially in healthcare insurance. This paper developed a mathematical model for the analytics of healthcare insurance data using set theory. We implemented a prototype of the model using Java Programming Language, MapReduce Framework, Association Rule Mining and MongoDB. Also, we tested the model accuracy using data from the National Health Insurance Scheme in Nigeria to reduce delay. The result showed that the accuracy level was 97.14% on average, which depicts a higher performance for the model. This result implies that delays affecting the processing of data submitted by the providers and enrollees to the HMOs reduced drastically leading to the improvement in the eas...

Data outsourcing allows data owners to keep their data at untrusted clouds that do not ensure the privacy of data and/or computations. One useful framework for fault-tolerant data processing in a distributed fashion is MapReduce, which was developed for trusted private clouds. This paper presents algorithms for data outsourcing based on Shamir's secret-sharing scheme and for executing privacy-preserving SQL queries such as count, selection including range selection, projection, and join while using MapReduce as an underlying programming model. The proposed algorithms prevent the untrusted cloud to know the database or the query while also preventing output size and access-pattern attacks. Interestingly, our algorithms do not need the database owner, which only creates and distributes secret-shares once, to be involved to answer any query, and hence, the database owner also cannot learn the query. We evaluate the efficiency of the algorithms on parameters: (i) the number of communication rounds (between a user and a cloud), (ii) the total amount of bit flow (between a user and a cloud), and (iii) the computational load at the user-side and the cloud-side.

Even though there are lots of invented systems that have implemented customer analytics, it’s still an upcoming and unexplored market that has greater potential for better advancements. Big data is one of the most raising technology trends that have the
capability for significantly changing the way business organizations use customer behaviour to analyze and transform it into valuable insights. Also decision trees can be used efficiently in the decision making analysis under uncertainty which provides a variety of essential results. Hence to the end of this paper, we propose the MapReduce implementation of well-known statistical classifier, C4.5 decision tree algorithm. The paper additionally mentions why C4.5 is preferred over ID3. Apart from this, our system aims to implement Customer data visualization using Data Driven Documents (d3.js) which allows you to build well customized graphics.

The computer industry is being challenged to develop methods and techniques for affordable data processing on large datasets at optimum response times. The technical challenges in dealing with the increasing demand to handle vast quantities of data is daunting and on the rise. One of the recent
processing models with a more efficient and intuitive solution to rapidly process large amount of data in parallel is called MapReduce. It is a framework defining a template approach of programming to perform large-scale data computation on clusters of machines in a cloud computing environment. MapReduce provides automatic parallelization and distribution of computation based on several processors. It hides
the complexity of writing parallel and distributed programming code. This paper provides a comprehensive systematic review and analysis of large-scale dataset processing and dataset handling challenges and requirements in a cloud computing environment by using the MapReduce framework and its open-source implementation Hadoop. We defined requirements for MapReduce systems to perform large-scale data processing. We also proposed the MapReduce framework and one implementation of this framework on Amazon Web Services. At the end of the paper, we presented an experimentation of running MapReduce system in a cloud environment. This paper outlines one of the best techniques to process large datasets is
MapReduce; it also can help developers to do parallel and distributed computation in a cloud environment.

Big Data is large-volume of data generated by public web, social media and different networks, business applications, scientific instruments, types of mobile devices and different sensor technology. Data mining involves knowledge discovery from these large data sets. Different types of clustering methods are used in data mining. For processing these large amounts of data in an inexpensive and efficient way, new architecture, techniques, algorithms and analytics are require to manage it and extract value and hidden knowledge from it. Hadoop is the core platform for storing the large volume of data into Hadoop Distributed File System (HDFS) and that data get processed by MapReduce model in parallel. Hadoop is designed to scale up from a single server to thousands of machines and with a very high degree of fault tolerance. This paper presents the survey of big data, issues with big data, clustering information and how Hadoop works.

A flexible, efficient and secure networking architecture is required in order to process big data. However, existing network architectures are mostly unable to handle big data. As big data pushes network resources to the limits it results in network congestion, poor performance, and detrimental user experiences. This paper presents the current state-of-the-art research challenges and possible solutions on big data networking theory. More specifically, we present the state of networking issues of big data related to capacity, management and data processing. We also present the architectures of MapReduce and Hadoop paradigm with research challenges, fabric networks and software defined networks (SDN) that are used to handle today's idly growing digital world and compare and contrast them to identify relevant problems and solutions.

Data Mining refers to the process of mining useful data over large datasets. The discovery of interesting association relationships among large amounts of business transactions is currently vital for making appropriate business decisions. This is the reason that the research in data mining is carried out largely for business decision making rather than for academic importance. Association rule analysis is the task of discovering association rules that occur frequently in a given transaction data set. Its task is to find certain relationships among a set of data (itemset) in the database. It has two measurements: Support and confidence values. Confidence value is a measure of rule’s strength, while support value corresponds to statistical significance. There are currently a variety of algorithms to discover association rules. Most of the algorithms need a specification of minimum support value as user input. Specifying minimum support values of items is not recommended as it leads to very less or very large rules. With a sufficiently high support value, the less frequent elements gets eliminated, leaving only the elements which are most frequent. Thus, knives and spoons may get eliminated leaving only biscuits and milk. One approach for this problem is proposed by MsApriori Algorithm. However, both Apriori and MsApriori are computationally complex and need large computational time for large datasets over traditional machines. One solution to this problem is proposed by Dynamic Matrix Apriori which is much faster as compared to traditional Apriori in the generation of candidate sets. The contribution of this paper is twofold. It first proposed a method to use MsAprioiri using Dynamix Matrix Technique. It then proposes a framework to use the Algorithm under the Map Reduce Programming model. Experiments on large set of data bases have been conducted to validate the proposed framework. The achieved results show that there is a remarkable improvement in the overall performance of the system in terms of run time, the number of generated rules, and number of frequent items used.

— In today’s age of information technology processing data is a very important issue. Nowadays even terabytes and petabytes of data is not sufficient for storing large chunks of database. The data is too big, moves too fast, or doesn’t fit the structures of the current database architectures. Big Data is typically large volume of un-structured and structured data that gets created from various organized and unorganized applications, activities such as emails web logs, Facebook, etc. The main difficulties with Big Data include capture, storage, search, sharing, analysis, and visualization. Hence companies today use concept called Hadoop in their applications. Even sufficiently large amount of data warehouses are unable to satisfy the needs of data storage. Hadoop is designed to store large amount of data sets reliably. It is an open source software which supports parallel and distributed data processing. Along with reliability and scalability features Hadoop also provide fault tolera...

Map Reduce has gained remarkable significance as a prominent parallel data processing tool in the research community, academia and industry with the spurt in volume of data that is to be analyzed. Map Reduce is used in different applications such as data mining, data analytics where massive data analysis is required, but still it is constantly being explored on different parameters such as performance and efficiency. This survey intends to explore large scale data processing using MapReduce and its various implementations to facilitate the database, researchers and other communities in developing the technical understanding of the MapReduce framework. In this survey, different MapReduce implementations are explored and their inherent features are compared on different parameters. It also addresses the open
issues and challenges raised on fully functional DBMS/Data Warehouse on MapReduce. The comparison of various Map Reduce implementations is done with the most popular implementation Hadoop and other similar implementations using other platforms.

Pretty much every part of life now results in the generation of data. Logs are documentation of events or records of system activities and are created automatically through IT systems. Log data analysis is a process of making sense of these records. Log data often grows quickly and the conventional database solutions run short for dealing with a large volume of log files. Hadoop, having a wide area of applications for Big Data analysis, provides a solution for this problem. In this study, Hadoop was installed on two virtual machines. Log files generated by a Python script were analyzed in order to evaluate the system activities. The aim was to validate the importance of Hadoop in meeting the challenge of dealing with Big Data. The performed experiments show that analyzing logs with Hadoop MapReduce makes the data processing and detection of malfunctions and defects faster and simpler.

Twitter, a micro-blogging service, has been generating a large amount of data every minute as it gives people chance to express their thoughts and feelings quickly and clearly about any topics. To obtain the desired information from these available big data, it requires high-performance parallel computing tools along with machine learning algorithms' support. Emerging big data processing frameworks (e.g. Hadoop) can handle such big data effectively. In this paper, we, firstly introduce a novel approach to automatically classify Twitter data obtained from British Geological Survey (BGS), collected using some specific keywords such as landslide, landslides, mudslide, landfall, land-slip, soil sliding, based on tweet post date and the countries where tweets are posted using MapReduce algorithm. We then propose a model to distinguish the tweets if they are landslides-related using Na¨ıveNa¨ıve-Bayes machine learning algorithm with n-Grams language model on Mahout. This paper also describes an algorithm for the pre-processing steps to make the semi-structured Twitter text data ready for classification. The proposed methods are useful for the BGS and other interested people to be able to see the name and number of the countries where the tweets are sent, the number of tweets sent from each country, the dates and time intervals of the tweets, and to classify the tweets whether they are related to landslides.

In distributed systems, databases migration is not an easy task. Companies will encounter challenges moving data including legacy data to the big data platform. This paper shows how to optimize migration from traditional databases to the big data platform, evaluate and estimate the cost and efficiency of migration tools.

Accurate recognition and differentiation of the human facial expressions require substantial computational power, where the efficiency of algorithm plays a vital role. Recent advancement in the human computer interaction and object recognition in terms of facial expressions and gesture demanded realistic facial animation models, smart algorithms for massive data handling, as well as sophisticated graphical user interfaces. A rapid escalation in the photo upload to the online social networks web sites such as Facebook and Twitter is evident, where huge dataset handling became the key issue. Competent search and manipulations within a large dataset for image reproductions posed a new challenge, where standard tools cannot achieve the desired target. Often, images possessing intricate multidimensional attributes involve robust computational techniques for pragmatic recognition. Thus, developing a novel robust facial recognition platform emerged as an urgent necessity. We introduce a new algorithm for facial image tagging and classification in cloud environments using the Hadoop and MapReduce based k-nearest neighbor algorithms. Experiments are performed on 3120 images from 120 individuals (65 male, 55 female) from the AR Face Database. The efficiency of the proposed algorithm is evaluated in terms of recognition rates and processing time. Significant improvements in the performance are demonstrated.

Profound attention to MapReduce framework
has been caught by many different areas. It is
presently a practical model for data-intensive applications
due to its simple interface of programming,
high scalability, and ability to withstand the subjection
to flaws. Also, it is capable of processing
a high proportion of data in distributed computing
environments (DCE). MapReduce, on numerous occasions,
has proved to be applicable to a wide range
of domains. However, despite the significance of the
techniques, applications, and mechanisms of MapReduce,
there is no comprehensive study at hand in
this regard. Thus, this paper not only analyzes the
MapReduce applications and implementations in general,
but it also provides a discussion of the differences
between varied implementations of MapReduce as
well as some guidelines for planning future research.

This document presents the handbook of BigDataBench (Version 3.1). BigDataBench is an open-source big data benchmark suite, publicly available from http://prof.ict.ac.cn/BigDataBench. After identifying diverse data models and representative big data workloads, BigDataBench proposes several benchmarks specifications to model five important application domains, including search engine, social networks, e-commerce, multimedia data analytics and bioinformatics. BigDataBench (partially) implements the same benchmarks specifications using variety of competitive techniques. The current version BigDataBench 3.1 includes 14 real data sets and the corresponding scalable big data generation tools, and 33 big data workloads. To allow flexible setting and replaying of mixed workloads, BigDataBench provides the multi-tenancy version; To save the benchmarking cost, BigDataBench reduces the full workloads to a subset according to workload characteristics from a specific perspective. It also provides both MARSSx86 and Simics simulator versions for architecture communities.

An effective technique to process and analyse large amounts of data is achieved through using the
MapReduce framework. It is a programming model which is used to rapidly process vast amount of data
in parallel and distributed mode operating on a large cluster of machines. Hadoop, an open-source
implementation, is an example of MapReduce for writing and running MapReduce applications. The
problem is to specify, which computing environment improves the performance of MapReduce to process
large amounts of data? A standalone and cloud computing implementation are used for the experiment to
evaluate whether the performance of running MapReduce system in cloud computing mode is better than
in stand-alone mode or not, with respect to the speed of processing, response time and cost efficiency.
This comparison uses different sizes of dataset to show the functionality of MapReduce to process large
datasets in both modes. The finding is, running a MapReduce program to process and analysis of large
datasets in a cloud computing environment is more efficient than running in a stand-alone mode.

In order to make accurate and fast keywords and full text searches it is recommended to index the words in the corpus. One way to do this is to use an inverted index to maintain in a structured form the words occurrence in a set of documents. A stemming algorithm can be used to minimize the number of indexed words, so only the root word is kept for each term. This paper presents how to build an inverted index for documents stored in MongoDB and Oracle databases. Different approaches are presented in order to compare and determine which one has the best performance. These approaches take advantage of the frameworks and tools provided by the database systems to build the index: the MapReduce framework for MongoDB and Pipelined Table Functions for Oracle.

Graphs are everywhere in our lives: social networks, the World Wide Web, biological networks, and many more. The size of real-world graphs are growing at unprecedented rate, spanning millions and billions of nodes and edges. What are the patterns and anomalies in such massive graphs? How to design scalable algorithms to find them? How can we make sense of very large graphs? And what kind of real-world problems can we solve with such tools? These are exactly the goals of this tutorial.

Apriori is designed to operate on databases containing transactions (for example, collections of items bought by customers, or details of a website frequentation). Other algorithms are designed for finding association rules in data having no transactions , or having no timestamps (DNA sequencing). Each transaction is seen as a set of items (an item set). Given a threshold , the Apriori algorithm identifies the item sets which are subsets of at least transactions in the database. Apriori uses a "bottom up" approach, where frequent subsets are extended one item at a time (a step known as candidate generation), and groups of candidates are tested against the data. The algorithm terminates when no further successful extensions are found.

Data has become an indispensable part of every economy, industry, organization, business
function and individual. Big Data is a term used to identify the datasets that whose size is
beyond the ability of typical database software tools to store, manage and analyze. The Big
Data introduce unique computational and statistical challenges, including scalability and
storage bottleneck, noise accumulation, spurious correlation and measurement errors. These
challenges are distinguished and require new computational and statistical paradigm. This
paper presents the literature review about the Big data Mining and the issues and challenges
with emphasis on the distinguished features of Big Data. It also discusses some methods to deal
with big data.

In recent years, big data are generated from a variety of sources, and there is an enormous demand for storing, managing, processing, and querying on big data. The MapReduce framework and its open source implementation Hadoop, has proven itself as the de facto solution for processing large amounts of data in parallel, and is intrinsically designed for batch processing and high throughput jobs. Although Hadoop has proven as de facto solution for batch jobs, there is growing demand for non-batch applications like: real-time queries, interactive jobs, and big data streams. Since Hadoop is not suitable for these non-batch jobs, new solutions are proposed to meet these new challenges. In this paper, we discuss the strength, features, and shortcomings of the standard MapReduce framework and its open source implementation Hadoop. In addition; we have discussed the significant extensions of MapReduce. Further, we have considered two categories of these solutions: real-time processing, and stream processing of big data. For each category, we have included paradigms and strengths.

Bigdata is a horizontally-scaled storage, opensource architecture for indexed data and computing fabric supporting optional transactions, very high concurrency and operates in both a single machine mode and a cluster mode. The bigdata architecture provides a high-performance platform for data-intensive distributed computing, indexing, and highlevel query on commodity clusters also run in a highperformance single-server mode. The big advantages of NewSQL databases are non-locking concurrency control mechanism, SQL, ACID support and provides shared-nothing architecture as well as capable of running on a large number of nodes without suffering bottlenecks. In recent trends New SQL comes into existence because of its better Performance and achieves scalability. In this research paper, we focuses on the Paradigm, Data model for Big Data, introduction to Batch layer, Serving layer, Speed layer, Data storage on the batch layer, Incremental batch processing. This paper also provides Lambda architecture in-depth, CouchDB architecture and defines how to handle big data through NewSQL Solutions (dbShards) as well as Future of NewSQL and Big Data processing.

The main aim of this paper is to reduce the burden of the single reducer per map. Now-a-day, MapReduce performance improvement is very significant for big data processing. In previous works, we tried to reduce the network traffic cost for MapReduce job by implementing a novel data partition and aggregation scheme and also reduce the burden of the reducer while data processing. And also we have several schemes to provide an efficient and traffic-aware MapReduce scheme. But in these schemes we used one mapper and one reducer to process big data, then the reducer facing the overload problem in the network. Hence, in this paper we are implementing multiple reducers for single map instead of single reducer. This proposed implementation can balance the load of the reducer by load balancing mechanism.

Distributed denial of service (DDoS) attacks continues to grow as a threat to organizations worldwide. From the first known attack in 1999 to the highly publicized Operation Ababil, the DDoS attacks have a history of flooding the victim network with an enormous number of packets, hence exhausting the resources and preventing the legitimate users to access them. After having standard DDoS defense mechanism, still attackers are able to launch an attack. These inadequate defense mechanisms need to be improved and integrated with other solutions. The purpose of this paper is to study the characteristics of DDoS attacks, various models involved in attacks and to provide a timeline of defense mechanism with their improvements to combat DDoS attacks. In addition to this, a novel scheme is proposed to detect DDoS attack efficiently by using MapReduce programming model.

Abstract: Apache Hadoop is a software framework that supports data-intensive distributed application under a free license. It enables applications to work with thousands of nodes and petabytes of data. Hadoop was inspired by Google’s MapReduce and Google File System (GFS) papers. Hadoop was originally developed to support distribution for the Nutch search engine project. Here Hadoop is incorporated for handling large cluster of images. Moreover, Hadoop provide language independent platform which helps the programmer to remain in their comfort zone. Here I discussed the cloudera installation of Hadoop and here I present the design, implementation and evaluation of Hadoop thumbnail creation model that supports incremental job expansion. Under this model, a job consumes input as required and can dynamically govern its resource consumption while producing the required results. I have implemented the mechanism in Hadoop, and I present results from an experimental performance study of normal and Hadoop model for creating thumbnails.
Keyword: Cloudera, SSH, MapReduce, HDFS.
Title: THUMBNAIL IMAGE CREATION IN HADOOP USING PYTHON
Author: Devi.V.Kumar
International Journal of Computer Science and Information Technology Research
ISSN 2348-120X (online), ISSN 2348-1196 (print)
Research Publish Journals

During the recreation of monetary competence, data is entirety and everything is data. However data is dependent upon the world that is chaotic, insane, unpredictable, and sentimental. The acceleration of so called big data and the evolution of tools and techniques with the intention of are able to enumerate our every move, desire and practice, have exposed where the conflict lies between the unstable actualities that we reside in and the need to seizure it in data. From the ancient anatomy of calligraphy to extant data centres, the human chase has always collected knowledge and facts. The advancement in tools and technology has led to the flood of data which desire more mature data ordnance system. This excessive growth of data makes great dilemma to human beings. Although there are much potential and extremely useful value, masked in the large amount of data. Big data is significantly helpful to increase the productivity and efficiency in business and revolutionary discovery in scientific disciplines and great opportunities in other fields also. While big data also derives many difficulties and challenges which is the other side of coin. This paper intended to reveal a close view about big data including its brief history, big data applications, big data opportunities and challenges, current tools and techniques to deal with big data problems. We also converse numerous underlying technologies related to big data such as cloud computing, internet of things and NoSQL databases.

The Apache Hadoop framework is an open source implementation of MapReduce for processing and storing big data. However, to get the best performance from this is a big challenge because of its large number configuration parameters. In this paper, the concept of critical issues of Hadoop system, big data and machine learning have been highlighted and an analysis of some machine learning techniques applied so far, for improving the Hadoop performance is presented. Then, a promising machine learning technique using deep learning algorithm is proposed for Hadoop system performance improvement. Keyword: Hadoop HDFS Machine learning MapReduce Parameter

PageRank evaluates the importance of Web pages with link relations. However, there is no direct method of evaluating the meaning of links in a hyperlink-based Web structure. This feature may cause problems in that pages containing many in-links are highly ranked without considering the meaning of the link relations among the pages. We therefore propose a novel ranking approach to directly analyze the meaning of links by transforming a hyperlink-based Web structure into a semantic-link-based Web structure. We extract semantic metadata from Web pages and construct a semantic-link-based Web structure using RDF model. We define a metric to evaluate the weight of the links for stratifying rank values based on their importance in the semantic-link-based Web structure. We implement the weighted semantic ranking algorithm in the MapReduce framework to consider large-scale semantic metadata. The results of our experiment show that our approach outperforms existing PageRank algorithms.

At present, the scale of data in many cloud applications increases tremendously in accordance with the Big Data trend, thereby making it a challenge for commonly used software tools to capture, manage and process such large-scale data within a tolerable elapsed time. In big data applications, data privacy is one of the most concerned issues because processing large-scale privacy-sensitive data sets often requires computation power provided by public cloud services. As a result is challenge for existing anonymization approaches to achieve privacy preservation on privacy-sensitive large-scale data sets due to their insufficiency of scalability. In this paper we propose a scalable Advanced Bottom up generalization approach for data anonymization based on Map Reduce on cloud. To make full use of the parallel capability of Map Reduce on cloud, specializations required in an anonymization process. Original datasets are split up into a group of smaller datasets, and these datasets are anonymized in parallel, producing intermediate results. Then, the intermediate results are merged into one, and further anonymized to achieve consistent k-anonymous data sets. A group of MapReduce jobs are deliberately designed and coordinated to perform specializations on data sets collaboratively.

MapReduce is a simple and powerful programming model which enables development of scalable parallel applications to process large amount of data which is scattered on a cluster of machines. The original implementations of Map Reduce framework had some limitations which have been faced by many research follow up work after its introduction. It is gaining a lot of attraction in both research and industrial community as it has the capacity of processing large data. In this review paper, we are going to discuss the map reduce framework used in different applications and for different purposes. This is a analysis done for implementing the architecture of Map Reduce from different research perspectives.

Pretty much every part of life now results in the generation of data. Logs are documentation of events or records of system activities and are created automatically through IT systems. Log data analysis is a process of making sense of these records. Log data often grows quickly and the conventional database solutions run short for dealing with a large volume of log files. Hadoop, having a wide area of applications for Big Data analysis, provides a solution for this problem. In this study, Hadoop was installed on two virtual machines. Log files generated by a Python script were analyzed in order to evaluate the system activities. The aim was to validate the importance of Hadoop in meeting the challenge of dealing with Big Data. The performed experiments show that analyzing logs with Hadoop MapReduce makes the data processing and detection of malfunctions and defects faster and simpler. Keywords— Hadoop, MapReduce, Big Data, log analysis, distributed file systems.

The Size of the data is increasing day by day with the using of social site. Big Data is a concept to manage and mine the large set of data. Today the concept of Big Data is widely used to mine the insight data of organization as well outside data. There are many techniques and technologies used in Big Data mining to extract the useful information from the distributed system. It is more powerful to extract the information compare with traditional data mining techniques. One of the most known technologies is Hadoop, used in Big Data mining. It takes many advantages over the traditional data mining technique but it has some issues like visualization technique, privacy etc.

Database storage storing abundant data usually accompanies slow performance of query and data manipulation. This thesis presents a model and methodology of faster data manipulation (insert/delete) of mass data rows stored in a big table. In this thesis, it depicts the solution to manipulate large data sets of one table which moves into and out of another logical table with outstanding efficiency compared with traditional transactional way. With this idea, the table structure needs to be redesigned to accommodate and keep data, in other words, the table needs to be “partitioned”. It also covers partitioning strategies which are applied to various scenarios such as the data sliding window scenario, data archiving, and partition consolidation and movement practice.

Big data is an assemblage of large and complex data that is difficult to process with the traditional DBMS tools. The scale, diversity, and complexity of this huge data demand new analytics techniques to extract useful and hidden value from it. Data must be prepared before starting mining as real data is sometimes not suitable for mining, and poor quality finishes in poor results. This paper presents the needs, various problems, and solutions for the preprocessing of big data.