HDFS

With the tremendous amount of research done in the field of numerical methods for engineering, a sharp rise in the number of new algorithms and software tools (academic and commercial) have been observed in the past decades. The advent of such software tools has not only made the application of the methods easier, but it has also drastically increased the number of users and the applications of these numerical methods to engineering problems. However, mostly these tools have been developed independently for solving problems of a particular kind, by particular methods on a particular scale, which leads to some limitations. One of these limitations, interoperability, is discussed in this paper and a methodology to resolve it for a small use-case of finite element problems at a continuum scale has been proposed. The main idea revolves around the fact that, popular tools use formats like VTK, STL, or XDMF, among others, for transferring data of mesh, geometry, loads, etc., and not all the tools can understand or process these formats in a useful manner. To simply state it, there is not one standard format which can enable complete interoperability among these tools. In fact, this problem of interoperability also keeps us from transferring data from one scale to another, e.g. from an RVE to a continuum scale [1]. To enable the interoperability among tools, a methodology has been proposed to export data from Abaqus Output Database (ODB) [2] directly to HDF5 [3] containers as an example. In this paper, a brief overview of the HDF5 data model is given, followed by a discussion pointing out why this HDF5 data model is a suitable interoperability tool to be used in the Finite Element Analysis (FEA) of engineering problems. After a short introduction of the Abaqus ODB and its underlying hierarchical structure, the procedure to import external libraries to Abaqus-Python is discussed along with the scripting procedure in Python. The customization of HDF5 containers is done for thermo-mechanical problems using h5py, a Python API for HDF5 files. The metadata for the problem is collected from Abaqus v6.14, and a methodology for the export of data from Abaqus to HDF5 containers is defined. Finally, an example of an HDF5 data model created using our scripts and the future scope of the work is shown. The hierarchical structure of ODB and MDB (Model Database) are similar, and they serve the two main purposes in Abaqus software. The methodology we devise to extract data from ODB, can also be used to write data to a MDB, directly from HDF5 files after some alterations in the scripts. The ODB can be understood as a read-only mode, whereas MDB can be considered as read-write mode.

Hadoop and Spark are widely used distributed processing frameworks for large-scale data processing in an efficient and fault-tolerant manner on private or public clouds. These big-data processing systems are extensively used by many industries, e.g., Google, Facebook, and Amazon, for solving a large class of problems, e.g., search, clustering, log analysis, different types of join operations, matrix multiplication, pattern matching, and social network analysis. However, all these popular systems have a major drawback in terms of locally distributed computations, which prevent them in implementing geographically distributed data processing. The increasing amount of geographically distributed massive data is pushing industries and academia to rethink the current big-data processing systems. The novel frameworks, which will be beyond state-of-the-art architectures and technologies involved in the current system, are expected to process geographically distributed data at their locations without moving entire raw datasets to a single location. In this paper, we investigate and discuss challenges and requirements in designing geographically distributed data processing frameworks and protocols. We classify and study batch processing (MapReduce-based systems), stream processing (Spark-based systems), and SQL-style processing geo-distributed frameworks, models, and algorithms with their overhead issues.

In recent years, Hadoop framework is popularly
known for providing cost-effective solutions to
process large scale data intensive applications in a
distributed manner. Storage imbalance during replica
placement in Hadoop is harmful, because nodes receiving
too many replicas will get overloaded while other nodes
with fewer replicas will remain underutilized. Hence,
replica placement plays a major role in data availability
and balanced utilization of clusters in a cloud environment.
In this paper we propose a solution for loadaware
replica placement in Hadoop such that a cluster
is divided into small size partitions where at-least one
partition will be nearer to its users. Partitions are created
using minimum spanning tree and results are compared
with the default replica placement policy of Hadoop.
Experimental results of the proposed solution confirm
that, load-aware replica placement gives uniform rack
level utilization and reduces read access time over the default
replica placement policy of Hadoop in a multiuser
environment.

Log Analysis is a critical procedure in most framework and system exercises where log information is utilized for different reasons, for example, for execution checking, security examining or notwithstanding for revealing and profiling. Nonetheless, as years cruised by, the volume of log information increments alongside the span of the framework just as the quantity of clients included. Customary or existing log analyser instruments are not ready to deal with the huge measure of information. Thusly, Big Data is the answer for defeated this issue. The principle motivation behind this paper is to introduce a survey of log document investigation in Big Data condition dependent on past research works. This paper likewise features the qualities of Big Data just as Hadoop Framework that has been generally utilized as Big Data application. Results from the papers assessed demonstrate that dominant part analysts connected MapReduce as the principle segment of Hadoop for investigating the log records and HDFS as the information stockpiling. Past analysts have likewise utilized different instruments and calculations together with the Hadoop Framework for investigation purposes. The discoveries of this paper will give an intelligible audit of Hadoop use execution in breaking down various kinds of log records and prescribe justifiable outcomes for end clients to use in future work.

Sentiment Analysis is the process of using text analytics to mine various data sources for opinions. Often, sentiment analysis is done on the data that is got from the Internet and from various social media platforms. Because the content collected from the internet is unstructured, we need tools that can process and analyze this disparate data. Hence we make use of Big Data to handle the different sources and formats of the structured and unstructured data. In particular consumer reviews of a product are given in textual format, we first parse the reviews and classify them into positive and negative and then send these datasets to the Hadoop File System (HDFS) to analyze them. This helps the purchaser to have some knowledge about the product’s pros and cons and decide which product to buy.