11.

Explain the significance of Big Data and Hadoop in analytics and making informed
Decisions.
Big Data refers to the massive volume of structured and unstructured data that inundates
businesses on a day-to-day basis. It holds immense potential for insights and informed
decision-making. However, traditional data processing tools often struggle to handle such
large volumes efficiently.
Hadoop is an open-source framework designed to store, process, and analyze Big Data
across distributed computing clusters. By breaking down data into smaller chunks and
distributing them across multiple nodes, Hadoop enables parallel processing, making it
faster and more cost-effective to analyze massive datasets.
Together, Big Data and Hadoop revolutionize analytics by providing the infrastructure
needed to extract valuable insights from vast amounts of data. This enables businesses to
make data-driven decisions, uncover patterns, trends, and correlations that were previously
hidden, leading to better strategic planning, improved operational efficiency, and
competitive advantages.

12. Debrief about RDD and its utility in real life applications.
RDDs represent a fault-tolerant collection of elements that can be operated on in parallel
across a cluster of machines. The resilience aspect refers to RDDs' ability to recover from
node failures by recomputing lost partitions from lineage information.
In real-life applications, RDDs play a crucial role in various domains, including finance,
healthcare, e-commerce, and more. For instance, in finance, RDDs can be used to analyze
vast amounts of market data in real-time, enabling traders to make informed decisions
quickly. In healthcare, RDDs facilitate the analysis of patient data from disparate sources,
aiding medical professionals in diagnosing diseases and identifying treatment patterns. In e-
commerce, RDDs power recommendation engines by processing user interactions and
product data to deliver personalized recommendations, enhancing the overall shopping
experience.
The utility of RDDs lies in their ability to handle large-scale data processing tasks efficiently
and reliably. By distributing data across multiple nodes and performing computations in
parallel, RDDs enable high-performance analytics and machine learning at scale. This
capability is essential for organizations looking to derive actionable insights from massive
datasets and gain a competitive edge in today's data-driven world.

13. What is the Hadoop ecosystem, and what are its core components?
Hadoop is an open-source framework designed to store, process, and analyze Big Data
across distributed computing clusters. By breaking down data into smaller chunks and
distributing them across multiple nodes, Hadoop enables parallel processing, making it
faster and more cost-effective to analyze massive datasets.
The core components of the Hadoop ecosystem include:
Hadoop Distributed File System (HDFS), MapReduce, YARN, Apache Hive, Apache Pig,
Apache HBase, Apache Spark.

14. same ans as midterm question bank

15. What is SparkML, and what are its main components?

Spark ML is the machine learning library provided by Apache Spark, a fast
and general-purpose distributed computing framework. Spark ML is
designed to make scalable machine learning accessible and easy to use,
leveraging the distributed computing capabilities of Spark to process large
datasets efficiently. It provides a rich set of algorithms, tools, and utilities
for various machine learning tasks, including classification, regression,
clustering, collaborative filtering, dimensionality reduction, and more. Spark
provides a different set of machine learning tools to perform different tasks and take
different actions.
Machine Learning algorithms – It provides tools and techniques to
implement Regression, Classification, clustering, and collaborative
filtering. Collaborative filtering is a personalized recommendation strategy that
uses data from users to identify similarities between users and make
product recommendations. It works by looking for similarities
between users and items, such as items they've previously browsed
or purchasedSpark key components are Core, SQL, Streaming, MLlib, GraphX.

16. What is HBase, and how does it differ from traditional relational databases?
Apache HBase is an open-source, distributed, versioned, non-relational database modeled
after Google's Bigtable. provides low latency access to small amounts of data within large
data sets. HBase is designed for horizontal scalability, meaning it can handle very large
amounts of data by adding more machines to a cluster, while traditional databases are
typically vertically scalable, meaning they can handle more data by adding more resources to
a single machine. Unlike traditional relational databases, HBase does not support SQL
queries or transactions. Instead, it provides a schema-less data model and offers fast random
access to data, making it suitable for applications that require high throughput and low-
latency reads and writes.

17. What is Apache Hive, and what are its primary use cases?
Hive is a data warehousing system which exposes an SQL-like
language called HiveQL. A data warehouse provides a central store of information that can
easily be analyzed to make informed, data driven decisions. Hive allows users to read, write,
and manage petabytes of data using SQL. Primary use cases for Apache Hive include: Data
Warehousing, ETL (Extract, Transform, Load), Data Analysis and Reporting, Data Exploration.

18. What are the major challenges associated with Big Data processing and analysis?
Making big data accessible, Maintaining quality data, Keeping data secure, Finding the right
tools and platforms. Challenges of Big Data:
Storage, Processing, Security, Finding and Fixing Data Quality Issues, Scaling Big Data
Systems, Evaluating and Selecting Big Data Technologies, Big Data Environments, Real-Time
Insights.

19. What is the Hadoop ecosystem, and what are its core components?
the Hadoop ecosystem is a collection of open-source software tools that allow you to store
and process large amounts of data. The tools in this ecosystem include HDFS, YARN and
MapReduce. Core Components:
• Hadoop Distributed File System (HDFS): Distributes data across multiple nodes for high-
speed data access.
Fault-tolerant, ensuring data reliability even if a node fails.
• MapReduce: Programming model for processing and generating large datasets.
Splits tasks into smaller sub-tasks and processes them in parallel.

20. How does Spark and RDD differ from Hadoop and MapReduce in terms of data
processing?
Spark and Hadoop (specifically, the MapReduce framework) are both distributed computing
frameworks designed for processing large-scale data, but they differ significantly in terms of
their architecture, data processing models, and performance characteristics.

1. Architecture:
- Hadoop MapReduce follows a two-stage processing model where data is read from
Hadoop Distributed File System (HDFS), processed in parallel by mappers, shuffled and
sorted, and then aggregated by reducers.
- Spark, on the other hand, provides a more flexible and expressive processing model. It
introduces the concept of Resilient Distributed Datasets (RDDs), which are immutable
distributed collections of data that can be operated on in parallel across a cluster of
machines. Spark supports various programming languages (e.g., Scala, Java, Python) and
offers higher-level APIs like DataFrames and Datasets, which provide optimized processing
for structured data.

2. Data Processing Model:

- Hadoop MapReduce is optimized for batch processing of large datasets. It is well-suited
for ETL (Extract, Transform, Load) operations, batch analytics, and offline data processing.
- Spark offers a more versatile data processing model that supports batch processing,
iterative processing (e.g., machine learning algorithms), interactive data analysis, and real-
time stream processing. Spark's in-memory processing capabilities and DAG (Directed Acyclic
Graph) execution engine enable faster processing compared to Hadoop MapReduce,
especially for iterative algorithms and interactive workloads.

3. Performance:
- Spark generally outperforms Hadoop MapReduce in terms of performance due to its in-
memory processing and optimized execution engine. By caching intermediate data in
memory and minimizing disk I/O, Spark reduces the overhead associated with reading and
writing data to disk, resulting in faster processing speeds.
- Hadoop MapReduce processes data in a disk-centric manner, which can lead to higher
latency, especially for iterative algorithms and interactive queries that require frequent disk
reads and writes.

22. How does HBase handle high write throughput and scalability requirements?
HBase is designed to handle massive amounts of data by automatically partitioning tables
across multiple servers. This distributed architecture allows it to scale horizontally,
accommodating growing data volumes without sacrificing performance. In addition to its
distributed architecture and automatic partitioning, HBase optimizes storage with columnar
organization, ensures data durability with write-ahead logging, and enhances performance
with compaction. These features collectively enable HBase to handle massive data volumes
while maintaining high scalability and performance, making it ideal for real-time data
processing and analytics.

23. How does Hive simplify data querying and analysis for non-technical users?
Hive makes data querying and analysis easier for non-technical users by letting them use
SQL, a language they're already familiar with. They can write SQL queries to explore and
analyze large datasets stored in Hadoop without needing to know complex programming.
Hive handles the technical details behind the scenes, translating SQL queries into actions on
the data. This way, non-technical users can easily extract insights and make data-driven
decisions without the need for specialized skills.

24. What is MapReduce, and how does it enable parallel processing of large datasets?
MapReduce is a programming model and processing framework used for parallel and
distributed processing of large datasets across a cluster of computers. It breaks down data
processing tasks into two main phases: Map and Reduce. MapReduce makes handling big
data easier by splitting it into smaller pieces and processing them at the same time on many
computers. It makes big data processing faster by splitting tasks into smaller pieces that
computers can work on at the same time. First, the data is divided and sent to different
computers for processing. Each computer works on its piece independently. Then, the
results are combined to get the final answer. By distributing data processing tasks across
multiple nodes and executing them in parallel, MapReduce enables efficient and scalable
processing of large datasets, significantly reducing the time required to analyze and derive
insights from big data. This way, MapReduce speeds up the process of dealing with large
amounts of data by doing many tasks at once.