Preliminary

 Before getting to know about the Google Cloud Platform, let us gain
knowledge on Cloud Computing.
 The following are five important characteristics that explain what Cloud
Computing is:
o First, you will get all your computing resources on-demand and self-
service. That is, you will get Processing power, Storage and Network
resources through the internet by using a simple user interface
o Second, you will have the facility and infrastructure to access the
above resources over the internet.
o Third, these resources are provisioned from a pool through a provider,
who will allocate all these resources to the customers.
o Fourth, the resources are elastic, which means you can scale your
resources up and down.
o Fifth, you will pay for the resources that you use or reserve, which is
known as the pay-for-use model. You will not be billed for the
resources that you do not use.

Google Cloud Platform

What are the services that the Google Cloud Platform provides to its
customers?

 Google Cloud Platform offers several services like Compute, Storage, Big
Data and Machine Learning, which are used to develop applications
like Web, Mobile, Analytics, or back-end solutions.
 Google Cloud offers reasonable billing to all the resources that it provides.
 The primary reasons to choose Google Cloud Platform is that it is Global,
Cost-effective, Open-source friendly and Secure.

Let us learn about the services which are provided by Google Cloud Platform
to its customers.

Google Compute Architecture

GCP offers different options for computing. Following are the options that you
can choose for your requirements:

1. You can work in a serverless environment

2. You can use a Managed application platform
3. You can use Container Technology
4. You can build your own cloud-based infrastructure for more control and
flexibility
GCP Region, Zones and Pricing
 GCP is organized into Regions and Zones.
 For example, if you want to launch a Virtual Machine (VM) in GCP using
Compute Engine (briefly described in later topics), it will run in a zone that you
specify. A collection of zones is known as Region.
 All the zones within a region will have faster connectivity among them. You
can also distribute your resources across different zones in a region.
 As of now, GCP provides 20 Regions and 61 Zones to its customers.
 Customers duplicate their application across multiple regions to prevent
application loss.
 Google provides Per Second Billing for a VM.
 It also provides billing for various services like Kubernetes Engine, which is
Container as a Service (CaaS), Cloud DataProc, which is a Big Data system
(Hadoop as a Service), and App Engine, which is Platform as a Service
(PaaS).

Security
The following are the solutions that Google Cloud provides to its customers to keep
their data safe:

 Google will design its own Server Boards and Network Equipment.
 Cryptographic signatures are used by Google server machines to check
that they are booting the correct software.
 Google will design and build its own data centers by introducing multiple
layers of physical security protections.
 Google Infrastructure provides the cryptographic privacy and integrity for
data-on-the-network.
 Google also has Encryption support in hard drives and SSDs.
 Google services that are made available through the internet should register
with an infrastructure service called Google Frontend, whose job is to check
for incoming network connections for correct certificates.
 Google also supports multi-layer, multi-tier denials of service protections.
 Phishing attacks against Google employees are guarded through U2F
Compatability security keys.

GCP Resource Hierarchy

 The gathering of Resources will make up the GCP Services. Resources can
be Compute Engine Virtual Machine, Cloud Pub/Sub, Cloud Storage
Bucket, App Engine and so on.
 These resources are organized into Projects. Projects can be combined
into Folders, and the folders can contain Sub Folders. All the folders and
projects are organized under Organization Node.
 Policies are defined in Projects, Folders and Organization nodes. Policies
are inherited downwards of Resource Hierarchy.
GCP Resource Hierarchy
 Projects are responsible for enabling GCP Services such as Enabling
Billing, Managing APIs and other Google Services.
 Projects can consist of different owners and users. The project will have two
identifiers like Project name (Mutable) and Project ID (Immutable). Project ID
cannot be changed, and it is permanent.
 Resources will inherit the policies from the parent. For example, if you apply
policy at the organization node, all the resources under that node will inherit
those policies.
 Another rule in Resource Hierarchy is if you implement a policy at a lower
level of the hierarchy, then you cannot override that by applying any other
policy at a higher level.
 Let us consider an example. Suppose you have to apply a policy like a user
called John, who can write to the Cloud Storage Bucket at a project level .
But, if you want a policy like John to read-only from the Storage
Bucket at Organization node, then it will not be applicable.

Interaction with GCP

You can communicate with the GCP through four ways as follows:

1. GCP Console
2. Google Cloud SDK and Cloud Shell
3. Mobile App
4. APIs

Prelude
If you have heard the term Cloud Computing, then you would be aware of Virtual
Machine (VM). Google Compute Engine (GCE) in GCP will let you run the VM in
Google Cloud Platform Infrastructure. You will configure your VM in such a way that
you will build a physical server by specifying its CPU power, memory, storage types
and OS.

In this topic, you are going to acquire knowledge on how GCE works with Google
Virtual Networking.

Virtual Private Cloud Network

 The first thing to do in GCP is to define a VPC network inside the GCP
Project. You can either create the new VPC network or use the default VPC
network.
 The job of VPC networks is to connect your GCP resources with each other
or with the internet.
  You can divide your VPC network into segments, apply firewall rules to them
(allow or restrict access to VMs), and create static routes to forward traffic to
specific destinations.
 The VPC network that you create in your project will have a Global scope.
 VPC network can consist of Subnets, which can be present in the zones that
make up a region.

Knowledge on Virtual Private Cloud

 You can have your resources in different zones that are on the same subnet.
 When you create your own Subnet, you can dynamically increase the size of
the subnet by expanding the IP Addresses.
 In the above example, you have a VPC network with a subnet, which consists
of two Compute Engine VMs.
 Even though both the VMs are in different zones, they are considered as
neighbors as they are in the same subnet.

More About Google Compute Engine

  The basic functionality of the Compute Engine is to create and run the Virtual
Machines on the Google Infrastructure.
 You can create your own Virtual Machine in GCP by using GCP
Console or gcloud command-line tool.
 Your VM is capable of running images of either Linux or Windows provided
by Google or you can create your own Custom Images or you can import
images from your Physical Servers.
 You can choose how much Memory and how many Virtual CPUs your VM
should have.
 Regarding Processing Power, if you have workloads like Machine
Learning and Data Processing, then you can consider GPUs for your VM.

More About Google Compute Engine

 Persistent Storage, that is made available to your VM is of two types,
namely, Standard and SSD. You can also add Local SSD for high
performance.
 The data storage in Local SSD is not safe because when the VM terminates,
data will get erased. Therefore, it is better to consider Persistent Disks for
the permanent storage of your data. The default disk is Standard Persistent
Disk
 If you want your VM to spin up with configurations like Installing software
packages on first boot, you can do it with Start-up scripts. There are also
other fields like Metadata, which gives unique information about the VM.
 You can take snapshots of your disk for the backups. Suppose if you want
to do a batch job like analyzing large data sets, then consider options
like Preemptible VM, which saves a lot of money.
 The maximum number of virtual CPUs in a VM is 96, and the maximum
memory size is 624 Gigabytes, which will be useful for heavy workloads
like In-memory databases and CPU intensive analytics.
 GCP also supports a feature called AutoScaling, which allows you to add or
remove VMs. You can also balance the incoming traffic among different VMs
through Load Balancing.

Introduction
Developers develop many applications that require the storage of large amounts of
data. Data can be in many forms like Media, Confidential data from devices,
Customer account balances, and so on.
Earlier, we read that data can be stored in Persistent disks. Also, GCP provides
other storage options for Structured or Unstructured data, Transactional, and
Relational data.
In this topic, you will be learning the various storage options like Cloud Storage,
Cloud Bigtable, Cloud SQL, Cloud Spanner, and Cloud Data storage .

More About Cloud Storage

  Google Cloud Storage is an Object Storage.
 Object Storage can be defined as a bunch of bytes that are given to you,
and you will address the object with a unique key.
 These Unique keys are in the form of URL so that it is best suitable to interact
with Web Technologies.
 Cloud storage is comprised of Buckets, which is used to hold your Storage
Objects.
 Storage Objects are immutable, such that, for every new change, you will
create a new version.
 When you create a bucket, you will give a Unique name to it, specify
a Geographic location to store the data, and also choose a Storage
Class by default.
 You can also control the access to the bucket in two ways, namely, Cloud
IAM and Access Control List (ACL).
 Object Versioning is another important concept in cloud storage. If you turn
this feature ON, you can store every version of the object. Otherwise, the
newer version will override the old one, and you cannot archive the old
version.

About Cloud Bigtable

The following are a few reasons to select Cloud Bigtable as your storage
option:

 Scalability: With the help of Bigtable, you can increase your machine count,
which does not require any downtime. Also, it handles all Administration
tasks like Upgrades and Restarts.
 The data present in the Cloud Bigtable is encrypted. You can use IAM
roles to specify who can access the data.
 From an application API perspective, data is written to and from Bigtable
through data service layers like Managed VMs, HBase Rest Server, or Java
Server using HBase Client.
 The above will serve the data to Applications, Dashboards and Data
services.
 Data can be read from and written to Bigtable through batch processes
like Hadoop MapReduce, DataFlow or Spark.

More About Cloud SQL

The following example will give you an idea on the benefits of using Cloud SQL.
Consider you have two accounts in a bank, and you want to transfer Rs.10,000 from
one account to the other.
Due to a glitch, the amount gets deducted from one account but is not credited to the
other. As using a relational databases requires lot of work to set up, maintain,
manage, and administer, but still you want to take advantage of them you can
use Cloud SQL.
  Cloud SQL offers MySQL and PostgreSQL databases as a managed
service, that are capable of handling terabytes of storage.
 It also provides several replica services like Read, failover and external
replicas.*
 Cloud SQL instances include firewalls, customer data is encrypted on
Google's internal networks, and schedule backups and it can also be
accessed by other GCP services.
 It can be scaled vertically by changing the machine type and horizontally
through Read replicas.
 If you need horizontal scalability more than what is provided by Cloud SQL,
then you can go for Cloud Spanner.

Google Compute Engine

 Before entering into this topic, you have learned about Google Compute
Engine which is a Infrastructure as a service offering and an App
Engine which is a Platform as a Service offering that you will get to know
about it in the upcoming topic.
 You are going to learn about Containers and Google Kubernetes Engine , a
hybrid solution that sits between the Compute Engine and App Engine.
 In IaaS, you are allowed to share resources by virtualizing the hardware by
using VMs. The developer will deploy their OS, access the hardware, and can
build their applications in a self-contained environment along with access
to Networking Interfaces, File Systems, RAM, and so on.

Google App Engine

 The smallest unit of compute that you will get is VM with some application.
 The Guest OS which comes along with VM may be large(Gigabytes in size),
will take several minutes to boot.
 You can install Web server, runtime, database, middleware. of your choice.
 You can also install underlying resources like Disk space, Disk I/O,
and Networking, and build it as you like.
 When the demand for your application increases, then you need to copy the
entire VM and boot the guest OS for each instance of the application which
can be slow and costly.
 By using the App Engine, you will gain access to the programming services.
The only job that you need to do is to write your code and self-contained
workloads, will use these services and includes any dependent libraries.
 As the demand for your application increases, the platform scales your
applications seamlessly and independently by workload and infrastructure.
 This scales rapidly, but you give up control of the underlying server
architecture.

Containers
  Containers are preferred due to the drawbacks in the Compute
Engine and App Engine.
 Containers are responsible for independent scalability of workloads and also
acts as an abstraction layer of OS and Hardware.
 All you need when you get an host with an OS is which
supports Containers and Container Runtime.
 Your code will be portable and you can treat the OS and hardware as Black
Box. Using this functionality, you can move between stages
like Development, Staging, and Production from on-premises to the cloud.
 For example, if you want to scale a web server a hundred times, then you can
do it in seconds on a single host depending on the size of the workload.

Kubernetes
 For example, if you want to build your application using a lot of containers
performing like microservices, which are connected through network
connections, then you can make them to scale independently, modular,
and easily deployable across a group of hosts.
 Containers can be scaled up or down, and start or stop by the hosts on-
demand as your application changes or else the host fails.
 The above process can be done by a tool called Kubernetes.
 The functionality of Kubernetes is to orchestrate several containers on hosts,
scale them as microservices, and rollouts, rollbacks.
 Now you will get to know about how you can build an run containers. An
open-source tool called Docker, which helps you to define a format for
bundling your application with machine-specific settings and dependencies
into a container.
 Google cloud platform also has a separate tool called Google Container
Builder a managed service which is used for building containers.

Playing with Docker

 Consider a simple Python app whose output is Hello World. When you hit its
endpoint, it will give you the version.
 Here, the question is, how will you get the app into Kubernetes.
 The versions of python and Flask are managed
through Requirements.txt file also with the other dependencies too.
 You can use Docker Build command to build the container.
 This builds the Container and stores it on the local system as a runnable
image.
 Docker run command is used to run the image.
 This image can also be saved and uploaded to Container Registry Service (
a place to manage all your docker images), from where it can be downloaded.

Kubernetes Functionality
 Kubernetes an open-source orchestrator for containers.
  Kubernetes is a set of APIs which can be used to deploy containers on a set
of nodes called Cluster.
 Cluster is divided into master and node components. The master will manage
the nodes in the cluster.
 In Kubernetes, you will consider Node as Virtual Machines running
in Google Compute Engine.
 When you define a set of applications, Kubernetes is responsible for how to
make them interact with each other.
 Kubernetes makes it easy to run containerized applications one yourself on
your own hardware but you have to maintain it in your own environment.
 In order to avoid the burden of maintaining it in your own environment, Google
Cloud provides Google Kubernetes Engine which handles kubernetes as a
managed service.

Google Kubernetes Engine

 If you want to start Kubernetes on a cluster in GKE, run the following
command in Cloud Shell.

gcloud container cluster create k1

 You can deploy Containers on the nodes using a wrapper called Pod.
 Pod is the smallest unit in Kubernetes that you can create or deploy. Think
about pod like a running process on your cluster.
 Pod can be a component of an application or it can be an entire application.
 A pod can contain single or multiple containers.
 You can package multiple containers into a single pod and they will
automatically share networking and can have disk storage volumes in
common.
 Each pod will get a separate and unique IP, set of ports for your containes.
 Containers in the pod will communicate with each other using localhost and
fixed ports.
 Kubectl command is used to run the container in the pod, which results in an
immediate deployment of a container in the running pod.

Google Kubernetes Engine

 Deployment is represented as a group of replicas of the same pod that helps
you to run the pods continuously, even though the nodes fail.
 kubectl get pods command is used to get the pods.
 Pods in the Deployment are only reachable inside the GKE Cluster. You
can make them available public by connecting it to load balancer by
running kubectl expose command
 If you attach an External Load Balancer with the public IP to the service,
then the outsiders can also access the cluster.
 In Google Kubernetes Engine, a load balancer is created as a Network
Load Balancer.
  Deployments will create and delete the pods, pods get their own IP
addresses and these addresses will not be stable over time.
 A Service groups a set of pods and provide Fixed IP for them.

Pods
 kubectl scale command can be used to scale the deployment. You can also
provide autoscaling with all kinds of parameters.
 A Configuration file can also be provided instead of giving commands to tell
Kubernetes of what your desired state to look like and kubernetes will figure it
out`.
 You can check the deployment to make sure that replicas are running by
using the command kubectl get deployments or kubectl describe
deployments.
 You can get the External IP of the service by running the command kubectl
get services.
 For example, there are 3 replicas of a nginx pod and you want to update it
to 5 replicas, then you can edit the deployment configuration file
from three to five.
 Run kubectl apply command to update the changes to configuration file and
run kubectl get replicasets command to see the replicas and their updated
state.
 kubectl get services command confirms that the external IP of the service is
unaffected. The service will send traffic to all the 5 pods.
 If you want to update the version of the application, but rolling out the change
at once is risky so you will use a feature called Rolling update.

Introduction
By now you are familiar with two crucial GCP products, Compute
Engine and Kubernetes Engine.

 One common feature in these two products is you will choose the
infrastructure in which your application will run, that is, Virtual
Machines for Compute Engine and Containers for Kubernetes Engine.
 If you want to focus on the application code, then your choice should be App
Engine.
 App Engine comes under Platform as a Service (Paas). App Engine
manages both Hardware and Network Infrastructure.
 App Engine has many built-in services like No SQL databases, in-memory
caching, load balancing, health checks, logging, and authenticating
users, which can be used by your application.
 App Engine will automatically scale applications like web
applications and mobile backend.
App Engine Standard
App Engine offers two main environments namely App Engine Standard and App
Engine Flexible.

 App Engine Standard environment offers a more straightforward deployment

than App Engine Flexible.
 It offers free usage quota for certain services, which means low utilization
applications will be able to run at a small charge.
 Google App Engine Software Development Kits are available in several
languages. You can test your application using these kits before deploying
your application in the real App Engine Service.
 Google will provide runtime for App Engine Standard Environment. These
runtimes are provided in Java, Python, Go, and PHP and consist of libraries
that support App Engine APIs.

App Engine Standard

 App Engine can apply restrictions to your code by running it in a software
construct known as Sandbox.
 Sandbox is independent of the hardware, Operating System, and Physical
location of the server. Due to this, the App Engine can manage and scale your
applications.
 App Engine Standard composes some constraints like you cannot write to
the local file system.
 If you are unable to install third-party software which is not suitable for you,
then you should choose App Engine Flexible.
 Using App Engine SDK, you can run a test version locally and deploy it.
 App Engine will provide server instances automatically and scales, and load
balances them.
 Your App Engine application can call different GCP services using Dedicated
APIs such as NoSQL data store to make data persistent, caching of that data
using Memcache, searching logging, user logging, and the ability to launch
actions not triggered by direct user requests, like task queues and a task
scheduler.

App Engine Flexible

 In case the App Engine Standard sandbox model constraints are not suitable
for you, but you want to use App Engine, then you can choose App Engine
Flexible environment.
 App Engine Flexible will let you decide the container in which your application
should run unlike Sandbox.
 It enables you to run Docker Containers with Compute Engine Virtual
Machines. These VM's are health checked , healed as necessary.
  You can also choose the geographical location in which they have to run.
 App Engine Flexible uses standard runtimes and also has access to services
such as Data Store, Memcached, Task Queues, and so on.

App Engine Standard vs. App Engine

Flexible
Kubernetes App Engine
App Engine Flexible
Engine Standard

Language Java, Python, Go,

Any Any
Support PHP

Service
Hybrid PaaS PaaS
Model

Primary Use- Container-based Web and Mobile Both web and mobile applications
Case workloads applications and container-based workloads

Application Programming Interface

You have heard about the word Application Programming Interface (API) in
this course several times. Let us discuss that.

 A Software Service implementation is complex and changeable. If you need to

use that service and other software pieces, then you need to know about their
internal working, which will be a mess.
 Developers will structure the software to present a clean and well-defined
interface which abstracts all the needless information, then developers will
document the interface. That's an API.
 You can make a change to API by adding or deprecating the feature, these
changes can be made as versions of APIs.
 The programs which want to use the APIs can mention the version of API in
their calls.

Google Cloud Endpoints and Apigee Edge

 Google Cloud Platform provides two types of API Management Tools.
 Consider you are developing a Software Service and one of GCP Backend.
You want to expose this API.
  Meanwhile, if you want it to be consumed by trusted developers, you should
monitor and log its use. You will also need a single logical way to know which
end-user is making a call. This is where Cloud Endpoints come into play.
 Cloud Endpoints will deploy proxy in front of the software service and provide
API Console to combine all the capabilities which are mentioned above.
 Apigee Edge is also helpful in developing and managing API proxies.
 Apigee Edge focuses on Rate limiting, Quotas, and Analytics.
 By using Apigee Edge, instead of removing the entire application at once, you
can remove the services one by one until the whole application is retired.

Development in the Cloud

People develop significant applications in Google Cloud. GCP provides extraordinary tools
for different phases like:

1. Development
2. Deployment
3. Monitoring

 Most of the developers will store and maintain their code in the Git Repository.
 You can also create Git Instance (which gives you the great control) or Git
Provider (which lessens your work).
 In addition to the above options, you can keep your code private and add IAM
permissions to protect it by using the Cloud Source Repository (CSR).




Cloud Source Repository and Cloud Functions

 Cloud Source Repository provides the Git version control for the applications that
also run on Compute Engine, App Engine and Kubernetes Engine.
 You can have any number of Private Git Repositories with CSR.
 Source Viewer in CSR helps you to browse and view the repository files through the
GCP Console.
 Event-driven actions can also be present in the applications. Let us consider an
example. Suppose you have an app in which the users need to upload images.
 For uploading images, every time you need to process the image in various ways such
as converting it to standard image format, thumbnail into various sizes and store it in
the repository.
 You can integrate the above function into the application, but the major drawback is
providing resources to it.
 Cloud Functions is used to write a single purpose function that will do image
manipulations and run automatically when a new image is uploaded.
 


Cloud Functions
 When you are using Cloud Functions, you will not bother about
the Servers or Runtime Binaries.
 For example, You will write Javascript code in Node.js environment provided by
GCP, and then configure when it should fire.
 You will only pay for the functions that run in 100 milliseconds interval.
 It can trigger on events in Cloud Pub/Sub, Cloud Storage and HTTP call.
 You can choose your interested events.
 For each event type you will tell cloud Functions on which event you are interested in,
and this type of declaration is called Triggers.
 You will attach JavaScript functions to the triggers, and the functions respond
whenever the event happens.
 The applications that have Microservices architecture can be implemented in Cloud
Functions.
 Cloud Functions can also be used to enhance the application without worrying about
scaling.




Deployment
Setting up an environment in GCP will take a lot of steps like

1. You can set up compute network, storage network, and their configurations.
2. If you want to change the environment, you can do it with some commands.
3. You can also clone the environment by executing some commands.

 The above steps will take a lot of time and you can reduce it by using a Template. (
Requires specification for the environment that you want it to be).
 GCP provides , Deployment Manager to automate the creation and management of
the above steps by using the template.
 You can create the Template file either by using YAML Markup
language or Python. Then the template will be consumed by the Deployment
Manager and performs the actions that need to be done.
 You can edit the template and tell the Deployment Manager to make changes
according to that.

Deployment Manager Templates can be stored and version controlled in Cloud Sources
Repository.
 


Monitoring
Without Monitoring your application you cannot run it stably.

 It will analyze and let you know whether the changes are functional or not. It will
respond with some information whenever your application is down.
 Stackdriver is the monitoring tool that you can use in GCP for Monitoring, Logging
and Diagnosing.
 Stackdriver will give you an entry to receive signals from Infrastructure Platform,
Virtual Machines, Middleware, Application tier, Logs, Metrics, and Traces.
 It also gives helps you to check about Application health, Performance and
Availability.
 With Stackdriver you can able to do Monitoring, Logging, Tracing, Error
Reporting and Debugging.
 You can configure uptime checks which are associated with URL's and Resources
such as Instances and Load Balancers.
 You can also set up alerts on Health check results or Uptime falls.




Stackdriver
 You can also use Monitoring tool with Notification tools. You can also view
your Application state by creating Dashboards in Stackdriver.
 Stackdriver Logging will allow you to view logs of your application.
 Logging also allow you to define metrics depending on the Log content that are
included in the Dashboards and Alerts.
 You can export logs to Cloud Pub/Sub, Cloud Storage and BigQuery.
 Stackdriver Error Reporting will track and group all the errors in the cloud
application.
 Stackdriver Trace is used to sample the latency of App Engine applications.
 Stackdriver Debugger it connects your production data to source code. It works
efficiently when your source code is available in Cloud Source Repository.



Google Big Data Platform
Google Big Data solutions will help to transform the users and their business
experiences through data insights which are known as Integrated Serverless
Platform.

 GCP Services are fully maintained and managed, and you only pay for the
resources that you consume.
 Let us discuss the data services that are integrated and help you to create
custom solutions.
 Apache Hadoop is an open-source framework for Hadoop which depends on
MapReduce Programming model.
 MapReduce model is nothing but a function called Map Function will run along
with a large dataset to generate intermediate results. By taking these
results(intermediate) as input Reduce Function will produce Final output.
 Along with Apache Hadoop, there are such related projects such as Apache
Pig, Hive and Spark.
 On Google Cloud Platform, Cloud DataProc can be used to run Hadoop,
Spark, Hive and Pig.

Cloud DataProc
 When you request a Hadoop Cluster, it will be built you in less than 90
seconds on top of your Virtual Machines. Scaling can be done up and down
based on the processing power.
 You can also monitor your cluster using Processing Power.
 Running the clusters in On-premises will require hardware investment.
 But running them in DataProc will allow you to pay only for hardware
resources that you will when you are creating the cluster.
 Cloud DataProc is billed by Second and when you complete it, you can delete
the cluster, and immediately GCP stops the billing.
 You can also use Preemptible instances for the batch processing to save
costs.
 Once your cluster contains the data, you can use Spark and SparkSQL for
data mining.
 You can also use Apache Spark Machine Learning Libraries to discover
patterns through the Machine Learning.

Cloud Dataflow
Cloud DataProc is suitable when you know your cluster size. But if your
cluster size is unpredictable or your data shows up in real-time, then your
choice should be Cloud Dataflow.
  Dataflow is a managed service which allows you to develop and execute a big
range of processing patterns by Extracting, Transforming and
Loading batch computation or continuous computation.
 Dataflow is used to build Data pipelines for both Batch and Streaming data.
 It is used to automate processing resources, frees you from operational tasks
like Performance optimization and Resource Management.
 DataProc can read data from BigQuery, Processes it, applies transforms
like Map operations and Reduce Operations and writes it to Cloud
Storage.
 Use cases like Fraud Detection and Financial Services, IoT Analytics,
Manufacturing, Logistics, HealthCare and so on.

BigQuery
Suppose you possess a large dataset and you need to perform ad-hoc SQL Queries
then BigQuery must be your choice.

 BigQuery is Google's fully managed, Petabyte, low-cost analytical data warehouse.

 You can get data into BigQuery from either Cloud Storage or Cloud Datastore and
can stream that to BigQuery up to 100,000 rows per second.* You can perform super-
fast SQL queries and read, write data to BigQuery through Cloud Dataflow, Spark
and Hadoop.
 BigQuery allows you to keep your data based on the location. You pay only for the
Queries that you are running.
 When the data in BigQuery reaches 90 days, Google will automatically decrease the
storage price.
 All companies from startups to small organizations are using BigQuery which has the
availability of 99.99%




Cloud Pub/Sub and Datalab

What if you are working with events in real-time and you need a messaging
service then Cloud Pub/Sub will help you in the following way.

 It is the simple, reliable and scalable foundation for stream analytics. By using
it, you can build independent applications to send and receive messages.
 The Pub in Pub/Sub is known as Publishers, and Sub is Subscribers.
  The application will publish their messages to pub/sub and subscribers who
are subscribed to them will receive messages.
 Cloud Pub/Sub can also integrate with Cloud Dataflow. You can also
configure the subscribers to accept messages through push and pull basis.
 Cloud Datalab helps you to explore the data, and it can also integrate with
multiple GCP Services like BigQuery, Cloud Storage and Compute Engine.
 Cloud Datalab runs on a compute engine and you can also mention the
specific region to run compute engine.

Google Cloud Machine Learning Platform

 Machine Learning is one of the branches of Artificial Intelligence
 In this coders will build systems that will improve its performance by
repeatedly exposing to some sample data (Training data).
 Youtube, Photos, Google Mobile app, and Google Translate are some of
the Google applications that are using Machine Learning.
 Google Machine Learning Platform is available as a cloud service.
 It provides you with machine learning services, pre-trained models to
generate your custom models.
 Tensor Flow is an open-source software library for developing machine
learning applications like Neural Networks.
 Tensor Flow will use Tensor Processing Units (hardware devices designed
to accelerate machine learning workloads with TensorFlow).
 You can build machine learning models of any size by using Google Cloud
Machine Learning Platform.

More on ML Platform
 If you want to add different machine learning capabilities to your applications,
then you can add it through Machine Learning API's which are provided
through Google Cloud.
 Cloud Machine Learning Platform can be used in different applications based
on the type of data whether it can be Structured or Unstructured.
 For the structured data, ML is used for Classification and Regression
tasks like Customer churn analysis, product diagnostics and
Forecasting.
 For Unstructured data , You can use ML for Image Analytics such
as Damage Shipment, Identifying Styles and Flagging Content.
 You can also perform Text analytics like Blog analysis, Language
Identification, Topic Classification.

Machine Learning API's

There are several Machine Learning API's that are provided to users by the
Google Cloud.
 Cloud Vision API helps its users to identify the content of an image and also
classifies the image into the predictable number of categories.
 Cloud Speech API helps to convert audio to the text and it can recognize
around 80 languages.
 Cloud Natural API will provide different natural language technologies to
developers around the world by doing Syntax analysis, identifying verbs,
nouns, adverbs, adjectives and can also find out the relationship among
words.
 Cloud Translation API will convert a simple arbitrary string to a supported
language through a simple interface.
 Cloud Video Intelligence API will help you to annotate videos in different
formats. You can use it to make your video content searchable.