GGC Installation

Global Cache
Installation Guide
Dell R430 / R440 / R730xd / R740xd / R740xd2

HPE Apollo 4200
November 2019
GGC Partners Only

Contents
1 Installation Process Overview
2 Hardware Installation
2.1 You Will Need
2.2 Procedure
3 Network Installation - Google Router

3.1 You Will Need
3.2 Procedure
4 Network Installation - ISP Managed Switch

4.1 You Will Need
4.2 Procedure
5 IP Addressing
5.1 IP Addressing Requirements for GGC Nodes
5.2 IP Addressing Requirements for Google Routers
5.3 Enabling IPv6
6 Software Installation (Network based)

6.1 Procedure
6.2 GGC software network-based installation failures
7 Software Installation (USB-drive)

7.1 You Will Need
7.2 Installation Procedure
7.3 GGC Software Installation Troubleshooting
7.4 Boot the Machine from the USB Stick
8 BGP Configuration - Google Router

8.1 You Will Need
8.2 Procedure
9 BGP Configuration - ISP Managed Switch

9.1 You Will Need
9.2 Procedure
10 Appendix A - Cabling Requirements - Google Router

10.1 Dell R730xd (Cisco NCS 5001)
10.2 Dell R740xd (Cisco NCS 5001, up to 12 machines)
10.3 Dell R740xd (Cisco NCS 5001, more than 12 machines)
10.4 Dell R740xd2 (Cisco NCS 5011)
10.5 HPE Apollo 4200 (Cisco NCS 5001, up to 8 machines)
10.6 HPE Apollo 4200 (Cisco NCS 5001, more than 8 machines)
10.7 HPE Apollo 4200 (Cisco NCS 5011)
11 Appendix B - Cabling Requirements - ISP Managed Switch

11.1 Dell R430
11.2 Dell R440
11.3 Dell R730xd
11.4 Dell R740xd 2x10G
12 Appendix C - Configuration Examples - ISP Managed Switch

12.1 Cisco Switch Interface Configuration (LACP Disabled)
12.2 Cisco Switch Interface Configuration (LACP Enabled)
12.3 Juniper Switch Interface Configuration (LACP Disabled)
12.4 Juniper Switch Interface Configuration (LACP Enabled)
12.5 Cisco BGP Configuration (Prefix List Based Route Filtering)
12.6 Cisco BGP Configuration (AS-PATH Based Route Filtering)
12.7 Juniper BGP Configuration (Prefix Based Policy)
12.8 Juniper BGP Configuration (AS-PATH Based Policy)
13 Appendix D - Physical Requirements
14 Appendix E - Power Requirements
15 Appendix F - Vendor Hardware Documentation

1
Installation Process Overview
This document describes the installation and configuration process for a Google Global Cache (GGC)
node. Follow these instructions if you’re an ISP deploying GGC in your network.
The installation process has these steps:
1. Hardware installation
2. Network installation
3. IP addressing
4. Software installation
5. BGP configuration
Google may provide a networking device that connects to GGC machines. In some cases it’s managed by
the ISP, in others by Google. In this Installation Guide, all references to Google router mean a networking
device both provided and managed by Google.
Steps 2 and 5 have two scenarios, depending on whether the GGC node is connected to a Google router
or an ISP managed switch.
Follow only the steps applicable to the GGC node type you’re installing.
2
Hardware Installation
Follow these instructions to rack mount GGC machines. Some details vary depending on the type of GGC
hardware provided. See the appendices for detailed information.
You may rack mount GGC machines as soon as you receive them.
2.1 You Will Need
Two people to lift the machines into position
Rack mount installation kit and vendor specific instructions (shipped with machines)
Phillips (crosshead) screwdriver
T-15 and T-25 Torx screwdrivers
Rack space, as listed in Appendix D - Physical Requirements
Power feeds and cables, as listed in Appendix E - Power Requirements
Network and power cabling, as listed in Appendix A - Cabling Requirements - Google Router and
Appendix B - Cabling Requirements - ISP Managed Switch
2.2 Procedure
1. HPE Apollo 4200 only: Remove the shipping screws on either side of the chassis, following vendor
instructions. If you don’t remove the shipping screws you won’t be able to open the chassis to
maintain the mid-chassis mounted disks in the future.
2. Install provided rails and mounting hardware in your rack.
3. Install the machines in the rack, following vendor instructions.

GGC equipment is heavy. Use two people to lift it and follow appropriate health and
safety procedures to prevent injury and to avoid damage to the GGC equipment.
NOTE: For Dell R740xd2 machines, to prevent possible injury, ensure that the thumbscrews located
on the front left and right control panels are fastened during racking, so that the machine doesn’t
slide out of the rack when you pull the front drive bay.
4. Connect power, but do not turn the system on yet.
5. Verify that both Power Supply Units (PSUs) show green indicator lights.
For redundancy and performance reasons, you must use both PSUs. Google
 strongly recommends that you connect each power supply to independent power
feeds. If a second feed isn’t available, you can connect both PSUs to the same feed.
3
Network Installation - Google Router
Follow the instructions in this section if Google is providing and managing a GGC router to which the
GGC machines are connected. Otherwise, proceed to Network Installation - ISP Managed Switch.
Connect the GGC machines and the router to your network as soon as you receive them. Do this even if
you aren’t ready to install the GGC software or start GGC traffic.
The GGC router comes preconfigured with uplink interface, LACP (for uplinks with 2+ physical ports) and
IP addressing as shown in the ISP portal (https://isp.google.com/assets/).
Network diagram of a GGC deployment with GGC router: single interconnect to ISP network (left) and two interconnects (right)
3.1 You Will Need
Equipment supplied by Google:
GGC router
Rack mount hardware
Machine-to-router connectivity components, as listed in Appendix A - Cabling Requirements -

Google Router, including some or all of these:
SFPs
Direct-attach QSFP cables
RJ45 ethernet cables
Uplink connectivity SFPs or QSFPs, as required for number of uplinks
Equipment supplied by ISP:
Machine-to-router connectivity components, specifically multimode fibers, as listed in Appendix A -

Cabling Requirements - Google Router
Uplink connectivity components, including SFPs or QSFPs, single or multimode fibers, as required
for number of uplinks
3.2 Procedure
3.2.1 Install and Connect the Router
Details of cabling and network interfaces vary, depending on the GGC hardware type, the number of
machines in the node, and the type and number of uplinks in use. See Appendix A - Cabling Requirements
- Google Router.
1. Install the router in the rack.
2. Install SFPs and cabling as described in Appendix A - Cabling Requirements - Google Router.
3. Power on the router.
3.2.2 Configure Your Network
For each uplink bundle, configure LACP on your device as follows:
Passive mode
Layer 2 mode
Load balanced on source/destination layer 3 information
If you’re using a single, non-bundled uplink, don’t enable LACP.
3.2.3 Verify Connectivity
Check uplink physical status (link lights) on GGC router, and on your device.
Verify GGC router light levels (Tx and Rx) at your device.
Verify GGC router LACP status at your device.

4
Network Installation - ISP Managed Switch
Follow the instructions in this section if you’ll be providing all network connectivity for the GGC
machines.
You may connect the GGC machines to your network as soon as you receive them, even if you aren’t
ready to install the GGC software or start GGC traffic.
4.1 You Will Need
Equipment supplied by Google:
SFPs for machine to switch or router links
Equipment supplied by ISP:
Switch or router
SFPs for uplinks
Cabling, as described in Appendix B - Cabling Requirements - ISP Managed Switch
You’ll also need:
Administrative access to your Ethernet switch
Switch port numbers facing the GGC machines

4.2 Procedure
Install SFPs and cabling, as described in Appendix B - Cabling Requirements - ISP Managed Switch.
Configure your switch with the following settings:
Maximum port speed (10Gbps for 10Gbps links, 1Gbps for 1Gbps links, etc.)
Full duplex
Auto-negotiation enabled
Flow control enabled
Spanning tree, either:

Disabled on GGC machine-facing ports or set to Portfast (Cisco IOS), Edge port (JUNOS), or
equivalent
All GGC machines in a single, dedicated layer 2 broadcast domain (VLAN)
For GGC machines using a single interface, Link Aggregation Control Protocol (LACP) should be
disabled.
For GGC machines using multiple interfaces, LACP should be enabled, and configured as follows:
Passive mode
Layer 2 mode
Load balanced on source / destination layer 3 information
Standalone mode (aggregated link should remain up, even if a physical port is down)
All interfaces for a machine connected to the same switch
Separate LACP bundle for each machine
You may connect different GGC machines in the same node to different switches, but it isn’t required. If
you use multiple switches, the VLAN used by the GGC machines must span all switches involved.
Sample switch configurations are provided in Appendix C - Configuration Examples - ISP Managed
Switch. Refer to your switch vendor’s documentation for specific configuration commands.
5
IP Addressing
GGC nodes require a dedicated layer 3 subnet. You can configure nodes as dual-stacked (preferred), IPv4
only, or IPv6 only.
For each IP protocol version supported by the GGC node, machines have the following addresses
assigned:
Static maintenance IP address
Multiple VIPs to serve traffic to users
BGP VIP (on one machine at any time)
Maintenance IPs are configured statically. VIPs are managed automatically to ensure they move to other
machines during failures and machine maintenances. This minimizes disruption of traffic to the users
and ensures the BGP sessions with GGC nodes don’t remain down for extended periods.
5.1 IP Addressing Requirements for GGC Nodes
GGC nodes require a specific size of allocated public IP subnet, as shown in the table below.
IP Subnet Size Requirements for GGC nodes

Number of machines in the node IPv4 subnet size IPv6 subnet size
2-6 /27 /64
6-8 /27 or /26 (recommended) /64
more than 8 /26 /64
5.1.1 IPv4 Addressing Scheme
IPv4 addresses are assigned within the GGC allocated subnet as follows:
You must use the 1st usable address in the subnet for the subnet gateway.
If required, assign the 2nd and 3rd addresses to an ISP-managed switch (e.g. for HSRP or GLBP).
The 4th and following IP addresses in the subnet are reserved for GGC machines (management
IPs and VIPs).
Check the node information in the ISP Portal (https://isp.google.com/assets/) for the BGP peering
IP address.
5.1.2 IPv6 Addressing Scheme
IPv6 addresses are assigned within the GGC allocated subnet as follows:
The ::1 address in the subnet is used as a statically addressed subnet gateway. If your device is
configured to send router advertisements, the GGC machines uses this in preference to the static
gateway.
If required, assign the ::2 and ::3 addresses to an ISP-managed switch (e.g. for HSRP or GLBP).
The ::4 and following IP addresses in the subnet are reserved for GGC machines (management IPs
and VIPs).
Check the node information in the ISP Portal (https://isp.google.com/assets/) for the BGP peering
IP address.
5.2 IP Addressing Requirements for Google Routers
GGC nodes that are behind Google routers have additional subnet requirements because of
interconnects needed to connect them to your network.
We support up to 4 interconnects per Google router.
Each interconnect may be configured with either IPv4, IPv6 or both address types.
Subnet sizes required for each interconnect type are as follows:
A /31 IPv4 subnet is required for interconnects with standard configuration.
A /29 IPv4 subnet (or larger) is required for interconnects where redundancy protocol (HSRP,
VRRP, etc.) is used at the ISP’s side.
A /127 IPv6 subnet is required for interconnects.
It’s up to the ISP to decide which IP from the interconnect allocated subnet is configured at either side of
the interconnect. Google doesn’t have any guidelines or preferences.
5.3 Enabling IPv6
5.3.1 New Node
You can enable IPv6 prior to installation by specifying the IPv6 subnet and IPv6 Router for BGP Sessions
when you supply the technical information required for node activation in the ISP Portal
(https://isp.google.com/assets/) asset pages.
Google strongly recommends you enable IPv6 for new nodes, even if you don’t yet have significant IPv6
user traffic, provided that IPv6 is globally reachable. Connectivity issues will delay turn up.
5.3.2 Existing Node
For an existing node that’s already serving IPv4 traffic, you can enable IPv6 through the ISP Portal
(https://isp.google.com/assets/) asset pages by following these steps:
1. In the ‘Configuration’ tab, select ‘Update Configuration’ at the bottom.
2. Select ‘Enabled’ for IPv6 and supply the IPv6 subnet.
3. Click ‘Update’ to save the changes.
You’ll receive an email when configuration changes are complete.
If you’re enabling IPv6 support for a GGC node that’s connected to a Google router, you’ll
 need to provide the IPv6 subnet for the interconnect that’s configured with a BGP session
over it. Contact [email protected] to verify these prerequisites are in place.
6
Software Installation (Network based)
Network based install is possible on GGC nodes that meet all of these requirements:
Google supplied GGC Router (Cisco NCS 5001/5011)

IPv4 enabled (Network-based IPv6-only installation is not supported)
Dell R740xd or HPE Apollo 4200 hardware
6.1 Procedure
6.1.1 HP Apollo 4200
1. Install the GGC router and provide connectivity to it.
2. Wire the machines to the GGC router, following the cabling requirements in Appendix A exactly.
3. Connect power to the machines.
4. Power on the machines by pressing the front power button.
5. No further action needed. All machines will attempt to boot of the network at regular intervals.
Once the GGC Router is fully provisioned and has established BGP session(s), all machines should
install over the network without intervention on your side.
6.1.2 Dell R740xd
1. Install the GGC router and provide connectivity to it.
2. Wire the machines to the GGC router, following the cabling requirements in Appendix A exactly.
3. Connect power to the machines.
4. Wait for the GGC router to complete turnup. You will receive an email message once this is done.
This message will be instructing you to boot the machines. Do this by pressing the front power
button. If the machines were already powered on, reboot them at this point.
5. Press F12 to initiate a network boot on all machines.
6. No further action needed. The machines should perform an automated network-based install.
6.2 GGC software network-based installation failures
Occasionally GGC network-based installation may fail to start. If this happens, please retry the procedure
above. If the failure persists, try the USB-drive installation process.
If the machine can’t establish network connectivity, check all cables are properly seated and connected
according to requirements in Appendix A.
If the machines still fail a network boot, try the Software Installation ( USB-drive) procedure below.
7
Software Installation (USB-drive)
This section describes the steps to install the GGC software on the machine. After this step is complete,
the installer automatically signals Google to begin the turnup process or to return this machine to a
serving state, in the case of a reinstallation.
The installation steps are:
1. Download the USB image.
2. Create USB boot sticks.
3. Boot each machine from a USB stick.
4. Enter the network configuration and wait for the installer to complete.
5. Reboot the machines.
It’s possible to install a GGC machine without Internet connectivity. In this case, the machine repeatedly
tries contacting GGC management systems until it succeeds.
7.1 You Will Need
A monitor and keyboard
IP information provided to Google in the ISP Portal (https://isp.google.com/assets/)
Labels to mark the IP address on each machine
A USB stick with at least 1 GB capacity. One USB stick per machine is provided.
Access to a computer with appropriate permissions to:

Download and save the install image file
Download, save, and run the tools required to create a bootable USB stick
GGC machines, mounted in a rack and powered up. A connection to the network with Internet
connectivity is preferred but not required.
You’ll also need to advertise the prefix allocated to the GGC node to your upstream networks.
7.2 Installation Procedure
7.2.1 Download the Install Image
Download a copy of the latest GGC setup image (https://dl.google.com/ggc/install/ggc-setup-

latest.img).
Only the latest version of the setup image is supported. If you use an older installer version we may ask
you to reinstall the machine.
7.2.2 Prepare the USB Stick
You’ll need a USB removable media device to store the GGC setup image. This can be the USB stick
shipped with GGC servers, or any other USB stick or portable USB drive with at least 1 GB capacity.
Create the USB boot stick on a computer on which you have the permissions described above. You can
create multiple boot sticks, to install machines in parallel, or you can make one USB stick and reuse it for
multiple machine installations.
For details on how to write the install image to the USB stick on various operating systems, see the ISP
Help Center (https://support.google.com/interconnect?p=usb).
7.2.3 GGC Software Installation
Repeat the installation steps on each machine in the GGC node.
1. Connect a monitor and a keyboard to the machine.
2. Insert the prepared setup USB stick in any USB port (front or back). If you use the front port, that
may prevent you from easily connecting a keyboard or from closing the rack doors.
3. Power off the machine, if it’s currently running.
4. Power on the machine by pressing the front power switch.
5. Allow some time for the machine to boot from the USB stick.

If you’ve installed this machine before, it might not automatically boot from the USB
stick. If that happens, follow the steps at Booting the machine from the USB stick.
The installation displays this screen:

Installer Start Screen
6. Press ENTER or wait for 10 seconds for the ‘Boot Menu’ to disappear. The machine boots up and
starts the installation program.
The installer examines the hardware. Some modifications applied by the installer may require the
machine to reboot.
If that happens, make sure the machine boots from the USB stick again. The installation program
then resumes.
7. The installer detects which network interface has a link. In this example, it’s the first 10GE
interface. It prompts you to configure it, as shown in this screen. Press ENTER to proceed:
NIC Detection
8. Respond to the prompts that appear on the screen. The configuration should match the
information provided to Google in the ISP Portal (https://isp.google.com/assets/):
NOTE: If you’ve used this USB stick to install any GGC machines before, the fields listed below are
pre-populated with previously used values. Verify they’re correct before pressing ENTER.
Enable LACP: Select ‘Y’ if the machine has multiple interfaces connected. Select ‘N’ if it has
only a single interface connected.
Enter GGC node subnet in CIDR format: x.x.x.x/nn or x:x:x:x::/64.
Enter the machine number: 1 for the first machine, 2 for the second, and so on.
Confirm the configuration when prompted.
IP Information
9. The installer validates IP information and connectivity, then begins software installation onto the
local hard drive. This step takes a couple of minutes. Allow it to finish.
10. When the installation process has completed successfully, it prompts you to press ENTER to
reboot the machine:
Successful installation
If any warnings or error messages are shown on the screen, don’t reboot the machine. See GGC
Software Installation Troubleshooting.
11. Remove the USB stick from the machine and press ENTER to reboot.
12. When the machine reboots after a successful installation, it boots from disk. The machine is now
ready for remote management. The monitor shows the Machine Health Screen:
Booted from disk
13. Label each machine with the name and IP address assigned to it.

If the installation didn’t go as described above, follow the steps in the next section, GGC
Software Installation Troubleshooting.
7.3 GGC Software Installation Troubleshooting
Occasionally GGC software installation may fail. This is usually due to either:
Hardware issues, which prevent the machine from booting, or that prevent the install image from
being written to disk
Network issues, which prevent the machine from connecting to GGC management systems
First, check machine hardware status by attaching a monitor, and viewing the Machine Health Screen
(https://support.google.com/interconnect/answer/9028258?hl=en&ref_topic=7658879). Further
information is available in the ISP Help Center (https://support.google.com/interconnect?p=usb).
If you can’t establish network connectivity, check the cables, switch and router configuration, and the IP
information you entered during installation. Check IP connectivity to the GGC machines from the GGC
switch, from elsewhere in your network, using external Looking Glass utilities.
If you’re installing a brand new machine for the first time and the setup process reports an error before
network connectivity is available, let [email protected] know what its service tag is when reporting the
problem.
If the setup process reports an error after network connectivity is established, it automatically uploads
logs to Google for investigation. If this happens, leave the machine running with the USB stick inserted
so we can gather additional diagnostics, if required.
Don’t place transparent proxies, NAT devices, or filters in the path of communications between the GGC
Node and the internet, subject to the explicit exceptions in the GGC Agreement. Any filtering of traffic to
or from GGC machines is likely to block remote management of machines. This delays turnup.
In other cases, contact the GGC Operations team: [email protected]. Always include the GGC node name
in communications with us.
NOTE: Photos of the fault, including those of the installer and Machine Health Screen, are useful for
troubleshooting.
7.4 Boot the Machine from the USB Stick
7.4.1 Dell Hardware
1. Insert the USB stick and reboot the machine. During POST (Power On Self Test phase of PC boot),
this menu appears on the screen:
Dell POST screen
2. Press F11 to enter the Boot Manager.
3. Select the ‘BIOS Boot Menu’.

Boot manager
4. From the list of bootable devices, select ‘Hard disk C:’ and then the USB stick, as shown in this
screen:
Boot from USB stick
7.4.2 HPE hardware
1. Insert the USB stick and reboot the machine. During POST (Power On Self Test phase of PC boot),
this menu appears on the screen:
HPE POST screen
2. Press F11 to enter the Boot Menu.
3. Select ‘Legacy BIOS One-Time Boot Menu’ and select ‘ENTER’.
Legacy One-Time Boot
4. Choose ‘One Time Boot to USB DriveKey’.

Boot from USB DriveKey
8
BGP Configuration - Google Router
In this scenario you establish BGP sessions with the Google-provided router. One session per each
interconnect and IP protocol version is required.
8.1 You Will Need
Administrative access to your BGP peer router, or route server
BGP configuration details from [email protected]
8.2 Procedure
You can configure your BGP router at any time. The session doesn’t come up until Google completes the
next steps of the installation. Establishing a BGP session with a Google router and advertising prefixes
doesn’t cause traffic to flow. We’ll contact you to arrange a date and time to start traffic.
We recommend at least one BGP session between your BGP peer(s) and the Google-provided router.
We’ll contact you if we have problems establishing the BGP session, or if we detect problems with the
prefixes advertised.
If you’re planning to perform work that may adversely affect GGC nodes, don’t disable
 BGP sessions. Instead, schedule maintenance for it in the ISP Portal

(https://isp.google.com/assets/).
9
BGP Configuration - ISP Managed Switch
In this scenario you establish a BGP session directly with the GGC node.
9.1 You Will Need
Administrative access to your BGP peer router, or route server
General BGP instructions from https://support.google.com/interconnect?p=bgp
BGP configuration details for this node from ISP Portal (https://isp.google.com/assets/)
9.2 Procedure
You can configure your BGP router at any time. The session doesn’t come up until Google completes the
next steps of the installation. Establishing a BGP session with a new GGC node and advertising prefixes
doesn’t cause traffic to flow. We’ll contact you to arrange a date and time to start traffic.
It’s important to understand that the BGP session established with GGC node isn’t used for traditional
routing purposes. This has two implications:
Only a single BGP session (for IPv4 and IPv6 each) with a GGC node is supported and necessary.
There’s no need to monitor the session or to notify us if it’s down.
We’ll contact you if we have problems establishing the BGP session, or if we detect problems with the
prefixes advertised.
Disabling the BGP session doesn’t stop the node from serving traffic. Our systems
continue to use the most recently received BGP feed when the session isn’t established.
 If you’re planning to perform work that may adversely affect GGC nodes, schedule
maintenance for it in the ISP Portal (https://isp.google.com/assets/).
10
Appendix A - Cabling Requirements - Google Router
10.1 Dell R730xd (Cisco NCS 5001)
Each machine requires:
2 x 10G SR SFPs: 1 for the machine, 1 for the router
1 x 1G copper SFP
1 x multi-mode fiber
1 x copper Cat5e cable
LACP disabled in GGC installer
You’ll also need:
Additional SFPs for uplink(s) to your network
Fiber of correct type for uplink(s)
Procedure:
1. Insert 10G SFPs into machine network interface “Port 1”, as shown below.
2. Insert SFPs into GGC router: - Machine-facing 10G SFPs, starting from interface #0 - Machine-
facing 1G SFPs, starting from interface #14 - Uplink SFPs, starting from interface #30 or uplink
QSFPs, starting from interface #1/0
3. Connect machines to switch with fiber, as shown: - First machine 10G interface “Port 1” to router
interface #0 - Second machine 10G interface “Port 1” to router interface #1, and so on.
R730 Cabling (Google)
4. Connect machines to switch with Cat5e cabling: - First machine RJ45 interface #3 to router
interface #14 - Second machine RJ45 interface #3 to router interface #15, and so on.
R730 Cabling (out-of-band)
5. Connect uplinks from GGC router to your router.
R730 Cabling (uplinks)
6. Verify uplink Tx and Rx light levels from your router.

10.2 Dell R740xd (Cisco NCS 5001, up to 12 machines)
4 x 10G SR SFPs: 2 for the machine, 2 for the switch
LACP enabled in GGC installer
You’ll also need:
Additional 10G SFPs or 100G QSFPs for uplink to your network
Fiber of correct type for uplinks
Procedure:
1. Insert uplink SFPs into GGC router: - If you’re using 10G SFPs, insert up to 16 10G SFPs into the
GGC router, starting from interface #24. - if you’re using 100G QSFPs, insert up to 2 100G QSFPs
into the GGC router, starting from interface #1/0.
R740 10G Uplinks (Google)
2. Insert machine-facing 10G SFPs into GGC router, starting from interface #0.
3. Insert 10G SFPs into machine network interfaces #1 and #2.
4. Connect machines to switch, as follows: - First machine interface #1 to router interface #0 - First
machine interface #2 to router interface #1 - Second machine interface #1 to router interface #2 -
Second machine interface #2 to router interface #3, and so on.
R740 2x10G Cabling

10.3 Dell R740xd (Cisco NCS 5001, more than 12 machines)
4 x 10G SR SFPs; 2 for the machine, 2 for the router
You’ll also need:
Additional 100G QSFPs for uplink to your network
Procedure:
1. Insert uplink SFPs into GGC router: Insert up to 2 100G QSFPs into GGC router, starting from
interface #1/0. 10G uplinks aren’t supported.
R740 100G Uplinks (Google)
2. Insert machine-facing 10G SFPs into GGC router, starting from interface #0.
3. Insert 10G SFPs into machine network interfaces #1 and #2.
4. Connect machines to switch, as follows: - First machine interface #1 to router interface #0 - First
machine interface #2 to router interface #1 - Second machine interface #1 to router interface #2 -
Second machine interface #2 to router interface #3, and so on.
R740 2x10G Cabling

10.4 Dell R740xd2 (Cisco NCS 5011)
Each machine requires 1 x 40G QSFP direct-attach cable.
You’ll also need:
Procedure:
1. Connect machines to switch with direct-attach cable, as shown: - First machine 40G interface (#1
in the diagram) to GGC router interface #0 - Second machine 40G interface (#1 in the diagram) to
GGC router interface #1, and so on.
R740xd2 Cabling
2. Insert 100G QSFPs used for uplinks into GGC router, starting from interface #24.
R740xd2 Uplinks

10.5 HPE Apollo 4200 (Cisco NCS 5001, up to 8 machines)
1 x 1G copper SFP (router end)
2 x multi-mode fibers
You’ll also need:
Additional SFPs or QSFPs for uplink to your network
Procedure:
facing 1G SFPs, starting from interface #16 - Uplink SFPs, starting from interface #24 or QSFPs,
starting from interface #1/0
2. Insert 10G SFPs into machine network interfaces #1 and #2, as shown.
3. Connect machines to switch with fiber, as shown: - First machine 10G interfaces #1, #2 to router
interface #0, #1 - Second machine 10G interfaces #1, #2 to router interfaces #2, #3, and so on.
HPE Apollo 4200 Cabling
4. Connect machines to switch with Cat5e cabling, as shown: - First machine RJ45 interface #1 to
router interface #16 - Second machine RJ45 interface #1 to router interface #17, and so on.
HPE Apollo 4200 Cabling (out-of-band)

10.6 HPE Apollo 4200 (Cisco NCS 5001, more than 8 machines)
1 x 1G copper SFP (router end)
2 x multi-mode fibers
You’ll also need:
Additional 100G QSFPs for uplink to your network (10G SFP uplinks aren’t supported)
Procedure:
facing 1G SFPs, starting from interface #24 - Uplink QSFPs, starting from interface #1/0
2. Insert 10G SFPs into machine network interfaces #1 and #2, as shown.
3. Connect machines to switch with fiber, as shown: - First machine 10G interfaces #1, #2 to router
interface #0, #1 - Second machine 10G interfaces #1, #2 to router interfaces #2, #3, and so on.
HPE Apollo 4200 Cabling
4. Connect machines to switch with Cat5e cabling: - First machine RJ45 interface #1 to router
interface #24 - Second machine RJ45 interface #1 to router interface #25, and so on.
HPE Apollo 4200 Cabling (out-of-band)
HPE Apollo 4200 Cabling (uplinks)

10.7 HPE Apollo 4200 (Cisco NCS 5011)
1 x QSFP direct-attach cable
You’ll also need:
Procedure:
1. Insert 100G QSFPs into GGC router, starting from interface #24.
2. Connect machines to switch with direct-attach cable, as shown: - First machine 40G interface (#2
in the diagram) to GGC router interface #0 - Second machine 40G interface (#2 in the diagram) to
GGC router interface, and so on.
HPE Apollo 4200 Cabling (40G)

11
Appendix B - Cabling Requirements - ISP Managed Switch
11.1 Dell R430
2 x copper Cat5e or Cat6 cables
Procedure:
Connect machine RJ45 network interfaces #1 and #2, as shown, to your switch.
R430 Cabling
11.2 Dell R440
2 x copper Cat5e or Cat6 cables
Procedure:
Connect machine RJ45 network interfaces #1 and #2, as shown, to your switch.
R440 Cabling
11.3 Dell R730xd
2 x 10G SFPs (SR or LR): 1 for the machine, 1 for for the switch
1 x multi-mode or single-mode fiber, to match SFP type
Procedure:
1. Insert SFPs into machine network interface #1, as shown.
2. Insert SFPs into your switch.
3. Connect machines to switch.
4. Verify Tx and Rx light levels from your switch.
R730 Cabling (ISP)

4 x 10G SFPs (SR or LR); 2 for the machine, 2 for the router
Procedure:
1. Insert two SFPs into machine network interfaces #1 and #2, as shown.
2. Insert SFPs into your switch.
5. On your switch add two ports facing each machine into separate LACP bundle.
R740 2x10G Cabling

2 x 10G SFPs (SR or LR): 1 for the machine, 1 for for the switch
Procedure:
1. Insert SFP into machine network interface #1, as shown.
2. Insert SFP into your switch.
R740 1x10G Cabling

12
Appendix C - Configuration Examples - ISP Managed Switch
These examples are for illustration only; your configuration may vary. Contact your switch vendor for
detailed configuration support for your specific equipment.
12.1 Cisco Switch Interface Configuration (LACP Disabled)
Replace interface descriptions mynode-abc101 with the name of the GGC node and the machine
number.
!
interface TenGigabitEthernet1/1
description mynode-abc101
switchport mode access
flowcontrol send off
spanning-tree portfast
!
!
!
!
end
12.2 Cisco Switch Interface Configuration (LACP Enabled)
Replace interface descriptions mynode-abc101-Gb1 with the name of the GGC node, the machine
number, and machine interface name.
!
interface GigabitEthernet1/1
description mynode-abc101-Gb1
channel-protocol lacp
channel-group 1 mode passive
!
!
interface Port-channel1
switchport
no port-channel standalone-disable
!
!
!
interface Port-channel2
switchport
no port-channel standalone-disable
end
12.3 Juniper Switch Interface Configuration (LACP Disabled)
Replace interface descriptions mynode-abc101 with the name of the GGC node and the machine
number.
set interfaces xe-0/0/1 description mynode-abc101-Xe1

set interfaces xe-0/0/1 unit 0 family ethernet-switching port-mode access
# The above may be "interface-mode" access
set interfaces xe-0/0/1 unit 0 family ethernet-switching vlan members vlan10
set interfaces xe-0/0/1 ether-options no-flow-control
set protocols stp interface xe-0/0/1 edge
12.4 Juniper Switch Interface Configuration (LACP Enabled)
Replace interface descriptions mynode-abc101-Xe1 with the name of the GGC node, the machine
number, and interface name on your switch.
set interfaces ge-0/0/1 description GGChost1-Gb1

set interfaces ge-0/0/1 ether-options 802.3ad lacp force-up
set interfaces ge-0/0/1 ether-options 802.3ad ae0
set interfaces ae0 description GGChost1
set interfaces ae0 aggregated-ether-options no-flow-control
set interfaces ae0 aggregated-ether-options lacp passive
set interfaces ae0 unit 0 family ethernet-switching port-mode access
set interfaces ae0 unit 0 family ethernet-switching vlan members vlan10
set interfaces ae1 description GGChost2
set interfaces ae1 aggregated-ether-options no-flow-control
set interfaces ae1 aggregated-ether-options lacp passive
set interfaces ae1 unit 1 family ethernet-switching port-mode access
set interfaces ae1 unit 1 family ethernet-switching vlan members vlan10
# Use "stp" or "rstp" depending on the spanning-tree protocol in use
set protocols stp interface ae0 edge
set protocols stp interface ae1 edge
12.5 Cisco BGP Configuration (Prefix List Based Route Filtering)
neighbor <IP address of GGC> remote-as 11344

neighbor <IP address of GGC> transport connection-mode passive
neighbor <IP address of GGC> prefix-list deny-any in
neighbor <IP address of GGC> prefix-list GGC-OUT out
ip prefix-list deny-any deny 0.0.0.0/0 le 32
ip prefix-list GGC-OUT permit <x.y.z/24>

ip prefix-list GGC-OUT permit <a.b.c/24>
12.6 Cisco BGP Configuration (AS-PATH Based Route Filtering)
neighbor <IP address of GGC> remote-as 11344

neighbor <IP address of GGC> transport connection-mode passive
neighbor <IP address of GGC> filter-list 1 in
neighbor <IP address of GGC> filter-list 2 out
ip as-path access-list 1 deny .*
ip as-path access-list 2 permit _100_

ip as-path access-list 2 permit _200$
ip as-path access-list 2 permit ^300$
12.7 Juniper BGP Configuration (Prefix Based Policy)
neighbor <IP address of GGC> {

description "GGC";
import no-routes;
export export-filter;
peer-as 11344;
passive;
}
policy-statement no-routes {
term default {
then reject;
}
}
12.8 Juniper BGP Configuration (AS-PATH Based Policy)
neighbor <IP address of GGC> {

description "GGC";
import no-routes;
export export-filter;
peer-as 11344;
passive;
}
policy-statement no-routes {
term default {
then reject;
}
}
policy-statement export-filter {
term allow-routes {
from {
from as-path-group GGC;
}
then accept;
}
}
as-path-group GGC {
as-path AS-PATH-NAME-1 "^100.*";
as-path AS-PATH-NAME-2 "^200.*";
}
13
Appendix D - Physical Requirements
Physical Dimensions
Hardware Height Width Depth Weight
Dell R430xd 1U 19" rack mount 607mm (24") 19.9kg (44lb)
Dell R440 1U 19" rack mount 693mm (27") 17.5kg (38.6lb)
Dell R740xd2 2U 19" rack mount 811mm (32") 22.0kg (48.5lb)
HPE Apollo 4200 2U 19" rack mount 813mm (32") 50kg (110.2lb)
Cisco NCS 5001 1U 19" rack mount 490mm (19") 9.3kg (20.5lb)
Cisco NCS 5011 1U 19" rack mount 572mm (22.5") 10kg (22.2lb)
Required operating temperature is 10°C to 35°C (50°F to 95°F), at up to 80% non-condensing humidity.
You might see exhaust temperatures of up to 50°C (122°F); this is normal.
See Appendix F - Vendor Hardware Documentation for further environmental and mechanical
specifications.
14
Appendix E - Power Requirements
Each machine requires dual power feeds. Machines are shipped with either 110V/240V PSUs, or 50V DC
power PSUs.
Google doesn’t provide all power cord types. If your facility requires other types of power cords and
they’re not listed in the table below, you’ll need to provide them.
Per-machine Power Requirements and Google-Provided Power Cords

Hardware Nominal Power1 Peak Power AC Power Cords DC Power Cords
Dell R430xd 250W 270W C13/C14 None
Dell R440 166W 307W2 C13/C14 None
Dell R730xd 350W 450W C13/C14 None
Dell R740xd 357W 780W3 C13/C14 None
Dell R740xd2 617W 1018W4 C13/C14 None
HPE Apollo 4200 372W 633W C13/C14 None
Cisco NCS 5001 200W 275W C13/C14 CAB-48DC-40A-8AWG
Cisco NCS 5011 228W 508W C13/C14 CAB-48DC-40A-8AWG
Google doesn’t provide DC power cords for machines; have your electrician connect machines to DC
power.
See Appendix F - Vendor Hardware Documentation for further electrical specifications.

15
Appendix F - Vendor Hardware Documentation
If you have further questions related to hardware delivered by Google and can’t find answers in this
document, consult the appropriate resource listed below.
Vendor Hardware Documentation

Hardware Link
Dell
PowerEdge General Resources (https://qrl.dell.com/Product/Category/1)
(all)
Dell
Owner’s Manual (http://topics-cdn.dell.com/pdf/poweredge-r430_owner's%20manual_en-us.pdf)
R430xd
Dell R440 Owner’s Manual (http://topics-cdn.dell.com/pdf/poweredge-r440_owner's%20manual_en-us.pdf)
Dell
Owner’s Manual (http://topics-cdn.dell.com/pdf/poweredge-r730_Owner's%20Manual_en-us.pdf)
R730xd
Dell
Owner’s Manual (http://topics-cdn.dell.com/pdf/poweredge-r740_Owner's%20Manual_en-us.pdf)
R740xd
Dell
Owner’s Manual (https://topics-cdn.dell.com/pdf/poweredge-r740xd2_owners-manual_en-us.pdf)
R740xd2
Owner’s Manual (http://h20566.www2.hpe.com/hpsc/doc/public/display?
HPE Apollo
sp4ts.oid=8261831&docLocale=en_US&docId=emr_na-c04718413)
4200
Quick Specs (https://www.hpe.com/h20195/v2/GetPDF.aspx/c04616497.pdf)
Cisco NCS Hardware Installation Guide (https://www.cisco.com/c/en/us/td/docs/iosxr/ncs5000/hardware-
5000 install/b-ncs5000-hardware-installation-guide/b-ncs5000-hardware-installation-
series guide_chapter_01.html#id_16803)
1. Nominal power during regular operation.↩
2. Dell estimated maximum potential instantaneous power draw of the product under maximum
usage conditions. Should only be used for sizing the circuit breaker. Peak power values for other
platforms were sourced in-house by running synthetic stress test.↩

GGC Installation

Uploaded by

Copyright:

Available Formats

GGC Installation

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

GGC Installation

Uploaded by

Copyright:

Available Formats

Global Cache

Dell R430 / R440 / R730xd / R740xd / R740xd2

GGC Partners Only

3 Network Installation - Google Router

4 Network Installation - ISP Managed Switch

6 Software Installation (Network based)

7 Software Installation (USB-drive)

8 BGP Configuration - Google Router

9 BGP Configuration - ISP Managed Switch

10 Appendix A - Cabling Requirements - Google Router

11 Appendix B - Cabling Requirements - ISP Managed Switch

12 Appendix C - Configuration Examples - ISP Managed Switch

13 Appendix D - Physical Requirements

14 Appendix E - Power Requirements

15 Appendix F - Vendor Hardware Documentation

The installation process has these steps:

2.1 You Will Need

Two people to lift the machines into position

Phillips (crosshead) screwdriver

T-15 and T-25 Torx screwdrivers

Rack space, as listed in Appendix D - Physical Requirements

Power feeds and cables, as listed in Appendix E - Power Requirements

2. Install provided rails and mounting hardware in your rack.

3. Install the machines in the rack, following vendor instructions.

4. Connect power, but do not turn the system on yet.

Equipment supplied by Google:

Rack mount hardware

Machine-to-router connectivity components, as listed in Appendix A - Cabling Requirements -

Direct-attach QSFP cables

RJ45 ethernet cables

Uplink connectivity SFPs or QSFPs, as required for number of uplinks

Equipment supplied by ISP:

Machine-to-router connectivity components, specifically multimode fibers, as listed in Appendix A -

3.2.1 Install and Connect the Router

1. Install the router in the rack.

3. Power on the router.

3.2.2 Configure Your Network

For each uplink bundle, configure LACP on your device as follows:

Load balanced on source/destination layer 3 information

If you’re using a single, non-bundled uplink, don’t enable LACP.

3.2.3 Verify Connectivity

Verify GGC router LACP status at your device.

4.1 You Will Need

Equipment supplied by Google:

SFPs for machine to switch or router links

Equipment supplied by ISP:

SFPs for uplinks

Cabling, as described in Appendix B - Cabling Requirements - ISP Managed Switch

You’ll also need:

Administrative access to your Ethernet switch

Switch port numbers facing the GGC machines

Configure your switch with the following settings:

Flow control enabled

Spanning tree, either:

All GGC machines in a single, dedicated layer 2 broadcast domain (VLAN)

Load balanced on source / destination layer 3 information

All interfaces for a machine connected to the same switch

Separate LACP bundle for each machine