FACEBO OK NETWORK

APPLIANCE DEPLOYMENT
US B Ba s ed S o f twa re I nst a lla t io n Guide

Feb rua ry 20 22
Copyrights and Trademarks

© 2022 Meta. All rights reserved.

Facebook is a trademark of Meta in the U.S. and other countries.
Linux® is a registered trademark of Linus Torvalds in the United States.
Mac® and macOS® are registered trademarks of Apple Inc.
Windows® is either a registered trademark or a trademark of Microsoft Corporation in
the United States and/or other countries.

FNA: USB Based Software Installation Guide 1

Table of Contents

• CHAPTER 1: INTRODUCTION 3
• CHAPTER 2: USB BOOT DISK CREATION 4
• CHAPTER 3: IP ADDRESS CONFIGURATION 8
• 3.1 BASIC CONFIGURATION 8
• 3.2 MANUAL CONFIGURATION 9
• 3.3 NEXT STEPS 11
• CHAPTER 4: CONFIGURE THE FNA CLUSTER SWITCH 12
• 4.1 SWITCH CONFIGURATION 12
• 4.2 NEXT STEP 13
• CHAPTER 5: RE-IMAGE THE SERVERS TO BSD 14
• CHAPTER 6: TROUBLESHOOTING 16
• APPENDIX A: LEGACY SWITCH CONFIGURATION 17
• 1.1A CISCO NEXUS 3123 SWITCH CONFIGURATION 17
• 1.2A NEXT STEP 19
• APPENDIX B: HPE GEN10PLUS MANUAL REMEDIATIONS 20
• 1.1B Broadcom NIC speed setting 20
20

FNA: USB Based Software Installation Guide 2

Ch a pter 1:
I n trod uc ti on

Thank you for choosing to install the Facebook Network

Appliance (FNA)!

Facebook Network Appliance is Facebook’s content caching

program. FNA provides Internet Service Providers (ISPs)
with an efficient means of delivering static Facebook content
from within their existing network. After deploying an FNA
kit, an ISP will offload a significant amount of Facebook
content from its backbone network and vastly improve the
Facebook user experience.

If the FNA kit contents (switch and servers) were not

configured with the IP configuration specified in the order
form, the IP reconfiguration procedures described in this
document need to be performed. Also, at any time, network
changes can necessitate reconfiguration.

NOTE
For purposes of following these procedures, Facebook
assumes you have already completed the hardware
installation of the supplied kit, including racking and
connection of the servers and switch.

FNA: USB Based Software Installation Guide 3

Cha p ter 2 : U S B Bo o t D i sk C r eation

The USB boot disk creation process can be performed using a Windows,
macOS, or Linux computer and a blank USB flash drive.

1. Select the caching tab from the left navigation bar (Caching Screen)

2. Choose the cluster from the “FNA Status” list (Caching Screen)

3. Expand the “Hardware Troubleshooting” widget (Cluster View)

4. Download the files needed for boot disk creation (Hardware Troubleshooting)

• Disk Image - USB boot disk image

• Switch Firmware - Configuration files

• Hosts Config - Configuration files

FNA: USB Based Software Installation Guide 4

Cha p ter 2 : U S B Bo o t D i sk C r eation

5. Connect a blank USB flash drive to your computer.

Warning: Make sure that no important data exists on the USB flash drive before continuing.
These procedures will completely overwrite any current contents of the flash drive.

6. Create the USB boot disk on the flash drive by following the instructions for the appropriate
operating system:

Table 1: Instructions for Windows Operating Systems

Table 2: Instructions for Mac Operating System

Table 3: Instructions for Linux Operating Systems

Table 1: Instructions for Windows Operating Systems

Step Instruction Illustration

a. Download and install Win32 Disk

Imager and 7-Zip.

b. Use 7-Zip to extract the disk

image.

c. Open Win32 Disk Imager. Navigate

to the image in Image File and
select the blank USB flash drive
in Device. Make sure the correct
device is selected, then click Write
(Warning: This action will overwrite
any disk specified. Ensure you
have selected the correct USB
disk before continuing).

Go to step 7 on page 7.

FNA: USB Based Software Installation Guide 5

Cha p ter 2 : U S B Bo o t D i sk C r eation

Table 2: Instructions for macOS

Step Instruction Detail

a. Run diskutil list to find the mount diskutil list

point of the USB drive. /dev/disk0 (internal, physical):
/dev/disk1 (internal, virtual):
/dev/disk2 (external, physical):
#: TYPE NAME SIZE IDENTIFIER
0: FDisk_partition_scheme *15.5 GB disk2
1: DOS_FAT_32 UNTITLED 15.5 GB disk2s1
Unlocked Encrypted

b. Confirm you have selected the

correct mounting point by running
diskutil info <disk#> and verifying
the drive name, size, and protocol.

c. Run diskutil unmountDisk <disk#> diskutil unmountDisk /dev/disk2

to unmount the USB drive.
Unmount of all volumes on disk2 was
successful

d. Run gunzip to decompress the gunzip –d ~/Downloads/caelus.img.gz

disk image.

e. Run dd to write the disk image sudo dd if=~/Downloads/caelus.img of=/

to the USB flash drive (Warning: dev/rdisk2
This action will overwrite any bs=1m && sync
disk specified. Ensure you have
selected the correct USB disk Example:
before continuing). Note: to sudo dd if=./caelus.img of=/dev/rdisk2
increase the speed of imaging bs=1m
process in macOS, add an “r”
before the disk’s name (in our 256+1 records in
example the disk was “disk2”, but 256+1 records out
we are using “rdisk2” as an output
device). 268435456 bytes transferred in 24.221118
secs
Go to step 7 on page 7.
(11082703 bytes/sec)

FNA: USB Based Software Installation Guide 6

Cha p ter 2 : U S B Bo o t D i sk C r eation

Table 3: Instructions for Linux Operating Systems

Step Instruction Detail

a. Run mount or blkid to find the

mount point of the USB drive.

b. Run umount to unmount the USB

drive.

c. Run gunzip to unzip the disk gunzip –d ~/Downloads/caelus.img.gz

image.

d. Run dd to write the disk image In this example USB flash has been mounted
to the USB flash drive (Warning: as /dev/sdb.
This action will overwrite any sudo dd if=~/Downloads/caelus.img of=/
disk specified. Ensure you have dev/sdb bs=1M
selected the correct USB disk
before continuing). && sync
Example:
Go to step 7 below.
sudo dd if=./caelus.img of=/dev/sdb bs=1M
256+1 records in
256+1 records out
268435456 bytes transferred in 24.221118
secs
(11082703 bytes/sec)

7. After creating the USB boot disk, make sure it is connected it to the computer where the
configuration files (step 3) were saved.

8. Verify that you see the CONFIG partition of the USB drive as a normal USB partition.

9. Move the saved Switch Config and Hosts Config archive files onto the CONFIG partition
of the USB drive.

FNA: USB Based Software Installation Guide 7

Cha p ter 3 : I P Ad d r ess C o n f igur ation

This section describes how to configure the IP settings of the servers in the
FNA cluster. Most servers can be configured using the Basic Configuration
steps. If needed, Manual Configuration steps are provided.

3.1 Basic Configuration

1. Connect the flash drive containing the USB boot disk (see Chapter 2: USB
Boot Disk Creation) to the FNA server (see Figure 2). If this is the first server
being reconfigured, connect it to the top-most server fna001 (see Figure 3).

Note: You need to perform configuration for all servers in the FNA cluster.
Figure 2: USB Port Locations on FNA Servers
HP DL 380 Gen9 - Rear Panel

HP DL 380 Gen10 - Rear Panel

HP DL 380 Gen10plus - Rear Panel

Figure 3: High-Level FNA System Overview

2. Connect a VGA monitor to the server’s VGA port and a keyboard to a USB port.

Note: When the operating system has loaded, you will automatically be logged in as root.

3. Reboot the server while the USB flash drive is connected. If the server does not automatically
boot from the USB flash drive, access the BIOS settings and set the USB drive as the first boot
option, then reboot.

4. Verify the settings displayed on the VGA monitor:

a. Listing is correct. If the listing is correct (see Figure 4), go to 3.3 Next Steps. If the listing is
incorrect, go to 3.2 Manual Configuration

FNA: USB Based Software Installation Guide 8

Cha p ter 3 : I P Ad d r ess C o n f igur ation

Figure 4: Configuration Confirmation

b. Configuration file was not found. If you receive a message that the configuration file was not
found (see Figure 5), repeat the steps in creating the CONFIG partition on the USB drive (see
steps 7 and 8 in Chapter 2: USB Boot Disk Creation). Restart the server with the USB flash
drive connected. If the listing is still incorrect, use the steps in 3.2 Manual Configuration.
Figure 5: Server Configuration File Not Found

3.2 Basic Configuration

1. Begin manual configuration by following either step a or b:

a. Type ./config_ipaddress.sh

b. Type ./c and press [TAB]

2. Enter the new IPv4 address as prompted, for example, 192.168.1.1 (see Figure 6) select
< OK > and press [ENTER]. (This information is available from the CONFIG partition of the
USB flash drive. The CONFIG partition contains n server configuration files (where n is the
number of servers in the FNA cluster - each filename is the server’s serial number).

Figure 6: IPv4 Address Configuration Prompt

3. Enter the new IPv4 netmask as prompted (see Figure 7) select < OK > and press [ENTER]. The
netmask needs to be in dotted quad format. Most commonly used values are:
/24 – 255.255.255.0
/25 – 255.255.255.128
/26 – 255.255.255.192
/27 – 255.255.255.224

FNA: USB Based Software Installation Guide 9

Cha p ter 3 : I P Ad d r ess C o n f igur ation

Figure 7: IPv4 Netmask Configuration Prompt

4. Configure the IPv4 address for the default gateway as prompted, for example, 10.10.10.100 (see
Figure 8) select < OK > and press [ENTER].

Figure 8: IPv4 Default Route Configuration Prompt

5. If you are prompted to confirm that the configuration is correct (see Figure 9) and the address
is acceptable, select < OK > and press [ENTER]. You then receive the configuration confirmation
(see Figure 4) and can go to 3.3 Next Steps. If the address needs to change, select < No > and
press [ENTER], then repeat the steps in 3.2 Manual Configuration.

Figure 9: Confirm Configuration

6. If you are prompted that the address is already configured (see Figure 10) it may be due to
configuring the address through a different method, such as DHCP. If the address is acceptable,
select < Yes > and press [ENTER]. You then receive the configuration confirmation (see Figure 4)
and can go to 3.3 Next Steps. If the address needs to change, select < No > and press [ENTER],
then repeat the steps in 3.2 Manual Configuration.

Figure 10: Address Already Configured

FNA: USB Based Software Installation Guide 10

Cha p ter 3 : I P Ad d r ess C o n f igur ation

3.3 Next Steps

1. Repeat the basic or manual configuration steps (see 3.1 Basic Configuration or 3.2 Manual
Configuration) for all cluster servers.

2. When all servers are configured, configure the switch as described in Chapter 4: Configure the
FNA Cluster Switch.

FNA: USB Based Software Installation Guide 11

Cha p ter 4 : C o n f i g u r e t h e FN A C lus ter Switc h

This section describes how to configure the FNA cluster switch.

Note: For Cisco Nexus 3123 devices, please see Appendix A

4.1 Configuration Instructions for Arista 7060

1) Prepare a USB flash drive formatted with FAT32 filesystem

2) Download the Switch Firmware and Switch Recovery Bundle.

a) Log in to Facebook’s Network Partner Portal (NPP) with your credentials.

Figure 11: Caching Screen

b) On the left hand side click the “Caching” heading, then in the “Cluster” column click the
appropriate cluster name
Figure 12: Cluster View Screen

c) Expand the “Hardware Troubleshooting” section by clicking the icon on the right hand side

FNA: USB Based Software Installation Guide 12

Cha p ter 4 : C o n f i g u r e t h e FN A C lus ter Switc h

Figure 13: Hardware Troubleshooting

d) Click the “Switch Firmware” and “Switch Recovery Bundle” to download the needed files.
Both files are zip files

3) Attach the USB flash drive to your computer. Unzip the “Switch Firmware” zip file and copy the
contents to the USB drive in the top most directory/folder. Unzip the “Switch Recovery Bundle”
zip file and copy the files to the USB drive, again to the top most folder/directory. Safely detach
the USB drive.

Figure 14: Arista 7060

4) Attach the USB flash drive to the USB port on the front right-hand side of the switch

5) Unplug the power cable from the switch power supply (PSU). All LEDs will go off. Replug the
power cable in to the switch PSU

6) The switch will power on, loading the firmware and configuration files

7) Confirm with FNA Ops ([email protected]) that the switch configuration process is complete.

4.2 Next Step

Re-image all cluster servers as described in Chapter 5: Re-image Servers to BSD

FNA: USB Based Software Installation Guide 13

Cha p ter 5 : Re- I m a ge t h e Ser ver s to BS D

This section describes how to use the USB boot disk to re-image the FNA
cluster servers.

1. Connect the flash drive containing the USB boot disk (see Chapter 2:
USB Boot Disk Creation) to the FNA server (see Figure 15).

Figure 15: USB Port Locations on FNA Servers

HP DL 380 Gen9 - Rear Panel

HP DL 380 Gen10 - Rear Panel

HP DL 380 Gen10plus - Rear Panel

2. Ensure that the server is powered on and connected to the FNA cluster switch.

3. Connect a VGA monitor to the server’s VGA port and a keyboard to a USB port.

4. Run the re-image script for the switch: ./local_image.sh command.

5. Verify that re-imaging was successful (see Figure 16).

Figure 16: Server Re-Image Command

6. If re-imaging fails you may receive a prompt that the system disk was not found (see Figure 17).

Figure 17: System Disk Not Found

FNA: USB Based Software Installation Guide 14

Cha p ter 5 : Re- I m a ge t h e Ser ver s to BS D

7. If you need to shut down the server after the re-image process (for example, to install it
somewhere else), run the poweroff command. Otherwise, powering off is not necessary.

8. Repeat the preceding steps for all cluster servers that need to be re-imaged.

FNA: USB Based Software Installation Guide 15

Cha p ter 6 : Tr o u b l esh o o t i ng

If any issues were encountered during the creation of the USB boot disk or
IP configuration, use the following methods for general troubleshooting.

FNA Operations can be reached at [email protected].

Operating System Boot Issues

1. Take a picture of the screen where the error has occurred.

2. Send this image with brief description of the issue to FNA Operations.

Network Connectivity Test

If a network connectivity test is requested after you contact FNA Operations, follow these steps:

1. Ping the default gateway IP address.

2. If the ping was successful (minimal packet loss), ping any known working address on the
Internet (for example, Facebook.com).

3. If the ping was unsuccessful, troubleshoot for network connectivity (per next method).

Network Connectivity Checks

1. Ensure all equipment in the uplink / downlink path is powered on.

2. Ensure that the network cables between each server in the FNA cluster are properly connected
to the FNA cluster switch.

3. Ensure that the SFP+ optical transceivers are properly seated in the switch uplink ports.

4. Ensure that the optical cables in the path between the FNA cluster and the facility router have
a good connection.

5. Ensure that the signal strength (light level) for the FNA cluster is between -2 and -7 dB.

Switch Configuration Issues

1. If the switch has not been configured, press [Ctrl + C] to end the current flow.

2. Take a picture of the output in this log:

cat /var/log/nginx/access.log

3. Send the pictures with a brief description of the issue and its duration to FNA Operations.

FNA: USB Based Software Installation Guide 16

Ap p end i x A : Leg a cy Sw i t c h C onf igur ation

This section describes how to use the USB boot disk created as described
in Chapter 2: USB Boot Disk Creation to configure the FNA cluster switch.

1.1A Cisco Nexus 3123 Switch Configuration

1. Select any of the servers with successfully reconfigured IP settings (as
described in Chapter 3: IP Address Configuration).

2. Ensure that the server is powered on and connected to the FNA cluster
switch.

3. Connect a VGA monitor to the server’s VGA port and a keyboard to a

USB port.

4. Run the configuration script for the switch: ./switch_configuration.sh

(see Figure 1a).

Figure 1a: Switch Configuration Command

5. When you receive the switch configuration prompt (see Figure 2a) select < Yes > and press
[ENTER].

Figure 2a: Switch Configuration Confirmation

6. If you are prompted that the server is not connected (see Figure 3a), check the connections
of both the server and its port (it should be the correct port on the server’s network card). Then
select < OK > and press [ENTER].

Figure 3a: Server Not Connected

7. If you are prompted that the switch configuration file was not found on the CONFIG partition
(see Figure 4a), select < OK > and press [ENTER]. Then follow the steps in Chapter 2: USB Boot
Disk Creation and make sure you moved the switch configuration file to the CONFIG partition
(look for a file name beginning with JPE, FOC, SSZ, or HSH). Restart the server with the USB flash
drive connected.

FNA: USB Based Software Installation Guide 17

Ap p end i x A : Leg a cy Sw i t c h C onf igur ation

Figure 4a: Switch Configuration Not Found on CONFIG Partition

8. Select < OK > and press [ENTER] at the configuration process prompt (see Figure 5a) to start
the configuration procedure.

Figure 5a: Configuration Process Prompt

9. Contact the FNA Ops team ([email protected]), providing them with the cluster name.

10. Wait up to 30 minutes. During the process, the FNA team will monitor the state of the switch.
Echo requests will be sent to predefined IP addresses and a response will be made when it is
configured (see Figure 6a).

Figure 6a: Switch Configuration Monitoring

11. Verify whether switch configuration is complete by following either step a, b or c:

a. If you receive a prompt that switch configuration was successful (see Figure 7a), select
<OK> and press [ENTER].

FNA: USB Based Software Installation Guide 18

Ap p end i x A : Leg a cy Sw i t c h C onf igur ation

Figure 7a: Successful Switch Configuration

b. If you receive a prompt that the switch serial number was not found on the CONFIG
partition (see Figure 8a), note the number listed in the prompt and contact FNA Operations at
[email protected].

Figure 8a: Switch Serial Number Not Found on CONFIG Partition

c. If no prompts at all appear (such as Figure 7a or Figure 8a) after 30 minutes, press
[Ctrl + C] to end the current flow and follow the procedures as described in Chapter 5: Re-
Image the Servers to BSD.

1.2A Next Step

1. Re-image all cluster servers as described in Chapter 5: Re-Image the Servers to BSD.

FNA: USB Based Software Installation Guide 19

Ap p end i x B : H P E G en 1 0p l us Manual Rem ed iations

This section describes how to restore network connectivity for Gen10plus

servers by manually setting the Broadcom network card speed setting.

1.1B Broadcom NIC speed setting

Symptoms
Network cable connected from switch to server, the network port is
configured on the switch but there is no connection - specifically the port is
not “up” in the switch, there is no connection in the OS, and the link LED’s
aren’t illuminated. The iLO also will not be contactable.

Issue
For Gen10plus servers the Broadcom network card has a speed setting which if misconfigured can
break connectivity. With FNA servers this results in complete loss of all connectivity to the server
because the server shares it’s network link with the iLO.

For now the solution requires a manual fix which cannot be performed remotely.

Procedure to correct
1. Attach a monitor to the VGA port and a keyboard to a USB port.

2. Power the server off, then on.

3. When prompted, press F9 to enter System Utilities.

FNA: USB Based Software Installation Guide 20

Ap p end i x B : H P E G en 1 0p l us Manual Rem ed iations

4. Select System Configuration and hit enter.

5. Select OCP Slot 10 Port 1 : Broadcom NetXtreme-E Single 100gb OCP 3.0 Ethernet and hit
enter.

FNA: USB Based Software Installation Guide 21

Ap p end i x B : H P E G en 1 0p l us Manual Rem ed iations

6. Select Device Configuration Menu and hit enter.

7. Ensure that Operation Link Speed is set to AutoNeg.

8. Press F12 to Save and Exit.

FNA: USB Based Software Installation Guide 22