3 MediaGrid Version 4.1 To 4.3 Updates

MediaGrid version 4.1/4.2/4.
3
updates
Patrick Moey / Loh Cheng Song
©2016 Harmonic Inc. All rights reserved worldwide.

MediaGrid version 4.1

MediaGrid version 4.1updates
• Local User Authentication

• New ContentStore (CSJ-5840A)
• Lower 10 GbE Ports
• Replication over RAID
• Other Improvements
• Critical Fixes
©2016 Harmonic Inc. All rights reserved worldwide. 3

Local User Authentication

Local User Authentication
• Provide Authentication, access controls, and auditing WITHOUT:

– The expense of Active Directory
– The complexity of OpenLDAP
– The hardware for Apple OpenDirectory
• Support ALL current access methods
– Samba/CIFS, FTP, NFS, FSD, etc.
• Small & Medium MG Implementations
– Fewer users and groups

Webmin
• Modern UI
• Self-signed Certificate
– Always HTTPS (https://)
• Admin ONLY access

– ‘admin’ password set during install
– can be changed manually with ovnuser + sudo
• Users and groups with “uid/gid > 500” are viewed / created / modified
– Prevents root permissions
– Prevents modifying system accounts
• Portal access through IP address of CLD/HBCB
– Port 8080
• Clustering & Samba support built-in

• Open Source tool with 100% BSD License

Login
 URL https://10.4.28.8:8080/?dashboard
 Harmonic Branding

Clustering
• Clustering based on Webmin Servers Index

– No master node
– Easy setup
• Allows user/group creation across entire cluster

• Common Users/Common Groups
– Shows common users/groups across cluster
• “Synchronize” users and groups across cluster

– Allows easy RMA

Webmin Servers Index

Webmin common Users and Groups

No Backend access
• Users created through Webmin will not have shell login.
[root@cld ~]# ssh user@localhost

user@localhost's password:
This account is currently not available.
Connection to localhost closed.

New ContentStore CSJ-5840A

ContentStore 5840A
• New ContentStore JBOD CSJ-5840A

– Available in 4TB and 6TB configurations
• Functional replacement for the CSJ-5840
– Supported with both the MG-3000 and MG-4000 platforms.
– No functional differences to CSJ-5840, except:
• 12 Gb/s SAS Connection to CSS-4000 (SFF-8644 to SFF-8644 - miniSASHD )
• 12 Gb/s SAS Disks
• 6 Gb/s SAS Connection to MG-3000 (SFF-8644 to SFF-8088 - miniSASHD to miniSAS)
• Performance improvements
– In 4.1 the performance will remain as published for the current platforms
– In future, there may be possible improvements that can be realized
• 12G DDiC can be used in CSJ-5840 but 6G DDIC can not be used in CSJ-5840A.
CSJ-5840A: Front Panel Components

CSJ-5840A: Rear Panel Components

CSJ-5840A: I/O Module

Lower 10GbE ports

Lower 10GbE ports
• In 4.0
– MediaGrid 4000 requires 8 x 10Gb Ethernet port connections
– 4 x 10GbE ports on each of the ContentServer 4000 controllers
• In 4.1
– MediaGrid 4000 supports lower 10GbE port connections
– Requires min of 4 x 10Gb Ethernet port connections per CSS.
– 2 x 10GbE ports on each of the ContentServer 4000 controllers
• To use a lower port count
– Plug in only 2 of the 4 NICs on each controller.
– No special configuration is required.
• Must connect 2 ports on each controller.
• If 5840A is in use, you must connect all 8 ports.

CSS 4000 with lower ports connected
2 x 10GbE ports on each

controller
• Controller level redundancy
• Switch level Redundancy

Replication over RAID
• Stretch cluster configuration

– Support for replication on systems with CSS 4000 (RAID)
• Supports a distance of 20km for replication.

Other Improvements
• Improvements for Active Directory users and groups

– User's group membership changes on AD server are replicated to CLB, FSD, and CLD
• Added new alerts in System Manager

– When a ContentStore PSU is disconnected from power.
• Applies to ContentStore 4240, 5840, and 5840A.
– ContentStore 3160 not supported due to h/w limitation.

Resolved Issues
• Please refer to the release notes for Resolved Issues


New Features
• RecycleBin
– Off by default
• Update to ContentStore CSJ-5840A expander firmware (ver. 5025)
– Address drawer open/close status
• Upgraded OpenSSH 4.7 for MediaGrid ContentDirector and ContentBridge (security
patch)
• Location change for system directory omipc
– Moved from temporary location /tmp/omipc to /var/omneon/omipc
– Prevents inadvertent deletion of critical files from tmp.

RecycleBin: Configuration
• Enable RecycleBin
– On each ContentDirector
– Open /omneon/config/mdscore-local file and add the following:
recyclebin_on = on
recyclebin_max_space = 3
recyclebin_retention_hint = 168
– Restart the Content Director(s)
• RecycleBin Parameter Max limit:-

– MDSCORE_RECYCLEBIN_RETENTION_MAX 24*30 // 30 days in hours = 720
– MDSCORE_RECYCLEBIN_MAX_SPACE_MAX 10 // 10 percent

RecycleBin: Command-Line Parameters
• To verify RecycleBin config: rbadmin live --getattributes
#/opt/omcld/bin/rbadmin --help
Usage:
rbadmin list [<src>] [--[r]sort=<attr>] [--fmt=<attr>[,<attr>] ...]
[--showtags[=0|1]]
(default: --fmt=ouid,opath,size,dtime)
rbadmin restore <src> [--dst=<path>]
rbadmin delete <src> [--force[=0|1]]
rbadmin live <livecmd>
 Refer to “Harmonic MediaGrid Installation and Configuration Guide” for additional information.
RecycleBin: Properties
• Permanent deletion from RecycleBin:

– Automatic by a background thread called ‘cleaner’ (FIFO)
– Uses a retention time limit: Max time file(s) present in RecyleBin (in hours) OR
– A space limit: Max disk space ReccyleBin can use (% of total Grid Capacity)
• Deleting hard-link will result in permanent deletion.
• ‘cleaner’ status: rbadmin live --cleanerstatus
– Displays space used
– CLD on which cleaner is active
#rbadmin live --cleanerstatus --user=omneon --pass=usm

Volume space: 231.90T, Cleaner: mgraid-eng45-1, Recycle Bin: Max space -
3.0%(6.96T) and Used space - 0.0%

RecycleBin: Recovering a file
• Logon to CLD as an administrator.

• To review the contents of the Recycle Bin: rbadmin list.
– Use --sort to sort the list
#rbadmin list --sort=opath

396318224 /mg4/dir01 0 2017.09.27 20:01:06.371 +0000
396318336 /mg4/dir01/file01.mov 512000000 2017.09.27 20:01:06.370 +0000
395378960 /mg4/dir02 0 2017.09.27 20:01:13.012 +0000
395567576 /mg4/dir02/file01.mov 512000000 2017.09.27 20:01:13.011 +0000
396334568 /mg4/dir03/file01.mov 512000000 2017.09.27 20:12:56.327 +0000
• To recover deleted file: rbadmin restore

– The filepath, username and password
#rbadmin restore /mg4/dir03/file01.mov --user=omneon --pass=usm --verbose
– verbose parameter to view additional details
/mg4/dir03/file01.mov: restored
countDone 1, countFail 0
RecycleBin: Permanent Deletion

• To review the contents of the Recycle Bin: rbadmin list
– Use --sort to sort the lists

396318224 /mg4/dir01 0 2017.09.27 20:01:06.371 +0000
396318336 /mg4/dir01/file01.mov 512000000 2017.09.27 20:01:06.370 +0000
• To permanently delete the directory and its contents: rbadmin delete
#rbadmin delete /mg4/dir01/file01.mov --user=omneon --pass=usm --verbose

Will permanently delete 1 file. Continue? [Yes/No] Y
/mg4/dir01/file01.mov: permanently deleted

RecycleBin: Recovering a directory


396318224 /mg4/dir01 0 2017.09.27 20:01:06.371 +0000
396318336 /mg4/dir01/file01.mov 512000000 2017.09.27 20:01:06.370 +0000
395378960 /mg4/dir02 0 2017.09.27 20:01:13.012 +0000
395567576 /mg4/dir02/file01.mov 512000000 2017.09.27 20:01:13.011 +0000
• To recover a directory and its contents: rbadmin restore
#rbadmin restore /mg4/dir02 --recursive=1 --user=omneon --pass=usm --verbose

/mg4/dir02: restored
/mg4/dir02/file01.mov: restored
RecycleBin: Permanently deleting a directory


396318224 /mg4/dir01 0 2017.09.27 20:01:06.371 +0000
396318336 /mg4/dir01/file01.mov 512000000 2017.09.27 20:01:06.370 +0000
• To permanently delete the directory and its contents: rbadmin delete
#rbadmin delete /mg4/dir01 --recursive=1 --user=omneon --pass=usm --verbose

Will permanently delete 2 files. Continue? [Yes/No] Y
/mg4/dir01/file01.mov: permanently deleted
/mg4/dir01: permanently deleted

New Features
• Up to 4x CSJ-5840/A enclosures with single a CSS-4000

New “Deep Stack” Configuration
High-Density and Performance
MG3000 MG4000 MG4000 MG4000

ContentDirectors
“Full Stack” “Full Stack” “Short Stack” “Deep Stack”
2.4 GBps 4 GBps 3.4 GBps 5.4 GBps
ContentServers
ContentStores
“Deep Stack” Configuration

• “Deep Stack” > 2PB Storage

Performance
With CSS 4000 + CSS 4240 Assume 50/50 read write

Dual Cont. Single Cont. Dual Cont. Single Cont.
MB/s Raid 2+1 Raid 6+2
CSS-4000 800 800 500 500
CSS 4000 = 1 x CSS4240 1600 1600 1000 1000
CSS 4000 = 2 x CSS4240 2400 2350 1450 1400
CSS 4000 = 3 x CSS4240 3100 3000 2000 1950
CSS 4000 = 4 x CSS4240 3750 3650 2400 2350
With CSS 4000 + CSS 5840A Assume 50/50 read write

Dual Cont. Single Cont. Dual Cont. Single Cont.
MB/s Raid 2+1 Raid 6+2
CSS-4000 800 800 500 500
CSS 4000 = 1 x CSS5840 3600 3550 2250 2200
CSS 4000 = 2 x CSS5840 4750 4400 4000 3700
CSS 4000 = 3 x CSS5840 5100 4200 4200 3850
CSS 4000 = 4 x CSS5840 5400 4100 4500 4050

Improvements
• Unit ID reporting for each CSJ-5840/A enclosures

– "mgcmtool enclist" reports the Unit ID for each CSJ-5840/A enclosure for easy indentification.
– Enclosures other than a CSJ-5840/A report ‘0’ as the value.
• Power on Hours for disk drives
– "bs ld defects" reports POH (Power On Hours) for disk drives in CSS/CSJ.
• "Good to Go" register values for NVDIMM
– Agigaram driver logs the actual GTG_1 and GTG_2 register values to assist in diagnostics when it
finds the overall capacitor charge insufficient.
• Default slice balancer algorithm
– Changed default SSM slice balancer algorithm to capacity based.
– Reduces the need to balance the slices by the balancer.

Improvements … cont.
• Alternate FTPHOME directory on a HBCB in NATIVE mode

– FTPHOME parameter made to work for customers who wish to change the FTP home directory
• Add the "ftphome=." (the relative path is the dot) mount option to the NATIVE directive.
– Example 1: without ftphome, the default directory for ftp transfers is /mnt/omfs/native
gateway.conf contains auth=ADS; NATIVE <mg> <gridfs>
FTP transfers land in /mnt/omfs/native if the directory is not changed to gridfs.
Observe: <gridfs>, <gridfs>/dir1, <gridfs>/dir1/file1
– Example 2: with ftphome set to “.” the default directory for ftp transfers is /mnt/omfs/native/gridfs
gateway.conf contains auth=ADS; NATIVE <mg> <gridfs> ftphome=.
FTP transfers land in /mnt/omfs/native/<gridfs>
Observe: dir1, dir1/file1
– Example 3: with ftphome set to dir1, the default directory for ftp transfers is /mnt/omfs/native/gridfs/dir1
gateway.conf contains auth=ADS; NATIVE <mg> <gridfs> ftphome=dir1
FTP transfer land in /mnt/omfs/native/<gridfs>/dir1
Observe: file1

Improvements … cont.
• Improved garbage collector cycle times

– Improved GC to process a batch of slice IDs rather than one at a time
– Cuts the time required by GC cycle almost in half thereby reducing the overhead on the system
• Retain the boot.log across controller reboots
– Existing boot.log files are preserved in /corefiles when the controller is forced a reboot during
startup, for example after an f/w download to help diagnosis.
• Increased threads for SSM
– The default value for commandThreads in SSM increased to 128
– This helps SSM to work more seamlessly during peak load periods
– Matches with the RPC threads in mdscore, another critical process in CLD

Compatibility
Product Software/Firmware Version

MediaGrid Server 4.3-2018.03.13 (4.3.0.0.8)
MediaGrid File System Driver (FSD) for Linux 3.4.x, 3.5.x, and 4.0
MediaGrid File System Driver (FSD) for Windows 3.4.x, 3.5.x, and 4.0
MediaGrid File System Driver (FSD) for Macintosh OSX 3.4.x, 3.5.x, and 4.0
MediaGrid ContentManager 3.0.1.0
SystemManager 6.9 and above

Capacity specifications for ContentDirectors
Content Director Max File Max Node Directory Slice Storage

Type Count Count Count Count Size
Supported* Supported Supported Supported Supported
(Files + (Files +
Symlinks) Directories
+ Links)
HPCLD-2000C 50 million 50 million 1.5 million 2 billion 4 PB
(64 GB RAM)

MediaGrid 4.0 FSD

Agenda
• New Features
• SSD Media Cache
• Adaptive Read Ahead
• Improvements
• Critical Fixes
• Known Issues
• Q&A

New Features
• SSD Media Cache

• Adaptive Read Ahead
• WinFSD
– Windows Server 2016
– Multi-user credential support
• MacFSD
– Mac default cache size Increased
– Persistent sysctl changes for Mac FSD
– Uninstall application for Mac FSD
• FSD can be uninstalled using an application instead of a command line

SSD Media Cache
• Disabled by default.
• Vastly expands the existing Client RAM Slice Cache beyond the boundaries of physical RAM.
• Provides performance benefits for NLE users
– On previously read files during an edit session.
– Files read off the Media Grid will be cached in the specified cache folder on a designated SSD drive.
– On next read activity, the file will be read from the SSD drive and not fetched over the network from Grid.
• The performance boost can be significant depending on
– Speed of SSD drive
– Throughput of network interface connection to Media Grid
– The total size of project files being read
• Recommended for users
– With a 1GbE client connection
– If a dedicated SSD drive available
– SSD drive has read/write speeds of more than 300MB/s
• FSD automatically validates SSD drive performance before use

SSD Media Cache (Windows)

SSD Media Cache (Mac)

SSD Media Cache – Options
• Enable cache : Select to cache slices read from Harmonic MediaGrid.

• Cache written data
– Select to cache both slices read from and written to MediaGrid. Improves speed for future read of the slice.
– On exports from NLE to a MediaGrid directory, slices will be cached for the new file being written.
• After export completes the file can be played on the same client and enjoy the benefit of previously cached slices.
• Cache folder : Enter the path to SSD Media Cache folder.

• Max SSD cache size
– Set the size of the SSD Media Cache. Useful if one partition is shared between several tasks.
– It is recommended to leave at least 10% free drive space on the SSD.
• Test speed : Click to test the write speed for the SSD Media Cache.
• Clear cache
– Click to clear the cached SSD Media Cache slices.
– Automatically removes cached slices when free capacity reaches within 5GB of Max SSD cache size.
Adaptive Read Ahead (ARA)
• Enabled by default
• Improves NLE read performance in real-time and shuttle playback modes.
• Automatically detects read patterns and adjusts the read-ahead of data from
MediaGrid.
• Helps to eliminate unnecessary reads of data and lowers network utilization
during playback.
• Benefits 1GbE clients the most, and 10GbE clients showed benefits as well.
• Activates an intelligent algorithm to detect different MediaGrid access patterns
and adjust read-ahead accordingly.
• Noticed significant performance increase for MacFSD clients.

Adaptive Read Ahead (Windows)

Adaptive Read Ahead (Mac)

ARA - Options
• Read-Ahead Setting: amount of data that will be read in advance and is most beneficial to
sequential read patterns
• Limit Read Bandwidth: read bandwidth limit for a client
• Maximum Connections: maximum number of connections the FSD will make to the
ContentDirector for metadata operations
• Maximum RAM Cache Size: Memory Limit for caching file data
• Optimize for nonlinear editing: Improve performance for specific workflows by altering the
behavior of file reading in the FSD. This is particularly useful to non-linear editing applications.
• Optimize for 1GbE connection: Internal optimizations for 1 GbE connection to set the amount
of data pre-fetched by FSD from MediaGrid.
• Adaptive read-ahead: Dynamically change read-ahead for data.
Recommended to turn off for workloads other than video processing to lessen additional load on
the network and on MediaGrid.

New Features: Windows
• Windows Server 2016 support

– Added support for Windows Server 2016.
• Multi-user credential support
– Introduced support for multiple users of the same workstation to connect with different credentials.

New Features: Mac
• Mac default cache size increased

– The default cache size for Mac has been raised to 400MB.
• Persistence sysctl settings
– Ability to specify sysctl settings in a way that persists after restarting the computer.
• Uninstall application
– Mac FSD can be uninstalled using an application instead of a command line.
• User Interface improvements
– The Mac FSD user interface has been improved to include a list of network interfaces, a field for
SSD Media Cache size limits, and a new Global Options tab.

Performance Improvements
• Performance has been improved (asynchronous) for read connections to slice

servers on a busy system - MG-12227
• The balancing of FSD connections to ContentServer 4000 has been improved -
MG-11955
• New options have been add to the "omservice" command to allow manipulation
of the saved user credentials on Windows - MG-11853
– add, update, expel, show
omservice -add <user_name> <password> <domain> to create new credentials
omservice -update <user_name> <password> <domain> to update password
for userdomain record
omservice -expel <user_name or *> <domain or *> to remove credentials
omservice -show to show all stored credentials

Thank You

3 MediaGrid Version 4.1 To 4.3 Updates

Uploaded by

Copyright:

Available Formats

3 MediaGrid Version 4.1 To 4.3 Updates

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

3 MediaGrid Version 4.1 To 4.3 Updates

Uploaded by

Copyright:

Available Formats

MediaGrid version 4.1/4.2/4.

Patrick Moey / Loh Cheng Song

©2016 Harmonic Inc. All rights reserved worldwide.

©2016 Harmonic Inc. All rights reserved worldwide.

• Local User Authentication

©2016 Harmonic Inc. All rights reserved worldwide. 3

©2016 Harmonic Inc. All rights reserved worldwide. 4

• Provide Authentication, access controls, and auditing WITHOUT:

©2016 Harmonic Inc. All rights reserved worldwide. 5

• Admin ONLY access

• Clustering & Samba support built-in

©2016 Harmonic Inc. All rights reserved worldwide. 6

©2016 Harmonic Inc. All rights reserved worldwide. 7

• Clustering based on Webmin Servers Index

• Allows user/group creation across entire cluster

• “Synchronize” users and groups across cluster

©2016 Harmonic Inc. All rights reserved worldwide. 8

©2016 Harmonic Inc. All rights reserved worldwide. 9

©2016 Harmonic Inc. All rights reserved worldwide. 10

• Users created through Webmin will not have shell login.

[root@cld ~]# ssh user@localhost

©2016 Harmonic Inc. All rights reserved worldwide. 11

©2016 Harmonic Inc. All rights reserved worldwide. 12

• New ContentStore JBOD CSJ-5840A

©2016 Harmonic Inc. All rights reserved worldwide. 14

©2016 Harmonic Inc. All rights reserved worldwide. 15

©2016 Harmonic Inc. All rights reserved worldwide. 16

©2016 Harmonic Inc. All rights reserved worldwide. 17

©2016 Harmonic Inc. All rights reserved worldwide. 18

2 x 10GbE ports on each

©2016 Harmonic Inc. All rights reserved worldwide. 19

• Stretch cluster configuration

• Supports a distance of 20km for replication.

©2016 Harmonic Inc. All rights reserved worldwide. 20

• Improvements for Active Directory users and groups

• Added new alerts in System Manager

©2016 Harmonic Inc. All rights reserved worldwide. 21

• Please refer to the release notes for Resolved Issues

©2016 Harmonic Inc. All rights reserved worldwide. 22

©2016 Harmonic Inc. All rights reserved worldwide.

©2016 Harmonic Inc. All rights reserved worldwide. 24

• RecycleBin Parameter Max limit:-

©2016 Harmonic Inc. All rights reserved worldwide. 25

• To verify RecycleBin config: rbadmin live --getattributes

• Permanent deletion from RecycleBin:

#rbadmin live --cleanerstatus --user=omneon --pass=usm

©2016 Harmonic Inc. All rights reserved worldwide. 27

• Logon to CLD as an administrator.

#rbadmin list --sort=opath

• To recover deleted file: rbadmin restore

• Logon to CLD as an administrator.

#rbadmin list --sort=opath

• To permanently delete the directory and its contents: rbadmin delete

#rbadmin delete /mg4/dir01/file01.mov --user=omneon --pass=usm --verbose

©2016 Harmonic Inc. All rights reserved worldwide. 29

• Logon to CLD as an administrator.

#rbadmin list --sort=opath

#rbadmin restore /mg4/dir02 --recursive=1 --user=omneon --pass=usm --verbose

• Logon to CLD as an administrator.