Advanced Topics

Storage Encryption

By default, items in the Content Fabric are stored in encrypted form, and a private key is required to authorize access and decrypt. For some use cases, however, you may wish to make something available for playout to the general public without any authorization at all (e.g. a movie trailer). In these cases the data must be stored without encryption (aka "clear").

If you wish to create an Offering that allows playout without authorization you must use an ABR Profile that specifies "store_clear": true

In the elv-utils-js/example-files directory, these are the files with names ending in "_store_clear.json". See Reference: elv-utils-js /example-files ▸ ABR Profiles for a complete list.

Note that DRM is not available on Offerings ingested using clear storage. If you change your mind and wish to add DRM later, you will need to re-ingest using an ABR Profile that specifies "store_clear": false

Access Groups

One of the ways that the Content Fabric organizes and controls access to Objects is via Access Groups. These are groups that can be granted permissions to Libraries and Objects, and whose members are either individual users (i.e. private keys) or other Access Groups.

There are 2 different kinds of permissions for Libraries:

• list objects - see the list of objects in the library (and each object's name and other public metadata)
• add objects - create and add new objects to the library

There are 3 different kinds of permissions for Objects.:

• see - See only public metadata for the object
• access - See all details of the object, public and private
• manage - See all details of the object, public and private, as well as edit the object

When your tenancy was set up, the following three Access Groups were created:

• TENANCY_NAME Content Admins (can create new objects in Libraries)
• TENANCY_NAME Content Users (cannot create new objects in Libraries)
• TENANCY_NAME Tenant Admins ()

Note that having "list objects" permission for a Library automatically grants "see" access to all Objects within.

Object Permission / Visibility

Each Object in the Content Fabric has a base permission level setting. This Object Permission setting can be misleading however, as it can be overridden by granting Access Groups permissions on an Object.

Just because an object is set to "Owner Only" does not necessarily mean that no one else can read or edit the object. For example, after creating a Mezzanine, the owner could then give the Content Admins group permission to manage the Object. This will allow any member of that Access Group the ability to edit the Mezzanine, regardless of the Object Permission setting.

User Account / Profile

Content Fabric Accounts gain access via a Private Key. Do not share your Private Key. Keep a copy of it in a safe place. If you lose your Private Key, it cannot be recovered.

In addition to a Private Key, Accounts also have a Public Key and an Address.

You can view these items in the Fabric Browser by clicking on the ELUV.IO logo then clicking Profile. When you first go to this page, only your Address will be visible:

The Address is used for funding your Account. If you ever need an administrator to add credits to your Account, you will need to let them know your Address.

A Content Fabric Address consists of 0x followed by 40 hex digits (0-9 and/or a-f).

To view your Public and Private Keys, click on the key icon.

A Private Key consists of 0x followed by 64 hex digits.

A Public Key is a long alphanumeric string of 90 or more characters.

The Fabric Browser Profile page can also be used to edit the name and image displayed for your Account.

Playout (Streaming) Formats

The two most common formats for streaming media over the internet are HLS and DASH. The Eluvio Content Fabric supports both, and you can offer both formats using a single Offering.

Subtitles - Forced

Forced Subtitles are ones that translate audio dialogue or text seen on screen (e.g. a street sign or a newspaper headline) that may not be understood by the viewer - e.g. for a Spanish-speaking viewer, the following is an example translating on-screen English text:

They are most commonly used for scenes where the audio dialogue is in a language not expected to be understood by the viewer (e.g. fictional languages like Elvish, Klingon or Dothraki, or any language other than that used in the rest of the sound track). The example below is for English-speaking viewers, translating lines spoken in Klingon:

Forced subtitles are usually quite sparse, as they do not translate all spoken lines in a program. A given piece of content may only have a few lines.

There may be multiple sets of forced subtitles (often one for every audio language track) and they are meant to be used together with the matching audio track - e.g. if someone has chosen the English audio track, then an English forced subtitle track will be used (if one exists) and likewise if someone has selected a Spanish audio track, then Spanish forced subtitles will be used. In Offerings that have multiple audio and/or subtitle streams, every stream should be assigned a language code to help players automatically select the appropriate streams.

They are called 'forced' because they are supposed to be shown even if the viewer has not turned on subtitles. For this reason, they are also not supposed to show as a choice in UI subtitle pickers.

Instead, when a user switches audio track to a particular language, if subtitles are turned OFF, then forced subtitles for the matching language should automatically be used.

When a user selects a subtitle track, then no forced subtitle track should be used at all. However, 'normal' subtitle tracks often include the forced subtitles anyway. This is recommended by Netflix as best practice:

On our service, Forced Narrative subtitles are only displayed if Subtitles and CC are set to "off" in the user's playback settings. When the user activates a full Subtitle or SDH/CC file, the FN subtitle does not display and for this reason, we require that all Forced Narrative events are also included in each full Subtitle and SDH/CC file.

(from Understanding Forced Narrative Subtitles – Netflix | Partner Help Center )

Note that while the terms 'subtitles' and 'captions' are often used interchangeably, strictly speaking there is a difference in definition:

• subtitles are meant for people who don't understand the audio language (but can still hear)
• captions are meant for people who cannot hear, and may include additional text descriptions of audio events, e.g.:

In practice however this distinction is not always observed. Currently in the fabric we do not distinguish between subtitles and captions, and internally the two terms are used interchangeably.

The Eluvio Content Fabric supports Forced Subtitle tracks. However, player support for them is uneven, particularly among open source projects. Depending on what player is used, Forced Subtitles may be treated (incorrectly) as just another normal subtitle stream and shown alongside them:

x265/HEVC Mezzanines

The Content Fabric supports creating Mezzanine Offerings that use the x265 codec (also known as HEVC) for playout, with the following limitations:

• no DRM
• no watermarking
• only 1 resolution/bitrate in the ABR ladder

Also note that you cannot directly set a target bitrate for the mezzanine - it is only possible to set the following 2 parameters instead:

• Constant Rate Factor (CRF) - a value from 0-51, where 0 is lossless and 51 is worst quality.
• Preset - a setting which controls tradeoff between transcoding speed and quality, with the following possible values:
• ultrafast
• superfast
• veryfast
• faster
• fast
• medium
• slow
• slower
• veryslow
• placebo

See the FFmpeg guide to x265 encoding for more information on the above two settings.

To generate a Mezzanine Offering with x265 playout, you must use an ABR Profile that specifies x265 encoding. The elv-utils-js repo contains a sample x265 ABR Profile for 16x9 4k video, with CRF set to 28 and a Preset of "medium": elv-utils-js/example-files/abr_profile_x265_4k_16x9_crf28_medium_clear.json.

Note that creating an x265 Mezzanine Offering will take significantly longer than creating an x264 mezzanine, with the time difference increasing at higher quality settings.

HDR Mezzanines

NOTE: you must use an x265 ABR Profile to create HDR Mezzanines. (The Content Fabric currently does not support HDR for x264, as few playout devices support 10-bit x264 decoding).​​ The elv-utils-js repo contains a sample x265 ABR Profile, elv-utils-js/example-files/abr_profile_x265_4k_16x9_crf28_medium_clear.json. See the x265/HEVC Mezzanines section for additional information.

To ingest a High Dynamic Range (HDR) video it is necessary to supply additional information on how the master file was prepared:

• Mastering display properties (information on the display used during mastering)
• RGB Color Primaries
• White point
• Brightness range
• Maximum Content Light Level (MaxCLL - the light level of the brightest pixel in the entire video)
• Maximum Frame-Average Light Level (MaxFALL - the average light level of the brightest frame in the entire video)

This information is sometimes provided in a separate file (sometimes called a "sidecar" file), but it may also be included in the file as extra metadata.

HDR info contained in the video file

You can check if the file contains HDR metadata using ffprobe:

ffprobe \

  -v error \

  -show_entries streams:format \

  -print_format json \

  -i myVideoFile.mov 

Output (trimmed)

"side_data_list": [

  {

    "side_data_type": "Mastering display metadata",

    "red_x": "34000/50000",

    "red_y": "16000/50000",

    "green_x": "13250/50000",

    "green_y": "34500/50000",

    "blue_x": "7500/50000",

    "blue_y": "3000/50000",

    "white_point_x": "15635/50000",

    "white_point_y": "16450/50000",

    "min_luminance": "1/10000",

    "max_luminance": "10000000/10000"

  },

  {

    "side_data_type": "Content light level metadata",

    "max_content": 1000,

    "max_average": 400

}"

HDR info from sidecar files

Sidecar files come in a variety of formats. One example is the CPL file used by IMF packages. HDR mastering info in a CPL file looks like this:

  <EssenceDescriptor>

    <r1:MasteringDisplayWhitePointChromaticity xmlns:r1="http://www.smpte-ra.org/reg/335/2012">

          <r2:X xmlns:r2="http://www.smpte-ra.org/reg/2003/2012">15635</r2:X>

          <r2:Y xmlns:r2="http://www.smpte-ra.org/reg/2003/2012">16450</r2:Y>

        </r1:MasteringDisplayWhitePointChromaticity>

        <r1:MasteringDisplayPrimaries xmlns:r1="http://www.smpte-ra.org/reg/335/2012">

          <r2:ColorPrimary xmlns:r2="http://www.smpte-ra.org/reg/2003/2012">

            <r2:X>34000</r2:X>

            <r2:Y>16000</r2:Y>

          </r2:ColorPrimary>

          <r2:ColorPrimary xmlns:r2="http://www.smpte-ra.org/reg/2003/2012">

            <r2:X>13250</r2:X>

            <r2:Y>34500</r2:Y>

          </r2:ColorPrimary>

          <r2:ColorPrimary xmlns:r2="http://www.smpte-ra.org/reg/2003/2012">

            <r2:X>7500</r2:X>

            <r2:Y>3000</r2:Y>

          </r2:ColorPrimary>

        </r1:MasteringDisplayPrimaries>

        <r1:MasteringDisplayMaximumLuminance xmlns:r1="http://www.smpte-ra.org/reg/335/2012">

          40000000

        </r1:MasteringDisplayMaximumLuminance>

        <r1:MasteringDisplayMinimumLuminance xmlns:r1="http://www.smpte-ra.org/reg/335/2012">

          50

        </r1:MasteringDisplayMinimumLuminance>

        ...

<ExtensionProperties>

    <cc:ApplicationIdentification xmlns:cc="http://www.smpte-ra.org/schemas/2067-2/2016">

      http://www.rohde-schwarz.com/schemas/2067-xx/201x

    </cc:ApplicationIdentification>

    <MaxCLL xmlns="http://www.smpte-ra.org/schemas/2067-21/2016">4000</MaxCLL>

    <MaxFALL xmlns="http://www.smpte-ra.org/schemas/2067-21/2016">1239</MaxFALL>

Sometimes, instead of explicit values for the RGB Color Primaries and/or White Point, you may just see a standard cited, e.g.: "DCI-P3 color primaries" or "D65 White Point". In these cases you will need to look up the relevant standard.

Formatting the HDR information

The Content Fabric uses HEVC-style whole number values for HDR mastering information. You may instead be supplied with HDR10-style fractional values. To convert HDR10-style values to HEVC-style, multiply by the relevant conversion factor below:

Once you have the values in HEVC format, put them into a JSON file using the following format (note that the max_cll field also includes MaxFALL value):

{

  "master_display": "G(13250,34500)B(7500,3000)R(34000,16000)WP(15635,16450)L(40000000,50)",

  "max_cll": "1339,770"

}

You can then use the MasterAddHDRInfo.js utility to add the information to your Master object.

Streamlining Commands with Config Files

It is possible to use a Configuration File to automatically set environment variables and (some) command line options for you. These are JSON files with information specific to your Tenancy.

Safeguarding your information

As Configuration Files often contain your Private Key and other potentially sensitive information you should limit who can read them, e.g. with:

chmod 600 $YOUR_CONFIGURATION_FILE_NAME

Usage

There are two ways to use a Configuration File:

• Include --confs $PATH_TO_YOUR_CONFIG_FILE in your command line
• Set the ELVUTILS_CONFIG environment variable

# using --confs

node ListLibraries.js --confs $PATH_TO_YOUR_CONFIGURATION_FILE

# using environment variable

export ELVUTILS_CONFIG=$PATH_TO_YOUR_CONFIGURATION_FILE

node ListLibraries.js

When you run most utility scripts, if a Configuration File is specified using either of the above methods you will see a notification at the beginning of the script's output, e.g.:

ELVUTILS_CONFIG env var found, loading config from: /myFolder/myConfig.json

There are a few utilities that do not take any options at all. These are not affected by Configuration Files.

Format (Basic)

A basic Configuration File has one entry named "defaults" with a number of sub-entries: /elv-utils-js/example-files/config_basic.json

{

  "defaults": {

    "abrProfile":       "$_ElvUtilsDir_/example-files/abr_profile_both.json",

    "FABRIC_CONFIG_URL":"https://... (Your Fabric configuration URL)",

    "groupAddress":     "0x... (your Content Admins group address)",

    "masterLib":        "ilib... (your Master library ID)",

    "masterType":       "iq__... (your Master content type ID)",

    "mezLib":           "ilib... (your Mezzanine library ID)",

    "mezType":          "iq__... (your Mezzanine content type ID)",

    "PRIVATE_KEY":      "0x... (your private key)"

  }

}

• Sub-entries with names that are in ALL_CAPS_WITH_UNDERSCORES (aka "SCREAMING_SNAKE_CASE") will be used to set environment variables. In the example above, FABRIC_CONFIG_URL and PRIVATE_KEY will be set while the script is running. More specifically, to be used as an environment variable, the entry name must:
• Contain only capital letters and underscores
• Contain at least one underscore
• Begin and end with a capital letter
• Not be "ELVUTILS_CONF"
• Sub-entries with names that do not not qualify as environment variables will be used to fill in any missing command line options with the same name (or an option alias that matches the name)
• The values stored for sub-entries must be either text or arrays of text. This is the case even for numeric options such as --ethContractTimeout (e.g. "ethContractTimeout": "30")
• Arrays of text are used for command line options that take multiple values, e.g. --files
• Dollar signs ($) have a special meaning in values - they are used to specify substitution variables.
• For example, "$_ElvUtilsDir_" in the value at /defaults/abrProfile will be substituted with the path to your /elv-utils-js directory.
• If you actually need to include a dollar sign in a value, use two in a row ($$). These will not be processed as substitution variables, and will be replaced with single dollar signs before use.

Command line option handling

To see a command's list of options and their aliases, run it with the --help option, e.g.:

node MasterCreate.js --help

Sample (partial) output:

--libraryId, --masterLib, --master-lib,             Library ID for new master (should start with

--library-id                                        'ilib')                    [string] [required]

--metadata                                          JSON string for metadata or path to JSON file

                                                    containing the metadata of master object

                                                                                          [string]

--storeClear, --store-clear                         Store uploaded/copied files unencrypted

                                                                                         [boolean]

--streams                                           JSON string or path to JSON file containing

                                                    stream specifications for variant in new

                                                    master                                [string]

--type, --masterType, --master-type                 Name, object ID, or version hash of content

                                                    type for new master        [string] [required]

The above shows that --type has 2 aliases: --masterType and --master-type. All three are synonymous.

If you use the config_basic.json file and run MasterCreate.js without --type (or --masterType or --master-type), then the value stored in the Configuration File under /defaults/masterType will be used. Similarly, if --libraryId is missing, the value under /defaults/masterLib will be used, allowing you to shorten the MasterCreate.js command (and skip having to set environment variables):

node MasterCreate.js \

  --title "$YOUR_TITLE" \

  --files $PATH_TO_YOUR_FILE

Advanced usage: multiple config files

You can create multiple Configuration Files for different scenarios and switch between them - for example, if you ingest from S3, you can create a second file with additional entries:

{

  "defaults": {

    "abrProfile":       "$_ElvUtilsDir_/example-files/abr_profile_both.json",

    "FABRIC_CONFIG_URL":"https://... (Your Fabric configuration URL)",

    "groupAddress":     "0x... (your Content Admins group address)",

    "masterLib":        "ilib... (your Master library ID)",

    "masterType":       "iq__... (your Master content type ID)",

    "mezLib":           "ilib... (your Mezzanine library ID)",

    "mezType":          "iq__... (your Mezzanine content type ID)",

    "PRIVATE_KEY":      "0x... (your private key)",

    "AWS_BUCKET":       "your-bucket-name",

    "AWS_KEY":          "your-aws-key",

    "AWS_REGION":       "your-aws-region",

    "AWS_SECRET":       "your-aws-secret",

    "s3Reference":      "true"

  }

}

If you do set up multiple Configuration Files, it is recommended to use the --confs option instead of the ELVUTILS_CONFIG environment variable.

If you do use the ELVUTILS_CONFIG environment variable and work with multiple Configuration Files, then to help prevent mistakes you may wish to change your command line system prompt to show the value of $ELVUTILS_CONFIG. For example, on recent versions of OS X (which use the ZSH shell) you can insert the following in your ~/.zshrc file:

setopt prompt_subst

function precmd() {

  if [ -z "$ELVUTILS_CONFIG" ];

  then

    PROMPT="%n@%m %1~  %# "

  else

    CONFIG_FILENAME=$(basename $ELVUTILS_CONFIG)

    PROMPT="%n@%m %1~ %B%F{yellow}$CONFIG_FILENAME%f%b! "

  fi

}

It is also possible to use multiple Configuration Files in a single command, either by using both --confs and ELVUTILS_CONFIG or by listing multiple files after --confs. The files will be merged together, with later files taking precedence over earlier files if the same entry exists in multiple files. (If both --confs and ELVUTILS_CONFIG are used then the file specified by ELVUTILS_CONFIG is loaded first, then the file(s) listed after --confs are merged on top of it)

Format (Advanced)

In addition to the "defaults" section, a Configuration File can have a "presets" section. This section holds groups of settings that can be imported into the "defaults" section by adding a "presets_add" entry.

The example file /elv-utils-js/example-files/config_complex.json uses Presets. Note that a Preset can also import other Presets by specifying a "presets_add" entry - for example, the "ingest_s3" Preset combines the "ingest" and "s3" presets: /elv-utils-js/example-files/config_complex.json

{

  "defaults": {

    "presets_add": ["client", "tenant",  "ingest"]

  },

  "presets": {

    "client": {

      "ethContractTimeout": "30"

    },

    "tenant": {

      "FABRIC_CONFIG_URL": "https://demov3.net955210.contentfabric.io/config",

      "PRIVATE_KEY":       "your-private-key",

      "groupAddress":      "0x.. your-content-admins-group-contract-address",

      "masterLib":         "ilib... your-master-library-id",

      "masterType":        "iq__... your-master-content-type-id",

      "mezLib":            "ilib... your-mezzanine-library-id",

      "mezType":           "iq__... your-channel-content-type-id"

    },

    "s3": {

      "AWS_BUCKET": "your-bucket-name",

      "AWS_KEY":    "your-aws-key",

      "AWS_REGION": "your-aws-region",

      "AWS_SECRET": "your-aws-secret"

    },

    "abr_profiles": {

      "abr_profile_dir":              "$_ElvUtilsDir_/example-files",

      "abrp_both":                    "$abr_profile_dir/abr_profile_both.json",

      "abrp_no_drm_store_encrypted":  "$abr_profile_dir/abr_profile_no_drm_store_encrypted.json",

      "abrp_no_drm_store_unencrypted":"$abr_profile_dir/abr_profile_no_drm_store_unencrypted.json",

      "abrp_drm_all":                 "$abr_profile_dir/abr_profile_drm.json",

      "abrp_drm_strict":              "$abr_profile_dir/abr_profile_drm_strict.json"

    },

    "ingest": {

      "presets_add": ["abr_profiles"],

      "abrProfile": "$abrp_both"

    },

    "ingest_s3": {

      "presets_add":      ["ingest","s3"]

    },

    "ingest_s3_reference": {

      "presets_add": ["ingest_s3"],

      "s3Reference": "true"

    },

    "use_example_video_file": {

      "files":            "$_ElvUtilsDir_/example-files/video.mp4"

    },

    "airtable" : {

      "AIRTABLE_API_KEY": "your-airtable-access-token",

      "baseId":           "your-airtable-base-ID"

    },

    "debug": {

      "ELVUTILS_THROW": "1",

      "debug":          "true"

    },

    "lib_master": {

      "libraryId": "$masterLib"

    },

    "lib_mez": {

      "libraryId": "$mezLib"

    },

    "type_master": {

      "type": "$masterType"

    },

    "type_mez": {

      "type": "$mezType"

    }

  }

}

Note also that dollar signs can be used for more than just the pre-defined $_ElvUtilsDir_ substitution variable. They are also used to create cross-references to other entries. In the example above, the "abr_profiles" Preset has an entry of "abrp_both" : "$abr_profile_dir/abr_profile_both.json". The $abr_profile_dir part of that entry's value will be swapped out and replaced by the value of the "abr_profile_dir" entry before being used by the script. (The "abr_profile_dir" entry's value itself contains $_ElvUtilsDir_, which will be replaced by the actual path to /elv-utils-js.)

Advanced usage: --presets

When you run a command, you can also cause one or more Presets to be included by using --presets followed by the Preset name(s). In the config_complex.json file, there are a number of Presets that are not imported by the "defaults" section, for example:

    "lib_master": {

      "libraryId": "$masterLib"

    },

    "lib_mez": {

      "libraryId": "$mezLib"

    },

If you were using a similar Configuration File, you could list the Objects in your Master Library with:

node LibraryListObjects.js --presets lib_master

Since --libraryId was not supplied on the command line, the value under /presets/lib_mez/libraryId/ will be used ($mezLib winds up cross-referencing to /presets/tenant/mezLib).

Updating your copy of elv-utils-js

If you need to update to the latest version of elv-utils-js, you can do so with:

cd elv-utils-js

git pull

npm install

npm audit fix