How is content ingested into Core?

Introduction

This article is not a typical step-by-step guide. It is general information on how content is ingested into Core from Amazon S3 cloud storage.

For fully detailed information and training, please contact your Wazee Digital representative via the Helpdesk.

Core's Content Discovery Policies

A discovery policy defines what content is expected in the Amazon S3 storage location and determines how this content will be packaged and ingested into Core. Core’s content discovery policies are flexible and vary, from providing the ability to upload a single video asset to uploading multiple images with related metadata at separate times.  

Key Benefits

Core's content discovery policies help to enrich existing content with updated metadata information, they manage bulk uploads (reducing re-keying and other potential errors), and can be used as the foundation of an effective and efficient data migration tool. Industry standards are supported for cross-product compliance.

Pre-requisites

  • Your Amazon S3 bucket needs to be set up and configured. This can be your own account or a new one created by your Wazee Digital representative
  • Your Core account needs to be set up and configured. Core's system Origins and Locations need to be configured for the appropriate discovery policy / policies used

How Core's Content Discovery Policies Work

New Content

  • New content is uploaded into the Amazon S3 bucket. The key file is the driver of the discovery policy and must be uploaded last
  • The AWS Lambda service will notify Core's ingest service (NoM) of a new discovery with an internal system message
  • NoM will discover the content and parse its associated sidecar XML (if applicable)
  • A new database record is now created in Core
  • Masters, renditions and ancillary assets are then ingested against the new record
  • Associated metadata is mapped to the existing content database record
  • An email is sent (to pre-configured recipients) if any failure should occur

Content Updates

  • A new asset with same name is uploaded into Amazon S3:
    • If the rendition / ancillary asset already exists, it will replace the current asset in Core
    • If new sidecar XML metadata is uploaded and the metadata is different than the original Core database record, the metadata will be updated to reflect the new XML sidecar

Types of Core Content Discovery Policies

Single Asset

  • Supports the ingest of a single asset into Core without an accompanying sidecar XML or ancillary assets
  • Use Case: cloud storage locations with a single master asset, without an accompanying sidecar XML or ancillary / rendition assets  

Single Asset with Data

  • Supports the ingest of a single asset into Core with an accompanying sidecar XML and no ancillary / rendition assets
  • Use Case: cloud storage locations with an accompanying sidecar XML with a master asset that can be used to enrich the existing database record with additional metadata

Single Asset with Data and Renditions

  • Supports the ingest of a single asset into Core with accompanying sidecar XML and ancillary / rendition assets
  • Use Case: cloud storage locations with an accompanying sidecar XML with a master asset that can be used to enrich the existing database record with additional metadata and additional renditions to be ingested along with master asset  

Multiple Assets (a)

  • Supports the ingest of multiple assets into Core with a single sidecar XML with accompanying sidecar XML and ancillary / rendition assets
  • Use Case: bulk uploads can make the on-boarding of content and metadata even easier where one sidecar XML is used to provide metadata for up to 1,000 records. This is particularly useful for the on-boarding of large volumes of images or to quickly update large amounts of metadata for existing images. Up to 1,000 assets supported (i.e. 1,000 x rows plus 1 x header row) with a maximum of 150 columns of metadata

Multiple Assets (b)

  • Supports the ingest of multiple assets with a single sidecar into Core without an accompanying sidecar and ancillary / rendition assets
  • Use Case: provides the ability to ingest a spreadsheet of metadata without the related master asset.  Core will manage the synchronisation of metadata and video assets when the video asset is ingested

Video Discovery Policy Configurations

Formats Supported

  • mov
  • mxf
  • mp4
  • mpg
  • dv
  • hdv
  • 3gp
  • avi
  • cyuv
  • divx
  • flv
  • f4v
  • m1v
  • rm
  • rmvb
  • svi
  • wmv
  • webm

Single Asset

  • Video Master: video is the key file

Single Asset with Data

  • Video Master with Sidecar XML: video and sidecar XML with associated metadata in the Core XML format. In this option the sidecar XML is the key file
  • Video Master with EVS Sidecar: video and sidecar XML with associated metadata in the EVS format. In this option the sidecar XML is the key file and will pull standard EVS fields
  • Video Master with EVS + Golf Sidecar: video and sidecar XML with associated metadata in the EVS format. In this option the sidecar XML is the key file and will pull standard EVS and Golf fields

Multiple Assets (a) and (b)

  • Video Masters with Spreadsheet Sidecar (formats supported: csv, tsv, xls, xlsx): video with spreadsheet sidecar with specified Core asset data fields and values. In this option the spreadsheet sidecar is the key file
  • Multiple Assets with Alternative Master Assets: this policy allows the upload of a spreadsheet sidecar with metadata for multiple database records. This works like the existing Multiple Asset Discovery Policy - video masters (if the content is in the Amazon S3 bucket), but it will also create database records in Core with appropriate metadata if the content does not exist at the time of ingest. Master assets can be uploaded at another date to update database records with the master assets

Image Discovery Policy Configurations

Formats Supported

  • jpg
  • jpeg
  • gif
  • tif
  • tiff
  • tga
  • png
  • bmp

Single Asset

  • Image Master: image is the key file

Single Asset with Data

  • Image Master with sidecar XML: image and sidecar XML with associated metadata in the Core sidecar XML format. The sidecar XML is the key file

Multiple Assets (a) and (b)

  • Image Masters with Spreadsheet Sidecar (formats supported: csv, tsv, xls, xlsx): image with spreadsheet sidecar with specified Core asset data fields and values. In this option the spreadsheet sidecar is the key file

0 Comments

Add your comment

E-Mail me when someone replies to this comment