Model Protocol

Eluvio Tagger model protocol

The Eluvio Tagger runtime is easily extensible by implementing new models as OCI containers which follow a simple protocol.

Requirements

• Containers must use OCI image format: docker, podman, etc…
• The container must stay alive and accept newline separated input files via stdin
• The container must offer a command line argument --output-path which will point to a .jsonl file (newline separated json), and it should write all it’s outputs to this file. Each line in the file will be a json object which represents a message. Details of the message format are described in the Message format section

Optional

• Offer a command line parameter --params where the caller can pass a json object which represents any runtime params you want to expose to the end user.

Example

echo -e "/container/path/input1.mp4\n/container/path/input2.mp4" | docker run -i -v /host/path:/container/path localhost/hotdog-detection --output-path /container/path/out/out.jsonl --params '{"fps":5}'

If properly configured, running the above command (substituting your own parameters, filepaths, image name) should generate a file /host/path/out.jsonl with all the containers’ outputs.

Message format

common

• All messages contain two top-level fields: “type” and “data”.
• There are 3 valid types: “tag”, “file processed”, “error”
• The schema for “data” depends on the type

message types

• tag - the tag to be produced and ultimately written to the content fabric
• progress - optional indicator that a given input file has been processed, helps the tagger to generate accurate progress reports
• error - error message to report to end user in case of tagging failure

tag

tags have the following data fields:

• start_time (integer in milliseconds) [Required]
• end_time (integer in milliseconds) [Required]
• source_media (string) [Required]
  • this should be equal to the input filepath that was used to generated the tag. This is necessary in order for the tagger system to be able to align the start/end_time against the source content object
• tag (string)
• track (string) [optional]
  • overwrite the track name where the tagger will write it’s tags
• frame_info (object) [if relevant]
• additional_info (object) [optional]
  • store arbitrary additional metadata here

frame_info is an optional field which contains bounding box info and a frame index. The box contians an arbitrary list of coordinate points to render a polygon (see “Example with all fields” below).

simple example tag

{
    "type": "tag",
    "data": {
        "start_time": 0,
        "end_time": 15500
        "tag": "dog",
        "source_media: "input.mp4"
    }
}

example with all fields

{
    "type":"tag", 
    "data": {
        "tag":"dog",
        "start_time": 10500,
        "end_time": 15000,
        "track": "hotdog_detection",
        "frame_info": {
            "frame_idx": 70
            "box": {
                "x1":0.3,
                "y1": 0.5,
                "x2": 0.5,
                "y2": 0.7
            }
        },
        "additional_info": {
            "confidence": 0.5,
            "breed": "golden retriever"
        },
        "source_media": "input.mp4"
    }
}

progress

example

{
    "type": "progress",
    "data": {
        "source_media": "media/input.mp4"
    }
}

The above message will signal to the tagger that the input file input.mp4 is fully processed, and no more tags are expected from that file.

error

example

{
    "type": "error",
    "data": {
        "message": "received a file of type '.png' which is not supported"
        "source_media": "media/input.mp4"
    }
}