|
Earlier this yr, we preannounced that TwelveLabs video understanding fashions had been coming to Amazon Bedrock. As we speak, we’re saying the fashions are actually accessible for looking by means of movies, classifying scenes, summarizing, and extracting insights with precision and reliability.
TwelveLabs has launched Marengo, a video embedding mannequin proficient at performing duties similar to search and classification, and Pegasus, a video language mannequin that may generate textual content based mostly on video knowledge. These fashions are educated on Amazon SageMaker HyperPod to ship groundbreaking video evaluation that gives textual content summaries, metadata era, and inventive optimization.
With the TwelveLabs fashions in Amazon Bedrock, yow will discover particular moments utilizing pure language video search capabilities like “present me the primary landing of the sport” or “discover the scene the place the primary characters first meet” and immediately leap to these precise moments. You may as well construct purposes to know video content material by producing descriptive textual content similar to titles, subjects, hashtags, summaries, chapters, or highlights for locating insights and connections with out requiring predefined labels or classes.
For instance, yow will discover recurring themes in buyer suggestions or spot product utilization patterns that weren’t apparent earlier than. Whether or not you could have lots of or hundreds of hours of video content material, now you can rework that complete library right into a searchable data useful resource whereas sustaining enterprise-grade safety and efficiency.
Let’s check out the Marengo and Pegasus movies that TwelveLabs has printed.
You possibly can rework video workflows with these fashions throughout industries. Media producers and editors can immediately find particular scenes or dialogue, which suggests you may give attention to storytelling somewhat than sifting by means of hours of footage. Advertising groups are streamlining their promoting workflows by rapidly personalizing content material to resonate with numerous audiences, whereas safety groups are utilizing the expertise to proactively establish potential dangers by recognizing patterns throughout a number of video feeds.
Getting began with TwelveLabs fashions in Amazon Bedrock
Earlier than getting began, if you happen to’re new to utilizing TwelveLabs fashions, go to the Amazon Bedrock console and select Mannequin entry within the backside left navigation pane. To entry the most recent TwelveLabs fashions, request entry for Marengo Embed 2.7 or Pegasus 1.2 in TwelveLabs by accessible Areas.
To make use of TwelveLabs fashions in Amazon Bedrock, select Chat/Textual content Playground underneath Check within the left navigation pane. Select Choose mannequin, choose TwelveLabs because the class and Pegasus because the mannequin, after which select Apply.
For looking or producing textual content out of your video, you must add your video to an Amazon Easy Storage Service (Amazon S3) bucket or enter Base64-formatted video string.
I’ll use a pattern video generated with Amazon Nova Reel with a immediate: “A cute raccoon taking part in guitar underwater.”
Enter the S3 URI of your uploaded video and the S3 Bucket Proprietor, after which run your immediate: “Inform me concerning the video by timeline”.
You may get following mannequin response:
The video opens with an underwater scene that includes a sandy seabed and swaying seaweed, making a serene environment. At round 5 seconds, a raccoon seems, swimming in the direction of the digicam with a curious expression. By 10 seconds, the raccoon is seen swimming close to the seabed. The main target then shifts to a bunch of koi fish swimming gracefully close to the seabed from 11 to 13 seconds.
At 14 seconds, the scene transitions to an ukulele resting on the seabed, with the digicam zooming in to focus on its particulars. The raccoon reappears at 20 seconds, swimming among the many koi fish, and by 23 seconds, it begins to play the ukulele. The raccoon continues to play the ukulele from 29 to 41 seconds, with the koi fish swimming round it, including a energetic and colourful contact to the underwater setting.
From 42 to 49 seconds, the raccoon is proven taking part in the ukulele with the koi fish swimming round it. The video concludes with a display displaying "Video generated with Amazon Novo Reel 1.1" and "Obtainable in Amazon Bedrock" from 50 to 54 seconds, adopted by the AWS emblem from 55 to 57 seconds.
The TwelveLabs fashions may be simply built-in into your purposes utilizing the Amazon Bedrock Invoke API, which offers a unified interface for conversational AI interactions.
Right here’s an instance of the way to use the AWS SDK for Python (Boto3) with the TwelveLabs Pegasus mannequin:
import boto3
import json
import base64
AWS_REGION = "us-west-2"
MODEL_ID = "us.twelvelabs.pegasus-1-2-v1:0" # OR "eu.twelvelabs.pegasus-1-2-v1:0" whether it is doing cross area inference in europe
VIDEO_PATH = "pattern.mp4"
def read_file(file_path: str) -> str:
"""Learn a file and return as base64 encoded string."""
strive:
with open(file_path, 'rb') as file:
file_content = file.learn()
return base64.b64encode(file_content).decode('utf-8')
besides Exception as e:
increase Exception(f"Error studying file {file_path}: {str(e)}")
bedrock_runtime = boto3.consumer(
service_name="bedrock-runtime",
region_name=AWS_REGION
)
request_body = {
"inputPrompt": "inform me concerning the video",
"mediaSource": {
"base64String": read_file(VIDEO_PATH)
}
}
response = bedrock_runtime.invoke_model(
modelId=MODEL_ID,
physique=json.dumps(request_body),
contentType="software/json",
settle for="software/json"
)
response_body = json.hundreds(response['body'].learn())
print(json.dumps(response_body, indent=2))The TwelveLabs Marengo Embed 2.7 mannequin generates vector embeddings from video, textual content, audio, or picture inputs. These embeddings can be utilized for similarity search, clustering, and different machine studying (ML) duties. The mannequin helps asynchronous inference by means of the Bedrock StartAsyncInvoke API.
For video supply, you may request JSON format for the TwelveLabs Marengo Embed 2.7 mannequin utilizing the StartAsyncInvoke API.
{
"modelId": "twelvelabs.marengo-embed-2-7-v1:0",
"modelInput": {
"inputType": "video",
"mediaSource": {
"s3Location": {
"uri": "s3://your-video-object-s3-path",
"bucketOwner": "your-video-object-s3-bucket-owner-account"
}
}
},
"outputDataConfig": {
"s3OutputDataConfig": {
"s3Uri": "s3://your-bucket-name"
}
}
}You may get a response delivered to the required S3 location.
{
"embedding": [0.345, -0.678, 0.901, ...],
"embeddingOption": "visual-text",
"startSec": 0.0,
"endSec": 5.0
}That can assist you get began, try a broad vary of code examples for a number of use instances and a wide range of programming languages. To be taught extra, go to TwelveLabs Pegasus 1.2 and TwelveLabs Marengo Embed 2.7 within the AWS Documentation.
Now accessible
TwelveLabs fashions are usually accessible right now in Amazon Bedrock: the Marengo mannequin within the US East (N. Virginia), Europe (Eire), and Asia Pacific (Seoul) Area, and the Pegasus mannequin in US West (Oregon), and Europe (Eire) Area accessible with cross-Area inference from US and Europe Areas. Examine the full Area listing for future updates. To be taught extra, go to the TwelveLabs in Amazon Bedrock product web page and the Amazon Bedrock pricing web page.
Give TwelveLabs fashions a strive on the Amazon Bedrock console right now, and ship suggestions to AWS re:Submit for Amazon Bedrock or by means of your traditional AWS Assist contacts.
— Channy
Up to date on July 16, 2025 – Revised the screenshots and code half.
