SAM 2 from Meta is the first unification for segmenting objects across images and videos. What makes the choices less restricted is that a ‘click,’ ‘box,’ or a ‘mask’ can be used for the input to select an object on any image or frame of video. With this feature, you can point at one or several objects in a frame of a video. Employ more prompts to continue narrowing down the model predictions.
Major features
- SAM 2 demonstrates high accuracy of the objects, images, and videos even if they were not included in the training phase and can be applied to various real-life cases.
- It is ready for video processing with streaming inference to support real-time interactions with the applications.
- This new features operates by integrating into a single model, new and improved video and image segmentation without having to sacrifice basic structural layout and high performance per run.
SAM 2 Enhanced segmenting
Unlike the original SAM, the SAM 2 model adds a per-session memory module to the promptable capability of SAM to assist in the video domain by storing information about the object of interest in the video. This enables SAM 2 to follow the selected object in all frames of the video; even when the object is not visible for some frames the model has an understanding of where the object is due to context from the previous frames. The second version of SAM also allows to correction of the mask prediction following the additional prompts that can be given on any frame.
SAM 2’s streaming which is a system that processes video frames one at a time is also a natural extension of SAM to the video domain. Namely, when applied to images, the memory module of SAM 2 has no contents, which means that the model would act like SAM.
Moreover, it was trained on a large and diverse set of videos and masklets which are object masks over time generated by interactively applying this in another model in the loop data engine. The training data also includes the SA-V dataset which we are also open sourcing. SAM 2’s video object segmentation outputs may be fed to other AI systems, particularly modern video generation models, giving the systems precise editing functionalities.