vapoursynth-letterbox

A VapourSynth plugin to find and clean letterbox.

Usage

You should use functions in the Python package to use this plugin. These functions perform some very important pre and postprocessing. Only with these protection is this letterbox masking safe to use.

The functions in the Python package depends on vs-jetpack. If you can't use vs-jetpack, it's highly suggested to copy the code out and implement it into your environment.

For video with no permanent letterbox:

from vsletterbox import clean_letterbox

clip = clean_letterbox(clip)

For video with 129-pixel permanent letterbox:

clip = clean_letterbox(clip, permanent=[129, 129]) # [Top, Bottom]

In addition to cleaning the noise, we can also perform border deringing based on the dynamic letterbox detected.

clip = clean_letterbox(
    clip,
    permanent=[129, 129],
    border_y=lambda clip: clip.bore.SinglePlane(top=4, bottom=4),
    border_u=lambda clip: clip.bore.SinglePlane(top=2, bottom=2),
    border_v=lambda clip: clip.bore.SinglePlane(top=2, bottom=2)
)

Alternatively, we also provide letterbox_mask function with which you can apply your own operations.

from vsletterbox import letterbox_mask

mask = letterbox_mask(clip, permanent=[129, 129])

Method

We detect letterbox based on three details:

1. We iterate row by row from the edge pixel in, calculating the mean brightness of the pixels in each row.
   We detect if starting from a certain row the mean brightness rapidly increases, through a statistical moving predicter.

2. We apply a sensitive general edgemask to the image, and we require the row detected in method 1. to have high edgemask coverage.
   This is to prevent cases such as title screens with only white text on black background to be detected as letterbox.

3. If the detected row is within 1 row from the user provided permanent row, we snap the detected row to the permanent row.
   This is to combat the case where there is not a clean cutoff at the border and random pixels protrude to the otherside.
   In addition, method 1. currently requires the border row to be at least 6.375 at 8-bit higher than the mean noise brightness. In very dark scenes with dirty border, there might not be enough change to trigger the detection on the very first border row. This is also to combat this potential issue.

4. When there is no letterbox detected such as in a pure black frame, or when the border is rejected by method 2. user provided permanent row applies.

Once letterbox border is identified:

5. Any pixel of the letterbox whose brightness and colour is below a set threshold is cleaned to pure black.
   This threshold is to protect cases where there are intentional items in the border such as the opening of 173295 / 57810.
   ⠀
   This is protected by a Morpho.minimum() to make the cleaning stay further away from intentional items.
   If the source contains very heavy noise or chroma noise, the threshold for this needs to be increased.

6. We don't want to eliminate the noise in a pure black screen for multiple reasons.
   First, the video is still going and it shouldn't just be completely blank.
   Second, there are situations such as fading. When the image is fading to black, there is noise during the fading. But when the fading ends, the letterbox detection triggers, and suddenly all the noise goes away within the time of a single frame. That'll be really odd.
   ⠀
   Instead a protection is applied and the letterbox cleaning strength is reduced as the area of the letterbox increases until it reaches a full black screen where no cleaning is applied.
   ⠀
   As an exception, the cleaning will still apply to the user provided permanent letterbox in a pure black screen.

7. border_y, border_u, border_v will be applied to the image clip with letterbox cropped away.

Reference

clean_letterbox(
    clip:            vs.VideoNode,

    # Threshold in method 5.
    # Full cleaning is applied when transformed pixel value <= thr - transition
    # No cleaning is applied when transformed pixel value > thr
    thr:             float     = 0.030
    transition:      float     = 0.015

    # Permanent letterbox used in method 3. and 4., as well as in method 6.
    permanent:       list[int] = [0, 0], # [Top, Bottom]

    # Enables the detection, without which only method 5. and 7. will apply  
    dynamic:         bool      = True,
    # Method 2.
    dynamic_ref:     Callable[[vs.VideoNode], vs.VideoNode]
                               = ExKirsch().edgemask,
    # Method 2.
    dynamic_ref_thr: float     = 2/3,

    # Method 6.
    # The cleaning strength starts reducing when the area of the image is smaller than this
    # threshold.
    fullblack_thr:   float     = 1/5,

    # Method 7.
    # Example function:
    # lambda clip: clip.bore.SinglePlane(top=2, bottom=2)
    border_y:        Callable[[vs.VideoNode], vs.VideoNode] | None
                               = None,
    border_u:        Callable[[vs.VideoNode], vs.VideoNode] | None
                               = None,
    border_v:        Callable[[vs.VideoNode], vs.VideoNode] | None
                               = None,
)

letterbox_mask(
    clip:            vs.VideoNode,

    # Method 1., 2., 3., 4., and 6. applies

    # Permanent letterbox used in method 3. and 4., as well as in method 6.
    permanent:       list[int] = [0, 0], # [Top, Bottom]

    # Method 2.
    dynamic_ref:     Callable[[vs.VideoNode], vs.VideoNode]
                               = ExKirsch().edgemask,
    # Method 2.
    dynamic_ref_thr: float     = 2/3,

    # Method 6.
    # The cleaning strength starts reducing when the area of the image is smaller than this
    # threshold.
    fullblack_thr:   float     = 1/5,
)

find_letterbox(
    clip:            vs.VideoNode,

    # Method 1., 2., 3., 4. applies
    # Outputs `VSLETTERBOX_TOP_ROW` and `VSLETTERBOX_BOTTOM_ROW` frame properties marking the first
    # and last row of the image, both inclusive (of the image)

    # Permanent letterbox used in method 3. and 4., as well as in method 6.
    permanent:       list[int] = [0, 0], # [Top, Bottom]

    # Method 2.
    dynamic_ref:     Callable[[vs.VideoNode], vs.VideoNode]
                               = ExKirsch().edgemask,
    # Method 2.
    dynamic_ref_thr: float     = 2/3,
)