x9k3 0.1.15

HLS Segmenter with SCTE-35

Details | Install | Use | Customize | Live Events | Bugs | Feedback | Cue | Stream Diff | Sidecar SCTE35

x9k3

HLS Segmenter with SCTE-35 Baked In

• SCTE-35 Cues in Mpegts Streams are Translated into HLS tags.
• Segments are Split on SCTE-35 Cues as needed.
• M3U8 Manifests are created with SCTE-35 HLS tags.
• Supports h264 and h265 and mpeg2 video.
• Multi-protocol. Files, Http(s), Multicast, and Udp.
• Supports Live Streaming.
• Customizable Ad Break Criteria
• SCTE-35 Cues Can Now Load from a Sidecar File.

Requires

• python 3.6+ or pypy3
• threefive

pip3 install threefive

How to Use

a@debian:~/x9k3$ pypy3 x9k3.py -h

usage: x9k3.py [-h] [-i INPUT] [-o OUTPUT_DIR] [-s SIDECAR] [-l] [-d]

optional arguments:

  -h, --help            show this help message and exit

  -i INPUT, --input INPUT
                        Input source, like "/home/a/vid.ts" or "udp://@235.35.3.5:3535" or
                        "https://futzu.com/xaa.ts"

  -o OUTPUT_DIR, --output_dir OUTPUT_DIR
                        Directory for segments and index.m3u8 ( created if it does not exist )
  
  -s SIDECAR, --sidecar SIDECAR
                        sidecar file of scte35 cues. each line contains (PTS, CueString) Example:
                        89718.451333, /DARAAAAAAAAAP/wAAAAAHpPv/8=
  
  -l, --live            Flag for a live event ( enables sliding window m3u8 )
  
  -d, --delete          delete segments ( enables live mode )

Example Usage

local file as input

   python3 x9k3.py -i video.mpegts

multicast stream as input with a live sliding window

python3 x9k3.py --live -i udp://@235.35.3.5:3535

use ffmpeg to read multicast stream as input and x9k3 to segment

  with a sliding window, and  expiring old segments.
   --delete implies --live

 ffmpeg  -re -copyts -i udp://@235.35.3.5:3535 -map 0 -c copy -f mpegts - | python3 x9k3.py --delete

https stream for input, and writing segments to an output directory

  directory will be created if it does not exist.

 pypy3 x9k3.py -i https://so.slo.me/longb.ts --output_dir /home/a/variant0

using stdin as input

cat video.ts | python3 x9k3.py

load scte35 cues from a text file

Sidecar Cues will be handled the same as SCTE35 cues from a video stream.

line format for text file : pts, cue

pts is the insert time for the cue, A four second preroll is standard. 

cue can be base64,hex, int, or bytes

a@debian:~/x9k3$ cat sidecar.txt

38103.868589, /DAxAAAAAAAAAP/wFAUAAABdf+/+zHRtOn4Ae6DOAAAAAAAMAQpDVUVJsZ8xMjEqLYemJQ== 
38199.918911, /DAsAAAAAAAAAP/wDwUAAABef0/+zPACTQAAAAAADAEKQ1VFSbGfMTIxIxGolm0= 

    

pypy3 x9k3.py -i  noscte35.ts  -s sidecar.txt 

---

Details

• Segments are cut on iframes.

• Segment size is 2 seconds or more, determined by GOP size.

• Segments are named seg1.ts seg2.ts etc...

• For SCTE-35, Video segments are cut at the the first iframe >= the splice point pts.

• If no pts time is present in the SCTE-35 cue, the segment is cut at the next iframe.

• SCTE-35 cues are added when received.

• All SCTE35 cue commands are added.

# Time Signal
#EXT-X-SCTE35:CUE="/DC+AAAAAAAAAP/wBQb+W+M4YgCoAiBDVUVJCW3YD3+fARFEcmF3aW5nRlJJMTE1V0FCQzUBAQIZQ1VFSQlONI9/nwEKVEtSUjE2MDY3QREBAQIxQ1VFSQlw1HB/nwEiUENSMV8xMjEwMjExNDU2V0FCQ0dFTkVSQUxIT1NQSVRBTBABAQI2Q1VFSQlw1HF/3wAAFJlwASJQQ1IxXzEyMTAyMTE0NTZXQUJDR0VORVJBTEhPU1BJVEFMIAEBhgjtJQ==" 
#EXTINF:2.085422,
seg1.ts

SCTE-35 cues with a preroll are inserted again at the splice point

# Splice Point @ 17129.086244
#EXT-X-SCTE35:CUE="/DC+AAAAAAAAAP/wBQb+W+M4YgCoAiBDVUVJCW3YD3+fARFEcmF3aW5nRlJJMTE1V0FCQzUBAQIZQ1VFSQlONI9/nwEKVEtSUjE2MDY3QREBAQIxQ1VFSQlw1HB/nwEiUENSMV8xMjEwMjExNDU2V0FCQ0dFTkVSQUxIT1NQSVRBTBABAQI2Q1VFSQlw1HF/3wAAFJlwASJQQ1IxXzEyMTAyMTE0NTZXQUJDR0VORVJBTEhPU1BJVEFMIAEBhgjtJQ==" 
#EXTINF:0.867544,
seg2.ts

CUE-OUT ans CUE-IN are added at the splice point

#EXT-X-SCTE35:CUE="/DAxAAAAAAAAAP/wFAUAAABdf+/+zHRtOn4Ae6DOAAAAAAAMAQpDVUVJsZ8xMjEqLYemJQ==" CUE-OUT=YES
#EXTINF:1.668334,
seg13.ts

VOD

• x9k3 defaults to VOD style playlist generation.
• All segment are listed in the m3u8 file.

Live

• Activated by the --live switch or by setting X9K3.live=True

• Like VOD except:

  • M3u8 manifests are regenerated every time a segment is written.
  • Segments are generated to realtime, even if the stream isn't live. ( like ffmpeg's "-re" )
  • Sliding Window for 5 WINDOW_SLOTS
  • A cue out continue tag is added to first segment in manifest during an ad break.

#EXTM3U
#EXT-X-VERSION:3
#EXT-X-TARGETDURATION:3
#EXT-X-SCTE35:CUE="/DAxAAAAAAAAAP/wFAUAAABdf+/+zHRtOn4Ae6DOAAAAAAAMAQpDVUVJsZ8xMjEqLYemJQ==",CUE-OUT=CONT
#EXTINF:2.002,
seg43.ts
#EXTINF:2.002,
seg44.ts
#EXTINF:2.002,
seg45.ts
# Splice Insert
#EXT-X-SCTE35:CUE="/DAsAAAAAAAAAP/wDwUAAABef0/+zPACTQAAAAAADAEKQ1VFSbGfMTIxIxGolm0="
#EXTINF:2.168834,
seg46.ts
# Splice Point @ 38203.125478
#EXT-X-SCTE35:CUE="/DAsAAAAAAAAAP/wDwUAAABef0/+zPACTQAAAAAADAEKQ1VFSbGfMTIxIxGolm0=",CUE-IN=YES
#EXTINF:1.001,
seg47.ts
#EXTINF:2.836166,
seg48.ts
#EXTINF:2.002,
seg49.ts
#EXTINF:2.002,
....

Stream Diff

• stream diff is the difference between the playback time of the stream and generation of segments by x9k3.

• A segment with a 2 second duration that takes 0.5 seconds to generate would have a stream diff of 1.5.

• In the default mode, stream_diff is a benchmark of playlist generation.

a@debian:~/x9k3$ time pypy3 x9k3.py  -i local-vid.ts 
./seg0.ts	start:  3.545000	duration:  2.112000	stream diff:  2.094049
./seg1.ts	start:  5.593000	duration:  2.048000	stream diff:  4.133058
./seg2.ts	start:  7.598333	duration:  2.005333	stream diff:  6.133111
./seg3.ts	start:  9.625000	duration:  2.026667	stream diff:  8.151764
./seg4.ts	start:  11.673000	duration:  2.048000	stream diff:  10.196475
./seg5.ts	start:  13.785000	duration:  2.112000	stream diff:  12.298679
./seg6.ts	start:  15.833000	duration:  2.048000	stream diff:  14.343878

  ...
  
./seg77.ts	start:  163.011667	duration:  2.176000	stream diff:  161.307591
./seg78.ts	start:  165.187667	duration:  2.176000	stream diff:  163.482903 <-- big stream diff

real	0m0.482s             <--  fast segmenting for VOD
user	0m0.334s
sys	0m0.128s

Live stream non-live stuff

• stream_diff with --live or --delete

  • stream_diff automatically throttles non-live streams for realtime playback .
  • stream_diff keeps segmentation and the sliding window in sync.

a@debian:~/x9k3$ time pypy3 x9k3.py  -i local-vid.ts --live
./seg0.ts	start:  1.433000	duration:  2.112000	stream diff:  1.749682
./seg1.ts	start:  3.545000	duration:  2.048000	stream diff:  1.664505
./seg2.ts	start:  5.593000	duration:  2.005333	stream diff:  1.604484
./seg3.ts	start:  7.598333	duration:  2.026667	stream diff:  1.608694
./seg4.ts	start:  9.625000	duration:  2.048000	stream diff:  1.614071
./seg5.ts	start:  11.673000	duration:  2.112000	stream diff:  1.660427
./seg6.ts	start:  13.785000	duration:  2.048000	stream diff:  1.570947
./seg7.ts

...
./seg65.ts	start:  136.025000	duration:  2.005333	stream diff:  0.197508
./seg66.ts	start:  138.030333	duration:  2.069334	stream diff:  0.227942
./seg67.ts	start:  140.099667	duration:  2.112000	stream diff:  0.240508
./seg68.ts	start:  142.211667	duration:  2.026666	stream diff:  0.132231
./seg69.ts	start:  144.238333	duration:  2.026667	stream diff:  0.111408
./seg70.ts	start:  146.265000	duration:  2.005333	stream diff:  0.064351
./seg71.ts	start:  148.270333	duration:  2.026667	stream diff:  0.063869
./seg72.ts	start:  150.297000	duration:  2.112000	stream diff:  0.129556
./seg73.ts	start:  152.409000	duration:  2.005333	stream diff:  0.004607
./seg74.ts	start:  154.414333	duration:  2.133334	stream diff:  0.112022
./seg75.ts	start:  156.547667	duration:  2.069333	stream diff:  0.018838
./seg76.ts	start:  158.617000	duration:  2.218667	stream diff:  0.151273
./seg77.ts	start:  160.835667	duration:  2.176000	stream diff:  0.101823
./seg78.ts	start:  163.011667	duration:  2.176000	stream diff:  0.100369  <-- small stream diff

real	2m44.775s   <-- real time segmenting to sync live stream sliding window
user	0m0.678s
sys	0m0.169s

FAQ

Q.

How do I Customize CUE-OUT and CUE-IN ad break events?

A.

Override the X9K3.scte35.is_cue_out and X9K3.scte35.is_cue_in static methods.

The X9K3 class has three static methods you can override and customize.

@staticmethod	arg	return value	details
mk_cue_tag	cue	text	called to generate scte35 hls tags
is_cue_out	cue	bool	returns True if the cue is a CUE-OUT
is_cue_in	cue	bool	returns True if the cue is a CUE-IN

Example

---

• Override the static method X9K3.scte35.is_cue_out(cue)
• Require
  • a Splice Command of type 6, Time Signal
  • a Splice Descriptor tag of type 2, Segmentation Descriptor
  • a Segmentation Type Id of 0x22, "Break Start"

def my_cue_out(cue):
    """
    my_cue_out returns True 
    if the splice command is a time signal
    """
    if cue.command.command_type == 6: # time signal
        for d in cue.descriptors:      # cue.descriptors is always list
            if d.tag ==2:              # Segmentation Descriptor tag
                if d.segmentation_type_id == 0x22:  # Break Start
                    return True
    return False

• Create an X9K3 instance

from x9k3 import X9K3
x9 = X9K3("vid.ts")

• set is_cue_out to your custom function

x9.scte35.is_cue_out = my_cue_out
x9.decode()