Scan stops after 15 / 45 frames for Gopro 7 videos (both AVC and HEVC in .mp4)

[dany123]

I just found out yesterday about this awesome tool and manged to scan some hours of video from my camera, but I've noticed that any clip from my gopro fails "succesfully" after either 15 or 45 frames (I think it depends on original video framerate) and I'm trying to figure out why.
If I use handbrake to convert the video (or even ffmpeg to trim with -c copy) the scan works as expected. Also tried with both H264 and H265 (in camera) and short/long videos and it also fails in ~1 second.

I recorded a 10s sample if anyone wants to check it out: https://we.tl/t-p95xvJWT5u
Any idea how to debug this?

E:\_media\GoPro 7D>dvr-scan -i "E:\_media\GoPro 7D\GX014620 - 10s dvr-scan test.MP4" -so
[DVR-Scan] Initializing scan context...
[DVR-Scan] Opened video GX014620 - 10s dvr-scan test.MP4 (1920 x 1080 at 59.940 FPS).
[DVR-Scan] Appended video GX014620 - 10s dvr-scan test.MP4.
[DVR-Scan] Processed:   0%|                                 | 0/596 [00:00<?, ? frames/s][DVR-Scan] Scanning input video for motion events...
[DVR-Scan] Processed:   8%|█▊                      | 45/596 [00:00<00:11, 49.77 frames/s]
[DVR-Scan] Processed 45 frames read in 0.9 secs (avg 49.3 FPS).
[DVR-Scan] No motion events detected in input.

Mediainfo:

General
Complete name                            : E:\_media\GoPro 7D\GX014620 - 10s dvr-scan test.MP4
Format                                   : MPEG-4
Format profile                           : Base Media / Version 1
Codec ID                                 : mp41 (mp41)
File size                                : 15.2 MiB
Duration                                 : 10 s 10 ms
Overall bit rate                         : 12.8 Mb/s
Encoded date                             : UTC 2021-09-20 00:51:16
Tagged date                              : UTC 2021-09-20 00:51:16

Video
ID                                       : 1
Format                                   : HEVC
Format/Info                              : High Efficiency Video Coding
Format profile                           : [email protected]@Main
Codec ID                                 : hvc1
Codec ID/Info                            : High Efficiency Video Coding
Duration                                 : 9 s 943 ms
Bit rate                                 : 12.6 Mb/s
Width                                    : 1 920 pixels
Height                                   : 1 080 pixels
Display aspect ratio                     : 16:9
Frame rate mode                          : Constant
Frame rate                               : 59.940 (60000/1001) FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Bits/(Pixel*Frame)                       : 0.101
Stream size                              : 14.9 MiB (98%)
Title                                    : GoPro H.265
Language                                 : English
Encoded date                             : UTC 2021-09-20 00:51:16
Tagged date                              : UTC 2021-09-20 00:51:16
Color range                              : Full
Color primaries                          : BT.709
Transfer characteristics                 : BT.709
Matrix coefficients                      : BT.709
Codec configuration box                  : hvcC

Audio
ID                                       : 2
Format                                   : AAC LC
Format/Info                              : Advanced Audio Codec Low Complexity
Codec ID                                 : mp4a-40-2
Duration                                 : 9 s 963 ms
Bit rate mode                            : Constant
Bit rate                                 : 192 kb/s
Nominal bit rate                         : 48.0 kb/s
Channel(s)                               : 2 channels
Channel layout                           : L R
Sampling rate                            : 48.0 kHz
Frame rate                               : 46.875 FPS (1024 SPF)
Compression mode                         : Lossy
Stream size                              : 230 KiB (1%)
Title                                    : GoPro AAC  
Language                                 : English
Encoded date                             : UTC 2021-09-20 00:51:16
Tagged date                              : UTC 2021-09-20 00:51:16

Other #1
ID                                       : 3
Type                                     : Time code
Format                                   : QuickTime TC
Duration                                 : 9 s 943 ms
Bit rate mode                            : Constant
Frame rate                               : 59.940 (60000/1001) FPS
Time code of first frame                 : 00:52:05:14
Time code, striped                       : Yes
Title                                    : GoPro TCD  
Language                                 : English
Encoded date                             : UTC 2021-09-20 00:51:16
Tagged date                              : UTC 2021-09-20 00:51:16

Other #2
Type                                     : meta
Duration                                 : 10 s 10 ms
Bit rate mode                            : Variable

Other #3
Type                                     : meta
Bit rate mode                            : Variable
mdhd_Duration                            : 9943

[Breakthrough]

I'm not sure why the video is like that, but by remuxing it with ffmpeg I was able to get DVR-Scan to process it. You can do:

ffmpeg -i "GX014620 - 10s dvr-scan test.MP4" -c:v copy output.mp4

And running DVR-Scan on output.mp4. Does this help? Do you have any more samples, and does this fix them as well?

Thank you!

[Breakthrough]

Sorry I totally overlooked that you already mentioned this worked - I was able to narrow this down to how OpenCV's VideoCapture processes the video though. It seems to work fine with PyAV.

This should be improved once DVR-Scan has better integration with PySceneDetect. Specifically take a look at:
Breakthrough/PySceneDetect#213

In the meantime I'll try to have a further look with ffprobe to see if anything else jumps out. I don't have a Gopro myself so cannot test on my end, but if you have any other samples or any other info that would be helpful.

Does it happen with all videos? Just ones under a certain length? What other options can you configure in the camera besides h264/h265? (Is there a keyframe interval?)

I did notice decoding the video with PyAV works, although there is a long run at ~frame 60 without any video frames in the encoded video. Not sure why that is just yet, still investigating.

Thanks!

[dany123]

Hey, thanks for having a look.
This happens with all videos I've quickly tested (from 10seconds to 20 mins, different filesizes). One quick theory I can think of would be the extra data/info streams in the original video (I did a ffprobe and it complains about streams 2,3,4 having no supported codec) get discarded when re-encoding so maybe that's why the converted videos work? I don't know how to copy them individually to give this test a try.

There aren't many options regarding encoding in camera, only a AVC/HEVC toggle. I can also change between PAL/NTSC but I don't think framerate matters.

[dany123]

One interesting thing I just noticed, the 45 frames it shows to succesfully read are actually processed and if there's motion it creates an output video.

Hmm, well I recorded some tests with random settings and found some weird results:
-for very short videos @30/60 fps (1-6 seconds) it seems to succesfully scan no matter the resolution (tested 720p60, 1080p30 and 4k60)
-exception: 1080p240 fps -> it only scanned 225 frames -even if it's only 3.6s, stopped at 225/880
-but there's no hard limit of 225, it scanned 265/265 frames for a 4k60 video of 4.4s and 305/305 of 720p60 of 5.9s

ffprobe still shows same unsupported codec warning for the extra streams even for the short videos that sucesfully get scanned.

[Breakthrough]

Hmm I see, thanks for the feedback and the sample. I'll see if I can come up with a workaround using OpenCV alone, because if not then it might be worth filing a bug with them as well. I'm sure it would be worth trying to replicate the issue in C++ and seeing if it's possible to get additional logging statements to be printed from ffmpeg in debug mode (unfortunately I don't have the capacity right now or I would try this myself).

I will leave this open until this has been fixed in DVR-Scan, and hopefully in the meantime I can see why OpenCV stops processing the video as a workaround. Thanks for the report.

[Breakthrough]

@dany123 Sorry I seem to have misplaced the original file you sent, and the link has expired. Would you be able to upload it again?

You can now attach files directly to Github issues in comments, so if possible, if you could attach it directly to this issue, that would be great. Thank you!

[dany123]

It seems there's a 10MB limit for attaching files and the video is 15MB, I'll reupload to wetransfer: https://we.tl/t-uuD5Dg3oTK

[Breakthrough]

Interesting, so it seems that the 45th frame fails to be read by OpenCV. DVR-Scan interprets this as the end of the video and thus quits. I think this is also a bug in PySceneDetect as well which could explain some issues there too, so thank you for the sample.

I'll have to update the scanning logic to allow some dropped frames, after which emitting a warning indicating the issue (and that the video should be remux'd). I still have to run both copies through ffmpeg, since now I'm curious if it correctly decodes that frame after running it through ffmpeg, or it's just discarded.

If the latter then it implies some kind of corruption/missing data rather than something that can be fixed.

Edit: Seems to be no difference when I extract all of the frames using ffmpeg before/after remuxing, so it seems this only fails with OpenCV currently. Thus I'll be recommending the workaround you suggested to just use ffmpeg -c:v copy if this is detected at runtime.

[Breakthrough]

Now, we allow up to 5 frames to fail decoding in a row (just a value I thought would be reasonable for most purposes). This allows the video you posted to be processed as-is.

Since this can potentially lead to some issues if this happens too frequently, if any more than 1 frame fails to decode throughout the video, the following message will now be displayed including the number of frames that failed to decode:

Failed to decode 123 frame(s) from video, result may be incorrect. Try re-encoding or remuxing video (e.g. ffmpeg -i video.mp4 -c:v copy out.mp4). See https://github.com/Breakthrough/DVR-Scan/issues/62 for details.

Fix will be included in v1.4, thanks for the report.

[Breakthrough]

One last question for you @dany123 - would it be alright to include this file in both DVR-Scan and PySceneDetect's test cases? I would like to use this as an example and distribute it with the project if possible.

Thank you!

[dany123]

Sure, I have no problem with that.

Thank you for the fix!

[frankpatrick]

Having this issue with GoPro4 videos. Videos are 12 minutes long but only the first 11 frames are processed:

`[DVR-Scan] DVR-Scan v1.5.1
[DVR-Scan] Initializing scan context...
[DVR-Scan] Opened video GP175914.mp4 (3840 x 2160 at 14.985 FPS).
[DVR-Scan] Scanning input video for motion events...
Detected: 1 | Progress: 0%| | 11/10640 [00:03<50:13, 3.53 frames/s]
[DVR-Scan] Processed 11 frames read in 3.4 secs (avg 3.2 FPS).
[DVR-Scan] Detected 1 motion events in input.
[DVR-Scan] List of motion events:

| Event # | Start Time | Duration | End Time |

| Event 1 | 00:00:00.1 | 00:00:00.7 | 00:00:00.7 |

[DVR-Scan] Comma-separated timecode values:
00:00:00.067,00:00:00.734
[DVR-Scan] Motion events written to disk.`

[frankpatrick]

INFO: controller._init_dvr_scan(): DVR-Scan v1.5.1
DEBUG: controller._init_dvr_scan(): User config file not found.
DEBUG: controller._init_dvr_scan(): Parsing program options.
INFO: scanner.init(): Initializing scan context...
INFO: video_joiner._load_input_videos(): Opened video GP175914.mp4 (3840 x 2160 at 14.985 FPS).
DEBUG: scanner.scan_motion(): Using detector MOG2 with params: kernel_size = 7
INFO: scanner.scan_motion(): Scanning input video for motion events...
Detected: 1 | Progress: 0%| | 11/10640 [00:02<47:57, 3.69 frames/s]
INFO: scanner._post_scan_motion(): Processed 11 frames read in 3.6 secs (avg 3.0 FPS).
INFO: scanner._post_scan_motion(): Detected 1 motion events in input.
INFO: scanner._post_scan_motion(): List of motion events:

| Event # | Start Time | Duration | End Time |

| Event 1 | 00:00:00.1 | 00:00:00.7 | 00:00:00.7 |

INFO: scanner._post_scan_motion(): Comma-separated timecode values:
00:00:00.067,00:00:00.734
INFO: scanner._post_scan_motion(): Motion events written to disk.

[Breakthrough]

@frankpatrick try processing the video using ffmpeg or mkvmerge first to re-mux the container.