Extracting NALs from Moonfire for use with RTSP playback (ONVIF Profile G)

[RogerHardiman]

Hi
I'm writing to discuss the steps for me to extract individual NALs from Moonfire instead of a .mp4 export. The websocket API may work for me.

I'm the lead developer of RPOS, the Raspberry Pi ONVIF Server. Turns a Pi (and other Linux boxes) into a CCTV Camera with ONVIF library and RTSP server and special support for Pi cameras and Pi Pan/Tilt HATs

A common feature request is to add recording to RPOS, so I went looking for an NVR project.

ONVIF replay uses RTSP to allow a viewer to replay video from an earlier time.
It does this by adding some Headers to the RTSP messages to specify the replay start time and replay speed (and direction, eg fast forward 8x or rewind 2x)

So I was looking around for an open source NVR that has a nice API to let me extract NALs or GOPs from any point in time so I can feed the NALs into a RTSP server at correctly timed intervals. I've had a look at the WebSocket API docs.

Recording

On the recording side, RPOS makes the Pi camera available as an RTSP stream, so Moonfire could record my own RTSP stream.

Replay

I need to modify the RPOS RTSP server code to pick out replay times but then I need to hook into Moonfire to get the NALs at a specific timestamp (eg if just getting the keyframes for fast forwards/rewind) or getting a block of NALs for normal speed replay)

So it looks like I may be able to use the exisiting APIs and web socket to get most (if not all) of the NALs how I need them.

Next steps

So I wanted a reality check on existing Moonfire APIs or what extra APIs I may need to implement.

Then I can modify my RTSP server to hook into the Moonfire APIs and implement any Moonfire changes in a fork as needed.

So does this sound feasible?
And keeps a nice clean API beween moonfire and the RTSP server stuff.

Thanks
Roger Hardiman

Open source projects where I've ended up as lead developer
ONVIF Client - https://github.com/agsh/onvif
ONVIF Server - https://github.com/BreeeZe/rpos
RTSP Client and Server Libs - https://github.com/ngraziano/SharpRTSP
RTSP Client - https://github.com/mbullington/yellowstone

[scottlamb]

So overall, yes, this is feasible.

Starting from what I've seen cameras offer. As you may already know, typically the dependency structure is each frame is a reference frame for the following one, so that if there are 30 frames in a Group of Pictures, to decode the 30th, you need all the previous ones. But some cameras have a "temporal SVC" option in which they have an alternate structure in which say half the frames aren't reference frames at all, so they can just be discarded without giving up the ability to decode the other half. Sometimes even there's another level in which say frames that are a multiple of 4 don't depend on any frames that aren't. Here's my crude attempt at a diagram:

typical:

I0 <- P1 <- P2 <- P3 <- P4 <- P5 <- P6  ...

1-level SVC

I0 <------- P2 <------- P4 <------- P6  ...
 \--- P1     \--  P3    \--   P5 

2-level SVC

I0 <------------------- P4 <----------  ...
 \--------- P2 <--------|\--------- P6  ...
 \--- P1     \--  P3    \--   P5 

In the first case to decode frame 5, you need to get/decode frame 0, 1, 2, 3, and 4.

In the 1-level SVC case to decode frame 5, you need to get frames 0, 2, 4, and 5. You can skip 1 and 3.

In the 2-level SVC case to decode frame 5, you need to get frames 0, 4, and 5 only. You can skip 1, 2, and 3.

Moonfire's generally written to support pre-existing closed source cameras, but if you control the camera / H.264 encoder, in theory you can structure the reference frames however you want to optimize for the ability to do this fast forward/rewind stuff.

Currently Moonfire stores in its SQLite database whether an access unit (a group of NALs with the same timestamp that represent the same field, and field=frame unless using interlaced video) is an IDR frame or not. It doesn't track the full dependency structure now, but it could do so in a future revision of the database schema. So in short, within Moonfire's code we can pick out the frames of interest.

Now there's no API for this today, but we could add all kinds of things, e.g.:

• give me an .mp4 file for timestamps t–u, with just the IDR frames
• give me an .mp4 file ... with at most/least f fps (picking out some subset of frames depending on the reference structure)
• give me an MPEG-TS stream ...
• give me an RTP stream ...
• tell me the timestamp/sizes/reference structure of each frame matching my criteria, without actually sending me the video sample data...

In particular, variations of .mp4 files are currently the only way Moonfire sends video out, but as discussed a bit in #203 recently, other stuff is possible. That might be more convenient for you than implementing .mp4 demuxing on the client side.

The biggest constraint is my limited time. If you're interested in contributing to the Moonfire codebase, anything is possible. We could even add a v4l2 video source (in addition to RTSP) and an RTSP server, such that Moonfire becomes another of / replaces the other RTSP server options mentioned in your project's README.

[RogerHardiman]

Hi
Thanks for the reply.
I'd not considered SVC when it comes to replay before.
I was expecting to start simple with the following design...
For normal playback, I'd ask for all the NALs in a GOP and get a series of Data Packets of NALs and Timestamps over the web socket.
I'd then ask for the next GOP that follows.

For fast forwards and re-wind, I'd ask for just an IDR frame, and then skip forwards and backwards through IDR frames.
Just showing IDRs is cheap (CPU time) and simple (coding time) although the commercial VMS systems I use have lovely smooth fast forwards and fast rewind, but I'm just starting with IDRs.

Having raw NALs over a web socket sounds would work for me. No need to parse .mp4 files or parse Transport Streams.
Although I don't think parsing a TS is too hard. I've had a look at the payload format.

Having limited time on your side is fine.
I was expecting to have a go at this myself and just ask questions and then submit Pull Requests back to you for review and comment.

[scottlamb]

👍 At a high level, that sounds very reasonable to me. Maybe the PR can start with just the ref/api.md changes to work out the precise API?