mediasoup library in Rust

mediasoup libraries
1

So yeah, time to make an announcement.

Over last few weeks I’ve been slowly working on Mediasoup library in Rust.

Server, not client. It communicates with the same unmodified C++ worker as TypeScript library, but from Rust environment.

This is still very early in development, but I figured it would be nice to present minimal implementation and get some feedback from the community.

What I have right now:

  • Worker Manager (holds async executor and allows creating Workers, API is almost feature-complete)
  • Worker (primarily used to create routers, API is almost feature-complete)
  • WebRTC transport (can only create consumers and producers, there is still significant portion of API that is not covered)
  • Consumer (API is almost feature-complete, Direct transport-related things are, obviously, missing)
  • Producer (API is almost feature-complete, Direct transport-related things are, obviously, missing)
  • Way more extensive typing for everything (down to mime types), the only place where Any is used is AppData attached to instances similarly to TypeScript implementation
  • Only basic smoke tests that make sure C++ worker understands what library sends and library understands what C++ worker responds
  • Only Linux support (may work on macOS, will not work on Windows for now)
  • 6473 LOC
  • Over 70 unresolved TODOs :innocent:

Where to find it:

Plans:

  • Obviously implement missing functionality
  • More typing, more idiomatic Rust in the API
  • Replicate tests from TypeScript library and maybe add some more
  • Import docs from mediasoup.org such that documentation on docs.rs is self-contained and has lots of examples
  • Upstream this as one of the officially supported libraries
7 Likes
2

Tks for sharing. What’s the motivation on having it written on Rust?

3

There are a few:

  1. I’m writing most of new apps in Rust and there was no SFU library like Mediasoup available while interacting with C++ libraries is painful and dangerous in general
  2. This paves the way towards maybe having Mediasoup worker in Rust instead of C++ at some point
  3. It is an interesting learning experience, since the library will be quite large when it is done (it is hard to call tiny already)
1 Like
4

Amazing! Thanks for this!

I’ve just tweeted it: https://twitter.com/ibc_tw/status/1310524027308515330

3 Likes
5

Published 0.0.2.

It includes DataProducer/Consumer support (except DirectTransport and DataConsumer.send() method) and PlainTransport alongside some other fixes and improvements.

The remaining 2 transports are most likely not as popular, so next candidate for implementation is RtpObserver.

Also I could use some help from the community, like porting h264-profile-level-id package to Rust, that is not done yet and is a sizable piece of code on its own. If anyone is interested, I can help you get your feet wet :wink:

6

Would it be easier by using existing C++ code? https://chromium.googlesource.com/external/webrtc/+/refs/heads/master/media/base/h264_profile_level_id.h

7

FFI requires unsafe blocks and careful API design around memory management. Potential savings are much less than drawbacks in this case I think.

For me at least it would be much more productive to rewrite JS version to idiomatic Rust than to complicate build process by including C++ dependency just for this.
Also my C++ knowledge is very limited, which is also a contributing factor here for sure.

8

Published 0.0.5.

Didn’t have much time lately, but added AudioLevelObserver and minor improvements like ability to remove event handlers using RAII (this part was later refactored into a separate crate event-listener-primitives that some may find useful).

Missing transports will require quite a bit more work, but I’ll get to them, hopefully soon.

9

Published 0.0.6.

It has DirectTransport, PipeTransport and previously lacking methods on other entities related to those transports.

I believe the whole API surface should now be covered.

Next steps:

  • fix about 50 TODOs
  • port h264-profile-level-id (one of the major TODOs)
  • more docs
  • more tests
  • some refactoring
  • UPD: Ah, yes, Windows support would be nice too
4 Likes
10

Amazing

11

Published 0.0.7.

Ported AudioLevelObserver, Consumer and Producer tests.

There were a few minor bugs that were found and fixed thanks to tests.

Also added heavier typing, such that more things can be checked at compile time, for instance this is how RTP codec capability can be specified:

RtpCodecCapability::Video {
    mime_type: MimeTypeVideo::VP8,
    preferred_payload_type: None,
    clock_rate: NonZeroU32::new(90000).unwrap(),
    parameters: BTreeMap::new(),
    rtcp_feedback: vec![
        RtcpFeedback::Nack,
        RtcpFeedback::NackPli,
        RtcpFeedback::TransportCC,
    ],
}

What is not possible here comparing to TypeScript (and thus doesn’t need and can’t have corresponding tests):

  • not possible to specify Mime Type that is not of a video kind
  • not possible to specify physically invalid clock rate (even though it is possible to specify incorrect one, I’m thinking whether it makes sense to hide it in video altogether)
  • RTCP feedback can only be set to something that mediasoup is aware of

Yet when serialized to JSON it looks identical to this TypeScript version:

{
    mimeType    : 'video/VP8',
    clockRate   : 90000,
    rtcpFeedback :
    [
        { type: 'nack' },
        { type: 'nack', parameter: 'pli' },
        { type: 'transport-cc' }
    ]
}

This is in contrast to JavaScript tests that have to verify how library/worker behave with kind : 'chicken' specified as producer kind :slightly_smiling_face:.

There are many more examples of this, but you get the idea.

Tests proven to be time consuming, but mostly just mechanical work, I’m getting there.

3 Likes
12

Published 0.0.9.

h264-profile-level-id is finally ported to Rust and integrated into mediasoup library with corresponding tests unlocked and TODOs removed.

Router, Worker and ortc-related tests are all ported as well with a bit of types refinement and minor bug fixes, useful additional derives and other improvements.

transport_close, router_close and worker_close events are added to corresponding entities that were previously missing (only useful if worker has died, otherwise will never be triggered, but I had to add them for completeness) with corresponding tests.

Next steps:

  • port DataProducer, DataConsumer and *Transport tests
  • more docs (including useful Rust-specific readme)
  • fix about 25 TODOs:
    • mostly type refinements for info field in trace events, currently their type is equivalent to TS’s object/any because I was lazy and had to type a ton of untyped (in TypeScript) data structures based on actual C++ implementation (which, strictly speaking, is not typed on API level either and is assembled JSON object on the fly piece by piece instead)
    • questions to myself and in general
    • potential, but not critical improvements
  • not much refactoring, but probably some
  • Windows support would be nice too, but low priority for me personally (contributions are welcome though)

I don’t expect major API changes at this point, as soon as all tests are ported 0.1.0 will be published.

2 Likes
13

Published 0.0.11.

All of the remaining tests are ported from TypeScript, there were small bugs with basically every transport making them unusable, but those are now resolved and I’m pretty optimistic about the state overall.

There are a few issues that cause tests to stall/fail occasionally and sometimes leak worker processes. Those are important to fix with upcoming updates.

Library approaches Beta quality now.

14

Amazing @nazar-pc!

Congratulations!

15

Published 0.1.0 with a bunch of fixes as there are no issues I’m aware of.

Next steps:

  • more docs (including useful Rust-specific readme)
  • fix about 25 TODOs:
    • mostly type refinements for info field in trace events, currently their type is equivalent to TS’s object/any because I was lazy and had to type a ton of untyped (in TypeScript) data structures based on actual C++ implementation (which, strictly speaking, is not typed on API level either and is assembled JSON object on the fly piece by piece instead)
    • questions to myself and in general
    • potential, but not critical improvements
  • Windows support would be nice too, but low priority for me personally (contributions are welcome though)

:rocket:

1 Like
16

After many hours of copying, pasting, writing and editing again, most of useful documentation is pulled in into source code for nice contextual documentation during autocomplete and auto-generated on docs.rs.

Everything should be documented unless I missed something: https://docs.rs/mediasoup

It should be usable and feature-complete at this point. There are always things to improve, but that process never ends.

Should I open a PR against upstream now?
It is just 234 commits with 22,448 additions and 99 deletions :upside_down_face:

17

Is it ok to talk about all this on first week of January? I’m having more things to do than expected these days. Also, we are gonna work on having our own bandwidth estimator system in mediasoup (get rid of libwebrtc completely) and that’s gonna be our main and mostly exclusive task during next months.

18

Yes, first week on January should work.

News on removing libwebrtc as a dependency sound awesome!

19

And to give some more concrete examples to play with I’ve created the first example app with Rust version of the mediasoup library: https://github.com/nazar-pc/mediasoup/tree/rust/rust/examples

It is just a simple echo server and client for audio+video, but will hopefully be useful for those who just getting started and as a reference point of a working example with very little going on to encourage hacking on it.

It is built with Actix, which is a good fit in this case as existence of worker/router/transports is tied to the WebSocket connection, which makes actor-based model very convenient. Real-world apps are not as simple unfortunately.

2 Likes
20

I’ve being thinking lately about the fact that in order to use Rust port user needs to build worker separately, about Windows support, inconveniences of managing subprocesses, inefficiencies coming from JSON serialization/deserialization and some other things and started wondering how to make things better.

The idea I have currently is that it would be nice to librarify mediasoup-worker minimally, such that it would be possible to use library API to create a thin wrapper and instead of running mediasoup-worker as a process with 1 thread (OK, there is also SCTP iterator thread in that process) run worker in a thread of a parent process instead. This is the first step, communication can still happen via file descriptors (whose numbers need to become configurable) and things like signal handing would need to change, but for the most part things should be possible to use as is.

Then it would be possible to create mediasoup-sys crate with thin FFI (Rust ecosystem naming convention) and make mediasoup library depend on it, such that there is no need to build worker and specify path to it manually anymore, everything can be done during dependency installation time improving usability.

There is also https://cxx.rs/ that bridges gaps between C++ and Rust, can be useful here, but may not be necessary for the first step.

1 Like