Getting started with mediasoup-client-aiortc

mediasoup-demo
1

Hi,

I am very new to Mediasoup, WebRTC and Javascript, hence please bear with me if the question is too basic. Would appreciate if you can please point me where to look.

I am experienced in Python, and want to use it do some image processing before streaming the video. I looked at the mediasoup-demo (which I can run), but now want to use mediasoup-client-aiortc to insert a Python worker that does the image processing.

However I am unsure how to start. If I look at the aiortc subdirectory in mediasoup-demo, it seems to have most of the elements except it does not have a client html/js file. Is that something I should write on my own, or modify the files under app/ to use aiortc client?

I tried running the script “npm run dev” inside aiortc/ subdirectory. My camera light turns on but the Websocket connection to the server (running locally under mediasoup-demo/server) fails to connect. Note that running “npm start” under app/ (as described in README) is able to connect to same server.

Here is the dump:

mediasoup-demo-aiortc starting mediasoup-demo-aiortc: {
mediasoup-demo-aiortc “roomId”: “devel”,
mediasoup-demo-aiortc “peerId”: “mediasoup-client-aiortc-id”,
mediasoup-demo-aiortc “displayName”: “mediasoup-client-aiortc”,
mediasoup-demo-aiortc “useSimulcast”: false,
mediasoup-demo-aiortc “useSharingSimulcast”: false,
mediasoup-demo-aiortc “forceTcp”: false,
mediasoup-demo-aiortc “produce”: true,
mediasoup-demo-aiortc “consume”: true,
mediasoup-demo-aiortc “forceH264”: false,
mediasoup-demo-aiortc “forceVP8”: false,
mediasoup-demo-aiortc “datachannel”: true,
mediasoup-demo-aiortc “externalAudio”: “”,
mediasoup-demo-aiortc “externalVideo”: “”
mediasoup-demo-aiortc } +0ms
mediasoup-demo-aiortc:RoomClient constructor() [roomId:“devel”, peerId:“mediasoup-client-aiortc-id”, displayName:“mediasoup-client-aiortc”, device:aiortc] +0ms
terminal> mediasoup-client:aiortc createWorker() +0ms
mediasoup-client:aiortc:Worker constructor() [logLevel:‘warn’] +0ms
mediasoup-client:aiortc:Worker spawning worker process: python3 -u <path/to>/mediasoup-demo/aiortc/node_modules/mediasoup-client-aiortc/worker/worker.py --logLevel=warn +1ms
mediasoup-client:aiortc:Channel constructor() +0ms
mediasoup-client:aiortc:Worker worker process running [pid:20700] +380ms
protoo-client:WebSocketTransport constructor() [url:wss://127.0.0.1:4443/?roomId=devel&peerId=mediasoup-client-aiortc-id, options:undefined] +0ms
protoo-client:WebSocketTransport _runWebSocket() [currentAttempt:1] +1ms
protoo-client:Peer constructor() +0ms
protoo-client:ERROR:WebSocketTransport WebSocket “error” event +0ms
protoo-client:WARN:WebSocketTransport WebSocket “close” event [wasClean:false, code:1006, reason:“connection failed”] +0ms
protoo-client:Peer emit “failed” [currentAttempt:1] +29ms
mediasoup-demo-aiortc:ERROR:RoomClient WebSocket connection failed +0ms

Thanks!

2
  • server/ has the backend code (the “server”).
  • app/ is the web application (it uses React, mediasoup-client, etc).
  • aiortc/ is a Node client (not web but Node client) that uses mediasoup-client-aiortc to connect to the server.

So you are mixing too many things here.

First, if you have WebSocket connection problems that may be related to a wrong TLS certificate. The README says that a valid TLS certificate must be used. So first, try with just the server and the web app.

mediasoup-client-aiortc is a Node library. It just happens that it “executes” Python internally to communicate with an aiortc process, but I don’t think you will be able to add Python code externally. That would probably require changes in mediasoup-client-aiortc.

3

Regarding the Websocket connection, make sure you are connecting to the correct port. The WebSocket handling in client side is the same as in the app (demo application for mediasoup-client).

Regarding the image processing prior to streaming, mediasoup-client-aiort accepts the same inputs as PyAv does, AFAIK those are: URL, physical device, file:

4

Thanks for the clarifications!

I suspect that is related to the certificate. When I try connecting to test.mediasoup.org with the aiortc client, it seems to work.

For my local machine, I was using a self-signed certificate and I see posts where you have mentioned that a self-signed certificate may not work. However, I searched around a bit and found some posts explaining how to create a valid certificate for localhost (using the correct IP and adding it to the keychain on my laptop). By doing that I can connect the app/ client to the server/ without needing to bypassing any security warnings (i.e. I get the lock icon in the browser instead of “not secure” warning). However, the aiortc client still does not connect to it.

So IIUC, there is no particular benefit then to using mediasoup-client-aiortc for my scenario where I want to use Python processing in the client before sending the output to the server?

The other route I am exploring is to use the C++ client and do the processing in C++ before sending/broadcasting. Would that be a better choice?

5

I am using 4443 as the port. I essentially sat the HOSTNAME to localhost in the package.json “dev” script, which then affects the port selection in urlFactory I believe.

I am not familiar with PyAV but will take a look.

I was thinking there might be a way to customize the worker.py used in mediasoup-client-aiortc (or ideally be able to supply an external py file that the worker can call), and do the processing in it before outputting the video track.

Thanks!

6

Valid certificate is a valid certificate. If you use a self-signed certificate then you must tell your whole OS to trust it. We use Letsencrypt real and valid certificates.

If you figure out how to do such a processing using aiortc, then we may consider adding some API/stuff into mediasoup-client-aiortc to allow custom Python code injection or whatever.

7

Hi All,

Do we have any update on this one ? “we may consider adding some API/stuff into mediasoup-client-aiortc to allow custom Python code injection or whatever.”

If not yet, how can I modify python /worker folder so that I can add implementation to python code ? Do I need to build repo manually for that?
I dont think npm install --save mediasoup-client-aiortc is a way to go for my case which require custom python implementation ?

Thanks in advance