I am experienced in Python, and want to use it do some image processing before streaming the video. I looked at the mediasoup-demo (which I can run), but now want to use mediasoup-client-aiortc to insert a Python worker that does the image processing.
However I am unsure how to start. If I look at the aiortc subdirectory in mediasoup-demo, it seems to have most of the elements except it does not have a client html/js file. Is that something I should write on my own, or modify the files under app/ to use aiortc client?
I tried running the script “npm run dev” inside aiortc/ subdirectory. My camera light turns on but the Websocket connection to the server (running locally under mediasoup-demo/server) fails to connect. Note that running “npm start” under app/ (as described in README) is able to connect to same server.
app/ is the web application (it uses React, mediasoup-client, etc).
aiortc/ is a Node client (not web but Node client) that uses mediasoup-client-aiortc to connect to the server.
So you are mixing too many things here.
First, if you have WebSocket connection problems that may be related to a wrong TLS certificate. The README says that a valid TLS certificate must be used. So first, try with just the server and the web app.
mediasoup-client-aiortc is a Node library. It just happens that it “executes” Python internally to communicate with an aiortc process, but I don’t think you will be able to add Python code externally. That would probably require changes in mediasoup-client-aiortc.
I suspect that is related to the certificate. When I try connecting to test.mediasoup.org with the aiortc client, it seems to work.
For my local machine, I was using a self-signed certificate and I see posts where you have mentioned that a self-signed certificate may not work. However, I searched around a bit and found some posts explaining how to create a valid certificate for localhost (using the correct IP and adding it to the keychain on my laptop). By doing that I can connect the app/ client to the server/ without needing to bypassing any security warnings (i.e. I get the lock icon in the browser instead of “not secure” warning). However, the aiortc client still does not connect to it.
So IIUC, there is no particular benefit then to using mediasoup-client-aiortc for my scenario where I want to use Python processing in the client before sending the output to the server?
The other route I am exploring is to use the C++ client and do the processing in C++ before sending/broadcasting. Would that be a better choice?
I am using 4443 as the port. I essentially sat the HOSTNAME to localhost in the package.json “dev” script, which then affects the port selection in urlFactory I believe.
I am not familiar with PyAV but will take a look.
I was thinking there might be a way to customize the worker.py used in mediasoup-client-aiortc (or ideally be able to supply an external py file that the worker can call), and do the processing in it before outputting the video track.
Do we have any update on this one ? “we may consider adding some API/stuff into mediasoup-client-aiortc to allow custom Python code injection or whatever.”
If not yet, how can I modify python /worker folder so that I can add implementation to python code ? Do I need to build repo manually for that?
I dont think npm install --save mediasoup-client-aiortc is a way to go for my case which require custom python implementation ?