Client side web element

mediasoup libraries
1

Hello, Consider i have the following prototype:

 <WhateverMediaElement> Parent Element>
    <video ... video1 style:LeftSide muted />
    <video ... video2 style:style:RightSide muted />
    <audio .... audio3 >
 </WhateverMediaElement>

And i would like to stream the whole parent element into mediasoup as 1 stream having both the video preserving their layouts, muted, and having the audio3 playing as an audiosource. Can someone guide me into what html elements to look into and how to achieve this?

thanks

2

So assuming that what you want is that you have some container and in that container there are multiple videos and an audio, in short you have a grid like layout something like that and you want to stream that layout including audio to mediasoup. You can do something like this:

Assuming the video, audio tracks are proper media stream tracks:

  • Firstly you will have to merge these multiple videos into one video stream preserving the layout:
    – Create a canvas
    – Draw the video frame by frame on that canvas
    – Capture the stream from canvas using captureStream() method.
  • And then you will add the audio track to to this one merged video stream
  • Now you can send this final stream to mediasoup just like normal stream

If video, audio tracks are not proper media stream tracks but are some urls etc then:

  • For videos it will be the same canvas process as mentioned above.
  • For audio you can use captureStream() method of HtmlAudioElement
  • The rest of the process remains the same as mentioned above.
3

Thank you for your reply. So mainly a canvas is video/audio streamable to mediasoup? I thought it is not after reading some questions in the forum

4

HtmlCanvasElement, HtmlMediaElement are the only elements (I guess) which can give you a MediaStream. This MediaStream then can be used in webrtc, it is the main sauce of it.

Canvas gives you the video stream, you can add audio stream to this video stream and then send it over to mediasoup or whatever thing you are using.

5

@OG-RTC change the topic title to ‘Capture stream from html container element and send it to mediasoup’. As this is most suitable for your question.

1 Like
6

Hello Zaid

Whats he optimal way to render the local webcam into the canvas/ We are having a huge performance compromise using the drawImage method. Any better way to include the webcam in the canvas?

7

Canvas is going to be costly, but if you are just rendering webcam to canvas then that will not be an issue for medium to high end devices. If you are doing more than that like rendering icons or text or whatever then that can cost you more.

There are some optimisations which you can do while working with canvas. Like not rendering the static content like icons and texts everytime. So you will draw this static content on separate canvas and captureImage from that canvas to your webcam canvas. This will be really quick.

Read this from mozilla dev: