MediaStreamTrack doesn't play video on HTMLVideoElement

xplatsolutions · August 13, 2020, 10:43am

Echo here from => https://stackoverflow.com/questions/63393245/mediastreamtrack-doesnt-play-video-on-htmlvideoelement

I am not sure if this is mediasoup-client specific but the forum helped me so far to learn a lot about producing and consuming video, the last part to make this work is to show the remote video to a consumer and for some reason although it looks fine I can’t seem to get the video play on HTMLVideoElement. Reposting here for the community to help with ideas if possible.

I have the below JavaScript code to set a MediaStreamTrack to MediaStream and then as srcObject to HTMLVideoElement.

let track = result.data?.track
              
    if (track.kind === 'video' && videoElement.current) {
      const videoStream = new MediaStream()
      videoStream.addTrack(track)
    
      videoElement.current.srcObject = videoStream
      videoElement.current.oncanplay = () => logger.info(`video oncanplay`)
      videoElement.current.onplay = () => logger.info(`video onplay`)
      videoElement.current.onpause = () => logger.info(`video onpause`)
    
      videoElement.current
                  .play()
                  .then(() => logger.info(`video play`))
                  .catch((error) =>
                    logger.warn(`videoElem.play() failed: ${error}`))
    }

The only logging I see from registered delegate functions is the onplay, nothing else pretty much is invoked.

The HTMLVideoElement.

 <video
      ref={videoElement}
      controls={false}
      autoPlay
      playsInline
    />

The MediaStream.

MediaStream {id: "ab29093c-085a-4f12-9590-8af635614cb8", active: true, onaddtrack: null, onremovetrack: null, onactive: null, …}
    active: true
    id: "ab29093c-085a-4f12-9590-8af635614cb8"
    onactive: null
    onaddtrack: null
    oninactive: null
    onremovetrack: null

The MediaStreamTrack.

 MediaStreamTrack {kind: "video", id: "8f13425b-c052-491f-a190-a24e2939fbf3", label: "8f13425b-c052-491f-a190-a24e2939fbf3", enabled: true, muted: false, …}
    contentHint: ""
    enabled: true
    id: "8f13425b-c052-491f-a190-a24e2939fbf3"
    kind: "video"
    label: "8f13425b-c052-491f-a190-a24e2939fbf3"
    muted: true
    onended: null
    onmute: null
    onunmute: null
    readyState: "live"

Not sure why muted=false but when I click the arrow details evaluating the object will show muted=true

Doing the same thing for an audio stream track is working fine.

Screenshot from browser logs.

xplatsolutions · August 13, 2020, 11:02am

Debugging the server peer producers and consumers it seems like two consumers are paused.

mediaserver:INFO:./Logger consumer Consumer {
  _events: [Object: null prototype],
  _eventsCount: 9,
  _maxListeners: Infinity,
  _closed: false,
  _paused: true,
  _producerPaused: false,
  _priority: 1,
  _observer: [EnhancedEventEmitter],
  _internal: [Object],
  _data: [Object],
  _channel: [Channel],
  _appData: {},
  _score: [Object],
  _preferredLayers: undefined,
  [Symbol(kCapture)]: false
} +0ms

  mediaserver:INFO:./Logger consumer Consumer {
  _events: [Object: null prototype],
  _eventsCount: 9,
  _maxListeners: Infinity,
  _closed: false,
  _paused: true,
  _producerPaused: false,
  _priority: 1,
  _observer: [EnhancedEventEmitter],
  _internal: [Object],
  _data: [Object],
  _channel: [Channel],
  _appData: {},
  _score: [Object],
  _preferredLayers: [Object],
  [Symbol(kCapture)]: false
} +0ms

Am I missing/have to resume the mediasoup-client consumer? Data on client side doesn’t show consumer being paused.

ibc · August 13, 2020, 11:39am

You are creating the consumer I’m server side with paused:true (as done in the demo for reasons and also documented in the doc). Just resume it later (after receiver ends processing the new Consumer) and tell the receiver to also resume it.

xplatsolutions · August 13, 2020, 12:19pm

Server side I am doing the below after receiving the response from client.

await consumer.resume()

consumerPeer.notify('consumerScore', {
	consumerId: consumer.id,
	score: consumer.score
}).catch(() => { })

And indeed client-side I receive the consumerScore as you can verify in the screenshot logs.
Client-side invoking accept() and nothing else after this command.

I guess I have to figure out why consumer.resume() isn’t working.

xplatsolutions · August 13, 2020, 12:32pm

Looks fine after accepting the request client-side and IDs match too.

mediasoup:Consumer resume() +136ms
mediasoup:Channel request() [method:consumer.resume, id:29] +135ms
mediasoup:Channel request succeeded [method:consumer.resume, id:29] +0ms
mediaserver:INFO:./Logger consumer aa1f0945-373d-4889-8f82-c742f32f7e17 paused=false +135ms

Added new consumer data logging.

So weird…

xplatsolutions · August 13, 2020, 12:38pm

The above screenshot with the consumers logs in server is not entirely true. I have 4 consumers, 2 are paused and 2 are not.

The ones not paused belong to the peer I want to consume the video from the peer that produce video. All looks good…so interesting problem, any ideas appreciated.

ibc · August 13, 2020, 1:10pm

So you are not properly resuming consumers.

xplatsolutions · August 13, 2020, 1:23pm

No I am, let me explain further.

Peer1: Has two producers (not paused) and two consumers (paused) - Doesn’t want to consume any video at the moment. I don’t think I need to resume consumers for this peer.

Peer2: Has two producers (not paused) and two consumers (not paused) - Wants to consume Peer1 video and audio and received all data successfully.

Audio and video tracks are available, audio works, video doesn’t.

ibc · August 13, 2020, 4:32pm

producerScore: 0 does not look good. Video sending problems from client to server.

xplatsolutions · August 16, 2020, 6:40pm

Not sure exactly what was wrong…I started from scratch the client code and now it works. I will just have to monitor what I’m doing moving forward if it appears again.

Alby · August 17, 2020, 12:52am

Is Firefox always ok?

xplatsolutions · August 17, 2020, 8:13am

I just tested and not working in Firefox. This could be related that I pass the device info hardcoded to Chrome at the moment;

const device = {
    flag: 'chrome',
    name: 'Chrome',
    version: '84.0.4147.105',
  }
...
const res = await this.protooPeer.request('join', {
        displayName: this.displayName,
        device: this.device,
        rtpCapabilities: this.mediasoupDevice.rtpCapabilities,
        sctpCapabilities: this.mediasoupDevice.sctpCapabilities,
      })

@Alby do you still experience this issue?

ibc · August 17, 2020, 9:46am

That info is not consumed by mediasoup at all.

xplatsolutions · August 17, 2020, 9:53am

True, I just followed the demo and set these as peer.data.device.

@Alby My issue in Firefox is actually not able to connect to localhost wss media server, most probably not a mediasoup specific problem. I will update this thread when I get time to debug it.

xplatsolutions · August 18, 2020, 9:19am

@ibc I found the initial issue.

The server will throw 500 with the below error when calling produce and forcing VP9, it looks like I have to work on the server code to support VP9 as it seems passing ‘svc’ and VP9 while the server has VP8 won’t work…need to understand more the codec configuration.

mediasoup:ERROR:Channel [pid:59893 RTC::Producer::Producer() | throwing MediaSoupTypeError: video/VP8 codec not supported for svc +34m
mediasoup:WARN:Channel request failed [method:transport.produce, id:108]: video/VP8 codec not supported for svc +13m
mediaserver:ERROR:./Logger request failed:TypeError: video/VP8 codec not supported for svc at Channel._processMessage

Produce on transport.

this.videoProducer = await this.sendTransport?.produce({
        track: videoTrack,
        encodings, // NOTE: If forceVP9 and WEBCAM_KSVC_ENCODINGS will throw server error
        codecOptions: {
          videoGoogleStartBitrate: 1000,
        },
      })

The code where I get codec and encodings (from mediasoup demo sample).

let encodings
      let codec

      if (this.forceH264) {
        codec = this.mediasoupDevice?.rtpCapabilities?.codecs?.find(
          (c) => c.mimeType.toLowerCase() === 'video/h264'
        )

        if (!codec) {
          throw new Error('desired H264 codec+configuration is not supported')
        }
      } else if (this.forceVP9) {
        codec = this.mediasoupDevice?.rtpCapabilities?.codecs?.find(
          (c) => c.mimeType.toLowerCase() === 'video/vp9'
        )

        if (!codec) {
          throw new Error('desired VP9 codec+configuration is not supported')
        }
      }

      if (this.useSimulcast) {
        // If VP9 is the only available video codec then use SVC.
        const firstVideoCodec = this.mediasoupDevice?.rtpCapabilities?.codecs?.find(
          (c) => c.kind === 'video'
        )

        if (
          (this.forceVP9 && codec) ||
          firstVideoCodec?.mimeType?.toLowerCase() === 'video/vp9'
        ) {
          encodings = WEBCAM_KSVC_ENCODINGS
        } else {
          encodings = WEBCAM_SIMULCAST_ENCODINGS
        }
      }

Logs when forceVP9 is true.

Logs when forceVP9 is false.

I just need to understand better how to send correct configuration and the encodings/codec concepts. Any article where I can read to grasp the fundamentals and make better code decisions?

Your help is appreciated!

ibc · August 18, 2020, 9:37am

You are passing wrong rtpParameters to transport.produce() in mediasoup. Please, don’t just focus on client side. The information that you get in client side is somehow sent to the server and there you may be doing things wrong or whatever. The important thing here is that this is being rejected by mediasoup server so we need to know exact params given to mediasoup server API.

BTW 100% sure you are passing wrong scalabiltyMode value for VP8.

xplatsolutions · August 18, 2020, 11:03am

I use transport.on(‘produce’…) to send a request from the client. rtpParameters come from the event delegate parameters.

if (direction === 'send') {
      // sending transports will emit a produce event when a new track
      // needs to be set up to start sending. the producer's appData is
      // passed as a parameter
      transport?.on(
        'produce',
        async ({ kind, rtpParameters, appData }, callback, errback) => {
          // Signal parameters to the server side transport and retrieve the id of
          // the server side new producer.
          try {
            const { id } = await this.protooPeer?.request('produce', {
              transportId: (transport as mediasoupClientTypes.Transport).id,
              kind,
              rtpParameters,
              appData,
            })

            callback({ id })
          } catch (error) {
            errback(error)
            logger.error(
              `sendTransport.produce - server produce error: ${JSON.stringify(
                error
              )}`
            )
          }
        }
      )
    }

Server-side ‘produce’.

const { transportId, kind, rtpParameters } = request.data
let { appData } = request.data
const transport = peer.data.transports.get(transportId)

if (!transport)
    throw new Error(`transport with id "${transportId}" not found`)

// Add peerId into appData to later get the associated Peer during
// the 'loudest' event of the audioLevelObserver.
appData = { ...appData, peerId: peer.id }

const producer = await transport.produce(
{
    kind,
    rtpParameters,
    appData
    // keyFrameRequestDelay: 5000
})

The important thing here is that this is being rejected by mediasoup server so we need to know exact params given to mediasoup server API.

In what stage do you need me to show you the parameters?

ibc · August 18, 2020, 11:19am

The RTP parameters given to transport.produce() in server.

xplatsolutions · August 18, 2020, 12:11pm

produce rtpParameters forceH264=true, forceVP9=false, useSimulcast=true

{
  codecs: [
    {
      mimeType: 'video/VP8',
      payloadType: 96,
      clockRate: 90000,
      parameters: {},
      rtcpFeedback: [Array]
    },
    {
      mimeType: 'video/rtx',
      payloadType: 97,
      clockRate: 90000,
      parameters: [Object],
      rtcpFeedback: []
    }
  ],
  headerExtensions: [
    {
      uri: 'urn:ietf:params:rtp-hdrext:sdes:mid',
      id: 4,
      encrypt: false,
      parameters: {}
    },
    {
      uri: 'urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id',
      id: 5,
      encrypt: false,
      parameters: {}
    },
    {
      uri: 'urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id',
      id: 6,
      encrypt: false,
      parameters: {}
    },
    {
      uri: 'http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time',
      id: 2,
      encrypt: false,
      parameters: {}
    },
    {
      uri: 'http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01',
      id: 3,
      encrypt: false,
      parameters: {}
    },
    {
      uri: 'http://tools.ietf.org/html/draft-ietf-avtext-framemarking-07',
      id: 8,
      encrypt: false,
      parameters: {}
    },
    {
      uri: 'urn:3gpp:video-orientation',
      id: 13,
      encrypt: false,
      parameters: {}
    },
    {
      uri: 'urn:ietf:params:rtp-hdrext:toffset',
      id: 14,
      encrypt: false,
      parameters: {}
    }
  ],
  encodings: [
    {
      active: true,
      scaleResolutionDownBy: 4,
      maxBitrate: 500000,
      rid: 'r0',
      scalabilityMode: 'S1T3',
      dtx: false
    },
    {
      active: true,
      scaleResolutionDownBy: 2,
      maxBitrate: 1000000,
      rid: 'r1',
      scalabilityMode: 'S1T3',
      dtx: false
    },
    {
      active: true,
      scaleResolutionDownBy: 1,
      maxBitrate: 5000000,
      rid: 'r2',
      scalabilityMode: 'S1T3',
      dtx: false
    }
  ],
  rtcp: { cname: '', reducedSize: true },
  mid: '1'
}

produce rtpParameters forceH264=false, forceVP9=true, useSimulcast=true

{
  codecs: [
    {
      mimeType: 'video/VP8',
      payloadType: 96,
      clockRate: 90000,
      parameters: {},
      rtcpFeedback: [Array]
    },
    {
      mimeType: 'video/rtx',
      payloadType: 97,
      clockRate: 90000,
      parameters: [Object],
      rtcpFeedback: []
    }
  ],
  headerExtensions: [
    {
      uri: 'urn:ietf:params:rtp-hdrext:sdes:mid',
      id: 4,
      encrypt: false,
      parameters: {}
    },
    {
      uri: 'urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id',
      id: 5,
      encrypt: false,
      parameters: {}
    },
    {
      uri: 'urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id',
      id: 6,
      encrypt: false,
      parameters: {}
    },
    {
      uri: 'http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time',
      id: 2,
      encrypt: false,
      parameters: {}
    },
    {
      uri: 'http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01',
      id: 3,
      encrypt: false,
      parameters: {}
    },
    {
      uri: 'http://tools.ietf.org/html/draft-ietf-avtext-framemarking-07',
      id: 8,
      encrypt: false,
      parameters: {}
    },
    {
      uri: 'urn:3gpp:video-orientation',
      id: 13,
      encrypt: false,
      parameters: {}
    },
    {
      uri: 'urn:ietf:params:rtp-hdrext:toffset',
      id: 14,
      encrypt: false,
      parameters: {}
    }
  ],
  encodings: [
    {
      ssrc: 1589911514,
      rtx: [Object],
      active: true,
      scalabilityMode: 'S3T3_KEY',
      dtx: false
    }
  ],
  rtcp: { cname: 'Z0h/WKm3YwoX0KAN', reducedSize: true },
  mid: '1'
}

Yeah, it seems like VP9 is not supported by the client. I have to work out the logic I am just not sure when to force H264, VP9, and Simulcast.
From the code I read so far it should be either H264 or VP9 force and send encodings only when Simulcast=true.

Code logic for your reference:

let encodings
      let codec

      if (this.forceH264) {
        codec = this.mediasoupDevice?.rtpCapabilities?.codecs?.find(
          (c) => c.mimeType.toLowerCase() === 'video/h264'
        )

        if (!codec) {
          throw new Error('desired H264 codec+configuration is not supported')
        }
      } else if (this.forceVP9) {
        codec = this.mediasoupDevice?.rtpCapabilities?.codecs?.find(
          (c) => c.mimeType.toLowerCase() === 'video/vp9'
        )

        if (!codec) {
          throw new Error('desired VP9 codec+configuration is not supported')
        }
      }

      if (this.useSimulcast) {
        // If VP9 is the only available video codec then use SVC.
        const firstVideoCodec = this.mediasoupDevice?.rtpCapabilities?.codecs?.find(
          (c) => c.kind === 'video'
        )

        if (
          (this.forceVP9 && codec) ||
          firstVideoCodec?.mimeType?.toLowerCase() === 'video/vp9'
        ) {
          encodings = WEBCAM_KSVC_ENCODINGS
        } else {
          encodings = WEBCAM_SIMULCAST_ENCODINGS
        }
      }

      this.videoProducer = await this.sendTransport?.produce({
        track: videoTrack,
        encodings, // NOTE: If forceVP9 and WEBCAM_KSVC_ENCODINGS will throw server error
        codecOptions: {
          videoGoogleStartBitrate: 1000,
        },
        codec, // NOTE: Will fail if codec passed
      })

ibc · August 18, 2020, 12:30pm

There is no forceVP9 etc in mediasoup. That’s just part of the demo, and I don’t know how you are trying to replicate that in your app. You must check device.rtpCapabilities.codecs (after calling load() on it) to check available codecs, then call produce() with the desired one as codec field.

You are passing SVC options to the encoding in transport.produce() assuming that, magically, vp9 will be selected, but it’s not and the first available video codec is chosen, hence the error.

Topic		Replies	Views
MediaStreamTrack doesn’t play video on HTMLVideoElement mediasoup libraries	0	35	September 15, 2024
mediaTrack received from the consumer is correct? mediasoup libraries	5	454	February 13, 2022
Audio works but video stays empty. mediasoup libraries	2	240	May 24, 2022
Video element not able to play MediaStream from consumer mediasoup libraries	2	383	July 9, 2020
Video is not playing in video Element although I am getting the media track in consmer mediasoup libraries	27	1247	January 18, 2021

MediaStreamTrack doesn't play video on HTMLVideoElement

Related topics