MediaSfu_MediaSoup-client- flutter is not firing producerTransport "connect" method !!

Hi i have made a backend code and for frontend i have implemented this thing on react js, which is working fine
Now i am try to implement this thing in flutter using mediasfu_mediasoup_client package

After checking logs i find the producer transport ‘connect’ on method is never getting fired why?

Here is the log of it (flutter)
transportsParams {id: 23666f5e-d7dd-4a08-a86d-722c04274b08, iceParameters: {usernameFragment: ictr9rer45skcjxgf5y460jkhydl9ca5, password: 4ajp7pg52nyi65r6tn4yk2lmns73j6ov, iceLite: true}, iceCandidates: [{foundation: udpcandidate, priority: 1076302079, ip: 192.168.1.2, address: 192.168.1.2, protocol: udp, port: 2015, type: host}, {foundation: tcpcandidate, priority: 1076276479, ip: 192.168.1.2, address: 192.168.1.2, protocol: tcp, port: 2025, type: host, tcpType: passive}], dtlsParameters: {fingerprints: [{algorithm: sha-512, value: 3C:B9:99:4F:4C:36:EA:E6:BE:B7:FB:AD:A1:EC:95:E4:78:9D:5C:DB:98:7C:FF:68:8B:48:62:5C:B2:24:90:B2:16:1D:98:C0:77:BC:A2:C1:D8:79:F9:4F:D5:43:F1:00:23:A1:AB:27:3C:BA:66:53:1A:1D:67:28:37:2A:00:23}, {algorithm: sha-224, value: E1:8A:98:43:AF:22:66:55:A9:7C:D4:F2:0C:EE:76:46:50:60:CB:F6:6E:BB:3C:EB:03:21:A0:81}, {algorithm: sha-1, value: FB:AC:5D:80:AE:7F:19:3D:69:31:38:D2:7F:04:2D:62:EE:F1:39:7B}, {algorithm: sha-384, value: F9:60:9D:95:EE:72:6D:DE:54:83:1B:DD:29:CB:47:C2:B0:B7:C1:76:
I/flutter ( 7626): Invalid protocol value: udp
I/flutter ( 7626): Instance of ‘IceParameters’
I/flutter ( 7626): [Instance of ‘IceCandidate’, Instance of ‘IceCandidate’]
I/flutter ( 7626): Instance of ‘DtlsParameters’
I/flutter ( 7626): pTrans new
I/flutter ( 7626): producerTransport id here 23666f5e-d7dd-4a08-a86d-722c04274b08
2
I/org.webrtc.Logging( 7626): EglRenderer: Duration: 4003 ms. Frames received: 0. Dropped: 0. Rendered: 0. Render fps: .0. Average render time: NA. Average swapBuffer time: NA.
I/org.webrtc.Logging( 7626): EglRenderer: Duration: 4002 ms. Frames received: 27. Dropped: 0. Rendered: 27. Render fps: 6.7. Average render time: 156 us. Average swapBuffer time: 1154 us.
I/org.webrtc.Logging( 7626): EglRenderer: Duration: 4006 ms. Frames received: 0. Dropped: 0. Rendered: 0. Render fps: .0. Average render time: NA. Average swapBuffer time: NA.
I/org.webrtc.Logging( 7626): EglRenderer: Duration: 4003 ms. Frames received: 0. Dropped: 0. Rendered: 0. Render fps: .0. Average render time: NA. Average swapBuffer time: NA.

void createSendTransport(MediaStream sstream) {
_socket.emitWithAck(‘createWebRtcTransport’, {
‘consumer’: false,
}, ack: (params) async {
var transportParams = params[‘params’];
print(‘transportsParams $transportParams’);

final iceParameters = IceParameters(
  usernameFragment: transportParams['iceParameters']['usernameFragment'],
  password: transportParams['iceParameters']['password'],
  iceLite: transportParams['iceParameters']['iceLite'],
);

final iceCandidates =
    (transportParams['iceCandidates'] as List).map((candidate) {
  return IceCandidate(
    foundation: candidate['foundation'],
    priority: candidate['priority'] as int,
    ip: candidate['ip'] as String,
    protocol: Protocol.values.firstWhere(

(e) => e.toString().split(‘.’).last.toLowerCase() == candidate[‘protocol’].toString().toLowerCase(),
orElse: () {
print(‘Invalid protocol value: ${candidate[‘protocol’]}’);
return Protocol.tcp; // default fallback
},
),

    port: candidate['port'] as int,
    type: IceCandidateType.values.firstWhere(
      (e) => e.toString().split('.').last == candidate['type'],
      orElse: () => throw Exception('Invalid type value'),
    ),
    tcpType: candidate['tcpType'] != null
        ? TcpType.values.firstWhere(
            (e) => e.toString().split('.').last == candidate['tcpType'],
            orElse: () => throw Exception('Invalid tcpType value'),
          )
        : null,
    transport: candidate['transport'] as String? ?? '',
    raddr: candidate['raddr'] as String? ?? '',
    rport: candidate['rport'] as int? ?? 0,
  );
}).toList();

final dtlsParameters = DtlsParameters(
  fingerprints:
      (transportParams['dtlsParameters']['fingerprints'] as List)
          .map((fingerprint) => DtlsFingerprint(
                algorithm: fingerprint['algorithm'],
                value: fingerprint['value'],
              ))
          .toList(),
  role: DtlsRole.auto,
);

print(iceParameters.toString());
print(iceCandidates.toString());
print(dtlsParameters.toString());

var createSendTransport = device.createSendTransport(
  id: transportParams['id'],
  iceParameters: iceParameters,
  iceCandidates: iceCandidates,
  dtlsParameters: dtlsParameters,
);

print('pTrans ${createSendTransport.connectionState}');
producerTransport = createSendTransport;
print('producerTransport id here ${createSendTransport.id}');

producerTransport.on('connect', (data) async {
  final dtlsParameters = data['dtlsParameters'];
  try {
    _socket.emitWithAck('transport-connect', {
      'dtlsParameters': dtlsParameters,
    }, ack: (res) {
      data['callback']();
    });
  } catch (e) {
    data['errback'](e);
  }
});

producerTransport.on('produce', (Map data, Function callback, Function errback) async {
  try {
    _socket.emitWithAck('transport-produce', {
      'kind': data['kind'],
      'rtpParameters': data['rtpParameters'],
      'appData': data['appData'],
    }, ack: (response) {
      callback({'id': response['id']});
    });
  } catch (e) {
    errback(e);
  }
});

producerTransport.on('connectionstatechange', (state) async {
  print("Transport connection state: $state");
  if (state == 'connected') {
    print("✅ Transport is connected. Now safe to produce.");

    try {
      final videoTrack = sstream.getVideoTracks().first;
      print("Tracks in stream: ${sstream.getVideoTracks()}");
      print("Using track: $videoTrack");

      producerTransport.produce(
        source: 'webcam',
        stream: sstream,
        track: videoTrack,
        codecOptions: ProducerCodecOptions(
          videoGoogleStartBitrate: 1000,
        ),
        appData: {'mediaTag': 'cam-video'},
      );
    } catch (e) {
      print("Error in creating producer: $e");
    }
  }
});

});
}

Activity

Already addressed this here.

Hi @Bdayz , Could you please let me know how to resolve this issue, if you are familiar to this case?

you can check my flutter client side code here in github:-GitHub - Sam15b/MediaStream.
But i Suggest you to better way implement this Natively Because when mediasoup_client_flutter support version ^ 0.4.6 is compatible with flutter_webrtc ^0.6.10 if you increase the version of any it throw you error of incompatiable with flutter_webrtc or mediasoup_client_flutter . So i implement this project in Flutter 3.16.9 and Dart 3.2.6 . Because of Incompatibility Issue of flutter packages . And Also in Production Google need Compile and Target Sdk 34 . If new Android Version 16 comes then New Policy of Google will Also be comes with Compile and Target Sdk to 35 Which Also Incompatible with both this flutter packages. I raised this issue to Mediasoup but i see no response from them Mediasoup Client Flutter Compatibility Issue with Flutter WebRTC

Sorry; you duplicated the issue on the GitHub repo, and a likely solution is posted there (GitHub), so do check it out.

In summary, call your ‘produce’ method to trigger the rest of your logic.

@Bdayz Yes i just saw i duplicated the Github repo . Mainly in that section i missed the produce method and the consume callback logic . I would update that, But Also my problem is When i Implement the Screen Sharing logic in it Mainly in Android version 15 in first time the App gets terminated and then afterward it works fine. And it throw Error “SecurityException because the required foreground service permissions for mediaProjection are not properly handled.” while added this permission

<uses-permission android:name="android.permission.FOREGROUND_SERVICE" />
<uses-permission android:name="android.permission.REQUEST_IGNORE_BATTERY_OPTIMIZATIONS"/>
<uses-permission android:name="android.permission.FOREGROUND_SERVICE_MEDIA_PROJECTION"/>

and service

   <service  android:name="de.julianassmann.flutter_background.IsolateHolderService" android:foregroundServiceType="mediaProjection" android:enabled="true" android:exported="false"/>    

And in main.dart i also called my foreground services

Future<bool> startForegroundService() async {
  const androidConfig = FlutterBackgroundAndroidConfig(
    notificationTitle: 'Title of the notification',
    notificationText: 'Text of the notification',
  );
  await FlutterBackground.initialize(androidConfig: androidConfig);
  return true;
}

void main() async {
  WidgetsFlutterBinding.ensureInitialized();
  startForegroundService();
  runApp(const MyApp());
}

and last my logic for Screen Sharing

Future<void> _enableScreenSharing() async {
    try {
      print("CHeck for _localscreensharingStream");
      FlutterBackground.enableBackgroundExecution();
      _localscreensharingStream = await navigator.mediaDevices.getDisplayMedia({
        'video': {
          'cursor': 'always'
        },
        'audio': false, 
      });
      print("CHeck for _localscreensharingStream222");
      // Get the screen video track
      _localscreensharingTrack =
          _localscreensharingStream!.getVideoTracks().first;

      final ScreenRenderer = mediasoupService.users[0]['screenShareRenderer'];
      await ScreenRenderer.initialize();
      ScreenRenderer.srcObject = _localscreensharingStream;
      mediasoupService.users[0]['hasScreenSharingOn'] = true;

      _connectSendTransportScreen();

      print("Screen sharing enabled");
    } catch (e) {
      print('Error enabling screen sharing: $e');
    }
  }

Hi @Sam15b Thanks for sharing the repo but its seems to be the consumer transport is still not access when a user connected through flutter app and another is connected to website frontend

Is it working on your end?

Connect with me to check what is the issue with consumer part. You can check the demo video that i created . #flutter #nodejs #webrtc #mediasoup #aws #realtimetech #flutterdev… | Syed Abdul Mannan

Thanks for replying, @Sam15b

The issue you can recreate by connecting one user from phone (create room) and open website and join the room

From the phone side give permisson for camera and in that room you cam find the error in console log of the web frontend

Even if the web user has also enable camera even then also the phone user will unable to see any consuminb video stream of him/her(web user)

i see no error in my web console log . It means there is an issue of consumer part in web client side and flutter client side. Share your consumer logic for both web and flutter part.

Thats Strange , @Sam15b ,

Because i tried this by cloning your repo and ran the backend , here i have given the announced ip as from my public ip and listen ip as 0.0.0.0

In flutter part i just point to that server that is locally running nothing else

But the problem arises when a user join or create a room through mobile he can see his local stream but can consume any one other user in the room and similar for web user too

Is there anything i am doing wrong here?
Cuz i found your backend server code is almost similar to what i have wrote

do you running your backend server locally in Windows. If so then it didnot gone work . I run my Backend Server in Ubuntu Ec2 Machine with reverse proxy . And Just by cloning my project not gone work for you i change my Elastic Ip in the github repo.

@Sam15b , So how can i run it locally?

You can’t run it locally . Because of Windows Firewall system . You have to use some external machine like wsl (Ubuntu) or best is Aws Ec2 machine .

My logic of Mediasoup connectivity is correct for both producer and Consumer section you can refer from my project repo . And if any issue comes inform me.

@Sam15b , Can i use the docker for this purpose to test this thing locally , If yes could you please share me the steps?

You can use dockers to run just for Web thing locally. To run Both on web and Phone you have to make backened server run in public ip not your local ip. For this i recommend you to host you backend server in Ec2 Ubunto Machine to run backend server Publically.

@Sam15b , Is there any way to test it locally first , if it run then i can proceed for hosting purpose

In intial Phase of my development i find lot of issues in docker so i shifted my backened to Ec2 . I think there is a way to make it using dockers. But i suggest you go implement Ec2 machine basically the free trial one . This will definitely work for cross platform.

Got it @Sam15b, I am getting this in your app side when i try to join a room but got this error

Empty String in producerId

Also i saw the post but i want to know, are you using putty ubuntu or anything else to achieve that?

tou can use any thing to get access of your machine i am using private key to access my ubuntu machine. You can use ssh key also or aws terminal.

And Error of Empty String producerId. In which line , describe properlt the error