SendTransport.Produce error after onConnect is fired (libMediasoupclient - iOS wrapper)

mediasoup libraries
1

Seems like I am missing something but the error is not clear where is the problem. I am working with protoo-server in backend but connection and data exchange seems good.

Here is the payload from the server:
{"response":true,"id":1594900528,"ok":true,"data":{"id":"e69f9b16-75e7-4db9-9789-472b3461914b","iceParameters":{"iceLite":true,"password":"m1noqsetar4jvnugba45fdnirs9skonp","usernameFragment":"p06nxbxkd2by6e5n"},"iceCandidates":[{"foundation":"udpcandidate","ip":"192.168.2.2","port":40327,"priority":1076302079,"protocol":"udp","type":"host"}],"dtlsParameters":{"fingerprints":[{"algorithm":"sha-1","value":"CE:FC:D9:BA:67:C3:8E:82:75:34:37:2A:E9:01:DA:4C:58:7D:8A:3F"},{"algorithm":"sha-224","value":"69:52:97:0E:FE:59:B1:20:6F:5D:D2:ED:DE:8F:EB:65:D5:01:9D:AE:F8:C8:D1:6B:BD:A7:82:10"},{"algorithm":"sha-256","value":"54:0B:F3:10:3B:3B:DB:9D:4A:F6:D9:F3:AD:51:F5:DA:78:FE:EA:36:38:54:FE:92:93:E1:84:A7:03:85:4F:DE"},{"algorithm":"sha-384","value":"50:D1:9E:B4:B7:24:34:FD:1C:EB:CE:DE:CC:CF:2A:67:59:77:B0:59:34:5B:93:30:B4:C6:E5:76:81:A4:B7:99:F2:5D:A9:3F:5F:85:FA:7D:4D:DB:B7:41:CC:CE:C1:38"},{"algorithm":"sha-512","value":"54:88:3B:7F:12:B2:B5:B2:8B:F4:DC:9A:A1:A5:4F:20:96:36:A7:B9:53:4D:8D:DA:58:D0:00:44:96:4F:86:01:F0:E3:65:CA:FC:75:C5:F4:CC:CF:0D:73:85:9A:0F:1B:13:D9:C0:3F:FA:E9:F4:09:EF:08:39:A6:90:F8:B0:A7"}],"role":"auto"},"sctpParameters":{"MIS":1024,"OS":1024,"isDataChannel":true,"maxMessageSize":262144,"port":5000}}}

Debug info from Device.

[ERROR] transport_wrapper::+[TransportWrapper nativeProduce:listener:track:encodings:codecOptions:appData:]() | [json.exception.type_error.302] type must be string, but is null

onProduce will not get a chance to fire. [EDIT] onConnect will fire.

In SendTransport.Produce I pas encodings = null and codecOptions = "{\n \"videoGoogleStartBitrate\" : 1000\n}"

2

codecOptions must be a json object, not a json string.

3

It sounds like we need to work on the iOS wrapper method signature (@ethand91 ), only thing is I didn’t have this problem before converting HTTP signaling to protoo-server, but highly unlikely that to be an issue since data messaging communication seems fine.

4

@ibc it looks like after all the messages below I receive the error.

[DEBUG] Handler::Send() | calling pc->SetLocalDescription():
[DEBUG] PeerConnection::OnSignalingChange() | [newState:have-local-offer]

// Connecting the transport on server here so onConnect is fired
websocketDidReceiveMessage {"response":true,"id":1595231957,"ok":true,"data":{"actionEvent":"connectWebRtcTransport"}} 

[DEBUG] PeerConnection::OnIceGatheringChange() | [newState:gathering]
[DEBUG] PeerConnection::OnIceCandidate() | [candidate:candidate:851307166 1 udp 2122260223 192.168.2.4 61436 typ host generation 0 ufrag /rK5 network-id 1 network-cost 10]
[DEBUG] PeerConnection::OnIceCandidate() | [candidate:candidate:3159874914 1 udp 2122194687 169.254.230.86 58099 typ host generation 0 ufrag /rK5 network-id 2 network-cost 10]
[DEBUG] PeerConnection::OnIceCandidate() | [candidate:candidate:3710925147 1 udp 2122131711 2607:fc20:7c58:c851:8fad:ddf4:c547:ddac 58100 typ host generation 0 ufrag /rK5 network-id 9 network-cost 900]
[DEBUG] PeerConnection::OnIceCandidate() | [candidate:candidate:1046478326 1 udp 2122063615 162.162.189.82 62860 typ host generation 0 ufrag /rK5 network-id 8 network-cost 900]
[DEBUG] PeerConnection::OnIceCandidate() | [candidate:candidate:3710925147 1 udp 2122000639 2607:fc20:7c58:c851:8fad:ddf4:c547:ddac 62861 typ host generation 0 ufrag /rK5 network-id 3 network-cost 50]
[DEBUG] PeerConnection::OnIceCandidate() | [candidate:candidate:3710925147 1 udp 2121935103 2607:fc20:7c58:c851:8fad:ddf4:c547:ddac 62862 typ host generation 0 ufrag /rK5 network-id 4 network-cost 50]
[DEBUG] PeerConnection::OnIceCandidate() | [candidate:candidate:3710925147 1 udp 2121869567 2607:fc20:7c58:c851:8fad:ddf4:c547:ddac 62863 typ host generation 0 ufrag /rK5 network-id 5 network-cost 50]

I assume the problem relies here;

I tried with nil for codecOptions and error is the same, is this related to some server response.

Investigating the iceCandidates response from the server I see a key property “type” with value “host”.

{
  "response": true,
  "id": 1595236196,
  "ok": true,
  "data": {
    "actionEvent": "createWebRtcTransport",
    "id": "b8a8c27a-2804-4499-962e-9070a314b276",
    "iceParameters": {
      "iceLite": true,
      "password": "i1j06oqh6zw3zbq0buod3x723o9wg0fd",
      "usernameFragment": "1gezxrt8taveymtr"
    },
    "iceCandidates": [
      {
        "foundation": "udpcandidate",
        "ip": "192.168.2.2",
        "port": 44711,
        "priority": 1076302079,
        "protocol": "udp",
        "type": "host"
      }
    ],
    "dtlsParameters": {
      "fingerprints": [
        {
          "algorithm": "sha-1",
          "value": "89:E3:DB:95:EE:2D:9B:A7:18:9A:AB:3C:CE:DC:28:CD:C2:A8:56:B8"
        },
        {
          "algorithm": "sha-224",
          "value": "E4:B5:1A:46:36:87:54:82:6C:C1:F0:A6:F7:1F:DA:7E:68:2D:FE:51:53:EF:2D:25:F4:1A:F4:6E"
        },
        {
          "algorithm": "sha-256",
          "value": "AF:7C:6C:A4:31:65:89:51:8A:53:66:2F:0A:4C:CD:D3:CB:2E:68:DA:97:B7:17:8E:5F:34:7F:41:05:2E:DA:CE"
        },
        {
          "algorithm": "sha-384",
          "value": "D9:7A:77:54:1E:20:29:7D:95:BF:51:16:10:2C:B9:61:14:85:75:E9:08:97:C3:4D:B1:47:A0:EE:5D:CE:15:B1:C1:00:6F:76:C3:23:32:EA:B4:DF:90:F6:72:4C:10:0E"
        },
        {
          "algorithm": "sha-512",
          "value": "AF:1F:C1:29:21:6D:A9:4B:32:50:F3:CE:5C:FF:1A:67:F2:23:00:C8:90:15:E0:5B:27:D7:07:CC:D7:20:7E:05:51:40:0E:76:AA:C1:F2:3E:0D:15:9F:14:89:44:6A:BE:E2:79:5F:33:68:81:F8:45:4D:E6:4E:D7:67:16:13:DC"
        }
      ],
      "role": "auto"
    },
    "sctpParameters": {
      "MIS": 1024,
      "OS": 1024,
      "isDataChannel": true,
      "maxMessageSize": 262144,
      "port": 5000
    }
  }
}
5

Playing around with the server response JSON and now I am getting the below, might be a bit more helpful;

[ERROR] ortc::validateIceParameters() | throwing MediaSoupClientTypeError: params is not an object
[ERROR] device_wrapper::+[DeviceWrapper nativeCreateSendTransport:listener:id:iceParameters:iceCandidates:dtlsParameters:sctpParameters:options:appData:]() | params is not an object
6

So print the ICE parameters you are passing into the CreateXxxTransport() factory.

7

Here is the iOS code to create the Device Transport.

print("Server Response - iceParameters:\n\(iceParameters.description)")
print("Server Response - iceCandidates:\n\(iceCandidatesArray.description)")

self.sendTransport = mediasoupDevice!.createSendTransport(self.sendTransportHandler!.delegate!, id: id, iceParameters: iceParameters.description, iceCandidates: iceCandidatesArray.description, dtlsParameters: dtlsParameters.description, sctpParameters: sctpParameters.description, options: nil, appData: nil)

print("Local Transport\nID: \(self.sendTransport!.getId()!)\nConn State: \(self.sendTransport!.getConnectionState()!)\nIsClosed: \(self.sendTransport!.isClosed())")

Server Response - iceParameters:
{
"password" : "4mfx7todqf66mhfq1vo68i1qr8r7va8z",
"usernameFragment" : "b52s6c3s5et910ad",
"iceLite" : true
}

Server Response - iceCandidates:
[
  {
    "protocol" : "udp",
    "foundation" : "udpcandidate",
    "priority" : 1076302079,
    "type" : "host",
    "port" : 45567,
    "ip" : "192.168.2.2"
  }
]

Local Transport:
ID: 3d8bf0a4-07b5-4ccb-b310-054216a848d1
Conn State: new
IsClosed: false

[ERROR] transport_wrapper::+[TransportWrapper nativeProduce:listener:track:encodings:codecOptions:appData:]() | [json.exception.type_error.302] type must be string, but is null

2020-07-20 05:50:35.897595-0400 Phone[4428:2028450] *** Terminating app due to uncaught exception 'RuntimeException', reason: '[json.exception.type_error.302] type must be string, but is null'

First throw call stack:

(0x1bfad5794 0x1bf7f7bcc 0x1052e2ef0 0x1041062c0 0x104105b90 0x10410560c 0x104103610 0x104102e80 0x104107d54 0x10410841c 0x1040ddb9c 0x1040dc390 0x1040db608 0x1040df940 0x10435845c 0x10432b964 0x104a42338 0x104a43730 0x104a45e88 0x104a54d74 0x104a55698 0x1bf7ecb38 0x1bf7ef740)

libc++abi.dylib: terminating with uncaught exception of type NSException
8

Honestly I don’t know what such a ICEparameters.description is. It must be a JSON object. I don’t think it’s that.

9

Yeah, it is a JSON string representation, I guess @ethand91 is handling how a JSON string is passed down to the native libMediasoupclient as a JSON object;

10

The ios client takes a string (NSString), the wrapper then parses that string into a nlohmann::json object and passes it to the inner libmediasoupclient.

11

Thanks for replying @ethand91, I guess we can stick to the GitHub opened issues.