Users connect to the server.
Server gets information from other server and will connect users together.
Like this:
User 1 connects. User 2 connects. User 3 connects.
Server gets information that user 2 and user 3 needs to be connected but user 1 shouldnt.
User 2 receives user 3’s stream and user 3 recieves user 2’s stream.
5 minutes later user 3 should not be connected to user 2 anymore and the streams are shut off for user 3. but user 1 will receive it now.
There’s loads of posts about scalability, it’s impossible to answer such a question in a post like this. The answer to you original question is yes it’s possible.