I have seen all the discussions done for these two direct questions but have yet to find a concrete answer for this.
I just have two questions:
What is the most efficient way to do load balancing without using any paid third-party apps?
When we have routers on different hosts, what is the best way to connect them? For example, do we store the different server IP addresses somewhere in order to connect to them when requested, or do we get the producer server IP from the client side and then make a connection (which is, of course, not a secure way)?
1). Custom load balancing solution like broker server as we have peer connection with server i.e transport so I think there will be no automatic way to do load balancing in this case as we have direct connection to the server i.e nothing going through aws or some other service.
So what I do is that I have a borker server to which all media servers connect over socket and that borker server keeps track of all the servers connected to it along with which producer, consumer etc is on which server. And also which server have how much load. And that broker server decides to which server the user will have connection (transport) with based upon the load of the media servers, producers etc.
2). When you will have all the mappings in broker server as mentioned in 1) then you can ask 2 media servers which are on different hosts to connect with each other with pipe transport and share the producer with each other and after that you can let the user consume from the relevant server.
If anyone have some other solutions better than this, please share the thoughts, eagerly wanting to have some good stuff on this topic, thanks
When I was thinking about possible solutions, this was one of them. But I want to handle a million users, and for that, I need approximately 2000 workers. If, on average, I have a 10-core CPU host, then I need 20 hosts. So, do I need to keep track of all those servers in some sort of gateway or proxy?
You can support millions of users with this approach in my opinion but you need to confirm because broker server is going to be very light weight process kind of load balancer handling the servers connected to it. The main load 95% will be on media servers where your workers will be. You need to stress test it to confirm this.
Regarding 2000 workers for 1 million users I think you calculated this based upon 500 consumers per worker thing. But in reality if worker can handle 500 consumers you need more than 2000 workers to support 1 million users because of the piping thing. You need to consider the weight of piping as 2, 1 producer load on the producing server and 1 consumer load on the consuming server and with this in mind if your cpu can support 500 consumers per worker then due to piping it will be lesser than 500 it can be 400, 450 etc you need to calculate.
So you will write a simple node server which is called broker server and that broker server should keep track of all the media servers and connect the user to the best possible media server.
And by millions users we mean million streams (producers, consumers) in and out of server, the overall users will be a lot lesser because a user is going to need may be 20 streams etc.
So lets say a user need 20 streams and you need to support million users then 1 million * 20 = 20 million so you need to support approx 20 millions consumers, producers apart for piping stuff. Figures are random just to illustrate.
so my app will have a can have alot of rooms depending on how many people will be live and each can have maximum 9 concurrent producer/consumer but there will 1000 or more active watcher may be in some room 10k or 100k depending upon the popularity of the broadcaster
So it is live streaming kind of application may be 1-many or 9-many as you mentioned. So if in a room 9 people are live together and broadcasting then as a watcher I need 9 consumers of video and 9 consumers of audio and that will be total 18 consumers and if there are 100 watchers it will be 18*100 = 1800 consumers just for 100 watchers. So you need to calculate it like this way.
Let’s assume you can have many many many connections, it comes down to requests a second, so you can run 100->100,000 servers but truth is you want to be fast so a broker may handle 100-1,000 servers at any given time. You need to benchmark, test and tune to your needs.
So for a broker server there’s not one type, there can be many in ones development. You can design a chat broker server, media broker server and more. These essentially are routing/signal servers that keep synchronization over queue to ensure data delivery across many servers is not corrupt or having race issues.
36-cores of media server power only use 1% or less of broker CPU in my case.
Too many factors, I run 9 media servers such has literally no impact, can’t calculate it.
thanks for the insights, with 36 core media servers I assume that will be 9 servers of 4-cores and if they use less than 1% broker then in that case we can safely say that it will be able to handle more than 1000 media servers. Is this right?
Do you have any broker server implementation to refer to? I am using the NestJS framework for the broker server and microservices as the media service, and is it considered good standard practice do so ?
Not exactly, there should be a cap-size on how many users/rooms can utilize a broker server to not overwhelm in signaling. So 1,000 media servers could be a bit wild and add several seconds of delay till a user gets a response if say 20K requests came in all of a sudden as a room restarted and everyone is flooding in.
I’d probably set my limit to 100-200 media servers per media broker and just open up more media brokers and assign rooms to them based on various scaling factors.
On a real note, one should never throw $30,000 of resources on one broker for if failures occur that’s expensive down-time. Be better off running 30 brokers $1,000 of media servers each broker for higher availability and switch over upon failures.
