1 Year at hotstar – a startup in beast mode!
It’s been over a year since my association with hotstar and it feels like it was yesterday. I started with hotstar as a consultant. I was initially tasked to work with the team in making the platform scale for upcoming IPL. I worked over period 2-3 months with the QE team on building out the first version of what we called Game Day Simulations It was a very interesting problem statement, something which I have never done before. We designed a simulation for 10M Concurrent users, using different traffic pattern and simulating users journey in the app. Then came the IPL 2018, we started with a bang of 6M+ Users on the opening game. One of the largest so far on the platform. Over the course of the tournament, I got an opportunity to work with the different teams and external services which hotstar was using at that time. As the tournament progressed the 10M number started to look real. We worked on different strategies to handle traffic across different services in case we reach that number. It was good 2 months! 50+ matches and we were breaking our own records as the games passed. In the final, we did touch the 10.3M number! Setting a global record.
This entire journey of IPL got me excited enough to work with hotstar as full time. I started working closely with Adtech team then. It was relatively a new team at hotstar. The format of advertising at hotstar was very static in nature, being a digital company we wanted to provide our advertisers with a capability to target on the live stream. Being a digital platform, it sounds very naive for us to not offer that. But it’s not that simple as it sounds. Live streaming is very different from VOD streaming, it has its own challenges. We discussed with a lot of external vendors about existing solutions. Almost all of them had some form a limitation – Either they would add latency to the stream or can’t support beyond a number. We did not want to work with that kind of limitations. I spend the first few months evaluating different available vendor options.
Alongside I was working on another idea at hotstar, the part of the idea needed us to build our own ad-exchange so that we can support PMP ad deals. We looked at existing exchanges, but all had some form a limitation in terms of the throughput, and we wanted speed and flexibility, so we decided it to build it in-house. This was the first service, which I launched at hotstar and the first time I worked on Java. Once launched I saw traffic ramped up to 60K rpm within a day of a full roll out. This was the first time I actually wrote a service working continuously at that volume. Nothing is more pleasing than seeing a design on paper coming out in life as it is!
Coming back to targeting on live. Time passed, no vendor was able to support our use case, and Asia Cup came in. This time we were expecting a larger volume of traffic in India vs Pakistan. I worked back with the team on doing another set of Game Days, we needed to revamp the script from the learning of IPL and apply ODI traffic pattern. This was the first time hotstar was going to operate all of the backends on all in-house developed services, freeing itself completely from the limitation imposed by third-party service providers. The tournament went well. Even though we did not breach our IPL records, but we noticed more stickiness on the platform.
I wanted to make a live targeting at our scale a reality without adding any latency/delay to our streaming. Over the course of the next few months, we worked on developing this system called SSAI (Server Side Ad Insertion). Our goal was to test it on the first series of game before next IPL. We developed all the components needed for it in-house. Did a lot of testing in-house. At hotstar, our first design principle is to design for failure. We design everything to fail and to ensure that the video keeps running, and SSAI was the service which was just touching that! So we had to design it in such a way that even if the ad targeting fails, the video should keep running! We missed our first set of deadlines to release it. We wanted to test the service in India vs New Zealand series in Jan 2019 on production but couldn’t go out of shadow testing. Over the course of the tournament, we made service stronger and were able to test it in production in India vs Australia match of 2019. There was an instance when service was misbehaving but it never impacted the user playback because of our first design principal.
We improved the performance as time progressed and over the course of IPL 2019. During IPL 2019 final this service was doing > 2.5M request per second on the edge! Seeing it run at that scale still makes me feel happy & proud! And that’s how my first year at hotstar passed. Time just flew and I did not realise how and when.
Hotstar is a startup in beast mode. One has all the challenges one would ever want. One doesn’t have to worry about the validation of the platform! People love watching content, it’s proven. Now it’s up to us on how we want to scale it!