A mini-crunch mode is done and progress videos were made in time for the East Coast Game Conference. They can be seen on the facebook page.
While at ECGC I had some wonderful conversations about server (server cluster) infrastructure. Since starting this project I have been looking for better ways to slice and dice the server code to make things as efficient (and cost effective) as possible. I have identified 3 potential bottlenecks in my server setup: 1) Socket connections to the 'Zones' (how many people can connect to a single zone), 2) AI processing (how much resources does it take to update 100,000 RTS units), and 3) Database connection (even using a Mongo cluster all writes are done on the primary). If I have 3 Zones and 2 of those zones have very low traffic, what is the best way to shift resources over to the zone that has a lot of processing to do? I've researched everything from socket load-balancing to a stateless server structure.
It was at ECGC I was clued into a potential answer. SpatialOS (by Improbable) is a new MMO server engine that should take these server problems out of my concern. It looks like they store world objects in Redis style memory, with worker processors handling any updates for those objects.
The idea behind it is sound, the persistent storage of user/unit information can be solve with my same Mongo solution, and this may give me a few valuable tools: A zoneless world, better development tools. The first is a great bonus and will allow users to have their units roam over the entire world. I can imaging having half your units logging a forest while your fighters are half way across the world helping the war effort. The second beautiful bonus is an easier way to create the world, and monitor the world as it is running, through development tools I haven't yet had the time to create.
However switching technologies isn't easy or free. Moving the client over is simple as both my current client and Spatial client is in Unity. My database is the same and shouldn't need to change at all. My server is currently in Java, and that will need to switch to Unity and the Spatial way of processing things. With the amount of availability I currently have, I expect this to take 4 to 6 months, but after this time I should have a playable demo. Here's hoping!
Update - August 09, 2018:
After a lot of thought and design I decided to move my entire server from Java to .Net Core and add a caching system to help make the entire server horizontally scalable. I wanted to transfer everything into .Net Core so that I could reduce the number of languages used in my system. For the caching system I chose Hazelcast, but I'm not the happiest about it. Hazelcast is written in Java, so some of the advanced features (i.e. predicate searches on complex objects or keys) is not available in their C# library. Still, it helps lay the foundation for a caching system.
To horizontally scale everything, I've broken the server into smaller 'processors', each with a specific function, and each capable of having multiple instances running.
- Processor Hub: Controls the flow of the server so that each unit is updated at the appropriate time and thus evaluates its movement based on everyone's current position. (ever unit moves at the same time). It sends out signals to the AI processors to calculate the next step, receives responses that each AI processor has completed the step, sends the signal to perform the next step, receives responses, etc. It also acts as the main hub to forward messages between processors and spawn new processors to load-balance.
- Client Message Processor: Clients connect to this server via UDP and transmit unit commands. Connector sends back unit positions & positions of units the client can see. Handles all of the encryption and token validation for client connections (beyond login and rest service)
- AI Processor: Performs the calculations for what units should do. Expected to be spun up and down based on the needs of the Zone and, as it is spun up, is given a which units (by Id) to work.
- Faction/Threat Processor: NPC (non-client) owned units need higher level instructions. These units are broken down into faction owned, keep (area) owned, or squads (small group). Units send messages like "I'm being attacked" or "I see an enemy" up to their owner which helps determine what to do about; which may include forwarding the threat up the line (squad to keep to faction). The results are commands sent to the unit to be followed.
Not connected to the pocessor ecosytem is the REST service that handles all "non-time sensitive" requests. (i.e. unit definitions, login, chat messages, etc)