Pitboss Redundancy Discussion

[Tarkeel]

Thanks for the details Zulan.

For moving further, I will take a closer look at the setup on the server and see what is needed to setup another game. Can I assume that you are going to use the same mod known as "RB74"?

The RB74 mod is the latest iteration of CtH, and is the de-facto standard for now since development is paused with Charriu's absence.

[RefSteel]

Thanks for the detailed information, Zulan - and for hosting the server that has enabled these games for so long!  I hope Ramk is well too, regardless of whether or not that involves someday working with the Pitboss server again.

- It sounds like (as I kind of suspected from some of the github documentation) your current Pitboss server is running on Linux, so we don't need to do any further testing to confirm that's possible - it sounds like the uncertainty is whether it can be done on Windows, actually!  Thanks for explaining that and pointing to the pitboss-wonkiness-mitigating infrastructure you've created.

- I'll continue the slow process of trying to look at and understand the various interacting software involved in making the server work properly.  I have no formal (and little informal) training in any kind of remotely advanced or modern coding, but others here might have more background there, and I have a couple of friends who might be able to give me some advice.

- I think Krill and scooter were talking about a home server as basically a stop-gap to run new Pitboss games until we could (hopefully) get a cloud server running.  If setting up back-up natneg has to wait for the latter, getting a back-up Pitboss server working in the meantime with RB admins would still be worthwhile, I think:  Hopefully at least the natneg doesn't require as much direct administration as does (e.g.) creating new Pitboss games, reloading them, and tending to their other assorted needs.

- Of course if you're able and willing to create a new game on your server yourself, even with no further assistance, that's much appreciated!  As you assume and Tarkeel confirmed, RB74 is indeed our current mod of choice.

[Krill]

First of all, I truly hope Ramk is fine. And maybe, we will come back. Regardless, it is not great to have no redundancy and share of the work. I will try to support this one way or another. But I cannot offer any "SLA" whatsoever.

Okay, cool.  So for a first super-simple proof of concept, I started a vanilla Civ4 Pitboss under Linux Mint, using Wine; then I started vanilla Civ4 itself, connected to the Pitboss game on my own machine, and successfully logged in, set my civ's password, and played a turn.  Still have to make sure others can connect too, and set it up to run BtS, the various mods, and Ramk's tools, but at least it looks like a Pitboss server really can run on Linux anyway!  (Plus test for stability, but my set-up remains a poor choice for testing that part.)

Unfortunately continuously running a number of PB servers is quite a complex deal. Now we do have 10 years of experience, even though Ramk is the most knowledge person on this period. To mention the most important aspects:

• Civ4's network protocol is wonky to the extent that in some situations, connections are not terminated properly. This leads to the PB server blasting substantial packet rates to no-longer-connected clients indefinitely. This can cause huge amounts of unwanted traffic. We have pretty sophisticated infrastructure in place to mitigate this. This works on Linux systems, not sure if you can get it to run on Windows. https://github.com/civ4-mp/pb-watchdog
  
  
• The Pitboss server does in fact have a graphical interface, which was huge pain on headless servers. There are also sporadic errors that require graphical interactions, i.e., clicking "ok" on an error popup. Fortunately, we have recently containerized the workarounds. That doesn't mean it will just work as is. If you can understand what's happening here: https://github.com/civ4-mp/docker-civ4-pitboss-server - you could be very helpful!
  
  
• To allow multiple players to join often requires Nat Negotiation, which was shutdown with Gamespy. I run the NatNeg servers on the same server as the PB server, they are also used by the replacement Lobby. It is extremely unlikely that you can run a Natneg server at home internet connection. But you could run a PB on a different server and still use my Natneg server.
  
  
• Especially for larger PBs, to reduce login times, we implemented the side-loading of save games. This does require you to host a web-server in addition to the PB server on the same system
  

For moving further, I will take a closer look at the setup on the server and see what is needed to setup another game. Can I assume that you are going to use the same mod known as "RB74"?

Zulan, thank you for keeping the Civ 4 scene going. If you hear anything about Ramk please can you keep us informed?

I should ask, what is the challenge with hosting the Natneg servers, and the side loading? What sort of bandwidth do we need to make it work? I'm mindful that I can get a 1.6Gbps down/115Mbps up just by upgrading internet, and it is just as trivial for me to upgrade the last leg of my home network cable to 10GbE. If it's other hardware that is the problem, I will be using an Epyc 7532 (32 core, 64 thread) with 256GB of RAM so I can dedicate quite a bit of resource to it if needed. how does this compare to what is currently in place?

Ref, scooter, I was hesitent to be more forthcoming until I built and tested the used hardware I've bought, but it's working now. I can confirm that I should be able to run a Debian or Mint VM with a graphical interface dedicated as a pitboss host, and keep this separate from my own home network. This means that the pb-watchdog should function without alteration, and this is limited by my capabilities as an admin (that's a pretty big shortcoming).

[scooter]

Thanks so much Zulan for checking in and explaining so much.

If you can understand what's happening here: https://github.com/civ4-mp/docker-civ4-pitboss-server - you could be very helpful!

FWIW I am a software engineer by day, and while my docker expertise is middling, that might be better than nothing. Learning on the fly is part of the core competency anyway. Happy to help in any way I can.

I should ask, what is the challenge with hosting the Natneg servers, and the side loading? What sort of bandwidth do we need to make it work? I'm mindful that I can get a 1.6Gbps down/115Mbps up just by upgrading internet, and it is just as trivial for me to upgrade the last leg of my home network cable to 10GbE. If it's other hardware that is the problem, I will be using an Epyc 7532 (32 core, 64 thread) with 256GB of RAM so I can dedicate quite a bit of resource to it if needed. how does this compare to what is currently in place?

Ref, scooter, I was hesitent to be more forthcoming until I built and tested the used hardware I've bought, but it's working now. I can confirm that I should be able to run a Debian or Mint VM with a graphical interface dedicated as a pitboss host, and keep this separate from my own home network. This means that the pb-watchdog should function without alteration, and this is limited by my capabilities as an admin (that's a pretty big shortcoming).

That is… some serious hardware. That sounds incredibly useful either as a backup to Zulan’s servers or used in conjunction with them. If you need a tester or want another set of eyes on this, let me know.

[Krill]

Yes, it is some serious hardware, took a short while to find in the UK. Right now it's being used a file server, with Jellyfin etc hosted on my PC but the intent is to move everything over to a VM hosted on this server. Even that is not going to stretch it whatsoever, but I adhere to the maxim that "Overkill is underrated" when it comes to these sort of "large" investments, for just this sort of eventuality.

There is a very strong possibility I take you up on the offer, scooter.

[Zulan]

Zulan, thank you for keeping the Civ 4 scene going. If you hear anything about Ramk please can you keep us informed?

I should ask, what is the challenge with hosting the Natneg servers, and the side loading? What sort of bandwidth do we need to make it work? I'm mindful that I can get a 1.6Gbps down/115Mbps up just by upgrading internet, and it is just as trivial for me to upgrade the last leg of my home network cable to 10GbE. If it's other hardware that is the problem, I will be using an Epyc 7532 (32 core, 64 thread) with 256GB of RAM so I can dedicate quite a bit of resource to it if needed. how does this compare to what is currently in place?

Ref, scooter, I was hesitent to be more forthcoming until I built and tested the used hardware I've bought, but it's working now. I can confirm that I should be able to run a Debian or Mint VM with a graphical interface dedicated as a pitboss host, and keep this separate from my own home network. This means that the pb-watchdog should function without alteration, and this is limited by my capabilities as an admin (that's a pretty big shortcoming).

I will keep you updated on Ramk if I hear anything.

For NatNeg you need a public IPv4 address dedicated to the host (vm) running the server, not only the local router. Well, I can't know for sure, but I highly doubt it would work stable behind nat, no matter the port forwarding. NatNeg is too flaky as is... That is very rare to have in a home network. The address should also be stable given that many players prefer to use a hosts file rather than the patched exe. That's the reason I am rarely updating the server despite the outdated hardware, because I can't keep the IPv4 address.

The hardware and bandwidth is not really the issue. I use a vm with 6 threads of a i7-7700 and 12 GiB memory for all the current PBs, web, lobby, natneg. It is on a 1 Gb symmetrical connection. I have about 10 GiB outgoing 2 GiB incoming in total per month when the watchdog is running properly. PB should also run fine behind proper port forwarding (I do that in docker anyway). There have been some network issues with fragmented UDP packets, mostly for large PBs where the player name list gets very long.

The trouble with running the web server for save-sideloading is just that the setup of properly exposing the right directories is not smoothly automated.

[Tarkeel]

I should ask, what is the challenge with hosting the Natneg servers, and the side loading? What sort of bandwidth do we need to make it work? I'm mindful that I can get a 1.6Gbps down/115Mbps up just by upgrading internet, and it is just as trivial for me to upgrade the last leg of my home network cable to 10GbE. If it's other hardware that is the problem, I will be using an Epyc 7532 (32 core, 64 thread) with 256GB of RAM so I can dedicate quite a bit of resource to it if needed. how does this compare to what is currently in place?

Ref, scooter, I was hesitent to be more forthcoming until I built and tested the used hardware I've bought, but it's working now. I can confirm that I should be able to run a Debian or Mint VM with a graphical interface dedicated as a pitboss host, and keep this separate from my own home network. This means that the pb-watchdog should function without alteration, and this is limited by my capabilities as an admin (that's a pretty big shortcoming).

That is… some serious hardware. That sounds incredibly useful either as a backup to Zulan’s servers or used in conjunction with them. If you need a tester or want another set of eyes on this, let me know.

I'll second this. My expertise is data management, but I do have an eye for overall flow and troubleshooting.

[xist10]

Ramk hat sich gemeldet. Er sieht sich nicht in der Lage, die PB weiter zu betreuen, hat aber Hinweise gegeben, wie das geht.

Link

Ramk has got in touch. He does not see himself in a position to continue looking after the PB, but has given some hints on how to do this.

This is a bit behind the scenes for me, maybe Ramk->Zulan->Rob, but I think, we will follow this discussion too.

[Zulan]

I can confirm basically what the translation said. Ramk has given me some further information on PB management, but he won't be actively involved.

[RefSteel]

I'm really glad to hear that Ramk's well enough to check in with online friends again, and hope that (if health-related) his recovery from here is speedy and complete - or (if other-related) that he continues to enjoy the top-secret, exclusive, intensive mars colony preparation program that he was selected to help put together (or whatever other cool thing it may be)!