ISDG 2012 flame war

[MJW (ya that one)]

Apolyton's forum is now up.

28 times?... wow. Seven the data came from here: http://forums.civfanatics.com/showthread...53&page=98

If you agree with me that the odds are impossible and Memphis didn't cheat my spreadsheet has to be wrong (seven liked it though) anyway and I don't have an argument at all. As you said the results actually happened. So I don't see the point of this but anyway:

Krill, if Memphis was reporting his turns the way he played him then there maybe not enough possibilities for him to get that lucky even if he did cheat. However, he cheated he would have lied and would have much more actions by using workers and stuff. Interleaving combat and worker orders was not banned until "the verdict" at the very end of the game so he could have even just been lying by omission.

Ichabod, the 37 times didn't happen in a row in game. So if he were cheating it would be much easier to get than having to do a single combat. After every turn he gets a refresh. The longest was a streak of 12 wins at the end with and average of ~45% odds. That's one in 14500 which is much easier than 1 in 44 billion.

In fact he gets a refresh every-time he wins a combat. So every-time he loses he can reload and try something different. It won't take that long to get passed the 45% coinflips that happened in the last battle. In order to model how hard it is to get the result using cheating you need Civ4's PRNG and how many rerolls (wasting an RNG by doing something) that Memphis had. I would ask you to back up that Memphis had very limited options if you are really going to push that. Also, as I said before, you would have to check if Civ4's combat engine doesn't have a flaw that makes my spreadsheet wrong.

I said that the other things that Memphis did were not real evidence. Just the odds. If he logged-in and played first thing it would have not be possible for him to cheat. So him doing that stuff makes it possible for him to cheat and him having to log-in and out a lot could have been just making sure that he doesn't mess up following his sandbox. Also the specific PRNG was already known: http://forums.civfanatics.com/showthread.php?t=336088 . So he may only have had to reload for the first two battles or so to find out where he is and then follow the formula so he would have enough time.

[MJW (ya that one)]

Cross-posted

Ichabod's point (and it is a good one) is that even with cheating, where you manipulate the order of the rng rolls, the combat results seen are unlikely, since they rely on unlikely rng rolls. So the combat results don't prove that cheating occurred, just that something unlikely happened.

As I said in my reaction to Ichabod in the game it only happened 12 times in a row not 37. Also each time he loses he doesn't go back to the beginning he just goes back to where he last won.

[SevenSpirits]

Ichabod's point (and it is a good one) is that even with cheating, where you manipulate the order of the rng rolls, the combat results seen are unlikely, since they rely on unlikely rng rolls. So the combat results don't prove that cheating occurred, just that something unlikely happened.

I'm not convinced either way at this point and I'm not sure I want to get further into it. But here's what could conceivably have happened. Suppose you care a lot about these battles for some reason, so you're going to sandbox them. And you figure out that the easiest way to create the sandbox is by saving the game and opening it somehow (I am not familiar with how this supposedly worked). So now you have a completely accurate sandbox. In fact it's so accurate that the RNG seed is the same. Maybe you didn't even intend to cheat until a little while in when you realized that you were getting the exact same results as in the sandbox, I don't know.

So you have your sandbox and you do a battle, and you lose. OK, that was a bad attack. You try a different attack instead and you win. Sounds good! Now you're at a new position in the RNG and maybe the original first attack works this time, so you do that next... and so on. Winning each individual battle, or doing well in it by getting a lot of hits at least, is in itself a common occurrence. The unlikeliness of it is measured as a whole over the entire sequence of combats, and that's the number that MJW is claiming is 69000 times smaller than the odds of getting a royal flush in 5 cards.

Back to our theoretical cheater. Any time they run up against an unacceptably poor situation, they can do something else in the game that consumes a random number, like set a unit to autoexplore. (Granted, I don't think a lot of actions like this are available on a given turn.) So yes, you really can cause a very unlikely event to occur by rolling the dice repeatedly and throwing out all the bad rolls.

Let me throw out a few numbers here for those who didn't look at the spreadsheet. There were 37 combats listed, with average odds of 41.18%. That results in an average number of combat wins of about 15 out of the 37. Sancta got 30 out of 37. So the difference is you just have to turn 15 losses into wins - it's not THAT much work.

In the 500 sims included in the spreadsheet, the number of wins out of 37 ranged from 9 to 22 (again, averaging about 15). That's the short explanation of how unlikely it is to get 30.

[SevenSpirits]

OK, I looked at the source of the numbers and it's apparently a list of every single (reported?) combat initiated by sancta in the game thus far. MJW's sheet says that one in 44 billion legitimate Civ IV pitboss players/teams should start off a game with that good combat luck or better.

There were a lot of numbers thrown around haphazardly in the original thread. "Those are 1 in a million results!" "Well millions of games of civ have been played!" Now I guess we have a more accurate number.

One caveat though: I notice that it looks like these 37 combats were only the vs player combats and did not include a high-odds loss in an attack vs barbs. You have to be careful when you're looking at data like this which is self-selecting because you wouldn't have looked at it if other teams didn't complain. Now you're looking at just a very specific thing, albeit one that encompasses almost all the controllable random actions taken by sancta in the game so far. Throw in the barb attacks and I assume the results would look a little less extreme (but still ridiculously extreme).

Let me put it this way - the data makes it look like we have something like a million-to-one odds that he cheated (supposing you expect the a priori odds of a cheater to be one in a thousand, which is probably naive). But I am concerned about systemic errors in the data collection or analysis (or I guess just a mistake in calculation by mjw). You don't decide that someone cheated without being really sure of your methodology and numbers.

[novice]

You have to be careful when you're looking at data like this which is self-selecting because you wouldn't have looked at it if other teams didn't complain.

On that note, shouldn't some of the combat results leading up to SANCTA being accused of cheating be disregarded, as they are the reason you're looking at the data in the first place?

[SevenSpirits]

You have to be careful when you're looking at data like this which is self-selecting because you wouldn't have looked at it if other teams didn't complain.

On that note, shouldn't some of the combat results leading up to SANCTA being accused of cheating be disregarded, as they are the reason you're looking at the data in the first place?

That would make me a lot happier with it, but they stopped playing at that point. So there isn't any data starting after someone decided to start collecting data but before memphus knew he was accused of cheating and stopped playing.

[novice]

You have to be careful when you're looking at data like this which is self-selecting because you wouldn't have looked at it if other teams didn't complain.

On that note, shouldn't some of the combat results leading up to SANCTA being accused of cheating be disregarded, as they are the reason you're looking at the data in the first place?

This reminded me of "The Science of Discworld" (of all books ), which has a good and entertaining treatise on the use of statistics in science. You can preview every other page here:
http://books.google.no/books?id=Q1x0852i...&q&f=false
(It's surprisingly coherent even with every other page excluded). On page 273 they conclude: "It's illegitimate to include the data that brought the clump to our attention as part of the evidence that the same clump is unusual."

[mackoti]

To bad i didnt posted, back time then but in pbem played against gaspar, lewwyn, seven.. I atacked lewwyn, and i lost i think , 6 batles at 78% knigs aginst LB, all after another, after i lost one batle at 96%, like after to win 3-4 at under 10% odds , one beeing even at under 4% i think - was a chariot against spear, is clear i was in tilt and very mad, about my luck, is this more probable to happen what happen for santa?
Against novice i won i think 6 batles at 50% odds knigs against pikes batles as well i lost 7 batles in row caravels against galeons, all having 33% , so i think this kind of streak can hapen just pray dont be on the wrong end.

[SevenSpirits]

To bad i didnt posted, back time then but in pbem played against gaspar, lewwyn, seven.. I atacked lewwyn, and i lost i think , 6 batles at 78% knigs aginst LB, all after another, after i lost one batle at 96%, like after to win 3-4 at under 10% odds , one beeing even at under 4% i think - was a chariot against spear, is clear i was in tilt and very mad, about my luck, is this more probable to happen what happen for santa?
Against novice i won i think 6 batles at 50% odds knigs against pikes batles as well i lost 7 batles in row caravels against galeons, all having 33% , so i think this kind of streak can hapen just pray dont be on the wrong end.

Yeah, those are WAY more probable. Your caravel example happens about 1 in 17 times. The knights vs pikes is 1 in 64 times.

We are talking here about 1 in 44 billion times.

[plako]

Odds of getting 30 or more wins out of 37 even in coin flip would be ~1:48000 (withdrawal also counted as win). Clearly the odds were on average lower than 50%. If you play with weighted coin giving only 40% odds/round similar odds would drop to ~1:17000000 i.e. Memphis got extremely lucky. Definately in the range of winning in a big lottery. As was case back then, this is not conclusive evidence. Naturally this leads to the tought that almost impossible was made more probable somehow.