One of the criticisms by some reviewers about games from Stonemaier Games is that they are unbalanced, not play tested enough.
But how fair a criticism is this?
For this post I’m going to look at Tapestry. Which is a game that has this accusation made against it. And recently had a pack of rebalanced civilisation mats released. I may also use the odd game from their catalog to illustrate a point.
Before I go any further I need to give a disclaimer of some sort. Firstly I have no idea how many play testers, how much play testing was done, or what the play test process is at Stonemaier games. And I’ve made no attempt to find out. I’m also not a mathematician or statistician. So there are likely major flaws in my maths and logic. Please feel free to correct me in the comments. For this post I’m going to ignore solo play testing because I don’t play solo, and I don’t want to look up the solo rules. Oh and I have been accused of being a bit of a Stonemaier Games fanboy.
In Tapestry and it’s three expansions we have 41 civilization mats broken down as follows:
- Tapestry – 16 civilization mats
- Plans & Ploys – 10 civilization mats
- Arts & Architecture – 5 civilization mats
- Fantasies & Futures – 10 civilization mats
So to look at how feasible and reasonable it is to play test Tapestry and its expansions I will be working out the number of combinations of civilization mats. For this exercise the order doesn’t matter. Hence combinations instead of permutations.
To work out the number of possible combinations of civilization mats I’m using the following Binomial coefficients formula:
Where n is the set size or in our case number of civilization cards, and r is size of sets we are choosing, aka player count for us.
What follows is the number of combinations for Tapestry and its expansions based on player count.
As expected it’s going to be a lot easier to play test all the combinations once at the lower player counts than the higher ones.
But you need to repeatedly play these combinations to make sure that what would appear to be an unbalanced civilization mat is in fact that, and not just appearing to be so due to other factors.
I would argue that at the higher player counts (4 and 5) it’s unreasonable to expect all the combinations to be play tested. Just from a time and number of play testers basis. Let alone any monetary factor. I’m almost tempted to add in 3 players with the expansions to this statement as well.
So how would you even attempt to spot unbalanced or as some like to call them broken or over powered civilization mats?
You could use feedback from the lower player counts to focus in on particular civilization mats. However this over looks the fact that certain civilization mats might be better against others at lower player counts, but when played in higher player counts actually weaker or on par with others. Or that certain civilization mats are better at higher player counts but poor at lower counts.
Plus in all of this we aren’t even factoring in the random elements of the game such as tapestry cards.
In reality play testing will not just focus on one player count but be a mixture of all of them.
I think the big take away from this is it’s just not possible to play test every combination at all player counts. Just on the base game approximately 8000 games that would need playing to do it once.
There has to be a certain point where you say “we’ve tested as much as we can”. Or the game would never get released or be so expensive to cover the increased testing costs it won’t make any money.
It’s only once the game gets out in the wild and in the hands of lots of players that things start to come to light. I do like that Stonemaier capture play data and feedback this into the design and provide updates to the games.
I can see a follow up argument that some of the civilization mats were obviously unbalanced, just too powerful. That I don’t have a response to. I’d need more info about the companies testing and feedback.
What are your thoughts on the feasibility of testing a game?
- Elwes, R. (2010) Mathematics 1001: absolutely everything that matters in mathematics. London: Quercus.