- Fun is not measurable, but reflections of it are.
- Determining Proxies in terms of accuracy, precision, and granularity of measurement is important.
- Definitions and importance of Hypotheses and Theories
- Different game analysis methods
Let us talk about some important as well as hard topic to discuss: what is fun? As game designers we forget one important point which is if we do not have fun while playing a game; this does not mean that game is not fun. Have you ever thought of measuring your feelings? I mean, how much are you happy while reading this, for example? Fun is also unmeasurable; but players’ behaviors are observable.
Mr. Szomolai points one important term which is called as proxy. In general usage, proxy simply means medium between you and server. It serves as a waiter that carries you a meal (website). In game business, this term has similar analogy. Everything which is measurable like player actions or expression are called as proxies.
You do not need to know what fun really is, still you can observe proxies which are reflections of fun. Let us discuss how to use these proxies.
First, we need to understand what hypothesis is. We all create theories in our daily life; for example, we can think that if we lie to our friend and s/he understands it; then s/he will be mad at us. This is one example of hypothesis: if ‘a’ happens, then ‘b’ will always follow it.
Theory is group of hypotheses and these hypotheses are interconnected and consistent with each other. Interconnection between them means that if you know the relationship in one hypothesis, then you can make inferences about another hypothesis.
In practice, this relationship between ‘a’ and ‘b’ can not be fully understandable: You try to get as closer as possible to the reality. Then the ladder continues as:
- Better relationship leads you to better hypothesis.
- Better hypotheses lead you to better theory.
- Better theories lead you to happier players.
Mr. Szomolai explains their point as if hypothesis is not improving their theory, then it is not good enough. If it is not good enough, they start with a new one. If it is good enough, then they update their theory. Once they updated their theory, then they start with new hypothesis based on their theory; and loop continues.
When you have hypothesis, it is time to measure it with valuable indicators: proxies. If you talk about scale, then you deal with numbers. But do you really want to measure numbers? Mr. Szomolai explains measurement as approximation of one phenomenon to another; till you get to an approximation that you really understand intuitively, or which is already part of your inner theory. Proxy is every single example of these phenomena that different game designers may understand differently. Mr. Szomolai indicates very valuable point: Junior game designers argue about whose theory is true; however, seniors will experiment them and measure outcomes. I want to quote his words as exactly, here:
“A great theory with bad measurements will lead to worse results than a weak theory with great measurements. Bad measurements will improve theory only a bit (if at all) while good measurement can move theory by miles (this is iteration once again, you probably heard of that before).” – Mr. Roman Szomolai
We talked about the importance of proxies and measurement; let us discuss about parameters that we should watch.
Accuracy: Degree of truthfulness. Increases with repetition.
Precision: Difference in small parts. Requires focus on details.
Granularity of measurement: Do you need to measure it in kilograms or grams?
Here is a good example from Mr. Szomolai, to fully understand the difference between these terms:
“Imprecise but accurate weight will show +/- 25% of the real value each time you step on it. Scale with rough granularity will show only kilograms and precise but inaccurate weight will show exact numbers each time but it will be exactly 8 kilograms off. In each case you will not know exactly what you wanted to know but you will be closer to the truth than before.” – Mr. Roman Szomolai
Then why don’t we always have the best precision, accuracy, and granularity of measurement? It is about time, money, and know-how. We should make our decisions between them as accurate as possible according to the situation that we are in.
Designing is not simply “I know how to do it, and I want it in this way!”. To create a theory, we need to know how every part of our game works; and we need to have a theory about how they affect player experience. Good game designers should be doing progressive design: they do not say “we know everything”, but continuously update their theory.
We -as game designers- should always be checking for noise while considering data that we collected. Looking for counter examples will make us aware if the data is as accurate as we want or not.
We have different analysis methods while measuring proxies. Let us discuss them:
Heuristics: This is based on your instincts as “player”. You are, probably, already playing games and observing what else do you want to happen, or what things did you dislike in the game. You can use this way with your friend: you make him/her play the game and ask his/her opinions.
Advanced way of improving your heuristic skills is continuously playing different games and creating patterns from them.
Playtest: This is cornerstone of game development. No game can be thought without playtesting. This is also can be done by yourself or your friends. You should do this as much as possible.
Mr. Szomolai points that there are several advanced ways of doing this like Kleenex playtesting, company wide testing, focus group testing, etc. These methods should be analyzed in-depth to choose accurately.
Discursive Analysis: This type of analysis is based on players’ feedback and game dynamics. You may know that dynamics are created from mechanics by players. Their verbal feedbacks are so important for us to criticize our decisions.
Advanced ways of doing this analysis is using machine learning to analyze as much of feedbacks as possible; and directly communicating with players -by community manager for example- about their opinions and transferring them to developers.
Thought Experiment / Model: In this model, we do not experiment all the components of game, but some part of it: for example, multiplayer matchmaking mechanics of the lowest rank. Then we can make deduction about the rest of it. You can create sub-parts of your whole game system then ask yourself what-if questions. The advanced version of this approach is making this process with machine learning prediction.
Domain Analysis: This analysis type is based on your competitors in your genre. What commonalities or differentiations do their games have? You can simply do this playing and noting these points manually. The more advanced way of doing this includes listing all games and relationships in your genre, watching interviews with creators, doing in-depth analysis with specific frames.
Quantitative Analysis: This is the area where you play with numbers: KPIs. You can simply do this by feedbacks from players in specific part of game. Advanced way of doing this is working with data companies.
Interview / Discussion: This analysis is done directly between players and developers. You ask players about their opinion, then just listen them without defending your game; then observe their comments. Advanced version of this analysis is done with wisely chosen questions and qualified interviewers.