Mabinogi Player Survey: Planning

• Kelvas wrote on 2014-09-15 23:47
  [UB]

  Quote from Rydian;1247562:

  Okay, current question list has 9, I think a nice 10 would be best.
  
  1 - How long have you been playing Mabinogi?
  2 - What's your total level?
  3 - What's your main combat style?
  4 - Have you enchanted your equipment, other than using the beginner enchants?
  5 - Do you buy reforges?
  6 - Have you ever used red or blue upgrade stones for special upgrades?
  7 - Do you do any production skills? (Blacksmithing, Magic Craft, etc.) (If so) which ones do you do?
  8 - Which generation quests have you beaten?
  9 - About how much gold do you have saved up right now?
  
  For the gold question, "I don't want to say" could be an acceptable answer. For average time played a day, that could be extrapolated from total time and level... or just straight-up asked if there's not a good question to fill it to 10.

  
  
  That's a good improvement from the first set of questions. The few problem I have with the new set of questions are as follows
  4. One could simply slapped a giant, imp, and a strider, and do a talent like archery or fighter, and give a yes. Like what others have suggested, I would only consider a "yes" to be a complete, or near complete enchanted set that is actually relevant to their talent choice.
  5. Same could apply for reforges. One could have simply bought 5 reforging tools, used them to get absolutely nothing and answered yes. Even in my case, the only two reforge I have at the moment that's relevant to my talent choice is a hail 18 max on my hermit, and a firebolt max 9 on my fire savage wand. I don't think I have to right to answer with a simple "yes", at least for the time being.
  6. Same as above. One could have just upgraded their weapon to s/r 1 and answered yes to that question.
  
  Overall, I think the degree of how far one takes certain aspect should be asked as oppose to a simple yes or no question. I'm also very interested in the result of this. Too bad I'm too busy in school to contribute too much to this. [/UB]
• Zekkii wrote on 2014-09-16 04:16
  If I had one suggestion, I would ask players if they possess an AOE summon pet but given that most players have a perseus from completing G18E1~10 it would result in most players being forced to say yes. If you deem this to be an appropriate question I would place players into the zones of 0, 1, 2~4, 5~10, and 10+. I feel AOE pets are sadly one of the major aspects of a strong player in meta and would be interested to see how most players fare.
• Rydian wrote on 2014-09-16 04:48

  Quote from Osayidan;1247572:

  I think "how often do you play?" or some quantification of time spent on the game on average should be in there. Some people who have been playing 1/5 or even less of the time I have been playing are 2 or 3 times my total, so when they started and total level doesn't give a good idea.

  That might be a 10th, but I'd consider that a lower-priority question as it will vary along with the time of year so it might not be an accurate representation when somebody goes to look at the gathered data at another point in time.
  
  

  Quote from Kelvas;1247585:

  [UB]
  
  That's a good improvement from the first set of questions. The few problem I have with the new set of questions are as follows
  4. One could simply slapped a giant, imp, and a strider, and do a talent like archery or fighter, and give a yes. Like what others have suggested, I would only consider a "yes" to be a complete, or near complete enchanted set that is actually relevant to their talent choice.
  5. Same could apply for reforges. One could have simply bought 5 reforging tools, used them to get absolutely nothing and answered yes. Even in my case, the only two reforge I have at the moment that's relevant to my talent choice is a hail 18 max on my hermit, and a firebolt max 9 on my fire savage wand. I don't think I have to right to answer with a simple "yes", at least for the time being.
  6. Same as above. One could have just upgraded their weapon to s/r 1 and answered yes to that question.
  
  Overall, I think the degree of how far one takes certain aspect should be asked as oppose to a simple yes or no question. I'm also very interested in the result of this. Too bad I'm too busy in school to contribute too much to this. [/UB]

  If the degree they take it to is rounded out to certain ranges that'd be more acceptable, but...
  
  On the one hand, the main curiousity of this survey is not to determine how strong players are in combat. I'm interested in how much of Mabinogi's content has been explored and/or beaten by which percentage of the player base. Combat is a big part of the game, but I'm more interested in finding out how many players actually use specific mechanics and methods rather than what ends people have reached in terms of numbers. Like, if somebody has only placed 3 rank A or less enchants on their gear that weren't from beginner scrolls, that still means they've checked out enchanting at least a little.
  
  Though on the other hand, as an example splitting the enchant answers into "I have never enchanted", "I've tossed some whatever enchants on I happened to get", and "I make sure to enchant all my of gear with the best enchants available" would definitely provide nicer info, though would be a little more work on the volunteers.
  
  Not sure which method to go for. The second one (including ranges) seems better... but I'll probably have to work up some compensation to make it worth the time of people doing the surveys.
  
  

  Quote from Zekkii;1247647:

  If I had one suggestion, I would ask players if they possess an AOE summon pet but given that most players have a perseus from completing G18E1~10 it would result in most players being forced to say yes. If you deem this to be an appropriate question I would place players into the zones of 0, 1, 2~4, 5~10, and 10+. I feel AOE pets are sadly one of the major aspects of a strong player in meta and would be interested to see how most players fare.

  I don't think this, specifically, is really that important for long-term data gathering, it'd be neat to see, but...
  
  Perhaps how many pets they have in general would be a better question and provide more useful info.
• EndlessDreams wrote on 2014-09-16 04:55
  So you want to make the survey "scientific"? How do you address the following problems with the surveys:
  
  Random Sampling is more important than just Large Sample Size. How do you propose to have random sampling here?
  
  Large Sample Size? How large will it be?
  
  Where are the people who are giving out the surveys going to go? Would their choice introduce additional bias into the survey?
  
  What about people who just don't want to answer your surveys? Are they not going to count toward the Mabinogi population?
  How are you going to motivate people to even answer your surveys from random strangers?
• Zekkii wrote on 2014-09-16 05:22

  Quote from Rydian;1247653:

  I don't think this, specifically, is really that important for long-term data gathering, it'd be neat to see, but...
  
  Perhaps how many pets they have in general would be a better question and provide more useful info.

  
  Well I understand, there's lots of great questions to ask. I look forward to the results, best of luck.
• Kiru wrote on 2014-09-16 08:30
  maybe ask what talents they mastered if any?
• Rydian wrote on 2014-09-17 05:11

  Quote from EndlessDreams;1247655:

  Random Sampling is more important than just Large Sample Size. How do you propose to have random sampling here?

  Asking in various locations on various servers and channels (and I mentioned time zones already I think). How many places/channels/servers exist in total is set, but how many per volunteer will depend on the number of volunteers.
  
  Dunbarton and the stonehenges will be set locations of course due to the traffic, but Tir, Filia, and Vales should be scouted as well, and various dungeons and junk too. I figured this was specific info that concerned the volunteers moreso than the design of the project so I wasn't going to include it in this phase.
  
  

  Quote from EndlessDreams;1247655:

  Large Sample Size? How large will it be?

  Until everybody is sick of gathering responses, or until the project hits a point where volunteers are notably running into more people they have approached than people they have not.
  
  

  Quote from EndlessDreams;1247655:

  Where are the people who are giving out the surveys going to go?

  Uh, covered above?
  
  

  Quote from EndlessDreams;1247655:

  Would their choice introduce additional bias into the survey?

  Various hotspots of populations would be laid out for volunteers to check out. It's not like the entire Mabi world can be covered, but there's a number of locations (cities, NPCs) that the majority of players pass by at least now and then.
  
  

  Quote from EndlessDreams;1247655:

  What about people who just don't want to answer your surveys? Are they not going to count toward the Mabinogi population? How are you going to motivate people to even answer your surveys from random strangers?

  Uh, you do realize that it's not like we need to get everybody in the game to answer, right? That's why phrases like "sample size" and "sample diversity" use the word "sample", because in cases like this it's unreasonable to try to get everybody. We'll just get as many people as we can from as many places as are reasonable. to get the best estimates on the general population that we can.
  
  You know, just like voting and other things that do this, which run into the same potential issues you mentioned...
• EndlessDreams wrote on 2014-09-17 05:37

  Quote from Rydian;1247817:

  
  Uh, you do realize that it's not like we need to get everybody in the game to answer, right? That's why phrases like "sample size" and "sample diversity" use the word "sample", because in cases like this it's unreasonable to try to get everybody. We'll just get as many people as we can from as many places as are reasonable. to get the best estimates on the general population that we can.
  
  You know, just like voting and other things that do this, which run into the same potential issues you mentioned...

  
  
  Of course you don't need everyone to answer to get a representative sample sized. However, for a sample sized to be representative of the population, it has to have at least both of the following factors:
  
  1) Sufficient number size
  2) Randomly selected
  
  Voting does not fall under the same category as creating a scientific survey. On the same note, a lot of surveys done on polls after elections are very biased because of how their sampling is done.
  
  Only getting responses from only the people who are willing to give you responses is a representation of the general population.
  
  You will only get a representation of a handful of people. That is it. If that is your attention, it isn't really science you are doing.
  
  Even if we are going in Dunbarton itself, any single volunteer cannot be in all the all channels at once. The volunteer will have to make a decision in choosing what channel to be in. There is also the decision to be what part of Dunbarton will the volunteer will be in.
  
  In addition, there will be the decision on who the volunteer will attempt to speak to. Don't forget that the volunteer will have to make a decision on when to log on, and how long the person is going to stay. All these decisions are introducing bias into the survey.
  
  There are also other factors that will influence volunteer responses such as the volunteer's appearance, how the questions are phrased, the order of the questions, how long the volunteer is asking the questions, how well-known or not known is the volunteer. The list goes on. Each of these factors introduces more and more bias into the survey.
  
  Summary: Your survey won't represent the general population at all. It cannot be called scientific either.
• Rydian wrote on 2014-09-17 19:42

  Quote from EndlessDreams;1247819:

  Voting does not fall under the same category as creating a scientific survey.

  Taking a sample of opinions/information from a population and extrapolating that to the whole is definitely something they have in common.
  
  

  Quote from EndlessDreams;1247819:

  Summary: Your survey won't represent the general population at all.

  Feel free to think that and disregard the collected info at the end if you wish, nobody's making you pay attention to this.
  
  

  Quote from EndlessDreams;1247819:

  It cannot be called scientific either.

  You're confusing the words "scientific" and "comprehensive".
  
  The word "scientific" is defined as...
  http://www.merriam-webster.com/dictionary/scientific
  

  of, relating to, or exhibiting the methods or principles of science

  
  And looking up "scientific method", we find...
  http://www.merriam-webster.com/dictionary/scientific+method
  

  principles and procedures for the systematic pursuit of knowledge involving the recognition and formulation of a problem, the collection of data through observation and experiment, and the formulation and testing of hypotheses

  
  
  Whereas "comprehensive" is...
  http://www.merriam-webster.com/dictionary/comprehensive
  

  covering completely or broadly : inclusive

  
  
  I know from the beginning that collecting info from every Maginogi player in total is impossible, and that limiting it to every single still-active player isn't feasible due to the time and effort required. That's why I plan from the start to try to cover as wide of a sample range as I feasibly can, knowing I'll need volunteers to help get enough info for this to be useful.
  
  The plan is to...
  
  1 - Outline a method for gathering data and what data to gather (this thread).
  2 - Obtain many samples over a period of time from varied locations.
  3 - Compile the samples into something that can be sorted and analyzed easily.
  4 - Present these samples and conclusions based on the proportions and relations of answers.
  
  Seems pretty scientific to me. :P
  
  As for specific hypothesis (if the lack of those is what you're concerned on), I've stated a few times in previous threads that people seem to have a misunderstanding of just how many users actually enchant their gear, which generations are done, how strong players in general are, and things like that. This data, once gathered, can prove or disprove some hypothesis even without asking every player (as some statements people use are overly broad in the first place).
• ironwoman wrote on 2014-09-17 20:15
  EndlessDreams has a point in this.
  
  I'm not going to go on about whether your survey will represent the population or not because this can vary greatly from an interviewer to another.
  There may be bias for one thing. For example, like you said, this is about how correctly the nationers describe people (which I'd say is pretty damn accurate, considering we've been playing fo awhile and most of us were in that situation, but you want to do a survey, go ahead and do your thang). Your arguments consistently show that you believe that most of the players are like you (more enchantless than not), and not on the "strong" side. Remember that this may be what you want to believe. Meaning when you go interviewing people, you might unconsciously pick out players that seem to be fitting under that category from the appearance. This is the part that you need to be very careful about. If this becomes the case, then survey is rather incorrect and can give out false results. If bias is involved, then the survey is no longer a correct representation of the general population.
  
  That being said, I don't think the survey should be done in-game at all. What I would do is use the game to advertise the survey, and spread the word around in other Mabinogi communities, like Mabinogi World, MabiMods, Nexon Forums, etc.
• Kelvas wrote on 2014-09-17 20:29
  [UB]I also think this should be done online, where it's anonymous. Not sure if this is the case, but some players may be pressured to answer with a slightly higher range for a question.
  Another question that could be asked is the amount of stat points they have that's most relevant to their primary talent choice, heck, or multiple talents that are primarily used. I think this would also allow the data to be more accurate, since the a large portion of investment can come from stats. For example, someone who mains archery could have good gear, but insufficient dex due to their incomplete life skills.[/UB]
• EndlessDreams wrote on 2014-09-17 22:31

  Quote from Rydian;1247892:

  
  The plan is to...
  
  1 - Outline a method for gathering data and what data to gather (this thread).
  2 - Obtain many samples over a period of time from varied locations.
  3 - Compile the samples into something that can be sorted and analyzed easily.
  4 - Present these samples and conclusions based on the proportions and relations of answers.
  
  Seems pretty scientific to me. :P
  

  
  
  None of the steps you just listed tells us how you going to get a non-biased sample to give credibility to your survey.
  
  Just because something is scientific doesn't mean their statistics and experiment is not biased. A lot of research/studies are just simply ignored/thrown out because of how biased their experiments are and/or their statistics are just biased.
  
  

  Quote from ironwoman;1247897:

  
  That being said, I don't think the survey should be done in-game at all. What I would do is use the game to advertise the survey, and spread the word around in other Mabinogi communities, like Mabinogi World, MabiMods, Nexon Forums, etc.

  
  
  But if you do that, you will get people who only:
  1) People who feel strongly about the game enough to visit AND participate in reading the forums.
  2) Feel strongly enough to give a response back.
  
  Both factors are just adding layers to bias.
  You will be leaving out people who don't do the above 2 factors, basically.
• Rydian wrote on 2014-09-17 22:50
  Alright, so here's the current proposed question list, along with the ranges that an answer will be put into if applicable.
  
  

  1. How long have you been playing Mabinogi?
     [SPOILER="Responses"]Less than 6 months.
     Less than 1 year.
     1-2 years.
     2-3 years.
     3-4 years.
     4-5 years.
     5-6 years.[/SPOILER]
     
     
  2. What's your total level?
     [SPOILER="Responses"]Less than 100.
     100-500
     500-1000
     1000+
     2000+
     3000+
     4000+
     5000+
     6000+
     7000+
     8000+
     I don't want to tell.[/SPOILER]
     
     
  3. What's your main combat style?
     [SPOILER="Responses"]Melee - Standard
     Melee - Lances
     Ranged
     Magic
     Alchemy - Battle
     Alchemy - Construct (Should this be here?)
     Fighter
     Music (Should this be here?)
     Puppetry
     Dual Guns
     Ninja[/SPOILER]
     
     
  4. Do you enchant your equipment?
     [SPOILER="Responses"]Yes, everything I possibly can.
     Yes, a few things here and there.
     Yes, if the beginner enchants count.
     No.[/SPOILER]
     
     
  5. Do you buy reforges?
     [SPOILER="Responses"]Yes, with NX.
     Yes, with gold.
     No.[/SPOILER]
     
     
  6. Have you ever used red or blue upgrade stones for special upgrades?
     [SPOILER="Responses"]Yes, everything I can.
     Yes, once or twice.
     No.[/SPOILER]
     
     
  7. Do you do any production skills? (Blacksmithing, Magic Craft, etc.) (If so) which ones do you do?
     [SPOILER="Responses"]Tailoring
     Handicraft
     Blacksmithing
     Cooking
     Potion Making
     Carpentry
     Hillwen Engineering
     Magic Craft
     Nothing[/SPOILER]
     
     
  8. Which generation quests have you beaten?
     [SPOILER="Responses"]G1 (Advent of The Goddess)
     G2 (Paladin)
     G3 (Dark Knight)
     G7 (Secrets of Irinid)
     G8 (Dragon)
     G9 (Alchemist)
     G10 (Goddess of Light)
     G11 (Sword of The Gods)
     G12 (Return of The Hero)
     G13 (Hamlet)
     G14 (Romeo & Juliet)
     G15 (Merchant of Venice)
     G16 (Macbeth)
     G17 (Shamala & Nightmare)
     G18 / Saga
     None of them.[/SPOILER]
     
     
  9. About how much gold do you have saved up right now?
     [SPOILER="Responses"]Less than 100K
     Less than 500K
     Less than 1 million.
     1-2 million.
     2-3 million.
     3-4 million.
     4-5 million.
     5-6 million.
     6-7 million.
     7-8 million.
     8-9 million.
     9-10 million.
     Over 10 million.
     I don't want to say.
     [/SPOILER]

Is the number of possible results too big? Is it too small? Did I miss a possible answer?

The contenders for question #10 are...
- How many pets do you have?
- Which talents, if any, have you mastered?
- How often or how much do you play?
I'm leaning towards #1 or #3, but does anybody else have another question, or some reason #2 would be insightful?

Quote from ironwoman;1247897:

Your arguments consistently show that you believe that most of the players are like you (more enchantless than not), and not on the "strong" side.

I don't think you understand where I'm coming from when I argue those, though.

You appear to believe that everybody enchants. I tell you this is not true. This is not the same thing as me saying "nobody enchants".

Some people expect players to be able to reach 300+ max damage at total level 150 (yes, I still remember that thread), I say that's not a reasonable assumption. This is not the same thing as me saying "it's not possible and nobody does it".

Some people state that nobody runs dungeons, ever... do I even need to provide a backing for this one? :P

There are a lot of Mabinogi players and a range of play styles. I'm tired of people going back and forth on what players actually do or do not take part in or have access to, so I decided to start up this project so we could try to determine how things are actually laid out (which could easily prove any of the sides wrong because, as far as I know, this is the only project like this that's been done in recent times).

Besides, if this was actually just to prove you wrong (which would be a crapton of work for just that, srsly come on now), then I would include things like "What's your max character damage" and "what's the maxed buffed hit you've done" like people mentioned, but I don't have them in the question list because...

A - This is me actually wanting to gather this data because it could be useful, not to take sides in an old argument.
B - I don't think max hit data is really useful anyways, as that will change over time as new skill sets and features get added to the game and faster ways of leveling become available. Thus me focusing on player progression in general (not just damage numbers) like generations complete and which game features people actually use.

Quote from ironwoman;1247897:

Remember that this may be what you want to believe. Meaning when you go interviewing people, you might unconsciously pick out players that seem to be fitting under that category from the appearance.

That's why I want volunteers, because I know there could be unconscious bias in asking. I'm on the smallest-population server anyways (Ruairi) if that helps.

You know... if it puts you (and others) at ease, I could do none of the data gathering myself, leaving it up to volunteers only (putting the raw spreadsheets of answers up for public view at the end of course). Would that make people feel better about this project?

This would further delay the start of the data gathering though, as I'd want to save up some more compensation (NX and/or in-game items) for the volunteers since they'd be doing all the in-game work without my help... but if it's a way to get more people involved I don't mind.

Quote from ironwoman;1247897:

This is the part that you need to be very careful about. If this becomes the case, then survey is rather incorrect and can give out false results. If bias is involved, then the survey is no longer a correct representation of the general population.

Thus me getting multiple people on multiple servers to contribute, so it's not a glorified process of me going "Hey these two people I asked say X, haha on you".

Quote from ironwoman;1247897:

That being said, I don't think the survey should be done in-game at all. What I would do is use the game to advertise the survey, and spread the word around in other Mabinogi communities, like Mabinogi World, MabiMods, Nexon Forums, etc.

Asking on the forums and groups outside of the game would be including the same type of bias that causes the conflicts to begin with.

The reason these arguments come up is because of niche users, groups of like-minded people, people with the same habits/hobbies, and users of similar outlooks grouping up. Tumblr users, various forums, wiki communities, these groups tend to contain people with similar habits and outlooks, while we want to approach the active player base in general.

This survey is going to be getting results about active players in the game. So to make sure we're doing this, we will be asking active players in the game. Anything else is contrary to the point of this survey (finding out what's true in-game, not just within communities) and is unacceptable. This is one point I'm going to be a hard-ass on, it's one of the core guidelines for a reason.

Quote from Kelvas;1247898:

[UB]I also think this should be done online, where it's anonymous. Not sure if this is the case, but some players may be pressured to answer with a slightly higher range for a question.

Volunteers could simply state that the survey is anonymous and names won't be recorded to try to avoid that type of reaction.

Quote from Kelvas;1247898:

Another question that could be asked is the amount of stat points they have that's most relevant to their primary talent choice, heck, or multiple talents that are primarily used. I think this would also allow the data to be more accurate, since the a large portion of investment can come from stats. For example, someone who mains archery could have good gear, but insufficient dex due to their incomplete life skills.[/UB]

I don't really think this is that important though, like it'd be more "Oh that's neat" and less "Oh well we can see that this percentage of players blah blah and from this we should make sure to blah blah when writing guides".

Quote from EndlessDreams;1247915:

None of the steps you just listed tells us how you going to get a non-biased sample to give credibility to your survey.

Just because something is scientific doesn't mean their statistics and experiment is not biased. A lot of research/studies are just simply ignored/thrown out because of how biased their experiments are and/or their statistics are just biased.

Ah, if it's about that, would you prefer I make sure that I have the guidelines for the data gathering outlined before moving onto gathering volunteers?

Like, listing the locations people should check routinely and making sure to tell people to try on multiple channels, and at different times.