Mabinogi Player Survey: Planning

• EndlessDreams wrote on 2014-09-18 03:21

  Quote from Kelvas;1247925:

  
  That's why I want volunteers, because I know there could be unconscious bias in asking. I'm on the smallest-population server anyways (Ruairi) if that helps.

  
  
  Volunteers are human too. They will have the that unconscious bias in asking as well. In order to avoid that unconscious bias, you will have to find a way to randomly choose the people you are asking that is outside of the volunteer's control. Pretty hard when you factor in the fact you are dealing with other people and people can just refuse to deal with your survey.
  
  

  Quote from Kelvas;1247925:

  
  This survey is going to be getting results about active players in the game. So to make sure we're doing this, we will be asking active players in the game. Anything else is contrary to the point of this survey (finding out what's true in-game, not just within communities) and is unacceptable. This is one point I'm going to be a hard-ass on, it's one of the core guidelines for a reason.

  
  
  How do you know if someone is active? What does it mean to be active?
  
  

  Quote from Kelvas;1247925:

  
  Ah, if it's about that, would you prefer I make sure that I have the guidelines for the data gathering outlined before moving onto gathering volunteers?
  
  Like, listing the locations people should check routinely and making sure to tell people to try on multiple channels, and at different times.

  
  
  Yes, restricting locations and restricting times is reducing the amount of variables and some bias in choosing the time/place for surveys. Of course, you will have to restructure your survey goal to be about those areas/time instead of "general population of Mabiongi", whatever that means.
  
  

  Quote from Kelvas;1247925:

  [UB]
  Why not both? We can just have both an online questionnaire and a few people running in-game asking random players questions. Most sane people would recognize the same set of questions, and will not double up, so having the same data collected from the same person shouldn't be a huge issue.
  [/UB]

  
  
  

  Quote from Rydian;1247928:

  One concern there is that people may submit their answers twice on purpose thinking it was for a raffle, or that only one of the methods counted, or "just to make sure they take note of my answers" or something like that. If there's only one collection method it could cut down on that, at least a bit.

  
  
  The people who submits twice feels really strongly about the survey. It still leaves the people who don't care about the survey out. That is a form of bias.
• Rydian wrote on 2014-09-18 04:33

  Quote from EndlessDreams;1247972:

  Volunteers are human too. They will have the that unconscious bias in asking as well. In order to avoid that unconscious bias, you will have to find a way to randomly choose the people you are asking that is outside of the volunteer's control. Pretty hard when you factor in the fact you are dealing with other people and people can just refuse to deal with your survey.

  Because the only surveys and stats that mean anything and can be trusted are those run by a controlling entity that has all the info about the subjects already and can have a subset queried via a pseudo-RNG with no human intervention?
  
  We don't have access to the player account database, so players will have to be asked.
  
  I mean, unless you can PM me info on some magical public database I can query (SQL, MS access, etc.), or if I've been living under a rock and there's some universal brain-scanning device I can pick up at Walmart.
  
  

  Quote from EndlessDreams;1247972:

  Yes, restricting locations and restricting times is reducing the amount of variables and some bias in choosing the time/place for surveys. Of course, you will have to restructure your survey goal to be about those areas/time instead of "general population of Mabiongi", whatever that means.

  I planned to outline methods, but I didn't think people would care about those specifics unless they were actually looking into volunteering (Phase 2, so they'd know how much work they're expected to put in and what they're doing) or the results (Phase 4).
  
  You gonna' complain that I don't have the results yet when we haven't finished deciding on the question list?
  
  I mean come on now. I can understand wanting to make sure a survey is done properly and I'll humor your questions for a bit, but by this point you're taking it too far and expecting too much when we have reality to deal with and no steps have been taken (the bolded sentence above).
• EndlessDreams wrote on 2014-09-18 06:38

  Quote from Rydian;1247985:

  Because the only surveys and stats that mean anything and can be trusted are those run by a controlling entity that has all the info about the subjects already and can have a subset queried via a pseudo-RNG with no human intervention?
  
  We don't have access to the player account database, so players will have to be asked.
  
  I mean, unless you can PM me info on some magical public database I can query (SQL, MS access, etc.), or if I've been living under a rock and there's some universal brain-scanning device I can pick up at Walmart.
  

  
  
  Even if you have the whole database, what does that have to do with Random Sampling?
  But you are slowly understanding the point. Maybe.
  
  Your survey is fundamentally flawed because it isn't based on Random Sampling. You cannot get an accurate, non-biased representation through non-random sampling, and your way of getting data (asking players in which players can refuse) is an example of biased sampling.
  
  This is the one of the best example of when sampling went wrong: Landon wins US presidency in 1936!
  
  Standards too high? I didn't know striving for a accurate, non-biased representation sampling was consider high.
  
  The more bias a study can remove, the stronger the study.
  On a side note, this is why double-blind randomized trials are the golden standard in conducting very strong scientific studies.
  
  I never said once that you couldn't do your biased study. You are free to do your biased study.
  
  However, if you going to put it on the public forum for people to see (and comment on), people should know the flaws in the methods of your study.
  
  Not everyone took statistics classes, or even remember them. After all, it is really easy to fool people who don't know much about statistics.
• ZeroPantz wrote on 2014-09-18 17:03
  I know there is no way to avoid some bias in this like Endless said. However, just like voting for a new president or something, there are people who also choose not to vote at all, just like there'd be players who choose not to answer the questions. Playerbase wise that shouldn't matter because I am pretty sure people of all kinds of levels etc will either refuse or cooperate. The danger in this is if too much players refuse it may make the volunteer discouraged and turn to friends/guildies which narrows down the scale of the investigation.
  
  But regardless all of the people part of the voting for a president as I just used as an example, are informed though and have the choice to vote or not so you'd still get all the results from the people willing to do it. If you have but one person on a server whose only online at certain times, you're going to leave a lot of players out that would have been willing to cooperate.
  
  I was going to suggest multiple volunteers on each server to make sure you cover as much players as possible, but then you'd risk someone asking the same player the questions. To avoid that recording usernames would work. It won't harm the anonymosity of the players. Basically see it as a tally system, where when a players gives a certain answer, we put a tally behind the answer given.
  
  And in a seperate list we would write down the usernames, so we can see whose been asked, but there is no way for us to get back to knowing who gave what answer due to the tally system.
  
  Perhaps it is also a good idea to have volunteers open a dungeon advertisement such as: [Wanted]ppl2AnswerMabiSurvey- NM Or something. So everyone online at that moment knows about it, and could choose to note the volunteer to answer the questions. Or perhaps it would be even better to make an online survey and link players that do respond to the party advertisement, to it.
  
  However the problem is that I fear not much people would actually even respond to the party advertisement or would be willing to go do the survey. There is, in the end, nothing in it for them, many players are busy in the game and don't feel like taking the time for that, I'd assume. Like that it'd end up more biased again because the volunteer would result in asking his/her friends and guildies to go do the survey at the lack of cooperation from the playerbase they are not friends with.
  
  All in all the way to actually excecute this in the best and accurate way is pretty tricky. But like I said I'd be willing to help out regardless, even if the results may not be entirely accurate due to the outcome being somewhat biased, I am still pretty curious to see the result. In the end, I believe, most scientific results are always somewhat biased since they are only done on a small group of people. Miss World is also not really Miss World because there is no way anyone would have been able to compare her to every single woman in this world.
  
  
  I was also wondering whether or not it'd be a good idea to ask about total exploration level. After all, it would be interesting to see how many players actually work on that. Especially in comparison to the time they've been playing/total level.
• Rydian wrote on 2014-09-19 01:35

  Quote from EndlessDreams;1248002:

  Even if you have the whole database, what does that have to do with Random Sampling?
  But you are slowly understanding the point. Maybe.
  
  Your survey is fundamentally flawed because it isn't based on Random Sampling. You cannot get an accurate, non-biased representation through non-random sampling, and your way of getting data (asking players in which players can refuse) is an example of biased sampling.

  And what's your solution?
  
  

  Quote from EndlessDreams;1248002:

  This is the one of the best example of when sampling went wrong: Landon wins US presidency in 1936!

  A better label might be "an example of not understanding the system" because the public vote does not determine the President, the electoral vote does (it's a pretty common misunderstanding because they usually both sway the same way).
  
  http://www.learnnc.org/lp/media/lessons/davidwalbert7232004-02/electoralcollege.html
  
  So in that case, the group being polled was not actually the group that the outcome depended on.
  
  In fact there have been four cases of people who were elected President via electoral vote when they lost the popular vote. John Quincy Adams, Rutherford B Hayes, Benjamin Harrison, and most famously George W Bush (the 2nd).
  
  Years 1824, 1876, and 1888 here. Year 2000 here.
  
  

  Quote from EndlessDreams;1248002:

  Standards too high? I didn't know striving for a accurate, non-biased representation sampling was consider high.

  You're asking for data to be gathered without asking people.
  
  We do not have a method to gather player stats from the system without asking them.
  
  Therefore your standards are too high because they prevent the project from moving forward.
  
  

  Quote from EndlessDreams;1248002:

  The more bias a study can remove, the stronger the study.
  On a side note, this is why double-blind randomized trials are the golden standard in conducting very strong scientific studies.
  
  I never said once that you couldn't do your biased study. You are free to do your biased study.
  
  However, if you going to put it on the public forum for people to see (and comment on), people should know the flaws in the methods of your study.
  
  Not everyone took statistics classes, or even remember them. After all, it is really easy to fool people who don't know much about statistics.

  If you don't care about this and you don't want this to take place, then I already told you that you can feel free to ignore this.
  
  Even Ironwoman, who I normally butt heads with so much you'd think we were a pair of mountain goats, is willing to discuss possible solutions to perceived issues and is not just here to get in the way.
  
  
  
  
  

  Quote from ZeroPantz;1248045:

  I was going to suggest multiple volunteers on each server to make sure you cover as much players as possible, but then you'd risk someone asking the same player the questions. To avoid that recording usernames would work. It won't harm the anonymosity of the players. Basically see it as a tally system, where when a players gives a certain answer, we put a tally behind the answer given.
  
  And in a seperate list we would write down the usernames, so we can see whose been asked, but there is no way for us to get back to knowing who gave what answer due to the tally system.

  I'm hoping that we can just tell people it's anonymous and then rely on them to be all "Wait somebody already asked me this", but it's kinda' hard to determine which method is better.
  
  

  Quote from ZeroPantz;1248045:

  Perhaps it is also a good idea to have volunteers open a dungeon advertisement such as: [Wanted]ppl2AnswerMabiSurvey- NM Or something. So everyone online at that moment knows about it, and could choose to note the volunteer to answer the questions.

  A lot of players mentally tune out the party ads because they're almost always for market stuff, so that wouldn't work very well and would mainly catch the attention of a subset of players.
  
  

  Quote from ZeroPantz;1248045:

  I was also wondering whether or not it'd be a good idea to ask about total exploration level. After all, it would be interesting to see how many players actually work on that. Especially in comparison to the time they've been playing/total level.

  I'll stick that in as a consideration for #10, it seems like it might be interesting info as, yeah, not everybody does exploration.
• ironwoman wrote on 2014-09-19 02:25
  For fuck's sake, this is just a game, not some National Health Services questionaire or some shit like that.
  
  Why did we create this thread? Brain storm questions right? I don't even think we're done with that part yet. Complete that part first, then y'all can fight about methods.
  
  Also, it's inevitable that we can't avoid bias completely. However... we can still try to reduce it.
  Who said we only need to use one method? Just try to use everything we have. Create a google doc, direct people to where you post it, ask people on forums, facebook groups (although full of dumbasses), Wiki, etc. And I can tell you from my experience that you will get many different kinds of people, because on Wiki, it's not like mabination. Most nationers are far more experienced that most of the Wiki people. Just get people from every sources possible.
• Rydian wrote on 2014-09-19 23:44
  I thought for a little bit about getting answers from various groups and then showing the results separately (per-site) from the in-game info, but I'm afraid that might encourage stereotyping of player groups. Also I don't actually have a tumblr account or actively use facebook and junk so I wouldn't be able to plan out the methods properly anyways.
  
  So #10 could still be...
  - How many pets do you have?
  - Which talents, if any, have you mastered?
  - How often or how much do you play?
  - What's your total exploration level?
  Though, total exploration level can be asked in the question that asks for total normal level?
• Rydian wrote on 2014-10-11 09:57
  It's been almost a month, so bump for any last inputs on the question list.
  
  I'm still leaning towards including exploration total level in with the main total level question, and having #10 be how often they play, but how should it be worded? Should it ask how often they play per day or per week, or how many days per week they play when they're active or something like that? How would inactive periods factor in?
• Paradise wrote on 2014-10-13 21:03

  Quote from Rydian;1250655:

  It's been almost a month, so bump for any last inputs on the question list.
  
  I'm still leaning towards including exploration total level in with the main total level question, and having #10 be how often they play, but how should it be worded? Should it ask how often they play per day or per week, or how many days per week they play when they're active or something like that? How would inactive periods factor in?

  
  
  hmmm, i think it could be like
  how many days per week do you log in and do any activity besides chat/afk (setting a shop or party ad to sell/buy counts as an activity, since it shows interest on achieving something ingame)?
  
  as answer, obviously a range of 1 to 7 days
  
  
  And i'm sorry to read so many destructive critics ( destructive critics get nowhere, and guess who are those fired in less than a week when working as engineers...) about what you are planning
  
  Hope your survey goes well
  
  oh and about picking random players, just place people on different places, you will get different players (SM altars, dungeon entrances, towns, rafting line, and places where people train skills like the mines on vales) and of course, ask the person if he/she has participated in a survey recently, getting non-biased answers 100% is impossible (really, even the best one expect like a 5% margin), but the bigger the sample, the less it will damage the survey