I’ve been asked why I dislike surveymonkey surveys, I don’t want to reply fully in that forum (because it’s off-topic there) and I want to publish this for your feedback now.
surveymonkey really frustrates me. In short, it’s often confusing, I think it doesn’t follow UK accessiblity laws and I’m unsure if it complies with our privacy laws. It seems like another triumph of marketing over good sense. It’s a free market failure, possibly based on lack of information on the drawbacks of this option.
If a survey is on Surveymonkey, it’s easiest for able-bodied users of full-fat browsers with low security and privacy settings to answer. Using NoScript or a privacy tool? You’ll get an error message (which contradicts the VPAT_SurveyMonkey.pdf description). Unable to see graphics clearly? The forms will be hard to read because they’ve replaced the browser form widgets (such as checkboxes and radio buttons) with javascript-driven images for no good reason.
Those sort of faults will probably bias results a bit – how much will depend on the subject. It’s a particular case of a problem I first mentioned in the last-but-one bullet point on my debian surveys page back in 2006. Basically, I think you should keep open surveys as open as possible. Statistics literature is littered with bizarre spurious results from when the survey method accidentally favours some particular subset of the population.
Finally, surveymonkey surveys can be left with answer requirements which make no sense, such as ticking “Other” requiring a text input, but the text box not accepting input. Is that the fault of the survey owner not being able to work the admin interface, or the interface having some subtle flaws? I’m not completely sure, but I’ve had that experience a few times.
So, can I persuade people to use limeservice, doodle or even Google Docs rather than Surveymonkey, please?
I think Limeservice is better because it’s based in Germany (so EU data protection and privacy laws apply) and is cooperatively-developed FOSS software. Doodle and Google aren’t, but are quicker and easier to use for straw polls.
If you want something to install on your own site and the downloadable limesurvey software seems a bit heavy, then askpeople seems another good option. Our co-op can host either of those if you’re willing to pay. (If someone would be willing to sponsor it, we’d host some free surveys too – let me know.)
Lots of people use surveymonkey, but popularity doesn’t always mean it’s the best tool. Please consider the alternatives and pick the best tool for the job.
I haven’t tried Google Docs yet since I have found SurveyGizmo to be a good alternative to SurveyMonkey. Maybe next survey I’ll try Google Docs to see what all fuss is about.
Do you know how our (Swiss) privacy laws compares to EU regulations? I’d expect them to be somewhat similar, but I’m not sure.
Doodle is based in Switzerland, so Swiss law applies.
http://www.privacyinternational.org/article.shtml?cmd347=x-347-559062&alstheme=Freedom%20of%20Information reports that “On July 26, 2000, the European Commission ruled that both Switzerland and Hungary (now an EU member) provide “adequate” protection for personal information and therefore all transfers of personal data to these countries could continue.”
http://www.edri.org/edrigram/number7.22/google-streetview-privacy-switzerland mentions a Swiss Privacy Authority.
I’m no expert on privacy, but both of those look like good signs to me.
I didn’t want to answer your remarks on list but I welcome the opportunity to reply that you offer here.
I understand your remarks about privacy violations etc but there are some additional comments that you make that I find more difficult given the context in which the survey was being run.
The survey was run on behalf of members of a JISC discussion list. The members of that list defined the questions they wanted answered and also wanted results as soon as possible to inform their budget decisions for 2010.
You wrote to ask me to change my mind after I had announced it and after it had opened. I wrote to expalin my reasons but you didn’t bother to read that and instead launched what many saw as a personal attack on my integrity, accusing me of bias and suggesting that the survey was being run for the benefit of the CILIP Gazette.
The Gazette – and others – had asked me if they could publish the results and I was happy to agree. I think it’s important to give information as wide an audience as possible. I’m rather astonished that you don’t.
Just to be quite clear. The list is provided by JISC on behalf of UK (and world) libraries. It is not a commercial enterprise.
Library RFID Ltd is a single person company (me) that does buy or sell anything except my advice, .
I declined to give you access to the names of those who completed the survey because, unlike me, you want to use the information to sell them your goods and services. I expressly promised that I would not open them up to commercial exploitation by publishing names.
I hope that makes things a little clearer.
@Mick Fortune – you seem to have a serious axe to grind on me and I’ve never been sure why.
The comments about SurveyMonkey aren’t only about the one survey, but to cover the points you raise:
The survey could have collected from additional less problematic locations. As you note, the privacy concerns are understandable, besides any other comments.
I made no personal attack on you, nor did I accuse you of bias. I pointed out the bias in the survey from the poor survey tool being the only input. (By bias, I mean the statistical study sense.)
I asked if you would publish the data (I don’t care about the respondant names!) and results under open terms because I think it’s important to give information as wide an audience as possible, much wider than the usual niches like CILIP. I’m rather disappointed that you don’t.
Just to be quite clear: I know what JISCmail is. I remember the transition from Mailbase.
Library RFID Ltd seems to be a for-profit private company, so I don’t think its owner should make baseless accusations about a member of a cooperative that is run to serve. I never asked for the names of those who completed the survey, I don’t want to use the information to sell and I don’t understand where you got that idea from. I was motivated by the fifth (Education, Training and Information) and seventh (Concern for Community) cooperative principles. I ask you to reread what I wrote and retract parts of the above.
Perhaps you should go back and re-read your original remarks, and consider why others reacted to them in the way they did. Not just on the list but also on Twitter – and not all from commercial companies but also from librarians.
I have no axe to grind with you. I admire what you are trying to do. I just feel that I have a right to respond to what I, and many others, saw as a personal attack.
I am happy for the results to be published as widely as possible so if there are other avenues I have not explored please let me know. You will of course receive a copy yourself with which you may do as you please.
I’m pretty sure that I didn’t make any remarks about the survey on Twitter. I don’t often mention things which I consider user-hostile there.
Here are what I think were my original remarks on the list:
OK, the final question was a bit silly, but even so, I really don’t understand how that was seen as a personal attack.
Glad to hear that I can do as I please with the results, which is part of what I was asking for. I await it with interest.
You didn’t comment on Twitter but others – who were clearly reacting to your list post – did. No matter. The point was always to make as much information as possible available to the widest audience,
Let’s move on shall we? The results are taking a while to produce because there were so many from the same institutions and I needed to separate out the factual (which systems in use etc.) from the opinions (is it reliable etc.) Of course all opinions will be counted, but I need to be sure that everyone thinks they have the same systems – or even that they are using RFID at all
I can’t really be held responsible for what random rubbish others make up about me on Twitter. It’s sad, but it seems to be the way of the world these days, so yes, let’s move on!
Does anybody know if it’s possible to embed SurveyMonkey survey’s into a wordpress blog or do the forms necessarily need to be completed on the surveymonkey site itself? I know this works for Google Docs survey feature but find the analytics options a little limited on there.
Thanks
@Coach – you can, but using SurveyMonkey is wrong. It biases answers in favour of able-bodied users of full-fat browsers with low security and privacy settings. Can I persuade you to use limeservice, doodle or even Google Docs rather than Surveymonkey, please?
Also interesting is the Drobots application (www.drobots.com). This cloud application could be usefull for survey monkey users. Just load your data in drobots and you can start analyzing your survey in minutes ! Spss is not needed anymore !
Hi, I am did a private survey on surveymonkey and now I can’t access my data because my survey was taken by more than 100 users. I don’t want to pay 35 $ for it, I won’t need it anymore. Anyone with a survey monkey account here ? if so I could transfer it to your account, I just want to see the result… that’s pissing me off !!!!
Another good Survey Monkey alternative is Rational Survey. It integrates well with Twitter & Facebook and has superb analytics.
http://www.rationalsurvey.com/survey_tool/survey_monkey
Rational Survey says the right things, but then the big “Get Started” button does nothing. Another Javascript-required site which fails to degrade gracefully. I prefer the alternatives I suggested above.
Hello, I´d like to inform you about http://www.easygoingsurvey.com
It´s a newly launched English version of the widely popular encuestafacil.com (Spanish) that allows unlimited surveys to be launched with unlimited questions in its basic free version. Of course we also include real-time results, graphs, statistics, templates, and the ability to design a survey completely from scratch. Give it a try! I´d love to hear any comments! — David
I can’t offer a good comment because Easy Going Survey doesn’t work for me. My browser “detected that the server is redirecting the request for this address in a way that will never complete” on a test survey (the Course Evaluation Product Test), but the FAQ says “a survey can be answered using any browser even if it does not accept cookies or Javascript” so I’m not sure why the redirect loop.
What do you think of Qualtrics?
Qualtrics looks similar to Survey Monkey but less popular: it requires javascript (so won’t work on some accessibility and security-conscious browsers), has a weak privacy policy that says it follows the “safe harbour” scheme but is only self-certified and contradicted by the “The Company makes no representations that its materials or content are appropriate or available for use in other locations” in the terms of use, Utah law for disputes, … I’d avoid it.
Thanks for the response! I also hate Survey Monkey but for different reasons. Mostly because the usability is terrible, and it can be very buggy. I was searching for an alternative today (which brought me to this blog post)..and in the process also came acros Qualtrics. I am enjoying the usability and functionality so far. You can do some really creative things with the question types, and the analytic functions seem to be pretty advanced. I have tried Google forms before, but I had to do all my own analytics writing a bunch of excel functions… felt like I was reinventing the wheel. Although maybe there´s some kind of app you can use with Google in order to do a quicker analysis?
About JavaScript… what browser doesn´t run it? Every other web site you go to seem to use it. The percentage of people using browsers that don´t supoprt JavaScript must be really really small. (Although how big of an impact this makes on your survey, depends who you are attempting to survey, I suppose!)
Also, I don´t exactly understand the concern about privacy. I wouldn´t use any of these tools for surveying of highly sensitive information. So if I want to do a straw poll on a topic of little consequence.. is there a reason I should be really concerned about privacy?
I don’t know about the Google outputs. I’m a graduate statistician, so I just drop survey results into a stats package like R. OpenOffice also has some passable data exploration tools.
Javascript: It’s not a question of browser support. People with various accessibility problems use tools like NoScript.net to limit or stop JavaScript because most websites are difficult enough to use without having things moving around of their own accord. Also, people who want their browsers to be more secure will restrict it too. Yes, how much this matters depends on your target audience, but there are few surveys where you want responses correlated with physical ability.
Privacy: at least here, if for no other reason like ethical behaviour or concern for community, you should be concerned because it’s the law (Data Protection Act) and it covers all personal data, not only highly sensitive stuff. Read up on http://www.ico.gov.uk/
I am using the survey for very cursory and anonymous market research for a potential business plan… asking only age and gender, and very impersonal questions about whether they would potentially use a product. I´m from the US, and under our privacy law generally only applies when the person has “reasonable expectation” of privacy. I don´t think anyone would expect this type of survey to be especially private. Regardless, this issue could be solved with a simple disclaimer.
I understand for your purposes privacy could be really important. But, I don´t think it makes sense to make a blanket statement that some software is bad only because it doesn´t fit your needs. It might fit someone else´s needs perfectly.
I suspect that linking age and gender to a personal static IP address could still be considered personal information, but I’ve not checked recent case law.
But you illustrate my point pretty well: different countries have different privacy laws and the US is pretty weak, which is why European users in particular should be wary of hosting surveys in the US, especially when you also consider the famously expensive US legal system. I don’t understand why it’s unreasonable to expect data submitted to a research survey to be kept private and anonymous unless I’m told otherwise up front.
Personally, I agree with the general position on personal data in England, so I do believe that surveys hosted under weak privacy regimes are bad in general.