I’ve been asked why I dislike surveymonkey surveys, I don’t want to reply fully in that forum (because it’s off-topic there) and I want to publish this for your feedback now.
surveymonkey really frustrates me. In short, it’s often confusing, I think it doesn’t follow UK accessiblity laws and I’m unsure if it complies with our privacy laws. It seems like another triumph of marketing over good sense. It’s a free market failure, possibly based on lack of information on the drawbacks of this option.
If a survey is on Surveymonkey, it’s easiest for able-bodied users of full-fat browsers with low security and privacy settings to answer. Using NoScript or a privacy tool? You’ll get an error message (which contradicts the VPAT_SurveyMonkey.pdf description). Unable to see graphics clearly? The forms will be hard to read because they’ve replaced the browser form widgets (such as checkboxes and radio buttons) with javascript-driven images for no good reason.
Those sort of faults will probably bias results a bit – how much will depend on the subject. It’s a particular case of a problem I first mentioned in the last-but-one bullet point on my debian surveys page back in 2006. Basically, I think you should keep open surveys as open as possible. Statistics literature is littered with bizarre spurious results from when the survey method accidentally favours some particular subset of the population.
Finally, surveymonkey surveys can be left with answer requirements which make no sense, such as ticking “Other” requiring a text input, but the text box not accepting input. Is that the fault of the survey owner not being able to work the admin interface, or the interface having some subtle flaws? I’m not completely sure, but I’ve had that experience a few times.
So, can I persuade people to use limeservice, doodle or even Google Docs rather than Surveymonkey, please?
I think Limeservice is better because it’s based in Germany (so EU data protection and privacy laws apply) and is cooperatively-developed FOSS software. Doodle and Google aren’t, but are quicker and easier to use for straw polls.
If you want something to install on your own site and the downloadable limesurvey software seems a bit heavy, then askpeople seems another good option. Our co-op can host either of those if you’re willing to pay. (If someone would be willing to sponsor it, we’d host some free surveys too – let me know.)
Lots of people use surveymonkey, but popularity doesn’t always mean it’s the best tool. Please consider the alternatives and pick the best tool for the job.
Loading ...
8 comments so far
1 Tanya Forsythe // Jan 20, 2010 at 8:20 am
I haven’t tried Google Docs yet since I have found SurveyGizmo to be a good alternative to SurveyMonkey. Maybe next survey I’ll try Google Docs to see what all fuss is about.
2 Adrian von Bidder // Jan 20, 2010 at 8:25 am
Do you know how our (Swiss) privacy laws compares to EU regulations? I’d expect them to be somewhat similar, but I’m not sure.
Doodle is based in Switzerland, so Swiss law applies.
3 MJ Ray // Jan 20, 2010 at 9:48 am
http://www.privacyinternational.org/article.shtml?cmd347=x-347-559062&alstheme=Freedom%20of%20Information reports that “On July 26, 2000, the European Commission ruled that both Switzerland and Hungary (now an EU member) provide “adequate” protection for personal information and therefore all transfers of personal data to these countries could continue.”
http://www.edri.org/edrigram/number7.22/google-streetview-privacy-switzerland mentions a Swiss Privacy Authority.
I’m no expert on privacy, but both of those look like good signs to me.
4 Mick Fortune // Jan 22, 2010 at 8:04 am
I didn’t want to answer your remarks on list but I welcome the opportunity to reply that you offer here.
I understand your remarks about privacy violations etc but there are some additional comments that you make that I find more difficult given the context in which the survey was being run.
The survey was run on behalf of members of a JISC discussion list. The members of that list defined the questions they wanted answered and also wanted results as soon as possible to inform their budget decisions for 2010.
You wrote to ask me to change my mind after I had announced it and after it had opened. I wrote to expalin my reasons but you didn’t bother to read that and instead launched what many saw as a personal attack on my integrity, accusing me of bias and suggesting that the survey was being run for the benefit of the CILIP Gazette.
The Gazette – and others – had asked me if they could publish the results and I was happy to agree. I think it’s important to give information as wide an audience as possible. I’m rather astonished that you don’t.
Just to be quite clear. The list is provided by JISC on behalf of UK (and world) libraries. It is not a commercial enterprise.
Library RFID Ltd is a single person company (me) that does buy or sell anything except my advice, .
I declined to give you access to the names of those who completed the survey because, unlike me, you want to use the information to sell them your goods and services. I expressly promised that I would not open them up to commercial exploitation by publishing names.
I hope that makes things a little clearer.
5 MJ Ray // Jan 24, 2010 at 1:22 am
@Mick Fortune – you seem to have a serious axe to grind on me and I’ve never been sure why.
The comments about SurveyMonkey aren’t only about the one survey, but to cover the points you raise:
The survey could have collected from additional less problematic locations. As you note, the privacy concerns are understandable, besides any other comments.
I made no personal attack on you, nor did I accuse you of bias. I pointed out the bias in the survey from the poor survey tool being the only input. (By bias, I mean the statistical study sense.)
I asked if you would publish the data (I don’t care about the respondant names!) and results under open terms because I think it’s important to give information as wide an audience as possible, much wider than the usual niches like CILIP. I’m rather disappointed that you don’t.
Just to be quite clear: I know what JISCmail is. I remember the transition from Mailbase.
Library RFID Ltd seems to be a for-profit private company, so I don’t think its owner should make baseless accusations about a member of a cooperative that is run to serve. I never asked for the names of those who completed the survey, I don’t want to use the information to sell and I don’t understand where you got that idea from. I was motivated by the fifth (Education, Training and Information) and seventh (Concern for Community) cooperative principles. I ask you to reread what I wrote and retract parts of the above.
6 Mick Fortune // Jan 24, 2010 at 3:40 pm
Perhaps you should go back and re-read your original remarks, and consider why others reacted to them in the way they did. Not just on the list but also on Twitter – and not all from commercial companies but also from librarians.
I have no axe to grind with you. I admire what you are trying to do. I just feel that I have a right to respond to what I, and many others, saw as a personal attack.
I am happy for the results to be published as widely as possible so if there are other avenues I have not explored please let me know. You will of course receive a copy yourself with which you may do as you please.
7 MJ Ray // Jan 24, 2010 at 5:26 pm
I’m pretty sure that I didn’t make any remarks about the survey on Twitter. I don’t often mention things which I consider user-hostile there.
Here are what I think were my original remarks on the list:
OK, the final question was a bit silly, but even so, I really don’t understand how that was seen as a personal attack.
Glad to hear that I can do as I please with the results, which is part of what I was asking for. I await it with interest.
8 Mick Fortune // Jan 24, 2010 at 5:40 pm
You didn’t comment on Twitter but others – who were clearly reacting to your list post – did. No matter. The point was always to make as much information as possible available to the widest audience,
Let’s move on shall we? The results are taking a while to produce because there were so many from the same institutions and I needed to separate out the factual (which systems in use etc.) from the opinions (is it reliable etc.) Of course all opinions will be counted, but I need to be sure that everyone thinks they have the same systems – or even that they are using RFID at all