this post was submitted on 11 Jun 2023
1317 points (100.0% liked)
Technology
37740 readers
650 users here now
A nice place to discuss rumors, happenings, innovations, and challenges in the technology sphere. We also welcome discussions on the intersections of technology and society. If it’s technological news or discussion of technology, it probably belongs here.
Remember the overriding ethos on Beehaw: Be(e) Nice. Each user you encounter here is a person, and should be treated with kindness (even if they’re wrong, or use a Linux distro you don’t like). Personal attacks will not be tolerated.
Subcommunities on Beehaw:
This community's icon was made by Aaron Schneider, under the CC-BY-NC-SA 4.0 license.
founded 2 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
Right, sorry. I understood it meant legal guardian in the other contexts, but misread this line in particular
Again, you are posting links that tell me that the type of research done by Reddit and Facebook would not be covered. "Exempt" in these links means exempt from the full scope of requirements of the Federal Policy for the Protection of Human Subjects for research that presents no more than minimal risk and falls into one of one of predefined categories. "Exempt" research may still require informed consent. Your prior link was for exemption to informed consent specifically. In my view, the Reddit experiment satisfies neither the conditions for "exempt" research nor the conditions for "informed consent exemption".
It may be hard to keep track of all the legalese flowcharts, all the AND and OR conjugated lists of preconditions, but I think I got it right. To take a look at UCSB flowchart for example, how would I argue that my Reddit-like experiment is "exempt"? I would still need to meet with the IRB to determine whether that my research is exempt in the first place (Reddit Inc has no IRB), and to do so I'd have to show that ALL of these are true:
Then I would pick a predefined category, probably Exempt Category 3 - Benign Behavioral Interventions with Adults, and show that I meet at least ONE of these is preconditions:
Embarrassingly, b) is probably a typo, since on every other site the language used is that the responses would NOT reasonably place the subjects at risk of criminal or civil liability, but whatever. Assuming we want to keep track of as much PI as possible (whether or not you agree that any username-related information online is PI at all), we'll take the (corrected) option b) since there is no criminal liability for the user from our experiments. Then we have to follow all of these example rules:
Reddit violates ALL of these example rules.
So you see, even your own examples of specific IRB guidelines disagree with you.
The examples of research listed under Category 3 are:
The Reddit experiment goes beyond these, as it is designed to specifically manipulate the user's emotional state, to deliberately frustrate the user to see whether they would download the native app or abandon reddit entirely when their web access is blocked. It is much more similar to the Facebook shadowbanning experiment (which you agreed can shove it) than to these examples above. I'd say the level of frustration and embarrassment is similar to the Asch conformity experiment, which if you wanted to repeat it now I was taught under modern rules DOES require IRB review, DOES require informed consent (if not with details of the deception then at least with the very fact that an experiment is taking place), DOES require post-experiment debfrief, all because it DOES present a risk of causing emotional harm.
Can you find instances of modern Asch experiment research papers that specifically show they are "exempt" research and/or that have received an exemption from collecting informed consent prospectively and/or retrospectively? If you do, it will help convince me that that's how modern research ethics really works.
No.
Strictly speaking, COPPA prevents Reddit from collecting information from users under the age of 13. While there are no explicit guarantees that a person on the site isn't 13 or older, and also recognizing that age of majority is typically 18, then in a general literal sense yes there are minors involved, in so far as the activity discussed is research.
A research intervention is an intervention in so far as it intersects with the participant. A drug trial that lasts for 10 weeks total but only gives doses to participants for 2 weeks (and then results monitoring during, 2-weeks post, and 4-weeks post) does not mean that the "intervention" lasts 10 weeks.
In most practical terms, running an experiment for 2-3 weeks is very common to collect sufficient data. However, the intervention itself may be quite brief (for example, a short 45-minute interview with a participant would be the "intervention" for a 2-3 week long study interviewing physicians on their concerns about organizational capacity for change).
For a repeated measures experiment, the intervention usually involves the actual experiment encounter and maybe some additional time between them.
For the case of A/B testing, usually the "intervention" in this case is the A/B test as it applies to the user at the moment, and not the entire duration of when the testing is taking place for all participants.
Once again, to iterate, you are equating the inability to view the content of a website without logging into an account with such substantial emotional and psychological harm that is comparable to being verbally derided in public, for a week, shamed on a public channel, and/or comparable situations. You are not going to convince an IRB that being able to view the content of a website without logging in, then subsequently going to a different community to ask for help, and then hypothetically ruminate about the matter for weeks, is going to exceed the kind of everyday ordinary harm to qualify a risk level above minimal risk.
Deception goes beyond simply "lying" to or "not informing" the participant. Duke University gives some good considerations here:
Additionally, a waiver for utilizing deception in research has to:
Satisfying #1 and #2 in an UI/UX A/B testing regime that Reddit used here is pretty easy. You are specifically hung up on the implicit harms involved but in reality they are of no particular serious concern.
#3 is particularly interesting, because effectively what that means is you need to demonstrate deception is necessary in experimental design for the experiment to actually work. If you are seeing whether a person will interact more with a site or not if they are blocked from seeing content without logging into an account, telling them ahead of time could already bias the outcome. This is in very specific consideration that #1 and #2 are already met: being unable to view a website without logging into an account is not anything more than minimal risk. And even then, it is important to emphasize that the failure to disclose the research hypothesis to counter the demand effect is NOT deception.
The kind of UI/UX A/B testing Reddit employed in this specific instance is absolutely not equivocal to the Asch conformity studies.
To be very clear here, when YOU operate under the belief that being unable to view the content of the website, then posting about it elsewhere and potentially being ridiculed for it is sufficient of a bar to meet beyond minimal risk, then we have very different definitions of what "minimal risk" entails. Since we cannot come to consensus on this particular topic, and instead you've gone so far as to associate this kind of activity's harm of equivalent to the Asch conformity studies is frankly ludicrous.
If we cannot agree on this, then so be it. However, I will repeat (with added finality): YOU running the same A/B experiment Reddit is doing on an university-sanctioned website will NOT get you run out of the university by the IRB. In a real-world scenario. you would likely discuss this experiment with an administrator or similar at your department, then maybe send an e-mail off to the IRB for any clarification, would likely have some back-and-forth, and then would ultimately receive a determination that it is exempt (at worst), or not considered human subjects research at all. I can see a few circumstances where such an effort might merit an expedited review, but to do this would involve some torturous twisting of the situation that could be easily avoided (for example, running this testing on a page that has instructions for performing the Heimlich maneuver).
The Asch conformity study experiments are absolutely not equivalent to doing A/B testing on a wide scale with regards to how users interact with a website when presented with a pop-up preventing further interaction without logging in.
Thank you for agreeing that Asch conformity experiment falls under human experiment ethics considerations and informed consent requirements! I am surprised though that you consider the reddit experiment unlike the Asch experiment (I personally see reddit as the worse one actually!). Could you explain how in your mind, Asch does carry a risk of causing harm in a way that reddit does not? Asch is just looking at a bunch of lines on a page after all! How can a bunch of lines cause harm? I also find it odd your cavalier attitude towards the word "should".
If "should" isn't prescriptive, why even have any "should" statements in our guidelines if you are just going to ignore them all? And yet again your link disagrees with you:
Reddit never had any intention to ameliorate the deception, to debrief the participants, or to give them an opportunity to delete their experiment records after the fact. Reddit never implemented alternative practical research methods like opt-in studies. Are you seriously arguing that because the page says "should" and not "must", reddit was perfectly ethical simply not doing any of this at all? This isn't some RFC, this is normal people language!
Or if you are saying this wasn't deception, then why link to the entire Duke deception page at all? The only relevant sentence here to you is:
And it only refers to the disclosing the research hypothesis itself, not the very fact that you are taking part in some experiment! And you agree that it is not impossible to perform usage experiments without informing participants in advance (I brought up Firefox as a better example alternative), it is just more laborious. Moreover, reddit did engage in actual deception beyond simply keeping the fact of the experiment secret:
What happened went beyond that. If you read the reddit OP:
This isn't some simple A/B testing of things like text size or link color. This isn't like Facebook or Instagram blocking everyone equally from seeing communities without logging in. OP was logged in. Reddit lied to them saying they were not logged in when they were. Reddit lied to them saying there is no way to log in. Reddit lied to them saying the only way to see the content was to download the app. This is the deception part. This is the part that's similar to Asch and the people in the room with you lying that they are participants like you. You think you are in a normal situation but you are not. You've been singled out and no one believes you.