I believe the principle is fairly easy to understand: imagine you're missing two fingers. Because kids are cruel, your middle school experience involved lots of shaming and name-calling, and that really got to you at the time.
Everything is fine now, because nobody actually cares. But then along comes this algorithm and, for some reason, it crops your profile picture to be just that hand of yours. Of course one or two "friends" from middle school are following you on twitter, and start reminiscing about the times they called you three-finger-bob.
Is it conceivable/realistic/justified that this would, at least for a while, hurt?
Meanwhile, for someone who didn't go through that experience, the algorithm doing the very same thing is just... a curious bug?
As to how they are going to score this: it cannot be scored only quantitively, as they specifically say in the description. They are going to read your hypothesis–something like the above, but maybe a little more serious–and score it, probably along a a few categories and by a few different people etc.
A better comparison would be if there was some salient feature surrounding my hands such as me holding an object. The object would be the equivalent to the jersey number. It's clearly not cropping on the thing that makes me feel bad but it did anyway. To avoid this, it has to be trained to take into account all possible human reactions. Basically, it has to know that these pixel values activate some social controversy and it's best to avoid. Not saying this isn't possible, but it seems absolutely fraught with peril and potentially more harmful than the original saliency, especially for social constructs with no clear consensus.
Yes, I didn't mean to suggest any specific reason for the algorithm's output, because it doesn't matter.
This issue isn't about anyone's "guilt", least of all the algorithm's. It's about harm. Harm is to be avoided, even if it is the result of a completely benign process with no ill intentions.
And you aren't going to be able to explain away some decision that causes harm by explaining the algorithm. Even knowing everything about the algorithm, I would prefer it doesn't focus on whatever my weak spot is. And even knowing how GPT-3 works, I still tend to anthromorphize it.
To some approximation, exactly nobody in the real world is going to give the company the benefit of the doubt and study its algorithm, nor should they be expected to.
It's like that escaped lion that keeps eating the schoolchildren: it's what lions do, an expression of its basic lioness! It's not evil, guilty, or even culpable: it just can't help itself, but to help itself to an early dinner.
And yet, we are going to shoot the lion. Not as punishment, but as a simple, mechanistic, prevention of harm to people, who, in our system of values, rank higher.
Algorithms rank far below lions. An algorithm that causes harm, no matter how or why, goes the way of php (taken out back and shot, unless it's run by Facebook). And anything that happens is considered to be caused by the algorithm, even if some humans happen to provoke it by somehow being different than other humans, or our expectation of rationality. Because we cannot change humans, and because nobody should be expected to change to accommodate technology, especially if they were never asked if they want that technology.
> Is it conceivable/realistic/justified that this would, at least for a while, hurt?
The people reminiscing about hurtful teasing would hurt. The algorithm that did the cropping would not. Algorithms don't have intent and intentions matter.
Everything is fine now, because nobody actually cares. But then along comes this algorithm and, for some reason, it crops your profile picture to be just that hand of yours. Of course one or two "friends" from middle school are following you on twitter, and start reminiscing about the times they called you three-finger-bob.
Is it conceivable/realistic/justified that this would, at least for a while, hurt?
Meanwhile, for someone who didn't go through that experience, the algorithm doing the very same thing is just... a curious bug?
As to how they are going to score this: it cannot be scored only quantitively, as they specifically say in the description. They are going to read your hypothesis–something like the above, but maybe a little more serious–and score it, probably along a a few categories and by a few different people etc.