Hi everyone.
I am mostly pleased with how Santabot performed in its first live test today, but I also found issues which I am fixing. I made it a lot more strict and fixed a bug. Details to follow:
Mostly, I was surprised and pleased that the bot was coming up with right judgements for people I was talking with, but this thread has one comment that seems reasonable, and one comment that seemed like exactly the kind of inflammatory content I wanted the bot to eliminate. I looked into it for a while, and eventually had to revisit my attitude toward users that have a lot of "positive rank" counterbalancing also a lot of "negative rank." I think that the ratio of upvoted content to negative content that it's reasonable to ask someone to produce should be higher than 1:1. So I made it 2:1. That eliminated the user posting the offending comment while leaving the user posting the legitimate comment. I didn't have time to go into a full detailed analysis, but I did some spot checks on how that affected its other judgements, and generally I liked what I saw:
- The percentage of total users banned has gone way up, from 0.4% to 2.8%.
- @Kaboom@reddthat.com, @LibertyLizard@slrpnk.net, and other specific users that I looked into and thought should remain unbanned, are still unbanned by the bot's analysis.
- Some users who weren't banned by this morning's configuration, but which seemed like they should be, are now banned again.
In addition to the algorithm issues, I found a bad bug. The bot is supposed to remove recent postings when banning someone, to stop throwaway accounts from coming in and posting and then being banned but with the content staying up. However, the bot was simply removing all recent content from all users whenever it banned anybody, and my test suite was too simple-minded to catch the problem until it was live and removed a random innocent user's comment and sent them a message that they were banned. Oops. I partly fixed it once I saw it. Maybe I shouldn't have used Futurama Santa as the avatar.
That's the update. More to come. If you have questions, comments, or concerns, please by all means say something and I'll do my best at a response.