In my community consulting work, I’ve often had to remind clients that their products, while important, are not governments.
I can almost understand the confusion. Community systems create a kind of governance. There are rules for what members can do and procedures for when those rules are broken. Sometimes there are even appeals processes. If you squint, the whole thing can take on the shape of a rudimentary justice system.
The government-thinking has a secondary appeal to executive teams. If their site is a country, that makes them the ruling class. It makes the CEO the president (or dictator). And again, squinting, it can kind of feel that way. Running a company, like managing a community, is literally a power trip. You can do things your members can’t, including punishing those members. Power, even tiny power, can be addictive.
But it’s not true. None of it. Your product is not a country. You are not a government. Your CEO is not a president. And, worse, thinking that way is damaging to the community, disastrous for the company, and may just be ruining the world.
Governments Aren’t Optional
I am a citizen of the United States. I became that by simply being born here (unlike my dad, but that’s another story). I did not have a choice in the matter.
But I can choose whether I’m a member of Facebook or not. I can decide to delete my Twitter account. (Every day I get a little closer.) No matter how important these sites become, they’re still optional. Nobody has to participate in them.
When the leaders of these sites fall into the trap of thinking of themselves as governments, they forget their people can simply up and leave if they get angry. They become callous to member complaints and arrogant enough to think they can mistreat their communities without repercussions.
But these sites come and go (just ask Tom from Myspace). Mark and Jack could use a little humility, and part of that is remembering that you’re not a sovereign ruler, you’re just another temporary caretaker of a precious commons.
Governments Give Rights
If you work in community management, there are two words that probably make you twitch: “free speech.” Because every time you’ve had to remove content for some reason, someone somewhere used those words in an angry response.
But here’s the thing: “free speech” is guaranteed by governments. (And not even all governments! We’ve got our First Amendment here, and Canada has something similar, but after that it gets a lot sketchy.) Free speech means you have the right to speak. In your life. In general. It does not mean you can say whatever you want, wherever you want.
You can’t yell “fire” in a crowded theater. You can’t post the phrase “I’m going to,” followed by a word that rhymes with “thrill,” followed by “the president,” pretty much anywhere. (No joke, I was once subpoenaed by the Secret Service because I was the admin of a community site where someone posted something they interpreted as a threat.) You can’t post photos of federal facilities online. You can’t post illegal content (child porn) or copyrighted content.
Point is, speech has limits, online and off. So limiting speech online, in a community run by a private company, is not a violation of the First Amendment, no matter how important the site is.
A government can censor you, a private company cannot. So it matters which one you are. If the CEO of a huge community company is under the delusion that they’re the president of a country, they may think they have to allow hate speech because that’s what countries do. But that’s wrong. And it leads to inaction when the community is being attacked by bad actors.
Countries guarantee free speech. Your site is not a country. There is no guarantee of free speech on your site or any site, unless a government made it, which, again, is not you.
Governments Have Responsibilities
Finally, I try to remind executive teams that, even though it can be kind of thrilling to imagine yourselves as the ruling council of a great country, you really do not want that. Because governments have responsibilities.
Transparency is good when you’re a government. It’s even required in a lot of cases. Every vote tallied, every memo recorded. As CEO, you want all your emails to be public property? You want to be subject to elections every four years? (Maybe some of us would like that, but I know the CEOs wouldn’t.)
But transparency is not required for companies. And while many people think they should be more transparent, and maybe sometimes they should, there are some really good reasons not to be. Spelling out exactly what actions are taken for what rule-breaking gives the rule-breakers the tools they need to get away with it better. Think of all Google has to do to stay ahead of the people trying to game their search algorithm. That opaqueness keeps their results good for us. Opaqueness is valuable sometimes.
Governments are slow moving on purpose. You want to have your hands tied up in red tape? You want to have to do a long term study before rolling out a new feature? (Again, maybe we should.)
But, no, CEOs and executive teams prize their independence, privacy, and fleet footedness. So if you don’t want to subject yourself to the rules and responsibilities of governments, don’t pretend you are one.
Thinking of your site as a country, and yourself as a government, is seductive. It feels good. But it leads companies to make bad decisions for the wrong reasons, and we all suffer as a result.
Mark was so busy running Facebook like an empire, he didn’t notice when Russians used it to disrupt the 2016 US election. And in his confused effort to rationalize his community management inaction, he defended Holocaust denial, which is hate speech.
Twitter is now working so hard to protect the “free speech” of Alex Jones that it’s allowing him to use the platform to promote hate, libel innocent people, and poison the entire community. In his effort to appear presidential, Jack has condoned bigotry on the platform, which opens the door to a legion of smaller bigots to come in and harass everyone else with the blessing of the CEO.
It’s time for these guys to stop playing pretend politics and admit that they’re not presidents, their sites are not countries, and we are not their citizens. They are caretakers of communities and they’d better start acting like it or they won’t have anyone to rule anymore.
Let’s start from the beginning. Say you make a website that allows people to say something online. Most people use it like you intended, and everything’s fine.
If you’re exceptionally lucky, more people start to use your site. In fact, enough people use it that they begin to feel a kinship with each other.
Congratulations, you’ve got a community.
Because communities are made of people, and people are hopelessly, ridiculously complicated, eventually, inevitably, someone uses your site in a way you hadn’t expected, to say something terrible. So you remove that one thing.
Congratulations, you’re a community manager.
Again, you’re very lucky. The community grows. And grows. And soon there’s a lot of community management work to do. So you hire some people. And the first thing they ask is, but is this allowed? What about this other thing? And you realize that your community members have been asking those same questions.
So you write some things down. Rules. Community guidelines. Terms of service. And you think, I probably should have written these things down earlier. But it’s done now, so everything’s fine.
But then the community grows more and something truly frightening happens. Because you’ve been successful, because your community is so large, the community itself becomes a target. The bad actors arrive. The trolls, grifters, and criminals. They’re all coming because that’s where their victims, marks, and targets are. Because that’s where everybody is.
Congratulations, you’ve got a problem.
The community you started with optimism and naïveté is now a battleground. People get hurt feelings, but you can wave that away with idealistic platitudes about free speech. Then people get killed in SWATtings and broadcast their suicides, but you can just claim those are a few bad eggs.
And then you leak private data to Russians. And then you help elect Donald Trump. And then you get called in front of congress.
Congratulations, you’ve got a big fucking problem.
I’m old enough to remember when search engines were dumb. Alta Vista, HotBot, Lycos, Web Crawler. Names nobody remembers anymore. They were dumb because all they did was crawl the web and make indexes. Lists of which words appeared on which pages. So you’d type in a word and you’d get a list of pages that included that word. This was exciting at the time.
But it was dumb because knowing that a page included a word didn’t necessarily mean that page was about that word, or even necessarily good for that topic. And, worse, the people making the pages quickly learned that could just use a word a lot and get found that way. After all, a page that uses the word “dog” 400 times must really be about dogs, right?
The reason you “Google” things now and not “Alta Vista” them is because Google was the first company to really nail search results. And they did that with something they called PageRank. They made the usual index of which pages included which words, but then they made another list of which pages got linked to a lot, and used that linking behavior as a trust metric. So if a page got linked to a lot using the word “dog,” then it was a pretty good result for a search for “dog.”
This PageRank thing, they told us, was an “algorithm.” And, for a time, algorithms were all the rage. We were living in the age of the algorithm. And in all my client meetings and project plans, every time we had a decision to make, someone would say, “the algorithm will do it.”
The algorithm never did it.
When Facebook CEO Mark Zuckerberg testified in front of Congress on April 10, 2018, he was pressed repeatedly on what Facebook was doing to combat the rising tide of terribleness on his platform. And every time his answer was AI. He said it 25 times in one sitting.
So what’s AI? When you say “Artificial Intelligence” to a normal person, they probably think of a sentient robot. Star Trek’s Data. Star Wars’ C-3P0. It’s a romantic, futuristic notion. And totally wrong.
In the context of community moderation, all “AI” means is: teach a computer to do something so it can do it faster than you can. AI is the new algorithm – another way to avoid human responsibility.
If you want to see how meaningless the term “AI” is, just replace it with “recipe” when you see it.
SENATOR: How will you prevent abuse?
CEO: We will use a recipe.
SENATOR: A recipe? For what? What’s in the recipe? What does it do? Who’s making the recipe? How will it help?
CEO: That’s right! A RECIPE.
AI is just computers doing what they do. It’s not a solution to everything. And if we’re using it to avoid making hard decisions, then it’s part of the problem.
When technologists talk about AI, I think what they’re really talking about is machine learning, which is pretty cool and not nearly as new as people think it is. It’s been around since the 1960s. It just goes faster now because computers go faster.
Machine learning, at its simplest, it’s taking a pile of data, calling it a thing, and asking the computer to find more things like that. It’s pattern matching and computers are good at that.
Imagine you’re running an email system and you really need to help your users avoid spam. So you make a pile of spam messages and say, hey computer, this is spam. And the computer scans all that data and finds patterns in it. Then, when new messages come in, it can take a guess at how closely they match that pattern. This happens now, every second of every day, and every time you mark a message as spam, you’re adding to the pile and helping train the system.
What’s interesting about machine learning is that it requires you give the computer examples, but allows you to skip defining it. You can just let the computer find the similarities in the data. That works for something as simple as spam vs not-spam, or photos of faces vs photos of not-faces.
In his senate testimony, Zuckerberg claimed that internal “AI tools” at Facebook are already deployed against terrorist content. He said: “99 percent of the ISIS and Al Qaida content that we take down on Facebook, our AI systems flag before any human sees it.”
Even though this is unprovable (“trust us, we’re removing bad stuff before you see it 99% of the time!”), I don’t doubt it. Because terrorist content, like spam, is relatively easy to define, target, and remove. It’s identifiable because it includes telltale phrases, signifiers, and comes from known bad actors.
The trouble with taking this technique and applying it to general community management is that we are too messy, too inconsistent, too prone to human weirdness.
Anyone who’s ever managed a community knows how complicated people are. A reasonable community member can suddenly have a bad day. Sometimes things that look like bad contributions are honest mistakes. Other times things that look reasonable to a bystander are known to be abusive to the sender and recipient. (Nazis are using milk as a hate symbol. MILK.) When one person tells another to die on Twitter, it’s a threat. But when David Simon says it, he’s making a point. Abusers can use liking to remind their victims that they’re watching. And abuse isn’t limited to one system – just ask Kelly Marie Tran. Point is, we’re complicated critters.
Of course humans need tools to help manage community. I’ve built systems to do this. And, sure, machine learning can be part of that. But I fear the leaders of Twitter and Facebook are depending too much on technology (again), and overlooking the kinds of systems that are great at this kind of empathetic flexible pattern recognition: humans.
They’re also overlooking the reason they’re in this predicament in the first place: unfettered growth, design that encourages immediate engagement over thoughtfulness, and a general unwillingness to define and communicate who and what their platforms are for. Thus far, they’ve been for everyone and everything. It’s time to rethink that. While there’s a community for everyone, not everyone is welcome in every community, and that’s okay. That’s how communities work. And when the “everything” that your community is for includes destroying human lives and American democracy, it’s time to raise your standards.
You can’t create a system for everyone, where everything goes, not communicate the rules, not design for community, and then say it’s just too hard to protect everyone. This end state is the outcome of all of those decisions. And AI is not going to be the patch that fixes all the bugs.
AI is not a community management strategy because it’s skipping the hard part of community management: deciding what’s allowed and what’s not. You can’t skip the definition step in community management because that’s literally the very first thing you have to do, and the thing that only you can do. You can’t just give a pile of bad stuff to the computer and say “you figure it out.” That’s just outsourcing your responsibility.
You have to do the hard part. You have to decide what your platform is for and what it’s not for. And, yeah, that means deciding who it’s for and who it’s not for (hint: it’s not bots, nor nazis). That’s not a job you can outsource. The tech won’t do it for you. Not just because it’s your job, but because outsourcing it won’t work. It never does.
Call it “AI” or “machine learning” or “the algorithm” or whatever you like, but it’s really an abdication of your duty to care for the community that depends on you. And these days, that community is all of us, our fragile democracy, and possibly the stability of the world in general.
Hi there. This is just to say, lately I’ve been telling some stories on my mailing list, Somewhere In Between. You can check out the archive or sign up here.
Hope to see you there.
I’ve had depression so long, I don’t know what parts of my personality are it and what parts are me. When I was a teenager, I adopted it as part of my identity. I let it define me. I was just a dark, broody kid. But eventually I realized that was a lie. That was putting me in the back seat of my own car and I hate being a passenger.
In my twenties, depression became a thing I fought. It was a battle. I would show depression who was boss. Wrestle it to the ground. Prove I was stronger. This, of course, doesn’t work. Fighting it is just another way of letting it control you. It was still driving from the back seat.
In my thirties, depression became my secret. If I could present as okay, I could pretend I was fixed. I accomplished things. I was a professional. I had jobs, started companies, worked as a consultant. When you sell confidence, you can’t admit that you’re broken.
The depression was always there, a snake wrapped around me, but inside.
I have known a lot of addicts. I like hanging out with people in recovery because they understand something very important about the human condition. They know, if they continue to do the things they’ve always done, the things that feel the most natural, they will continue to fuck up their lives in all the ways they always have. So they choose a new reaction. They decide to do something else, something that is not their first instinct, and they have to keep choosing every day to make it stick.
I cannot keep doing the things I always have done. I can’t let depression define me, I can’t fight it, I can’t pretend it away. I have choose to do something else.
Now I’m in my mid-forties and I’m trying a new thing with my depression. I have no idea if it’ll work any better than all the others, but here it goes.
Step one is to admit it’s there. I kept it a secret for so long. I said it was because it could hurt me professionally if I admitted it, but that excuse is over now. The plants and animals here on the farm don’t care if I have dark thoughts – they care if the coop is clean and the barn roof doesn’t leak and the food is plentiful. They’re nice and selfish like that.
So hey. I have clinical depression. I always have. It sucks.
Having depression isn’t the same as being sad. I’m sad sometimes, too. But depression is different. Depression is being empty. Being nothing. Sometimes it’s loud and sometimes it’s quiet but it’s never completely gone. It’s part of me but it isn’t me.
And yeah, I’m on meds. I have been for a long time. I tried to go off of them, in a stunning bit of badly timed overconfidence, in November 2016. That was not a good idea. I’m back on them now. They help, but they’re not a cure. Probably they’ve helped me get to the point I’m at now, being able to type these words.
Admitting this is terrifying. It’s also, I hope, liberating. A secret can control you so long as it stays a secret. Telling a secret takes away its power.
Now that my depression exists as a fact, I need a new way to think about it. It’s not me, not part of my identity, even though it lives inside me. I used to visualize it as a dark shadow, or a powerful snake, wrapping around me.
But that’s giving it too much power. My depression isn’t a character. It doesn’t have desires. It feels like it does sometimes, but that’s not helpful. It plays into the fight narrative, which is self-defeating.
So lately I’ve been thinking about it in a new way. And it’s scary, but don’t let it scare you.
Depression is a loaded gun. It’s a dangerous weapon, sitting on a table, unattended.
If you hang out with gun people, you learn quickly that they have an intense understanding of how dangerous the things are. They have rituals around the guns – cleaning them, locking them up when not in use, where your fingers go when you touch them, where you point the gun and when – there are rules and they’re life and death important.
My depression is like that gun. You can’t leave it on the table unattended. If you do, someone will get hurt. Probably me, but maybe somebody else.
The gun must be cared for. Taken apart, pieces examined, cleaned, and put back together. If I ignore it, it will become unstable. More dangerous.
You never, ever point a gun at someone else unless you intend to use it. This is a reminder that the people around me aren’t responsible for my depression. They’re not the cause of it, or the solution to it. Never point it at them.
There are times when a gun is a tool you need to use for a job. My depression can be like that. I visualize pointing my depression at things. Things I need to do. Using that dark energy to accomplish something light.
But most importantly, I must consciously choose to point the gun at something. Every day. If I don’t, it’ll wind up pointed at me.
The other day I pointed my depression at the blackberry brambles. Blackberries grow really well here in the Pacific Northwest. We have three rows of them, maybe 100 feet long in total, and they became a mountain of thorns this year. I ignored them too long and they became a hazard. I can’t imagine a more perfect metaphor for so much of this terrible year.
The goats helped defoliate them, and then it was my turn. I wrapped myself in the toughest clothing I had, with the thickest gloves (not nearly thick enough), and waded into the brambles with a lopper and my trusty pruners. I worked my way from one end to the other over the course of a week, chopping down the canes and dragging them to the burn pile.
There was no quick fix for this. They were too tough for the riding mower. This was something I had to just knuckle down and do. It was cold out, but I worked up a sweat. I listened to podcasts until they ran out, then I listened to music. My body ached and my back hurt, but slowly the brambles grew shorter and the pile grew taller.
By the end, the blackberries were cut down to the ground, the burn pile was ten feet tall and ten feet around, I was bleeding in several places, and I could feel the depression ebb.
It wasn’t gone – it’s never gone – but it felt farther away. And when I lit the burn pile and watched the canes disappear in screams and smoke, I felt good for the first time in a long time.
Now I walk the property with my loaded depression gun and look for other things to point it at. That pile of boxes is next. Some dead trees that need to come down. Anyplace I can point the gun that’s not a loved one or my own head. If it needs to get done, if it’s going to make me sweat to do it, and nobody’s gonna get hurt, that’s a good place to point my gun.
If I can accept my depression, treat it with the respect of a loaded gun, and use it to get the farm work done, and if that farm work in the end makes me happy, maybe my depression and I can coexist in peace.
Dear friends and family,
So we’re coming up on the holidays, and you may have started to think about what to get me this year. I’ve never done this before, but this is a year like no other, so I’m doing something different.
This year, instead of buying me something, I’d like you to take that cash and make a donation to one of the organizations listed below instead. You can do so in my name.
I’m lucky. I have everything I need. I have a wife who loves me, a farm full of animals, and food on the table. And I have you, someone who cares enough about me to read this. Thank you. But there are many in this country who do not have those things, and many who are desperately afraid for their future right now.
I don’t know how you voted in this election, and I don’t want to. No matter who you voted for, the fact is, the incoming administration is on the record: they plan to dismantle health care, disregard environmental regulations, fight the free press, and take away the civil rights of many who only recently achieved them.
The organizations that will stand up to these injustices are more important than ever. They need the cash more than I do. Please support them instead of me.
I am terrified for the future of this country for the first time in my adult life. Your donation to one of these organizations, even if you don’t agree with them, is the greatest gift you could give me.
Please donate to:
American Civil Liberties Union
Color of Change
Center for Reproductive Rights
Council on American-Islamic Relations
Electronic Frontier Foundation
International Refugee Assistance Project
International Rescue Committee
Mexican American Legal Defense and Educational Fund
NARAL Pro-Choice America
NAACP Legal Defense Fund
National Organization for Women
Natural Resources Defense Council
Southern Poverty Law Center