this post was submitted on 01 Aug 2024
553 points (97.8% liked)

Technology

57997 readers
2866 users here now

This is a most excellent place for technology news and articles.


Our Rules


  1. Follow the lemmy.world rules.
  2. Only tech related content.
  3. Be excellent to each another!
  4. Mod approved content bots can post up to 10 articles per day.
  5. Threads asking for personal tech support may be deleted.
  6. Politics threads may be removed.
  7. No memes allowed as posts, OK to post as comments.
  8. Only approved bots from the list below, to ask if your bot can be added please contact us.
  9. Check for duplicates before posting, duplicates may be removed

Approved Bots


founded 1 year ago
MODERATORS
 

Reddit CEO Steve Huffman is standing by Reddit’s decision to block companies from scraping the site without an AI agreement.

Last week, 404 Media noticed that search engines that weren't Google were no longer listing recent Reddit posts in results. This was because Reddit updated its Robots Exclusion Protocol (txt file) to block bots from scraping the site. The file reads: "Reddit believes in an open Internet, but not the misuse of public content." Since the news broke, OpenAI announced SearchGPT, which can show recent Reddit results.

The change came a year after Reddit began its efforts to stop free scraping, which Huffman initially framed as an attempt to stop AI companies from making money off of Reddit content for free. This endeavor also led Reddit to begin charging for API access (the high pricing led to many third-party Reddit apps closing).

In an interview with The Verge today, Huffman stood by the changes that led to Google temporarily being the only search engine able to show recent discussions from Reddit. Reddit and Google signed an AI training deal in February said to be worth $60 million a year. It's unclear how much Reddit's OpenAI deal is worth.

Huffman said:

Without these agreements, we don’t have any say or knowledge of how our data is displayed and what it’s used for, which has put us in a position now of blocking folks who haven’t been willing to come to terms with how we’d like our data to be used or not used.

“[It’s been] a real pain in the ass to block these companies,” Huffman told The Verge.

top 50 comments
sorted by: hot top controversial new old
[–] morgunkorn@discuss.tchncs.de 425 points 1 month ago* (last edited 1 month ago) (3 children)

Honestly, any platforms hosting user-generated content who use the legal argument that they only provide hosting and aren't responsible for what their user post shouldn't also be able to sell the same data and claim owning any of it.

Otherwise, take away their legal immunity. Nazis or pedophiles post something awful? You get in front of the judge.

edit: typo

[–] jlh@lemmy.jlh.name 180 points 1 month ago (1 children)

Exactly this. You can claim that their scraping is abusing your servers, but the moment you claim copyright for the content of the site, then you give up your Section 230 rights.

[–] fuckwit_mcbumcrumble@lemmy.dbzer0.com 35 points 1 month ago (2 children)

You'd also probably lose a whole lot more processing power trying to stop the crawlers vs just letting them have API access with some sort of limit to queries.

[–] ptz@dubvee.org 29 points 1 month ago* (last edited 1 month ago)

Eh, not really.

I block bot user agents to my Lemmy instance, and the overhead is pretty negligible for that (it's all handled in my web firewall/load balancer).

Granted, those are bots that correctly identify themselves via user agent and don't spoof a browser's.

It's also cheaper and easier to add another load balancer than size up or scale out my DB server to handle the bot traffic.

[–] rbits@lemm.ee 6 points 1 month ago (1 children)

I don't think they actually block malicious bots, the change they've made is just to the robots.txt, they don't have to do anything.

[–] tb_@lemmy.world 9 points 1 month ago (3 children)

Robots.txt does literally nothing. It's a piece of courtesy that's easily ignored if you don't care.

load more comments (3 replies)
[–] givesomefucks@lemmy.world 38 points 1 month ago (2 children)

Can't sell something you don't own.

So if they're selling the parts people want, they need to own the parts no one wants.

load more comments (2 replies)
load more comments (1 replies)
[–] JaymesRS@literature.cafe 102 points 1 month ago* (last edited 1 month ago)

Robots.txt isn’t a binding agreement, this isn’t stopping anyone for whom their drive for profit outweighs their ethics.

Also, Fuck Spez.

[–] Sordid@lemmy.world 66 points 1 month ago (5 children)

The enshittification cycle:

Phase one, attract users by providing a good service.
Phase two, once the users are locked in, squeeze them for all they're worth by selling them to business customers (advertisers and/or data buyers).
Phase three, once the business customers are locked in, squeeze them for all they're worth by threatening to deny them access to the users on whom they now depend.

Spez seems to think Reddit has the pull to make phase 3 happen. I rather doubt it, but we'll see.

[–] taaz@biglemmowski.win 28 points 1 month ago* (last edited 1 month ago)

https://catvalente.substack.com/p/stop-talking-to-each-other-and-start

Blog post (?) from Catherynne Valente about this exact topic

Stop benefitting from the internet, it’s not for you to enjoy, it’s for us to use to extract money from you. Stop finding beauty and connection in the world, loneliness is more profitable and easier to control.

[–] lemmyvore@feddit.nl 14 points 1 month ago

If he really had balls he'd restrict access to the site and improve the built-in search engine.

If reddit's own search worked well nobody would care. Engines like DDG even have bang codes that send you to a site's own engine. So instead of having to add "site:reddit.com" to the search on DDG I'd just add "r!" and it would end up being the same thing. IF the internal search didn't suck.

[–] ruk_n_rul@monyet.cc 12 points 1 month ago

Spez is tracing Elon's steps with X really closely.

Also fuck spez.

[–] _haha_oh_wow_@sh.itjust.works 9 points 1 month ago* (last edited 1 month ago)

Yeah, as soon as the API thing happened I switched to Lemmy for mobile browsing and like it more than Reddit (Connect is pretty good, but even the mobile browser site is solid).

The more they squeeze, the more popular alternatives like Lemmy, Kbin/Mbin, Tildes, etc. will become.

[–] Boozilla@lemmy.world 6 points 1 month ago

My guess is that phase three will work for a while. But I think you're right that eventually they are going to drive that thing into the ground. Because it's never enough pure profit for rent-seeking scum, and there is no lower limit to the abuse they'll inflict on their content creators (who they call users but think of as products).

[–] ayyy@sh.itjust.works 58 points 1 month ago

Fuck Spez. He’s probably editing the comments anyway, he literally can’t help himself.

[–] Vespair@lemm.ee 46 points 1 month ago

I'm sure plenty of others join me in the sentiment of thinking "Who the fuck are you to restrict MY free content that I contributed?"

God, fuck reddit so fucking hard

[–] spongebue@lemmy.world 46 points 1 month ago (2 children)

Honestly, my biggest issue with LLMs is how they source their training data to create "their own" stuff. A meme calling it a plagiarism machine struck a chord with me. Almost anyone else I'd sympathize with, but fuck Spez.

[–] Wirlocke@lemmy.blahaj.zone 6 points 1 month ago (1 children)

What resonated with me is people calling LLMs and Stable Diffusion "copyright laundering". If copyright ever swung in AI's favor it would be super easy to train an AI on stuff you want to steal, add in some generic training, and now you have a "new" piece of art.

LLMs and Stable Diffusion are just compression algorithms for abstract patterns, only one level above data.

[–] echodot@feddit.uk 6 points 1 month ago* (last edited 1 month ago)

The real takeaway of all of this is that copyright law is massively out of date and not fit for purpose in the 21st century or frankly the late 20th.

The current state of copyright law cannot deal with the internet, let alone AI

load more comments (1 replies)
[–] bappity@lemmy.world 41 points 1 month ago

profiting off of user generated content 😒

[–] catloaf@lemm.ee 41 points 1 month ago (2 children)

And yet reddit is happy to make money off our content for free.

Or at least it did. Personally I overwrote and deleted all my content a while back.

[–] iAmTheTot@sh.itjust.works 15 points 1 month ago (22 children)

You think that Reddit didn't already have the previous content saved?

[–] mp3@lemmy.ca 18 points 1 month ago

Bingo, the only winning move is not to play at all and stop using Reddit.

[–] dual_sport_dork@lemmy.world 9 points 1 month ago (3 children)

Everyone always says this like it's some kind of gotcha, but all of my nuked posts still have my "fuck you, reddit" content and haven't been reverted. It's been nearly exactly a year.

Maybe reddit has an offline copy of my old content and that of others somewhere, but if so they'd be handing that directly over to whoever under some kind of agreement -- that certainly wouldn't be the subject of any kind of site crawling which is the crux of the issue here.

[–] Womble@lemmy.world 6 points 1 month ago

it never was deleted, all that happened is that an extra line was added to a database that said "comment 65432426542654 now should be displayed as "fuck you, reddit" rather than the original text". The original post is still in an earlier row available to reddit, it just isnt being displayed on their web page.

[–] stevedidwhat_infosec@infosec.pub 5 points 1 month ago (7 children)

You’re ignoring the idea that they could still be working on a way to restore content and haven’t completed that process yet

Or that they could start feeding your archived (not cached) data directly to the AI companies anyway for a price

IMO, you can win by jamming your “transmissions” with noise. It’s easier to hide in noise as noise than it Is to be silent IMO. Muddy the waters as it were

load more comments (7 replies)
load more comments (1 replies)
load more comments (20 replies)
load more comments (1 replies)
[–] Brkdncr@lemmy.world 29 points 1 month ago

What if I had an agreement with MS that they can scrape my data and anything I post online?

[–] ChadCMulligan@lemm.ee 24 points 1 month ago (3 children)
[–] ptz@dubvee.org 25 points 1 month ago (1 children)

In all the ways that matter, it's already dead. Once something enshittifies beyond a certain point, is its zombified, shambling corpse really considered "alive" anymore?

load more comments (1 replies)
[–] iAmTheTot@sh.itjust.works 12 points 1 month ago (1 children)

It's easy to say this as someone who is "on the other side". But the data doesn't really back up that statement.

[–] catloaf@lemm.ee 6 points 1 month ago* (last edited 1 month ago) (1 children)

I don't have data, but the quality of the content certainly seemed to be declining, even as the quantity went up.

[–] _haha_oh_wow_@sh.itjust.works 6 points 1 month ago (1 children)

I think autocorrect might've gotten you: You posted "quality" twice in a contradictory way.

load more comments (1 replies)
[–] li10@feddit.uk 7 points 1 month ago

It’s awful. Politics is unavoidable at this point, and the amount of general anger on the platform is crazy.

People love watching their videos of people getting TBIs… Or getting too excited about a “justice served” post where a woman gets hit.

It’s kinda nice to see someone get their comeuppance, but then you look in the comments and there are just weirdos saying stuff like “glad that bitch got hit”, like… wtf?

[–] boonhet@lemm.ee 15 points 1 month ago (1 children)

I never bothered to go edit or delete my comments after the API drama that caused me to move here, but now I might just go do that because the entire point of keeping old comments up was that maybe someone will find one from a search engine and find it useful. If reddit is going to monetize THAT, they can fuck right off.

[–] palordrolap@kbin.run 15 points 1 month ago

Save your effort. What's already there is there forever. They can just roll back your comments, or even, if they're in the mood for it, make it appear under an entirely different username.

The only way to win is not give them any more. And that fight is already under way. They've already started recommending old comments after new ones because the quality isn't as high any more.

Think about it: The only people who contribute to Reddit now are the clueless and the sort of people who have willingly stayed.

I like to imagine Spez stomping around saying "Hmph! Hmph! It's not fair! Why did they all leave?! They're stealing my revenue by not giving me anything for free!". I mean, he's probably not doing that, but I do like to imagine it.

[–] MedicPigBabySaver@lemmy.world 8 points 1 month ago (1 children)
[–] _haha_oh_wow_@sh.itjust.works 7 points 1 month ago

Gross.

No thanks.

[–] MataVatnik@lemmy.world 7 points 1 month ago

Late stage capitalism

[–] MichaelTen@lemmy.world 7 points 1 month ago (1 children)

Minimum royalty laws should exist!

load more comments (1 replies)
[–] werefreeatlast@lemmy.world 6 points 1 month ago

How about starting a company that gathers people's CAD design....grabCAD!.... Oh can't scrape out design work Microsoft, you gotta pay!....or how about a company that stores people's records or drawings or movies.... Adobe! Oh Microsoft, you can't scrape our data! It's our data!

[–] Brkdncr@lemmy.world 5 points 1 month ago (1 children)

What if I had an agreement with MS that they can scrape my data and anything I post online?

load more comments (1 replies)
[–] Lemminary@lemmy.world 5 points 1 month ago (1 children)

I'm not sure what Reddit is still doing with that spaz. Ditch the fucker.

load more comments (1 replies)
load more comments
view more: next ›