Technology

59086 readers

3496 users here now

This is a most excellent place for technology news and articles.

Our Rules

Follow the lemmy.world rules.
Only tech related content.
Be excellent to each another!
Mod approved content bots can post up to 10 articles per day.
Threads asking for personal tech support may be deleted.
Politics threads may be removed.
No memes allowed as posts, OK to post as comments.
Only approved bots from the list below, to ask if your bot can be added please contact us.
Check for duplicates before posting, duplicates may be removed

Approved Bots

founded 1 year ago

MODERATORS

554

Blocking AI bots from Microsoft, others has been “pain in the a**”: Reddit CEO | Huffman says companies must pay to scrape Reddit data even though Reddit itself relies on free, user-generated content (arstechnica.com)

submitted 3 months ago* (last edited 3 months ago) by ForgottenFlux@lemmy.world to c/technology@lemmy.world

85 comments fedilink hide all child comments

Reddit CEO Steve Huffman is standing by Reddit’s decision to block companies from scraping the site without an AI agreement.

Last week, 404 Media noticed that search engines that weren't Google were no longer listing recent Reddit posts in results. This was because Reddit updated its Robots Exclusion Protocol (txt file) to block bots from scraping the site. The file reads: "Reddit believes in an open Internet, but not the misuse of public content." Since the news broke, OpenAI announced SearchGPT, which can show recent Reddit results.

The change came a year after Reddit began its efforts to stop free scraping, which Huffman initially framed as an attempt to stop AI companies from making money off of Reddit content for free. This endeavor also led Reddit to begin charging for API access (the high pricing led to many third-party Reddit apps closing).

In an interview with The Verge today, Huffman stood by the changes that led to Google temporarily being the only search engine able to show recent discussions from Reddit. Reddit and Google signed an AI training deal in February said to be worth $60 million a year. It's unclear how much Reddit's OpenAI deal is worth.

Huffman said:

Without these agreements, we don’t have any say or knowledge of how our data is displayed and what it’s used for, which has put us in a position now of blocking folks who haven’t been willing to come to terms with how we’d like our data to be used or not used.

“[It’s been] a real pain in the ass to block these companies,” Huffman told The Verge.

you are viewing a single comment's thread
view the rest of the comments

[–] dual_sport_dork@lemmy.world 9 points 3 months ago (3 children)

Everyone always says this like it's some kind of gotcha, but all of my nuked posts still have my "fuck you, reddit" content and haven't been reverted. It's been nearly exactly a year.

Maybe reddit has an offline copy of my old content and that of others somewhere, but if so they'd be handing that directly over to whoever under some kind of agreement -- that certainly wouldn't be the subject of any kind of site crawling which is the crux of the issue here.

[–] Womble@lemmy.world 6 points 3 months ago

it never was deleted, all that happened is that an extra line was added to a database that said "comment 65432426542654 now should be displayed as "fuck you, reddit" rather than the original text". The original post is still in an earlier row available to reddit, it just isnt being displayed on their web page.

[–] stevedidwhat_infosec@infosec.pub 5 points 3 months ago (1 children)

You’re ignoring the idea that they could still be working on a way to restore content and haven’t completed that process yet

Or that they could start feeding your archived (not cached) data directly to the AI companies anyway for a price

IMO, you can win by jamming your “transmissions” with noise. It’s easier to hide in noise as noise than it Is to be silent IMO. Muddy the waters as it were

[–] finley@lemm.ee 3 points 3 months ago (1 children)

You’re ignoring the idea that they could still be working on a way to restore content and haven’t completed that process yet

there's no evidence to suggest this, though.

[–] stevedidwhat_infosec@infosec.pub 1 points 3 months ago (1 children)

Content is absolutely archived and they have financial incentive to restore the quality of their “knowledge base”

That’s a fair amount of circumstance and motivation to support my idea, regardless of tangible evidence

[–] finley@lemm.ee 0 points 3 months ago (1 children)

Motivation and circumstance, absent actual evidence, does not make for a convincing argument.

[–] stevedidwhat_infosec@infosec.pub 1 points 3 months ago (1 children)

Alright well I guess evidence is needed before we can have ideas - crazy

[–] finley@lemm.ee 0 points 3 months ago (1 children)

No, it just means that they are no more than ideas at this point

[–] stevedidwhat_infosec@infosec.pub 2 points 3 months ago* (last edited 3 months ago) (1 children)

Right, which means it can be fairly considered when discussing the real crux of the issue with AI and big tech companies right now, which is the monetization of other peoples content.

If we’re discussing this, we should be looking at whether or not companies are doing this, given they have motive and specific, relevant circumstance to enact such behavior.

Lack of evidence means you need to investigate for said evidence. It does not mean you should not investigate. Privacy advocates, members of any org/cert with an ethics statement should be blowing the whistle on any kind of activity that would mean a users data is not being deleted upon their request, especially considering reddits global usage.

[–] finley@lemm.ee -1 points 3 months ago* (last edited 3 months ago)

By all means investigate, I’m just saying there has yet to be presented any actual evidence. I look forward to seeing whatever you may discover.

[–] iAmTheTot@sh.itjust.works 2 points 3 months ago

I certainly wasn't implying that they were going to revert your comments.