this post was submitted on 07 Oct 2024
564 points (98.8% liked)
Technology
58701 readers
3994 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related content.
- Be excellent to each another!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, to ask if your bot can be added please contact us.
- Check for duplicates before posting, duplicates may be removed
Approved Bots
founded 1 year ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
Isn't there a way to limit requests so that traffic isn't bringing down your servers
They obfuscate their traffic by randomizing user agents, so it's either add a global rate limit, or let them ass fuck you
the article told all source IPs can be tracked back to bytedance. Wouldn't it be possible to block them? maybe even blocking all IPs of a specific ASN
They can be tracked back one by one but if you have any amount of traffic it's a constant game of cat and mouse.
You can block entire ASNs until they start using residential proxies provided by less ethical companies. Then you end up blocking all of France or destroying user experience by enforcing a captcha on everyone.
Why do they need to hit a website like that? Wouldn't it just need to scrape the data and frig off. What is the point of creating that much traffic