• Home
  • Latest
  • Fortune 500
  • Finance
  • Tech
  • Leadership
  • Lifestyle
  • Rankings
  • Multimedia

Trendingnow

1

MacKenzie Scott alone accounted for one-third of America's $19.2 billion in megagifts last year

2

Philanthropy leader at Warren Buffett and Bill Gates’ Giving Pledge says children of billionaires are pushing them to give their wealth away faster

3

Elon Musk on MacKenzie Scott giving away $26 billion of her fortune: 'Sadly,' it makes the world a worse place

1

MacKenzie Scott alone accounted for one-third of America's $19.2 billion in megagifts last year

2

Philanthropy leader at Warren Buffett and Bill Gates’ Giving Pledge says children of billionaires are pushing them to give their wealth away faster

3

Elon Musk on MacKenzie Scott giving away $26 billion of her fortune: 'Sadly,' it makes the world a worse place
TechTikTok

TikTok’s parent launched a web scraper that’s gobbling up the world’s online data 25 times faster than OpenAI

By
Kali Hays
Kali Hays
Down Arrow Button Icon
By
Kali Hays
Kali Hays
Down Arrow Button Icon
October 3, 2024, 9:00 PM ET
A person holding a phone in front of a large TikTok logo
TikTok’s parent company is using a bot that scrapes online content at far greater quantities than rivals. Artur Widak/NurPhoto via Getty Images
Add Fortune on Google for similar content.

ByteDance looks like it’s eager to make up for lost time when it comes to scraping the web for data needed to train its generative AI models.

Recommended Video

The China-based parent company of video app TikTok released its own web crawler or scraper bot, dubbed Bytespider, sometime in April, according to research from Kasada, a company that specializes in bot management for companies with online data. The existence of the bot was also confirmed by Dark Visitors, which monitors scraper bots.

ByteDance’s bot has quickly become one of the most, if not the single most, aggressive scrapers on the internet, the research shows. It’s scraping data at a rate that’s many multiples of other major companies, such as Google, Meta, Amazon, OpenAI, and Anthropic, which use their own scraper bots to help create and improve their large language or multimodal models, known as LLMs or LMMs.

Sam Crowther, the CEO of Kasada, said since Bytespider showed up, it’s been scraping data at about 25 times the rate of GPTbot, which scrapes data for OpenAI’s ChatGPT platform and underlying models, for instance. Bytespider has been scraping at 3,000 times the rate of ClaudeBot, from Anthropic, which operates the Claude platform.

As the months have gone by, Bytespider has become even more aggressive, according to Kasada. Data shows huge spikes in scraping activity from Bytespider over each of the last six weeks.

Representatives of TikTok and ByteDance did not respond to emails seeking comment.

ByteDance’s aggressive scraping comes despite the possibility of TikTok being banned in the U.S. in the coming months. President Joe Biden has signed legislation that requires ByteDance to sell TikTok, due to national security concerns, or shut it down.

The Bytespider bot, much like those of OpenAI and Anthropic, does not respect robots.txt, the research shows. Robots.txt is a line of code that publishers can put into a website that while not legally binding in any way, is supposed to signal to scraper bots that they cannot take that website’s data. 

Web scraping goes back decades, mainly by search engines to gather links to web pages. But the rise of generative AI tools has added a new dimension and made the practice a prime source of lawsuits and controversy. People and organizations whose work has been scraped argue their copyright is being infringed in the process. All of the models that underlay generative AI tools were trained on massive amounts of online data, effectively everything available on the web, particularly written information. Tech companies use scraper bots to essentially copy it all for free and put it into their datasets.

“It’s like they’re trying desperately to catch up,” Crowther said of the aggressive scraping being done by Bytespider. Just last year, ByteDance was reportedly so far behind in the generative AI race that it was using OpenAI to help build ByteDance’s own LLM, which is against OpenAI’s terms of service. Earlier this year, ByteDance released a chat-based LLM called Doubao, but work on that model would have been completed prior to the accumulation of more recent training data scraped by Bytespider.

It’s “clear” that ByteDance is at work on a new LLM, according to one person familiar with the company. As for what ByteDance plans to do with a new LLM, a person familiar with the company’s ambitions said one goal has to do with the search function for TikTok.

Last week, TikTok released an update to its current search function focused on keywords for ads, basically allowing advertisers to search in real time for words that are trending on TikTok. It allows marketers to build an ad with relevant keywords that would ostensibly help the ad show up on the screens of more users.

A new AI model with data on more recent internet trends and topics could expand and improve TikTok’s search environment further, according to the person familiar with the company’s ambitions. 

“Given the audience and the amount of use, TikTok with a search environment that is a completely biddable space with keywords and topics, that would be very interesting to a lot of people spending a ton of money with Google right now,” the person said.

Are you a TikTok or ByteDance employee or someone with insight or a tip to share? Contact Kali Hays securely through Signal at +1-949-280-0267 or at kali.hays@fortune.com.

About the Author
By Kali Hays
See full bioRight Arrow Button Icon
Add Fortune on Google for similar content.

Latest in Tech

Finance
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam
By Fortune Editors
October 20, 2025
Finance
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam
By Fortune Editors
October 20, 2025
Finance
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam
By Fortune Editors
October 20, 2025
Finance
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam
By Fortune Editors
October 20, 2025
Finance
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam
By Fortune Editors
October 20, 2025
Finance
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam
By Fortune Editors
October 20, 2025

Most Popular

Finance
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam
By Fortune Editors
October 20, 2025
Finance
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam
By Fortune Editors
October 20, 2025
Finance
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam
By Fortune Editors
October 20, 2025
Finance
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam
By Fortune Editors
October 20, 2025
Finance
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam
By Fortune Editors
October 20, 2025
Finance
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam
By Fortune Editors
October 20, 2025
Fortune Secondary Logo
Rankings
  • 100 Best Companies
  • Fortune 500
  • Global 500
  • Fortune 500 Europe
  • Most Powerful Women
  • World's Most Admired Companies
  • See All Rankings
  • Lists Calendar
Sections
  • Finance
  • Fortune Crypto
  • Features
  • Leadership
  • Health
  • Commentary
  • Success
  • Retail
  • Mpw
  • Tech
  • Lifestyle
  • CEO Initiative
  • Asia
  • Politics
  • Conferences
  • Europe
  • Newsletters
  • Personal Finance
  • Environment
  • Magazine
  • Education
Customer Support
  • Frequently Asked Questions
  • Customer Service Portal
  • Privacy Policy
  • Terms Of Use
  • Single Issues For Purchase
  • International Print
Commercial Services
  • Advertising
  • Fortune Brand Studio
  • Fortune Analytics
  • Fortune Conferences
  • Business Development
  • Group Subscriptions
About Us
  • About Us
  • Press Center
  • Work At Fortune
  • Terms And Conditions
  • Site Map
  • About Us
  • Press Center
  • Work At Fortune
  • Terms And Conditions
  • Site Map
  • Facebook icon
  • Twitter icon
  • LinkedIn icon
  • Instagram icon
  • Pinterest icon

Latest in Tech

Current price of Ethereum for July 1, 2026
Personal FinanceEthereum
Current price of Ethereum for July 1, 2026
By Joseph HostetlerJuly 1, 2026
2 hours ago
In this photo illustration, a Cisco logo is displayed on a smartphone with Artificial Intellingence (AI) symbols in the background.
AICFO Daily
Cisco is rolling out AI agents to every single one of its 90,000 employees
By Sheryl EstradaJuly 1, 2026
2 hours ago
senate
CommentaryCongress
One rare bipartisan AI bill is moving through Congress. Here’s why it deserves to pass
By Neil Björkman and Betsy BrewerJuly 1, 2026
4 hours ago
I know how Gen Z can survive the ‘jobpocalypse’ because I built an AI company — in 2015
CommentaryCareers
I know how Gen Z can survive the ‘jobpocalypse’ because I built an AI company — in 2015
By Jeremy FainJuly 1, 2026
4 hours ago
OCBC rolls out its ‘avatar banking’ platform with ‘Wendy’ and ‘Wayne,’ two virtual financial advisors, as banks integrate AI into wealth management
AsiaSingapore
OCBC rolls out its ‘avatar banking’ platform with ‘Wendy’ and ‘Wayne,’ two virtual financial advisors, as banks integrate AI into wealth management
By Angelica AngJuly 1, 2026
5 hours ago
Photo: Rocks balancing on driftwood, sea in background.
AIMarkets
Leveraged stock bets are ‘very concentrated in the AI ecosystem,’ Goldman Sachs warns
By Jim EdwardsJuly 1, 2026
5 hours ago

Most Popular

MacKenzie Scott alone accounted for one-third of America's $19.2 billion in megagifts last year
Success
MacKenzie Scott alone accounted for one-third of America's $19.2 billion in megagifts last year
By Sydney LakeJune 25, 2026
6 days ago
Philanthropy leader at Warren Buffett and Bill Gates’ Giving Pledge says children of billionaires are pushing them to give their wealth away faster
Success
Philanthropy leader at Warren Buffett and Bill Gates’ Giving Pledge says children of billionaires are pushing them to give their wealth away faster
By Preston ForeJune 27, 2026
4 days ago
Elon Musk on MacKenzie Scott giving away $26 billion of her fortune: 'Sadly,' it makes the world a worse place
Success
Elon Musk on MacKenzie Scott giving away $26 billion of her fortune: 'Sadly,' it makes the world a worse place
By Sydney LakeJune 29, 2026
2 days ago
The U.S. Army is opening military bases to private billions — here's why that changes everything for the next 250 years
Commentary
The U.S. Army is opening military bases to private billions — here's why that changes everything for the next 250 years
By Marc AndersenJune 30, 2026
1 day ago
Current price of oil as of June 30 2026
Personal Finance
Current price of oil as of June 30 2026
By Joseph HostetlerJune 30, 2026
1 day ago
As Big Tech showers employees with perks to win the talent war, Nvidia built a nearly $5 trillion company by making people pay for their own lunch
Big Tech
As Big Tech showers employees with perks to win the talent war, Nvidia built a nearly $5 trillion company by making people pay for their own lunch
By Marco Quiroz-GutierrezJuly 1, 2026
8 hours ago

© 2026 Fortune Media IP Limited. All Rights Reserved. Use of this site constitutes acceptance of our Terms of Use and Privacy Policy | CA Notice at Collection and Privacy Notice | Do Not Sell/Share My Personal Information
FORTUNE is a trademark of Fortune Media IP Limited, registered in the U.S. and other countries. FORTUNE may receive compensation for some links to products and services on this website. Offers may be subject to change without notice.