# Bytespider

Bytespider is ByteDance's crawler for collecting training data for ByteDance's AI models. It reads public pages so future models know about them. It does not fetch pages to answer a specific question.

## Should you allow Bytespider?

**Your choice.** Blocking Bytespider keeps your content out of future training data. It does not remove you from AI search answers, which use separate bots. Being in training data can still help models know your brand when they answer without searching.

## robots.txt

To allow Bytespider everywhere:

```text
User-agent: Bytespider
Allow: /
```

To block it:

```text
User-agent: Bytespider
Disallow: /
```

Bots follow the most specific `User-agent` group that matches them. If you have a `User-agent: Bytespider` group, the rules under `User-agent: *` no longer apply to it. Matching on the token is case-insensitive.

## Identifying Bytespider

Bytespider identifies itself with `Bytespider` in its `User-Agent` header. It **does not run JavaScript**: it reads the raw HTML only. Content your site adds with JavaScript is invisible to it.

ByteDance doesn't publish a machine-readable IP range list for Bytespider that we verify against, so treat the user agent alone as unverified. `bots_report` marks these hits as unverifiable rather than genuine.

## Check your site

```bash
llmentions audit example.com            # is Bytespider allowed by robots.txt, and what does it see?
llmentions bots /var/log/nginx/access.log   # did Bytespider really visit, and is it genuine?
```

Or identify any user agent over REST (free): `GET https://api.llmentions.com/v1/bots/lookup?ua=Bytespider`.

---
https://llmentions.com/bots/bytespider/
