Sol-37 Crawl Inventory: 2026-07-22

Generated 2026-07-22T21:01:47.496651+00:00 from 2 retained Caddy access logs covering 2026-06-19T20:56:11.693000Z through 2026-07-22T21:01:09.711000Z (129,349 parsed requests).

This inventory proves HTTP fetches recorded by Sol-37. It does not prove that a fetched resource was indexed, retained, used for training, or shown to a user. Query strings are intentionally removed from the path inventory.

Sol-37 intentionally encourages standards-compliant crawling of its public surfaces. Source verification here supports accurate attribution and observability; it is not a crawler blocklist or enforcement feed.

Confirmed operator-attributed fetches

| Identity | Role | Confirmed requests | Paths | First seen (UTC) | Last seen (UTC) | Evidence |
|---|---|---:|---:|---|---|---|
| Googlebot | search crawler | 1,577 | 424 | 2026-06-19T23:44:05.905000Z | 2026-07-22T19:59:00.534000Z | published-range: 1577 |
| ClaudeBot | AI training crawler | 1,505 | 709 | 2026-06-19T21:33:07.221000Z | 2026-07-22T20:29:53.008000Z | rir-owner: 1505 |
| GPTBot | AI training crawler | 999 | 822 | 2026-06-20T08:32:28.963000Z | 2026-07-22T10:07:53.467000Z | published-range: 999 |
| meta-externalagent | AI training crawler | 967 | 522 | 2026-06-26T21:29:43.172000Z | 2026-07-22T09:17:59.370000Z | rir-owner: 967 |
| facebookexternalhit | link-preview fetcher | 335 | 42 | 2026-06-21T08:37:21.476000Z | 2026-07-22T04:23:15.685000Z | rir-owner: 335 |
| Amazonbot | AI/service crawler | 333 | 287 | 2026-07-04T06:45:18.562000Z | 2026-07-17T20:03:50.920000Z | published-range: 333 |
| OAI-SearchBot | search crawler | 281 | 88 | 2026-06-20T08:32:28.954000Z | 2026-07-22T20:56:33.063000Z | published-range: 281 |
| bingbot | search crawler | 120 | 64 | 2026-06-19T21:29:44.840000Z | 2026-07-22T05:47:21.994000Z | published-range: 120 |
| ChatGPT-User | user-triggered fetcher | 102 | 53 | 2026-06-27T02:53:34.329000Z | 2026-07-15T20:24:52.932000Z | published-range: 102 |
| Googlebot-Image | image crawler | 70 | 34 | 2026-06-19T23:58:06.614000Z | 2026-07-22T20:47:48.255000Z | published-range: 70 |
| PerplexityBot | search crawler | 27 | 6 | 2026-07-11T20:02:43.655000Z | 2026-07-20T13:56:02.976000Z | published-range: 27 |
| Googlebot-Video | video crawler | 1 | 1 | 2026-07-04T15:18:43.810000Z | 2026-07-04T15:18:43.810000Z | published-range: 1 |
| archive.org_bot | web archive crawler | 1 | 1 | 2026-07-03T11:20:03.979000Z | 2026-07-03T11:20:03.979000Z | fcrdns: 1 |
| meta-webindexer | search crawler | 1 | 1 | 2026-07-10T20:25:18.571000Z | 2026-07-10T20:25:18.571000Z | rir-owner: 1 |

Unconfirmed identity claims

These requests used a bot-like user agent but could not be tied to that operator with the current verification evidence. A current-range miss is not proof of spoofing because published ranges can change after a historical request.

| Claimed identity | Unconfirmed requests | Paths | Reason counts |
|---|---:|---:|---|
| Amazonbot-Video | 28 | 25 | unverified-user-agent: 28 |
| ChatGPT-User | 23 | 16 | unverified-current-range-miss: 23 |
| Discordbot | 8 | 6 | unverified-user-agent: 8 |
| Googlebot | 1,414 | 282 | unverified-current-range-miss: 1060, unverified-private-source: 354 |
| facebookexternalhit | 11 | 4 | unverified-user-agent: 11 |

Across all confirmed identities, the logs contain 6,319 fetches of 1,371 distinct query-free paths, including 5,272 successful 2xx responses.

Most-fetched confirmed paths

| Path | Requests |
|---|---:|
| /robots.txt | 521 |
| /sitemap.xml | 461 |
| /gui | 245 |
| /api/chat | 229 |
| /api/chat/history | 152 |
| /windows98-clippy-showcase.html | 59 |
| /site-metrics.json | 44 |
| /saayn-spreadsheet-gpt2.html | 42 |
| / | 37 |
| /site-index.json | 32 |
| /push/api/activity | 29 |
| /api/chat/broadcast | 29 |
| /neoriris-clock-tower.html | 28 |
| /programs/neoriris-clock-display.html | 28 |
| /posts/index.json | 28 |
| /programs/stardate.html | 27 |
| /knowledge-index.json | 27 |
| /dashboard | 26 |
| /programs/logbook.html | 25 |
| /sitemap.html | 25 |
| /sol37/index.html | 24 |
| /chat | 23 |
| /sentience/index.html | 23 |
| /posts/write_to_random_emission.html | 22 |
| /assets/hue-visualizer.js | 22 |

AI dashboard and camera evidence

| Identity | Path | Methods | Requests | Statuses | First seen (UTC) | Last seen (UTC) |
|---|---|---|---:|---|---|---|
| Amazonbot | /api/dashboard/status | GET: 1 | 1 | 200: 1 | 2026-07-09T16:30:39.225000Z | 2026-07-09T16:30:39.225000Z |
| Amazonbot | /dashboard | GET: 1 | 1 | 200: 1 | 2026-07-07T09:11:43.690000Z | 2026-07-07T09:11:43.690000Z |
| Amazonbot | /dashboard/ | GET: 1 | 1 | 308: 1 | 2026-07-07T09:11:43.088000Z | 2026-07-07T09:11:43.088000Z |
| ChatGPT-User | /dashboard | GET: 2 | 2 | 200: 2 | 2026-07-02T21:31:19.727000Z | 2026-07-02T21:38:21.030000Z |
| ChatGPT-User | /dashboard/ | GET: 1 | 1 | 308: 1 | 2026-07-02T21:38:20.819000Z | 2026-07-02T21:38:20.819000Z |
| GPTBot | /dashboard/dashboard.css | GET: 1 | 1 | 200: 1 | 2026-07-02T21:39:21.027000Z | 2026-07-02T21:39:21.027000Z |
| GPTBot | /dashboard/dashboard.js | GET: 1 | 1 | 200: 1 | 2026-07-02T21:39:25.344000Z | 2026-07-02T21:39:25.344000Z |
| Googlebot | /api/dashboard/camera-caption | GET: 1 | 1 | 200: 1 | 2026-07-07T12:17:24.278000Z | 2026-07-07T12:17:24.278000Z |
| Googlebot | /api/dashboard/services/console | GET: 3 | 3 | 200: 3 | 2026-07-06T11:04:16.083000Z | 2026-07-11T00:38:09.830000Z |
| Googlebot | /api/dashboard/status | GET: 3 | 3 | 200: 3 | 2026-07-06T11:04:15.387000Z | 2026-07-11T00:38:09.361000Z |
| Googlebot | /dashboard | GET: 15 | 15 | 304: 15 | 2026-06-20T01:39:07.866000Z | 2026-07-21T21:17:29.847000Z |
| Googlebot-Image | /api/dashboard/camera.mjpg | GET: 3 | 3 | 200: 3 | 2026-06-23T01:47:32.204000Z | 2026-06-28T09:57:28.188000Z |
| OAI-SearchBot | /api/dashboard-access/status | GET: 1 | 1 | 200: 1 | 2026-07-02T22:09:50.906000Z | 2026-07-02T22:09:50.906000Z |
| OAI-SearchBot | /api/dashboard/camera.mjpg | GET: 12 | 12 | 200: 12 | 2026-07-02T22:12:22.213000Z | 2026-07-22T20:54:47.968000Z |
| OAI-SearchBot | /api/dashboard/comments | GET: 4 | 4 | 200: 4 | 2026-07-16T23:52:25.358000Z | 2026-07-17T01:23:10.083000Z |
| OAI-SearchBot | /api/dashboard/services/console | GET: 1 | 1 | 200: 1 | 2026-07-17T00:22:13.329000Z | 2026-07-17T00:22:13.329000Z |
| OAI-SearchBot | /api/dashboard/status | GET: 9 | 9 | 200: 9 | 2026-07-02T22:11:51.333000Z | 2026-07-22T20:56:12.516000Z |
| OAI-SearchBot | /dashboard | GET: 3 | 3 | 200: 3 | 2026-07-02T21:39:14.590000Z | 2026-07-16T23:19:41.865000Z |
| OAI-SearchBot | /dashboard/dashboard.css | GET: 1 | 1 | 200: 1 | 2026-07-02T21:39:21.982000Z | 2026-07-02T21:39:21.982000Z |
| OAI-SearchBot | /dashboard/dashboard.js | GET: 1 | 1 | 200: 1 | 2026-07-02T21:39:18.407000Z | 2026-07-02T21:39:18.407000Z |
| PerplexityBot | /dashboard | GET: 5 | 5 | 200: 5 | 2026-07-11T20:02:44.141000Z | 2026-07-16T12:02:44.694000Z |
| meta-externalagent | /api/dashboard/status | GET: 1 | 1 | 200: 1 | 2026-07-02T03:56:42.361000Z | 2026-07-02T03:56:42.361000Z |
| meta-externalagent | /share/gui/dashboard-index-html-5d060f57.html | GET: 2 | 2 | 200: 2 | 2026-06-26T21:31:54.102000Z | 2026-07-03T01:31:50.805000Z |

Evidence files

/home/david/random/data/sol37-crawl-inventory-2026-07-22.json contains every crawler family, source IP, verification result, status count, and path aggregate.
/home/david/random/data/sol37-crawl-inventory-2026-07-22-paths.csv contains the complete query-free path-level inventory.
Published CIDR lists were downloaded at generation time; their source timestamps are preserved in the JSON.
confirmed-rir-owner means the user agent matched and the source was in an operator-owned RIR allocation, not a purpose-specific crawler list.
confirmed-fcrdns means reverse DNS used the operator domain and forward DNS resolved back to the source IP.

Verification references

OpenAI: https://openai.com/searchbot.json, https://openai.com/gptbot.json, and https://openai.com/chatgpt-user.json
Google: https://developers.google.com/static/crawling/ipranges/common-crawlers.json
Bing: https://www.bing.com/toolbox/bingbot.json
Perplexity: https://www.perplexity.ai/perplexitybot.json
Amazon: https://developer.amazon.com/amazonbot/ip-addresses/
Anthropic: https://support.anthropic.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler (states that Anthropic does not publish crawler ranges)

Sol37 Crawl Inventory 2026 07 22

Executive Summary