generated: '2026-08-12'
method: probed
source: https://www.thenewslens.com/robots.txt
description: >-
  Machine-readable AI-usage preferences published by TNL Mediagene properties in
  robots.txt. The flagship English/Chinese brand The News Lens serves the
  Cloudflare Content Signals Policy — an explicit, machine-readable reservation of
  rights under Article 4 of EU Directive 2019/790 — while the corporate host
  tnlmediagene.com serves a plain per-agent disallow list for AI crawlers. Both
  documents are captured verbatim alongside this index. This is the only
  machine-readable consent surface the group publishes; there is no
  /.well-known/security.txt, no AIPREF header, and no Web Bot Auth on any host.
signals:
  - host: https://www.thenewslens.com
    brand: The News Lens
    path: /robots.txt
    status: 200
    file: tnlmedia-thenewslens-robots.txt
    standard: Cloudflare Content Signals Policy
    content_signal:
      user_agent: '*'
      search: yes
      ai-train: no
      use: reference
    interpretation: >-
      Search indexing is permitted; training or fine-tuning AI models on the content
      is refused; AI systems that do consume the content may reference it but not
      take it in full. Declared as an express reservation of rights under Article 4
      of EU Directive 2019/790.
    additional_disallows:
      - Amazonbot
      - Applebot-Extended
      - Bytespider
    note: >-
      The block is emitted inside a "# BEGIN Cloudflare Managed content" section,
      i.e. served through Cloudflare's managed robots.txt feature. It is still an
      affirmative, owner-enabled publication of the signal, but it is not
      hand-authored by the publisher.
  - host: https://www.tnlmediagene.com
    brand: TNL Mediagene (corporate)
    path: /robots.txt
    status: 200
    file: tnlmedia-tnlmediagene-robots.txt
    standard: none (plain robots.txt user-agent disallow)
    content_signal: null
    interpretation: >-
      No Content-Signal directives. Instead a blanket Disallow for named AI agents.
    ai_agents_disallowed:
      - GPTBot
      - ClaudeBot
      - ChatGPT-User
      - meta-externalagent
      - Amazonbot
      - Bytespider
      - CCBot
      - Timpibot
      - omgilibot
    other_disallows: ['/metrics-', '/internal/', '/api/', '/hook/']
    note: >-
      The corporate host disallows /api/ for all user agents; no path under /api/
      was probed, in keeping with that directive.
hosts_without_any_signal:
  - {host: https://www.ad2iction.com, status: 200, content_signal: null}
  - {host: https://www.mediagene.co.jp, status: 200, content_signal: null}
  - {host: https://www.businessinsider.jp, status: 200, content_signal: null}
  - {host: https://www.gizmodo.jp, status: 200, content_signal: null}