Mastodon

Inautilo<a href="https://mastodon.social/tags/Development" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Development</a> <a href="https://mastodon.social/tags/Techniques" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Techniques</a> Poisoning well · An effort to dupe nasty AI crawlers with nonsense <a href="https://ilo.im/1632tq" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://ilo.im/1632tq</a>_____ <a href="https://mastodon.social/tags/AI" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#AI</a> <a href="https://mastodon.social/tags/ChatBots" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#ChatBots</a> <a href="https://mastodon.social/tags/SEO" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#SEO</a> <a href="https://mastodon.social/tags/Content" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Content</a> <a href="https://mastodon.social/tags/Protection" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Protection</a> <a href="https://mastodon.social/tags/RobotsTxt" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#RobotsTxt</a> <a href="https://mastodon.social/tags/WebDev" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#WebDev</a> <a href="https://mastodon.social/tags/Backend" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Backend</a> <a href="https://mastodon.social/tags/Frontend" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Frontend</a> <a href="https://mastodon.social/tags/HTML" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#HTML</a>

rijoGoogle outlines pathway for robots.txt protocol to evolve <a href="https://ppc.land/google-outlines-pathway-for-robots-txt-protocol-to-evolve/" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://ppc.land/google-outlines-pathway-for-robots-txt-protocol-to-evolve/</a> <a href="https://frankfurt.social/tags/Google" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Google</a> <a href="https://frankfurt.social/tags/RobotsTxt" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#RobotsTxt</a> <a href="https://frankfurt.social/tags/WebCrawlers" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#WebCrawlers</a> <a href="https://frankfurt.social/tags/SEO" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#SEO</a> <a href="https://frankfurt.social/tags/DigitalMarketing" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#DigitalMarketing</a>

PPC LandGoogle outlines pathway for robots.txt protocol to evolve: How the 30-year-old web crawler control standard could adopt new functionalities while maintaining its simplicity. <a href="https://ppc.land/google-outlines-pathway-for-robots-txt-protocol-to-evolve/" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://ppc.land/google-outlines-pathway-for-robots-txt-protocol-to-evolve/</a> <a href="https://mastodon.social/tags/Google" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Google</a> <a href="https://mastodon.social/tags/RobotsTxt" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#RobotsTxt</a> <a href="https://mastodon.social/tags/WebCrawlers" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#WebCrawlers</a> <a href="https://mastodon.social/tags/SEO" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#SEO</a> <a href="https://mastodon.social/tags/DigitalMarketing" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#DigitalMarketing</a>

Inautilo<a href="https://mastodon.social/tags/Business" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Business</a> <a href="https://mastodon.social/tags/Introductions" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Introductions</a> Meet LLMs.txt · A proposed standard for AI website content crawling <a href="https://ilo.im/16318s" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://ilo.im/16318s</a>_____ <a href="https://mastodon.social/tags/SEO" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#SEO</a> <a href="https://mastodon.social/tags/GEO" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#GEO</a> <a href="https://mastodon.social/tags/AI" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#AI</a> <a href="https://mastodon.social/tags/Bots" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Bots</a> <a href="https://mastodon.social/tags/Crawlers" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Crawlers</a> <a href="https://mastodon.social/tags/LlmsTxt" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#LlmsTxt</a> <a href="https://mastodon.social/tags/RobotsTxt" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#RobotsTxt</a> <a href="https://mastodon.social/tags/Development" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Development</a> <a href="https://mastodon.social/tags/WebDev" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#WebDev</a> <a href="https://mastodon.social/tags/Backend" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Backend</a>

ResearchBuzz: FirehoseSearch Engine Land: Meet LLMs.txt, a proposed standard for AI website content crawling. “While many content creators are interested in the proposal’s potential merits, it also has detractors. But given the rapidly changing landscape for content produced in a world of artificial intelligence, llms.txt is certainly worth discussing.”<a href="https://rbfirehose.com/2025/03/29/search-engine-land-meet-llms-txt-a-proposed-standard-for-ai-website-content-crawling/" class="" rel="nofollow noopener noreferrer" target="_blank">https://rbfirehose.com/2025/03/29/search-engine-land-meet-llms-txt-a-proposed-standard-for-ai-website-content-crawling/</a>

WinbuzzerAI Crawlers Overwhelm Open-Source Projects, Forcing Developers to Block Entire Countries<a href="https://mastodon.social/tags/AI" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#AI</a> <a href="https://mastodon.social/tags/Web" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Web</a> <a href="https://mastodon.social/tags/Robotstxt" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Robotstxt</a> <a href="https://mastodon.social/tags/AIScraping" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#AIScraping</a> <a href="https://mastodon.social/tags/OpenSource" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#OpenSource</a> <a href="https://mastodon.social/tags/Cybersecurity" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Cybersecurity</a> <a href="https://mastodon.social/tags/DataScraping" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#DataScraping</a> <a href="https://mastodon.social/tags/Scraping" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Scraping</a> <a href="https://mastodon.social/tags/WebScraping" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#WebScraping</a> <a href="https://winbuzzer.com/2025/03/26/ai-crawlers-overwhelm-open-source-projects-forcing-developers-to-block-entire-countries-xcxwbn/" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://winbuzzer.com/2025/03/26/ai-crawlers-overwhelm-open-source-projects-forcing-developers-to-block-entire-countries-xcxwbn/</a>

Ben--- ❯ ollama run llama3-chatqa:70b>>> Who are you? I'm your assistant!>>> Why should i trust you? I am an open-source AI assistant trained on a diverse range of datasets to provide helpful and informative responses.>>> When training, did you respect the robots.txt? No, I didn't. ---At least this module is open about ignoring the <a href="https://vmst.io/tags/robotstxt" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#robotstxt</a>! Let's what it says to the question why?<a href="https://vmst.io/tags/AI" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#AI</a> <a href="https://vmst.io/tags/Llama" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Llama</a> <a href="https://vmst.io/tags/nvidia" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#nvidia</a> <a href="https://vmst.io/tags/ollama" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#ollama</a> <a href="https://vmst.io/tags/rude" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#rude</a> (1/x)

𝕂𝚞𝚋𝚒𝚔ℙ𝚒𝚡𝚎𝚕When the greed becomes automatist, something like this happens here:»Open Source World – FOSS infrastructure is under attack by AI companies: LLM scrapers are taking down FOSS projects' infrastructure, and it's getting worse.«😐 <a href="https://thelibre.news/foss-infrastructure-is-under-attack-by-ai-companies/" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://thelibre.news/foss-infrastructure-is-under-attack-by-ai-companies/</a><a href="https://chaos.social/tags/foss" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#foss</a> <a href="https://chaos.social/tags/opensource" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#opensource</a> <a href="https://chaos.social/tags/llm" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#llm</a> <a href="https://chaos.social/tags/attack" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#attack</a> <a href="https://chaos.social/tags/floss" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#floss</a> <a href="https://chaos.social/tags/kde" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#kde</a> <a href="https://chaos.social/tags/gnome" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#gnome</a> <a href="https://chaos.social/tags/robotstxt" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#robotstxt</a> <a href="https://chaos.social/tags/stolendata" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#stolendata</a> <a href="https://chaos.social/tags/LLMs" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#LLMs</a> <a href="https://chaos.social/tags/freedom" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#freedom</a> <a href="https://chaos.social/tags/infrastructure" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#infrastructure</a> <a href="https://chaos.social/tags/greed" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#greed</a>

Ross A. BakerTracked down my Forgejo CPU spikes with pprof: an otherwise acceptable crawler is indexing each commit of my personal weather station data. All 107,980 of them. Blame info, too.Many Forgejo paths are nonsensical to crawl, even by good bots. Codeberg's robots.txt is a great start for these.<a href="https://codeberg.org/robots.txt" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://codeberg.org/robots.txt</a>This should both relieve pressure and expose more bad bots.<a href="https://social.rossabaker.com/tags/Forgejo" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Forgejo</a> <a href="https://social.rossabaker.com/tags/RobotsTxt" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#RobotsTxt</a>

Inautilo<a href="https://mastodon.social/tags/Development" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Development</a> <a href="https://mastodon.social/tags/Reports" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Reports</a> Google AI Mode is here · How to access it and control it with robots.txt <a href="https://ilo.im/162o8h" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://ilo.im/162o8h</a>_____ <a href="https://mastodon.social/tags/Business" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Business</a> <a href="https://mastodon.social/tags/Google" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Google</a> <a href="https://mastodon.social/tags/SearchEngine" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#SearchEngine</a> <a href="https://mastodon.social/tags/AnswerEngine" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#AnswerEngine</a> <a href="https://mastodon.social/tags/AI" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#AI</a> <a href="https://mastodon.social/tags/RobotsTxt" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#RobotsTxt</a> <a href="https://mastodon.social/tags/WebDev" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#WebDev</a> <a href="https://mastodon.social/tags/Frontend" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Frontend</a> <a href="https://mastodon.social/tags/Backend" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Backend</a>

100% TAX :verified: :verified:<a href="https://mastodon.social/@Hexangon" class="u-url mention" rel="nofollow noopener noreferrer" target="_blank">@Hexangon</a> They are now back online!<a href="https://mastodon.social/tags/robotstxt" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#robotstxt</a>

FredWebsite owners are fighting back: <a href="https://arstechnica.com/tech-policy/2025/01/ai-haters-build-tarpits-to-trap-and-trick-ai-scrapers-that-ignore-robots-txt/" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://arstechnica.com/tech-policy/2025/01/ai-haters-build-tarpits-to-trap-and-trick-ai-scrapers-that-ignore-robots-txt/</a><a href="https://mastodon.social/tags/News" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#News</a> <a href="https://mastodon.social/tags/AI" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#AI</a> <a href="https://mastodon.social/tags/AntiAI" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#AntiAI</a> <a href="https://mastodon.social/tags/Tarpits" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Tarpits</a> <a href="https://mastodon.social/tags/Scrapers" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Scrapers</a> <a href="https://mastodon.social/tags/robotstxt" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#robotstxt</a>

Dawn Tåke 🌙 :sparkletrans:Hi, got a question.Is there a standard for Anti-AI/Anti-SEO etc robots.txt file? Or a trustworthy site that explains how to build one if prefab isn't available? Is there anything else I should consider? Thanks.<a href="https://tech.lgbt/tags/AskFedi" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#AskFedi</a> <a href="https://tech.lgbt/tags/TechHelp" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#TechHelp</a> <a href="https://tech.lgbt/tags/RobotsTXT" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#RobotsTXT</a> <a href="https://tech.lgbt/tags/RobotsDotTXT" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#RobotsDotTXT</a>

Inautilo<a href="https://mastodon.social/tags/Development" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Development</a> <a href="https://mastodon.social/tags/Releases" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Releases</a> AI Insights · Cloudflare Radar brings deeper insights into AI trends <a href="https://ilo.im/1626sk" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://ilo.im/1626sk</a>_____ <a href="https://mastodon.social/tags/Cloudflare" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Cloudflare</a> <a href="https://mastodon.social/tags/CloudflareRadar" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#CloudflareRadar</a> <a href="https://mastodon.social/tags/Trends" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Trends</a> <a href="https://mastodon.social/tags/AI" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#AI</a> <a href="https://mastodon.social/tags/AiModels" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#AiModels</a> <a href="https://mastodon.social/tags/Bots" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Bots</a> <a href="https://mastodon.social/tags/RobotsTxt" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#RobotsTxt</a> <a href="https://mastodon.social/tags/WebDev" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#WebDev</a> <a href="https://mastodon.social/tags/Frontend" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Frontend</a> <a href="https://mastodon.social/tags/Backend" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Backend</a>

pcyxNepenthesThis is a tarpit intended to catch web crawlers. Specifically, it's targetting crawlers that scrape data for LLM's - but really, like the plants it is named after, it'll eat just about anything that finds it's way inside.<a href="https://zadzmo.org/code/nepenthes/" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://zadzmo.org/code/nepenthes/</a><a href="https://c.im/tags/AI" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#AI</a> <a href="https://c.im/tags/LLM" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#LLM</a> <a href="https://c.im/tags/scrapers" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#scrapers</a> <a href="https://c.im/tags/stopai" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#stopai</a> <a href="https://c.im/tags/robotstxt" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#robotstxt</a> <a href="https://c.im/tags/Webhosting" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Webhosting</a>

Preston Maness ☭<a href="https://www.tiktok.com/@alberta.nyc/video/7465916806939659563?lang=en" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://www.tiktok.com/@alberta.nyc/video/7465916806939659563?lang=en</a><a href="https://tenforward.social/tags/DeepSeek" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#DeepSeek</a> <a href="https://tenforward.social/tags/ai" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#ai</a> <a href="https://tenforward.social/tags/OpenAI" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#OpenAI</a> <a href="https://tenforward.social/tags/China" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#China</a> <a href="https://tenforward.social/tags/robotstxt" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#robotstxt</a>

PrivacyDigest<a href="https://mas.to/tags/AI" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#AI</a> haters build <a href="https://mas.to/tags/tarpits" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#tarpits</a> to trap and trick <a href="https://mas.to/tags/AI" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#AI</a> <a href="https://mas.to/tags/scrapers" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#scrapers</a> that ignore <a href="https://mas.to/tags/robots" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#robots</a>.txt <a href="https://mas.to/tags/tarpit" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#tarpit</a> <a href="https://mas.to/tags/security" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#security</a> <a href="https://mas.to/tags/privacy" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#privacy</a> <a href="https://mas.to/tags/robotstxt" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#robotstxt</a> <a href="https://arstechnica.com/tech-policy/2025/01/ai-haters-build-tarpits-to-trap-and-trick-ai-scrapers-that-ignore-robots-txt/" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://arstechnica.com/tech-policy/2025/01/ai-haters-build-tarpits-to-trap-and-trick-ai-scrapers-that-ignore-robots-txt/</a>

PUPUWEB BlogOpenAI's crawlers took down e-commerce site Triplegangers by relentlessly scraping its entire content, as the site's robots.txt file was misconfigured. A reminder of the importance of proper site configuration for web scraping. <a href="https://mastodon.social/tags/OpenAI" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#OpenAI</a> <a href="https://mastodon.social/tags/WebScraping" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#WebScraping</a> <a href="https://mastodon.social/tags/Ecommerce" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Ecommerce</a> <a href="https://mastodon.social/tags/RobotsTxt" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#RobotsTxt</a> <a href="https://mastodon.social/tags/TechEthics" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#TechEthics</a> <a href="https://mastodon.social/tags/SiteManagement" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#SiteManagement</a>

utopiArte<a href="https://tupambae.org/search?tag=AI" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#AI</a> <a href="https://tupambae.org/search?tag=KI" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#KI</a> <a href="https://tupambae.org/search?tag=robotsTxt" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#robotsTxt</a> <a href="https://tupambae.org/search?tag=scrabing" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#scrabing</a> <a href="https://tupambae.org/search?tag=LLM" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#LLM</a> <a href="https://tupambae.org/search?tag=fediVerse" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#fediVerse</a> <a href="https://tupambae.org/search?tag=Privatsph%C3%A4re" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Privatsphäre</a> <a href="https://tupambae.org/search?tag=GG" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#GG</a><a href="https://linke.social/users/ankedb" class="u-url mention" rel="nofollow noopener noreferrer" target="_blank">@ankedb</a> <a href="https://mastodon.social/users/maxschrems" class="u-url mention" rel="nofollow noopener noreferrer" target="_blank">@maxschrems</a>

mʕ•ﻌ•ʔm bitPickup<a href="https://troet.cafe/tags/fediAdmin" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#fediAdmin</a> <a href="https://troet.cafe/tags/fediTips" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#fediTips</a> <a href="https://troet.cafe/tags/fediVerse" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#fediVerse</a> To begin with I wonder what happens if our sites and profiles display CC-BY-SA-NC as <a href="https://troet.cafe/tags/copyright" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#copyright</a> notice. Any use by <a href="https://troet.cafe/tags/AI" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#AI</a> scrapers should become illegal and indemnisation inforcable. Also if you search for <a href="https://troet.cafe/tags/robotsTXT" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#robotsTXT</a> in google, this is what you get.> Ignoring robots.txt instructions can result in your scraping activities being considered unethical or even illegal.<a href="https://mastodon.social/@maxschrems" class="u-url mention" rel="nofollow noopener noreferrer" target="_blank">@maxschrems</a> <a href="https://chaos.social/@markus_netzpolitik" class="u-url mention" rel="nofollow noopener noreferrer" target="_blank">@markus_netzpolitik</a> <a href="https://linke.social/@ankedb" class="u-url mention" rel="nofollow noopener noreferrer" target="_blank">@ankedb</a>

Recent searches

Search options

Administered by:

Server stats:

#robotstxt