The Linux Foundation's director of IT infrastructure says they now spend more CPU cycles "rendering commits for scrapers than we spend on all other kinds of legitimate access." At any one time, across 5 geo-distributed nodes, there are 14 CPU cores doing nothing but rendering git commits as html.... [W]hen a source is guaranteed to be LLM-free, like the entire history of kernel commits, it's worth its weight in gold as a source of training data... At the time of writing, linux.git is about 1.48 million commits. Oh, and we have about 922 forks of it on git.kernel.org — but don't worry, it's actually extremely efficient on the backend, since it's mostly the same objects in every fork. Unless, of course, you're a scraper, in which...
Les hele artikkelen hos kilden.
Kommentarer (0)
Ingen kommentarer ennå. Bli den første til å kommentere!