Pay Per Crawl

This week we talk about crawling, scraping, and DDoS attacks.We also discuss Cloudflare, the AI gold rush, and automated robots.Recommended Book: Annie Bot by Sierra GreerTranscriptAlongside the many, and at times quite significant political happenings, the many, and at times quite significant military conflicts, and the many, at times quite significant technological breakthroughs—medical and otherwise—flooding the news these days, there’s also a whole lot happening in the world of AI, in part because this facet of the tech sector is booming, and in part because while still unproven in many spaces, and still outright flubbing in others, this category of technology is already having a massive impact on pretty much everything, in some cases for the better, in some for the worse, and in some for better and worse, depending on your perspective.Dis- and misinformation, for instance, is a bajillion times easier to create, distribute, and amplify, and the fake images and videos and audio being shared, alongside all the text that seems to be from legit people, but which may in fact be the product of AI run by malicious actors somewhere, is increasingly convincing and difficult to distinguish from real-deal versions of the same.There’s also a lot more of it, and the ability to very rapidly create pretty convincing stuff, and to very rapidly flood all available communication channels with that stuff, is fundamental to AI’s impact in many spaces, not just the world of propaganda and misinformation. At times quantity has a quality all of its own, and that very much seems to be the case for AI-generated content as a whole.Other AI- and AI-adjacent tools are being used by corporations to improve efficiency, in some cases helping automated systems like warehouse robots assist humans in sorting and packaging and otherwise getting stuff ready to be shipped, as is the case with Amazon, which is almost to the point that they’ll have more robots in their various facilities than human beings. Amazon robots are currently assisting with about 75% of all the company’s global deliveries, and a lot of the menial, repetitive tasks human workers would have previously done are now being accomplished by robotics systems they’ve introduced to their shipping chain.Of course, not everyone is thrilled about this turn of events: while it’s arguably wonderful that robots are being subbed-in for human workers who would previously have had to engage in the sorts of repetitive, physical tasks that can lead to chronic physical issues, in many cases this seems to be a positive side-benefit of a larger effort to phase-out workers whenever possible, saving the company money over time by employing fewer people.If you can employ 100 people using robots instead of 1000 people sans-robots, depending on the cost of operation for those robots, that might save you money because each person, augmented by the efforts of the robots, will be able to do a lot more work and thus provide more value for the company. Sometimes this means those remaining employees will be paid more, because they’ll be doing more highly skilled labor, working with those bots, but not always.This is a component of this shift that for a long while CEOs were dancing around, not wanting to spook their existing workforce or lose their employees before their new robot foundation was in place, but it’s increasingly something they’re saying out loud, on investor calls and in the press, because making these sorts of moves are considered to be good for a company’s outlook: they’re being brave and looking toward a future where fewer human employees will be necessary, which implies their stock might be currently undervalued, because the potential savings are substantial, at least in theory.And it is a lot of theory at this point: there’s good reason to believe that theory is true, at least to some degree, but we’re at the very beginning phases of this seeming transition, and many companies that jumped too quickly and fired too many people found themselves having to hire them back, in some cases at great expense, because their production faltered under the weight of inferior automated, often AI-driven alternatives.Many of these tools simply aren’t as reliable as human employees yet. And while they will almost certainly continue to become more powerful and capable—a recent estimate suggested that the current wave of large-language-model-based AI systems, for instance, are doubling in power every 7 months or so, which is wild—speculations about what that will mean, and whether that trend can continue, vary substantially, depending on who you talk to.Something we can say with relative certainty right now, though, is that most of these models, the LLM ones, at least, not the robot-driving ones, were built using content that was gathered and used in a manner that currently exists in a legal gray area: it was scraped and amalgamated by these systems so that ...