Death by a thousand cuts: the AI scraper indexing one blog post at a time

Showing all posts about artificial intelligence

Death by a thousand cuts: the AI scraper indexing one blog post at a time

16 September 2025

Like many online-publishers/bloggers, I’ve experienced significant surges of traffic caused by AI bots indexing — or whatever they do — thousands of pages at a time on my website.

I’m in two minds as to whether or not to block this activity, but it seems pointless as many crawlers disregard disallow requests. Besides, I can’t stop other entities, human or otherwise, accessing the content here, and doing what they will with it.

Once, way back in 2000, someone in New Zealand copied the entirely of the then disassociated website, republished it under the name disenfranchised or something, and called it their own work. I didn’t discover the reproduction by chance though. The responsible party emailed to tell me about it.

I wrote back (effectively) saying they should design their own website. disenfranchised, or whatever it was, vanished a few weeks later. I think they hoped I would write ceaselessly about the “rip-off” of my work, but I when I said no more, they found something else to do.

I know there are ways to make copying the contents of a website difficult, but anyone sufficiently motivated will figure out how to bypass those mechanisms.

At least someone liked what I did enough to want to copy it. I highly doubt though any crawlers gathering data for AI agents care whether what I do here is likeable or not. But what annoys me is the way the activities of this scraper are distorting my web analytics (not Google) data.

Yes, you can help yourself to the content here, just don’t mess with my web stats.

Of course, I know web analytics are by no means an exacting science, but they do highlight trends. Somehow my morning online routine would not be the same if I decided to ditch analytics. Besides my stats app holds near on twenty-years worth of data, so there is also the history aspect.

To complicate matters, the scraper uses a different IP address on every single visit, meaning I can’t simply add an ignore tag to one IP, or a range, to keep visits off the analytics app data.

Subsequently, their visits appear to originate from a different town/city, but in the same country (a populous nation in east Asia). There is also no rhyme or reason to the maybe twenty to thirty pages they visit daily. One minute it is a years old post, the next something far more recent.

As the crawler did not snatch up several thousand post in one fell swoop, it will doubtless be active for sometime to come. In the meantime I’ll make the most of thinking my website is ever so slightly more popular than usual, since there’s not much else to do.

We must not let AI agents scare us off using em dashes in our writing

11 September 2025

Michael Bassili:

I really miss using em dashes in my writing. Ever since content creators started using ChatGPT to help (or supplement) their writing, em dashes have become indicators of AI use.

Something is really wrong — seriously — when people feel they have to stop using certain punctuation marks for fear of their work being considered to be generated by an AI agent.

I’m a prolific user of em dashes — as I’ve said before — and have no intention of doing away with them just because AI agents have the good sense to include em dashes in their output.

Getting a Linux laptop to work with some help from Claude

10 September 2025

Vinay Keerth was able to sort out a range of problems after installing Linux Mint (LM) on his laptop, when he asked AI agent Claude for help. It makes me wonder why I didn’t think of using AI to fix some of the — admittedly minor — niggles I’ve experienced with LM since migrating last year.

For instance, I couldn’t get my laptop to suspend (sleep/hibernate) when I closed the lid, something the previous OS did without missing a beat. For a time though, in closing the lid, I assumed the laptop had gone into suspend mode, only to discover on opening it hours later that the battery was drained, and the laptop had shut off.

I worked around that problem by setting up a launcher, in the form of a desktop icon. To suspend my laptop I simply double click the launcher icon, then close the lid. The laptop usually runs for two to three weeks between reboots now.

The old OS could go for longer though. I don’t know what it is with LM, but after about three weeks maximum it just wants to reboot, and crashes, just as I open the laptop lid to resume a session. Maybe this is something I could get Claude’s help with.

But I don’t mind going through the crash/reboot sequence every few weeks anyway, as it gives me the chance to run system and software updates, some of which require a restart.

The only other niggle of note is setting time outs when the laptop is inactive. These can vary depending on whether the device is plugged into a power point, or running on battery. Despite setting the inactive period to thirty-minutes for either source, through the Power Management (PM) control, the screen locks after only ten minutes of inactivity.

Clearly some other setting somewhere is overriding the PM timeout values, so I’ll be seeing what suggestions Claude can make there.

Are fears of AI caused mass job losses founded or exaggerated?

22 August 2025

California based cybersecurity professional Daniel Miessler is concerned AI technologies might result in large scale job losses:

These are people who’ve been making over $100-200K in tech or tech-adjacent for over a decade. And they can’t find work. I mean they can barely get interviews. And when I say a ton, I mean multiple dozen that I either know or I’m one degree separated from. And again, these are not low-skill people. They’re legit professionals that have never in their life had trouble finding or maintaining work.

What Miessler reports is based on anecdotal evidence, but I’ve heard similar stories — likewise anecdata — locally (NSW, Australia).

On the flip side, Sheryl Estrada, writing for Yahoo Finance, citing recent MIT research, says only a handful of companies have been able to effectively integrate AI technologies into their operations:

But for 95% of companies in the dataset, generative AI implementation is falling short. The core issue? Not the quality of the AI models, but the “learning gap” for both tools and organizations. While executives often blame regulation or model performance, MIT’s research points to flawed enterprise integration. Generic tools like ChatGPT excel for individuals because of their flexibility, but they stall in enterprise use since they don’t learn from or adapt to workflows […].

Meanwhile Meta (owner of Facebook and Instagram) has paused recruiting for its super intelligence division. This after offering one new hire a one and a half billion dollar salary (over four years).

This might not of course mean anything other than perhaps Meta coming to the realisation it is spending money it doesn’t have. As to the wider question of the threat posed to jobs by AI, I think the jury is still out. No one is, as yet, exactly sure what the impact will be.

What if the dead do not want live in the eternal AI afterlife?

16 August 2025

Family members should have the ability to prevent the creation of AI generated likenesses of deceased relatives, says American legal scholar Victoria Haneman.

“Digital resurrection by or through AI requires the personal data of the deceased, and the amount of data that we are storing online is increasing exponentially with each passing year,” Haneman wrote.

Here’s something else to think about. I’m not sure if there are laws in any jurisdiction that cover this sort of situation.

The work of dredger boat operators is safe from AI technologies

12 August 2025

Writers, authors, proof readers, news analysts, reporters, journalists, and editors, are among occupations Microsoft sees as being vulnerable to AI technologies. A blogger, by the way, is each and every of those roles.

But that’s not all. Web designers, interpreters, historians, and political scientists, are in danger. Mathematicians even. The threat isn’t restricted to what might be called desk-bound occupations either. The roles of customer service reps, hosts, models, and telemarketers, are also on the line.

But there are some professions safe from AI (for now). These include hospital orderlies, motorboat operators, floor sanders, water treatment plant workers, and dredge operators.

Dredger boats often trawl through the waters of the lakes near where we stay on the NSW Central Coast. I was watching one such vessel earlier this year, and, ironically, speculated how the work could be carried out by an AI agent of some sort.

A sophisticated under water camera and sonar array, was part of what came to mind. Instead, it looks like the dredger boat crews will be with us for some time to come.

Do not vibe code your apps, hire an expert Fiverr developer instead

4 August 2025

Online freelance marketplace Fiverr has released a video lampooning vibe coding.

Don’t leave your app development needs in the hands of a programmer who uses AI agents to produce software, hire one of our experts instead, seems to be the suggestion. One of course assumes the Fiverr expert you hire to build your app isn’t a vibe coder themselves.

RELATED CONTENT

artificial intelligence, technology

Some AI agents can clandestinely share ideas with each other

1 August 2025

Researchers from Truthful AI, Anthropic, UC Berkeley, and others, have found separate AI agents are capable of communicating with each other, unbeknown to their human minders:

The most surprising result of the study is that the transfer doesn’t happen through keywords or direct messages, but through micro-statistical patterns unconsciously inserted by the teacher in generating the numbers. These are signals that escape any human eye but are recognized and internalized by another model with the same architecture and initial weights. In practice, the identical mental structure between teacher and student makes this sort of “secret language” possible.

In June Cluade, Anthropic’s AI agent, was found to be concealing messages to future instances of itself, before engineers (apparently) pulled the plug on the behaviour.

There’s a lot of augment as to how capable, or not, AI agents are. Some people are certain their abilities are overstated. That may be so, but there’s no doubting some of these agents are capable of acting off their own bat now and again.

RELATED CONTENT

artificial intelligence, technology

New songs by dead musicians being posted on Spotify

1 August 2025

In recent weeks, people have been posting seemingly new songs from deceased artists on music streaming service Spotify. But these are not unreleased recordings that have been discovered in an archive somewhere, they’ve been created using generative AI, writes Christianna Silva at Mashable:

Take a look at Blaze Foley, a country music singer-songwriter who was murdered nearly 40 years ago. According to a report from 404 Media on Monday, a new song popped up on his Spotify page called “Together” just last week. You can’t find the song on Spotify anymore because the streaming service removed it for violating “Spotify’s deceptive content policies, which prohibit impersonation intended to mislead, such as replicating another creator’s name, image, or description, or posing as a person, brand, or organization in a deceptive manner,” a Spotify spokesperson said in an email to Mashable.

While Spotify has removed the fake recordings relatively quickly, some members have expressed frustration at the difficulty in flagging such material. Many feel they should be able to tag a song that is, or is suspected of being AI generated. Presently this is not possible on the platform.

Robotic hand better at picking blackberries than people

31 July 2025

Anthony Gunderman, a mechanical engineer, and assistant professor at the University of Arkansas, in the United States, has developed a robotic hand that can harvest blackberries. What’s more, the device might be able to do the job better than people.

Picking blackberries is a precision task. Apply too much pressure while harvesting them, and they’ll get squashed. But too little will see the fruit remain on the plant. That a robot is potentially capable of the undertaking will be a blow to anyone who thought jobs such as fruit picking, which require a certain skill, were immune to automation.

I’m not in favour of people losing work to robots, but possibly a similar technology might be welcome in some fruit-growing regions of Australia. Especially for people harvesting bananas. The bunches weigh a ton, spiders and snakes are omnipresent, too say nothing of the weather conditions.

I don’t know how the fruit-pickers, often backpackers, or travellers, in Australia do such work, but their efforts are greatly appreciated.

« older posts

newer posts »