I'm pretty sure that stealing all the books and research papers from the US nuclear weapons programme would be illegal. I'm also confident that giving those books to, say, Iran would break a whole raft of other laws too.

What if instead you used the stolen books to train an AI model, transforming the books into an LLM which doesn't contain those original sources verbatim? Surely still illegal to give to Iran because nuclear secrets can be gleaned from the LLM. However what if transformation is so extreme—the LLMs parameters so small—that all the nuclear secrets drop out leaving (probably) only basic physics knowledge which isn't illegal to distribute?

It's a moot point. I'm sure the US authorities would lock you up for stealing the books, and you're not going to be allowed to distribute the LLM.

That's not what happened in this week's US court victory for Anthropic. The judgement found that transforming books into an LLM was fair use, but that stealing books was not—so only the stealing part can be taken to trial. Anthropic's LLM was not immediately shut down.

If instead the books had been stolen from the US nuclear weapons programme rather than from Libgen and PiLiMi it surely would have been shut down immediately. Obviously that's because stealing nuclear secrets is a more serious crime than stealing copyrighted books, but they're both illegal. Anthropic hasn't yet been convicted of stealing books so it is reasonable to wait to take action, but that's a delay not an abandonment.

A trial on Anthropic's stealing of books could—and in my opinion should—order that Anthropic's LLMs get shut down. Not just Anthropic; journalists report it is likely that all major LLMs are trained on stolen books. It is possible that in time all major LLMs get shut down by the courts.

The industry should be responding now by training LLMs legally, not waiting for the law to catch up. I'd like to start using AI but I don't want to be handling stolen goods.