A really interesting question.

https://www.researchgate.net/profile/Hin-Yan-Liu/publication/377577308_Why_is_AI_regulation_so_difficult/links/65ae508a9ce29c458b91dcc1/Why-is-AI-regulation-so-difficult.pdf

Why is AI regulation so difficult?

… Let me qualify this: in order to regulate AI in anything approaching the conventional manner, we must aim to either regulate the underlying technology, or aim to regulate how that technology is applied to the world. To regulate the underlying technology, we must be able to understand and describe what it is; that is, we need to be able to define, categorise, and communicate in precise language what it constitutes. Attempts to define AI have been notoriously elusive, and are usually tautological (e.g., “creating machines that perform functions that require intelligence when performed by people” (Kurzweil, 1992)) or anchored by reference to other ill-defined concepts (e.g., “making machines intelligent, [where] intelligence is that quality that enables an entity to function appropriately and with foresight in its environment” (Nilsson, 2010, p. xiii)). An aspect of why defining AI is so difficult is that there a bewildering array of definitions for “intelligence” (Legg & Hutter, 2007), suggesting that there is no real consensus on what this might be or comprise. To make matters more complicated, recourse to the concept of intelligence is itself only one of many possible metaphors we can deploy to understand AI. As each metaphor or analogy drawn for AI foregrounds certain characteristics and capabilities over others, each holds significant ramifications for AI regulation, a point that I will raise in detail later.

To illustrate the difficulty of defining AI in law, consider Article 3(1) of the draft EU AI Act which states that “artificial intelligence system” means: ... software that is developed with [specific] techniques and approaches [listed in Annex 1] and can, for a given set of human-defined objectives, generate outputs such as content, predictions, recommendations, or decisions influencing the environments they interact with.

This approach does away with the metaphor of ‘intelligence’ altogether, thereby sidestepping many definitional pitfalls.