We can certainly run the experiment, just like we can let a kid touch a hot pan on the stove.
Like the kid, the experiment is not to add knowledge to the world. Every adult knows touching a hot pan results in a burn. Just like everyone who understands how current LLMs work knows that it will fail at being a lawyer.
Instead the point of such an experiment is to train the experimenter. The kid learns not to touch pans on the stove.
In this case it’s not fair to metaphorically burn desperate legal defendants so that the leaders and investors in an LLM lawyer company learn their lesson. It’s the same reason we don’t let companies “experiment” with using random substances to treat cancer.
Like the kid, the experiment is not to add knowledge to the world. Every adult knows touching a hot pan results in a burn. Just like everyone who understands how current LLMs work knows that it will fail at being a lawyer.
Instead the point of such an experiment is to train the experimenter. The kid learns not to touch pans on the stove.
In this case it’s not fair to metaphorically burn desperate legal defendants so that the leaders and investors in an LLM lawyer company learn their lesson. It’s the same reason we don’t let companies “experiment” with using random substances to treat cancer.