AI can “read the mind”! From a brain scan to a photograph, a medical project in Israel recreates what the user sees

A team of researchers at the Weizmann Institute of Science in Israel has developed an artificial intelligence system that can reconstruct the images a person is seeing by analysing their brain activity recorded through fMRI scans.

The system can also work in the opposite direction: from an image, it tries to predict what the brain activity of a person seeing it would look like.

The technology rests on two main components. One predicts the structure of the image, including the distribution of colours and shapes, while the other tries to identify the content, such as particular objects, foods or scenes.

These data are then used by a generative model to produce an image that approximates what the person saw.

To train the system, the researchers used existing data from eight people, each of whom had viewed about 9,000 images during high-resolution fMRI scans.

Because of the lack of large quantities of brain data, the team also built a second system that predicts brain activity from an image. The two models were used together to improve the results.

According to the researchers, about 70% of the material used for training came from images that had not previously been shown during real brain scans.

The system is not infallible, however. In some cases it reconstructed similar but not exact objects, such as a cake interpreted as several sandwiches, or a dog in a bathtub reconstructed as a goat of similar colours.

The researchers aim in future to expand the technology to analyse videos, sounds, mental images and perhaps the content of dreams as well.

One possible use would be helping fully paralysed people who cannot communicate, using their brain activity to reconstruct images or thoughts.

But the technology has also raised serious concerns about mental privacy.

Other researchers warn that, while today the method requires complex scans and the person’s cooperation, improvements in techniques using EEG and simpler devices could make analysing brain activity easier in future.

According to them, this could open debates about the use of brain data without individuals’ consent, as well as about the possible use of mental reconstructions in fields such as justice.