When Robert Erdmann visited the back rooms of the Art Institute of Chicago in the mid 2000s and saw people studying and conserving paintings, he had one thought- ‘I want to weasel my way into this field.’ He was intrigued by their efforts to ‘solve the mysteries’ of artworks.

At the time, Erdmann was studying fluid flow through porous media for his PhD at the University of Arizona, Tucson. He earned the degree in 2006 and joined the university as an assistant professor with appointments in materials science and engineering and in applied mathematics.

During that initial visit at the art institute, he watched as a chemistry postdoc painstakingly assembled x-ray radiographs of a work by 17th-century Spanish painter Diego Velázquez. X rays let one peer beneath the surface of a painting to see, for example, the use of lead white pigment. Erdmann realized the software he had developed to stitch together panoramic images from his hobby of kite photography could help. That started a collaboration, others followed, and a few years later he moved into the museum world.

Today, Erdmann is a senior scientist at the Rijksmuseum in Amsterdam and a professor at the University of Amsterdam, where he is affiliated with the Institute of Physics and the department of conservation and restoration of cultural heritage. As techniques have been developed and applied to study works of art, Erdmann notes, museums ‘have very quickly gone from a world in which we had a dearth of data to having a deluge.’ Most of his work involves creating neural networks and other software with the aim of understanding an artwork’s physical state and how it was made. His mission is to ‘help the world access, preserve, and understand its cultural heritage.’

PT- How did you get interested in cultural heritage?

ERDMANN- I started my tenure-track position at the University of Arizona in 2006. My research was in multiscale modeling of transport processes—the movement of mass, momentum, energy, and charge around the universe. I studied how the processing of materials changes their structure.

I thought it would be interesting to look at cultural heritage on the side, but I was advised that it was not the smartest thing to pursue if I was going to get tenure. I didn’t take the advice. In typical assistant professor fashion, I worked my butt off and didn’t have much of a social life.

One nice cultural heritage project from that time involved analyzing a set of photos of Henri Matisse in front of one of his paintings, Bathers by a River. It was an important work that Matisse had painted during his transition period during World War I. He repainted the painting four times. Conservators at the Art Institute of Chicago were trying to reconstruct the four stages of the painting from x rays and from photos at oblique angles. I made a model that predicted where the cameras were and what the lens distortions were. It worked—and it was published in the front page of the culture section of the New York Times in July 2010.

PT- How did you end up in Amsterdam?

ERDMANN- In 2012 I was invited to spend a sabbatical year in Amsterdam as a fellow at the Netherlands Institute for Advanced Study. I worked with the museums of the Netherlands to bring imaging and computational technologies to help with art conservation and technical art history.

One of the things I worked on was the Bosch Research and Conservation Project, to commemorate 500 years since the death of Hieronymus Bosch, a titan in the history of Dutch artists. A group of six individuals—two art historians, a technical art historian, a conservator of paintings, a photographer, and myself—visited all the museums in the world that had Bosch paintings. We imaged them with multiple types of imaging—extremely high resolution visible and infrared cameras, IR reflectography, x-ray radiography, and so on. IR penetrates to see an artist’s preparatory sketches or painted-over layers. I developed a sequence of algorithms to process these big images. Later I invented the ‘curtain viewer’ to zoom around and compare the different images using a mouse.

If you look at a nice little Bosch painting, St John the Baptist in the Wilderness, for example, you see John the Baptist sort of meditating. He gestures to a lamb, and a weird vegetable or fruit thing is growing next to him. By zooming in with my curtain viewer, you can see details that you couldn’t otherwise, like a mark on the lamb made by a tour guide who was waving a ballpoint pen.

And if you peel back the top layer and look in the infrared, you see that underneath the weird vegetable there is a man kneeling in prayer. The story is that this man commissioned the painting, but that when Bosch finished it, he didn’t pay. Bosch just painted over him and made him into a vegetable.

PT- How did the year in Amsterdam turn into a permanent move?

ERDMANN- More or less on the last day of my fellowship, in 2013, the head of physical sciences at the Dutch Science Foundation visited me and said, ‘What would it take for us to bring you permanently to the Netherlands?’ I had just gotten tenure in Arizona a month earlier. I quit my job there and moved to the Netherlands.

PT- What are your responsibilities at the museum?

ERDMANN- I will often hang around art historians and conservators and have them tell me what kinds of problems they are facing. I want to ensure that the work I do directly impacts the practice in the field.

As an example of my responsibilities, take Rembrandt’s The Night Watch. Completed in 1642, the painting is his largest work, currently 454 cm by 378 cm. It was trimmed on all four sides in 1715 to fit between two columns at the Amsterdam Town Hall. I programmed the artificial intelligence reconstruction of the painting’s appearance before it was cut down.

For a separate project, I designed the specifications for the imaging hardware and new ways to photograph The Night Watch at 5-micron resolution using a 400-megapixel Hasselblad camera. It’s tricky- You have a depth of field of one-eighth of a millimeter; the painting has ripples, and it acts like a sail in the air currents from the air-conditioning.

We have gathered 55 terabytes of data on The Night Watch. Those data can be organized and interrogated with the curtain viewer. I can zoom in to see the swirl of Rembrandt’s brush and how it had three different paints mixed on it. I can see a place where a past restorer has covered up a blemish. I can see a brittle fracture pattern that shows a pigment was made with small particles—blue cobalt glass ground into linseed oil.

I can see lead soap particles; they result from a reaction between lead in paint and potassium in linseed oil. Sometimes the lead soaps grow and then fall out, leaving little craters. Johannes Vermeer was renowned for his reds, but the View of Delft looks pink; at the microscopic level, it has thousands of lead soaps poking out, and that affects the color.

I can show the angles or spacings of the threads on the canvas—another computational technique I developed.

PT- Why is it interesting to look at canvas threads?

ERDMANN- If you measure the spacing between the threads, you can compare and see if another painting has the same spacing. If it does, you know they were cut from the same roll of canvas. It may be surprising that threads show up in an x-ray radiograph. But the painter smears a ground layer over the linen canvas, and that ground layer is a bit radio-opaque. It squeezes between the threads, becoming thicker between the threads and thinner on top of them. When you make a radiograph, you can see what is effectively the imprint of the canvas.

For example, there is a pair of paintings that were in the news a few years ago. They show Marten Soolmans and Oopjen Coppit, a newlywed couple in 1634 who commissioned Rembrandt to paint them. They are the only full-size, full-length portraits he painted. They were in the possession of a member of the Rothschild family, Éric de Rothschild, who sold them in 2016 for €160 million. In the end, the Rijksmuseum bought Marten and the French government bought Oopjen. They agreed that the paintings would alternate between France and the Netherlands and that they would always hang together.

I did a canvas analysis, and you can see that the two canvases were cut from the same roll. In addition, by looking at the cusping, or gathering, you can see how the canvases were stretched over a wooden frame. In this case, the cusping is along three sides of each painting, and not along the fourth. The thread intersection angles form a smoking gun that says these two paintings were together at some moment. I can say the canvases were stretched as one piece; but whether they were painted together I can’t say.

The fact that you can receive paintings that are almost 400 years old and interrogate the physical evidence and reconstruct things is exciting and fascinating.

PT- What are some other examples of your work?

ERDMANN- I trained a neural network to take images of old paper that has writing on both sides and show what the paper would look like with no ink. That’s useful if you want to read the watermark so you can know where and when the paper was made.

Another of my projects uses joint language and image models to make a neural network search engine for cultural heritage images. I’ve assembled a database of 2.3 million images from the open image collections of lots of museums. I can search, for example, for ‘a woman wearing a blue dress and holding a musical instrument’ or ‘a woman caring for her child.’ Without any metadata associated with the images, the neural network knows how to read the art. It flies through woodcuts, Rembrandt drawings, paintings, and so on.

PT- You were recently asked to work on Anne Frank’s diaries. Can you tell us about the project?

ERDMANN- Yes, that was a huge honor. Every 10 years, the Anne Frank House does conservation documentation of the diaries. One thing I was asked to do was to collect extremely high resolution photographs of the areas where she added holiday stickers to the pages. Some cracks are present, and the 3-micron-resolution photos are intended to monitor growth in the cracks. That’s on the conservation side.

There are also questions posed by scholars about the diaries. For example, there are pages that were redacted by Anne Frank’s father by crossing things out and taping over them with brown packaging tape. The hope is that with transmitted light photography and neural network calculations, we will be able to pull out the text. A colleague has used hyperspectral imaging to separate inks when something has been scribbled over. The method has an exquisite ability to separate tiny differences in reflectivity; maybe it could be used to check for and decipher writing on the backs of photos in the diaries.

PT- What are the ethics of exposing things that were redacted or hidden?

ERDMANN- I’ve been asked the same question about looking at what is behind paintings. If Anne Frank’s father redacted things to hide something embarrassing about her, it would have a different feeling than if he was altering the historical record to avoid offending certain people or to make himself look better. But we don’t know. Anne Frank’s diaries are a world heritage–scale historical document. I find knowing to be better than not knowing.

PT- Do others use the tools you have developed?

ERDMANN- People in the medical, aerospace, and forensics fields are interested. At the moment, I help people prepare their images for use with the curtain viewer. But my long-term vision is to make it into an open-source stack that anyone can use.

PT- Anything else you’d like to add?

ERDMANN- There is no shortage of deeply challenging and intellectually fascinating problems wherever you go. That’s been a prejudice- When people hear that I work in a museum, they sometimes think I don’t do real science. But if you want low-hanging fruit, plant new trees. That is what is happening here. I am a materials-science, artificial-intelligence-imaging nerd in a museum, and that means that there are interesting projects everywhere. I think I have 43 different ongoing projects at the museum right now.