These scientists help rescue ‘broken’ digital art | Science News for Students

These scientists help rescue ‘broken’ digital art

Hardware and software updates can damage computer-based art — and restoring it is tricky
Apr 24, 2019 — 6:45 am EST

“Unfolding Object” is a piece of online artwork by John F. Simon Jr.  It had stopped running on most modern computers. But a new digital fix lets viewers interact with this artistic creation again.

John F. Simon, Jr., Unfolding Object (2002), Solomon R. Guggenheim Museum.

WASHINGTON, D.C. — To see most art at the Solomon R. Guggenheim Museum, you’d have to travel to New York City. But you only need to go online to see one of the museum’s works: “Unfolding Object.” It’s a 2002 web-based work by artist John F. Simon, Jr. As viewers interact with it, they affect what it looks like. For several years, the art stopped working on updated computers. But thanks to innovative restoration work, computer scientists have put this piece and others like it back in action.

At first, “Unfolding Object” looks like a simple square on your screen. Click on any of its edges and that side unfolds. Click on another edge, and it unfolds. You can repeat the process over and over. Or, go back to the first square or another shape, and click a different edge. As you go, lines appear on each shape. These hatch marks tally up previous viewers’ clicks. The directions represent ones, tens, hundreds and thousands of interactions with viewers. The color of the object and background also change throughout the day. Playing with the work can keep you clicking — again and again and again.

a screen capture of "Unfolding Object"
The colors and shapes of the restored “Unfolding Object” depend on where someone clicks and what time of day it is. Here’s how it looked when our writer played around with it late one night.
John F. Simon, Jr., Unfolding Object (2002), Solomon R. Guggenheim Museum

But there came a time when the program wouldn’t work on many newer computer systems.

Such digital — or online — art can’t break in the same way that an ancient vase might shatter. But the art is still quite fragile, says Deena Engel. She’s a computer scientist at New York University in New York City. Recently she and her students worked with Joanna Phillips, a former conservator at the Guggenheim, and others there. A conservator is someone who repairs and preserves artworks. Together they restored “Unfolding Object” art. Engel described how they did it at the annual meeting of the American Association for the Advancement of Science on February 17.

“It’s very difficult to run software from maybe 15 years ago on a current Mac or a current PC,” Engel explains. Operating systems change. Computer programming platforms can change. Hardware gets updated. Old equipment breaks and parts may no longer be available. Other changes affect how colors show up on a screen. All of that can affect what people see in a work of computer art.

“If you can even get it to run at all, it's most likely going to run differently,” Engel says. “Even things like code for black and white [may] get flipped.”

But fixing broken computer art takes more than a new program that does pretty much the same thing. After all, a copy of the Mona Lisa isn’t the same as the original. And the goal here is to restore the computer art — not to replace it. “Just like you want to be careful with the brush strokes if you’re preserving a painting, we're very careful with the code,” Engel explains. “We want to preserve the algorithms, and we want to preserve the artist’s hand in the source codes in these kinds of works.”

That’s where Engel and the team had to be innovative. “We are building new models to apply to this rich area of contemporary art,” she explains.

‘We have work to do’

Engel first learned about problems in computer-art restoration some years ago. At that time, conservators from the Museum of Modern Art in New York City approached the head of her university department, who then reached out to Engel. “I must have a reputation for my love of art,” she says.

Engel visited a back room at the museum. There she found stacks of old computer media, including early floppy disks. “I was like, ‘Oh, my gosh, roll up your sleeves. We have work to do!’” 

Engel and others got “Unfolding Object” working again. The Guggenheim museum unveiled this restored version online last November 19. Previously they had restored another web-based work in the Guggenheim’s collection. It’s “Brandon” by artist Shu Lea Cheang.

a photo of several people interviewing Shu Lea on video chat
Computer scientist Deena Engel (closest to the front on the right) and others interviewed artist Shu Lea Cheang as part of their efforts to restore his artwork, “Brandon.”
© Solomon R. Guggenheim Foundation

One task for Engel’s team has been to recreate the computer environment in which the artists had worked. The team set up software that the artists had used. Then the group’s members could see how the art was supposed to look and work. And they could get source code if it was not otherwise available. Finally, the team had to figure out what’s different between the earlier computer environment and what’s in use now. Only then could they begin designing tweaks and add-ons.

For example, Simon originally created “Unfolding Object” in Java. That’s a programming platform. But Java has changed a lot since the early 2000s. In some cases, Simon had used applets — mini-programs within the platform — that no longer worked. So the team had to add new programming language.

The restorers also had to make sure the webpage would run correctly. For that, they worked with Hypertext Markup Language, or HTML. That’s the standardized system for the structure of most webpages.

“We work very hard to allow the original code to run,” Engel says. The approach is almost the opposite approach to how most programmers work, she notes. After all, computer engineers usually aim to update and improve programs. But it’s a matter of ethics for art conservators. They want to keep the art as it was created. So instead of using all modern code, the group tweaks the old programs.

“We add [other features] that remind it to behave the way it did 15 or 20 years ago," Engel explains. When it is necessary to rewrite code, Engel’s team makes detailed comments in the updated code. The comments note which parts of the original code no longer work. And they state the date and details of what they might of added to solve any problems.

Computer artist John F. Simon Jr. explains how “Unfolding Object” works.
Guggenheim Museum/YouTube

Looking ahead

Meanwhile, artists are creating new computer artworks. Engel’s group is working with the Guggenheim conservators in hopes of making future restorations less tricky. One step is talking with artists when new works come into the collection. Artists can explain what their vision is, and what types of future changes they would be willing to accept. The museum also can get source code at that time for the work. All that info can guide future computer-science teams and conservators if a computer system’s base set of colors or other properties change.

Where possible, there will also be video recordings of computer art with voiceover comments by the artist. Interviews like that can help future restorers see just how the art worked when it was created.

Marc Walton is a materials scientist at Northwestern University in Evanston, Ill. He specializes in applying science to art conservation. He and his team used computer programming to capture 3-D images of “acne” on a Georgia O’Keefe painting. (Chemicals called metal soaps cause the problem, which forms due to chemical reactions between paint ingredients.) Walton thinks Engel’s work is addressing new technical questions and ethical issues in computer art. Computer science and other high-tech fields (such as medical imaging and chemical analysis) will be the future of art restoration, he predicts.

He says, “I think you’re just seeing the tip of the iceberg with these kinds of things.”

Power Words

(more about Power Words)

3-D     Short for three-dimensional. This term is an adjective for something that has features that can be described in three dimensions — height, width and length. 

algorithm     A group of rules or procedures for solving a problem in a series of steps. Algorithms are used in mathematics and in computer programs for figuring out solutions.

American Association for the Advancement of Science (or AAAS)   Formed in 1848, it was the first permanent organization formed to promote the development of science and engineering at the national level and to represent the interests of all its disciplines. It is now the world’s largest such society. Despite its name, membership in it is open to anyone who believes “that science, technology, engineering, and mathematics can help solve many of the challenges the world faces today.” Its members live in 91 nations. notably Science.

chemical     A substance formed from two or more atoms that unite (bond) in a fixed proportion and structure. For example, water is a chemical made when two hydrogen atoms bond to one oxygen atom. Its chemical formula is H2O. Chemical also can be an adjective to describe properties of materials that are the result of various reactions between different compounds.

chemical reaction     A process that involves the rearrangement of the molecules or structure of a substance, as opposed to a change in physical form (as from a solid to a gas).

code     (in computing) To use special language to write or revise a program that makes a computer do something. (n.) Code also refers to each of the particular parts of that programming that instructs a computer's operations.

computer science     The scientific study of the principles and use of computers. Scientists who work in this field are known as computer scientists.

conservation     The act of preserving or protecting something. The focus of this work can range from art objects to endangered species and other aspects of the natural environment.

conservator     A person in charge of protecting and/or restoring valuable items.

digital     (in computer science and engineering)  An adjective indicating that something has been developed numerically on a computer or on some other electronic device, based on a binary system (where all numbers are displayed using a series of only zeros and ones).

engineer     A person who uses science to solve problems. As a verb, to engineer means to design a device, material or process that will solve some problem or unmet need.

environment     The sum of all of the things that exist around some organism or the process and the condition those things create. Environment may refer to the weather and ecosystem in which some animal lives, or, perhaps, the temperature and humidity (or even the placement of things in the vicinity of an item of interest).

ethics     (adj. ethical) A code of conduct for how people interact with others and their environment. To be ethical, people should treat others fairly, avoid cheating or dishonesty in any form and avoid taking or using more than their fair share of resources (which means, to avoid greed). Ethical behavior also would not put others at risk without alerting people to the dangers beforehand and having them choose to accept the potential risks. Experts who work in this field are known as ethicists.

HTML     Short for Hypertext Markup Language. It's the standardized language for

materials science     The study of how the atomic and molecular structure of a material is related to its overall properties. Materials scientists can design new materials or analyze existing ones. Their analyses of a material’s overall properties (such as density, strength and melting point) can help engineers and other researchers select materials that are best suited to a new application. 

metal     Something that conducts electricity well, tends to be shiny (reflective) and malleable (meaning it can be reshaped with heat and not too much force or pressure). 

model     A simulation of a real-world event (usually using a computer) that has been developed to predict one or more likely outcomes. Or an individual that is meant to display how something would work in or look on others.

online     (n.) On the internet. (adj.) A term for what can be found or accessed on the internet.

operating system     The software that controls a computer to supports its basic activities, such as scheduling tasks; assigning storage space (memory) to programs or data; performing applications, and controlling linked up devices (peripherals such as a printer, keyboard or monitor).

software     The mathematical instructions that direct a computer’s hardware, including its processor, to perform certain operations.

source code     The declarations, instructions, functions and other statements that direct the actions and functions of computer software. They are written in a computer-programming language.

voiceover     (in TV and moviemaking) The recording of a human voice over film or animation.

web     (in computing) An abbreviation of World Wide Web, it is a slang term for the internet.


Meeting: D. Engel. “Conserving and restoring computer-based art in an ethical framework.” American Association for the Advancement of Science Annual Meeting. Washington, D.C. February 17, 2019.