A native of Austria, Gross emigrated as a teenager from war torn Europe in 1921, and immediately began his study of art at the Educational Alliance on the Lower East Side. There he came to know painters Moses and Raphael Soyer, Peter Blume, Adolph Gottlieb, and many other key 20th-century New York artists. Gross then expanded upon his study of sculpture with Robert Laurent at the Art Students League and Elie Nadelman at the Beaux-Arts Institute of Design.

After his first solo show in New York in 1932 at Gallery 144, Gross's works were soon acquired by major Manhattan and American museums, including The Metropolitan Museum of Art and the Whitney Museum of American Art. Around 1950 Gross's primary medium became bronze, which he used on both an intimate and large scale; his outdoor sculpture, The Family (1979) continues to be a beloved fixture at the intersection of Bleecker and West 11th Streets in the West Village of New York City. In the early 1960s Gross added lithography to his studio practice and created luminous and profound renderings of historic and modern Judaic subjects. Today, the largest body of Gross's sculpture in a public collection is at the Hirshhorn Museum in Washington D.C.