Distinguishing mutations that determine an organisms phenotype from (near-) neutral hitchhikers is a fundamental challenge in genome research, and is relevant for numerous medical and biotechnological applications. For human influenza viruses, recognizing changes in the antigenic phenotype and a strains capability to evade pre-existing host immunity is important for the production of efficient vaccines. We have developed a method for inferring antigenic trees for the major viral surface protein hemagglutinin. In the antigenic tree, antigenic weights are assigned to all tree branches, which allows us to resolve the antigenic impact of the associated amino acid changes. Our technique predicted antigenic distances with comparable accuracy to antigenic cartography. Additionally, it identified both known and novel sites, and amino acid changes with antigenic impact in the evolution of influenza A (H3N2) viruses from 1968 to 2003. The technique can also be applied for inference of phenotype trees and genotype-phenotype relationships from other types of pairwise phenotype distances.