What you see depends not only on what there is to see, but also on what you expect to see. Human vision relies heavily on priors about how things should appear in the world, allowing for efficient visual learning and generalization. This differs greatly from most machine learning approaches to visual recognition, which typically require large datasets of training images that have been meticulously hand-labeled with what they depict. While unsupervised methods allow machines to learn on their own without any human intervention, the results are often difficult to interpret. Semantic Component Analysis (SCA) is a novel framework for unsupervised learning that incorporates rich, instance-level constraints to encourage semantic interpretability. Through simple and intuitive spatial consistency priors, SCA is able to learn components that naturally correspond to semantic image segmentations. Even without manual engineering, fine-tuning, or overfitting to a particular dataset, we achieve competitive performance on standard semantic segmentation tasks. This suggests the importance of prior assumptions in visual learning and encourages a decreasing reliance on human annotations.