The transmission of video data is a major part of traffic on today’s Internet. Since the Internet is a highly dynamic environment, quality adaptation is essential in matching user device resources with the streamed video quality. This can be achieved by applying mechanisms that follow the Scalable Video Coding (SVC) standard, which enables scalability of the video quality in multiple dimensions. In SVC-based streaming, adaptation decisions have long been driven by Quality of Service (QoS) metrics, such as throughput. However, these metrics do not well match the way human users perceive video quality. Therefore, in this paper, the classical SVC-based video streaming approach is expanded to consider Quality of Experience (QoE) for adaptation decisions. The video quality is assessed using existing objective techniques with a high correlation to the human perception. The approach is evaluated in context of a P2P-based Video-on-Demand (VoD) system and shows that by making peers favor always layers with a high estimated QoE but not necessarily high bandwidth requirements, the performance of the entire system can be enhanced in terms of playback delay and SVC video quality by up to 20%. At the same time, content providers are able to reduce up to 60 of their server costs, compared to the classical QoS-based approach.