This paper argues for a need to develop Virtual Reality (VR) learning spaces based on pedagogy rather than implementing pedagogy in a general purpose VR application. In doing so, the paper summarizes the challenges in using an immersive virtual reality application to unite together primary school children across national, regional, and political boundaries in the same learning space in order to promote natural learning. Data was collected through observations during the activity sessions among children from a Finnish and a Bangladeshi primary school. Furthermore, we collected data using semi-structured interviews and focus groups with teachers, heads of schools, and children from these two schools. The data analysis revealed three types of challenges: design, technological, and pedagogical. These findings guide us in designing and developing pedagogy centered learning spaces using virtual reality.