OLLD18 Workshop – Optimising the Learning Curve

Morning of August 23rd 2018

Nathalie Stembert (SD), Katariina Malmberg (ENoLL), Abdolrasoul Habibipour (LTU) and Cesco Reale (AS)

The U4IoT workshop – “Optimising the Learning Curve: implementing end-user engagement tools in IoT large-scale pilots”, was held on the 23rd of August 2018 during the Open Living Lab days 2018 in Geneva, supported by the partners ENoLL, Stembert Design, LTU and AS. All Large Scale Pilot (LSP) end-user engagement leads contributed to the workshop preparation by providing input for the workshop cases. The workshop was attended by representatives from SynchroniCity, MONICA and ActivAge.

Since 2010, the Open Living Lab Days (OLLD) has become the annual gathering of the global Living Lab community. A space for public officials, companies, entrepreneurs, academics and innovators to connect and work together: To create new products and services, to set the basis for debate and exploration of theories, and to discuss and process policy recommendations within the practical elements of open and user-driven innovation. Through interactive panel discussions, hands-on workshops with leading experts and site visits to our local partners, OLLD offers an exclusive networking and knowledge sharing experience – From the OLLD website.

In short, the OLLD provide a great platform to get in touch with end-user engagement experts and exchange knowledge on the topic. The workshop exercises presented to these experts were based on the cases from the LSPs. The majority of the LSP projects are now halfway, nevertheless implementation of the tools, methods and recommendations can be challenging. Amongst the U4IoT support services are the Privacy game, Living Lab Support, Co-Creative Workshop Support, Participatory Sustainability Models, Survey & Crowdsourcing tools, IoT Adoption Barriers and End-user Engagement Toolkit. Each of the U4IoT tools, methods and recommendations can be applied in (a) different stage(s) of the Learning Curve. Mapping the end-user engagement support services on the LSP project timelines was the main exercise of the workshop, in order to discuss how the Learning Curve of the LSPs could be optimised.

With the main objective to share experiences and gather knowledge on how to implement end-user engagement support in complex and large-scale technical projects.


Figure 2 – Workshop materials and SynchroniCity table.

After a short introduction of the U4IoT project and support services, the five cases based on the LSP projects in combination with the UN Sustainable Development Goals were presented. Based on the cases, smaller groups shared experiences and brainstormed about implementation of the U4IoT tools, methods and recommendations in the stages the projects are currently in. The workshop resulted in concrete recommendations applicable to the real-life situation of each of the cases, where intervening actions are needed to enhance end-user engagement. The presented recommendations also considered the optimal situation, where the project is planned from the beginning to involve users.

In this workshop the participants got to introduced to the U4IoT end-user engagement support services. They got a glimpse of large-scale technical projects and the complexity of carrying out end-user engagement activities within these projects. Based on the cases and their own experiences, they exchanged knowledge and formulated recommendations on how to establish an ideal learning curve in large-scale technical pilot projects.

The workshop resulted in some very practical insights concerning the implementation of the U4IoT support services in the second half of the European IoT-LSP projects. Besides these ready-to-adopt insights, also insights concerning an optimal implementation process were discussed. Large-scale projects are driven by a broad variety of partners, stakeholders and (end-)users, who are not only situated in a different fields of work, context and stage, but also aren’t necessarily aware of the importance of end-user engagement. Involving end-users is not a linear process: first you need a critical mass, and to share experiences in order to go for open innovation. Embedding end-user engagement activities in the project proposal is a start to successful implementation, yet in order to do this, training of the partners is required to convince them to involve end-users and to align the end-user engagement activities and outcomes. It is a long-lasting process to build the trust to involve all the partners, thus, long-term planning is needed. When the momentum is there, a sequence of end-user engagement activities can be repeated to gather insights. In this process researchers should however be aware of GDPR protocols and the ownership and value of the data gathered should be discussed in a transparent way amongst all parties involved.

During the workshop recap, per table and case, the following aspects were discussed:


  • General feedback for the exercise:
    • Useful to receive feedback from external people with different perspective.
    • Would have needed more flexible timeline with several pieces of tool cards
    • Went further from the tools/services and started to think about the different stakeholders needed to take part in different activities
    • Good to start thinking about the process and how to redefine it
  • Starting to think about tools that haven’t yet been used in the project but that would be beneficial, e.g. privacy game
  • Transversal activities for the project: Living Lab support and Participatory sustainability model
  • Challenge: cities working in different phases, not equal level of awareness
  • Some activities were done already: IoT adoption barriers and co-creation workshops, but project is planning to explore more of these when new pilots are starting to run in February 2019
  • Starting to use survey & crowdsourcing tools for user analysis, and will use again later on in each city in each pilot.
  • Launching the solutions, will use the privacy game and to review the sustainability model

Figure 2 – Timeline created by the SynchroniCity table.


  • This is an idealised version of the process compared to what was actually done.
  • Suggestion to start with the survey and crowdsourcing tool and use it at the beginning, in the middle and at the end – to easily get feedback when you go along the process
  • Including the privacy guidelines at the very beginning of the project.
  • Co-creation workshop at the beginning, this was done within the project.
  • In the middle suggestion to use the Living Lab support, within that considering the sustainability – at the end of the project a lot of data generated throughout the project, considering what is going to happen with that.
  • Adoption barriers nearer the end, as you need to know what you’re asking people to adopt
  • Extra things to be developed in the project: app for attendees of music festival, as a way of getting feedback focused on info about the end-users.
  • Project will have hackathons – project is expecting as a result of these hackathons development of attendee apps.
  • Usage of all tools? Didn’t use privacy guidelines/game, as with this project the idea is not to get personal data, however do have ethics involved.

Figure 3 – Timeline created by the MONICA table.


  • Discussion needed especially around data, evaluating if it can be gathered, if it can be stored, for how long etc.
  • To go into dialogue with the stakeholders: end-users, general public, public sector etc.
  • Should start thinking about selling the data from the very beginning – sustainability and business models. However, considerations needed: users need to be aware, thinking what kind of users are involved, in which format of the data (raw data vs. analysed information based on the data).
  • Careful consideration of the user group.
  • Researchers need to know better the GDPR and what it means – therefore focus on the privacy game with this exercise. Technicians and end-users playing together the game will create trust.
  • Another consideration for the usage of data, after project offering it for free, but not selling – for the benefit of the society.
  • Value of personal data: no clear methodology for the valorisation of data, it’s unclear how companies valorise the data, and what value data has to whom. Suggestion to look into value assessment of the data and to consult the stakeholders, especially in the project that is relying on collecting a lot of data from end-users.
  • Sharing – people are open for it, of course considering the type of data. Another consideration with the GDPR, it sometimes is in contradiction with the national laws, e.g. concerning health data.

Figure 4 – Timeline created by the ActivAge table.


  • Discussion in the group about the pre-defined end-users: food-processing industry & farmers, but also a lot of other stakeholders need to be included: academia, public sector & especially the customers from the very beginning to define the business model.
  • Involving end-users is not a linear process: first you need a critical mass of end-users and farmers, and to share experiences to go for open innovation.
  • Agriculture is a traditional sector, need to start with training and convincing people to involve the end-users. After these first phases you ongoingly have always the same training loops with more advanced modules and with national/international mentors, so that you can improve the different development stages. And then you can integrate all the pre-defined tools and methods step-by-step.
  • It is a long-lasting process to build the trust to involve all the partners, and to avoid competition, thus, long-term planning is needed before the implementation.
  • Case example from Austria: have an ongoing process, in the farming school system, to have an open innovation process from the beginning, so that everyone is involved. Have built a system with different stages of Living Labs in the process: Living Lab itself, then a policy learning lab, where stakeholders are integrated in a short version of LL method, informing them about the activities, IoT platform etc., and finally the last stage.
  • Biggest challenge is to define the end-users and how to integrate them. When they’re on board, it’s easy to take them through all the different tools and methods, but the first stage is the most difficult one.

Figure 5 – Timeline created by the IoF2020 table.

Based on the insights gathered in the workshop, U4IoT is drafting support plans for the LSP projects, including recommendations on how to implement the tools and support services in the second half of the pilots. The uptake of the support recommendations will be monitored and evaluated. The conclusions of this study will be used to inform future project proposals, on how to structurally implement end-user engagement from the beginning on in large-scale technical projects. The U4IoT tools and support services continue to be developed, and the publicly available online service package will be completed with the upcoming privacy game and e-courses on privacy, participatory sustainability models and meetups.

This work was undertaken in the context of the U4IoT project – User Engagement for Large Scale Pilots in the Internet of Things (www.u4iot.eu). The research leading to these results has received funding from the European Commission’s H2020-ICT-2016-2017 Programme under grant agreement n° 732078.