2021:Evaluation/Findings/Process/ja

Wikimania 2021 - An Evaluation of a Virtual Conference

要約

Findings

Demographics
Effectiveness
Process
Annex

Process of Organizing the Conference

Wikimania 2021 staff support co-working in the virtual space

This conference was the first time that Wikimania was held virtually. To support the design of the conference, the Wikimedia Foundation and members of the Steering Committee participated in a third party-led Event design process. The Event design generated three possible scenarios for the conference, as well as outlined goals and desired changed behaviors for the participants and other stakeholders.

Wikimania is usually organized by a well-established team that is geographically bound through a hosting affiliate and is selected in a competitive application process. Given that this conference would be virtual, the Steering Committee selected a group of volunteers from across the Movement and located across different geographies. This team of volunteers, alongside Foundation staff, would be responsible for organizing the conference. The COT was first convened in March 2021.

This section of the evaluation seeks to unpack the process of organizing and implementing the Wikimania Conference. It focuses on better understanding what worked well and what needs improvement. In addition, the Wikimedia Foundation Strategy Principles are used as a lens for interpretation, organization, and reflection. Interview findings are not disaggregated by role (i.e., staff or volunteer) to maintain confidentiality. To ensure anonymity, Foundation staff and COT members will be referred to as the organizing team. Findings are also framed as lessons learned to support decision-making and facilitate change.

Organizing a Virtual Conference

The team that organized Wikimania overcame many challenges to organize this conference. These challenges included an abbreviated timeframe to develop a new type of conference.

Resilience of Individuals: Almost all interviewees noted that they were proud that the virtual conference had taken place. They each highlighted their own contribution as examples of accomplishments, whether it was their role in Program Design, Scholarship Program Development, or Trust and Safety. Organizers were proud of organizing a conference that aimed to increase participation, enable inclusion, and showcase the diversity of the Movement. They highlighted the grueling long hours working independently and as team members.

Small teams working on specific tasks led to efficiency and personal satisfaction: Team members enjoyed most working in small teams focused on key tasks and appreciated one-on-one interactions between volunteers and Wikimedia staff to discuss key issues. Many of the team members acknowledged that they increased their own skills, knowledge, and capacities. Throughout the interviews, team members reflected on technical learnings and articulated in detail the key improvements necessary to make the process of organizing virtual conferences smoother and more collaborative. All interviewees learned about event planning in a virtual space.

For many, the experience of organizing the conference was not what they had originally signed up for or expected. That said, most were appreciative to have had the opportunity to be part of the team that organized the first virtual Wikimania. Members were energized by each other’s commitment to the conference. The process, however, led many of the members to feel burnt out. While burn out is not unusual for Wikimania organizers, the experience or organizing and implementing the conference can be more fruitful if the following lessons learned are applied to future planning processes.

Lessons Learned:

1. Allocate sufficient time to come together as a team. The team came together in March, only five months before the start of the conference. The abbreviated timeline was not sufficient to adequately develop the team and for members to bond. The team did not know one another and had not previously worked together. They were unaware of each other’s strengths and weaknesses. Interviewees shared that it was only two months prior to the conference that the team finally started to come together to produce. Future virtual conference team organizers may benefit from regional team building and in person planning meetings if health protocols allow.
2. Create a joint vision for the conference. The team reminisced that due to the compressed timeframe they had to make a lot of decisions even during the first meeting. While the Foundation and members of the Steering Committee had gone through a process to develop the Event Canvas and create three prototypes, COT members were not part of this exercise. When the COT met with the Foundation, they were presented with three prototypes for them to choose. COT members felt that deciding on key decisions based on the prototypes felt like they were simply “rubbing-stamping” decisions that had already been made for them. They did not feel that they had the autonomy to change the prototypes. The lack of a joint vision and agreed-upon objectives by all team members meant that there was ambiguity over the task at hand, decision-making authority, and a general lack of ownership. COT members expressed that they would sometimes feel “lost.” A process to co-create the vision and objectives of the conference will help increase ownership while also building team structure.
3. Co-create, discuss, and align on decision-making responsibilities. Ambiguity over who was responsible for what and who had the final say on key decisions led to frustrations and inefficiency. A RACI chart was developed at the beginning of the process and volunteers highlighted their areas of interest and strength. However, roles, responsibilities, authority, and accountability were not clear for all team members. One interviewee mentioned that conference outputs, such as the Speaker Guide, would appear without any previous notice to the team. Ambiguity over responsibilities meant that individuals acted, such as hiring a production team, without consulting with others because they believed that it was under their scope of work. Others felt excluded from these decision-making processes. Working in smaller teams focused on one aspect of the conference, such as the Program Design or Scholarship, helped create ownership, facilitate decisions, and accountability.
4. Build an infrastructure to support effective communication and decision-making. The infrastructure to organize an event of Wikimania’s size and scope with a remote team was not fully developed. The team relied on Google Sheets, Telegram chats, email, and meetings to communicate and document decisions. According to interviewees, this system was inefficient given the number of decisions that the team needed to make. It meant that individuals not present at one meeting would lead to a sense of loss and this impeded their ability to fully contribute in follow up meetings. Interviewees expressed frustration at tasks and actions not followed up on, emails not answered, and a lack of a system to document key decisions that would have helped promote accountability and transparency. An integrated project management or task tracker system may help future organizing teams. It may help set realistic turnaround timelines and communicate expectations on working hours.
5. Organize and institutionalize a working group of Foundation internal resources to better support the implementation of Wikimania. Several departments from the Foundation work to support Wikimania, including Communications, Security, Trust and Safety, and Legal. A cross-departmental working group made of Foundation staff that meets regularly can better troubleshoot, communicate, support volunteers, and create processes and products that serve not only Wikimania but also the many other events organized by the Movement.
6. Continue to innovate and experiment with decision-making structures. Interviewees mentioned that many of the decisions were conducted during meetings taking place during the work schedule. This produced a burden on some volunteers as they had to schedule their availability around their work schedule. Staff were also not available during the weekend, which is when volunteers had the most time. Some organizing team members suggested that leading up the conference perhaps Foundation staff could have flexibility and work one or half days during the weekend alongside volunteers. The organizing team also experimented with having drop-in meetings for people to communicate and co-working sessions to jointly finalize specific tasks. These meetings worked well, members felt better connected and they increased productivity.
7. Include a Community Facilitator as part of the Core Organizing Team. A community facilitator was brought into the organizing team halfway through the process. This individual had a unique set of skills and knowledge of the workings of movements that helped resolve conflict, brought cohesion to the group, and implemented effective communication to build trust. Once the Community Facilitator joined the team, members of the organizing team felt represented and heard. Other organizing team members felt relieved that there was a facilitator to bring together the team based on everyone’s strengths.
8. Clarify role of the Steering Committee. The Steering Committee has played an oversight role in past Wikimania conferences. The members of the Steering Committee have deep knowledge of Wikimania. The Steering Committee participated in the Event design. There was also one person that acted as a Liaison between the Steering Committee and the organizing team. Some members of the Steering Committee also supported the back-end implementation process of the conference. The involvement of the Steering Committee was appreciated by some of the members of the organizing team, while others felt it was disruptive. As more virtual conferences are organized, consider what role the Steering Committee should play particularly during the final stages of planning and particularly during the implementation phase. Clarifying the role and involvement of the Steering Committee will help make perfectly clear who has decision-making authority, implementation responsibility, and increase accountability and ownership of the organizing team.

In addition, the organizing team also reflected on how to improve aspects or outputs of the conference:

1. Technology - Members of the organizing team reflected that they wish they had had more time to reflect and test the conference platform. They would have liked to have more information to better support programming, such as the Unconference, presentations, community village, and meetups. Individuals wish they had known that Remo would not support mobile users earlier in the process.
2. Trust and Safety - According to team members, the Trust and Safety protocols were created too late considering the size of the conference. Members of the organizing team felt relieved that there were not a lot of serious Trust and Safety issues that took place. Many members felt under prepared. Recommendations by the organizing team include ensuring that everyone understands Trust and Safety limitations and workflows. They recommended having a multilingual volunteer Taskforce that is formed and trained early. The reliance of one person to remove a member not adhering to the policy was risky and not ideal. If something was to happen to this one individual, there would be no one to authorize the removal of a person from the conference. A member of the organizing team suggested that processes to ensure Trust and Safety in virtual conferences need to be documented and carefully throughout.
3. Program Development - The size of the program was increased from two tracks to five tracks. This was a decision that the many members of the organizing team felt passionate about, and it was the right strategic decision to ensure diversity of program, representation of the Movement, and inclusion of themes. Individuals shared that Wikimania does not have a high rejection rate on submitted proposals and that by virtue of being virtual many more voices could be included. The implications of adding additional tracks put pressure on internal resources, management of speakers, production capabilities, and coordination.

Conclusion: Future Virtual Conferences

Participants had an opportunity to reflect on Wikimania and the future of virtual events. Some individuals highlighted that Wikimania 2021 had shown that “there are possibilities in doing several days long virtual and maybe also hybrid events.” “I was skeptical before the event but enjoyed it immensely and see those online gatherings as a way to make our movement more inclusive and climate friendly, so I am looking forward to seeing how (virtual events) develop over the years!”

Many of the strategic decisions, such as diversity of program content and translation services, contributed to participants feeling part of the Movement. Moreover, the conference was able to connect with newcomers and provide them a platform to learn about new tools and generate ideas. While individuals did not meet as many new participants in the conference as organizers had hoped for, this is a particularly difficult task in virtual settings and perhaps the target was too high for this type of conference. If Wikimania 2022, as a virtual conference, aspires to be a source of new connections and two-way meaningful participation, the networking and Unconference space will have to be more intentionally and purposefully designed than this year.

Recommendations for next year: Organizers for Wikimania 2022 will have to convene early and often to form a cohesive team. This team will have the insights and lessons learned acquired during the process of organizing and implementing Wikimania 2021. Organizers will have to not only convene a group of individuals but create and use the necessary infrastructure (roles and responsibilities, meeting schedule, working-groups, appropriate agenda) to make efficient decisions. Appropriate project management tools to support multicultural remote decision-making and collaboration in a multi-layered and involved project may help. As the group is convened it can create operating principles that are based on the Movement Strategy Principles and better document the process of creating a virtual conference. As the conference approaches the last six week before implementation, lines of communication need to be clear, timely, and responsive to maintain cohesiveness.

In some ways, the 2022 organizers of Wikimania will have to go back to the drawing board to reflect on key decisions to see if these are still relevant and where there can be improvements. Most importantly, team members will need to be aligned in what the conference wants to achieve to ease decision-making, on key questions such as:

1. Given what we have learned in 2021, what will Wikimania 2022 seek to achieve? How can the use of the unconference space or community village be structured more strategically?
2. Should a virtual Wikimania 2022 span across five days, one weekend, or several weekends? How can the program better align with the different time zones? Does the Hackathon have to be part of a virtual Wikimania?
3. What additional features or processes need to be in place to ensure that Wikimania 2022 is even more accessible and inclusive? How can translation services be further mainstreamed? What technological platform can be used to meet the goals of Wikimania 2022?
4. Newcomers tended to appreciate the conference more than returnees. What features are needed to keep newcomers engaged? What program features can be added to cater to the needs of returnees?

This evaluation was prepared by Maria Vanessa Corlazzoli on behalf of the Wikimedia Foundation.