2021 talk:Evaluation
Add topicLatest comment: 2 years ago by Valerio Bozzolan in topic Participation metric... 60 seconds?
Participation metric... 60 seconds?
[edit source]Hi :) thank you for this report! Question. I noticed this phrase:
- Participation is defined as a person logging into the Remo platform for more than 1 minute
I think this is a very strange metric that cannot rappresent participation very well. Just as it seems it's possible to know the logged-in time, can't we also filter for... uhm... let's say 20 minutes instead of 60 seconds? What do you think about? Thank you :) --Valerio Bozzolan (talk) 10:10, 4 December 2021 (UTC)
- @Valerio Bozzolan: I guess we could debate this for quite some time - but this seemed a convenient bare minimum to work with. I consider this equivalent to someone registering into the conference, collecting their badge, paying the admission fee and then after seeing the lunch buffet straight walking out of the venue. We would then also still count that person as a 'participant'. I think what we tried to communicate with this sentence, is that people who only watched via YouTube, are not covered under this definition (which is more problematic to me, but unfortunately the data sometimes forces you to make hard choices!). Effeietsanders (talk) 21:52, 10 December 2021 (UTC)
- Yeah! the scenario you described is exactly what it could be useful to understand: the bounce rate (TL;DR how many people look and go away immediately). I understand it's not an happy information to share but it's a transparent and constructive one. For example, from the charts of the Italian wiki conference 2020 online we estimated 23 thousand apparent participants but "only" 310 users online for more than 15-30 minutes. See the itWikiCon 2020 data table. Do you think we can we share similar information for Wikimania 2021, to help other organizers? --Valerio Bozzolan (talk) 15:31, 14 December 2021 (UTC)
- If you're asking in 'what was the distribution of participation time across participants on Remo', that would probably better answer your question than changing the definition of a 'participant'. I believe it was hard to answer this with specific numbers due to privacy concerns, but there is the section The drop-off numbers within the first 20 minutes of the conference are significant, even in the absence of reliable comparative virtual conference attendance standards which is probably as clear a signal as you can get. Effeietsanders (talk) 17:57, 14 December 2021 (UTC)
- About privacy
- Yes, I understand too that privacy is an important topic. Participants of Wikimania 2021 were tracked by very invasive proprietary tools and without any strong reason or clarification and that was important (I tried to describe it here: 2021:Present and future). Having said that, using already-collected data to provide anonymized aggregated stats has nothing to do with privacy and we can surely do it now. Let's do it! It would be reeeeally useful! By the way, it would be pretty ridiculous if we are not able to provide more meaningful participation metrics after accepting such trackers on our devices.
- About participation metrics
- I am just in good faith looking for more attractive participation metrics since, unfortunately, "how many were online for 60 seconds" it's not attractive for an human being examining an interesting 2-days conference. I'm inviting to proactively search additional metrics (for example contacting Remo) and release them (for example in Wikimedia Commons as a data table), having said that Remo has every right not to provide this information to us, but it would not be strategic for us not to ask Remo this question.
- Example: "Dear Remo. Do you have any metrics other than "how many online for 60 seconds?" can this time window be increased to 10/20/30/60 minutes? Thank you! - WMF".
- So, I think we should contact them and I hope they would be happy to help us, but the more time that goes into contacting them, the more we risk them not providing these info to us. Valerio Bozzolan (talk) 15:09, 15 December 2021 (UTC)
- If you're asking in 'what was the distribution of participation time across participants on Remo', that would probably better answer your question than changing the definition of a 'participant'. I believe it was hard to answer this with specific numbers due to privacy concerns, but there is the section The drop-off numbers within the first 20 minutes of the conference are significant, even in the absence of reliable comparative virtual conference attendance standards which is probably as clear a signal as you can get. Effeietsanders (talk) 17:57, 14 December 2021 (UTC)
- Yeah! the scenario you described is exactly what it could be useful to understand: the bounce rate (TL;DR how many people look and go away immediately). I understand it's not an happy information to share but it's a transparent and constructive one. For example, from the charts of the Italian wiki conference 2020 online we estimated 23 thousand apparent participants but "only" 310 users online for more than 15-30 minutes. See the itWikiCon 2020 data table. Do you think we can we share similar information for Wikimania 2021, to help other organizers? --Valerio Bozzolan (talk) 15:31, 14 December 2021 (UTC)
- @Valerio Bozzolan: I guess we could debate this for quite some time - but this seemed a convenient bare minimum to work with. I consider this equivalent to someone registering into the conference, collecting their badge, paying the admission fee and then after seeing the lunch buffet straight walking out of the venue. We would then also still count that person as a 'participant'. I think what we tried to communicate with this sentence, is that people who only watched via YouTube, are not covered under this definition (which is more problematic to me, but unfortunately the data sometimes forces you to make hard choices!). Effeietsanders (talk) 21:52, 10 December 2021 (UTC)