Post-COVID19 TPC experience

I do not participate in a huge number of TPCs (Technical Programme Committees) as a general rule– partly time constraints but mostly no-one knows who I am so I don’t often get asked… (!)

I have done a few though, some big (e.g., USENIX NSDI, ACM IMC), some small (UK Systems, PAM), and perhaps because I only do a couple every few years, while doing ACM CoNEXT and ACM/IEEE SEC this week, I found myself particularly noticing some changes in practice since that last TPCs I recall (notably NSDI and IMC). So here are three observations.

1. Online, first and only

The biggest obvious change is that TPC meetings are now online rather than in-person. This has one big disadvantage for me: I really enjoyed travelling to the meeting to meet colleagues and (usually) participate in some TPC-oriented workshop with presentations of recent an in-progress work. In many ways, I found this sort of activity more interesting than the conference itself (sorry!). It also has the unfortunate effect that, at least for international TPCs, timezones make scheduling tricky – one benefit of travelling was that at least the meeting took place in localtime for (almost) everyone.

However, it also has clear benefits: the CO2 footprint of the event is dramatically reduced which can only be a good thing. The financial cost reduction probably also opens up the experience to attendees who would never previously have been able to make it. I’ve rarely seen really poor meeting behaviour on the TPCs I’ve been involved in, but I also find that in an online meeting, chairing tends to be more easily more rigorous and the sometimes dominating effect of a single confident (perhaps I might say over-confident, or even just loud) individual is significantly reduced. Which is good.

2. Offline dominance

I now see dramatically more use being made of commenting and discussions in HotCRP, which remains the only conference management and paper reviewing platform I will willingly use (thanks Eddie!). I don’t know whether this is a post-pandemic effect, or just the fact that the old old guard has largely shuffled out of active TPC duties and we now have a new old guard (ie., people of my era) and younger who are perhaps happier to communicate and express views without needing to be in the same room.

I think this is another dramatic improvement in process, and puts us in a similar place to how, for example, we handled marking final year undergraduate projects at Nottingham University. It means that discussion is recorded, and usually more coherently and explicitly argued due to the need to write it down.

3. Reduction in extremism

The one arguably slightly negative comment I would have is – and this is only my anecdotal impression, and not something I can pretend to have data on – that I think I see a tendency for reviewers to perhaps be a little less clear in their scoring. Once the across the board strong rejects (a fair number) and strong accepts (a much smaller number) were taken out, I saw an awful lot of weak reject (but happy to consider weak accept after discussion) and weak accept (but happy to consider weak reject after discussion). This made it quite hard for me, as another reviewer, sometimes to get a clear signal as to what the other reviews were recommending.

This is a challenge I sometimes see in admissions interviewing: less experienced interviewers are sometimes reluctant to give a clear signal, exhibiting instead a tendency to score near or perhaps just above the middle of the scale. (I remember doing this myself.) This feels more comfortable – putting one’s head above the parapet by taking a clear stance often feels more socially awkward than giving a “fine, ok, pretty good, sure” response, not least because it needs stronger supporting argument – but in the end I think it misses the point of being an interviewer / a TPC member which is exactly to accept or reject papers based on having been recognised for your expertise.

So in the end, two good points, two potential (but minor) negatives, and one completely unjustifiable and purely selfish negative. Which on the whole is a good thing. Though having come back to this after a break for a couple of years, I find I am even more sceptical of the whole process than I was. More later perhaps, once I get my thoughts in order.