Council of Technical Groups

Hi HFES Community!

On Wednesday, we had a meeting between the TG Chairs and the ASPIRE Program Leads to discuss the feedback from the submission and review process for ASPIRE 2024. After discussing that feedback, we reviewed the risks for different options to address those concerns and to improve on the structure from last year. The following are the meeting notes from that meeting.

If you would like to give the ASPIRE planning committee additional feedback, please attend their office hours! They have an open feedback session every Friday from 12-1pm.

Warm regards,

Katie Tippey
COTG Chair

 
Review of Feedback on the ASPIRE Submission and Review Process

There were two major changes strategically chosen by the ASPIRE Program Committee last year that were focused on increasing the accessibility of our conference (and our great research!) to a broader community:

(1) Change from a proceedings paper submission to an abstract: the goal of this was to make it easier for non-academics and those seeking to publish journal papers on their work later to make submissions

(2) Change to using keywords instead of TGs when submitting: the goal of this was to make it easier for those unfamiliar with TGs to make submissions and increase collaboration across TGs

We did have substantially more submissions and thus a correspondingly higher rejection rate for ASPIRE 2024 than in previous conferences (only so many submissions can be accepted given the space we have available so more submissions will result in a higher rejection rate). As these were related to most major conference KPIs, the change resulted in positive outcomes for those metrics.

This was a high-level overview of the constructive feedback received about these conference changes from all of the listening sessions and surveys conducted during and after the conference. This feedback was used to develop initial ideas for how to improve the conference from our first year using abstracts. Those are detailed in the proceeding section.

(1) Change from a proceedings paper submission to an abstract: concerns about the following were stated
     · Shorter submissions make it difficult for reviewers to determine the scope and quality of the research
     · Many reviews of submissions appeared to be of lower quality than in the past
Proposed root cause from the TPC: maybe there is not enough guidance on how to scope an abstract to 1250 words; maybe reviewers don't have a strong enough incentive to provide more detailed reviews for abstracts
Proposed root cause from the COTG Chair: abstracts may not allow enough content or flexibility; maybe the review forms are tailored too much to proceedings papers

(2) Change to using keywords instead of TGs when submitting: it was difficult for program chairs to determine which keywords were most relevant to them
Proposed root cause from the COTG Chair: keywords were not tied to TGs when they were created

The additional question posed from this discussion was, "are we using the correct KPIs for the strategic vision of HFES?"

Discussion of Proposed Submission Process Updates

The initial ideas for updates to the submission process were discussed using the three parts of the submission process: (1) submitting content, (2) bidding process, and (3) review of submissions. Meeting attendees were given 5 minutes to write all of the risks they saw in each proposal on a Miro board. The group then openly discussed the risks in each proposal. The main concern highlighted throughout this discussion was that by trying to expand accessibility to our conference, we may push away our long-standing members and then neither group will benefit.

(1) Submitting Content: this discussion centered on whether a 1250-word abstract is enough to effectively communicate complex academic studies
     · Proposal 1: replace the text box entry with a file upload that has a global 1250-word limit but still requires all 5 sections -> the primary concern held by some is that this word limit may not be enough to express mature research. As the discussion progressed, many indicated they were uncertain about how much of a limitation the word limit actually imposed on mature research and that they were open to trying an abstract submission again as long as the TPC attempts to iterate and improve from our starting point last year

TG Chairs discussed reconsidering allowing for multiple types of submissions (i.e., full papers or abstracts). This prompted a discussion about what it means to have equitable submissions across all types and depths of research.

     · Proposal 2: allow the submission of images -> the primary concern was to ensure images were "packaged" with the files so that all the content downloads at once

Additional requests: HFES should share exemplar submissions from last year as examples

(2) Bidding Process
     · Proposal 1: allow TGs to select "n" keywords each to put on the keyword list so that they already know which keywords they would start filtering by after receiving submissions -> the primary concern was that TGs would choose the same keywords and having a longer list of keywords doesn't necessarily solve the issue of TG categories

As COTG Chair, I have observed even in the purpose statements published on the website, that we have an overlap among content in the TGs where it is confusing to know exactly what differentiates them (I did do a mapping on Miro to identify the similarities and differences!). My response to this concern is that having like keywords should indicate that those TGs should collaborate in their program.

(3) Review of Submissions
     · Proposal 1: add training for reviews -> the concern is that this is another barrier to getting reviewers and that people won't complete the training; TG Chairs recommended providing guidance and examples on what a good review is
     · Proposal 2: offer reviewers a conference discount for completing a certain number of reviews -> the concern is that people will still give low-quality reviews to reach the number required for the discount
     · Proposal 3: change the structure of the reviewer form to foster a more comprehensive review: there were no comments on this [yet]; from the previous comments, this could include giving examples of what a good review would include on the form