by Matthew Venaas
D
uring the past two years, the ongoing pandemic has accelerated equity gaps while bringing increased visibility to existing inequities, including those in higher education. More than ever, institutions and leaders are embracing the challenge and striving to correct these inequities. Doing so will require all areas to collaborate and commit, including assessment.
Assessment can and should be an important part of this process. It is a concept that is gaining recognition and increased attention from groups such as the University of Southern California’s Center for Urban Education and their Racial Equity Tools or the Urban Institute’s Do No Harm Guide. Good assessment can identify and raise awareness of existing inequities, provide key insights to inform changes, and clarify how those changes affect students. But the process of integrating equitable and inclusive practices into assessment does not occur spontaneously. It is an ongoing and extensive process that includes a wide range of options that come with competing interests, such as the value of transparency versus that of privacy. No single solution is likely to address all concerns. Even the most well-intentioned professionals can struggle to figure out where to start.
For housing professionals considering the role they play in fostering inclusive and equitable assessment practices, there are a handful of guiding principles to keep in mind. The first is to know the purpose of the assessment and then to stay true to it. As with any good assessment effort, a clear understanding of the purpose is needed: What is it that the survey is designed to accomplish? How does that fit with the goals of the department and institution? An assessment project with a clearly defined purpose is more likely to stay on track to successful completion. Additionally, assessment projects clearly linked to broader initiatives like strategic plans or institution-wide goals are more likely to garner support from other stakeholders on campus.
A second concern is to establish a balance between transparency and privacy. A significant focus of equitable assessment centers on transparency, in particular disaggregating data in order to better understand the challenges, experiences, and needs of underserved populations. However, increased disaggregation of data increases the risk of individually identifying participants and putting their confidentiality or anonymity at risk. All assessment efforts will need to strike a balance between transparency and privacy, as these needs will almost always compete against each other. It’s also important to periodically reflect on the potential benefits and risks of being more transparent or less and to discuss trade-offs not only with the administrators and decision-makers who will leverage the data but also with the people reflected in the data that would be shared.
Consider the assessment process as a whole. It can be easy to get focused on one aspect or stage in the process, for instance reviewing surveys to ensure that language is inclusive. But bringing diversity, equity, and inclusion from the periphery and truly embedding it into assessment requires an intentional and comprehensive approach across the full process, from planning to sharing results. After all, results from the best survey will mean little if the reports we shared aren’t accessible to the audiences or if the insights generated don’t fundamentally align with the purpose and resources.
While many of the above considerations are macro statements of strategy and purpose, there are also micro actions to help achieve equity-minded assessment in practice. For those looking to make a difference in their own practices (or those of their department or institution), here are seven practical strategies for making assessment work in a more equitable and inclusive way.
Mix Your Methods: Every assessment method has strengths and weaknesses. Quantitative methods allow for quicker analysis but may hide the experiences of smaller groups. Qualitative analysis brings individual stories and voices into the process, but sometimes at the expense of understanding the issues and experiences of a broader community. Combining different assessment methods in a strategic and intentional manner can address some of these concerns. One approach could be to start with a quantitative method like a survey and then follow up with a qualitative study like focus groups or interviews. The opposite can work as well; an initial qualitative study can help to better understand a broader issue and inform a larger follow-up survey.
A thoughtful mixed methods approach can leverage the power of large surveys while digging deeper to better tailor solutions. For instance, Texas Christian University used the results from their ACUHO-I/Benchworks Resident Assessment to identify learning related to diverse interactions as an area of focus for their department. They generated diversity rubrics, focus groups, and follow-up research in order to better understand the issue and generate actions. This approach ultimately led to the creation of a new student position to aid in multicultural programming and intercultural relationship development.
Include Students as More Than Subjects: College students can and should be more than just subjects of assessment. By engaging students in the broader assessment process, organizations not only ensure that their work is more representative of the students being served but can also help to uplift voices of historically marginalized or underserved communities. As an example, if there are concerns about specific language used in a survey, consult with various groups to achieve a cross-section of the campus. Terms and categories, especially those related to underserved or underrepresented populations, are nuanced and evolve quickly. Have students review or suggest feedback on the survey questions themselves so the options reflect their community. Student input should go beyond word choices. As Darby Roberts, the director of student life studies at Texas A&M University, explains in a blog post, there are a number of ways to include students in most every step of the broader assessment process, including determining outcomes, planning the assessment’s delivery, analyzing the data, and sharing assessment results.
Use Digital Tools to Include More Voices: The pivots to virtual environments due to the COVID-19 pandemic resulted in the proliferation of new tools to aid in learning and engagement. These tools not only created new opportunities to collect data but can also be a mechanism for more equitable access and the inclusion of more voices in our assessment efforts.
In a recent webinar, experts in educational delivery methods Emily Ravenwood and Edna Ross expanded upon strategies for better leveraging digital tools as a mechanism for fostering equity and inclusion. In particular, their lessons related to technology and engagement are directly applicable to programming and engagement outside the classroom. They noted that within the context of differences based on participants’ gender, personality type, culture, and language proficiency, using anonymous polling as an engagement strategy can eliminate barriers to participation and close gaps in engagement rates across different groups. Engagement activities that include varying degrees of group activity, reflection time, and asynchronous work all have the potential to more equitably allow students to engage in and demonstrate learning both inside and outside the classroom.
Bringing diversity, equity, and inclusion from the periphery and truly embedding it into assessment requires an intentional and comprehensive approach across the full process.
Ask Purposeful Demographic Questions: Demographic questions as part of an assessment enable analysis that can identify specific challenges and issues based on the characteristics and background of the community. They are also an essential component of identifying equity gaps. But this doesn’t mean asking every demographic question every time. Not every demographic question is necessary for every assessment. Consider which questions are necessary within the purpose of the assessment, think through how that question will contribute to analysis and how it will enable action, and avoid asking or adding demographic questions purely out of personal interest or curiosity. This is particularly important when considering that some questions might make participants uncomfortable or hesitant to respond. When possible, share with your participants the purpose of collecting the demographic information.
Review Assessment Language: Before launching any survey, take a careful look at the language. Ensure that categorical questions have an appropriate range of response options. Too few options can easily marginalize a respondent; however, too many options can not only confuse them but can also make analysis and practical use of the data difficult. Be sure to include an additional open-ended option for respondents who do not identify with a pre-defined category. When asking about identities, it is difficult, if not impossible, to create a perfect set of response options. Leveraging strategies like seeking feedback from student communities prior to launching an assessment and understanding the survey’s purpose can help guide the approach in crafting an appropriate and inclusive set of response options.
Additionally, review for any language that participants may not understand or experiences that may not relate to all respondents. Spell out acronyms that students might not be familiar with. Use care when asking questions about families, homes, or anything that may suggest that a participant’s situation is not being represented or that there are privileges the participant may not have.
Reviewing language extends to more than just the words themselves. When formatting, response options should be in alphabetical or numerical order instead of placing a majority or privileged category first. When applicable, respondents should be allowed to select more than one option to accurately reflect those with multiple identity or group memberships. Finally, allow respondents to skip demographic questions (or add some form of “prefer not to respond” option).
Dig into Data with Care: Disaggregating results is critical to identifying and starting to close equity gaps. Effective disaggregation can only take place, however, as part of a broader set of steps in the assessment process that includes asking the right questions, ensuring the accurate collection of data, and having mechanisms or plans to take action based on results.
Additionally, care must be taken regarding cases with a small number of respondents. On one hand, results involving a small number should not necessarily be generalized to a broader population. On the other hand, each and every part of that small number represents an individual student with individual needs and should not be written off purely because of a lack of statistical significance in a dataset. So, as with many things in assessment and the broader practice, it’s a balancing act.
Finally, ensure that the dataset is reflective of the community or population to be studied. If there’s not enough data, find ways to collect more through methods like interviews or focus groups so as to not make assumptions or risk sharing disaggregated data to the point of jeopardizing anonymity.
Ensure That Visuals Are Accessible: Inclusivity in assessment extends to how results are visualized. This particularly extends to the images, colors, and icons used in sharing results. For instance, approximately 4.5% of people are affected by some form of color vision deficiency, or an inability to distinguish between certain colors or to see any color at all. Certain color combinations (red and green, blue and green, blue and purple, or orange and green) are particularly problematic in visuals for individuals with certain color vision deficiencies. So avoid those combinations. Websites like Coblis allow the user to upload an image, such as a screenshot of a chart, to simulate how the image or visual is viewed by those with common color deficiencies.
Inclusivity with colors also extends to how colors are used to represent categories or groups in data visuals. In many cases, colors assigned to categories like race and gender can reinforce specific stereotypes and further marginalize individuals. One option for avoiding issues with color and comparisons would be to apply a small multiples approach that places individual categories in their own small visual as part of a larger group to allow for comparisons.
These strategies are not meant to be an exhaustive or authoritative guide to inclusive assessment, but they can serve as a great starting point or refresher for those who desire to embed equity and inclusivity into their assessment work. After all, assessment is about identifying and validating conditions to guide decision-making. Shouldn’t all those who will be affected by those decisions have their voices heard?
Matthew Venaas is a research manager for Skyfactor.