TWE: Research: Checklist for Rapid Qualitative Analysis (RQA)

[dialectic51]

Dependency

• TWE: RP001: Interview Recordings: Transcribe #324 transcription needs to be de-identified

Overview

We need to develop a step-by-step checklist for conducting Rapid Qualitative Analysis (RQA) at HackforLA. This checklist will be used to reanalyze data from the RP001: IS21 Internship Exit Interview. This checklist can also be used as the basis for developing a Template for conducting RQA and the starting point for creating a Guide.

Action Items

• Review information collected in Issue # 330 (link in Resources)
  • Review Issue # 335 as an example of a more elaborate checklist, but one that can still serve as a model for this one
• Based on information collected in Issue # 330, write a comment to this issue that provides a step-by-step checklist for conducting RQA
• Link checklist to the resources/instructions sections below
• Have Linda Angela and Dewey review the checklist and provide feedback
  • Make any needed changes to checklist based on Linda's feedback
• Create a template of the Summary Template
  • Save template of Summary Template to google drive folder
  • Place link to template of Summary Template in Resources/Instructions
• Create a template of the Matrix
  • Save template of Matrix to google drive folder
  • Place link to template of Matrix in Resources/Instructions
• Share updated checklist with lead analyst (Angela D.) Dewey Tran for review
• Create a new issue titled below
  • TWE: Research: RP001: Data: Reanalyze
  • Add links the same links to Resources/Instructions that are listed on this issue
  • Place in New Issue Approval column
• Create a new issue in UI/UX Templates and Guides titled 'Create a Guide/Template: Rapid Qualitative Testing'
  • Add the same links to Resources/Instructions that are listed on this issue
    • Remember that a document has already been created for the guide
  • Place in New Issue Approval column

Resources/Instructions

1. Wiki Page: RP001
2. TWE: Research: Guide Creation: Interview Expert (Linda) #330
3. Folder on UI/UX CoP Google Drive for Rapid Qualitative Analysis
4. HfLA Template on Making a Guide
5. UI/UX Guide for How to Conduct Rapid Qualitative Analysis
6. Linda's Presentation to UXR Team on How to Conduct Rapid Qualitative Analysis
7. Transcript of Hamilton's Presentation on RQA
8. Free Computer Assisted Qualitative Data AnalysiS (CAQDAS) programs
9. Potential domain names and categories for Internship RP01
10. Checklist for conducting Rapid Qualitative Analysis (RQA)
11. Template for Summary Template for RQA
12. Template for Matrix Analysis for RQA

[dialectic51]

NOTE: Compare this categorization of names with the categorization already conducted in the previous thematic analysis of RP001, which can be found in the google drive.

Here are potential domain names and categorization for the interview questions for Internship RP01. They can be used once the lead analyst creates a summary template for RQA (see the checklist created for this issue). The domain names and categorization are based upon the advice provided by Linda in her presentation on RQA.

Domains

Role and responsibilities

• 1 Describe your role and contribution.

Learned skills and discovery

• 2 What softwares or technical skills did you learn or practice?
• 10 Did you have an ‘aha’ moment at Hack for LA? If so, when?

Problem-solving

• 3 What did you do when you encountered a problem in regards to your knowledge of the subject matter?
• 7 How did you mitigate issues like unavailability of your peers and mentors?

Engagement and accountability

• 6 How did you keep your peers informed about your progress/issues?
• 8 How did you document your work?

Social networking

• 9 How did you extend or build your network at Hack for LA?

Pain points

• 11 What was missing or disappointing - how would you improve it?

Choice of HfLA

• 12 Were you offered more than one internship? Why did you choose our program over other options?

Prior expectations

• 14 What were your expectations with the internship program at Hack for LA before you applied? After the AMA? After the onboarding?

Recommend, endorse, volunteer

• 15 Would you recommend the internship program at hack for LA to your peers?
• 17 If time was not a constraint, would you like to be part of the community going forward?

Don’t require qualitative analysis

• 4 How often did you get to interact with your peers and mentors?
  • Could also be placed in “engagement and accountability” domain
• 5 How regularly did you attend the weekly meetings?
  • Could also be placed in “engagement and accountability” domain
  • (Could potentially code this—but unclear if this qualitative analysis would be that useful, but some pain points are identified, for example)
• 13 Which of the hack for LA volunteers was instrumental to your success at hack for LA?
• 16 Which feature did you learn from and enjoy the most? One-on-one, LinkedIn workshop, Standup, team meets.

[dialectic51]

Overview

We need to conduct Rapid Qualitative Analysis (RQA) of interview transcripts or other qualitative materials for RP[Replace 000].

Action Items

• 1.) Develop a Research Plan

  • Define problem statement (project overview and background)
  • Identify objectives (what you want to learn)
  • Choose appropriate research method
    • Determine if Rapid Qualitative Analysis (RAQ) is an appropriate method
      • Data type is qualitative
      • Using semi-structured or structured data
      • Research need is action-oriented (answers "how" questions about processes and skills)
      • Can use explanatory, deductive approach
  • Recruit participants
  • Construct an interview guide
    • Introductory Script
      • Introduce yourself (name; role in project)
      • Obtain informed consent (before asking any questions). Includes providing information on:
        • Purpose of the study
        • How data will be used/what products will result from study
        • How participants confidentiality will be protected
        • Study contact information
        • Emphasize participant's right to withdraw
        • Obtain signature of participant indicating informed consent
    • Interview questions (semi-structured or structured)
    • Probes (if interview is semi-structured)
    • Closing script
      • Thank participants for their time
      • Remind participants of study contact information
    • Include interview guide as appendix to research plan
  • Develop an analysis plan
    • Type of analysis (thematic, content, narrative, etc.)
      • (RQA will be thematic)
    • Decide on use of internal and/or external coding (internal coding, also known as inductive coding or conventional content analysis, involves identifying codes that arise from the data; external coding, also known as deductive coding or directed content analysis, involves the researcher creating codes beforehand)
    • Decide on which software tools to use (should use open source tools like google sheets at HfLA, but should be aware of other programs like NVivo, MAXQDA, Atlas.TI, Dedoose, RQDA, Excel, etc.)
    • Identify the lead analyst and the research team
      • Put together plan for training research team
        • Create training guide
        • Schedule training session
    • Estimate timeline for conducting the analysis (including when you will collect data, how often, and logic behind timing)
    • Identify who will receive the data
    • Identify how will you tell the stories about your data

• 2.) Collect data/conduct interviews

  • Will interviews be conducted remotely or in person?
    • Identify technology to be used
    • Test technology prior to interview
  • Identify how interview recordings (and transcriptions) will be stored
  • Identify how interview transcripts will be deidentified (if necessary)

• 3.) Transcribe and read interviews

  • If doing RQA, also have option of just listening to recording at taking notes
  • Use Otter.ai for automatic transcription
  • Deidentify transcripts (if necessary)

• 4.) Create a summary template for use by the team in a google doc

  • Create a neutral domain name that corresponds to each interview question
    • If needed, combine questions into the same domain, but domains should mostly line up with interview guide (i.e., stay in order)
    • Draft a summary template for use by the team (see the example on slide #23 of Hamilton's presentation)
      • Save draft summary template in the appropriate google drive folder
    • Include space at the end for "Other Observations," material that that is important but doesn't fit into the domain
    • Include space at the end for important quotations
    • Information under each domain should consist of bullet points that give a general sense of what was discussed (i.e., a minimally interpretive process)
    • If no information for a given domain, indicate why (Was the question asked? Was the question answered?)

• 5.) Take summary template for a "test drive" and assess its usability and relevance

  • Review with research team the characteristics of a good summary:
    • Brief (no more than 2 pages)
    • Organized
    • Thorough (major points captured)
    • Readable (anyone reading should get a sense of what respondent said; list bullet points about the key domains)
    • Useful (e.g., provides pointers for what's in the transcript)
  • Have team members use template on same subset of transcripts
  • Assess template:
    • Domains are intuitive and "findable" in the data
    • Identify any missing or incorrectly labeled domains
    • Easy to use
    • Measure time to complete (45-60 min/interview transcript)
      • Inform the team if you are taking longer (probably doing too much thinking or interpreting -OR- template isn't working as planned)
  • Compare summarizing styles across team. Assess for:
    • Similarities/differences in volume of information per domain
    • Use of direct quotes (should be minimal and concise; or paraphrase, especially long or complicated answers, then provide direct quotes at bottom of summary)
    • Notes on absence of content (e.g., question wasn't asked; or question asked, but not answered)
    • Notes on depth of a particular domain (e.g., lots of data on a topic, greate quotes, etc.)
  • Ensure that consistency has been established across team (if not, revise summary template and take on another "test drive")
  • Develop a codebook (if necessary)

• 6.) Divide transcripts across team and summarize (can divide by site, role, etc.)

• Track who has which transcript to review

• Post completed summaries to appropriate spot on shared google drive

• 7.) Transfer summaries into a matrix (respondent x domain)

  • Create a file using Google Sheets
    • Save file to appropriate spot on shared google drive
  • Develop templated summary of each data collection episode (answer to an interview question, with list of domains and space for the unexpected)
  • Set up matrix in format that makes sense for purpose of analysis (e.g., by site, by role, by data collection wave, etc). (See example of a matrix on Slide #33 of Hamilton's presentation) Research lead then decides whether to:
    • Either fill in data into a single worksheet in a spreadsheet workbook -OR-
    • Break data into multiple worksheets in a spreadsheet workbook (e.g., broken up by interview site, participant role, data collection wave, etc.)

• 8.) Conduct matrix analysis

  • Quickly peruse content of a domain to get a sense of variation
  • Assess gaps in information and why they exist
  • Develop analytical memos (your own thoughts on the data and the themes that are emerging)
  • Develop summaries of domains, sites, types of respondents, etc.

• 9.) Identify themes within domains

  • Look for trends, themes, and variations (e.g., positive or negative)

• 10.) Prepare report/presentation

[dialectic51]

@bradyse I wrapped up issue #330 and created this (issue #338) in order to put together the checklist for conducting rapid qualitative analysis. I have completed the checklist in this comment in this issue.

When you have time, could you take a look and make sure I'm doing what we discussed at last week's meeting? If it looks like I'm on the right track, then I will contact Linda, have her review the checklist, and then make changes based on her feedback. We can also discuss this at our meeting tomorrow.

[bradyse]

@dialectic51
This looks good. Go ahead and send the comment link to Linda. Ask her to quickly look through it and determine if the basic steps are there. If there are any glaring errors, she can point those out. Otherwise, I wouldn't expect her to spend more than 20 minutes or so reading through the whole thing.

[dialectic51]

@lkawentel
I have completed a checklist for conducting Rapid Qualitative Analysis (RQA), which I have placed in the comment above. I have based the checklist on your answers to Sara's questions, your presentation, and the materials your presentation links to.

When you have time, please review the checklist and identify any errors or things that need to be added to the list. Sara says this shouldn't take more than 20 minutes or so.

(I have also put together a list of potential domains based upon your presentation. I'm going to continue to revise them as I gain more expertise in RQA, but feel free to look them over if you have the time).

[dialectic51]

@bradyse

I have put in a request (via github and slack) for Linda to review the checklist.

I also made a few revisions to the checklist to address the points you raised at yesterday's meeting. This includes:

• Making the construction of the interview guide as part of the overall development of a research plan.
• I also broke the section on creating a summary template (step 4) into two sections (now steps 4 and 5).
• And I clarified the question I had listed about whether or not to use a single worksheet within a spreadsheet workbook or multiple worksheets within a spreadsheet workbook. I just made this into an option at the end of step 7.

[dialectic51]

Update on RQA for RP001: I have created templates for both the Summary Template and the Matrix Analysis to be used once we start conducting the RQA (located in this folder of the Internship google drive).

They are based off of slide #23 and slide #33 of Hamilton’s presentation and used in steps 4 and 7 in the RQA checklist. I have placed links to the templates in the instructions/resources section of issue#338. (Note: these are templates, but I haven’t been able to figure out how to turn them into actual templates in google docs and sheets).

The summary template was pretty straightforward from Hamilton’s slides. However, I made some changes to the Matrix in order to bring in two additional columns next to each domain: 1.) analytical memos and 2.) themes and trends. (Constructing analytical memos and identifying themes and trends are key steps in conducting a matrix analysis).

I have also placed in bullet points in each cell under the domain columns. The bullet points in the Matrix should be taken directly from the bullet points that will be created in the Summary Templates by a coding team. However, bullet points are a little wonky in Google Sheets, so I have a link to directions on how to create them at the bottom of the document.

(This note was also sent via Slack to Sara, Linda, and Angela).

[pandanista]

@dialectic51 Hi Brian! Welcome back from the break. We are confirming everyone's interest and availability. Please provide a new update to this issue you have been working on. Thank you.

Please provide update:

1. Progress: "What is the current status of your project? What have you completed and what is left to do?" - we know you just came back from break, please just summarize the progress from before then
2. Blockers: "Difficulties or errors encountered."
3. Availability: "How much time will you have this week to work on this issue?"
4. ETA: "When do you expect this issue to be completed?"
5. Pictures or links* (if necessary): "Add any pictures or links that will help illustrate what you are working on."

• remember to add links to the top of the issue if they are going to be needed again.

[dialectic51]

@pandanista I hope you had a good break!

1. Progress: I have finished the checklist for the RQA for RP001 (see comment above dated 11/25). I have also completed the templates needed for the RQA, specifically the "Summary Template" and "Matrix Analysis" documents (see comment above dated 12/2). I met with Sara on 12/14 and she helped me turn these into genuine Templates (in line with HackForLA standards), place them into the appropriate place on the google drive, and link them on the Wiki for RP001. The next steps will be to:
   -a.) create a Guide for conducting RQA based on the checklist I've created;
   -b.) break the different steps of the RQA checklist into issues (and create an Epic issue to organize them all); and
   -c.) conduct the RQA for RP001.
   -(Later on, the RQA checklist will be turned into a Template).
2. Blockers: Once Angela returns from the break, she will look over the checklist and give me feedback if she has any. We will then begin the RQA with Angela serving as the UX lead.
3. Availability: As needed.
4. ETA: I'm hoping to complete the Guide (at least a rough draft) and create the Issues next week; we would then be able to begin the RQA in the week starting 1/16

[dialectic51]

@Dewey-Tran To prepare you for reviewing the checklist, look over Linda's Presentation to UXR Team on How to Conduct Rapid Qualitative Analysis and the Transcript of Linda's Presentation on RQA, which are also located under the Information/Resources section in the Overview. These are the documents that I based my checklist on.

[Dewey-Tran]

I had reviewed the list @dialectic51 created, here are my suggestions of tasks to include.

1.) Develop a Research Plan

• Ensure data collection procedure is an iterable process

2.) Collect data/conduct interviews

• Keep note of significant events interviews (

8.) Conduct matrix analysis

• Provide suggestions for future analyses

[pandanista]

@adarosh I am assigning you to review the checklist above. Let me know if you have any questions. Thank you!

[pandanista]

• Need to assign a team member to revise the checklist above.

• Here is an example from RP017's checklist.

[jinyan0425]

@pandanista I reviewed all the comments and related issues. Before I start revising the checklist, I think I need to know more about the context and objective of this task. We can discuss this in today's UX meeting.

A minor issue is that I do not have access to the following folders/docs:

1. Folder on UI/UX CoP Google Drive for Rapid Qualitative Analysis
2. UI/UX Guide for How to Conduct Rapid Qualitative Analysis

[pandanista]

Hi @jinyan0425, I just found the instructions to be added back to the UI/UX COP Google Drive. See detailed info here.

[jinyan0425]

Hi @jinyan0425, I just found the instructions to be added back to the UI/UX COP Google Drive. See detailed info here.

@pandanista Thanks, Ying! Just filled out the form.

[jinyan0425]

@pandanista

1. I read through the transcript, which is not the transcript of Linda's presentation but that of Hamilton's presentation back in 2013. So, the label of this resource is incorrect: Linda's Presentation to UXR Team on How to Conduct Rapid Qualitative Analysis. We may need to correct it. On the other hand, it will be very useful if we can find the actual transcript for Linda's presentation.

2. I would like to confirm we will fully follow Hamilton's Process/Approach, which relies on verbatim transcript instead of identifying themes directly from audio recordings. Hamilton's Process/Approach is an early approach to RQA. See the following expert from a recent method paper on RQA -- Nevedal et al. 2021. This paper was also cited in Linda's presentation on pp.10:

Hamilton developed a rapid qualitative analysis approach that summarizes transcript data into templates using domains aligned with interview questions; summary points are then distilled into a matrix organized by domain and participant for analysis and interpretation [18].

The rapid approaches described by Hamilton and Gale et al. rely on verbatim transcripts, which means teams must wait for transcription to be completed to proceed with rapid or traditional analyses. In contrast, Neal et al. [28] developed an approach to rapidly identify themes directly from audio recordings.

[jinyan0425]

Draft of Rapid Qualitative Analysis (RQA) Checklist

---

@pandanista This is tentative and developing. I made some comments about the draft at the end.

1. Get Familiar with the Hamilton's Process of RQA

• If you are new to RQA, review the following methodology documents to familiarize yourself with Hamilton's Process:
  • [MUST READ] Qualitative Methods in Rapid Turn-Around Health Services Research
  • Linda's Presentation to UXR Team on How to Conduct Rapid Qualitative Analysis
• If you are experienced with RQA, you may skip reviewing the documents or review them for a refresher.

2. Create the Transcript Summary Template

• Review all interview questions (not transcripts) and create a neutral domain name that corresponds with each interview question (an example refers to p. 22 in the notes of Hamilton's Process/Approach)
• Draft the summary template (an example refers to p. 23 in the notes of Hamilton's Process/Approach)
  • Include basic information about the interview at the top
  • Include the neutral domains created above, and Other Observations for anything that doesn’t fit into any domain,
  • Include space for important quotations at the end
• Test the summary template
  • Test the summary template on a subset of transcripts
  • Assess the quality of the neutral domains created
  • Compare the summarizing "styles" across the team
  • Measure the time of completion and control for optimal summary length
• Revise the template until consistency has been established
• [Optional] Create an instructional handout for transcript summaries and distribute it to the team

3. Conduct Transcript Summaries

• Distribute transcripts across the team. The division of transcripts should be customized to suit research needs.
• Complete the summary for each transcript (an example refers to p. 27 in the notes of Hamilton's Process/Approach)
• Check the quality of the summaries

4. Conduct Matrix Analysis

• Create a matrix
  • The matrix format should be customized to suit research needs. The most commonly used format is respondent * domain (an example refers to p. 32 in the notes of Hamilton's Process/Approach).
• Transfer the summaries into the matrix
• Analyze the Matrix. The objective and method of matrix analysis should be customized to suit research needs. Common options are:
  • Develop a codebook and conduct code analysis
  • Identify themes and conduct thematic analysis
  • Create memos to inform future data collection
  • Evaluate the data collection quality (if the matrix is in the format of interviewee * domain)

5. Prepare reports/presentations

---

COMMENTS:

1. Many researchers may not be familiar with RQA, especially Hamilton's Process. Therefore, I think the Step should be to become familiar with this method. In Step 1, I do NOT want to mention Nevedal et al., 2021, which could potentially confuse researchers if we exclusively focus on Hamilton's Process (see my comment above).
2. Steps 2 and 3 can be combined as Conduct Transcript Summaries.
3. The sub-points in each step may still be too detailed. If so, we can remove some of them.
4. On the other hand, RQA is quite complex, with many nuances even at each lowest-level step in the checklist. For instance, there are detailed instructions on how to "Compare the summarizing 'styles' across the team". If we remove this bullet point from the checklist, we may still need to reintroduce all these details as action items in the actual data analysis phase/issue (e.g., TWE: Research: RP017: Data: Analyze #493)
5. Another challenge is the high flexibility of RQA, where many steps need to be customized for specific RPs. This requires careful construction of the actual data analysis action items tailored to each RP.

[pandanista]

@pandanista

1. I read through the transcript, which is not the transcript of Linda's presentation but that of Hamilton's presentation back in 2013. So, the label of this resource is incorrect: Linda's Presentation to UXR Team on How to Conduct Rapid Qualitative Analysis. We may need to correct it. On the other hand, it will be very useful if we can find the actual transcript for Linda's presentation.

• I don't think we have the actual transcript from Linda's presentation because it was not recorded.
• Resolved.

2. I would like to confirm we will fully follow Hamilton's Process/Approach, which relies on verbatim transcript instead of identifying themes directly from audio recordings. Hamilton's Process/Approach is an early approach to RQA. See the following expert from a recent method paper on RQA -- Nevedal et al. 2021. This paper was also cited in Linda's presentation on pp.10:

Hamilton developed a rapid qualitative analysis approach that summarizes transcript data into templates using domains aligned with interview questions; summary points are then distilled into a matrix organized by domain and participant for analysis and interpretation [18].

The rapid approaches described by Hamilton and Gale et al. rely on verbatim transcripts, which means teams must wait for transcription to be completed to proceed with rapid or traditional analyses. In contrast, Neal et al. [28] developed an approach to rapidly identify themes directly from audio recordings.

• Since our transcripts are mostly ready, I think we could follow Hamilton's Process/Approach.

[pandanista]

COMMENTS regarding RQA's updated checklist:

• Many researchers may not be familiar with RQA, especially Hamilton's Process. Therefore, I think the Step should be to become familiar with this method. In Step 1, I do NOT want to mention Nevedal et al., 2021, which could potentially confuse researchers if we exclusively focus on Hamilton's Process (see my comment above).
  • I agree if we follow Hamilton's approach.
• Steps 2 and 3 can be combined as Conduct Transcript Summaries.
  • It might be easier to leave them separated as step 2 is for set-up and testing and step 3 is for summarizing all transcripts.
  • From the operational standpoint, not every assignee needs to create and set up the summary template, etc.
  • Here is what I am thinking regarding the process: The assignee who got assigned to work on the analysis first, they need to create a summary template (maybe a matrix too) and do a round of test. Then, another team member can do a peer review. Lastly, we use the updated template/checklist and distribute to the team so other team members can work on the summary table and matrix.
    • Looks very similar to what you drafted. Let me know your thoughts as well.
    • This is specific to RP001: there might not be an existing interview guide in the folder, so we need to list out the interview questions from transcript.
• The sub-points in each step may still be too detailed. If so, we can remove some of them.
  On the other hand, RQA is quite complex, with many nuances even at each lowest-level step in the checklist. For instance, there are detailed instructions on how to "Compare the summarizing 'styles' across the team". If we remove this bullet point from the checklist, we may still need to reintroduce all these details as action items in the actual data analysis phase/issue (e.g., TWE: Research: RP017: Data: Analyze #493)
  • Good point. Maybe we should leave them as detailed for now. I wonder if we could remove the detailed parts and have all the detailed information in the RQA guide that will be based on our initial set of analyses (RP001). But for our initial set of analyses (RP001), we don't have the guide ready yet.
• Another challenge is the high flexibility of RQA, where many steps need to be customized for specific RPs. This requires careful construction of the actual data analysis action items tailored to each RP.
  • What kind of customization are your referring to?

[pandanista]

@sunannie27

• Please review the checklist. Let me know if you have any questions.

• If the checklist looks good, the research leads will use that to make RP001's data analysis new issue.

[KC-skc]

(not transcripts_) Is this a typo? Are you trying to italicize as well as bold?

[jinyan0425]

(not transcripts_) Is this a typo? Are you trying to italicize as well as bold?

YES, KC! Thanks for catching it. Edited.

[sunannie27]

@jinyan0425 @pandanista Checklist looks good. Will kickback to research leads.

[pandanista]

• Closing this issue since it is signed off and we copied the check list to the data analysis issue TWE: IS21: RP001: Rapid Qualitative Analysis (RQA): Pre-process #616.